Compare commits

...

69 Commits

Author SHA1 Message Date
Oliver Kieselbach 1c3d29f30b refactor(agent): read WiFi info via native WLAN API instead of netsh parsing
netsh wlan show interfaces output is localized: "Radio type"/"Channel"
(and on some languages even "Signal") never matched on non-English
images, silently dropping fields from wifi_signal_info and network
change snapshots. It also spawned a child process per collection.

Replace both netsh call sites with wlanapi.dll P/Invoke
(WlanQueryInterface: current_connection + channel_number):

- New WlanNativeMethods (raw interop) + WifiInfoReader (fail-soft,
  prefers the active NIC's interface GUID, falls back to the first
  connected WLAN interface; returns null on machines without WLAN).
- DeviceInfoCollector.CollectWiFiSignalInfo and
  NetworkChangeDetector.CollectWiFiInfo now use the reader; RadioType
  strings stay netsh-compatible ("802.11ax" ...), event fields
  (wifiSsid/wifiSignalPercent/wifiRadioType/wifiChannel) unchanged.
- SystemPaths.Netsh removed (no remaining references).

V2.Core suite 2085/2085 green incl. new WifiInfoReader tests; struct
marshaling verified against native sizeof (604/68/36/532).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 13:20:12 +02:00
Oliver Kieselbach bb4accfe96 feat: API Latency as optional column in dashboard session list
Off by default; sortable via avgApiLatencyMs. Data comes from the
SessionsIndex mirror the list already reads — no extra query or table.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 13:11:20 +02:00
Oliver Kieselbach 0ec481b479 feat(agent): outcome-calibrated agent_late_start severity and phrasing
Fleet data (2026-07-28, agent 2.0.1380) showed the late-start detector was
outcome-blind: tenant 659c3a90 collected 12 Warnings with the post-mortem/
hung-script note on enrollments that succeeded — the bootstrap merely ran
late in the ESP queue and the agent observed the healthy final minutes live.

New pure helper ObservationCoverage.DescribeLateStart:
- Succeeded -> Info with a tail-only-coverage note (IME-replay hint,
  bootstrap-ordering review if recurring); no alarm framing.
- Failed/TimedOut -> Warning with the existing post-mortem/hung-script note.
- oobeStateAtAgentStart refines the message only: "completed" = pure
  post-mortem, "in_progress" = live tail observed; not_started/unavailable
  add no claim (value flips to not_started after mid-OOBE reboots and
  "completed" is routine on post-desktop restarts).

The coverage gate (10 min boot-to-start / 5 min uptime) is unchanged, and
the OOBE state is deliberately NOT a gate: the canonical hung-script
failure (f148976f) terminates on the still-shown ESP screen, i.e. with
OOBE reading in_progress.

8 new tests (outcome x oobe-state matrix); V2.Core suite 2067 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:53:17 +02:00
Oliver Kieselbach 3ec716a3ec feat: expose API latency to MCP field projection and tool docs
- SearchSessionsFunction fields whitelist += avgApiLatencyMs,
  apiRequestCount so search_sessions lean sweeps (e.g.
  fields=geoCountry,avgApiLatencyMs,apiRequestCount) can aggregate
  latency without full SessionSummary payloads
- search_sessions tool doc lists the new fields incl. the weighting
  hint; get_geographic_metrics doc names avgApiLatencyMs /
  apiLatencyVsGlobalPct as the backend-region distance metric

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:28:01 +02:00
Oliver Kieselbach b0bb7233e4 chore: drop Outbound IP row from Connectivity section
The outbound_ip trace event stays available for analysis (timeline,
MCP); only the UI row is removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:20:35 +02:00
Oliver Kieselbach 6758dc59e4 refactor: move API latency into Device Details "Connectivity" section
The extra grid cell in the Session Info card forced a fourth row once
Device Validation was present. The latency reading is a "when needed"
detail, so it moves into the collapsed Device Details card as a new
Connectivity section instead — together with values that were collected
but never displayed anywhere: the estimated internet bandwidth
(network_bandwidth_estimate incl. DO-throttle hint), total telemetry
upload volume (agent_metrics_snapshot), and the outbound IP.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:12:08 +02:00
Oliver Kieselbach ffd70e0383 feat: geographic API-latency view (per-location weighted avg by geo)
Surfaces the persisted per-session AvgApiLatencyMs in the geographic
views so operators can compare regions when deciding on new backend
locations (Asia/US):

- ComputeGeographicMetrics: per-location and global request-weighted
  latency (sum(avg*requests)/sum(requests)) + ApiLatencyVsGlobalPct;
  serves both the tenant and the GA cross-tenant endpoint
- GeoMetricsSessionProjection extended (column-projected scan would
  otherwise drop the new fields); equivalence test pins it
- Drilldown: LocationSessionRow carries the per-session values; sessions
  table + summary tile show them
- geographic-performance page: sortable "API Latency" column with
  absolute color buckets (<250 green .. >=800 red), global benchmark
- PlatformMetricsService: session AvgLatency now exact from cumulative
  counters when present (fallback: mean of interval averages)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:58:53 +02:00
Oliver Kieselbach c18154d76d feat: persist per-session agent-to-backend API latency (AvgApiLatencyMs)
The V2 agent already measures HTTP round-trip latency per request
(NetworkMetrics, cumulative _totalLatencyMs) but only emitted the
per-interval delta average. Operators deciding on new backend regions
(Asia/US) need the session-wide average, aggregatable by GeoCountry.
Measured baseline: Jakarta sessions average ~600-1050 ms client-side vs
94 ms server-side processing — the gap is pure network distance to GWC.

- Agent: agent_metrics_snapshot now carries cumulative net_total_latency_ms
  (total/requests = whole-session average; idempotent by construction)
- Ingest: last matching snapshot of a batch projects AvgApiLatencyMs +
  ApiRequestCount onto the Session row (fire-and-forget, fail-soft,
  last-write-wins; agents predating the field simply never match)
- SessionsIndex: both fields mirrored (merge + BuildSessionIndexEntity
  superset) so list/GA views can aggregate latency per geo
- Web: session detail shows "API Latency" when present

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:44:46 +02:00
Oliver Kieselbach f524473305 Merge branch 'main' of https://github.com/okieselbach/Autopilot-Monitor 2026-07-28 09:54:07 +02:00
Oliver Kieselbach 8fe420b902 updated commands 2026-07-28 09:53:57 +02:00
dependabot[bot] 2e8c2cdb2f chore(deps): bump @modelcontextprotocol/sdk (#110)
Bumps the mcp group in /src/McpServer/autopilot-monitor-mcp with 1 update: [@modelcontextprotocol/sdk](https://github.com/modelcontextprotocol/typescript-sdk).


Updates `@modelcontextprotocol/sdk` from 1.29.0 to 1.30.0
- [Release notes](https://github.com/modelcontextprotocol/typescript-sdk/releases)
- [Commits](https://github.com/modelcontextprotocol/typescript-sdk/compare/v1.29.0...1.30.0)

---
updated-dependencies:
- dependency-name: "@modelcontextprotocol/sdk"
  dependency-version: 1.30.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: mcp
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-28 09:44:13 +02:00
dependabot[bot] e04b75dd53 chore(deps): bump the web group (#111)
Bumps the web group in /src/Web/autopilot-monitor-web with 3 updates: [next](https://github.com/vercel/next.js), [@next/bundle-analyzer](https://github.com/vercel/next.js/tree/HEAD/packages/next-bundle-analyzer) and [eslint-config-next](https://github.com/vercel/next.js/tree/HEAD/packages/eslint-config-next).


Updates `next` from 15.5.21 to 15.5.22
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/compare/v15.5.21...v15.5.22)

Updates `@next/bundle-analyzer` from 16.2.11 to 16.2.12
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/commits/v16.2.12/packages/next-bundle-analyzer)

Updates `eslint-config-next` from 15.5.21 to 15.5.22
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/commits/v15.5.22/packages/eslint-config-next)

---
updated-dependencies:
- dependency-name: next
  dependency-version: 15.5.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: "@next/bundle-analyzer"
  dependency-version: 16.2.12
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: eslint-config-next
  dependency-version: 15.5.22
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-28 09:41:30 +02:00
Oliver Kieselbach d55b0ff864 test(backend): pin the cross-tenant redaction on metrics/ime-versions
GET metrics/ime-versions is MemberRead and the archive is one global
partition, so every tenant member reads the same platform-wide rows. The only
thing between an ordinary tenant user and another tenant's identifiers was an
inline projection with no test behind it.

There is no HTTP-level harness in this suite - nothing constructs an
HttpRequestData with a RequestContext - so the projection was unreachable from
a test while it lived inside Run. Extracted it to a static
BuildResponsePayload(versions, hasGlobalScope), matching the pattern
GetTpmPssUnsupportedFunction already uses for the same reason. Behaviour is
unchanged: the payload is returned as `object`, and System.Text.Json resolves
an `object` declared type via the runtime type, so both branches keep the wire
format they had inline. A test asserts that rather than trusting it.

10 tests, asserting on serialized JSON in camelCase (Program.cs's naming
policy) because a property dropped from an anonymous type is only a real
redaction once it is absent from the wire:

  - non-global: firstSeenTenantId / firstSeenSessionId absent, and their GUID
    VALUES absent anywhere in the payload (a model rename must not let the
    identifier survive under another key)
  - non-global: exactly {version, firstSeenAt, lastSeenAt, sessionCount}
  - non-global: sessionCount survives - the counts are platform-wide by
    decision, pinned so it cannot drift either way unnoticed
  - global (Admin or Reader): full entry including both identifiers
  - both scopes: array shape, empty archive, row order preserved

Mutation-checked both directions: forcing the global branch fails 3 tests,
dropping sessionCount from the projection fails 2. Full suite 3521 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:33:56 +02:00
Oliver Kieselbach 123ba0e55e fix(web): fleet-health crash in GA all-tenants scope - repeatDevices absent
The worker serializer omits null fields (WhenWritingNull), so the
aggregated global/metrics/device-journeys response carries no
repeatDevices at all. The strict `=== null` check missed undefined and
fell through to repeatDevices.length, crashing the page. Verified
against the deployed chunk: crash column 11464 is the `0===o.length`
read behind the `null===o` guard.

Loose `== null` check + optional DTO field; absent and null both render
the "select a tenant" hint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 19:28:14 +02:00
Oliver Kieselbach c8175ff7b1 fix(mcp): drop npm from the runtime image, bump tar to 7.5.22
Defender for Cloud reports CVEs against the mcp container image that the
repo scanners cannot see. The subassessment evidence paths split them in two:

  /usr/local/lib/node_modules/npm/**  - the npm 10.9.8 that node:22-slim
  bundles, vendoring @sigstore/core 2.0.0, brace-expansion 2.0.2, tar 7.5.11
  (critical), sigstore, picomatch, ip-address, plus npm itself (high, no fix
  available). Present in all 20 images including the current one. Dependabot
  reads package-lock.json, so none of this is visible to it.

  /app/node_modules/**  - body-parser 2.2.2, fast-uri 3.1.2, hono, protobufjs,
  tar 7.5.16, all confined to images from 2026-07-21 and earlier and already
  fixed by 7df208fa/c3e9621a. Those findings age out with the old ACR tags.

npm is build-time only here: CMD is `node dist/index.js`, the HEALTHCHECK is
`node -e`, the container app spec overrides no command and probes over HTTP,
and src/ contains no child_process call. So remove npm rather than upgrade it
- npm 11.x still vendors @sigstore/core 3.1.0, itself inside the vulnerable
<= 3.2.0 range, and would clear nothing.

Separately, tar 7.5.20 -> 7.5.22 in our own tree (GHSA-r292-9mhp-454m, via
@huggingface/transformers -> onnxruntime-node). 7.5.22 satisfies the declared
^7.0.1, so no override is needed. fast-uri deliberately stays on 3.1.4: it is
patched against all four fast-uri advisories, and ajv 8.18.0 declares ^3.0.1,
so forcing 4.x would be an override against a major range.

npm audit --omit=dev: 0 vulnerabilities. tsc clean, 455 tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:00:07 +02:00
Oliver Kieselbach 0efeb0f734 fix(backend): alert-webhook 500 on non-seekable body - bounded read replaces Length check
With ASP.NET Core integration the HttpRequestData body is a
forward-only stream whose Length property throws NotSupportedException
(verified in prod: 500 + System.NotSupportedException on the first
POST). The size cap is now enforced by reading at most cap+1 chars via
ReadBlockAsync instead of asking the stream for its length.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:52:59 +02:00
Oliver Kieselbach 7b975d46b6 feat(backend): Azure Monitor alert webhook - platform alerts join the ops-event bus
New anonymous POST ops/alert-webhook receives Common Alert Schema
notifications from an Action Group webhook receiver and re-emits them
as ops events (category Platform, type AzureMonitorAlert), so Azure
Monitor alerts flow through the existing OpsEventService ->
OpsAlertDispatchService fan-out (Telegram/Teams/Slack + portal
ops-events list) like every internally raised operational event. The
delivery path is independent of the portal: a SWA outage cannot
suppress the notification; the action group's e-mail receiver stays as
the backend-independent fallback.

Security: action-group webhooks cannot send custom headers, so the
shared secret travels as ?secret= and is compared in constant time
(FixedTimeEquals) against the OpsAlertWebhookSecret app setting -
fail-closed while unset. 64 KB payload cap, defensive essentials-only
parsing (only alertRule is mandatory), severity mapped Sev0->Critical,
Sev1->Error, Sev2->Warning, Sev3/4->Info with unknown values falling
back to Info so schema drift can never escalate.

Dual-registered per convention: EndpointAccessPolicyCatalog entry
(PublicAnonymous), OPS_EVENT_TYPES in OpsAlertRulesSection (rule
wiring) and Platform category in OpsEventsSection (badge + filter).

Backend suite 3511 green (incl. new helper tests + catalog
completeness), web tsc + vitest 651 green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:39:22 +02:00
Oliver Kieselbach 7680bdff7b fix(web): disable RSC prefetch on nav links - portal freeze root cause
Navbar + GlobalSidebar render the full navigation on every page, so the
App Router's default viewport prefetch fired one ?_rsc= request per
visible link against the SWA runtime (~10.4k requests / 14 days). That
origin serves document, chunks and RSC payloads over HTTP/1.1 (6
connections per origin); when the runtime queued the prefetches (p95
896ms, tail up to 234s), they held every socket and the portal froze
until the queue drained. Measured on the 2026-07-27 17:25 incident:
/admin/presence?_rsc= at 151s while the backend API answered in <2.2s.

All 16 nav links now set prefetch={false}. Every target route is a
client component behind ProtectedRoute that fetches its own data after
mount, so the prefetched RSC payload carried no benefit.

New OKF doc docs/web/portal-navigation-prefetch.md records the full
evidence chain and where browser telemetry lives (separate App Insights
resource, not reachable via MCP query_backend_logs).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:27:24 +02:00
Oliver Kieselbach 93094653a2 fix(web): tenant-scope seed races on usage-metrics/sla - own-tenant stats shown for override
A GA with a persisted tenant override opening /usage-metrics (or /sla) saw
their OWN tenant's stats until a manual refresh. useGlobalAdminScope seeded
the persisted selection in a post-mount effect, so the page effect first saw
effectiveTenantId = own tenant and fired an own-tenant fetch immediately
followed by the override fetch; with raw authenticatedFetch (no latest-wins
guard, unlike useAuthenticatedFetch since 82934a37) the last-RESOLVED
response won.

- useGlobalAdminScope: seed the GA selection render-time (same pattern as
  the delegated seed) so consumers never observe a transient own-tenant
  effectiveTenantId - the wrong fetch no longer starts at all.
- usage-metrics/sla pages: latest-wins sequence guard in fetchMetrics so a
  superseded request can neither write data/notifications nor clear the
  newer request's loading state (covers rapid selector switches).

Verified: tsc 0 errors, vitest 651/651.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 16:08:12 +02:00
Oliver Kieselbach ba7d14b070 fix(insights): post-deploy MCP findings - projection drop, aggregate starvation, duration artifacts
Three defects found by measuring production after the 1.5.1012 deploy:

- AppMetricsProjection dropped TerminalState, so the PR0 skip
  classification never reached metrics/app (totalSkipped stuck at 0).
  Column added; equivalence fixture now populates TerminalState/AppId/
  EspBlocking/AppIdCollision and asserts them (proven: test fails
  without the projection fix).
- BuildTimeAttributionAggregates excluded breakdowns on ANY quality
  flag; 903/1000 carry BlockingSetUnknown, starving fleet stats
  (Clean=12 vs 331 excluded on the largest tenant). New
  TimeAttributionFlagQuality.DurationCriticalFlags mask - only
  ClockSkew/PartialObservation/WGAnchors exclude; blocking-set flags
  keep their measured spans (positive-evidence semantics unchanged).
- HasMeasuredDuration gets an upper plausibility bound (6h = default
  AgentMaxLifetimeMinutes): 490 rows carried back-stamped completions
  (up to 117 days; 60175s app rows in a 2697s session) that dominated
  slowestApps with six-figure averages. Read-side, heals old + new rows.

Backend 3484/3484 green; solution build clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 14:50:55 +02:00
Oliver Kieselbach 7f7330e9e5 feat(web): auto-recover from post-deploy stale-chunk failures (one reload, loop-guarded)
A tab still running the previous deployment's bundle requests lazily-loaded
chunks (next/dynamic charts, route chunks) whose content-hashed filenames no
longer exist after the SWA swap - the user is stuck on "Loading chart..." or a
dead navigation until a manual refresh. Standard remedy: reload once so the
fresh document references the new bundle.

- utils/chunkReloadRecovery.ts: chunk-error detection (webpack ChunkLoadError +
  Chromium/Firefox/Safari dynamic-import variants) and a sessionStorage guard -
  at most one auto-reload per tab per 5 minutes; without working storage no
  auto-reload at all, so a reload loop is structurally impossible. Emits a
  ChunkReloadRecovery App Insights event before reloading.
- components/ChunkReloadRecovery.tsx (root layout): catches rejected dynamic
  imports via unhandledrejection and failed /_next/ script/link tags via a
  capture-phase error listener (third-party scripts never trigger a reload).
- app/error.tsx: a chunk error reaching the boundary reloads once instead of
  showing the error card; inside the guard window it falls through to the card.

Verified: tsc clean, vitest 651/651 (8 new tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 12:47:34 +02:00
Oliver Kieselbach 82934a3719 fix(web): tenant-aware unrestricted badge + latest-wins fetch guard on rules pages
GA override on gather-rules called GET /api/global/config/{tenantId}, a route
that does not exist in the backend; the swallowed 404 left the unrestricted
badge showing the GA's HOME tenant's mode for every foreign tenant. Both paths
now read the member-readable config/{tenantId}/feature-flags (MemberRead has a
global-scope bypass and the route is RouteParam-scoped), with a reset-on-switch
so a failed fetch never carries the previous tenant's mode over. The dead
globalConfig.tenant entry is removed from the URL register.

useAuthenticatedFetch now enforces latest-wins: the scope hook seeds a
persisted foreign tenant only after mount, so an own-tenant fetch races the
override fetch and the last-RESOLVED response used to win, leaving the rules
list on the home tenant while the selector showed the override. Stale requests
no longer write data/error/loading; the spinner is owned by the last-started
non-silent request.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 12:26:01 +02:00
Oliver Kieselbach 372f0c7c98 feat(web): source pill for non-Intune install rows + per-source row key
Customers who additionally deploy Microsoft 365 Apps as their own Win32 app saw two
rows in Install Progress (the IME view and the Office C2R lifecycle) with nothing
telling them apart, which read as a duplicate.

- New lib/installProgress.ts: buildInstallItems() extracted from the component so the
  event-folding rules are unit-testable; component is now render-only.
- InstallItem gains `source` (ime | office-c2r | realmjoin) and a source-scoped `key`.
  Rows were keyed by app name alone, so a Win32 app named exactly "Microsoft 365 Apps"
  (= OfficeInstallDetector.AppName) collided with the C2R row and overwrote its state.
- Pill only for non-default sources: "Click-to-Run" / "RealmJoin", neutral grey so it
  does not compete with the state badge, with a tooltip explaining what the row measures.
  Plain Intune app rows stay unlabelled.
- Drops the "RJ: " name prefix, which broke sorting and truncation.

Verified: tsc clean, vitest 643/643 (16 new).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 11:24:53 +02:00
Oliver Kieselbach 8057b44e48 fix(agent): serialize ESP-config collects so a later emission never carries an older snapshot
Codex review (P2): the ESP-config refresh triggers each run on their own
Task.Run (as do CollectAll's startup sweep and CollectAtEnrollmentStart),
so two concurrent collects could interleave read-before-emit - the LATER
emission then carried the OLDER (smaller) registry snapshot. The backend
uses the latest list-bearing esp_config_detected emission as positive
blocking evidence, so an out-of-order emission would silently shrink it.

CollectEspConfiguration now runs its read->emit sequence inside a
dedicated lock covering ALL callers; a later emission always carries an
equal-or-newer snapshot. The race predates the Q2 refresh triggers (they
only raised its frequency).

Deterministic concurrency test: 12 parallel refreshes against a probe
whose list grows per read - the emission sequence must be exactly 1..12.

Verified: Agent V2 Core 2059/2059.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 10:52:56 +02:00
Oliver Kieselbach 7842d121a0 fix(insights): codex review findings - WG pause clamping, late-event healing, stale buckets, date windows, banner count
All six review findings verified against code and real; five fixed here,
the agent-side race lands in the companion agent commit.

[P1] WhiteGlove pause counted as time: ClampToWindows clipped intervals
against the HULL of both WG windows, but the excluded pause lies inside
the hull - a reboot gap or app interval straddling the pause counted the
entire pause (a ~2-day "reboot" in a 30-minute enrollment was possible).
Seconds are now the per-window intersection sum (chronological endpoints
kept); an interval wholly inside the pause makes no claim and no longer
raises ClockSkewDropped. What-if savings are additionally capped at the
session's WallClockSeconds - endpoint deltas can still straddle the
pause, and no removal can save more than the enrollment took.

[P1] Late-arriving events never updated breakdowns: the terminal-seam
compute is one-shot and the sweep only healed missing/version-stale rows.
SessionTimeBreakdown now stores EventCountAtCompute (0 = pre-fix row,
recomputed once); the sweep recomputes on EventCount drift and then also
re-runs ResolveEspBlockingForSessionAsync (newly exposed on
IMetricsRepository) so late esp_config_detected re-emits and app batches
reach both the breakdown and the blocking evidence.

[P2] Stale aggregate buckets: the sweeps now reconcile every partition
they wrote - buckets that existed but were not regenerated (sessions
deleted, class left the window) are deleted via new targeted deletes;
rolling30 rows additionally get a 48h ComputedAt freshness filter at
read time, so a fully quiet tenant stops serving year-old numbers as
"last 30 days".

[P2] Date windows included an extra calendar day: both inclusive range
ends meant days=30 returned 31 keys and days=1 summed yesterday AND
today. InclusiveWindowStart(today, days) = today-(days-1) in both
response builders; the sweep's rolling30 source is filtered to the same
30 calendar days (its query window stays one day wider for backfill).

[P3] Device-history banner counted later sessions as "previous
enrollments" when viewing an older session: selectPriorSessions
(deviceHistoryPrior.ts) anchors on the session's own chain entry (live
sessions: new sessionStartedAt prop), sessionId tie-break mirroring the
server chain sort.

Verified: solution build clean, Backend 3480/3480 (+6: three WG-pause
golden fixtures, window-start facts, RK builder/delete consistency,
EventCountAtCompute roundtrip), Web tsc + vitest 633/633 (+6).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 10:52:39 +02:00
Oliver Kieselbach 4bc2435ec5 fix(agent): audit Q1/Q2/Q5 source fixes - timing re-arm, esp_config re-emits, serial trim
Q1 (AppInstallTiming): a terminal->active transition (IME retry path,
e.g. Error->Downloading or Skipped->Downloading) re-arms CompletedAtUtc
so the final terminal payload covers first-start -> last-terminal instead
of freezing at the first terminal state. Plus a negative-duration guard
in AppInstallTiming.DurationSeconds (inverted endpoints -> null, field
omitted) - observed in production as durationSeconds=-2342 on a
Skipped-then-retried user-scope app (session 66f1c5e1).

Q2 (esp_config_detected): the ESP blocking lists are read too early -
device lists grow progressively, user-scope S-<SID> lists appear only
after sign-in (88 of 89 prod sessions had an empty user list at last
emission). DeviceInfoHost now re-collects the ESP configuration as
phase-driven one-shots: apps sub-phase declaration (AppsDevice/AppsUser
phase_transition = first app activity) and AccountSetup detection.
The StartupEventGate keeps unchanged payloads silent (fingerprint covers
list contents), so only grown lists actually re-emit. F1 EspBlocking
resolution (latest list-bearing emission) benefits without backend change.

Q5 (HardwareInfo): WMI values (serial, manufacturer, model) trimmed at
the source via NormalizeWmiValue; whitespace-only collapses to the
"Unknown" sentinel the retry logic already treats as unresolved.

Attach-window investigation (11a3b67a): NO agent fix - hypothesis
disproved against two real agent logs (diag bundles db5ce007/2.0.1337,
4a9e9273/2.0.1379). "completed without started" first apps were already
installed (WinGet detection-only, zero install lines in AppWorkload.log);
pre-attach replay provably works (pattern-match log). PR0's
HasMeasuredDuration already handles the class server-side.
Duplicate app_install_completed (66f1c5e1): mechanism narrowed to an
in-process re-read of exactly one AppWorkload rotation window (<24h-old
lines, historic suppression not applicable) - documented as follow-up,
no fix without a bundle-bearing repro per lessons.md.

Verified: solution build clean, Agent V2 Core 2058/2058 (14 new tests),
DecisionCore 493/493. No ConfigVersion bump, no new event types,
decision rail untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:59:20 +02:00
Oliver Kieselbach 1bf37f6761 feat(insights): F3 regression radar - Wilson-gated rule hit-rate drift detection (PR6)
Analyze rules only (gather stats stay per-batch-deduped until Q6 is
fixed - never silently included). Per (tenant, rule) the radar compares
the trailing 7-day hit rate (FireCount/SessionsEvaluated from the daily
RuleStats rows) against the prior 28-day baseline and fires only when
the window has >=5 hit sessions and >=20 evaluated sessions, the rate
is >=2x baseline AND the Wilson 95% intervals are disjoint
(MetricsMath.WilsonInterval/RateIncreaseSeparated, hand-computed test
vectors). Suppressed for deleted rules, rules inside the CreatedAt
grace period (<35d), rules edited inside the window, empty baselines,
and the RuleRegressionRadarDisabled app-setting kill switch.

Alert episodes live as ruleregression|{ruleId} rows in the notification
tracker (third keyspace): AddEntity-409 dedup = one bell + one
RuleFrequencyRegression ops event (category Tenant, Warning, full
numbers, dual-registered in the web OPS_EVENT_TYPES) per episode;
numbers refresh while active without moving FirstNotifiedAt; episodes
re-arm under 1.5x baseline, on fire-stop, or via the tracker's 30d
retention prune (spec-intended reminder). On-fire dimension correlation
(osBuild/model/agentVersion/imeVersion; hit set from a RuleResults
range scan intersected with the window sessions) reports both shares
with the "correlated, not necessarily causal" wording contract or
honestly "no clear dimension concentration". Radar runs in the daily
maintenance pass after the rule-stats aggregation, anchored on
yesterday (whole days only).

Surfaces: regressions[] in the rule-stats response (MCP get_rule_stats
inherits it), tenant bell deep-linking to the rule card, and the
analyze-rules page gains a densified 30-day fire sparkline plus a
regression badge while an episode is active. OKF doc
backend/rule-regression-radar.md + index/log.

Backend 3474/3474 (27 new: Wilson vectors, every suppression branch,
window boundary dates, re-arm matrix, dimension gates, tracker
round-trips incl. tri-states, keyspace dedup/prune, wording pins),
web tsc + vitest 627/627, MCP tsc + 455 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:10:52 +02:00
Oliver Kieselbach 88da317bf4 feat(insights): F2 surfaces - device-history banner, fleet FTR panel, API routes, MCP tool (PR5)
Three read routes over the PR4 data: metrics/device-history
(MemberRead + tenant query param - one route serves the web banner AND
MCP, GA cross-tenant via ?tenantId= with a sessionId-based owner
fallback; history:null is a normal outcome), metrics/device-journeys
and global/metrics/device-journeys (days selector honored because FTR
counts are additive - window rate = ratio of summed daily rows; the
tenant-less global variant serves the "global" partition and discloses
repeatDevices:null). Attempt numbers are server-computed
(DeviceJourneyCalculator.ComputeAttemptNumber: chain position within
the journey; live sessions via a virtual non-successful attempt so the
redeploy and 30d-gap rules apply) - no consumer re-derives journey
semantics. Repeat-devices violator list: last journey >=2 attempts with
newest terminal ref in-window, attempts desc then recency, cap 10,
failure reason from the newest failed attempt (bounded point-reads).

Web: DeviceHistoryBanner on session detail ("Attempt N for this device
- View history", expandable terminal chain, durations strictly the
authoritative durationSeconds, WG/DevPrep/admin-marked badges) and the
Fleet Health "First-time-right" section (rate gated at >=20 completed
journeys with disclosed n, junk-serial exclusion note, Monday-UTC
weekly trend, attempt histogram, repeat devices) following the 7/30/90
selector. MCP: get_device_history (serial mode / fleet FTR mode,
role-aware routing, all roles - GA_FULL privilege snapshot extended
deliberately). Docs: OKF backend/device-journeys.md + index/log.

Backend 3447/3447 (14 new: attempt-number vectors incl. gap boundaries
and live sessions, additive sums, days clamping, repeat-device
gate/sort/cap), web tsc + vitest 627/627, MCP tsc + 455 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 08:38:17 +02:00
Oliver Kieselbach 1e6de47433 feat(insights): F2 journey grouping + DeviceHistories chains + daily FTR aggregates (PR4)
Device-key normalization (trim+casefold, junk-serial exclusion incl. the
agent's "Unknown" WMI sentinel), pure journey grouping (terminal set
Succeeded/Failed/Incomplete; journey ends at first success; >30d gap from
the previous attempt's CompletedAt starts a new journey; WG Pending stays
an open session, never an attempt), and the DeviceHistories table
(PK tenant, RK encoded normalized serial): terminal-only session-ref
chains capped at 20 with derived journey counts, written inline at the
session-terminal seam (all three terminal statuses, DeletionState-guarded)
and healed by a rolling 30d maintenance sweep. The sweep prunes chain refs
of deleted sessions tombstone-driven (SessionTombstones cover cascade AND
retention deletions), deletes emptied rows, and recomputes daily FTR
aggregates (DeviceJourneyAggregates, PK tenant/"global", RK date):
completed journeys, first-time-right count, attempt histogram and
junk-serial disclosure — additive counts, bucketed on the completing
success session's StartedAt date. Chain refs carry the authoritative
DurationSeconds verbatim (never CompletedAt-StartedAt) plus an
adminMarked flag. Both tables registered in Constants.All + offboarding
TenantPartitionTables; aggregates under the 180d retention block;
deliberately NO per-session deletion-manifest step for DeviceHistories.

Backend 3433/3433 (51 new tests: junk/normalization vectors, terminal
set, exact 30d gap boundaries, WG one-row-one-attempt, redeploy split,
deletion pruning, entity round-trips incl. row-key escaping, FTR
aggregation core), Agent 2044+493 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 08:14:13 +02:00
Oliver Kieselbach 6de21e821a feat(insights): F1 surfaces — session lane, fleet panel, API routes, MCP tool (PR3)
Insights expansion PR3 (tasks/insights-expansion-spec.md §F1 "Surfaces").
Completes F1: schema (PR1) -> persistence/aggregation (PR2) -> surfaces.

- Rolling range rows: the sweep now also writes "rolling30|{class}"
  aggregate rows — range statistics over the window's actual sessions. A
  median of per-day medians is not the range median, so the fleet panel
  reads these and is deliberately fixed to "last 30 days"; the daily rows
  stay in the payload for trends/MCP.
- Routes (all registered in EndpointAccessPolicyCatalog):
  GET sessions/{sessionId}/time-attribution (MemberRead + tenant query
  param, GA fallback via FindSessionTenantId; breakdown=null is SUCCESS —
  a missing lane is a normal state, not an error), GET
  metrics/time-attribution (MemberRead) and GET
  global/metrics/time-attribution (GlobalReadOrAdmin + tenantId filter),
  both returning { windowDays, classes (rolling), daily }.
- Web: TimeAttributionLane under the session-detail phase timeline —
  proportional stacked bar of segment totals incl. the explicit
  unattributed remainder (bar always sums to the session's authoritative
  duration), ESP-blocking apps with critical-path occupancy, reboot badge
  (cross-cutting, never a bar slice), quality-flag chips; fetched
  fail-soft for terminal sessions only. Fleet Health gains the "Time
  attribution" section: per-class stacked medians gated at n>=20
  ("insufficient data (n=...)" below), flagged/missing exclusions
  disclosed, and a top-blocking-apps table with class-scoped rows and
  "up to X (p75 Y)" what-if wording. Deliberate deviation: the spec's
  per-app trend column is omitted — per-day app rows are too sparse under
  the daily n>=5 gate; revisit once rolling history exists.
- MCP: get_time_attribution (one tool, session + fleet mode, role-aware
  routing like get_app_install_metrics, registered for all roles) — the
  privilege-leak-guard catalog snapshot extended deliberately.
- Docs: OKF concept doc backend/time-attribution.md (invariant, windows,
  segment mapping, positive-evidence blocking, Q7 reboot spans,
  persistence + surfaces) registered in index.md + log.md.

Backend 3382/3382 (+1 rolling-aggregate test), web tsc clean + vitest
627/627, MCP tsc clean + 455 passed. Post-deploy: web deploy +
MCP redeploy (new tool + docs corpus).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 18:33:16 +02:00
Oliver Kieselbach 55c898bb9c feat(insights): F1 breakdown persistence + daily attribution aggregates (PR2)
Insights expansion PR2 (tasks/insights-expansion-spec.md §F1 "Data & compute
changes"):

- SessionTimeBreakdowns table (PK=TenantId, RK=SessionId): span/reboot/
  blocking-app JSON columns, QualityFlags, AttributionVersion,
  EspAppsOccupancySeconds kept tri-state (absent column = unknown, never 0).
  Written once at the terminal single-writer seam (TableSessionRepository,
  next to the counter reconcile + EspBlocking resolution; Succeeded/Failed
  only — Incomplete deliberately stores no duration). Deleted with its
  session: new deletion-manifest step 16 (PkRkExact; tombstone moves to 19)
  plus tenant-offboarding partition wipe — no separate age sweep.
- Self-healing 30d maintenance sweep instead of a one-off backfill (rides
  the existing 2h tick + manual parity): backfills pre-deploy sessions,
  heals inline-compute misses, folds late-terminating sessions into their
  StartedAt-date aggregates, and enforces AttributionVersion purity — a
  stale-version breakdown whose events aged out counts as missing rather
  than mixing algorithm semantics (truthfulness rule 8). Skips sessions
  with a deletion cascade in flight.
- TimeAttributionAggregates table (PK=TenantId or "global",
  RK="{date}|{class}"): per enrollment class (whiteglove /
  device_preparation / self_deploying / user_driven — never mixed),
  median/p75/p90 per segment over the fixed six-segment stack (incl.
  unattributed; absent segment = honest 0), clean/flagged/missing counts
  disclosed (rule 7), rows written below the n>=20 UI gate on purpose
  (the UI needs the n — rule 4). 180d retention alongside UsageMetrics.
- What-if bound: WhatIfSavingSeconds implements the spec formula
  max(0, cpEnd - cpEndWithoutX) including the idle gap before X (the ESP
  waits for X to start, too); sole-interval fallback = own start. Per-app
  aggregate rows gate at >=5 sessions, top 20 by median interval, carry
  median/p75 of interval and saving ("up to" wording is the consumer
  contract).

Backend 3381/3381 (13 new tests: entity round-trips incl. tri-state,
aggregation bucketing/disclosure/gates, what-if vectors, class precedence,
manifest step-count pins 17/19), solution build clean, web untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 18:03:14 +02:00
Oliver Kieselbach e637317bdb feat(insights): F1 schema + time attribution calculator (PR1)
Insights expansion PR1 (tasks/insights-expansion-spec.md §F1, audit §0.5):

- AppInstallSummary + AppId / EspBlocking (bool?, tri-state) / AppIdCollision:
  sentinel-gated columns + mapper + both projections. appId adopted from app
  events (first-seen wins); a second appId under the same display name flags
  the name-keyed row as a collision (audit Q3) — in-batch and cross-batch
  (sticky in the reconcile). Collision rows leave every per-app fleet
  aggregate (app metrics payload, apps list, app analytics) with a disclosed
  exclusion count; the fleet DO rollup and per-app session drill-down keep
  them.
- EspBlocking resolution at the session-terminal single-writer seam
  (TableSessionRepository.UpdateSessionStatusAsync, next to the counter
  reconcile — covers ingest, admin marks, maintenance sweep, rule engine):
  joins the session's app rows against the LATEST list-carrying
  esp_config_detected emission as a positive-evidence set (audit Q2) —
  listed => true, absent => stays unknown, never false. EspBlockingSets
  parser handles truncation via the uncapped espTracked*Count fields.
- TimeAttributionCalculator (pure, AttributionVersion 1) + SessionTimeBreakdown
  models: exact partition of the session's authoritative DurationSeconds into
  device_prep / esp_apps / identity_hello / user_esp / desktop_handoff spans
  plus an explicit unattributed remainder — invariant sum(spans)+unattributed
  == wall clock enforced in every test. WhiteGlove = two observation windows
  in ONE session (part1 anchored on whiteglove_part1_complete, pause never
  attributed). Per-app blocking intervals from EVENT timestamps (first start
  -> LAST terminal, audit Q1 retry freeze), occupancy = merged union clipped
  to esp_apps. Reboot spans from event gaps around lastBootUtc (audit Q7),
  deduped per boot. QualityFlags: ClockSkewDropped, PartialObservation,
  BlockingSetUnknown, BlockingSetTruncated, WhiteGloveAnchorsIncomplete.
  Persistence (SessionTimeBreakdowns) and surfaces follow in PR2/PR3.

Backend 3368/3368 (35 new tests: 19 calculator golden fixtures + 16
identity/blocking/aggregate-exclusion), solution build clean, web untouched
(new response fields are additive).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 17:29:36 +02:00
Oliver Kieselbach 4eac7d042c fix(metrics): app duration/rate stats over real installs only — TerminalState column + out-of-order reconcile
Source-data audit (tasks/insights-expansion-spec.md §0.5) verified against
production: 17.9% of Succeeded AppInstallSummaries rows carry no duration —
78% are genuine skips (app_install_completed with payload state=Skipped,
"Update for X" WinGet policies not applicable) whose state the ingest
discarded, the rest are real installs whose start the agent never observed
(attach window). Averaging them as 0s understated avg install duration by
18% and distorted the slowest-apps ranking.

- AppInstallSummary.TerminalState (Installed/Skipped/Postponed/Error) from
  the closing event's payload state; sentinel-gated column + mapper +
  AppsDashboardProjection. Status stays "Succeeded" for compatibility.
- MetricsMath.IsSkipTerminalState/HasMeasuredDuration: durations computed
  over measured rows only; skips leave the failure/success rate (decision
  2026-07-26 — a skip is not an install attempt); per-app skipped/unmeasured
  counts + totals exposed; slowest ranking gates on measured installs.
  Applied consistently: app metrics payload, apps dashboard (list/analytics/
  time series/trend/versions/device models), SLA weekly/snapshot/breach.
- StoreAppInstallSummaryAsync: reconcile extracted to internal static
  ReconcileAppInstallSummaryWithExisting (test seam). Fixes Q4 (terminal
  batch before started batch left duration permanently unset) and hardens
  terminal status against late/replayed InProgress batches.
- Web: measured-only hints on app detail cards, skip suffix in apps list,
  fleet-health "measured installs" caption.

Backend 3333/3333 (20 new tests), web tsc clean, vitest 627/627.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 16:24:45 +02:00
Oliver Kieselbach c7ed3ec27d more safe SecureBootCertReader and section plan layout update 2026-07-25 12:48:27 +02:00
Oliver Kieselbach f311e8746b docs(trust): on-demand log collection in customer-facing claims + tracker dedup tests
The "Collect Logs" feature (b382e567) added a second opt-in path into hosted
diagnostics upload — the quick-config dialog on a session — and a new collection
trigger, neither of which the customer-facing pages described. The trust FAQ still
said hosted upload is enabled "in the settings UI", and the dialog itself named
hosted storage without stating that packages leave the tenant boundary, so the
"clearly marked disclosure" promise was not met on that path.

- CollectLogsButton: marked data-boundary disclosure in the quick-config dialog
  (leaves your Azure tenant, per-tenant isolation, blob-scoped write-only token,
  removal per Data Retention Days)
- privacy + about pages: both opt-in paths named; on-demand collection described
  (Admin/Operator of the own tenant, same content as the automatic package,
  delivered with the agent's next check-in, visible on the session timeline)
- docs repo counterpart in trust/security-faq.md (separate commit)

Distress-bell dedup: the TpmPssUnsupported one-shot bell had key-shape tests only.
New HardwareRejectionNotificationTrackerDedupTests pins the 409 dedup branch, the
fail-soft branch, the blank-input short-circuit, the disjoint hardware/tpmpss key
spaces, and the retention sweep — which also prunes tpmpss rows and thereby re-arms
that device's bell, now stated in the interface and implementation docs.

Backend tests 3313/3313, tsc clean, web vitest app/sessions 60/60.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 12:43:42 +02:00
Oliver Kieselbach e0b6ce83cf fix null warnings 2026-07-25 00:17:31 +02:00
Oliver Kieselbach 7f7b590db4 fix(web): mobile readability for rule cards + phase timeline overlap
Rule list cards (analyze rules, gather rules, IME log patterns) used a
single-line flex header where every badge was flex-shrink-0 and only the
title could shrink — on phone widths the title truncated to a few
letters. The header now wraps: on mobile the title drops onto its own
full-width row (order-last + w-full) under the badge row and may wrap;
on >=sm the layout is unchanged (title inline, truncated).

PhaseTimeline squeezed up to 8 flex-1 columns with 48px circles and
whitespace-nowrap labels into the viewport, so circles and label text
overlapped completely on mobile. The timeline now has a minimum width
of 80px per phase inside an overflow-x-auto container (swipe on phone,
no scrollbar on desktop) and labels wrap inside their own column
instead of overlaying neighbours.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 00:04:04 +02:00
Oliver Kieselbach 4c2c447a2e fix(web): dark-mode slate/hover-tint overrides + mobile wrap for script executions
Dark mode: globals.css was missing the entire slate family (the
"Incomplete" status filter chip rendered white-on-white) and all light
hover tints outside gray/indigo/cyan (hover:bg-purple-50 in the tenant
suggestion dropdown flashed near-white under light text). Added slate
bg/text/border mappings mirroring gray, plus dark equivalents for every
hover:bg-*-50/100/200 class in use (blue, purple, red, amber, yellow,
green, orange, sky, teal, emerald, slate, gray-200/300).

Mobile: ScriptExecutions rows were single-line flex with flex-shrink-0
right clusters, so badges and status text overlapped/overflowed on
narrow screens. Header counters, cycle card headers and item rows now
flex-wrap with gap spacing instead of space-x.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 23:46:03 +02:00
Oliver Kieselbach 8a794a1b52 fix(rules): cross-tenant GA rule edits no longer misroute to home tenant
A Global Admin editing a gather/analyze rule while viewing a FOREIGN
tenant (override) got a green "saved" banner, but the write landed in
the GA's OWN home tenant: the JWT-scoped rules/{type}/{ruleId} routes
resolve the tenant from the caller's token, and no cross-tenant write
route existed (reads had /api/global/rules/*, writes did not). The
reload re-read the foreign tenant and showed the unchanged rule.

Fix: symmetric cross-tenant WRITE routes mirroring the global reads.

Backend:
- GlobalRulesFunction: PUT/DELETE global/rules/gather/{ruleId} and
  global/rules/analyze/{ruleId}; tenant from ?tenantId= (400 if
  missing), same 1 MB guard + ValidateScopeAndEmitMode as the JWT
  variants, same service calls (toggle-partial/built-in-state/full-edit
  semantics are tenant-parameterized and identical).
- Catalog: 4 entries GlobalAdminOnly + TenantScoping.QueryParam — a
  write tier, so the delegated-read rescue never fires; only a platform
  admin may mutate another tenant's rules. No global CREATE route:
  create stays JWT-only and the portal hides Create in override.

Web:
- api.rules.globalGatherRule/globalAnalyzeRule helpers; both rule pages
  route save/toggle/mark-failed/notify/delete through a
  *RuleWriteUrl(ruleId) switch on isGlobalOverride.
- Analyze template-enable (a create) is blocked in override with a
  clear message instead of silently writing to the wrong tenant.

Tests: CrossTenantAccessTests guards the 4 routes are GA-only +
QueryParam and that a GA reaches the named target tenant. Backend
3295/3295, web vitest 627/627, tsc clean.

Note: the old bug planted orphan copies of foreign rules into the GA's
home tenant on each attempted cross-tenant edit; those are not
auto-cleaned.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 23:21:37 +02:00
Oliver Kieselbach 6c53dc33e7 feat(distress): tenant-visible TpmPssUnsupported devices + one-shot bell
TpmPssUnsupported distress reports (TPM cannot RSA-PSS-sign, Schannel drops
the client cert on Win11 25H2+, agent can never authenticate) were operator-
only, yet the remediation is tenant-side: TPM firmware update or device
replacement.

- New GET audit/tpm-pss-unsupported (MemberRead), aggregated per serial,
  mirroring audit/device-not-registered
- Bell notification "tpm_pss_unsupported" (Admin audience) once per
  (tenant, serial); dedup via the existing rejection tracker table with a
  disjoint "tpmpss|{serial}" row-key space; no webhook by design
- Web: TpmIncompatibleDevicesInsights panel in the Hardware Whitelist
  section; renders only when a device is affected (no permanent card)

No agent change, no ConfigVersion bump. Backend tests 3287/3287, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 20:47:06 +02:00
Oliver Kieselbach 801891d57a layout update section plan comparison 2026-07-24 20:30:28 +02:00
Oliver Kieselbach 1f6efca95b fix(backend): stamp DiagnosticsBlobName from on-demand server_action_executed events
Session 8e4cc4ae: two "Collect Logs" round trips completed on the agent (both
ZIPs landed in hosted storage) but the portal never flipped to Download and
the button hung at "Collecting..." — the ServerActionDispatcher confirms an
on-demand upload only via server_action_executed(actionType=request_diagnostics,
blobName=...) and emits NO diagnostics_uploaded (only the terminal upload path
does), so UpdateSessionDiagnosticsBlobAsync never ran.

The ingest classification now treats such an executed event as the upload
confirmation (a real diagnostics_uploaded in the same batch still wins — it
carries destination). Because the executed event has no destination field,
InferDiagnosticsDestination derives "Hosted" from the backend-built
"{tenantId}/{filename}" blob-name shape (CustomerSas blobs are bare
filenames); without it the download route would look in the customer's
container for a hosted blob.

Backend-only — works with the already-deployed agent fleet, no agent release.
Backend suite 3274/3274.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 19:23:47 +02:00
Oliver Kieselbach 9ec8644be2 feat(settings): operator read-only settings view with real values
Operators previously saw editable settings forms bound to hardcoded defaults
whose Save always 403'd. Now they see the tenant's real configuration
read-only, with no save affordances at all.

Backend:
- GET config/{tenantId}: TenantAdminOrGlobalReader -> MemberRead. Safe because
  the handler redacts by default (CanViewSecrets: only GA or own-tenant Admin
  see secrets); Operators/Viewers receive RedactedCopyForReader().
- POST diagnostics/files: TenantAdminOrGA -> TenantAdminOrOperator (submitting
  logs is a troubleshooting action; Viewer and read-only GlobalReader excluded).
- Tests: policy pins updated; new middleware tests (own-tenant Operator allowed
  + redacted, cross-tenant Operator 403); CanViewSecrets theory for
  Operator/Viewer.

Web:
- TenantConfigContext loads the full config for all settings users (403
  fallback to feature-flags minimal config for deploy-order safety) and
  exposes canEditConfig; saveConfiguration guards on it.
- New ReadOnlyFieldset wraps section content (native fieldset disable, values
  stay copyable); SaveResetBar is not rendered at all for read-only viewers.
  Applied to Contact, SLA Targets, Hardware Whitelist (+ rejection-insights
  toggle), Notifications, Agent Settings, Agent Analyzers, Diagnostics, Data
  Management.
- Hidden entirely from Operators (in-page admin-only card + sidebar sub-item
  gate): Autopilot Validation, Access Management, Offboarding, Unrestricted
  Mode. GlobalReader sidebar parity kept via hasGlobalScope.

Tests: backend 3257/3257, web vitest 627/627, tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 18:40:27 +02:00
Oliver Kieselbach 9acd10e1dc docs(okf): ServerActions & on-demand diagnostics concept doc
Covers the queue-on-session/deliver-on-ingest channel, the
TenantAdminOrOperator tier with in-function per-type re-gating, the portal
Collect Logs flow (quick-config orders rotate_config before
request_diagnostics), sequence-based progress matching, and the hosted-prefix
sweep in the offboarding cascade. Registered in index.md, logged in log.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 18:15:23 +02:00
Oliver Kieselbach b382e5678e feat(web): "Collect Logs" on-demand diagnostics collection on the session detail page
Queues a request_diagnostics ServerAction (delivered with the agent's next
ingest response) and tracks progress live: server_action_received flips the
button to "Collecting…", the DiagnosticsBlobName SignalR delta completes it
(Download Diagnostics appears), server_action_failed or a 3-min quiet timeout
surface as toasts.

- Button is ALWAYS rendered — disabled states carry tooltips (inactive
  session, cross-tenant view, missing role, diagnostics not configured) as a
  deliberate nudge towards enabling diagnostics upload.
- Tenant Admin on an unconfigured tenant gets a quick-config dialog: switches
  the tenant to Hosted + OnFailure (GET config -> patch two fields -> PUT
  verbatim), then queues rotate_config BEFORE request_diagnostics so the agent
  collects against the fresh config. Operators/Viewers never see the dialog.
- Operator may collect when configured (new TenantAdminOrOperator backend tier).
- useSessionTenantConfig picks up diagnosticsUploadConfigured from the
  member-readable feature-flags endpoint.
- Progress matching is sequence-based (backend-assigned) so agent clock skew
  and at-least-once action redelivery cannot produce false matches.

Pure state logic in collectLogsLogic.ts with vitest matrix (21 tests).
tsc clean; web suite 627/627.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 18:12:49 +02:00
Oliver Kieselbach 2fee50276e feat(backend): operator-tier session actions + diagnostics groundwork for on-demand log collection
- New EndpointPolicy.TenantAdminOrOperator (Admin/Operator/GA, no Viewer) with
  middleware evaluator; POST sessions/{sessionId}/actions moves to this tier so
  troubleshooting operators can queue request_diagnostics ("Collect Logs").
- QueueSessionActionFunction re-gates per type: terminate_session and
  rotate_config stay Admin/GA-only (403 for operators).
- GET config/{tenantId}/feature-flags exposes diagnosticsUploadConfigured so
  members below Admin can see whether an on-demand upload would work.
- Offboarding cascade: hosted diagnostics deletion now sweeps the session
  prefix {tenantId}/AgentDiagnostics-{sessionId}- (plus the manifest-named
  blob) so on-demand packages overwritten on the Sessions row by the terminal
  package no longer survive deletion. CustomerSas branch unchanged.
- Notification dismiss routes deliberately NOT migrated: dismissal is still
  tenant-shared; operators would clear admin-tier notifications they cannot see.

Tests: middleware role matrix on the actions route, per-type gate matrix,
feature-flag matrix, prefix-sweep + real hosted-branch cascade tests.
Backend suite 3253/3253.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 18:05:39 +02:00
Oliver Kieselbach 8bb02acc4a fix(web): bump postcss to >=8.5.12 (GHSA-6g55-p6wh-862q, high)
Dependabot alert #74: PostCSS <= 8.5.11 allows arbitrary file read /
information disclosure via an attacker-controlled sourceMappingURL in CSS
comments. Raise the floor (devDependency + override) from ^8.5.10 to
^8.5.12; the lockfile now resolves postcss 8.5.22 (its nanoid dep moves
3.3.11 -> 3.3.16 with it). npm audit reports 0 vulnerabilities.

Next.js pins postcss 8.4.31 transitively, but the existing root override
collapses the tree to a single postcss node, so the bump covers it too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:54:57 +02:00
Oliver Kieselbach fd7d3973e7 fix(agent): esp_policy_provider_stalled gains sidecar-missing arm (issue #106)
Customer follow-up corrected the field-case mechanics: TrackingPoliciesCreated=1
WAS set under the foreign ConfigMgr provider - the ESP ignores a foreign
provider's completion values and waits on the Intune-registered "Sidecar"
provider by name (renaming the key unblocks instantly without creating any
tracking entries). The completeness-keyed detector would have missed exactly
the incident it was built for.

- Arm 2: >=1 Setup\Apps provider registered without Sidecar, continuously for
  the 15-min dwell -> reason=sidecar_provider_missing; clock resets when
  Sidecar registers later (legitimate startup ordering); DevicePreparation-only
  snapshots don't activate it
- Arm 1 (continuously-incomplete providers) unchanged -> reason=provider_incomplete;
  sidecar-missing wins when both arms cross in one round (single event)
- Cross-restart fingerprint gains a "sidecarMissing" sentinel; payload adds
  reason + sidecarMissingForMinutes
- Regression test: ConfigMgr complete + Sidecar absent -> fires (previously
  classified healthy); probe/doc headers and OKF doc corrected, web event-type
  description updated

Evidence (research session c7e3db08): Intune's ESP-policy WAP payload always
registers Sidecar (ConfigMgr only via co-management settings policy, per
Niehaus' documented payload); Microsoft's ESP troubleshooting models app
tracking exclusively through the Sidecar subkey.

Agent tests 2043/2043, web tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:48:50 +02:00
Oliver Kieselbach fd2b5c2da2 fix(ci): drop invalid SWA input, gate deploy on token via env guard
Azure/static-web-apps-deploy@v1 has no `skip_deploy_on_missing_secrets`
input (confirmed against its action.yml), so passing it did nothing but
emit an "Unexpected input(s)" warning — the intended "skip green on
Dependabot PRs" behaviour was never actually in effect.

Replace it with the documented mechanism: surface the deploy token as a
job-level env var and gate the deploy/close step with `if: env.DEPLOY_TOKEN
!= ''` (secrets cannot be referenced directly in an `if:`). On Dependabot
PRs the token is absent, so the step skips and the job stays green; real
deploys on main and internal PRs still run. Applied to both jobs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:38:58 +02:00
Oliver Kieselbach a44e37a7c2 fix(metrics): keep full failure-reason text for fleet-health display
BuildFleetFailureReasons truncated each reason to a 50-char prefix + "..."
and used that as both the grouping key AND the display value. The full text
never reached the UI, so its tooltip/tap-to-expand had nothing to reveal and
near-identical rows were indistinguishable past the cut.

Group by the 50-char prefix as before (variants still collapse into one row),
but carry the longest full reason as the display value. The UI truncates and
expands it on demand, so the complete text is now available end-to-end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:25:11 +02:00
Oliver Kieselbach d8a252562d fix(web): pin engines.node to 22.x so SWA/Oryx does not grab Node 24
The prior ">=22" made Oryx build with the highest available Node (24.13),
which the Static Web Apps runtime rejects: "Node version 24.13.0 is not
supported. Please use one of the following versions 18, 20, 22." Pin to
22.x so Oryx selects the SWA-supported major. CI setup-node '22' already
resolves to latest 22.x, so build and deploy now agree.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:23:17 +02:00
Oliver Kieselbach a5d12ba1d0 chore(ci): unify Node toolchain on 22 across CI + engines pins
Node 20 reached EOL (2026-04-30); Node 22 is the current maintenance LTS
and already the runtime the MCP container ships (node:22-slim). Straighten
the version skew so per-component CI major == deploy runtime == engines floor:

- ci.yml: web, mcp and guardrails setup-node 20 -> 22
- combine-rules.yml: setup-node 20 -> 22
- mcp package.json: add engines.node ">=22" (matches Dockerfile)
- web package.json: add engines.node ">=22" (first explicit pin; SWA/Oryx
  no longer picks a Node version implicitly)

engine-strict stays off (no .npmrc), so the pins are advisory and cannot
hard-break local installs on other Node versions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:15:54 +02:00
Oliver Kieselbach 530c978bf8 fix(web): TruncatedLabel expands to full text, not just two lines
The expanded state used line-clamp-2, which re-truncated long values with an
ellipsis after two lines — unreadable on touch where the hover tooltip is
unavailable. Expand now wraps the entire string over as many lines as needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 15:11:54 +02:00
Oliver Kieselbach 09898d7ba0 feat(web): consistent truncate-with-tooltip + tap-to-expand across dashboard cards
Add reusable TruncatedLabel component: a single-line label that truncates
with an ellipsis, shows the full text as a native hover tooltip, and on
click/Enter/Space expands to two wrapped lines (touch-friendly). An
interactive={false} mode provides tooltip-only behaviour for use inside
already-clickable rows/links where an inner onClick would hijack navigation.

Apply it consistently to the long, dynamic label columns across:
- fleet-health: failure reasons, model + app ranked lists
- sla: top failing apps, violators failure reason
- geographic-performance: fastest/slowest location cards
- vulnerability exposure panel: CVE sample software
- agent metrics: top exception types
- progress + download/install: app-name labels
- admin software tabs: software names
- session reports (comment/adminNote) + ops events (message): tooltip-only

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 15:03:34 +02:00
Oliver Kieselbach 71eda93e8b fix(metrics): app-install failure rate over finished installs only
Second half of the terminal-only rate convention (follows 1cfc32c9, which
converted the enrollment success rates): app-install failureRate divided
by ALL install rows including "InProgress" ones — installs still running,
or orphaned when a session died mid-install — diluting every failure rate
customers see on the Apps dashboard and Fleet Health.

- New MetricsMath.TerminalFailureRatePct (Failed / (Failed + Succeeded),
  0 when nothing finished) shared by both aggregation sites
- MetricsMath.BuildAppMetricsPayload: per-app failureRate terminal-based
  (Fleet Health "Top Failing Apps"; subtitle now "% of N finished")
- AppsAnalyticsHelper: list, analytics summary, versionBreakdown,
  timeSeries and deviceModelBreakdown rates terminal-based; the
  duplicated trend block is deduplicated into ComputeFailureTrend, whose
  rates and >= 5-sample gate now run on finished installs per half; the
  device-model >= 5 sample floor likewise counts finished installs
- Web: InstallsTab "Avg Failure Rate" card computed over summed finished
  installs instead of totalInstalls

Tests: new AppsAnalyticsFailureRateTests (list/trend/analytics/model
cases incl. in-flight dilution and gate regressions) and a terminal-rate
case in MetricsMathAppPayloadTests. Backend 3223/3223, tsc clean,
vitest 602/602.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 10:01:28 +02:00
Oliver Kieselbach 1cfc32c9aa fix(metrics): success rate over finished enrollments only (geo + fleet health)
Customer feedback: the geographic performance pane showed a poor success
rate for a site mid-rollout because in-progress/pending sessions counted
against the rate. Success rate is an outcome quota — align Geo and Fleet
Health with the SLA convention already used by dashboard stats, usage
snapshots and the fleet rollup: Succeeded / (Succeeded + Failed).
Incomplete (terminal, non-failure) stays excluded.

- Geo: per-location SuccessRate over finished sessions; table badge and
  map tooltip render an em dash instead of a red 0% when nothing has
  finished yet
- Fleet Health: Stats.SuccessRate terminal-based; card shows "X of Y
  finished enrollments" and a neutral em dash state; per-model health
  gets a Failed count (additive wire field) so the client rate excludes
  in-flight devices; TopFailingModels FailureRate now failed/finished
- Tests: FleetHealthPayloadTests flipped to the new convention plus
  zero-finished, model-failed and in-flight-dilution cases; new
  GeographicMetricsSuccessRateTests pin the geo semantics

Backend 3216/3216, web tsc clean, vitest 602/602.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 09:47:31 +02:00
Oliver Kieselbach 11339c9868 fix(test): de-flake SessionDeletionWorkerTests via condition-gated runs
CI run 30047512186 failed Worker_drops_envelope_missing_required_fields_
without_invoking_handler: RunForAsync starts the BackgroundService poll loop
and waits a fixed 500ms wall-clock window — on a loaded CI runner (xUnit runs
test classes in parallel) the loop may not have processed the message before
the positive assertions (DeleteMessageAsync at least once) run. All ten
positive tests in the file shared that latent race; this run it hit the
malformed-envelope drop test.

Fix mirrors the watchdog event-gate de-flake: new Harness.RunUntilAsync
(condition, 10s ceiling, 25ms poll) runs the worker until the awaited side
effect is observable, then stops. Gates: ops-event recorded (last step of the
poison path, so send+delete+CAS are already done), main-queue delete for the
drop tests, >=3 visibility heartbeats for the heartbeat test. The kill-switch
test keeps the fixed window — a Times.Never expectation has no event to wait
for. OpsEventCalls capture is now lock-guarded so the condition poll can read
it while the worker thread appends.

16/16 worker tests green across 4 consecutive runs; full backend suite
3211/3211.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 00:03:10 +02:00
Oliver Kieselbach 540f543834 feat(agent): extend historic IME-replay suppression to app events + app aggregation
Part 2 of the eaf3d8c4 fix (follows 088c3b04). In that session 147 of 157 app
events were replayed from a previous enrollment's IME log — all 10 "installed
apps" were phantoms, polluting the timeline, AppInstallSummaries, MCP counts,
the app_tracking_summary header, culprit lists and final-status.json.

Rename (nothing deployed yet): historic_script_replay_detected →
historic_ime_replay_detected; SuppressIfHistoricReplay / IsHistoricImeReplay.

Agent:
- EmitAppState suppresses stale-replay transitions BEFORE the sub-phase
  declaration, timing bookkeeping and terminal-signal dedup — covers the
  informational event, the AppInstallCompleted/Failed DecisionSignal, the
  shadow download_progress and the summary snapshot. Behavior note: phantom
  terminal signals can no longer spuriously re-arm the AdvisoryCompletion
  window (app signals are otherwise observation-only — verified no completion
  gate reads them). EmitDoTelemetry gets the same guard.
- Tracker-level guard at the HandlePatternMatch choke point skips the 23
  app-mutating actions (incl. espphasedetected — a stale AccountSetup line
  would make fresh DeviceSetup lines bounce as "backward") for source lines
  >24h stale, so replayed apps never enter PackageStates, persistence,
  app_tracking_summary, culprit lists or final-status. New internal
  UtcNowProvider seam (no ctor change), threshold single-sourced as
  ImeLogTracker.HistoricReplayThreshold.
- SimulationMode now bypasses all replay suppression — 088c3b04 had broken
  the --replay-log-dir dev tool, which feeds historic lines on purpose.

Backend: IsHistoricImeReplay guard in AggregateAppInstallEvent keeps replayed
events out of AppInstallSummaries (SLA/dashboards/metrics inherit).

Web: generic lib/historicReplay.ts partition (scriptExecutions becomes a
wrapper); InstallProgress hides replayed installs with a muted note;
DownloadProgress filters silently.

MCP: get_session_summary filters replayed events — cleans appInstalls stats,
errorCount (replayed app_install_failed are Error-severity) and keyEvents.

Tests: +13 agent (app suppression, tracker guard), +1 backend, +7 web,
+6 MCP. Agent 2039, backend 3211, web 602, MCP 455 all green; drift test
validates the rename.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 23:43:59 +02:00
Oliver Kieselbach 088c3b0479 fix(agent): suppress historic IME-log script replay + timeline-consistent durations
Session eaf3d8c4: a previous enrollment's IME logs survived on disk (dated
2026-07-16, 7 days old) and the agent replayed them at startup — 156 phantom
script_completed events in a 7-minute session, each showing ~170h durations
(clock-clamped completion minus RAW stale start), plus an immediateUpload
flood (3x ingress_backpressure) and RemediationScriptCount inflated to 156.

Agent:
- Script events whose source line is >24h stale are suppressed entirely; a
  one-shot historic_script_replay_detected Info event (new EventType) dates
  the replayed window via earliestRejectedSourceTimestamp. Future-skew
  rejections and clock-only fallbacks still emit.
- durationSeconds now uses a timeline-consistent pair: on rejected-timestamp
  emissions the raw source completion pairs with the raw start (true runtime
  instead of cross-timeline garbage). 24h plausibility backstop omits
  implausible values; same consistency for script_timeout_suspected.
- Stale pending platform-script slots (>24h older than the current start
  line) are discarded so an old orphaned start can't pair with a fresh run
  (extracted HandlePlatformScriptStarted as internal test seam).

Web: partitionHistoricScriptEvents hides legacy-agent replay events (muted
"N historic executions hidden" note) and the reducer blanks >24h durations —
covers sessions recorded before the agent fix rolls out.

Backend: IsHistoricScriptReplay guard keeps replayed events from inflating
Platform-/RemediationScriptCount during the rollout window.

Registered historic_script_replay_detected in Constants.EventTypes, MCP
resource-catalog (script_events) + KEY_EVENT_TYPES, gather-rules eventTypes.

Tests: 8 agent (suppression/one-shot/raw-pair/backstop/slot-hardening),
8 web (partition/blanking), 10 backend (predicate). Agent 2026, backend
3210, web 595, MCP 449 all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 18:39:11 +02:00
Oliver Kieselbach 3224774752 fix(test): EDR-safe straggler replay — no more renamed system executable
The straggler integration test copied powershell.exe under a synthetic
agent name and launched it — textbook MITRE T1036.003 ("Rename Legitimate
Utilities"). Defender for Endpoint flagged the run ("System executable
renamed and launched", initiating process testhost.net48.exe) and ISOLATED
the dev machine.

Replaced with function shadowing: a harness script defines Get-Process /
Stop-Process stub functions (PowerShell resolves functions before cmdlets)
and dot-sources the cleanup script. The stubs report a live agent-named
straggler on every probe and record the kill, pinning the identical
contract — poll by NAME until the deadline, force-kill by NAME, then
delete — with zero process spawns, zero PE copies, nothing EDR-visible.

The production routine never renames or copies executables; the flagged
technique existed only in this test. Lesson recorded in tasks/lessons.md.

22/22 cleanup tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:24:07 +02:00
Oliver Kieselbach e7ea431ccf feat(esp): recovery story for terminally reported ESP failures (session 4910a5a5)
DeviceSetup/Apps failed, the user pressed "Try again" ~23 min later, apps
recovered to 10/10 and DeviceSetup resolved - yet the advisory_completion
window un-defanged the stale failure and failed the live enrollment,
because the advisory variant was exempt from all re-arm/rebase guards.

Engine fix (DecisionCore):
- New facts EspAdvisoryFailureResolvedUtc + EspAdvisoryFailureCategory
  (state-borne; deadline payload would be lost on re-arm/rebase).
- TryResolveAdvisoryOnCategoryRecovery in both ProvisioningComplete
  handlers: category match sets the fact + emits
  esp_failure_advisory_resolved.
- Guards now (!hasAdvisoryAnchor || advisoryResolved): enforcement-progress
  re-arm, hello-never-observed promote and the reboot rebase apply again
  once the failure demonstrably un-happened; unresolved advisories keep
  the legacy un-defang exactly.
- Truthful terminal reason esp_recovered_without_completion_evidence
  (+ failureRecovered=true, LastFailureTrigger=DeadlineFired so the
  likely-stuck app promotion stays off).

Story events (agent tracker):
- esp_failure_retry_detected: failed subcategory leaves "failed" after the
  terminal fire - consistent with the user pressing "Try again".
- esp_failure_recovered: previously failed category reaches success.
- EspFailureDetectedEventArgs.LikelyCulpritApps (never-started first,
  cap 3) rides the EspTerminalFailure payload so esp_failure_advisory /
  enrollment_failed name the culprit app directly.

Surfacing: MCP resource-catalog, web timeline badges (Recovered /
User retry pills, culprit chip on advisory), gather-rule autocomplete,
ANALYZE-ESP-006 "ESP Failure Recovered After User Retry".

Tests: AdvisoryRecoveryTests (13, incl. the 527/611/635/671 field-ordinal
replay that no longer fails the session), EspFailureRecoveryStoryTests (5),
snapshot-builder guards 18 -> 20. All suites green (DecisionCore 493,
V2.Core 2018, Backend 3201, web vitest 589, MCP 449 incl. drift).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:03:44 +02:00
Oliver Kieselbach 9e24b92fbf feat(agent): ESP policy-provider stall detector + Autopilot registry guardrail
Customer field case: pre-provisioned devices with the ConfigMgr client as a
blocking app occasionally park the user ESP at "Apps (Identifying)" for days.
Root cause (verified against the EnrollmentStatusTracking CSP doc): only a
"ConfigMgr" provider is registered under Device\Setup\Apps\PolicyProviders
(no "Sidecar" = IME) and TrackingPoliciesCreated never becomes 1 - a wait the
CSP defines WITHOUT any timeout. Renaming the key to Sidecar unblocks
immediately. Microsoft documents that ESP providers are not aware of each
other and that pre-provisioning is unsupported with co-management.

New always-on esp_policy_provider_stalled tripwire, generalized to any
provider in both kinds (Setup/Apps TrackingPoliciesCreated != 1,
DevicePreparation InstallationState not in {2,3}):

- EspPolicyProviderProbe: fail-soft registry read of the
  EnrollmentStatusTracking mirror (device + user-SID scopes), defensive
  value parsing (missing/unparseable = incomplete), ScopedOverride test seam.
  Research and CSP contract quotes captured in the XML doc header.
- EspPolicyProviderStallDetector: pure EvaluateStallTransition on a 15-min
  continuous-incompleteness dwell (resets on completion/disappearance;
  absent key = normal early enrollment), one-shot per provider,
  StartupEventGate fingerprint over the sorted stalled key set for
  cross-restart dedup. Warning + ImmediateUpload via InformationalEventPost;
  purely observational, no decision-engine coupling, no config toggle,
  no ConfigVersion bump (disk_space_low precedent).
- EspPolicyProviderStallHost: 60-s wall-clock tick, deliberately outside the
  StallProbe idle gate (idle clock resets on session activity) and the
  idle-stopped periodic collectors.
- Registered in Constants.EventTypes, MCP resource-catalog (health_events),
  web gather-rule autocomplete (stall category).

Guardrail companion: SOFTWARE\Microsoft\Windows\Autopilot joins the
gather-rule registry allowlist ("Autopilot / OOBE / Setup") so tenants can
build custom rules on the subtree (e.g. not_exists on the Sidecar provider
key); generated web mirror regenerated via combine.js; positive + sibling-
spoof guard tests added.

Docs: OKF concept doc agent/esp-policy-provider-stall.md (+ index/log).

Tests: 22 new detector tests (pure transition matrix + emission surface incl.
the ConfigMgr-only customer scenario and gate-across-restart dedup); agent
suite 2013/2013, web vitest 589/589 + tsc clean, MCP 449 passed incl.
event-types-drift.

Both the detector and the embedded allowlist take effect with the next agent
release.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 15:20:47 +02:00
Oliver Kieselbach 45d7c03ccf fix(web): WhiteGlove per-block durations use sequence-canonical edges, not timestamp min/max
Customer report (session fcb44595): dashboard said 115 min, session detail
said 5h 2m for the same pre-provisioned enrollment. The detail's per-block
duration calc took min/max over ALL event timestamps inside each
sequence-block, which two real-world timestamp shapes break:

- an app_install_started re-emitted by the ImeLogTracker after the Part-2
  resume legitimately carries its historical Part-1 event time -> the User
  Enrollment block stretched back across the sealed pause (56m -> 3h18m)
- a do_telemetry with a +1h source timestamp (IME log line with a wrong
  timezone bias, just under the agent's 1h future-skew clamp) stretched
  Pre-Provisioning (58m -> 1h44m)

Sum: exactly the "5h 2m" the customer saw.

Block spans are now anchored on the first/last event BY SEQUENCE (sequence
is the canonical order), with Part 1 starting at session.startedAt for
backend DurationSeconds parity. Extracted as pure helper
computeWhiteGloveDurations with tests modeled on the real session; replay
against the customer's 880 events yields 1h 54m == backend 6872s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:26:33 +02:00
Oliver Kieselbach 90012d9a66 feat(agent): verify Secure Boot CA-2023 state against firmware, not just servicing registry
Customer report: ANALYZE-SEC-001 warned on devices whose firmware already
carried the Windows UEFI CA 2023 certificate — the agent only read the
SecureBoot\Servicing registry value, which mirrors the Windows Update
rollout and lags behind reality (OEM factory image, WinCS, WU not run yet).

Agent: new UefiSecureBootCertReader reads the UEFI db/KEK variables
natively (GetFirmwareEnvironmentVariableExW; SeSystemEnvironmentPrivilege
enabled on the process token and restored — first privilege-adjustment
code in the repo). Fail-soft, never throws; legacy BIOS maps to not_uefi.
secureboot_status gains per-certificate booleans, uefiFirmwareReadStatus,
and the one-sided marker uefiCa2023FirmwareConfirmed (emitted only on
positive confirmation, never as false).

Rule v3.0.0: not_exists precondition on the marker suppresses the rule
when the firmware confirms the certificate; the marker's absence keeps
old agents on exact v2 behavior, so the rule is safe to seed before the
agent rollout. Firmware-verified-missing raises confidence (+20 db,
+10 KEK 2K CA 2023).

Verified: full solution build; backend 3201, agent 1988, web 582 tests
green; tsc/ajv clean; elevated cross-check on real firmware matches
Get-SecureBootUEFI on all seven certificate probes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:07:01 +02:00
Oliver Kieselbach e8ef4f53ae feat(agent): OOBE-state observability via WinRT SystemSetupInfo
Sample Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState
(build >= 17763, SYSTEM-callable, CLR-native WinRT projection — no NuGet)
as a purely observational signal, mirroring the bootstrap guard marker:

- oobeStateAtAgentStart on agent_started and agent_late_start payloads:
  makes late starts provable instead of heuristic ("completed" on a
  first_run = OOBE was already over when the agent arrived) and validates
  the bootstrap OOBE guard fleet-wide.
- One-shot oobe_state_completed trace event on the in_progress->completed
  flip, piggybacked on the DesktopArrivalDetector 30s poll: an
  owner-independent desktop corroboration for DAD-starved sessions
  (4d5a0b78 class — WTS+WMI owner resolution dead, desktop_arrived never
  fires). Often absent in healthy sessions since the DAD timer is
  disposed at desktop arrival, which is when the flip lands.

OobeStateReader is fail-soft (never throws, "unavailable" on missing
contract); the flip check has its own try/catch so it cannot pollute the
DAD wmiErrorCount liveness diagnostics, "unavailable" never overwrites
the baseline, and the one-shot flag survives ResetForRealUserSwitch.
No decision-engine involvement, no config/backend changes. Event type
registered in Constants.EventTypes, MCP resource-catalog (phase_events)
and web gather-rule autocomplete.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:39:10 +02:00
Oliver Kieselbach a0d10b8471 test(cleanup): end-to-end incident replays for the self-destruct script
Six integration tests execute the generated cleanup script with a real
powershell.exe against GUID-unique scratch trees — proving what the string
pins cannot:

- locked tree ⇒ complete abort, marker + binaries survive (the 2026-07-22
  regression: the old script force-deleted around the locks)
- full incident replay: a renamed second process holds a file in the tree,
  is found by NAME, force-killed after the wait deadline, cleanup then
  completes with nothing left behind
- fresh foreign lock ⇒ loser backs off without touching the tree; stale
  lock (>10min) is taken over and cleanup proceeds
- happy path + keepLogs variant

BuildCleanupScript gains test-only timing parameters (defaults unchanged
for production) so the replays run in seconds. Agent suite 1969/1969.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 10:06:29 +02:00
Oliver Kieselbach ad3632e9fb fix(emergency-break): survivable self-destruct, reliable break report, ops visibility
Incident 2026-07-22: the 48h emergency break left exe + loaded DLLs orphaned
forever — a second agent instance held the images while the cleanup script
(waiting on the launcher PID only) force-deleted marker + state around the
locks AFTER removing the Scheduled Task, destroying the retry path.

Agent — cleanup script (LIFE-F7):
- wait for ALL agent processes by name (PID-reuse-proof), then force-kill
  stragglers; single-flight lock file serializes concurrent cleanup scripts
- probe-rename (GUID target) BEFORE task removal; any failure renames back
  and aborts with task + marker + files intact for a next-boot retry; the
  destructive Remove-Item-on-original fallback is gone
- script generation extracted to BuildCleanupScript, pinned by 13 tests;
  all 4 variants parser-validated + E2E-smoked (happy path, locked tree)

Agent — break report delivery:
- EmergencyReporter counts DELIVERED reports: reservation rolls back on
  failure (mark-before-send permanently suppressed a key after its first
  transport failure); optional in-reservation retries
- break path: NIC wait <=15s, 3 attempts x 15s timeout (survives Functions
  cold starts observed as >5s 503s), 75s overall bound

Backend:
- backend-materialized events (agent_emergency_break, session_timeout) now
  upsert the cross-session EventType index — they were invisible to every
  search-by-eventType surface (portal + MCP)
- new AgentEmergencyBreak ops event (Agent/Warning, once per session) so
  operators see silently-lost agents; dual-registered in OPS_EVENT_TYPES
- materialization extracted to a static seam, covered by 4 new tests

Agent 1963/1963, Backend 3198/3198, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:56:46 +02:00
Oliver Kieselbach 05e1569503 feat(agent): wildcard support for Local Admin Analyzer allowed accounts
Allowed Local Accounts entries may now contain glob wildcards (* / ?),
matching the hardware-whitelist semantics: Regex.Escape conversion,
case-insensitive, 100ms ReDoS timeout. Applies to both the account and
the profile-folder check. Entries without wildcard chars keep exact
matching; * and ? are invalid in Windows account names, so patterns and
literals cannot collide. UI documents the syntax and rejects patterns
with fewer than 2 literal characters (a bare * would disable the
analyzer). No schema change - old agents treat patterns as literals and
keep alerting (fail-safe).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:47:13 +02:00
Oliver Kieselbach ba3581e253 bootstrapper added download stopwatch 2026-07-23 09:35:30 +02:00
283 changed files with 23396 additions and 1207 deletions
+3 -3
View File
@@ -63,7 +63,7 @@ jobs:
- uses: actions/setup-node@v7
with:
node-version: '20'
node-version: '22'
cache: npm
cache-dependency-path: src/Web/autopilot-monitor-web/package-lock.json
@@ -86,7 +86,7 @@ jobs:
- uses: actions/setup-node@v7
with:
node-version: '20'
node-version: '22'
cache: npm
cache-dependency-path: src/McpServer/autopilot-monitor-mcp/package-lock.json
@@ -106,7 +106,7 @@ jobs:
- uses: actions/setup-node@v7
with:
node-version: '20'
node-version: '22'
- name: Regenerate
run: node rules/scripts/combine.js
+1 -1
View File
@@ -26,7 +26,7 @@ jobs:
- uses: actions/setup-node@v7
with:
node-version: '20'
node-version: '22'
- name: Install ajv-cli
run: npm install -g ajv-cli ajv-formats
+11 -3
View File
@@ -21,6 +21,11 @@ jobs:
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.action != 'closed')
runs-on: ubuntu-latest
name: Build and Deploy Job
# Dependabot PRs run without repo secrets, so the deploy token is absent.
# Surface it as an env var so the deploy step can gate on it — secrets
# cannot be referenced directly in an `if:` expression.
env:
DEPLOY_TOKEN: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
steps:
- uses: actions/checkout@v7
with:
@@ -28,6 +33,9 @@ jobs:
fetch-depth: 0 # Full history needed for git-based sitemap lastmod dates
- name: Build And Deploy
id: builddeploy
# Skip (green) instead of failing when the token is missing (Dependabot
# PRs); real deploys on main and internal PRs still have it.
if: env.DEPLOY_TOKEN != ''
uses: Azure/static-web-apps-deploy@v1
env:
NEXT_PUBLIC_ENTRA_CLIENT_ID: ${{ vars.NEXT_PUBLIC_ENTRA_CLIENT_ID }}
@@ -39,9 +47,6 @@ jobs:
with:
azure_static_web_apps_api_token: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
repo_token: ${{ secrets.GITHUB_TOKEN }} # Used for Github integrations (i.e. PR comments)
# Dependabot PRs run without repo secrets, so the deploy token is absent.
# Skip (green) instead of failing the check; real deploys on main still have the token.
skip_deploy_on_missing_secrets: true
action: "upload"
###### Repository/Build Configurations - These values can be configured to match your app requirements. ######
# For more information regarding Static Web App workflow configurations, please visit: https://aka.ms/swaworkflowconfig
@@ -54,9 +59,12 @@ jobs:
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
name: Close Pull Request Job
env:
DEPLOY_TOKEN: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
steps:
- name: Close Pull Request
id: closepullrequest
if: env.DEPLOY_TOKEN != ''
uses: Azure/static-web-apps-deploy@v1
with:
azure_static_web_apps_api_token: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
+151
View File
@@ -0,0 +1,151 @@
---
type: Concept
title: ESP Policy-Provider Stall Detection — the no-timeout Setup/Apps wait and the co-management Sidecar gap
description: Why the ESP's Apps wait is keyed to the Intune-registered "Sidecar" provider by name — a foreign provider (ConfigMgr) with TrackingPoliciesCreated=1 does not satisfy it — plus the classic never-completes stall, and how the always-on two-arm esp_policy_provider_stalled tripwire (reason=provider_incomplete | sidecar_provider_missing, 15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces both.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs
tags:
- agent
- esp
- enrollment-status-tracking
- co-management
- stall-detection
timestamp: 2026-07-24T00:00:00+02:00
---
# ESP Policy-Provider Stall Detection
## Schema
The Enrollment Status Page decides when its "Apps" category may progress from
"Identifying" using the **EnrollmentStatusTracking CSP**
(<https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>),
mirrored in the registry under
`HKLM\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\{Device|<UserSID>}\…`.
Two contracts matter:
1. **`Setup/Apps/PolicyProviders/<Name>`** — per the CSP doc: *"Existence of this node
indicates to the ESP that it shouldn't show the tracking status message until the
TrackingPoliciesCreated node has been set to true."* This wait has **no timeout**. A
registered provider that never sets `TrackingPoliciesCreated=1` parks the ESP at
"Apps (Identifying)" indefinitely — observed for days in the field. Exists in device
scope (`Device`) and user scope (`S-<SID>`, appears at sign-in).
2. **`DevicePreparation/PolicyProviders/<Name>/InstallationState`** — must reach 2
(NotRequired) or 3 (Completed); 1=NotInstalled, 4=Error. Device scope only. The ESP
applies a default 15-minute timeout here, so these stalls self-surface as ESP errors —
the detector still records them for the diagnostic trail.
Known provider names: **`Sidecar` is the Intune Management Extension** (the CSP doc names
"Intune's agents, such as SideCar"); **`ConfigMgr`** is the Configuration Manager client,
registered during Autopilot-into-co-management — the troubleshooting section of
<https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment> queries
exactly these registry paths. The same document states that ESP policy providers **are not
aware of each other** and that **pre-provisioning is unsupported** with Autopilot into
co-management.
## The field case (issue #106)
A tenant pre-provisions devices and installs the ConfigMgr client as a blocking app.
Occasionally the user-phase ESP reaches "Apps (Identifying)" and never leaves it — for
hours to days. On those devices `Device\Setup\Apps\PolicyProviders` contains **only a
`ConfigMgr` key (no `Sidecar`)** — and, per the customer's follow-up, **`TrackingPoliciesCreated=1`
IS set under it**. The provider is "complete" by the CSP value contract, yet the ESP
still waits. Manually renaming the key to `Sidecar` — which creates no tracking entries,
only satisfies the name — unblocks the ESP instantly.
The naive CSP-doc reading ("any provider with TrackingPoliciesCreated=1 satisfies the
wait") is therefore wrong for this case. What the evidence supports instead: provider
nodes are registered **by the Intune service at enrollment**, and its ESP-policy WAP
payload always registers `Sidecar` (ConfigMgr joins only via a co-management settings
policy — see the payload documented by Niehaus). The ESP waits on the provider it
expects **by name**; a foreign provider occupying the slot does not satisfy it,
regardless of its completion values. On the affected devices the CM client (installed
as a plain Win32 app, no co-management settings policy) evidently displaced the
Sidecar node — consistent with Microsoft's warning that providers "aren't currently
aware of others" and that pre-provisioning is unsupported with co-management. The exact
CloudExperienceHost mechanism is not publicly documented; the registry signature —
setupApps providers present, `Sidecar` absent — is deterministic either way.
## Detection
`EspPolicyProviderProbe` (fail-soft, static, `ScopedOverride` test seam — structural clone
of `EspTrackingInfoProbe`) enumerates all providers in both kinds and scopes.
`EspPolicyProviderStallDetector` evaluates a pure transition function on a 60-second tick
from the always-on `EspPolicyProviderStallHost` — **two arms**, same 15-minute wall-clock
dwell, one shared event type discriminated by `reason`:
* **Arm 1, `reason=provider_incomplete`** — a provider **continuously incomplete for
≥ 15 minutes**. The dwell clock starts at first incomplete observation and resets on
completion or disappearance. Missing values parse as incomplete
(fail-toward-observation). Catches e.g. Sidecar itself never reporting.
* **Arm 2, `reason=sidecar_provider_missing`** — **≥ 1 `Setup\Apps` provider registered
but none named `Sidecar`**, continuously for ≥ 15 minutes. Deliberately ignores
provider completeness (the field case had `TrackingPoliciesCreated=1`). Sidecar
registering later — the legitimate startup ordering — clears the condition and resets
the clock. DevicePreparation-only snapshots don't activate it.
* When both arms cross in the same round, ONE event is emitted and
`sidecar_provider_missing` wins as `reason` (it explains the hang regardless of the
foreign provider's state).
* Missing root key / empty provider list = normal early enrollment: clears all dwell
state (both arms), never fires.
* One `esp_policy_provider_stalled` **Warning** per newly stalled condition set
(`ImmediateUpload`), one-shot per condition in-process, cross-restart deduped via a
`StartupEventGate` fingerprint over the sorted currently-active fired conditions
(provider keys + a `sidecarMissing` sentinel) — a restart into the same stall stays
silent, a new condition joining re-reports.
* Payload: `reason`, `providers[]` (full table: name, scope, kind, raw values, complete
flag), `stalledProviders[]` with `stalledForMinutes`, `sidecarMissingForMinutes`
(arm 2 only), `sidecarRegistered`, `dwellMinutes`.
* **Purely observational**: emitted via `InformationalEventPost` (dispatch-guard exempt
pass-through) — never touches decision state. No config toggle, no ConfigVersion bump
(precedent: `disk_space_low`).
Deliberately **not** built on `StallProbeCollector` (its idle clock resets on any session
activity — a provider stall coexists with a chatty session) and not on the periodic
collectors (idle-stopped after 15 minutes — exactly the dormant window the dwell must
observe).
## Guardrail companion
`rules/guardrails.json` allow-lists `SOFTWARE\Microsoft\Windows\Autopilot` (category
"Autopilot / OOBE / Setup") so tenants can build their own gather rules on the subtree —
e.g. a `not_exists` registry rule on the `Sidecar` provider key. The subtree carries
deployment state only (provider registrations, tracking lists, app IDs, per-user SIDs);
sensitivity is at or below existing entries. Enforcement is agent-side (embedded
resource), so both the detector and the guardrail change ship with the agent release.
## Examples
Customer-case event (abbreviated) — note the provider is "complete" by CSP values, the
hang is carried entirely by `reason=sidecar_provider_missing`:
```json
{
"eventType": "esp_policy_provider_stalled",
"severity": "Warning",
"data": {
"reason": "sidecar_provider_missing",
"dwellMinutes": 15,
"sidecarRegistered": false,
"sidecarMissingForMinutes": 15.0,
"providers": [
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps",
"trackingPoliciesCreated": 1, "installationState": null, "complete": true }
],
"stalledProviders": []
}
}
```
`reason=sidecar_provider_missing` is the co-management signature;
`reason=provider_incomplete` with `sidecarRegistered=true` covers a stalled Sidecar or
third-party provider — the generalized contract violation.
# Citations
* EnrollmentStatusTracking CSP — <https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>
* Windows Autopilot into co-management (requirements, provider-conflict warning, troubleshooting registry paths) — <https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment>
* Troubleshoot the ESP (models app tracking exclusively via the `Sidecar` subkey) — <https://learn.microsoft.com/en-us/troubleshoot/mem/intune/device-enrollment/understand-troubleshoot-esp>
* Niehaus: the Intune ESP-policy WAP payload registering the provider nodes (`Sidecar` always, `ConfigMgr` only with a co-management settings policy) — <https://oofhours.com/2023/08/25/run-an-sccm-task-sequence-during-autopilot/>
* Niehaus: ESP tracks arbitrarily named providers when they are properly registered — <https://oofhours.com/2023/09/18/track-anything-using-esp/>
* Agent code: `EspPolicyProviderProbe.cs`, `EspPolicyProviderStallDetector.cs`, `EspPolicyProviderStallHost.cs` (SystemSignals / Orchestration.Hosts)
* Related: [MDM Reboot Coalescing](mdm-reboot-coalescing.md), [Gather-Rule Guardrails](../rules/gather-rule-guardrails.md)
+100
View File
@@ -0,0 +1,100 @@
---
type: Concept
title: Secure Boot CA-2023 Detection — firmware evidence vs. Windows servicing status
description: Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly (GetFirmwareEnvironmentVariableExW + SeSystemEnvironmentPrivilege) instead of trusting the SecureBoot\Servicing registry key, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant rule suppression across old and new agents.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Interop/UefiSecureBootCertReader.cs
tags:
- agent
- secureboot
- ca2023
- analyze-rules
- backward-compatibility
timestamp: 2026-07-23T00:00:00+02:00
---
# Secure Boot CA-2023 Detection
## Schema
Two independent evidence sources describe the Windows UEFI CA 2023 rollout state:
1. **Servicing registry** (`HKLM\SYSTEM\CurrentControlSet\Control\SecureBoot\Servicing\UEFICA2023Status`)
— reflects whether the *Windows Update* rollout ran and self-verified. Lags reality:
devices whose firmware received the certificate another way (OEM factory image, WinCS
CLI, manual deployment) or where WU simply has not run yet show `unknown`/`notfound`.
2. **Firmware itself** — the UEFI variables `db` (signature database, GUID
`{d719b2cb-3d3a-4596-a3bc-dad00e67656f}`) and `KEK` (GUID
`{8be4df61-93ca-11d2-aa0d-00e098032b8c}`). Authoritative. Certificate subject CNs are
ASCII bytes inside the DER signature lists, so a byte-substring search for
`Windows UEFI CA 2023` / `Microsoft Corporation KEK 2K CA 2023` decides presence —
the same check `Get-SecureBootUEFI` enables from PowerShell.
`UefiSecureBootCertReader` (agent, `Monitoring/Interop`) reads both variables natively via
`GetFirmwareEnvironmentVariableExW` — no PowerShell runspace during OOBE. The call needs
`SeSystemEnvironmentPrivilege`: SYSTEM holds it default-disabled, so the reader enables it
on the process token (`AdjustTokenPrivileges`) and restores the previous state afterwards.
Fail-soft contract like `OobeStateReader`: never throws; failures map to status strings
(`not_uefi` = legacy BIOS via `ERROR_INVALID_FUNCTION`, `variable_not_found`,
`privilege_denied`, `error_<code>`).
This is the repo's only privilege-adjustment code, so it also carries the ownership rules
any future one should copy:
* The process token lives in a `SafeHandle` assigned by the P/Invoke marshaler, so the CLR
owns it from the moment `OpenProcessToken` returns — the critical finalizer releases it
even if `Dispose` never runs (async exception before the instance reaches its `using`).
* The restore obligation is recorded immediately after the successful `AdjustTokenPrivileges`,
and only when the privilege was not *already* enabled; nothing that can throw sits between
the adjustment and that assignment. `Dispose` is idempotent, swallows a failed restore, and
releases the handle in a `finally`.
* The privilege is adjusted on the **process** token, so enable → read → restore is a
process-wide side effect and runs under a static gate. Without it a concurrent reader would
see "already enabled", correctly skip its own restore, and then lose the privilege mid-read
when the first reader disposes — a spurious `privilege_denied` on a machine that holds it.
Fields land on the existing `secureboot_status` event (once per agent start,
restart-deduped by the StartupEventGate): `uefiFirmwareReadStatus`, per-cert booleans
(`uefiDbHasWindowsUefiCa2023`, `uefiDbHasWindowsProductionPca2011`,
`uefiKekHasMicrosoftKek2kCa2023`, …) — emitted only when the respective read succeeded —
plus the marker below.
## The one-sided marker (absence-tolerant suppression)
Rule-engine constraints (pinned by `RuleEnginePreconditionTests`):
* **Conditions** never match on an absent field (null guard) — new fields must stay
`required: false` or the rule dies for every old agent.
* **Preconditions**: `not_exists` passes while the field is absent/empty and fails as soon
as the field is present with *any* value; comparison operators fail closed on absence.
A two-sided boolean therefore cannot drive suppression: `not_exists` on
`uefiDbHasWindowsUefiCa2023` would also suppress the rule when the field is present with
`false`. Hence the **one-sided marker** `uefiCa2023FirmwareConfirmed = true`, emitted
ONLY when the db read succeeded AND the 2023 certificate was found — never as `false`.
ANALYZE-SEC-001 v3 adds a `not_exists` precondition on it:
| Device state | Marker | Rule |
|---|---|---|
| Old agent (no firmware fields) | absent | evaluates exactly like v2 |
| New agent, cert missing or firmware unreadable | absent | evaluates; optional conditions `uefiDbHasWindowsUefiCa2023=false` (+20) / `uefiKekHasMicrosoftKek2kCa2023=false` (+10) raise confidence |
| New agent, firmware confirms cert | present | silently suppressed (the customer-reported false positive) |
Because the marker exists nowhere until the new agent ships, the v3 rule is safe to seed
**before** the agent rollout.
## Examples
Customer-equivalent verification from PowerShell (authoritative, matches the reader):
```powershell
[Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023')
[Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name KEK).Bytes).Contains('Microsoft Corporation KEK 2K CA 2023')
```
## Citations
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Interop/UefiSecureBootCertReader.cs` — native reader
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/DeviceInfo/DeviceInfoCollector.NetworkAndSecurity.cs` — `CollectSecureBootStatus` / `AppendFirmwareCertFields`
* `rules/analyze/ANALYZE-SEC-001.json` — v3.0.0 rule definition
* `src/Backend/AutopilotMonitor.Functions.Tests/RuleEnginePreconditionTests.cs` — pinned precondition semantics + v3 mirror tests
* Microsoft: [Act now — Secure Boot certificates expire June 2026](https://techcommunity.microsoft.com/blog/windows-itpro-blog/act-now-secure-boot-certificates-expire-in-june-2026/4426856)
+122
View File
@@ -0,0 +1,122 @@
---
type: Concept
title: Device Journeys & First-Time-Right (F2)
description: How a device's terminal enrollment attempts are grouped into journeys (device key, junk-serial exclusion, first-success end, 30-day gap), where the DeviceHistories chain and the daily FTR aggregates are computed, stored and pruned, and which surfaces serve them.
resource: src/Backend/AutopilotMonitor.Functions/Helpers/DeviceJourneyCalculator.cs
tags:
- backend
- metrics
- device-history
- first-time-right
- insights
timestamp: 2026-07-27T12:00:00+02:00
---
# Device Journeys & First-Time-Right (F2)
Answers "how many devices enrolled right on the first try?" and "show me every attempt
this device made". Session success rate hides wipe-and-retry loops — every retry that
eventually succeeds counts as a success; First-Time-Right (FTR) exposes the real pain.
All grouping semantics live server-side in `DeviceJourneyCalculator` (JourneyVersion 1);
no consumer re-derives them.
# Schema
## Device key
`(TenantId, normalized serial)` — normalization is trim + lower-case. Junk/placeholder
serials are excluded entirely (never a chain, never in FTR, disclosed as a per-day
exclusion count): `System Serial Number`, `To Be Filled By O.E.M.`, `Default string`,
`0`, `None`, `INVALID`, `Unknown` (the agent's WMI-failure sentinel), empty, or shorter
than 4 characters. A motherboard swap changes the serial and therefore the device key;
re-enrollment under a different tenant is a separate key by design.
## Journey
Terminal sessions of a device key ordered by StartedAt. The terminal set is
**Succeeded / Failed / Incomplete** — `Pending` (a sealed WhiteGlove part 1),
`AwaitingUser`, `Stalled` and `InProgress` are OPEN sessions and never attempts.
* A journey **ends** with the first terminal success; the next session starts a new
journey (redeployment).
* A gap of **more than 30 days** since the previous attempt's terminal moment
(`CompletedAt`, fallback StartedAt) starts a new journey even without a success
(device shelved/repurposed). Constant, not a setting — production gaps: median 4.9 h,
92 % < 7 d.
* WhiteGlove part 1 + part 2 share ONE session row, so a completed WG enrollment is ONE
attempt; a WG device still waiting for its user session leaves the journey open —
never a failed attempt.
* **Attempt count** = terminal sessions in the journey (`Failed` and `Incomplete` are
non-successful attempts).
* **FTR rate** = completed journeys with attempt count 1 ÷ all completed journeys.
Open and gap-abandoned journeys never enter numerator or denominator.
Durations shown for attempts are the sessions' authoritative `DurationSeconds` verbatim
(WhiteGlove pause excluded; `Incomplete` deliberately has none) — never
`CompletedAt − StartedAt`, which is later in ~25 % of terminal sessions.
## Persistence & compute
* `DeviceHistories` (PK = TenantId, RK = percent-encoded normalized serial): the chain of
the device's terminal session refs `{sessionId, startedAt, completedAt, status,
enrollmentType, isPreProvisioned, durationSeconds, adminMarked}` capped at the 20 most
recent, plus derived `CurrentJourneyAttempts` (the LAST journey's attempts — the
banner's "Attempt N"), `JourneyCount` (chain-scoped) and `JourneyVersion`. Written
inline at the session-terminal single-writer seam
(`TableSessionRepository.UpdateSessionStatusAsync`, all three terminal statuses,
DeletionState-guarded) so the session banner is fresh; healed by the maintenance sweep.
Deliberately NOT part of the per-session deletion manifest — the row aggregates many
sessions. Wiped on tenant offboarding.
* Sweep (`SweepDeviceJourneysAsync`, 30-day rolling window, 2h tick): (1) drops chain
refs of deleted sessions **tombstone-driven** — every session deletion (cascade and
retention both run through the cascade worker) leaves a `SessionTombstones` marker for
~7 days, far longer than the sweep cadence, and even an expired-but-unpruned marker is
valid deletion evidence; a chain left empty deletes its row. (2) Merges the window's
terminal sessions into their chains (backfill, inline-miss healing, re-terminal
reclassifications). (3) Recomputes the daily FTR aggregates idempotently.
* `DeviceJourneyAggregates` (PK = TenantId or `global`, RK = `{yyyy-MM-dd}`): per day —
completed journeys, first-time-right count, attempt histogram, junk-serial exclusion
count, `JourneyVersion`. A journey buckets on the StartedAt date of its completing
success session (the platform's StartedAt-date convention; late terminals converge via
the sweep). All counts are **additive**, so a window rate is the ratio of summed daily
rows — there is no rolling-window row (unlike the median-based time-attribution
aggregates, which cannot merge). 180-day retention. Rows exist below the ≥20 UI gate
on purpose (the UI needs the n for "insufficient data (n=…)").
# Surfaces
* `GET api/metrics/device-history?serialNumber=&sessionId=` (MemberRead + tenant query
param — the same route serves the web banner and MCP; a Global Admin passes `tenantId`)
— the chain plus, with `sessionId`, that session's server-computed attempt number
(live sessions get their would-be position via a virtual non-successful attempt, so the
redeploy and gap rules apply). `history: null` = no recorded history.
* `GET api/metrics/device-journeys?days=` (MemberRead) /
`GET api/global/metrics/device-journeys?days=&tenantId=` (GlobalReadOrAdmin) —
`{ windowDays, totals, daily, repeatDevices }`. The days selector is honored because
counts are additive. Without `tenantId` the global variant serves the cross-tenant
aggregate rows and `repeatDevices: null` — a per-device list would require scanning
every tenant's partition.
* Repeat devices (violator list, capped 10): devices whose LAST journey took ≥2 attempts
and whose newest terminal session falls inside the window, ordered by attempts then
recency; the failure reason comes from the newest failed attempt (bounded point-reads).
* Web: "Attempt N for this device · View history" banner on session detail (expandable
chain); "First-time-right" section on Fleet Health (rate + weekly trend + attempt
histogram + repeat devices). MCP: `get_device_history` (serial mode / fleet FTR mode).
# Examples
A device fails ESP twice and succeeds on the third try within one week: one journey,
three attempts, completed — counts once in the FTR denominator, not in the numerator,
and the histogram's bucket `3` gains one. The same device redeployed a month later and
succeeding immediately starts journey 2 with attempt count 1 — first-time-right. A
device with `serialNumber = "Unknown"` (WMI failure) never gets a chain; its terminal
sessions surface only as that day's disclosed exclusion count.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/DeviceJourneyCalculator.cs` — device key, junk list, grouping, attempt numbers (JourneyVersion 1).
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.DeviceHistory.cs` — persistence + inline seam entry.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.DeviceJourneys.cs` — sweep, tombstone pruning, daily aggregation.
* `src/Backend/AutopilotMonitor.Functions/Functions/Metrics/GetDeviceJourneyFunctions.cs` — read surfaces.
* `src/Backend/AutopilotMonitor.Functions.Tests/DeviceJourneyAndFtrTests.cs` — table-driven grouping/aggregation/attempt-number tests.
* `tasks/insights-expansion-spec.md` — F2 specification and source-data audit (§0.5).
+112
View File
@@ -0,0 +1,112 @@
---
type: Concept
title: Rule Regression Radar (F3)
description: How the radar detects analyze rules whose hit rate suddenly rose (7-day window vs 28-day baseline, Wilson-separated), which suppressions keep it quiet, how alert episodes live in the notification tracker, and which surfaces show them.
resource: src/Backend/AutopilotMonitor.Functions/Helpers/RuleRegressionRadar.cs
tags:
- backend
- rules
- regression-radar
- insights
timestamp: 2026-07-27T16:00:00+02:00
---
# Rule Regression Radar (F3)
Detects "this analyze rule suddenly fires much more often" before helpdesk volume does, and
says what the spike correlates with — with honest statistics and honest wording. Analyze
rules only in v1: their daily stats are audit-verified session-deduplicated
(`FireCount` ≈ distinct sessions with a hit, `SessionsEvaluated` the same-source
denominator). Gather rules are excluded until their per-batch stats dedup is fixed —
never silently included with bad math.
# Schema
## Signal & detection
Per (tenant, ruleId), daily hit rate = `FireCount ÷ SessionsEvaluated` from the
`RuleStats` rows — RATES, not counts (absolute counts spike with every rollout wave).
A regression fires only when ALL hold for the trailing 7-day window vs. the prior
28-day baseline:
1. window has ≥ 5 hit sessions AND ≥ 20 evaluated sessions;
2. window rate ≥ 2× baseline rate;
3. the Wilson 95 % intervals are disjoint in the increase direction
(`MetricsMath.WilsonInterval` / `RateIncreaseSeparated` — deterministic, unit-pinned),
so small-n noise never alerts.
## Suppression (false positives are trust damage)
* Rule entity missing (deleted) — never alerts.
* Rule younger than baseline + window (`CreatedAt` within 35 days — grace period for new
rules; the entity timestamp is the honest gate, a stats-row count would fail on
low-volume tenants whose quiet days simply have no rows).
* Rule edited inside the window (`UpdatedAt` ≥ window start) — edits legitimately change
hit rates.
* Empty baseline denominator (rule disabled throughout the baseline).
* Platform kill switch: app setting `RuleRegressionRadarDisabled=true` skips the radar
(fail-open — it only notifies, never mutates data).
## Alert episodes (tracker keyspace)
An episode lives as `ruleregression|{ruleId}` in the notification-tracker table
(the `tpmpss|` pattern): the row IS the dedup (bell + ops event fire exactly once per
episode), the badge state and the `regressions[]` payload — window/baseline counts,
rates, lift (null for a zero-baseline "new signal" — no finite lift is invented), window
dates, and the dimension concentration captured at first fire. Numbers refresh on every
radar pass while the episode stays active; `FirstNotifiedAt` never moves. The episode
closes (badge clears, dedup re-arms) when the window rate falls back under 1.5× baseline
or fires stop entirely; the tracker's 30-day retention sweep also re-arms, so a
month-old still-burning regression rings once more by design.
## Dimension correlation (on fire only)
Hit sessions (RuleResults scan: tenant partition range + RowKey = ruleId + DetectedAt in
window, intersected with the window's sessions) are compared against all window sessions
across `osBuild`, `model` (manufacturer + model), `agentVersion` and `imeVersion`. The
top value by lift is reported with BOTH shares — wording contract: "X % of affected
sessions are on … vs Y % of all sessions — correlated, not necessarily causal". No value
with ≥ 5 hit sessions at ≥ 2× lift → "no clear dimension concentration"; a correlation
failure also yields no claim. App-version correlation for `ANALYZE-APP-*` rules is a
follow-up (needs rule→app matching semantics), tracked with the gather-rule fix.
## Compute & alert path
Runs in the daily maintenance pass right after the rule-stats aggregation
(`RunRuleRegressionRadarAsync`), anchored on YESTERDAY — whole days only, a partial
today would understate the window rate. One 35-day stats read covers all tenants;
evaluation is the pure `RuleRegressionRadar` core. On a NEW episode:
1. `RuleFrequencyRegression` ops event (category Tenant, Warning) with the full numbers
— dual-registered in the web `OPS_EVENT_TYPES` catalog for alert-rule routing.
2. Tenant bell notification (type `rule_frequency_regression`, Admin audience) with the
same numbers and a deep link to the rule's card (`/analyze-rules#rule-card-{ruleId}`).
A sessions-list pre-filtered to rule hits needs a rule filter the session search does
not have yet — follow-up alongside the gather-rule work.
3. v1 channel: bell + ops event only. Provider fan-out (webhook/Telegram) follows once
alert quality is proven — a noisy first version burns trust permanently.
# Surfaces
* `GET api/metrics/rule-stats` (and the global variant with `tenantId`) carries
`regressions[]` — the active episodes. MCP `get_rule_stats` inherits the block.
* Analyze-rules page: per-rule 30-day fire-count sparkline (densified from the `trend`
rows the endpoint already served) and a red "↑ Regression" badge while an episode is
active, with the full numbers in the tooltip.
# Examples
A rule firing on 2 % of sessions for a month jumps to 15 % over a week (15/100 vs
10/500): lift 7.5×, Wilson-separated → one bell, one ops event, badge until the rate
falls back under 3 %. The same jump on 5/20 sessions with a 12 % baseline stays silent —
the intervals overlap, so the spike is not statistically real yet. A rule that never
fired in the baseline and now hits 15 % alerts as a "new signal" with no lift figure.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/RuleRegressionRadar.cs` — gates, suppression, re-arm, dimension concentration.
* `src/Backend/AutopilotMonitor.Functions/Helpers/MetricsMath.cs` — Wilson interval + separation primitive.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.RuleRegression.cs` — sweep, episode reconcile, bell/ops wiring.
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableHardwareRejectionNotificationTracker.cs` — episode keyspace.
* `src/Backend/AutopilotMonitor.Functions.Tests/RuleRegressionRadarTests.cs` — hand-computed Wilson vectors + every suppression branch.
* `tasks/insights-expansion-spec.md` — F3 specification and source-data audit (§0.5).
@@ -0,0 +1,102 @@
---
type: Concept
title: ServerActions & On-Demand Diagnostics Collection
description: The queue-on-session, deliver-on-ingest ServerAction channel (terminate_session / rotate_config / request_diagnostics), the operator-tier authorization split, the portal's Collect Logs flow including the admin quick-config path, and why the offboarding cascade sweeps the hosted diagnostics prefix.
resource: /src/Backend/AutopilotMonitor.Functions/Functions/Sessions/QueueSessionActionFunction.cs
tags:
- backend
- agent
- web
- server-actions
- diagnostics
- authorization
timestamp: 2026-07-24T00:00:00+02:00
---
# Schema
**Channel.** A `ServerAction` (`Type`, `Reason`, `RuleId`, `Params`, `QueuedAt`) is queued onto
`Sessions.PendingActionsJson` by server-side writers — the admin endpoint
`POST /api/sessions/{sessionId}/actions`, the RuleEngine, maintenance — and delivered
piggy-backed on the agent's next ingest response. No polling channel exists; delivery latency
equals the agent's telemetry cadence (seconds to ~1 minute during enrollment). Delivery is
at-least-once: the agent's `ServerActionDispatcher` dedups on `Type|RuleId|QueuedAt` and
processes a batch sequentially in queue order. Unknown types are logged and skipped, so new
types roll out server-first. Every action emits `server_action_received` and then
`server_action_executed` or `server_action_failed` into the session timeline.
**Authorization (two layers).** The route sits on `EndpointPolicy.TenantAdminOrOperator`
(tenant Admin, tenant Operator, or Global Admin — no Viewer, no Global Reader: it is a write
tier). Because the route serves multiple action kinds, `QueueSessionActionFunction`
re-gates per type via `IsTypeAllowedForCaller`: Operators may queue only
`request_diagnostics`; `terminate_session` and `rotate_config` require
`RequestContext.IsTenantAdmin || IsGlobalAdmin` (403 otherwise). Any route on this tier with
a multi-kind body MUST follow this same in-function re-gating pattern.
**On-demand collection (`request_diagnostics`).** The dispatcher calls
`DiagnosticsPackageService.CreateAndUploadAsync(enrollmentSucceeded: false, suffix:
"server-requested")` — producing `AgentDiagnostics-{sessionId}-{timestamp}-server-requested.zip`.
Passing `enrollmentSucceeded: false` is what makes on-demand collection work under
`DiagnosticsUploadMode=OnFailure`; only `Off` (or an unconfigured destination) blocks it,
surfacing as `server_action_failed`. The `diagnostics_uploaded` event stamps
`DiagnosticsBlobName` + `Destination` onto the Sessions row (latest wins — an on-demand name
is later replaced by the terminal package), and the SignalR `sessionUpdate` delta carries the
field, so the portal's Download button flips live.
**Portal flow (Collect Logs, session detail header).** The button is always rendered; the
disabled states carry tooltips (inactive session, cross-tenant view, missing role, diagnostics
not configured) as a deliberate nudge. Members below Admin cannot read the tenant config, so
the member-readable feature-flags endpoint exposes `diagnosticsUploadConfigured`
(`mode != Off && (customer SAS present || destination Hosted)`). Progress tracking is
sequence-based (backend-assigned, monotonic), never timestamp-based — agent clock skew and
at-least-once redelivery of older actions cannot produce false matches
(`collectLogsLogic.ts`). Cross-tenant views are gated off because `TenantHelper.GetTenantId`
resolves the tenant strictly from the caller's JWT.
**Admin quick-config.** For an unconfigured tenant, a Tenant Admin's click opens a dialog
that switches the tenant to `Destination=Hosted`, `Mode=OnFailure` (GET full config → patch
exactly those two fields → PUT verbatim), then queues `rotate_config` **before**
`request_diagnostics` — the dispatcher's sequential ordering guarantees the agent refetches
the now-enabled config (the `RemoteConfigMerger` applies diagnostics fields live, remote wins)
before building the package. This works against the already-deployed agent fleet; a
"one-time override" variant would not, because the `mode=Off` check is client-side in
`CreateAndUploadAsync` and the deployed handler reads no params.
**Offboarding cascade.** Latest-wins on `DiagnosticsBlobName` means a session can leave more
hosted blobs behind than the deletion manifest references. The cascade's Hosted branch
therefore sweeps the canonical prefix `{tenantId}/AgentDiagnostics-{sessionId}-`
(`HostedDiagnosticsBlobService.DeleteBySessionPrefixAsync`) plus the manifest-named blob as
belt-and-braces; a non-GUID legacy `SessionId` degrades to the single-name delete instead of
poisoning the cascade. The CustomerSas branch is unchanged (delete only with the `d`
permission; the customer's lifecycle rules stay authoritative).
# Examples
Queue an on-demand collection (Admin or Operator, own tenant):
```
POST /api/sessions/{sessionId}/actions
{ "type": "request_diagnostics", "reason": "On-demand log collection from portal" }
→ 202 { "success": true, "queuedAt": "..." }
```
Operator queueing an admin-only type:
```
POST /api/sessions/{sessionId}/actions
{ "type": "terminate_session" }
→ 403 { "message": "Action type 'terminate_session' requires the Tenant Admin role" }
```
Timeline for a successful round trip: `server_action_received` → `server_action_executed`
(with `blobName`) → `diagnostics_uploaded` → Sessions row + SignalR delta carry the new
`DiagnosticsBlobName`.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Functions/Sessions/QueueSessionActionFunction.cs` — allowed types, per-type caller gate.
* `src/Backend/AutopilotMonitor.Functions/Security/EndpointAccessPolicyCatalog.cs` — `TenantAdminOrOperator` tier definition.
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Runtime/ServerActionDispatcher.cs` — dedup, sequential dispatch, telemetry contract.
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Runtime/DiagnosticsPackageService.cs` — mode gates, archive build, suffix.
* `src/Backend/AutopilotMonitor.Functions/Services/Diagnostics/DiagnosticsBlobCascadeDeleter.cs` — hosted prefix sweep.
* `src/Web/autopilot-monitor-web/app/sessions/[sessionId]/components/collectLogsLogic.ts` — button gating matrix, sequence-based progress evaluation.
+124
View File
@@ -0,0 +1,124 @@
---
type: Concept
title: Enrollment Time Attribution (F1)
description: How a terminal session's wall clock is partitioned into named segments with an explicit unattributed remainder, how ESP-blocking apps and reboots are attributed, and where breakdowns and fleet aggregates are computed, stored and served.
resource: src/Backend/AutopilotMonitor.Functions/Helpers/TimeAttributionCalculator.cs
tags:
- backend
- metrics
- time-attribution
- insights
timestamp: 2026-07-26T20:00:00+02:00
---
# Enrollment Time Attribution (F1)
Answers "where did the 47 minutes go?" per session and per fleet. Every number is
traceable to observed events; a missing signal produces "unknown" or an explicit
unattributed remainder — never a guess, and the partition is never normalized to 100 %.
# Schema
## Invariant
For every computed breakdown:
```
sum(span seconds) + UnattributedSeconds == WallClockSeconds == session DurationSeconds
```
`DurationSeconds` is the authoritative wall clock — for WhiteGlove it is part 1 + part 2
WITHOUT the pause, and re-terminal stamps move `CompletedAt`, so `CompletedAt − StartedAt`
is never used (it diverges in ~25 % of terminal sessions). Span seconds are floored;
sub-second dust lands in the unattributed remainder.
## Observation windows
Windows are end-anchored on authoritative timestamps and constructed so their lengths sum
exactly to `DurationSeconds`:
* Non-WhiteGlove: one window `[CompletedAt − DurationSeconds, CompletedAt]`.
* WhiteGlove (ONE session row, two windows): part 2 = `[ResumedAt, CompletedAt]`; part 1 =
`[part1End − (DurationSeconds − part2), part1End]` where `part1End` is anchored on the
`whiteglove_part1_complete` event. The pause between the windows is never attributed.
Fallback anchors set the `WhiteGloveAnchorsIncomplete` quality flag.
## Segments
Spans are sliced at phase-declaration events (only declaration events carry
`Phase != Unknown`) plus `desktop_arrived`, in canonical Sequence order; anchors whose
timestamp runs backward are dropped (`ClockSkewDropped`).
| EnrollmentPhase | Segment key |
|---|---|
| Start / DevicePreparation / DeviceSetup | `device_prep` |
| AppsDevice | `esp_apps` |
| AccountSetup / FinalizingSetup | `identity_hello` |
| AppsUser | `user_esp` |
| Complete, `desktop_arrived` event | `desktop_handoff` |
| Failed | attribution ends — tail stays unattributed |
## ESP-blocking apps (positive evidence only)
Per-app intervals use EVENT timestamps (first started/download event → LAST terminal
event, covering IME retries) — never the agent payload timing, which freezes at the first
terminal transition. Blocking membership joins the app id against the session's latest
list-carrying `esp_config_detected` emission: listed ⇒ blocking; absent ⇒ unknown, never
false (`BlockingSetUnknown` / `BlockingSetTruncated` flags). Critical-path occupancy =
overlap-merged union of blocking intervals clipped to the `esp_apps` spans;
`EspAppsOccupancySeconds` is null (unknown) when no lists were observed.
The fleet what-if bound per app X is `max(0, cpEnd − cpEndWithoutX)` (latest interval end
with vs. without X; the idle gap before X counts — the ESP also waits for X to start).
It is an upper bound by construction; every consumer says "up to", never "you will save".
## Reboots
`system_reboot_detected` is detection-time-stamped by the NEXT agent run; the real boot
moment is its `lastBootUtc` payload. Reboot spans are therefore the event-stream gap
bracketing `lastBootUtc`, deduped per boot, clipped to the observation windows, and carried
as a cross-cutting annotation (they overlap the segment they started in — never a slice of
the partition).
## Persistence & compute
* `SessionTimeBreakdowns` (PK = TenantId, RK = SessionId): spans, reboot spans, blocking
intervals (top 20 + uncapped count), quality flags, `AttributionVersion`. Written once at
the session-terminal single-writer seam (`TableSessionRepository.UpdateSessionStatusAsync`,
Succeeded/Failed only — Incomplete deliberately has no duration). Deleted with its session
(deletion-manifest step) and on tenant offboarding.
* Self-healing 30-day maintenance sweep (2h tick): backfills missing breakdowns, folds
late-terminating sessions into their StartedAt-date aggregates, and enforces version
purity — a stale-version row whose events aged out counts as missing, never mixed.
* `TimeAttributionAggregates` (PK = TenantId or `global`, RK = `{yyyy-MM-dd}|{class}` and
`rolling30|{class}`): per enrollment class (`user_driven`, `whiteglove`, `self_deploying`,
`device_preparation` — never mixed) median/p75/p90 per segment over the fixed six-segment
stack (incl. `unattributed`; a session without a segment contributes 0), plus top blocking
apps (≥5 sessions per app row). Clean/flagged/missing counts are part of every row; rows
exist below the ≥20 UI gate on purpose (the UI needs the n for "insufficient data (n=…)").
Daily rows retain 180 days; `rolling30` rows are refreshed whole (a median of per-day
medians is not the range median — the rolling rows are the honest range statistics).
## Surfaces
* `GET api/sessions/{sessionId}/time-attribution` (MemberRead + tenant query param) —
breakdown row or `breakdown: null` (pre-feature / non-terminal / Incomplete = no lane).
* `GET api/metrics/time-attribution` (MemberRead) / `GET api/global/metrics/time-attribution`
(GlobalReadOrAdmin, optional `tenantId`) — `{ windowDays, classes (rolling), daily }`.
* Web: attribution lane under the session-detail phase timeline; "Time attribution" section
on Fleet Health. MCP: `get_time_attribution` (sessionId mode / fleet mode).
# Examples
A 30-minute user-driven session: `device_prep` 5m → `esp_apps` 10m (occupancy 4m by one
blocking app) → `identity_hello` 8m → `user_esp` 5m → `desktop_handoff` 2m, unattributed 0.
A WhiteGlove session with a 2-day pause reports wall clock = part1 + part2 only; the resume
gap before the first part-2 phase declaration is unattributed, not guessed.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/TimeAttributionCalculator.cs` — calculator + semantics (AttributionVersion 1).
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.TimeAttribution.cs` — persistence.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.Aggregation.cs` — sweep + aggregation.
* `src/Backend/AutopilotMonitor.Functions.Tests/TimeAttributionCalculatorTests.cs` — golden fixtures pinning the invariant.
* `tasks/insights-expansion-spec.md` — F1 specification and source-data audit (§0.5).
+10
View File
@@ -21,6 +21,8 @@ https://docs.autopilotmonitor.com (separate repository).
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait is keyed to the Intune-registered "Sidecar" provider by NAME: co-management's "ConfigMgr" parks the user ESP at "Apps (Identifying)" for days even with TrackingPoliciesCreated=1 (issue #106); the always-on two-arm esp_policy_provider_stalled tripwire (provider_incomplete | sidecar_provider_missing) and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
# Rules
@@ -30,11 +32,19 @@ https://docs.autopilotmonitor.com (separate repository).
# Backend
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
* [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract.
* [ServerActions & On-Demand Diagnostics](backend/server-actions-on-demand-diagnostics.md) - The queue-on-session, deliver-on-ingest action channel, the TenantAdminOrOperator tier with in-function per-type re-gating, the portal's Collect Logs flow (quick-config orders rotate_config before request_diagnostics), and the hosted-prefix sweep in the offboarding cascade.
# MCP Server
* [MCP Docs Corpus](mcp/docs-corpus.md) - How the published customer documentation becomes the `search_docs` corpus: CRLF-safe heading chunking, build-time embedding as a third precomputed section, and why a docs edit needs an MCP redeploy.
# Web Portal
* [Portal Navigation Prefetch & the SWA Runtime Bottleneck](web/portal-navigation-prefetch.md) - Why every navigation Link sets `prefetch={false}`: the App Router's viewport prefetch fans out one RSC request per visible link against an HTTP/1.1 Static Web Apps runtime, and its latency tail starves the six connections that also serve the document and the route chunks. Includes where browser telemetry actually lives.
# Security & Identity
* [MCP OAuth Flow](mcp-oauth-flow.md) - Who authenticates where when connecting an AI client to the MCP server; two identities, three parties plus Entra ID.
+25 -1
View File
@@ -1,6 +1,26 @@
# Log
## 2026-07-23
## 2026-07-27
* **Creation**: Added `web/portal-navigation-prefetch.md` (and a `Web Portal` section to the index) — root cause of the minute-long portal freezes. `GlobalSidebar` and `Navbar` render the whole navigation on every page, so the App Router's default viewport prefetch issued one `?_rsc=` request per visible link against `portal.autopilotmonitor.com` (~10,400 in 14 days). That origin serves the document, the route chunks and the RSC payloads over **HTTP/1.1**, so the six-connection-per-origin cap applies; when the Static Web Apps managed runtime queued the prefetches (p50 91 ms, p95 896 ms, but max 234 s / 21 samples over 30 s), the in-flight prefetches held every socket and the navigation the user wanted could not get one. Measured on the 2026-07-27 17:25 CEST incident: `/admin/presence?_rsc=` 151 s, three siblings finishing within 340 ms of each other at ~37.5 s, all `200`, while the backend API answered in 1.6–2.2 s and `browserTimings` showed `networkDuration` 16 ms against `processingDuration` 225 s. Every navigation Link now sets `prefetch={false}` — these routes are client components behind `ProtectedRoute` that fetch their own data, so the prefetched RSC payload carried no benefit. The doc also records that browser telemetry lives in a **separate** App Insights resource (`AUTOPILOT_MONITOR_WEB_APPINSIGHTS_ID`) that the MCP `query_backend_logs` tool cannot reach, which is why the freezes left no trace in the backend logs.
* **Creation**: Added `backend/rule-regression-radar.md` — F3 regression radar (analyze rules only; gather rules wait for their per-batch stats dedup fix): per (tenant, rule) daily hit rate from `RuleStats`, firing only when the trailing 7-day window has ≥5 hit sessions and ≥20 evaluated sessions, the rate is ≥2× the prior 28-day baseline AND the Wilson 95 % intervals are disjoint (deterministic `MetricsMath` primitive with hand-computed test vectors). Suppressed for deleted rules, rules younger than baseline+window (`CreatedAt` grace), rules edited inside the window (`UpdatedAt`), empty baselines, and the `RuleRegressionRadarDisabled` app-setting kill switch. Alert episodes live as `ruleregression|{ruleId}` tracker rows (one bell + one `RuleFrequencyRegression` ops event per episode; numbers refreshed while active; re-armed under 1.5× baseline, on fire-stop, or by the 30-day tracker retention). On-fire dimension correlation over osBuild/model/agentVersion/imeVersion with the "correlated, not necessarily causal" wording contract. Surfaces: `regressions[]` in the rule-stats response (MCP inherits), the analyze-rules page's 30-day sparkline and regression badge.
* **Creation**: Added `backend/device-journeys.md` — F2 device history / First-Time-Right: the device key (trim + case-fold serial, junk/placeholder exclusion incl. the agent's `Unknown` WMI sentinel), journey grouping over the terminal set Succeeded/Failed/Incomplete (first success ends a journey, a >30-day gap from the previous attempt's CompletedAt starts one, WhiteGlove's single session row is ONE attempt, open sessions are never attempts), the `DeviceHistories` chain (cap 20, written inline at the terminal seam, healed by the sweep, deleted-session refs pruned tombstone-driven, deliberately not in the per-session deletion manifest) and the additive `DeviceJourneyAggregates` daily rows (window rate = ratio of sums — no rolling row needed, 180-day retention). Surfaces: the session-detail "Attempt N" banner with server-computed attempt numbers (virtual-attempt rule for live sessions), the Fleet Health First-time-right section (≥20-journey gate, attempt histogram, repeat-devices violator list), and the MCP `get_device_history` tool (serial mode / fleet FTR mode).
## 2026-07-25
* **Update**: `agent/secureboot-ca2023-detection.md` — review follow-up on 90012d9 hardened the privilege scope, the repo's only privilege-adjustment code and therefore the precedent for any future one. The process token moved from a raw `IntPtr` to a `SafeHandle` assigned by the P/Invoke marshaler, so the CLR owns it from the moment `OpenProcessToken` returns and the critical finalizer releases it even when `Dispose` never runs — previously an async exception between the successful open and the caller's `using` leaked the handle *with the privilege enabled and no owner left to restore it*. `Dispose` became idempotent and now releases the handle in a `finally`, so a throwing restore cannot strand it; the restore obligation is recorded immediately after the successful `AdjustTokenPrivileges` with nothing throwable in between. The whole enable → read → restore sequence runs under a static gate because the privilege is adjusted on the *process* token: a concurrent reader would see "already enabled", correctly skip its own restore, and then lose the privilege mid-read when the first reader disposed, reporting `privilege_denied` on a machine that holds it.
## 2026-07-24
* **Creation**: Added `backend/server-actions-on-demand-diagnostics.md` — the ServerAction channel (queue on `Sessions.PendingActionsJson`, deliver piggy-backed on the next ingest response, at-least-once with `Type|RuleId|QueuedAt` dedup and sequential dispatch) and the on-demand log collection built on it: the portal's always-visible **Collect Logs** button, the new `EndpointPolicy.TenantAdminOrOperator` tier with the mandatory in-function per-type re-gate (Operators queue only `request_diagnostics`), the member-readable `diagnosticsUploadConfigured` feature flag, the admin quick-config path that must queue `rotate_config` before `request_diagnostics` (works against the deployed fleet — a one-time override would not, the mode gate is agent-side), sequence-based progress matching immune to agent clock skew, and the offboarding cascade's hosted-prefix sweep closing the latest-wins orphan-blob gap.
* **Update**: `agent/esp-policy-provider-stall.md` — the customer's follow-up on issue #106 corrected the field-case mechanics: `TrackingPoliciesCreated=1` WAS set under the foreign `ConfigMgr` provider, so the ESP ignores a foreign provider's completion values and waits on the Intune-registered `Sidecar` provider by name (the rename to `Sidecar` creates no tracking entries yet unblocks instantly). The completeness-keyed detector would have missed exactly this — the incident it was built for. The tripwire is now two-armed under one event type discriminated by `reason`: `provider_incomplete` (unchanged dwell on continuously-incomplete providers) and `sidecar_provider_missing` (≥ 1 Setup/Apps provider registered without `Sidecar`, completeness deliberately ignored, clock reset when Sidecar registers later). Sidecar-missing wins the `reason` when both arms cross in one round; the cross-restart fingerprint gains a `sidecarMissing` sentinel; payload adds `reason` and `sidecarMissingForMinutes`. Evidence for the name-keyed wait: Niehaus' documented Intune WAP payload always registers `Sidecar` (ConfigMgr only via a co-management settings policy) and Microsoft's ESP troubleshooting models app tracking exclusively through the Sidecar subkey.
* **Creation**: Added `agent/esp-policy-provider-stall.md` — a customer diagnosed that pre-provisioned devices with the ConfigMgr client as a blocking app occasionally park the user ESP at "Apps (Identifying)" for days: only a `ConfigMgr` provider is registered under `EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders` (no `Sidecar` = IME) and `TrackingPoliciesCreated` never becomes 1 — a wait the CSP defines without any timeout; renaming the key to `Sidecar` unblocks immediately. The agent now ships the always-on `esp_policy_provider_stalled` tripwire (60-s tick, 15-min continuous-incompleteness dwell, one-shot per provider, `StartupEventGate` fingerprint across restarts, purely observational via the InformationalEvent rail — no toggle, no ConfigVersion bump), generalized to any provider in both `Setup/Apps` and `DevicePreparation` kinds rather than hardcoding the ConfigMgr case. Companion change: `SOFTWARE\Microsoft\Windows\Autopilot` joins the gather-rule registry allowlist so tenants can target the subtree with custom rules (e.g. `not_exists` on the Sidecar key).
* **Creation**: Added `agent/secureboot-ca2023-detection.md` — ANALYZE-SEC-001 warned on devices whose firmware already carried the 2023 certificate, because the agent only read the `SecureBoot\Servicing\UEFICA2023Status` registry value, which mirrors the Windows Update rollout rather than the firmware (customer report). The agent now reads the UEFI `db`/`KEK` variables natively (`GetFirmwareEnvironmentVariableExW` with `SeSystemEnvironmentPrivilege` enabled on the process token — first privilege-adjustment code in the repo) and emits per-certificate booleans on `secureboot_status`. Suppression rides on a one-sided marker, `uefiCa2023FirmwareConfirmed`, emitted only on positive confirmation: the engine's `not_exists` precondition fails on any present value, so a two-sided boolean could not distinguish "confirmed" from "checked and missing", while the marker's absence keeps old agents and firmware-missing devices on the v2 path. Rule v3.0.0 is therefore safe to seed before the agent release.
* **Update**: `url-registry.md` — the first release after the alias migration surfaced Front Door caching as a correctness hazard: the alias served the previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) after the origin had rotated, and since manifest and ZIP are independent cache entries, a stale manifest with a fresh ZIP fails the bootstrap SHA-256 check. Route caching on `/agent/*` is now disabled and every blob upload stamps `Cache-Control: no-cache` (CI `Send-Blob`, local `build.ps1`, `versions/*.json` manifests) — all agent-container blobs rotate in place, so nothing there is safely cacheable.
@@ -40,6 +60,10 @@
* **Creation**: Added `agent/endpoint-migration.md` — config-channel agent re-home (`MigrateToApiBaseUrl` on `GET agent/config`, ConfigVersion 33): backend or per-tenant API-URL moves without DNS indirection (impossible due to mTLS/Flex-TLS). Live-fetch-only with cache strip, `AgentEndpointMigrationRules` allowlist validated on both sides, one hop, kill-switch wins. Built as the lesson from the 2026-07-18 WEU→GWC cutover, which hard-cut all in-flight agents.
## 2026-07-26
* **Creation**: Added `backend/time-attribution.md` — the F1 enrollment time attribution: exact-partition invariant against the session's authoritative DurationSeconds, two-window WhiteGlove model, segment mapping from phase-declaration events, positive-evidence ESP-blocking join with "up to" what-if bounds, reboot spans from event gaps around `lastBootUtc`, and the SessionTimeBreakdowns / TimeAttributionAggregates tables with their terminal-seam + maintenance-sweep compute paths and read surfaces (web lane, fleet panel, MCP `get_time_attribution`).
## 2026-07-17
* **Creation**: Added `agent/build-counter-blob.md` — agent build numbers now come from a shared public counter blob (`buildcounter-v{N}.txt`, reserve-before-build with ETag-CAS) so local `build.ps1` and CI `build-agent.yml` builds can never mint the same number. Documents the CI parity fixes (SummaryDialog.exe.config in the ZIP, EXE-vs-manifest version check, hash-oracle update gated on stable cutover) and the Sigstore provenance attestation on CI-built ZIPs.
+88
View File
@@ -0,0 +1,88 @@
---
type: Concept
title: Portal Navigation Prefetch & the SWA Runtime Bottleneck
description: Why every navigation Link in the portal sets prefetch={false} — the App Router's viewport prefetch fires one RSC request per visible link against an HTTP/1.1 Static Web Apps runtime, and its latency tail starves the connection pool that serves the document and the route chunks.
resource: src/Web/autopilot-monitor-web/components/GlobalSidebar.tsx
tags:
- web
- performance
- nextjs
- static-web-apps
- observability
timestamp: 2026-07-27T18:00:00+02:00
---
# Context
The portal is a Next.js 15 App Router application hosted on Azure Static Web Apps. Every page under
the authenticated shell is a client component behind `ProtectedRoute` that fetches its own data from
the Function App (`autopilotmonitor-api-eu`) after mount. The RSC payload a route returns on
navigation therefore carries no page data — only the shell.
`GlobalSidebar` and `Navbar` render the full navigation on every page. With the App Router's default
prefetch behaviour, each visible `<Link>` triggers a background `GET /<route>?_rsc=<hash>` as soon as
it enters the viewport.
# Mechanism
Three properties combine into a portal-wide freeze:
1. **Fan-out.** The navigation is fully rendered on each page load, so a single navigation issues one
RSC request per visible link — measured at ~10,400 requests over 14 days.
2. **Shared origin over HTTP/1.1.** `portal.autopilotmonitor.com` serves the document, the JS route
chunks *and* the RSC payloads, and it negotiates HTTP/1.1 — not HTTP/2. Browsers cap concurrent
connections per origin at six.
3. **A heavy latency tail on the SSR runtime.** RSC responses are prerendered and edge-cached
(`x-nextjs-cache: HIT`, `s-maxage=31536000`), so the median is fast. On a cache miss or cold start
the Static Web Apps managed runtime queues them.
When the tail hits, the in-flight prefetches hold all six connections. The document request and the
route chunks of the page the user actually wants then have no socket available, and the tab shows a
loading spinner or a blank page until the queue drains. Nothing fails — the requests eventually
return `200`, which is why the failure leaves no error trace anywhere.
# Evidence
Incident of 2026-07-27, 17:25–17:29 CEST. Browser-side dependencies during the freeze:
| Request | Duration | Status |
| --- | --- | --- |
| `/admin/presence?_rsc=…` | 151,461 ms | 200 |
| `/gather-rules?_rsc=…` | 52,283 ms | 200 |
| `/progress?_rsc=…` | 37,878 ms | 200 |
| `/analyze-rules?_rsc=…` | 37,559 ms | 200 |
| `/audit?_rsc=…` | 37,539 ms | 200 |
| Backend API, same window | 1,600–2,200 ms | 200 |
The three ~37.5 s responses complete within 340 ms of each other — a queue draining, not independent
slow requests. The corresponding `browserTimings` row for `/geographic-performance` reports
`networkDuration` = 16 ms against `processingDuration` = 225,095 ms: the network delivered the
document immediately and the browser then waited nearly four minutes for the `load` event.
Fourteen-day distribution of RSC prefetches: n = 10,428, p50 = 91 ms, p95 = 896 ms, max = 234,230 ms,
38 over 10 s, 21 over 30 s.
# Diagnosing this class of problem
The backend Application Insights resource is the wrong place to look — a freeze appears there as an
*absence* of requests, never as an error. Browser telemetry lives in a **separate** Application
Insights resource addressed by `AUTOPILOT_MONITOR_WEB_APPINSIGHTS_ID`, which the MCP
`query_backend_logs` tool does not reach. Query it through
`.claude/commands/scripts/query-appinsights.sh` with that ID exported, and read `dependencies`
(client-side fetches, including RSC prefetches) and `browserTimings` (the network/processing split of
a document load).
# Contract
Navigation `<Link>` elements in `GlobalSidebar` and `Navbar` set `prefetch={false}`. The prefetched
RSC payload buys nothing for these routes, so the only remaining cost of a click is the route's JS
chunk, which the browser caches after first use.
Do not remove the prop to "speed up navigation" without first re-measuring the tail in the web
Application Insights resource — the prefetch is cheap only while the runtime is warm.
# Citations
* `src/Web/autopilot-monitor-web/components/GlobalSidebar.tsx` — nav link rendering, rationale comment
* `src/Web/autopilot-monitor-web/components/Navbar.tsx` — top-bar and notification links
* [SWA and Oryx node pinning](../architecture.md) — hosting model for the web app
+90
View File
@@ -0,0 +1,90 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-ESP-006",
"title": "ESP Failure Recovered After User Retry",
"description": "Detects the recovery story: the ESP reported a terminal failure (typically DeviceSetup/Apps), the user most likely pressed 'Try again' on the failure page, the failed step re-ran and completed successfully. The earlier failure did not stick — the enrollment continued past it.",
"severity": "info",
"category": "esp",
"version": "1.0.0",
"enabled": true,
"baseConfidence": 85,
"conditions": [
{
"signal": "advisory",
"source": "event_type",
"eventType": "esp_failure_advisory",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "recovered",
"source": "event_type",
"eventType": "esp_failure_recovered",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "retry",
"source": "event_type",
"eventType": "esp_failure_retry_detected",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "category",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "category",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "failedSubcategory",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "failedSubcategory",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "likelyCulpritApps",
"source": "event_type",
"eventType": "esp_failure_advisory",
"dataField": "likelyCulpritApps",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "minutesSinceFailure",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "minutesSinceFailure",
"operator": "exists",
"value": "",
"required": false
}
],
"confidenceFactors": [
{ "signal": "retry", "condition": "exists", "weight": 10 },
{ "signal": "likelyCulpritApps", "condition": "exists", "weight": 5 }
],
"explanation": "The Enrollment Status Page reported a **failure** in the `{{category}}` phase (`{{failedSubcategory}}` subcategory, most likely app(s): `{{likelyCulpritApps}}`) — and then **recovered**: the failed step left its failed state and completed successfully about `{{minutesSinceFailure}}` min later.\n\nThat pattern is consistent with the user pressing **'Try again'** on the ESP failure page (the profile allows failure-page actions; the agent cannot observe the click itself). The failure the user saw was real, but it did not stick — the ESP re-ran the step, it succeeded, and the enrollment continued.\n\nRead the session timeline as one arc: `esp_failure_advisory` (the failure, with the likely culprit app) → `esp_failure_retry_detected` (the retry) → `esp_failure_recovered` (the recovery). If this pattern repeats across devices for the same app, that app's install is flaky on first attempt — fix the app rather than the devices. Any literal `{{…}}` placeholder means that detail was not recorded for this session.",
"remediation": [
{
"title": "Make the first install attempt stick",
"steps": [
"Check the likely culprit app(s) (`{{likelyCulpritApps}}`) in Intune: an app that fails on the first ESP pass but succeeds on retry usually has a dependency/ordering or detection-timing problem.",
"If the app shows 'never started installing' in esp_failure_settle_started, verify its assignment and applicability rules — ESP counted it as required, but IME never processed it.",
"If the pattern repeats across the tenant, consider removing the app from the ESP blocking list until the first-attempt install is reliable — users are seeing an avoidable failure screen.",
"No device-side action is needed for this session: the failure recovered and the enrollment continued."
]
}
],
"tags": ["esp", "recovery", "try-again", "retry", "continue-anyway", "enrollment"]
}
+33 -5
View File
@@ -2,10 +2,10 @@
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-SEC-001",
"title": "Secure Boot UEFI CA 2023 certificate not deployed",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed.",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed. Since v3 the agent reads the firmware db/KEK variables directly: when the firmware itself confirms the 2023 certificate, the rule is suppressed even if the Windows servicing status has not caught up yet.",
"severity": "warning",
"category": "security",
"version": "2.0.0",
"version": "3.0.0",
"enabled": true,
"baseConfidence": 70,
"preconditions": [
@@ -16,6 +16,14 @@
"operator": "equals",
"value": "false",
"description": "Skip on virtual machines — Secure Boot DB is host-controlled (Hyper-V/VMware/etc), the inside-the-VM UEFICA2023Status registry value is not the right indicator and not actionable for the VM admin."
},
{
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiCa2023FirmwareConfirmed",
"operator": "not_exists",
"value": "",
"description": "Skip when the firmware db itself confirms the Windows UEFI CA 2023 certificate — the marker is only emitted on positive firmware confirmation, so old agents (field absent) and unconfirmed devices keep evaluating the rule. The servicing registry status lags behind actual firmware state (OEM factory image, WinCS, manual deployment, or WU simply not run yet)."
}
],
"conditions": [
@@ -54,13 +62,33 @@
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "db_ca2023_firmware_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiDbHasWindowsUefiCa2023",
"operator": "equals",
"value": "false",
"required": false
},
{
"signal": "kek_2023_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiKekHasMicrosoftKek2kCa2023",
"operator": "equals",
"value": "false",
"required": false
}
],
"confidenceFactors": [
{ "signal": "ca2023_notfound", "condition": "exists", "weight": 20 },
{ "signal": "ca2023_error", "condition": "exists", "weight": 10 }
{ "signal": "ca2023_error", "condition": "exists", "weight": 10 },
{ "signal": "db_ca2023_firmware_missing", "condition": "exists", "weight": 20 },
{ "signal": "kek_2023_missing", "condition": "exists", "weight": 10 }
],
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nCurrent UEFI CA 2023 status: `{{uefiCA2023Status}}`",
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nOn devices running a current agent this finding is verified against the firmware itself: the agent reads the UEFI `db` signature database directly, and this warning is automatically suppressed when the firmware already contains the 2023 certificate — regardless of the Windows servicing status below.\n\nCurrent UEFI CA 2023 servicing status: `{{uefiCA2023Status}}`",
"remediation": [
{
"title": "Deploy UEFI CA 2023 certificate now",
@@ -69,7 +97,7 @@
"Use Intune policy to enable Secure Boot DB update (Windows UEFI CA 2023)",
"Check with the device OEM whether a firmware update is required for the DB update to apply on this model",
"Alternatively, use the WinCS CLI tool from Microsoft for manual deployment",
"Verify deployment: check registry HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated'",
"Verify deployment against the firmware directly: [Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023') — this is authoritative; the registry value HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated' only reflects the Windows Update rollout",
"After DB update: ensure the boot manager is updated to a 2023-signed version via monthly cumulative update"
]
}
+144 -4
View File
@@ -1884,6 +1884,110 @@
"user-experience"
]
},
{
"ruleId": "ANALYZE-ESP-006",
"title": "ESP Failure Recovered After User Retry",
"description": "Detects the recovery story: the ESP reported a terminal failure (typically DeviceSetup/Apps), the user most likely pressed 'Try again' on the failure page, the failed step re-ran and completed successfully. The earlier failure did not stick — the enrollment continued past it.",
"severity": "info",
"category": "esp",
"version": "1.0.0",
"enabled": true,
"baseConfidence": 85,
"conditions": [
{
"signal": "advisory",
"source": "event_type",
"eventType": "esp_failure_advisory",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "recovered",
"source": "event_type",
"eventType": "esp_failure_recovered",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "retry",
"source": "event_type",
"eventType": "esp_failure_retry_detected",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "category",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "category",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "failedSubcategory",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "failedSubcategory",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "likelyCulpritApps",
"source": "event_type",
"eventType": "esp_failure_advisory",
"dataField": "likelyCulpritApps",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "minutesSinceFailure",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "minutesSinceFailure",
"operator": "exists",
"value": "",
"required": false
}
],
"confidenceFactors": [
{
"signal": "retry",
"condition": "exists",
"weight": 10
},
{
"signal": "likelyCulpritApps",
"condition": "exists",
"weight": 5
}
],
"explanation": "The Enrollment Status Page reported a **failure** in the `{{category}}` phase (`{{failedSubcategory}}` subcategory, most likely app(s): `{{likelyCulpritApps}}`) — and then **recovered**: the failed step left its failed state and completed successfully about `{{minutesSinceFailure}}` min later.\n\nThat pattern is consistent with the user pressing **'Try again'** on the ESP failure page (the profile allows failure-page actions; the agent cannot observe the click itself). The failure the user saw was real, but it did not stick — the ESP re-ran the step, it succeeded, and the enrollment continued.\n\nRead the session timeline as one arc: `esp_failure_advisory` (the failure, with the likely culprit app) → `esp_failure_retry_detected` (the retry) → `esp_failure_recovered` (the recovery). If this pattern repeats across devices for the same app, that app's install is flaky on first attempt — fix the app rather than the devices. Any literal `{{…}}` placeholder means that detail was not recorded for this session.",
"remediation": [
{
"title": "Make the first install attempt stick",
"steps": [
"Check the likely culprit app(s) (`{{likelyCulpritApps}}`) in Intune: an app that fails on the first ESP pass but succeeds on retry usually has a dependency/ordering or detection-timing problem.",
"If the app shows 'never started installing' in esp_failure_settle_started, verify its assignment and applicability rules — ESP counted it as required, but IME never processed it.",
"If the pattern repeats across the tenant, consider removing the app from the ESP blocking list until the first-attempt install is reliable — users are seeing an avoidable failure screen.",
"No device-side action is needed for this session: the failure recovered and the enrollment continued."
]
}
],
"tags": [
"esp",
"recovery",
"try-again",
"retry",
"continue-anyway",
"enrollment"
]
},
{
"ruleId": "ANALYZE-ID-001",
"title": "Expected Machine Certificate Not Found",
@@ -2178,10 +2282,10 @@
{
"ruleId": "ANALYZE-SEC-001",
"title": "Secure Boot UEFI CA 2023 certificate not deployed",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed.",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed. Since v3 the agent reads the firmware db/KEK variables directly: when the firmware itself confirms the 2023 certificate, the rule is suppressed even if the Windows servicing status has not caught up yet.",
"severity": "warning",
"category": "security",
"version": "2.0.0",
"version": "3.0.0",
"enabled": true,
"baseConfidence": 70,
"preconditions": [
@@ -2192,6 +2296,14 @@
"operator": "equals",
"value": "false",
"description": "Skip on virtual machines — Secure Boot DB is host-controlled (Hyper-V/VMware/etc), the inside-the-VM UEFICA2023Status registry value is not the right indicator and not actionable for the VM admin."
},
{
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiCa2023FirmwareConfirmed",
"operator": "not_exists",
"value": "",
"description": "Skip when the firmware db itself confirms the Windows UEFI CA 2023 certificate — the marker is only emitted on positive firmware confirmation, so old agents (field absent) and unconfirmed devices keep evaluating the rule. The servicing registry status lags behind actual firmware state (OEM factory image, WinCS, manual deployment, or WU simply not run yet)."
}
],
"conditions": [
@@ -2230,6 +2342,24 @@
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "db_ca2023_firmware_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiDbHasWindowsUefiCa2023",
"operator": "equals",
"value": "false",
"required": false
},
{
"signal": "kek_2023_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiKekHasMicrosoftKek2kCa2023",
"operator": "equals",
"value": "false",
"required": false
}
],
"confidenceFactors": [
@@ -2242,9 +2372,19 @@
"signal": "ca2023_error",
"condition": "exists",
"weight": 10
},
{
"signal": "db_ca2023_firmware_missing",
"condition": "exists",
"weight": 20
},
{
"signal": "kek_2023_missing",
"condition": "exists",
"weight": 10
}
],
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nCurrent UEFI CA 2023 status: `{{uefiCA2023Status}}`",
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nOn devices running a current agent this finding is verified against the firmware itself: the agent reads the UEFI `db` signature database directly, and this warning is automatically suppressed when the firmware already contains the 2023 certificate — regardless of the Windows servicing status below.\n\nCurrent UEFI CA 2023 servicing status: `{{uefiCA2023Status}}`",
"remediation": [
{
"title": "Deploy UEFI CA 2023 certificate now",
@@ -2253,7 +2393,7 @@
"Use Intune policy to enable Secure Boot DB update (Windows UEFI CA 2023)",
"Check with the device OEM whether a firmware update is required for the DB update to apply on this model",
"Alternatively, use the WinCS CLI tool from Microsoft for manual deployment",
"Verify deployment: check registry HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated'",
"Verify deployment against the firmware directly: [Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023') — this is authoritative; the registry value HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated' only reflects the Windows Update rollout",
"After DB update: ensure the boot manager is updated to a 2023-signed version via monthly cumulative update"
]
}
+2 -1
View File
@@ -43,7 +43,8 @@
"prefixes": [
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon"
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
"SOFTWARE\\Microsoft\\Windows\\Autopilot"
]
},
{
@@ -272,13 +272,16 @@ try {
$downloadAttempt++
try {
Write-Log "Download attempt ${downloadAttempt}/${maxDownloadAttempts}"
$downloadStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Invoke-WebRequest `
-Uri $AgentDownloadUrl `
-OutFile $zipPath `
-UseBasicParsing `
-TimeoutSec 30 `
-ErrorAction Stop
Write-Log "Downloaded agent to $zipPath"
$downloadStopwatch.Stop()
$downloadSeconds = [math]::Round($downloadStopwatch.Elapsed.TotalSeconds, 1)
Write-Log "Downloaded agent to $zipPath (took ${downloadSeconds}s)"
break
}
catch {
@@ -0,0 +1,157 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
using SharedEventTypes = AutopilotMonitor.Shared.Constants.EventTypes;
using DeviceInfoCollector = AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo.DeviceInfoCollector;
namespace AutopilotMonitor.Agent.V2.Core.Tests.DeviceInfo
{
/// <summary>
/// Audit Q2 — <see cref="DeviceInfoCollector.RefreshEspConfiguration"/>
/// re-reads the ESP blocking lists and re-emits <c>esp_config_detected</c> exactly when the
/// payload changed: the registry lists grow progressively (one timestamped subkey per CSP
/// status write) and the user-scope lists appear only after sign-in, so the early emissions
/// are structurally partial. The StartupEventGate keeps an unchanged re-read silent.
/// Probe overrides are static slots → serialized with the other probe-driven suites.
/// </summary>
[Collection("SerialThreading")]
public sealed class DeviceInfoCollectorEspRefreshTests
{
private static readonly DateTime T0 = new DateTime(2026, 7, 27, 10, 0, 0, DateTimeKind.Utc);
private const string DeviceApp1 = "11111111-1111-1111-1111-111111111111";
private const string DeviceApp2 = "22222222-2222-2222-2222-222222222222";
private const string UserApp1 = "33333333-3333-3333-3333-333333333333";
private static EspTrackingInfoSnapshot Snapshot(string[] deviceApps, string[] userApps)
{
var all = deviceApps.Concat(userApps).Distinct().ToArray();
return new EspTrackingInfoSnapshot(
msiProductCodes: Array.Empty<string>(),
modernAppPfns: Array.Empty<string>(),
win32AppIds: all,
userWin32AppIds: userApps,
msiCount: 0,
modernCount: 0,
win32Count: all.Length);
}
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> EspConfigEvents(FakeSignalIngressSink ingress) =>
ingress.Posted
.Where(p => p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == SharedEventTypes.EspConfigDetected)
.ToList();
private static IReadOnlyList<string> TypedList(FakeSignalIngressSink.PostedSignal signal, string key)
{
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(signal.TypedPayload);
return Assert.IsAssignableFrom<IReadOnlyList<string>>(data[key]);
}
[Fact]
public void Refresh_reemits_when_lists_grew_and_stays_silent_when_unchanged()
{
using var tmp = new TempDirectory();
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
var ingress = new FakeSignalIngressSink();
var clock = new VirtualClock(T0);
var collector = new DeviceInfoCollector(
"session-1", "tenant-1",
new InformationalEventPost(ingress, clock), logger,
signalIngress: null, clock: clock,
startupGate: new StartupEventGate(tmp.Path, logger));
using var skip = new EspSkipConfigurationProbe.ScopedFullOverride(
_ => new EspFirstSyncSnapshot(
skipUser: false, skipDevice: false,
blockInStatusPage: null, syncFailureTimeoutMinutes: null));
// Early read (agent start / DeviceSetup detection): partial device list, empty user list.
using (new EspTrackingInfoProbe.ScopedOverride(_ => Snapshot(new[] { DeviceApp1 }, Array.Empty<string>())))
{
collector.CollectEspConfiguration();
}
var afterEarly = EspConfigEvents(ingress);
Assert.Single(afterEarly);
Assert.Empty(TypedList(afterEarly[0], "espTrackedUserWin32AppIds"));
// Apps sub-phase opened: the device list grew and sign-in populated the user list.
using (new EspTrackingInfoProbe.ScopedOverride(_ => Snapshot(new[] { DeviceApp1, DeviceApp2 }, new[] { UserApp1 })))
{
collector.RefreshEspConfiguration(DeviceInfoHost.EspConfigTriggerAppsUser);
}
var afterGrowth = EspConfigEvents(ingress);
Assert.Equal(2, afterGrowth.Count);
Assert.Contains(DeviceApp2, TypedList(afterGrowth[1], "espTrackedWin32AppIds"));
Assert.Equal(new[] { UserApp1 }, TypedList(afterGrowth[1], "espTrackedUserWin32AppIds"));
// A later trigger with an UNCHANGED registry state re-reads but emits nothing —
// the StartupEventGate suppresses the identical payload.
using (new EspTrackingInfoProbe.ScopedOverride(_ => Snapshot(new[] { DeviceApp1, DeviceApp2 }, new[] { UserApp1 })))
{
collector.RefreshEspConfiguration(DeviceInfoHost.EspConfigTriggerAccountSetup);
}
Assert.Equal(2, EspConfigEvents(ingress).Count);
}
[Fact]
public async Task Concurrent_refreshes_never_emit_an_older_snapshot_after_a_newer_one()
{
// Codex review: the refresh triggers each run on their own Task.Run — without the
// collector-side lock, two collects can interleave read-before-emit and the LATER
// emission carries the OLDER (smaller) list. The backend keeps the latest
// list-bearing emission as positive evidence, so that would silently shrink it.
// The probe returns a strictly growing list per read; with the read→emit critical
// section, emission order must equal read order — sizes strictly ascending.
using var tmp = new TempDirectory();
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
var ingress = new FakeSignalIngressSink();
var clock = new VirtualClock(T0);
var collector = new DeviceInfoCollector(
"session-1", "tenant-1",
new InformationalEventPost(ingress, clock), logger,
signalIngress: null, clock: clock,
startupGate: new StartupEventGate(tmp.Path, logger));
using var skip = new EspSkipConfigurationProbe.ScopedFullOverride(
_ => new EspFirstSyncSnapshot(
skipUser: false, skipDevice: false,
blockInStatusPage: null, syncFailureTimeoutMinutes: null));
var readCounter = 0;
using var tracking = new EspTrackingInfoProbe.ScopedOverride(_ =>
{
var n = Interlocked.Increment(ref readCounter);
var ids = Enumerable.Range(1, n)
.Select(i => $"{i:d8}-1111-2222-3333-444444444444")
.ToArray();
return Snapshot(ids, Array.Empty<string>());
});
var tasks = Enumerable.Range(0, 12)
.Select(i => Task.Run(() => collector.RefreshEspConfiguration($"stress-{i}")))
.ToArray();
await Task.WhenAll(tasks);
var sizes = EspConfigEvents(ingress)
.Select(e => TypedList(e, "espTrackedWin32AppIds").Count)
.ToList();
// Every refresh saw a strictly larger list (gate never suppresses) and each
// read→emit is atomic, so the emission sequence is exactly 1..12.
Assert.Equal(Enumerable.Range(1, 12), sizes);
}
}
}
@@ -39,6 +39,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Harness
logFolder: Tmp.Path,
patterns: new List<ImeLogPattern>(),
logger: Logger);
// Mirror ImeLogHost wiring: the historic-replay guard judges staleness against the
// (virtual) agent clock so tests stay deterministic.
Tracker.UtcNowProvider = () => Clock.UtcNow;
}
public IReadOnlyList<FakeSignalIngressSink.PostedSignal> DecisionSignals(DecisionSignalKind kind) =>
@@ -0,0 +1,71 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Analyzers
{
/// <summary>
/// LocalAdminAnalyzer.MatchesAllowedEntry — the allowed-list matching contract.
/// Entries without wildcard characters match exactly (case-insensitive); entries
/// containing * or ? are glob patterns with the same semantics as the hardware
/// whitelist. The same matcher is applied to local accounts and profile folders.
/// </summary>
public sealed class LocalAdminAnalyzerTests
{
// -------------------------------------------------------------- literal entries
[Theory]
[InlineData("SupportAdmin", "SupportAdmin")]
[InlineData("supportadmin", "SupportAdmin")]
[InlineData("SUPPORTADMIN", "supportadmin")]
public void Literal_entry_matches_exactly_case_insensitive(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("SupportAdmin2", "SupportAdmin")]
[InlineData("SupportAdmin", "SupportAdmin2")]
[InlineData("Support", "SupportAdmin")]
public void Literal_entry_does_not_substring_match(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
// -------------------------------------------------------------- glob patterns
[Theory]
[InlineData("adm-12345", "adm-*")]
[InlineData("adm-", "adm-*")]
[InlineData("ADM-9F3B2C", "adm-*")]
[InlineData("lapsAdmin01", "lapsAdmin??")]
[InlineData("svc-backup-01", "svc-*-01")]
public void Wildcard_entry_matches_glob(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("administrator2", "adm-*")] // no literal "adm-" prefix
[InlineData("xadm-123", "adm-*")] // pattern is anchored at start
[InlineData("lapsAdmin1", "lapsAdmin??")] // ? is exactly one char
[InlineData("adm.123", "adm-*")] // "-" is literal, not a regex class
public void Wildcard_entry_does_not_overmatch(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Fact]
public void Regex_metacharacters_in_entry_are_treated_literally()
{
// "adm.123" as an entry has no wildcard chars → exact match only,
// the dot must not act as a regex any-char.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.123"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.123"));
// Mixed: wildcard present, remaining metacharacters stay literal.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.*"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.*"));
}
}
}
@@ -405,5 +405,144 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
var started = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorStarted);
Assert.Equal(0, started.Data["noCandidateThresholdMinutes"]);
}
// ============================================================================
// OOBE-state flip (oobe_state_completed) — observational one-shot on the poll
// ============================================================================
// The WinRT OutOfBoxExperienceState flip in_progress->completed is an owner-
// independent desktop corroboration for DAD-starved sessions (4d5a0b78 class).
// Tests drive polls synchronously via PollOnceForTest after Stop() so the real
// timer cannot race, and pin the owner seams to an excluded user so a real
// explorer.exe on the test machine can never set _desktopArrived (which would
// short-circuit PollForDesktop before the flip check).
private static DesktopArrivalDetector BuildFlipDetector(TempDirectory tmp, TraceCapture trace, System.Func<string> oobeProvider)
{
var detector = new DesktopArrivalDetector(new AgentLogger(tmp.Path, AgentLogLevel.Info), noCandidateTimeoutMinutes: 0)
{
OnTraceEvent = trace.Sink,
OobeStateProvider = oobeProvider,
SessionOwnerResolver = _ => "SYSTEM",
ProcessOwnerResolver = _ => "SYSTEM",
};
return detector;
}
/// <summary>Returns the queued states in order, then repeats the last one forever.</summary>
private static System.Func<string> StateQueue(params string[] states)
{
var queue = new Queue<string>(states);
var last = states[states.Length - 1];
return () => queue.Count > 0 ? queue.Dequeue() : last;
}
[Fact]
public void OobeFlip_emits_exactly_once_with_payload()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
// Baseline sample happens synchronously inside Start()/ArmTimer.
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // reads "completed" -> flip
detector.PollOnceForTest(); // still "completed" -> no second emit
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
var flip = trace.Events.Find(e => e.EventType == SharedEventTypes.OobeStateCompleted);
Assert.Equal("in_progress", flip.Data["previousState"]);
Assert.True(flip.Data.ContainsKey("pollCount"));
Assert.True(flip.Data.ContainsKey("minutesSinceStart"));
Assert.False((bool)flip.Data["desktopArrived"]);
Assert.True(flip.Data.ContainsKey("wmiErrorCount"));
Assert.True(flip.Data.ContainsKey("wtsErrorCount"));
}
[Fact]
public void OobeFlip_does_not_emit_when_state_is_already_completed()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest();
detector.PollOnceForTest();
// Productive machines read "completed" from the very first sample — the flip
// event must never fire there (the agent_started field already records the state).
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_does_not_emit_when_in_progress_was_never_seen()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("unavailable", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest();
// Old builds without the WinRT contract report "unavailable"; a later
// "completed" without a positively observed "in_progress" is not a flip.
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_survives_transient_unavailable_between_polls()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "unavailable", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // transient reflection failure — must NOT clear the baseline
detector.PollOnceForTest(); // "completed" -> flip still detected
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_provider_throwing_never_emits_and_never_pollutes_wmi_error_count()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, () => throw new System.InvalidOperationException("boom"));
detector.Start(); // baseline sample throws -> swallowed
detector.Stop();
detector.PollOnceForTest();
detector.PollOnceForTest();
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
// The flip check has its own try/catch — a throw must not fall into
// PollForDesktop's outer catch, which increments the WMI error counter the
// session-4d5a0b78 liveness diagnostics rely on. first_poll carries that counter.
var firstPoll = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorFirstPoll);
Assert.Equal(0, firstPoll.Data["wmiErrorCount"]);
}
[Fact]
public void OobeFlip_one_shot_flag_survives_ResetForRealUserSwitch()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace,
StateQueue("in_progress", "completed", "in_progress", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // flip #1
detector.ResetForRealUserSwitch(); // re-arms + re-samples baseline ("in_progress")
detector.Stop();
detector.PollOnceForTest(); // "completed" again — must NOT re-emit
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
}
}
@@ -290,9 +290,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
[InlineData(@"SAM\SAM\Domains\Account\Users")]
[InlineData(@"SECURITY\Policy\Secrets")]
[InlineData(@"HKLM\SOFTWARE\Microsoft\EnrollmentsSomethingElse")] // prefix spoofing
[InlineData(@"HKLM\SOFTWARE\Microsoft\Windows\AutopilotSomethingElse")] // sibling spoof of the Autopilot prefix
public void RegistryCollector_OffAllowlistPath_IsBlockedBeforeTheHiveIsOpened(string target)
=> AssertBlocked(new RegistryCollector().Execute(Rule("registry", target), Context()), target);
// Session 5d735290: the EnrollmentStatusTracking subtree (ESP policy providers / tracking
// state) is allow-listed so tenants can build gather rules on the co-management stall
// signature the esp_policy_provider_stalled detector reports (e.g. not_exists on the
// Sidecar provider key).
[Theory]
[InlineData(@"SOFTWARE\Microsoft\Windows\Autopilot")]
[InlineData(@"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders\Sidecar")]
public void RegistryGuard_AutopilotEnrollmentTrackingSubtree_IsAllowed(string subPath)
=> Assert.True(GatherRuleGuards.IsRegistryPathAllowed(subPath));
[Theory]
[InlineData("SELECT * FROM Win32_UserAccount")]
[InlineData("SELECT * FROM Win32_ShadowCopy")]
@@ -0,0 +1,166 @@
using System;
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
{
/// <summary>
/// Tracker-level historic-replay guard (session eaf3d8c4 part 2): source lines > 24 h older
/// than now are content from a previous enrollment whose IME log survived on disk. The guard
/// in <c>HandlePatternMatch</c> skips app-mutating actions for such lines so replayed apps
/// never enter <c>_packageStates</c> / phase snapshots / persistence — keeping
/// app_tracking_summary, culprit lists and final-status clean. Script actions pass through
/// (the adapter suppresses their emissions), and SimulationMode bypasses the guard entirely.
/// </summary>
public sealed class ImeLogTrackerHistoricReplayGuardTests
{
private static readonly DateTime Now = new DateTime(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
private static List<ImeLogPattern> TestPatterns() => new List<ImeLogPattern>
{
new ImeLogPattern
{
PatternId = "T-POLICIES", Category = "always", Enabled = true,
Pattern = @"policies discovered = (?<policies>\[.*\])",
Action = "policiesDiscovered",
Parameters = new Dictionary<string, string>(),
},
new ImeLogPattern
{
PatternId = "T-INSTALLED", Category = "always", Enabled = true,
Pattern = @"app (?<id>[\w-]+) installed",
Action = "updateStateInstalled",
Parameters = new Dictionary<string, string>(),
},
new ImeLogPattern
{
PatternId = "T-ESP-PHASE", Category = "always", Enabled = true,
Pattern = @"In EspPhase: (?<espPhase>\w+)",
Action = "espPhaseDetected",
Parameters = new Dictionary<string, string>(),
},
new ImeLogPattern
{
PatternId = "T-SCRIPT-START", Category = "always", Enabled = true,
Pattern = @"script (?<id>[\w-]+) started",
Action = "scriptStarted",
Parameters = new Dictionary<string, string> { ["scriptType"] = "platform" },
},
};
private static ImeLogTracker BuildTracker(TempDirectory tmp)
{
var tracker = new ImeLogTracker(
logFolder: tmp.Path,
patterns: TestPatterns(),
logger: new AgentLogger(tmp.Path, AgentLogLevel.Info));
tracker.UtcNowProvider = () => Now;
return tracker;
}
private const string PoliciesLine = @"policies discovered = [{""Id"":""app-1"",""Name"":""App One""}]";
[Fact]
public void Stale_policies_line_does_not_populate_package_states()
{
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddDays(-7));
Assert.Empty(tracker.GetAllKnownPackageStates());
}
[Fact]
public void Fresh_relog_of_same_policies_is_tracked_after_stale_skip()
{
// IME re-logs the full policy JSON on every app-policy check-in — skipping the
// stale copy cannot starve the current enrollment.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddDays(-7));
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddMinutes(-2));
var all = tracker.GetAllKnownPackageStates();
var pkg = Assert.Single(all);
Assert.Equal("app-1", pkg.Id);
}
[Fact]
public void Stale_state_update_against_fresh_app_is_skipped()
{
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
var stateChanges = 0;
tracker.OnAppStateChanged = (_, __, ___) => stateChanges++;
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddMinutes(-5)); // fresh discovery
tracker.ProcessLogMessageForTest("app app-1 installed", Now.AddDays(-7)); // stale mutation
Assert.Equal(0, stateChanges);
var pkg = Assert.Single(tracker.GetAllKnownPackageStates());
Assert.NotEqual(AppInstallationState.Installed, pkg.InstallationState);
}
[Fact]
public void Stale_esp_phase_line_does_not_block_fresh_phase_detection()
{
// A stale "AccountSetup" would advance _currentPhaseOrder and make the fresh
// enrollment's DeviceSetup bounce as "backward" — the guard must skip it.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
var phases = new List<string>();
tracker.OnEspPhaseChanged = p => phases.Add(p);
tracker.ProcessLogMessageForTest("In EspPhase: AccountSetup", Now.AddDays(-7));
tracker.ProcessLogMessageForTest("In EspPhase: DeviceSetup", Now.AddMinutes(-1));
Assert.Equal(new[] { "DeviceSetup" }, phases);
}
[Fact]
public void Stale_script_line_still_reaches_script_handlers()
{
// Script tracker state is harmless for stale lines (adapter suppresses the
// emissions; the stale-slot hardening covers leftovers) — the guard must not
// swallow script actions.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
var started = new List<ScriptStartedInfo>();
tracker.OnScriptStarted = i => started.Add(i);
tracker.ProcessLogMessageForTest("script aaaa1111-0000-0000-0000-000000000000 started", Now.AddDays(-7));
Assert.Single(started);
}
[Fact]
public void SimulationMode_bypasses_the_guard()
{
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.SimulationMode = true;
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddDays(-7));
Assert.Single(tracker.GetAllKnownPackageStates());
}
[Fact]
public void Lines_without_timestamp_are_not_treated_as_stale()
{
// Non-CMTrace lines carry no source timestamp (entry == null) — never stale.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest(PoliciesLine, sourceTimestampUtc: null);
Assert.Single(tracker.GetAllKnownPackageStates());
}
}
}
@@ -0,0 +1,97 @@
#nullable enable
using System.Text;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using Xunit;
using static AutopilotMonitor.Agent.V2.Core.Monitoring.Interop.WlanNativeMethods;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Interop
{
/// <summary>
/// Tests for the native-WLAN-API WiFi reader. The WLAN state of the test machine is
/// unknown (CI runners/VMs have no WLAN service), so the live path only pins the
/// fail-soft contract; the pure helpers (SSID decode, PHY-type mapping) are pinned
/// exactly — their output feeds wifi_signal_info event payloads and must stay
/// consistent with the strings netsh historically reported.
/// </summary>
public sealed class WifiInfoReaderTests
{
[Fact]
public void TryGetCurrentConnection_never_throws_and_returns_sane_values()
{
var result = WifiInfoReader.TryGetCurrentConnection();
if (result != null)
{
Assert.InRange(result.SignalPercent, 0, 100);
if (result.Channel.HasValue)
Assert.True(result.Channel.Value > 0, "channel must be positive when present");
}
}
// ----- DecodeSsid ---------------------------------------------------
[Fact]
public void DecodeSsid_decodes_utf8_bytes()
{
var raw = new byte[32];
var ssid = Encoding.UTF8.GetBytes("Contoso-Büro");
ssid.CopyTo(raw, 0);
Assert.Equal("Contoso-Büro", WifiInfoReader.DecodeSsid(raw, (uint)ssid.Length));
}
[Fact]
public void DecodeSsid_returns_null_for_empty_or_missing()
{
Assert.Null(WifiInfoReader.DecodeSsid(null, 5));
Assert.Null(WifiInfoReader.DecodeSsid(new byte[32], 0));
}
[Fact]
public void DecodeSsid_clamps_length_to_buffer_size()
{
var raw = Encoding.UTF8.GetBytes("abc");
// A corrupt native length beyond the buffer must not throw.
Assert.Equal("abc", WifiInfoReader.DecodeSsid(raw, 200));
}
[Fact]
public void DecodeSsid_uses_only_declared_length()
{
var raw = new byte[32];
Encoding.UTF8.GetBytes("HomeNet-Rest-Ignored").CopyTo(raw, 0);
Assert.Equal("HomeNet", WifiInfoReader.DecodeSsid(raw, 7));
}
// ----- PhyTypeToRadioType -------------------------------------------
// int parameters (cast inside): DOT11_PHY_TYPE is internal and may not appear in a
// public test method signature (CS0051).
[Theory]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_ofdm, "802.11a")]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_hrdsss, "802.11b")]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_erp, "802.11g")]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_ht, "802.11n")]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_vht, "802.11ac")]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_dmg, "802.11ad")]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_he, "802.11ax")]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_eht, "802.11be")]
public void PhyTypeToRadioType_maps_known_generations(int phyType, string expected)
{
Assert.Equal(expected, WifiInfoReader.PhyTypeToRadioType((DOT11_PHY_TYPE)phyType));
}
[Theory]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_unknown)]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_fhss)]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_dsss)]
[InlineData((int)DOT11_PHY_TYPE.dot11_phy_type_irbaseband)]
[InlineData(9999)]
public void PhyTypeToRadioType_returns_null_for_legacy_or_unknown(int phyType)
{
Assert.Null(WifiInfoReader.PhyTypeToRadioType((DOT11_PHY_TYPE)phyType));
}
}
}
@@ -0,0 +1,40 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Smoke tests for the WinRT <c>SystemSetupInfo.OutOfBoxExperienceState</c> reader.
/// The actual state on the test machine is unknown (CI runners are post-OOBE; Server
/// SKUs may lack the UniversalApiContract entirely), so only the fail-soft contract
/// is pinned: never throw, always return a member of the known value set.
/// </summary>
public sealed class OobeStateReaderTests
{
[Fact]
public void Read_never_throws_and_returns_known_value()
{
var value = OobeStateReader.Read();
Assert.False(string.IsNullOrEmpty(value));
Assert.True(
value == "not_started"
|| value == "in_progress"
|| value == "completed"
|| value == OobeStateReader.Unavailable
|| value.StartsWith("unknown_"),
$"unexpected reader value '{value}'");
}
[Fact]
public void Read_is_stable_across_repeated_calls()
{
// Exercises the cached-PropertyInfo path (second call skips resolution).
var first = OobeStateReader.Read();
var second = OobeStateReader.Read();
Assert.Equal(first, second);
}
}
}
@@ -0,0 +1,208 @@
using System;
using System.Diagnostics;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Executes the generated cleanup script against REAL scratch trees with a real
/// powershell.exe — end-to-end replays of the 2026-07-22 incident mechanics that the
/// string pins in <see cref="CleanupServiceScriptTests"/> cannot prove:
/// <list type="bullet">
/// <item>a locked tree must abort WITHOUT deleting anything (old script force-deleted
/// marker + state around the locks and orphaned the binaries),</item>
/// <item>a straggler agent process (the incident's second instance) must be found by
/// NAME, killed, and the cleanup must then complete — simulated via function
/// shadowing, NEVER by launching a renamed executable (EDR isolation, see the test),</item>
/// <item>concurrent cleanup scripts must be serialized by the lock file.</item>
/// </list>
/// Timings are dialed down via the test-only BuildCleanupScript parameters so each case
/// runs in seconds. Every test uses GUID-unique process/task/tree names so nothing real
/// (a dev-box agent, a real Scheduled Task) can ever be touched.
/// </summary>
[Collection("SerialThreading")]
public class CleanupServiceScriptIntegrationTests : IDisposable
{
private readonly string _root;
private readonly string _basePath;
private readonly string _taskName = $"AmTest-NoSuchTask-{Guid.NewGuid():N}";
private readonly string _fakeProcessName = $"AmTestAgent{Guid.NewGuid().ToString("N").Substring(0, 8)}";
public CleanupServiceScriptIntegrationTests()
{
_root = Path.Combine(Path.GetTempPath(), $"am-cleanup-it-{Guid.NewGuid():N}");
_basePath = Path.Combine(_root, "AutopilotMonitor");
Directory.CreateDirectory(Path.Combine(_basePath, "Agent"));
Directory.CreateDirectory(Path.Combine(_basePath, "State"));
Directory.CreateDirectory(Path.Combine(_basePath, "Logs"));
File.WriteAllText(Path.Combine(_basePath, "Agent", "fake.exe"), "x");
File.WriteAllText(Path.Combine(_basePath, "State", "enrollment-complete.marker"), "x");
File.WriteAllText(Path.Combine(_basePath, "Logs", "agent.log"), "x");
}
public void Dispose()
{
try { Directory.Delete(_root, recursive: true); } catch { }
}
private static string LockPath => Path.Combine(Path.GetTempPath(), "AutopilotMonitor-Cleanup.lock");
private string BuildScript(bool keepLogs = false, int processWaitSeconds = 1)
=> CleanupService.BuildCleanupScript(
agentProcessName: _fakeProcessName,
agentBasePath: _basePath,
keepLogs: keepLogs,
scheduledTaskName: _taskName,
rebootOnComplete: false,
initialDelaySeconds: 0,
processWaitSeconds: processWaitSeconds,
renameRetries: 2,
renameRetryDelaySeconds: 1);
/// <summary>Runs the script synchronously in a real powershell.exe. Returns the script path.</summary>
private string RunScript(string script)
{
var scriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
File.WriteAllText(scriptPath, script);
using (var proc = Process.Start(new ProcessStartInfo
{
FileName = SystemPaths.PowerShell,
Arguments = $"-NoProfile -ExecutionPolicy Bypass -File \"{scriptPath}\"",
UseShellExecute = false,
CreateNoWindow = true,
}))
{
Assert.True(proc.WaitForExit(90_000), "cleanup script did not finish within 90s");
}
return scriptPath;
}
[Fact]
public void Happy_path_deletes_tree_and_removes_lock_and_script()
{
var scriptPath = RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "tree should be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(scriptPath), "script should self-delete");
Assert.False(File.Exists(LockPath), "cleanup lock should be released");
}
[Fact]
public void Locked_tree_aborts_completely_marker_and_files_survive()
{
// The incident regression: with a file locked inside the tree the OLD script
// deleted everything else (incl. the enrollment-complete marker) and left the
// binaries orphaned with the task already gone. The hardened script must treat
// the failed probe-rename as "someone still holds this" and walk away entirely.
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.None))
{
RunScript(BuildScript());
}
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")), "binaries must stay in place");
Assert.True(File.Exists(Path.Combine(_basePath, "State", "enrollment-complete.marker")),
"marker must survive so the next boot retries cleanup");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(LockPath));
}
[Fact]
public void Straggler_agent_process_is_waited_on_and_force_killed_via_name_backstop()
{
// Incident replay of the second-instance mechanics WITHOUT spawning a process:
// an earlier version of this test copied powershell.exe under the fake agent name
// and launched it — textbook MITRE T1036.003 ("Rename Legitimate Utilities"),
// which Defender for Endpoint answered with a machine ISOLATION on the dev box
// (see tasks/lessons.md). Instead, a harness script shadows Get-Process /
// Stop-Process with functions (PowerShell resolves functions before cmdlets, so
// the dot-sourced cleanup script binds to the stubs): Get-Process reports a live
// agent-named straggler on every probe; Stop-Process records the kill. This pins
// the same contract — the script polls by NAME until the deadline, then
// force-kills by NAME before deleting — with zero EDR-visible behavior.
var probeLog = Path.Combine(_root, "probe.log");
var cleanupScriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
File.WriteAllText(cleanupScriptPath, BuildScript(processWaitSeconds: 2));
var harness = @"
$probeLog = 'LOG_PATH'
$global:AmProbeCount = 0
function Get-Process {
[CmdletBinding()]
param([string[]]$Name)
if ($Name -contains 'FAKE_NAME') {
$global:AmProbeCount++
Add-Content -Path $probeLog -Value ('probe ' + $global:AmProbeCount)
return [pscustomobject]@{ Name = 'FAKE_NAME'; Id = 424242 }
}
Microsoft.PowerShell.Management\Get-Process @PSBoundParameters
}
function Stop-Process {
[CmdletBinding()]
param([Parameter(ValueFromPipeline = $true)]$InputObject, [switch]$Force)
process { Add-Content -Path $probeLog -Value ('kill ' + $InputObject.Id + ' force=' + $Force) }
}
. 'SCRIPT_PATH'
"
.Replace("LOG_PATH", probeLog)
.Replace("FAKE_NAME", _fakeProcessName)
.Replace("SCRIPT_PATH", cleanupScriptPath);
RunScript(harness);
var log = File.ReadAllLines(probeLog);
Assert.Contains("kill 424242 force=True", log);
Assert.True(Array.FindAll(log, l => l.StartsWith("probe")).Length >= 2,
"script must poll by name in the wait loop AND re-probe for the kill backstop");
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(LockPath));
}
[Fact]
public void Fresh_foreign_lock_backs_off_without_touching_the_tree()
{
File.WriteAllText(LockPath, "held by another cleanup");
try
{
var scriptPath = RunScript(BuildScript());
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")),
"loser script must not touch the tree — the lock holder owns the deletion");
Assert.True(File.Exists(LockPath), "foreign lock must not be deleted by the loser");
Assert.False(File.Exists(scriptPath), "loser still self-deletes its script");
}
finally
{
try { File.Delete(LockPath); } catch { }
}
}
[Fact]
public void Stale_foreign_lock_is_taken_over_and_cleanup_proceeds()
{
File.WriteAllText(LockPath, "crashed cleanup from a previous boot");
File.SetLastWriteTime(LockPath, DateTime.Now.AddMinutes(-15));
RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "stale lock must be taken over, cleanup must run");
Assert.False(File.Exists(LockPath));
}
[Fact]
public void KeepLogs_variant_deletes_everything_except_logs()
{
RunScript(BuildScript(keepLogs: true));
Assert.True(File.Exists(Path.Combine(_basePath, "Logs", "agent.log")), "Logs must survive");
Assert.False(Directory.Exists(Path.Combine(_basePath, "Agent")));
Assert.False(Directory.Exists(Path.Combine(_basePath, "State")));
Assert.Empty(Directory.GetFileSystemEntries(_basePath, "*.del-*"));
}
}
}
@@ -0,0 +1,135 @@
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Pins the LIFE-F7 hardening invariants of the self-destruct cleanup script
/// (incident 2026-07-22: a second agent instance held exe + loaded DLLs while the
/// old script — waiting on the launcher PID only — force-deleted around the locks
/// AFTER removing the Scheduled Task, orphaning the binaries with no retry path).
/// These are string-level pins on <see cref="CleanupService.BuildCleanupScript"/>:
/// weakening any of them silently re-introduces the orphaned-binaries failure mode.
/// </summary>
public class CleanupServiceScriptTests
{
private const string ProcessName = "AutopilotMonitor.Agent";
private const string BasePath = @"C:\ProgramData\AutopilotMonitor";
private const string TaskName = "AutopilotMonitorAgent";
private static string Script(bool keepLogs = false, bool reboot = false)
=> CleanupService.BuildCleanupScript(ProcessName, BasePath, keepLogs, TaskName, reboot);
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Waits_by_process_name_not_launcher_pid(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #2: name-based wait covers marker-retry / self-update / manual instances
// and is immune to PID reuse. Wait-Process -Id saw exactly one PID and nothing else.
Assert.Contains($"Get-Process -Name '{ProcessName}'", script);
Assert.DoesNotContain("Wait-Process", script);
// Backstop: stragglers are force-killed — their images are deleted right after.
Assert.Contains("Stop-Process -Force", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Never_deletes_the_original_path(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #3: deletion happens exclusively on successfully renamed targets
// ($entry.Renamed). A Remove-Item against the original tree is what deleted
// marker + state around locked binaries in the incident.
Assert.DoesNotContain($"Remove-Item -Path '{BasePath}'", script);
Assert.Contains("Remove-Item -Path $entry.Renamed", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Task_removal_is_gated_on_probe_and_deletion_on_task_gone(bool keepLogs)
{
var script = Script(keepLogs);
// Probe-rename first; only a successful probe reaches task removal, and only a
// removed task reaches deletion. Both failure branches rename BACK so the next
// boot retries with task + marker + files intact.
Assert.Contains("if ($probeOk)", script);
Assert.Contains("if ($taskGone)", script);
Assert.Contains("Rename-Item -Path $entry.Renamed -NewName $entry.Original", script);
// LIFE-F6 task-removal loop + module-independent fallback stay intact.
Assert.Contains("Unregister-ScheduledTask -TaskName $taskName", script);
Assert.Contains("schtasks.exe /Delete /TN $taskName /F", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Concurrent_cleanup_scripts_are_serialized_by_lock_file(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #1: CreateNew is the atomic single-flight gate; the loser exits without
// touching the tree. Stale locks (>10 min) are taken over.
Assert.Contains("AutopilotMonitor-Cleanup.lock", script);
Assert.Contains("[System.IO.FileMode]::CreateNew", script);
Assert.Contains("$lockAge.TotalMinutes -gt 10", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Rename_targets_carry_a_guid_suffix(bool keepLogs)
{
// A fixed '.del' target collides with leftovers of earlier attempts and with a
// concurrent script's rename — the GUID makes every attempt's target unique.
Assert.Contains(".del-' + [Guid]::NewGuid().ToString('N')", Script(keepLogs));
}
[Fact]
public void KeepLogs_variant_excludes_logs_directory()
{
var script = Script(keepLogs: true);
Assert.Contains("-Exclude 'Logs'", script);
Assert.DoesNotContain($"Rename-Item -Path '{BasePath}' -NewName", script);
}
[Fact]
public void Full_variant_renames_the_whole_tree()
{
var script = Script(keepLogs: false);
Assert.Contains($"Rename-Item -Path '{BasePath}' -NewName $renamedPath", script);
Assert.DoesNotContain("-Exclude 'Logs'", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Reboot_flag_controls_shutdown_invocation(bool reboot)
{
var script = Script(keepLogs: false, reboot: reboot);
if (reboot)
Assert.Contains("shutdown.exe /r /t 10", script);
else
Assert.DoesNotContain("shutdown.exe", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Script_and_lock_are_released_in_finally(bool keepLogs)
{
var script = Script(keepLogs);
Assert.Contains("finally", script);
Assert.Contains("$lockStream.Dispose()", script);
Assert.Contains("Remove-Item -Path $scriptPath -Force", script);
}
}
}
@@ -0,0 +1,256 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// Session 4910a5a5 (2026-07-23) — post-terminal recovery story. Field sequence:
/// DeviceSetup/Apps failed (culprit app never started installing) → settle window expired
/// → terminal EspFailureDetected → the user pressed "Try again" ~23 min later → Apps left
/// the failed state → 10/10 installed → DeviceSetup resolved. Pre-fix, that recovery only
/// appeared as a generic <c>subcategory_state_change</c>. Coverage:
/// <list type="bullet">
/// <item><c>esp_failure_retry_detected</c> fires once when the failed subcategory leaves
/// the failed state AFTER the terminal fire.</item>
/// <item><c>esp_failure_recovered</c> fires once when the category then resolves to
/// success.</item>
/// <item>An in-settle-window recovery keeps emitting only
/// <c>esp_failure_settle_recovered</c> — the post-terminal pair stays silent.</item>
/// <item>Apps failures carry <c>LikelyCulpritApps</c> on the EspFailureDetected args,
/// never-started apps ranked first.</item>
/// </list>
/// </summary>
public sealed class EspFailureRecoveryStoryTests
{
private static readonly DateTime Fixed = new DateTime(2026, 7, 23, 4, 30, 0, DateTimeKind.Utc);
private sealed class Fixture : IDisposable
{
public TempDirectory Tmp { get; } = new TempDirectory();
public FakeSignalIngressSink Sink { get; } = new FakeSignalIngressSink();
public List<EspFailureDetectedEventArgs> EspFailures { get; } = new List<EspFailureDetectedEventArgs>();
public ProvisioningStatusTracker Tracker { get; }
public Fixture(Func<IReadOnlyList<AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState>>? packageStatesProbe = null)
{
var clock = new VirtualClock(Fixed);
var post = new InformationalEventPost(Sink, clock);
var logger = new AgentLogger(Tmp.Path, AgentLogLevel.Info);
Tracker = new ProvisioningStatusTracker(
sessionId: "S1",
tenantId: "T1",
post: post,
logger: logger,
appxScanner: new FakeAppxDeploymentFailureScanner(),
backgroundDispatcher: action => action(),
packageStatesProbe: packageStatesProbe);
Tracker.EspFailureDetected += (_, args) => EspFailures.Add(args);
}
public IReadOnlyList<(string EventType, IReadOnlyDictionary<string, object> Data)> CapturedEvents()
{
return Sink.Posted
.Where(p => p.Kind == DecisionSignalKind.InformationalEvent && p.Payload != null)
.Select(p =>
{
var eventType = p.Payload!.TryGetValue(SignalPayloadKeys.EventType, out var et) ? et : "";
var data = p.TypedPayload as IReadOnlyDictionary<string, object>
?? (IReadOnlyDictionary<string, object>)new Dictionary<string, object>();
return (eventType, data);
})
.ToList();
}
/// <summary>Drive the session-4910a5a5 shape up to the terminal fire.</summary>
public void DriveToTerminalDeviceSetupAppsFailure()
{
Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (9 of 10 installed)""}
}");
Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
}
public void Dispose()
{
Tracker.Dispose();
Tmp.Dispose();
}
}
[Fact]
public void PostTerminalRetry_EmitsRetryDetected_Once()
{
using var f = new Fixture();
f.DriveToTerminalDeviceSetupAppsFailure();
Assert.Single(f.EspFailures);
// "Try again": Apps leaves the failed state well after the settle window.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
var retry = Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_retry_detected");
Assert.Equal("DeviceSetup", (string)retry.Data["category"]);
Assert.Equal("Apps", (string)retry.Data["subcategory"]);
Assert.Equal("failed", (string)retry.Data["previousState"]);
Assert.Equal("inProgress", (string)retry.Data["newState"]);
Assert.Equal("Provisioning_DeviceSetup_Apps_Failed", (string)retry.Data["failureType"]);
Assert.True(retry.Data.ContainsKey("minutesSinceFailure"));
// A later unrelated transition does not re-emit.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed)""}
}");
Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_retry_detected");
}
[Fact]
public void PostTerminalRecovery_EmitsRecovered_OnCategorySuccess()
{
using var f = new Fixture();
f.DriveToTerminalDeviceSetupAppsFailure();
// Retry → all apps installed → Windows resolves the category boolean.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed)""}
}");
var recovered = Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_recovered");
Assert.Equal("DeviceSetup", (string)recovered.Data["category"]);
Assert.Equal("Apps", (string)recovered.Data["failedSubcategory"]);
Assert.Equal("Provisioning_DeviceSetup_Apps_Failed", (string)recovered.Data["failureType"]);
Assert.True(recovered.Data.ContainsKey("minutesSinceFailure"));
// Duplicate category-success writes do not re-emit (fire-once).
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed, take 2)""}
}");
Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_recovered");
}
[Fact]
public void InSettleWindowRecovery_KeepsPostTerminalPairSilent()
{
using var f = new Fixture();
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (0 of 1 installed)""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
// Recovery INSIDE the settle window — terminal fire is suppressed.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
f.Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
Assert.Empty(f.EspFailures);
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (1 of 1 installed)""}
}");
var events = f.CapturedEvents();
Assert.Contains(events, e => e.EventType == "esp_failure_settle_recovered");
Assert.DoesNotContain(events, e => e.EventType == "esp_failure_retry_detected");
Assert.DoesNotContain(events, e => e.EventType == "esp_failure_recovered");
}
[Fact]
public void AppsFailure_TerminalArgs_CarryLikelyCulprits_NeverStartedFirst()
{
var neverStarted = BuildApp("app-bg", "Teams Backgrounds v2",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: false);
var startedButPending = BuildApp("app-slow", "Slow Installer",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: true);
var installed = BuildApp("app-ok", "Falcon Sensor",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Installed,
downloadingSeen: true);
using var f = new Fixture(() => new[] { startedButPending, neverStarted, installed });
f.DriveToTerminalDeviceSetupAppsFailure();
var fired = Assert.Single(f.EspFailures);
Assert.Equal(2, fired.LikelyCulpritApps.Count);
// Never-started app ranked first — it starved the ESP gate without any IME event.
Assert.Equal("Teams Backgrounds v2", fired.LikelyCulpritApps[0]);
Assert.Equal("Slow Installer", fired.LikelyCulpritApps[1]);
}
[Fact]
public void NonAppsFailure_TerminalArgs_CarryNoCulprits()
{
var neverStarted = BuildApp("app-bg", "Teams Backgrounds v2",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: false);
using var f = new Fixture(() => new[] { neverStarted });
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Working""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Failed""}
}");
f.Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
var fired = Assert.Single(f.EspFailures);
Assert.Empty(fired.LikelyCulpritApps);
}
private static AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState BuildApp(
string id, string name,
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState state,
bool downloadingSeen)
{
var pkg = new AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState(id, 0);
pkg.UpdateName(name);
pkg.UpdateIntent(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppIntent.Install);
pkg.UpdateTargeted(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppTargeted.Device);
// DownloadingOrInstallingSeen is derived inside UpdateState (Downloading..Installing
// range) — drive it through a Downloading transition instead of setting it directly.
if (downloadingSeen)
pkg.UpdateState(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Downloading);
if (state != AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown
&& state != AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Downloading)
pkg.UpdateState(state);
return pkg;
}
}
}
@@ -0,0 +1,465 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// Session 5d735290 — pure detection logic of <see cref="EspPolicyProviderStallDetector"/>:
/// the CSP completion rule and the dwell/one-shot transition. No registry, no timers.
/// </summary>
public sealed class EspPolicyProviderStallLogicTests
{
private static readonly DateTime T0 = new DateTime(2026, 7, 23, 10, 0, 0, DateTimeKind.Utc);
private static readonly TimeSpan Dwell = TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes);
private static PolicyProviderState SetupApps(string name, int? tracking, string scope = "device") =>
new PolicyProviderState(name, scope, EspPolicyProviderProbe.KindSetupApps, tracking, null);
private static PolicyProviderState DevicePrep(string name, int? installState) =>
new PolicyProviderState(name, "device", EspPolicyProviderProbe.KindDevicePreparation, null, installState);
private static EspPolicyProviderSnapshot Snapshot(params PolicyProviderState[] providers) =>
new EspPolicyProviderSnapshot(providers);
// ------------------------------------------------- completion rule (CSP contract)
[Theory]
[InlineData(1, true)] // TrackingPoliciesCreated=true
[InlineData(0, false)] // explicit false (the CSP default)
[InlineData(null, false)] // value missing — the field case
[InlineData(2, false)] // anything else is not "true"
public void SetupApps_provider_complete_only_at_tracking_1(int? tracking, bool expected)
=> Assert.Equal(expected, EspPolicyProviderProbe.IsProviderComplete(
EspPolicyProviderProbe.KindSetupApps, tracking, null));
[Theory]
[InlineData(2, true)] // NotRequired
[InlineData(3, true)] // Completed
[InlineData(1, false)] // NotInstalled
[InlineData(4, false)] // Error
[InlineData(null, false)]
public void DevicePreparation_provider_complete_at_state_2_or_3(int? state, bool expected)
=> Assert.Equal(expected, EspPolicyProviderProbe.IsProviderComplete(
EspPolicyProviderProbe.KindDevicePreparation, null, state));
[Fact]
public void TryReadInt_tolerates_dword_string_and_bool_forms()
{
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(1));
Assert.Equal(3, EspPolicyProviderProbe.TryReadInt(3L));
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(" 1 "));
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(true));
Assert.Equal(0, EspPolicyProviderProbe.TryReadInt(false));
Assert.Null(EspPolicyProviderProbe.TryReadInt(null));
Assert.Null(EspPolicyProviderProbe.TryReadInt("not-a-number"));
Assert.Null(EspPolicyProviderProbe.TryReadInt(new byte[] { 1 }));
}
// ------------------------------------------------- dwell transition
[Fact]
public void Empty_or_missing_snapshot_clears_dwell_state_and_never_stalls()
{
var previous = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var missing = EspPolicyProviderStallDetector.EvaluateStallTransition(
EspPolicyProviderSnapshot.Empty, previous, new HashSet<string>(), T0, false, T0.AddHours(2), Dwell);
Assert.Empty(missing.UpdatedFirstSeenUtc);
Assert.Empty(missing.NewlyStalled);
Assert.Null(missing.UpdatedSidecarMissingSinceUtc); // arm 2 resets too
Assert.Null(missing.SidecarMissingStalledForMinutes);
var registeredButEmpty = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(), previous, new HashSet<string>(), T0, false, T0.AddHours(2), Dwell);
Assert.Empty(registeredButEmpty.UpdatedFirstSeenUtc);
Assert.Empty(registeredButEmpty.NewlyStalled);
Assert.Null(registeredButEmpty.UpdatedSidecarMissingSinceUtc);
}
[Fact]
public void Incomplete_provider_accrues_dwell_but_does_not_stall_before_threshold()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Equal(T0, round1.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]);
Assert.Empty(round1.NewlyStalled);
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(),
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(14), Dwell);
Assert.Equal(T0, round2.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]); // first-seen preserved
Assert.Empty(round2.NewlyStalled);
Assert.Null(round2.SidecarMissingStalledForMinutes); // 14 min < dwell on arm 2 as well
}
[Fact]
public void Provider_stalls_at_dwell_threshold_with_correct_minutes()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: 0));
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, seen, new HashSet<string>(), null, false, T0.AddMinutes(15), Dwell);
var stalled = Assert.Single(eval.NewlyStalled);
Assert.Equal("ConfigMgr", stalled.Provider.Name);
Assert.Equal(15.0, stalled.StalledForMinutes, precision: 3);
}
[Fact]
public void Completion_or_disappearance_resets_the_dwell_clock()
{
var key = "setupApps|device|ConfigMgr";
var seen = new Dictionary<string, DateTime> { [key] = T0 };
// Completes mid-dwell → dropped from the map.
var completed = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet<string>(), null, false, T0.AddMinutes(10), Dwell);
Assert.Empty(completed.UpdatedFirstSeenUtc);
// Re-appears incomplete much later → dwell restarts from the new now, no stall yet.
var reappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 0)), completed.UpdatedFirstSeenUtc,
new HashSet<string>(), null, false, T0.AddHours(3), Dwell);
Assert.Equal(T0.AddHours(3), reappeared.UpdatedFirstSeenUtc[key]);
Assert.Empty(reappeared.NewlyStalled);
// Disappearance behaves the same as completion.
var disappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("Sidecar", tracking: 1)), new Dictionary<string, DateTime> { [key] = T0 },
new HashSet<string>(), null, false, T0.AddMinutes(20), Dwell);
Assert.False(disappeared.UpdatedFirstSeenUtc.ContainsKey(key));
}
[Fact]
public void Already_fired_provider_is_never_reported_again()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var fired = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "setupApps|device|ConfigMgr" };
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, seen, fired, null, false, T0.AddHours(5), Dwell);
Assert.Empty(eval.NewlyStalled);
Assert.True(eval.UpdatedFirstSeenUtc.ContainsKey("setupApps|device|ConfigMgr")); // still tracked
}
[Fact]
public void Mixed_kinds_and_scopes_are_tracked_under_distinct_keys()
{
var snapshot = Snapshot(
SetupApps("ConfigMgr", tracking: null),
SetupApps("ConfigMgr", tracking: null, scope: "user:S-1-5-21-1111"),
DevicePrep("ConfigMgr", installState: 1));
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Equal(3, eval.UpdatedFirstSeenUtc.Count);
Assert.Contains("setupApps|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
Assert.Contains("setupApps|user:S-1-5-21-1111|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
Assert.Contains("devicePreparation|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
}
[Fact]
public void SidecarRegistered_matches_setupApps_sidecar_only()
{
Assert.True(Snapshot(SetupApps("Sidecar", tracking: 0)).SidecarRegistered);
Assert.True(Snapshot(SetupApps("sidecar", tracking: 1, scope: "user:S-1-5-21-1")).SidecarRegistered);
Assert.False(Snapshot(SetupApps("ConfigMgr", tracking: 1)).SidecarRegistered);
Assert.False(Snapshot(DevicePrep("Sidecar", installState: 3)).SidecarRegistered); // wrong kind
Assert.False(EspPolicyProviderSnapshot.Empty.SidecarRegistered);
}
// ------------------------------------------------- arm 2: sidecar-missing transition
[Fact]
public void Issue106_foreign_provider_complete_but_sidecar_missing_stalls_after_dwell()
{
// The actual field case: ConfigMgr with TrackingPoliciesCreated=1 — "complete" by the
// CSP value contract, arm 1 never fires — yet the ESP waits on the absent Sidecar.
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: 1));
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Empty(round1.NewlyStalled); // arm 1 stays silent...
Assert.Equal(T0, round1.UpdatedSidecarMissingSinceUtc); // ...but arm 2 starts its clock
Assert.Null(round1.SidecarMissingStalledForMinutes);
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(),
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(15), Dwell);
Assert.Empty(round2.NewlyStalled);
Assert.Equal(15.0, round2.SidecarMissingStalledForMinutes!.Value, precision: 3);
}
[Fact]
public void Sidecar_registering_resets_the_sidecar_missing_clock()
{
// ConfigMgr first (legitimate startup ordering) → clock runs...
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Equal(T0, round1.UpdatedSidecarMissingSinceUtc);
// ...Sidecar joins → condition clears, state resets.
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1), SetupApps("Sidecar", tracking: 0)),
round1.UpdatedFirstSeenUtc, new HashSet<string>(),
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(10), Dwell);
Assert.Null(round2.UpdatedSidecarMissingSinceUtc);
Assert.Null(round2.SidecarMissingStalledForMinutes);
// Sidecar gone again much later → clock restarts from the new now, no instant stall.
var round3 = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
round2.UpdatedFirstSeenUtc, new HashSet<string>(),
round2.UpdatedSidecarMissingSinceUtc, false, T0.AddHours(3), Dwell);
Assert.Equal(T0.AddHours(3), round3.UpdatedSidecarMissingSinceUtc);
Assert.Null(round3.SidecarMissingStalledForMinutes);
}
[Fact]
public void Sidecar_missing_one_shot_and_deviceprep_only_scoping()
{
// Already fired → never re-reported, but the clock keeps running.
var latched = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
new Dictionary<string, DateTime>(), new HashSet<string>(), T0, true, T0.AddHours(5), Dwell);
Assert.Null(latched.SidecarMissingStalledForMinutes);
Assert.Equal(T0, latched.UpdatedSidecarMissingSinceUtc);
// DevicePreparation providers alone don't constitute the Setup/Apps wait — arm 2 off.
var devicePrepOnly = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(DevicePrep("ConfigMgr", installState: 1)),
new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Null(devicePrepOnly.UpdatedSidecarMissingSinceUtc);
}
}
/// <summary>
/// Emission surface of the detector via the real <see cref="InformationalEventPost"/> rail.
/// Static probe override → serial collection (same contract as
/// <c>EspConfigDetectedTrackingFieldsTests</c>).
/// </summary>
[Collection("SerialThreading")]
public sealed class EspPolicyProviderStallDetectorEmissionTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 7, 23, 12, 0, 0, DateTimeKind.Utc);
private readonly TempDirectory _tmp = new TempDirectory();
public void Dispose() => _tmp.Dispose();
private (EspPolicyProviderStallDetector sut, FakeSignalIngressSink sink, VirtualClock clock)
BuildDetector(StartupEventGate? gate = null)
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
var sink = new FakeSignalIngressSink();
var clock = new VirtualClock(At);
var post = new InformationalEventPost(sink, clock);
var sut = new EspPolicyProviderStallDetector("S1", "T1", post, logger, clock, gate);
return (sut, sink, clock);
}
private static EspPolicyProviderSnapshot ConfigMgrOnly() => new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps,
trackingPoliciesCreated: null, installationState: null),
});
private static EspPolicyProviderSnapshot SidecarCompleteConfigMgrIncomplete() => new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("Sidecar", "device", EspPolicyProviderProbe.KindSetupApps, 1, null),
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, 0, null),
});
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> StalledEvents(FakeSignalIngressSink sink) =>
sink.Posted.Where(p =>
p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == "esp_policy_provider_stalled").ToList();
[Fact]
public void CustomerScenario_configmgr_without_sidecar_emits_one_warning_after_dwell()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly());
var (sut, sink, clock) = BuildDetector();
sut.Tick(); // first sighting — dwell starts
Assert.Empty(StalledEvents(sink));
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick(); // threshold crossed → emit
var evt = Assert.Single(StalledEvents(sink));
Assert.Equal(DecisionSignalKind.InformationalEvent, evt.Kind);
Assert.Equal("Warning", evt.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", evt.Payload[SignalPayloadKeys.ImmediateUpload]);
Assert.Equal(EspPolicyProviderStallDetector.SourceLabel, evt.Payload[SignalPayloadKeys.Source]);
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
// Both arms cross in the same round (ConfigMgr incomplete AND Sidecar absent) →
// ONE event; sidecar-missing is the dominant reason.
Assert.Equal(EspPolicyProviderStallDetector.ReasonSidecarMissing, data["reason"]);
Assert.Equal(false, data["sidecarRegistered"]);
Assert.Equal(15.0, data["sidecarMissingForMinutes"]);
Assert.Equal(EspPolicyProviderStallDetector.DwellMinutes, data["dwellMinutes"]);
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
var entry = Assert.Single(stalled);
Assert.Equal("ConfigMgr", entry["name"]);
Assert.Equal(EspPolicyProviderProbe.KindSetupApps, entry["kind"]);
Assert.Equal(15.0, entry["stalledForMinutes"]);
}
[Fact]
public void Issue106_configmgr_complete_without_sidecar_still_emits_sidecar_missing()
{
// Regression test for the customer's follow-up on issue #106: TrackingPoliciesCreated=1
// WAS present under ConfigMgr — the provider looks "complete", arm 1 never fires, yet
// the ESP hangs because Sidecar is absent. Arm 2 must catch exactly this.
var configMgrComplete = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps,
trackingPoliciesCreated: 1, installationState: null),
});
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => configMgrComplete);
var (sut, sink, clock) = BuildDetector();
sut.Tick();
Assert.Empty(StalledEvents(sink));
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
var evt = Assert.Single(StalledEvents(sink));
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
Assert.Equal(EspPolicyProviderStallDetector.ReasonSidecarMissing, data["reason"]);
Assert.Equal(false, data["sidecarRegistered"]);
Assert.Equal(15.0, data["sidecarMissingForMinutes"]);
Assert.Empty(Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]));
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
Assert.Equal(true, Assert.Single(providers)["complete"]); // the misleading "complete" is visible
}
[Fact]
public void Sidecar_complete_configmgr_incomplete_reports_only_configmgr_with_sidecar_flag()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => SidecarCompleteConfigMgrIncomplete());
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
var evt = Assert.Single(StalledEvents(sink));
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
Assert.Equal(EspPolicyProviderStallDetector.ReasonProviderIncomplete, data["reason"]); // arm 2 off — Sidecar is there
Assert.Equal(true, data["sidecarRegistered"]);
Assert.False(data.ContainsKey("sidecarMissingForMinutes"));
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
Assert.Equal("ConfigMgr", Assert.Single(stalled)["name"]);
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
Assert.Equal(2, providers.Count); // full table always attached
}
[Fact]
public void Latch_prevents_second_emission_for_the_same_provider()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly());
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(30));
sut.Tick();
sut.Tick();
Assert.Single(StalledEvents(sink));
}
[Fact]
public void Startup_gate_suppresses_same_stall_across_restart_but_reports_new_provider_set()
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
// Run 1: fires and persists the fingerprint.
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly()))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Single(StalledEvents(sink));
}
// Run 2 (simulated restart — fresh detector, gate reloaded from the same state dir):
// identical stall re-accrues dwell but lands on the same fingerprint → suppressed.
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly()))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
// Run 3: a SECOND provider joins the stall → fingerprint changes → reported.
var widened = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, null, null),
new PolicyProviderState("Contoso", "device", EspPolicyProviderProbe.KindSetupApps, 0, null),
});
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => widened))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Single(StalledEvents(sink));
}
}
[Fact]
public void Healthy_or_absent_tracking_never_emits()
{
var healthy = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("Sidecar", "device", EspPolicyProviderProbe.KindSetupApps, 1, null),
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindDevicePreparation, null, 3),
});
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => healthy))
{
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromHours(4));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => EspPolicyProviderSnapshot.Empty))
{
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromHours(4));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
}
}
}
@@ -0,0 +1,143 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
{
/// <summary>
/// Pins the anti-flood semantics of <see cref="EmergencyReporter.TrySendAsync"/> after the
/// delivery-hardening change (2026-07-23): the once-per-session dedup and the per-session
/// budget count DELIVERED reports. The previous mark-before-send permanently suppressed an
/// error key after its first transport failure — fatal for single-shot callers like the 48h
/// emergency break, whose process exits right after the attempt.
/// </summary>
public class EmergencyReporterTests
{
private sealed class FakeApiClient : BackendApiClient
{
public readonly List<TimeSpan?> Timeouts = new List<TimeSpan?>();
public readonly Queue<bool> Results = new Queue<bool>();
public FakeApiClient() : base(
httpClient: new HttpClient(),
baseUrl: "http://localhost",
manufacturer: string.Empty,
model: string.Empty,
serialNumber: string.Empty,
useBootstrapTokenAuth: false,
bootstrapToken: null,
agentVersion: "0.0.0",
logger: null)
{
}
public int Calls => Timeouts.Count;
public override Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
Timeouts.Add(timeout);
return Task.FromResult(Results.Count > 0 && Results.Dequeue());
}
}
private static EmergencyReporter Reporter(FakeApiClient client)
=> new EmergencyReporter(client, "session", "tenant", "0.0.0", logger: null);
[Fact]
public async Task Failed_send_rolls_back_reservation_so_a_later_call_retries()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // first call fails
client.Results.Enqueue(true); // second call succeeds
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
// Old behaviour: second call suppressed as "already reported" ⇒ 1 call.
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Delivered_send_is_deduplicated_per_key()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(1, client.Calls);
}
[Fact]
public async Task Retries_stay_inside_one_reservation_and_stop_on_success()
{
var client = new FakeApiClient();
client.Results.Enqueue(false);
client.Results.Enqueue(true); // second attempt delivers — third must not fire
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, perAttemptTimeout: TimeSpan.FromMilliseconds(50), retryDelay: TimeSpan.Zero);
Assert.Equal(2, client.Calls);
// Delivered ⇒ dedup holds for the session.
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Per_attempt_timeout_is_forwarded_to_the_api_client()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
var timeout = TimeSpan.FromSeconds(15);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 1, perAttemptTimeout: timeout);
Assert.Equal(timeout, Assert.Single(client.Timeouts));
}
[Fact]
public async Task All_attempts_failing_rolls_back_after_exhausting_attempts()
{
var client = new FakeApiClient(); // queue empty ⇒ every attempt fails
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, retryDelay: TimeSpan.Zero);
Assert.Equal(3, client.Calls);
// Rolled back ⇒ a later call may try again (budget + dedup + cooldown restored).
client.Results.Enqueue(true);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(4, client.Calls);
}
[Fact]
public async Task Different_keys_do_not_suppress_each_other_after_rollback()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // break report fails → rollback (incl. cooldown restore)
client.Results.Enqueue(true); // unrelated report must still go out
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.ConfigFetchFailed, "m");
Assert.Equal(2, client.Calls);
}
}
}
@@ -0,0 +1,212 @@
#nullable enable
using System.Collections.Generic;
using System.Text;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Pins the pure parts of the UEFI Secure Boot certificate probe: the ASCII substring
/// search over raw signature-list bytes and the <c>secureboot_status</c> payload shaping
/// (in particular the one-sided <c>uefiCa2023FirmwareConfirmed</c> marker whose ABSENCE
/// is load-bearing for ANALYZE-SEC-001's not_exists precondition). The actual firmware
/// read is only smoke-tested: test machines may be VMs, non-elevated, or legacy BIOS,
/// so only the fail-soft contract is asserted.
/// </summary>
public sealed class UefiSecureBootCertReaderTests
{
// --- ContainsAscii ---
[Fact]
public void ContainsAscii_finds_needle_embedded_in_binary_noise()
{
var bytes = Concat(new byte[] { 0x00, 0xA1, 0x30, 0x82 },
Encoding.ASCII.GetBytes("Windows UEFI CA 2023"),
new byte[] { 0x00, 0xFF });
Assert.True(UefiSecureBootCertReader.ContainsAscii(bytes, "Windows UEFI CA 2023"));
}
[Fact]
public void ContainsAscii_finds_needle_at_the_very_end()
{
var bytes = Concat(new byte[] { 0x01, 0x02 }, Encoding.ASCII.GetBytes("KEK 2K CA 2023"));
Assert.True(UefiSecureBootCertReader.ContainsAscii(bytes, "KEK 2K CA 2023"));
}
[Fact]
public void ContainsAscii_rejects_partial_overlap_and_absent_needle()
{
var bytes = Encoding.ASCII.GetBytes("Windows UEFI CA 2011 something else");
Assert.False(UefiSecureBootCertReader.ContainsAscii(bytes, "Windows UEFI CA 2023"));
// Repeated prefix bytes must not confuse the scan.
var tricky = Encoding.ASCII.GetBytes("ababac");
Assert.True(UefiSecureBootCertReader.ContainsAscii(tricky, "abac"));
}
[Fact]
public void ContainsAscii_handles_null_empty_and_oversized_needle()
{
Assert.False(UefiSecureBootCertReader.ContainsAscii(null, "x"));
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[0], "x"));
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[] { 1, 2 }, ""));
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[] { 1, 2 }, "abc"));
}
// --- Payload shaping ---
[Fact]
public void Marker_emitted_only_when_db_ok_and_ca2023_present()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusOk,
KekStatus = UefiSecureBootCertReader.StatusOk,
DbHasWindowsUefiCa2023 = true,
KekHasMicrosoftKek2kCa2023 = true
});
Assert.Equal(true, data["uefiCa2023FirmwareConfirmed"]);
Assert.Equal("ok", data["uefiFirmwareReadStatus"]);
Assert.Equal(true, data["uefiDbHasWindowsUefiCa2023"]);
Assert.Equal(true, data["uefiKekHasMicrosoftKek2kCa2023"]);
}
[Fact]
public void Marker_absent_when_ca2023_missing_even_though_read_succeeded()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusOk,
KekStatus = UefiSecureBootCertReader.StatusOk,
DbHasWindowsUefiCa2023 = false,
DbHasWindowsProductionPca2011 = true
});
// The marker must be OMITTED, not false — not_exists fails on any present value.
Assert.False(data.ContainsKey("uefiCa2023FirmwareConfirmed"));
Assert.Equal(false, data["uefiDbHasWindowsUefiCa2023"]);
Assert.Equal(true, data["uefiDbHasWindowsProductionPca2011"]);
}
[Fact]
public void Failed_db_read_emits_status_but_no_cert_booleans_and_no_marker()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusPrivilegeDenied,
KekStatus = UefiSecureBootCertReader.StatusPrivilegeDenied
});
Assert.Equal(UefiSecureBootCertReader.StatusPrivilegeDenied, data["uefiFirmwareReadStatus"]);
Assert.False(data.ContainsKey("uefiDbHasWindowsUefiCa2023"));
Assert.False(data.ContainsKey("uefiKekHasMicrosoftKek2kCa2023"));
Assert.False(data.ContainsKey("uefiCa2023FirmwareConfirmed"));
}
[Fact]
public void Db_ok_but_kek_failed_keeps_db_booleans_and_reports_kek_status()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusOk,
KekStatus = UefiSecureBootCertReader.StatusVariableNotFound,
DbHasWindowsUefiCa2023 = true
});
Assert.Equal(UefiSecureBootCertReader.StatusVariableNotFound, data["uefiFirmwareReadStatus"]);
Assert.Equal(true, data["uefiDbHasWindowsUefiCa2023"]);
Assert.Equal(true, data["uefiCa2023FirmwareConfirmed"]);
Assert.False(data.ContainsKey("uefiKekHasMicrosoftKek2kCa2023"));
}
[Fact]
public void Null_snapshot_leaves_payload_untouched()
{
var data = new Dictionary<string, object> { { "uefiSecureBootEnabled", true } };
DeviceInfoCollector.AppendFirmwareCertFields(data, null);
Assert.Single(data);
}
[Fact]
public void DescribeFirmwareVerdict_covers_all_shapes()
{
Assert.Equal("unavailable", DeviceInfoCollector.DescribeFirmwareVerdict(null));
Assert.Equal(UefiSecureBootCertReader.StatusNotUefi, DeviceInfoCollector.DescribeFirmwareVerdict(
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusNotUefi }));
Assert.Equal("CA2023 confirmed", DeviceInfoCollector.DescribeFirmwareVerdict(
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusOk, DbHasWindowsUefiCa2023 = true }));
Assert.Equal("CA2023 missing", DeviceInfoCollector.DescribeFirmwareVerdict(
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusOk, DbHasWindowsUefiCa2023 = false }));
}
// --- Reader smoke (real firmware; machine state unknown) ---
[Fact]
public void Read_never_throws_and_reports_a_status_for_both_variables()
{
var snapshot = UefiSecureBootCertReader.Read();
Assert.NotNull(snapshot);
Assert.False(string.IsNullOrEmpty(snapshot.DbStatus));
Assert.False(string.IsNullOrEmpty(snapshot.KekStatus));
}
/// <summary>
/// SeSystemEnvironmentPrivilege is adjusted on the PROCESS token, so the enable →
/// read → restore sequence is a process-wide side effect. Unserialized, a second
/// reader observes "already enabled", skips its own restore, and then loses the
/// privilege mid-read when the first reader disposes — surfacing as a privilege_denied
/// that disagrees with the other readers. The machine's actual privilege state does
/// not matter here: whatever it is, every reader must report the SAME status.
/// </summary>
[Fact]
public void Read_is_serialized_across_concurrent_callers()
{
const int readers = 8;
var snapshots = new UefiSecureBootCertSnapshot[readers];
var failures = new System.Exception[readers];
var threads = new System.Threading.Thread[readers];
for (var i = 0; i < readers; i++)
{
var index = i;
threads[i] = new System.Threading.Thread(() =>
{
try { snapshots[index] = UefiSecureBootCertReader.Read(); }
catch (System.Exception ex) { failures[index] = ex; }
});
threads[i].Start();
}
foreach (var thread in threads) thread.Join();
Assert.All(failures, Assert.Null);
Assert.All(snapshots, s => Assert.False(string.IsNullOrEmpty(s.DbStatus)));
Assert.All(snapshots, s => Assert.Equal(snapshots[0].DbStatus, s.DbStatus));
Assert.All(snapshots, s => Assert.Equal(snapshots[0].KekStatus, s.KekStatus));
}
private static byte[] Concat(params byte[][] parts)
{
var total = 0;
foreach (var part in parts) total += part.Length;
var result = new byte[total];
var offset = 0;
foreach (var part in parts)
{
System.Array.Copy(part, 0, result, offset, part.Length);
offset += part.Length;
}
return result;
}
}
}
@@ -37,5 +37,41 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
Assert.False(DeviceInfoHost.IsEndTrigger(DecisionSignalKind.EspPhaseChanged, null));
Assert.False(DeviceInfoHost.IsEndTrigger(DecisionSignalKind.InformationalEvent, Phase("FinalizingSetup")));
}
// Audit Q2 — the payload shape the adapter's apps sub-phase declaration produces on the
// ingress rail (InformationalEventPost: eventType + declared phase enum name).
private static IReadOnlyDictionary<string, string> SubPhaseDeclaration(string declaredPhase) =>
new Dictionary<string, string>
{
{ SignalPayloadKeys.EventType, AutopilotMonitor.Shared.Constants.EventTypes.PhaseTransition },
{ AutopilotMonitor.Agent.V2.Core.Telemetry.Events.EventTimelineEmitter.PhaseParamKey, declaredPhase },
};
[Fact]
public void Apps_subphase_declarations_classify_as_esp_config_refresh_triggers()
{
Assert.Equal(DeviceInfoHost.EspConfigTriggerAppsDevice,
DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent, SubPhaseDeclaration("AppsDevice")));
Assert.Equal(DeviceInfoHost.EspConfigTriggerAppsUser,
DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent, SubPhaseDeclaration("AppsUser")));
// Non-apps phase declarations (e.g. the reducer's Finalizing entry) never classify.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent, SubPhaseDeclaration("FinalizingSetup")));
// A phase_transition without a declared phase never classifies.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent,
new Dictionary<string, string> { { SignalPayloadKeys.EventType, AutopilotMonitor.Shared.Constants.EventTypes.PhaseTransition } }));
}
[Fact]
public void AccountSetup_phase_classifies_as_the_user_setup_refresh_trigger()
{
Assert.Equal(DeviceInfoHost.EspConfigTriggerAccountSetup,
DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.EspPhaseChanged, Phase("AccountSetup")));
// DeviceSetup is the enrollment-start trigger, not an ESP-config refresh.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.EspPhaseChanged, Phase("DeviceSetup")));
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.EspPhaseChanged, null));
// Unrelated informational events never classify.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent,
new Dictionary<string, string> { { SignalPayloadKeys.EventType, "esp_config_detected" } }));
}
}
}
@@ -0,0 +1,33 @@
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
{
/// <summary>
/// Audit Q5 — WMI hardware values are trimmed at the source. BIOS serials become device
/// identity keys downstream (F2 device history joins on TenantId + serial), so padding
/// must never leave the agent; whitespace-only values collapse to the "Unknown" sentinel
/// the retry logic already treats as unresolved.
/// </summary>
public sealed class HardwareInfoNormalizeTests
{
[Theory]
[InlineData(" PF60H941 ", "PF60H941")]
[InlineData("PF60H941", "PF60H941")]
[InlineData("\tLENOVO \r\n", "LENOVO")]
[InlineData("Unknown", "Unknown")]
public void Trims_whitespace_at_the_source(string raw, string expected)
=> Assert.Equal(expected, HardwareInfo.NormalizeWmiValue(raw));
[Theory]
[InlineData("")]
[InlineData(" ")]
[InlineData("\t\r\n")]
public void Empty_and_whitespace_only_collapse_to_the_Unknown_sentinel(string raw)
=> Assert.Equal("Unknown", HardwareInfo.NormalizeWmiValue(raw));
[Fact]
public void Null_collapses_to_the_Unknown_sentinel()
=> Assert.Equal("Unknown", HardwareInfo.NormalizeWmiValue(null));
}
}
@@ -0,0 +1,157 @@
using System;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.SignalAdapters;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared.Models;
using SharedEventTypes = AutopilotMonitor.Shared.Constants.EventTypes;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
{
/// <summary>
/// Historic-replay suppression for APP events (session eaf3d8c4 part 2): 147 of 157 app
/// events in that session were replayed from a previous enrollment's IME log — all 10
/// "installed apps" were phantoms. Stale app-state transitions and DO telemetry must be
/// suppressed BEFORE the sub-phase declaration, the timing bookkeeping and the terminal
/// DecisionSignal dedup, so fire-once flags stay unconsumed and no phantom
/// AppInstallCompleted/Failed reaches the engine (which could spuriously re-arm the
/// AdvisoryCompletion window). The script twin lives in
/// <see cref="ImeLogTrackerAdapterHistoricReplayTests"/>.
/// </summary>
public sealed class ImeLogTrackerAdapterHistoricAppReplayTests
{
private static readonly DateTime ClockNow = new DateTime(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
[Fact]
public void Stale_terminal_app_transition_is_fully_suppressed()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-old", 0);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Installed);
Assert.Empty(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Empty(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted));
// Shadow download_progress (status=completed) and the summary snapshot ride the
// same emit — both must be gone too.
Assert.Empty(f.InfoEvents(SharedEventTypes.DownloadProgress));
Assert.Empty(f.InfoEvents(SharedEventTypes.AppTrackingSummary));
// Timing bookkeeping must not be poisoned (feeds FinalStatusBuilder).
Assert.False(adapter.AppTimings.ContainsKey("app-old"));
Assert.Single(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void Fresh_run_after_suppressed_stale_run_fires_signal_timing_and_subphase()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-X", 0);
// Fresh ESP phase so the sub-phase declaration is armed.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-5);
adapter.TriggerEspPhaseFromTest("DeviceSetup");
// Stale replay of the same app — suppressed, must consume NEITHER the sub-phase
// fire-once NOR the terminal-signal dedup for this appId.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Installed);
Assert.Empty(f.InfoEvents(SharedEventTypes.PhaseTransition));
Assert.Empty(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted));
// The app genuinely runs in THIS enrollment.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-2);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Installing);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-1);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Installed);
Assert.Single(f.InfoEvents(SharedEventTypes.PhaseTransition)); // sub-phase fired fresh
Assert.Single(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted)); // signal fired fresh
var timing = adapter.AppTimings["app-X"];
Assert.Equal(ClockNow.AddMinutes(-2), timing.StartedAtUtc); // fresh stamps, not stale
Assert.Equal(ClockNow.AddMinutes(-1), timing.CompletedAtUtc);
}
[Fact]
public void Stale_do_telemetry_is_suppressed_fresh_emits()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-do", 0);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerDoTelemetryFromTest(app);
Assert.Empty(f.InfoEvents(SharedEventTypes.DoTelemetry));
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-1);
adapter.TriggerDoTelemetryFromTest(app);
Assert.Single(f.InfoEvents(SharedEventTypes.DoTelemetry));
}
[Fact]
public void Oneshot_summary_is_shared_between_script_and_app_suppressions()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "aaa11111",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
adapter.TriggerAppStateFromTest(new AppPackageState("app-old", 0),
AppInstallationState.Installing, AppInstallationState.Installed);
Assert.Single(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void FutureSkew_app_transition_still_emits()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-skew", 0);
// Mid-enrollment clock jump — genuine current activity, stamped to the clock.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddHours(2);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Installed);
var info = Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Equal(ClockNow, info.OccurredAtUtc);
Assert.Equal("true", info.Payload!["derivedTimestamp"]);
Assert.Single(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted));
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void SimulationMode_bypasses_suppression_for_apps_and_scripts()
{
// The --replay-log-dir dev tool intentionally feeds historic lines.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.SimulationMode = true;
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerAppStateFromTest(new AppPackageState("app-sim", 0),
AppInstallationState.Installing, AppInstallationState.Installed);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "bbb22222",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
}
}
@@ -0,0 +1,257 @@
using System;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.SignalAdapters;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using SharedEventTypes = AutopilotMonitor.Shared.Constants.EventTypes;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
{
/// <summary>
/// Historic-replay handling (session eaf3d8c4): a previous enrollment's IME log surviving
/// on disk made the agent replay week-old script activity — 156 phantom script_completed
/// events with ~170 h durations (clock-clamped completion minus RAW stale start) and an
/// immediate-upload flood. Script events whose source line is &gt; 24 h stale are now
/// suppressed entirely (one-shot <c>historic_ime_replay_detected</c> summary instead),
/// and durations are computed from a timeline-consistent timestamp pair with a 24 h
/// plausibility backstop.
/// </summary>
public sealed class ImeLogTrackerAdapterHistoricReplayTests
{
private static readonly DateTime ClockNow = new DateTime(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
// ---------------------------------------------------------------------
// Suppression: stale (> 24 h past) source lines are a previous enrollment
// ---------------------------------------------------------------------
[Fact]
public void Stale_completion_is_suppressed_and_emits_oneshot_summary()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var ancient = ClockNow.AddDays(-7);
f.Tracker.LastMatchedLogTimestamp = ancient;
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "d78c1822",
ScriptType = "platform",
ExitCode = 0,
Result = "Failed",
StartedAtUtc = ancient.AddMinutes(-30),
// Bootstrap marker in stdout — week-old evidence must not drive the
// bootstrap-detected trace either.
Stdout = "Bootstrap script version: v2.0",
});
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptFailed));
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptTimeoutSuspected));
Assert.Empty(f.InfoEvents(SharedEventTypes.AgentTrace));
var summary = f.InfoEvent(SharedEventTypes.HistoricImeReplayDetected);
Assert.Equal(ancient.ToString("o"), summary.Payload!["earliestRejectedSourceTimestamp"]);
}
[Fact]
public void Summary_is_oneshot_across_started_and_completed_suppressions()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerScriptStartedFromTest(new ScriptStartedInfo { PolicyId = "aaa11111", ScriptType = "platform" });
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7).AddMinutes(1);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "aaa11111",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7).AddMinutes(2);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "bbb22222",
ScriptType = "remediation",
ScriptPart = "detection",
ExitCode = 0,
ComplianceResult = "True",
});
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptStarted));
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptCompleted));
// Exactly ONE summary regardless of how many replayed lines followed — and it dates
// the replay window from the FIRST (earliest) suppressed line.
var summary = Assert.Single(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
Assert.Equal(ClockNow.AddDays(-7).ToString("o"), summary.Payload!["earliestRejectedSourceTimestamp"]);
}
[Fact]
public void Fresh_events_after_suppression_emit_normally()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "aaa11111",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptCompleted));
// The agent catches up to current log content — normal emission resumes.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-2);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "ccc33333",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
StartedAtUtc = ClockNow.AddMinutes(-3),
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Equal("ccc33333", info.Payload!["policyId"]);
Assert.Equal("60.00", info.Payload["durationSeconds"]);
}
[Fact]
public void Clock_fallback_without_source_timestamp_still_emits()
{
// derivedFromClock with NO rejected source ts (synthetic/callback path) is not a
// replay — must emit, and must not trigger the summary.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "ddd44444",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Equal("true", info.Payload!["derivedTimestamp"]);
Assert.False(info.Payload.ContainsKey("rejectedSourceTimestamp"));
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
// ---------------------------------------------------------------------
// Duration: timeline-consistent pair + plausibility backstop
// ---------------------------------------------------------------------
[Fact]
public void FutureSkew_completion_emits_with_clamped_stamp_and_raw_pair_duration()
{
// Mid-enrollment clock jump (WhiteGlove +1h CMTrace skew case): the source ts is
// rejected as future-skewed — NOT suppressed — and the duration must come from the
// raw source pair (both ends on the same skewed timeline), not clamped-now − raw-start.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var skewed = ClockNow.AddHours(2);
f.Tracker.LastMatchedLogTimestamp = skewed;
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "eee55555",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
StartedAtUtc = skewed.AddSeconds(-360),
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Equal(ClockNow, info.OccurredAtUtc); // stamp: clamped to clock
Assert.Equal("true", info.Payload!["derivedTimestamp"]);
Assert.Equal(skewed.ToString("o"), info.Payload["rejectedSourceTimestamp"]);
Assert.Equal("360.00", info.Payload["durationSeconds"]); // duration: raw pair
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void FutureSkew_failed_platform_script_timeout_heuristic_uses_raw_pair()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var skewed = ClockNow.AddHours(2);
f.Tracker.LastMatchedLogTimestamp = skewed;
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "fff66666",
ScriptType = "platform",
ExitCode = 0,
Result = "Failed",
StartedAtUtc = skewed.AddMinutes(-26), // past the 25-min suspicion threshold
});
var suspected = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptTimeoutSuspected));
Assert.Equal("1560.00", suspected.Payload!["durationSeconds"]);
Assert.Equal(ClockNow, suspected.OccurredAtUtc);
}
[Fact]
public void Implausible_duration_above_24h_is_omitted()
{
// Fresh (accepted) completion paired with an ancient start — e.g. a stale pending
// slot a legacy path left behind. The duration would be a cross-run lie; omit it.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-2);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "abc12345",
ScriptType = "platform",
ExitCode = 0,
Result = "Failed",
StartedAtUtc = ClockNow.AddHours(-30),
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptFailed));
Assert.False(info.Payload!.ContainsKey("durationSeconds"));
Assert.False(info.Payload.ContainsKey("durationBasis"));
// The 30 h "duration" must not read as a hung script either.
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptTimeoutSuspected));
}
// ---------------------------------------------------------------------
// Tracker slot hardening: stale pending start must not pair with a fresh run
// ---------------------------------------------------------------------
[Fact]
public void Fresh_start_discards_stale_pending_slot_from_previous_enrollment()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
// Previous enrollment: replayed start line whose matching result never appeared
// (the script was mid-run when that enrollment ended).
f.Tracker.SeedPendingPlatformScriptForTesting("beef7777", exitCode: null, exitObservedAtUtc: null,
startedAtUtc: ClockNow.AddDays(-7));
// Same policy genuinely re-runs now — the fresh start must begin a NEW execution.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddSeconds(-90);
f.Tracker.HandlePlatformScriptStarted("beef7777");
f.Tracker.LastMatchedLogTimestamp = ClockNow;
f.Tracker.CompletePlatformScriptFromImeResultForTesting("beef7777", "Success", ClockNow);
var info = f.InfoEvent(SharedEventTypes.ScriptCompleted);
// Without the hardening the slot kept the 7-day-old start: the duration would have
// been dropped by the 24 h backstop (or, before that, shown as ~168 h).
Assert.Equal("90.00", info.Payload!["durationSeconds"]);
}
}
}
@@ -243,5 +243,87 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
Assert.Equal(firstStamp, adapter.AppTimings["app-P"].StartedAtUtc);
}
[Fact]
public void Terminal_to_active_retry_rearms_CompletedAt_and_final_terminal_covers_total_occupancy()
{
// Audit Q1: the IME retry path (Error -> Downloading -> Installed is permitted by
// AppPackageState.UpdateState). Pre-fix, CompletedAt froze at the first Error, so
// the final Installed payload carried the first failure's completedAt/duration.
using var f = new ImeLogTrackerAdapterFixture(T0);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-R", 0);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Installing);
f.Clock.Advance(TimeSpan.FromSeconds(5));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Error);
f.Clock.Advance(TimeSpan.FromSeconds(10));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Error, AppInstallationState.Downloading);
// Re-armed while active again: the retry's download event carries no completedAt,
// and the FIRST start stamp is kept.
Assert.Null(adapter.AppTimings["app-R"].CompletedAtUtc);
var retryDownload = f.InfoEvent(SharedEventTypes.AppDownloadStarted);
Assert.False(retryDownload.Payload!.ContainsKey("completedAt"));
Assert.False(retryDownload.Payload.ContainsKey("durationSeconds"));
Assert.Equal(T0.ToString("o"), retryDownload.Payload["startedAt"]);
f.Clock.Advance(TimeSpan.FromSeconds(20));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Downloading, AppInstallationState.Installed);
var timing = adapter.AppTimings["app-R"];
Assert.Equal(T0, timing.StartedAtUtc);
Assert.Equal(T0.AddSeconds(35), timing.CompletedAtUtc);
Assert.Equal(35.0, timing.DurationSeconds);
var completed = Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Equal("35.00", completed.Payload!["durationSeconds"]);
Assert.Equal(T0.AddSeconds(35).ToString("o"), completed.Payload["completedAt"]);
}
[Fact]
public void Skipped_then_retried_app_emits_positive_duration_from_retry_start()
{
// Production regression (session 66f1c5e1, audit Q1): a user-scope app evaluated
// Skipped during DeviceSetup (terminal, no start observed), retried after sign-in
// (Skipped -> Downloading is permitted), Installed later. Pre-fix payload:
// completedAt frozen at the early Skip + startedAt from the retry ->
// durationSeconds=-2342. Post-fix: retry re-arms, duration covers retry-start ->
// final terminal.
using var f = new ImeLogTrackerAdapterFixture(T0);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-S", 0);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Skipped);
Assert.Null(adapter.AppTimings["app-S"].StartedAtUtc);
Assert.Equal(T0, adapter.AppTimings["app-S"].CompletedAtUtc);
f.Clock.Advance(TimeSpan.FromSeconds(60));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Skipped, AppInstallationState.Downloading);
f.Clock.Advance(TimeSpan.FromSeconds(30));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Downloading, AppInstallationState.Installed);
var timing = adapter.AppTimings["app-S"];
Assert.Equal(T0.AddSeconds(60), timing.StartedAtUtc);
Assert.Equal(T0.AddSeconds(90), timing.CompletedAtUtc);
Assert.Equal(30.0, timing.DurationSeconds);
var completedEvents = f.InfoEvents(SharedEventTypes.AppInstallComplete);
Assert.Equal(2, completedEvents.Count);
Assert.Equal("30.00", completedEvents[1].Payload!["durationSeconds"]);
}
[Fact]
public void Inverted_endpoints_yield_null_DurationSeconds()
{
// Negative guard (audit Q1): endpoints stamped from out-of-order source lines must
// never surface as a negative durationSeconds (observed -2342s in production). The
// payload emitter omits the field when DurationSeconds is null.
var inverted = new AppInstallTiming(startedAtUtc: T0.AddSeconds(100), completedAtUtc: T0);
Assert.Null(inverted.DurationSeconds);
var wellOrdered = new AppInstallTiming(startedAtUtc: T0, completedAtUtc: T0.AddSeconds(100));
Assert.Equal(100.0, wellOrdered.DurationSeconds);
}
}
}
@@ -1,5 +1,7 @@
using System;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Termination;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Termination
@@ -70,5 +72,82 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Termination
Assert.Equal(0, bootToStart, 0);
Assert.Equal(10, uptime, 0);
}
// --- DescribeLateStart: outcome-calibrated severity + phrasing (fleet 2026-07-28) ---
private static void Describe(
EnrollmentTerminationOutcome outcome, string oobeState,
out EventSeverity severity, out string message, out string note)
=> ObservationCoverage.DescribeLateStart(
outcome, oobeState, bootToStartSeconds: 34 * 60, uptimeSeconds: 30,
out severity, out message, out note);
[Fact]
public void Failed_outcome_is_a_warning_with_the_hung_script_post_mortem_note()
{
Describe(EnrollmentTerminationOutcome.Failed, "in_progress",
out var severity, out var message, out var note);
Assert.Equal(EventSeverity.Warning, severity);
Assert.Contains("low coverage of the enrollment window", message);
Assert.Contains("post-mortem", note);
Assert.Contains("hung ahead of the bootstrap", note);
}
[Fact]
public void TimedOut_outcome_is_a_warning_like_a_failure()
{
Describe(EnrollmentTerminationOutcome.TimedOut, "in_progress",
out var severity, out _, out var note);
Assert.Equal(EventSeverity.Warning, severity);
Assert.Contains("post-mortem", note);
}
[Fact]
public void Succeeded_outcome_is_an_info_about_tail_only_coverage_without_alarm_framing()
{
// The 659c3a90 shape: healthy enrollment, bootstrap merely ran late in the ESP queue.
Describe(EnrollmentTerminationOutcome.Succeeded, "in_progress",
out var severity, out _, out var note);
Assert.Equal(EventSeverity.Info, severity);
Assert.Contains("succeeded", note);
Assert.Contains("IME log replay", note);
Assert.DoesNotContain("post-mortem", note);
Assert.DoesNotContain("hung", note);
}
[Fact]
public void Oobe_completed_at_start_phrases_a_pure_post_mortem()
{
Describe(EnrollmentTerminationOutcome.Failed, "completed",
out _, out var message, out _);
Assert.Contains("OOBE was already completed when the agent started", message);
}
[Fact]
public void Oobe_in_progress_at_start_phrases_the_live_tail()
{
Describe(EnrollmentTerminationOutcome.Failed, "in_progress",
out _, out var message, out _);
Assert.Contains("OOBE was still in progress at agent start", message);
Assert.Contains("final 30s live", message);
}
[Theory]
[InlineData("not_started")] // flips in after a mid-OOBE reboot — not interpretable
[InlineData("unavailable")] // WinRT contract absent / read failed
[InlineData("unknown_3")] // unexpected enum value
public void Uninterpretable_oobe_states_add_no_claim_to_the_message(string oobeState)
{
Describe(EnrollmentTerminationOutcome.Failed, oobeState,
out _, out var message, out _);
Assert.DoesNotContain("OOBE", message);
Assert.EndsWith("low coverage of the enrollment window.", message);
}
}
}
@@ -10,9 +10,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// the <c>app_tracking_summary</c> terminal event.
/// <para>
/// <b>StartedAtUtc</b> is recorded on the first lifecycle transition into Downloading /
/// Installing / InProgress. <b>CompletedAtUtc</b> is recorded on the first transition to a
/// terminal state (Installed / Skipped / Postponed / Error). Both are "set-once" — subsequent
/// identical-state events do not overwrite an earlier stamp.
/// Installing / InProgress and never overwritten. <b>CompletedAtUtc</b> is recorded on a
/// transition to a terminal state (Installed / Skipped / Postponed / Error) and re-armed
/// to <c>null</c> when the app goes terminal → active again (IME retry / re-evaluation,
/// audit Q1) — so the final terminal event carries first-start → last-terminal.
/// </para>
/// </summary>
public sealed class AppInstallTiming
@@ -30,10 +31,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// <summary>
/// Elapsed time from first Downloading/Installing transition to terminal state, in
/// seconds. <c>null</c> when either endpoint is missing (e.g. app still installing,
/// or the adapter missed the start event because it came online mid-install).
/// or the adapter missed the start event because it came online mid-install) — and
/// <c>null</c> when the endpoints are inverted (CompletedAt before StartedAt: log lines
/// from different source files can be processed out of chronological order; observed in
/// production as durationSeconds=-2342 on a Skipped-then-retried user-scope app,
/// audit Q1). A negative duration is never a real measurement, so it is not emitted.
/// </summary>
public double? DurationSeconds =>
StartedAtUtc.HasValue && CompletedAtUtc.HasValue
StartedAtUtc.HasValue && CompletedAtUtc.HasValue && CompletedAtUtc.Value >= StartedAtUtc.Value
? (CompletedAtUtc.Value - StartedAtUtc.Value).TotalSeconds
: (double?)null;
}
@@ -208,39 +208,60 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
}
else
{
// Platform script: AgentExecutor.log entries create/enrich, IME log entries also create
if (!_pendingPlatformScripts.ContainsKey(id))
{
// A fresh start for this policy (no pending entry) begins a NEW execution.
// Clear any emitted-marker from a prior run so IME re-evaluations / retries of
// the same platform-script policy within one agent lifetime are not silently
// deduped away. Within a single run the start always precedes exit/result, so
// this never clears the current run's own marker.
_platformScriptResultEmitted.Remove(id);
_pendingPlatformScripts[id] = new ScriptExecutionState
{
PolicyId = id,
ScriptType = "platform",
// Prefer the source CMTrace timestamp so a script that started before the
// agent launched (replayed log content) is dated to its real start, not now.
// Set only at slot creation: the start line fires twice (agentexecutor + ime
// source) but the earliest wins.
StartedAtUtc = LastMatchedLogTimestamp ?? DateTime.UtcNow,
};
// Live "running" indicator — same signal health scripts emit. Gated on slot
// creation so the duplicate start line (agentexecutor + ime source) doesn't
// double-emit for the same execution.
try { OnScriptStarted?.Invoke(new ScriptStartedInfo { PolicyId = id, ScriptType = "platform" }); }
catch (Exception ex) { _logger.Warning($"ImeLogTracker: OnScriptStarted handler threw: {ex.Message}"); }
}
_lastPlatformScriptPolicyId = id;
// Started lines fire twice per script (agentexecutor + ime source) and carry no
// outcome — the matching `platform script completed` line below carries result+exit
// and stays on Info. Keep starts on Debug so Info reflects script outcomes only.
_logger.Debug($"ImeLogTracker: platform script started: {id} (source: {source ?? "ime"})");
HandlePlatformScriptStarted(id, source);
}
}
// Platform script: AgentExecutor.log entries create/enrich, IME log entries also create.
// Internal so the stale-slot hardening is directly testable (the public path arrives via
// a regex Match).
internal void HandlePlatformScriptStarted(string id, string source = null)
{
// Stale-slot hardening (session eaf3d8c4): a replayed start line from a previous
// enrollment whose matching result never appeared (script was mid-run when that
// enrollment ended) leaves a pending slot with a days-old StartedAtUtc. When the
// same policy genuinely re-runs now, the fresh start must begin a NEW execution —
// otherwise the ancient start pairs with the fresh completion and the duration
// lies. > 24 h mirrors the adapter's source-timestamp staleness clamp.
var currentStartTs = LastMatchedLogTimestamp ?? DateTime.UtcNow;
if (_pendingPlatformScripts.TryGetValue(id, out var pendingSlot)
&& pendingSlot.StartedAtUtc.HasValue
&& currentStartTs - pendingSlot.StartedAtUtc.Value > TimeSpan.FromHours(24))
{
_logger.Debug($"ImeLogTracker: discarding stale pending platform-script slot for {id} (started {pendingSlot.StartedAtUtc:o})");
_pendingPlatformScripts.Remove(id);
}
if (!_pendingPlatformScripts.ContainsKey(id))
{
// A fresh start for this policy (no pending entry) begins a NEW execution.
// Clear any emitted-marker from a prior run so IME re-evaluations / retries of
// the same platform-script policy within one agent lifetime are not silently
// deduped away. Within a single run the start always precedes exit/result, so
// this never clears the current run's own marker.
_platformScriptResultEmitted.Remove(id);
_pendingPlatformScripts[id] = new ScriptExecutionState
{
PolicyId = id,
ScriptType = "platform",
// Prefer the source CMTrace timestamp so a script that started before the
// agent launched (replayed log content) is dated to its real start, not now.
// Set only at slot creation: the start line fires twice (agentexecutor + ime
// source) but the earliest wins.
StartedAtUtc = currentStartTs,
};
// Live "running" indicator — same signal health scripts emit. Gated on slot
// creation so the duplicate start line (agentexecutor + ime source) doesn't
// double-emit for the same execution.
try { OnScriptStarted?.Invoke(new ScriptStartedInfo { PolicyId = id, ScriptType = "platform" }); }
catch (Exception ex) { _logger.Warning($"ImeLogTracker: OnScriptStarted handler threw: {ex.Message}"); }
}
_lastPlatformScriptPolicyId = id;
// Started lines fire twice per script (agentexecutor + ime source) and carry no
// outcome — the matching `platform script completed` line below carries result+exit
// and stays on Info. Keep starts on Debug so Info reflects script outcomes only.
_logger.Debug($"ImeLogTracker: platform script started: {id} (source: {source ?? "ime"})");
}
// Routes line-by-line script-data handlers (context / exitCode / output) to the right
// accumulator based on the pattern's scriptType parameter. Health-script lines feed the
// single-slot _pendingHealthScript; platform-script lines feed _pendingPlatformScripts.
@@ -194,6 +194,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
}
}
// Actions that mutate app/phase tracking state (_packageStates, _phasePackageSnapshots,
// _currentPhaseOrder, DO telemetry) — the historic-replay guard skips these for source
// lines from a previous enrollment so replayed apps never enter tracked state,
// persistence, app_tracking_summary, culprit lists or final-status. Script actions are
// deliberately NOT here: their tracker state is harmless (stale-slot hardening covers
// leftovers) and the adapter suppresses their emissions. espphasedetected IS here — a
// stale "In EspPhase: AccountSetup" would advance _currentPhaseOrder and make the fresh
// enrollment's DeviceSetup lines bounce as "backward"; IME re-logs the phase constantly,
// so fresh lines re-deliver it immediately.
private static readonly HashSet<string> AppMutatingActions = new HashSet<string>(StringComparer.Ordinal)
{
"imeshutdown", "espphasedetected", "setcurrentapp",
"updatestateinstalled", "updatestatedownloading", "updatestateinstalling",
"updatestateskipped", "updatestateerror", "updatestatepostponed",
"captureexitcode", "capturehresult", "captureappversion",
"captureapptypewinget", "captureapptypemsi", "captureattemptnumber",
"capturedetectionresult", "esptrackstatus", "policiesdiscovered",
"ignorecompletedapp", "updatename", "updatewin32appstate",
"cancelstuckandsetcurrent", "updatedotelemetry",
};
private void HandlePatternMatch(CompiledPattern pattern, Match match, string message, CmTraceLogEntry entry)
{
LastMatchedPatternId = pattern.PatternId;
@@ -206,6 +227,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
try { OnPatternMatched?.Invoke(pattern.PatternId); }
catch (Exception ex) { _logger?.Warning($"OnPatternMatched handler threw: {ex.Message}"); }
// Historic-replay guard (session eaf3d8c4): a source line > 24 h older than now is
// content from a previous enrollment whose IME log survived on disk. App-mutating
// actions are skipped BEFORE _seenAppIds so replayed apps poison neither the tracked
// state nor the phase-change ignore list. SimulationMode (--replay-log-dir dev tool)
// replays historic logs on purpose and bypasses the guard.
var isStaleReplayLine = !SimulationMode && entry != null
&& (UtcNowProvider() - NormalizeUtc(entry.Timestamp)) > HistoricReplayThreshold;
if (isStaleReplayLine && AppMutatingActions.Contains(pattern.Action?.ToLower() ?? string.Empty))
{
_logger.Debug($"ImeLogTracker: skipped app action '{pattern.Action}' for historic line ({entry.Timestamp:o})");
return;
}
try
{
var id = match.Groups["id"]?.Value;
@@ -71,6 +71,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
public double SpeedFactor { get; set; } = 50;
private DateTime _lastLogTimestamp = DateTime.MinValue;
// Source log lines older than this relative to now are content from a previous
// enrollment (IME logs surviving a re-enrollment, session eaf3d8c4). Single source of
// truth shared with the adapter's ResolveOccurredAt clamp and replay suppression.
internal static readonly TimeSpan HistoricReplayThreshold = TimeSpan.FromHours(24);
// Deterministic-time seam for the historic-replay guard; production default is the
// system clock, ImeLogHost rewires it to the agent IClock, tests to their VirtualClock.
internal Func<DateTime> UtcNowProvider { get; set; } = () => DateTime.UtcNow;
/// <summary>
/// CMTrace timestamps from IME are emitted in UTC, but DateTimeKind may arrive as
/// Unspecified depending on the parser path. Normalize via SpecifyKind=Utc rather than
/// ToUniversalTime() to avoid double-conversion when Kind is already UTC.
/// </summary>
internal static DateTime NormalizeUtc(DateTime value) =>
value.Kind == DateTimeKind.Utc ? value : DateTime.SpecifyKind(value, DateTimeKind.Utc);
// Background task
private Task _pollingTask;
private CancellationTokenSource _cts;
@@ -61,6 +61,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
internal Func<int, string> SessionOwnerResolver { get; set; }
internal Func<int, string> ProcessOwnerResolver { get; set; }
// Test seam for the WinRT OOBE-state sample (null = OobeStateReader.Read).
internal Func<string> OobeStateProvider { get; set; }
// OOBE-state flip tracking (observational only — see OobeStateReader contract).
// _oobeCompletedEmitted is one-shot per agent lifetime and deliberately NOT reset
// by ResetForRealUserSwitch: the OOBE flip is a global OS transition, not per-user.
private string _lastOobeState;
private bool _oobeCompletedEmitted;
/// <summary>
/// System/service account names that should NOT be considered real users.
/// </summary>
@@ -161,6 +170,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
_startedAtUtc = DateTime.UtcNow;
// Baseline OOBE-state sample so the first poll can already detect a flip
// (self-contained try/catch inside, like the Emit* helpers).
CheckOobeStateFlip();
_pollingTimer?.Dispose();
_pollingTimer = new Timer(
PollForDesktop,
@@ -186,6 +199,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_pollCount++;
CheckOobeStateFlip();
try
{
var explorerProcesses = Process.GetProcessesByName("explorer");
@@ -381,6 +396,66 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
catch (Exception ex) { _logger.Verbose($"DesktopArrivalDetector: detector-started emit failed: {ex.Message}"); }
}
/// <summary>
/// Samples the WinRT OOBE state (<see cref="OobeStateReader"/>) and emits the one-shot
/// <c>oobe_state_completed</c> trace event on the in_progress-&gt;completed flip.
/// Observational only — an owner-independent desktop corroboration for sessions where
/// WTS+WMI owner resolution starves and <c>desktop_arrived</c> never fires. Often absent
/// in healthy sessions: the polling timer is disposed at desktop arrival, which is when
/// the flip lands (empirical probe, session 9c404ae9).
/// <para>
/// Self-contained try/catch: a throw here must NOT reach <see cref="PollForDesktop"/>'s
/// outer catch, which would inflate <see cref="_wmiErrorCountSinceStart"/> and skew the
/// session-4d5a0b78 liveness diagnostics. "unavailable" readings never overwrite the last
/// known state, so a transient reflection failure between two polls cannot mask the flip.
/// One-shot flag survives <see cref="ResetForRealUserSwitch"/> — the flip is a global OS
/// transition, not per-user.
/// </para>
/// </summary>
private void CheckOobeStateFlip()
{
try
{
var current = (OobeStateProvider ?? OobeStateReader.Read)();
if (string.IsNullOrEmpty(current) || current == OobeStateReader.Unavailable)
return;
if (!_oobeCompletedEmitted
&& _lastOobeState == "in_progress"
&& current == "completed")
{
_oobeCompletedEmitted = true;
var minutesSinceStart = Math.Round((DateTime.UtcNow - _startedAtUtc).TotalMinutes, 1);
_logger.Info("DesktopArrivalDetector: OOBE state flipped in_progress->completed (WinRT SystemSetupInfo)");
OnTraceEvent?.Invoke(
SharedEventTypes.OobeStateCompleted,
$"OOBE state flipped in_progress->completed (WinRT SystemSetupInfo, poll {_pollCount}) — owner-independent desktop corroboration.",
new Dictionary<string, object>
{
{ "previousState", "in_progress" },
{ "pollCount", _pollCount },
{ "minutesSinceStart", minutesSinceStart },
{ "desktopArrived", _desktopArrived },
{ "wmiErrorCount", _wmiErrorCountSinceStart },
{ "wtsErrorCount", _wtsErrorCountSinceStart },
});
}
_lastOobeState = current;
}
catch (Exception ex)
{
_logger.Verbose($"DesktopArrivalDetector: OOBE state check failed: {ex.Message}");
}
}
/// <summary>
/// Drives a single synchronous poll for unit tests — the production poll is private and
/// timer-driven with a fixed 30 s interval (precedent: <see cref="ResolveOwner"/> is
/// likewise internal for direct testing).
/// </summary>
internal void PollOnceForTest() => PollForDesktop(null);
/// <summary>
/// Resolves the owning user of an explorer.exe candidate. Session 4d5a0b78 fix
/// (2026-06-11): the WMI <c>Win32_Process.GetOwner</c> call failed on every poll on
@@ -1,4 +1,5 @@
using System;
using System.Collections.Generic;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
@@ -38,16 +39,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// </summary>
public string Category { get; }
/// <summary>
/// Session 4910a5a5: names of the tracked apps most likely behind an Apps-subcategory
/// failure (capped, never-started apps ranked first — those starve ESP's app gate
/// without ever producing an IME install event). Snapshot from
/// <c>ProvisioningStatusTracker.SnapshotTrackedAppsNotCompleted</c> at failure-arming
/// time; empty for non-Apps failures and event-log-derived failures. Propagated onto
/// the EspTerminalFailure signal payload so the DecisionEngine's
/// <c>esp_failure_advisory</c> / <c>enrollment_failed</c> events can name the culprit
/// directly.
/// </summary>
public IReadOnlyList<string> LikelyCulpritApps { get; }
public EspFailureDetectedEventArgs(
string failureType,
string errorCode = null,
string failedSubcategory = null,
string category = null)
string category = null,
IReadOnlyList<string> likelyCulpritApps = null)
{
FailureType = string.IsNullOrEmpty(failureType) ? "unknown" : failureType;
ErrorCode = string.IsNullOrEmpty(errorCode) ? null : errorCode;
FailedSubcategory = string.IsNullOrEmpty(failedSubcategory) ? null : failedSubcategory;
Category = string.IsNullOrEmpty(category) ? null : category;
LikelyCulpritApps = likelyCulpritApps ?? Array.Empty<string>();
}
}
}
@@ -0,0 +1,307 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using Microsoft.Win32;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Session 5d735290 (2026-07-23) — reads the ESP's registered policy providers from
/// <c>HKLM\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking</c>, the registry
/// mirror of the EnrollmentStatusTracking CSP
/// (https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp).
/// <para>
/// CSP contract the ESP enforces (quotes from the doc above):
/// <list type="bullet">
/// <item><c>Setup/Apps/PolicyProviders/&lt;Name&gt;</c> — "Existence of this node indicates
/// to the ESP that it shouldn't show the tracking status message until the
/// TrackingPoliciesCreated node has been set to true." This wait has NO timeout: a
/// registered provider that never sets <c>TrackingPoliciesCreated=1</c> parks the ESP at
/// "Apps (Identifying)" indefinitely (observed for days in the field).</item>
/// <item><c>DevicePreparation/PolicyProviders/&lt;Name&gt;/InstallationState</c> — must reach
/// 2 (NotRequired) or 3 (Completed); 1=NotInstalled, 4=Error. ESP applies a default
/// 15-minute timeout here, so stalls in this stage are self-surfacing — still reported
/// for the diagnostic trail.</item>
/// </list>
/// Registry layout: <c>Device</c> subkey for device scope, <c>S-&lt;SID&gt;</c> subkeys for user
/// scope (appear at sign-in), each containing <c>Setup\Apps\PolicyProviders\&lt;Name&gt;</c>;
/// <c>DevicePreparation\PolicyProviders</c> exists in device scope only.
/// </para>
/// <para>
/// Known provider names: <c>Sidecar</c> is the Intune Management Extension ("Intune's agents,
/// such as SideCar" per the CSP doc); <c>ConfigMgr</c> is the Configuration Manager client
/// (registered during Autopilot-into-co-management, see the troubleshooting registry paths in
/// https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment). Microsoft
/// warns that ESP policy providers are not aware of each other and that pre-provisioning is
/// unsupported with co-management. Field case that motivated this probe (issue #106): a
/// pre-provisioned device with the ConfigMgr client installed as a blocking app ended up with
/// ONLY <c>ConfigMgr</c> registered under <c>Device\Setup\Apps\PolicyProviders</c> (no
/// <c>Sidecar</c>) — and although <c>TrackingPoliciesCreated=1</c> WAS set under it, the user
/// ESP hung at "Apps (Identifying)" for hours to days; manually renaming the key to
/// <c>Sidecar</c> unblocked it immediately. So the ESP's Apps wait is keyed to the provider
/// the Intune service registered at enrollment (always <c>Sidecar</c>; ConfigMgr only joins
/// via a co-management settings policy — see the WAP payload documented in
/// https://oofhours.com/2023/08/25/run-an-sccm-task-sequence-during-autopilot/), and a
/// foreign provider's completion values do not satisfy it. Detection therefore keys on the
/// provider NAME topology, not only on completeness:
/// <see cref="EspPolicyProviderStallDetector"/> turns this contract into the two-arm
/// <c>esp_policy_provider_stalled</c> tripwire.
/// </para>
/// <para>
/// Fail-soft like <see cref="EspTrackingInfoProbe"/>: any failure is Debug-logged and yields
/// <see cref="EspPolicyProviderSnapshot.Empty"/>; a missing root key (early enrollment,
/// non-Autopilot device) is not an error. Values are parsed defensively — a missing or
/// unparseable value counts as incomplete (fail-toward-observation).
/// </para>
/// </summary>
internal static class EspPolicyProviderProbe
{
internal const string TrackingRootKeyPath =
@"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking";
/// <summary>Provider kind: <c>Setup\Apps\PolicyProviders</c> (TrackingPoliciesCreated wait, no timeout).</summary>
internal const string KindSetupApps = "setupApps";
/// <summary>Provider kind: <c>DevicePreparation\PolicyProviders</c> (InstallationState wait, 15-min OS timeout).</summary>
internal const string KindDevicePreparation = "devicePreparation";
/// <summary>The IME's provider name — its absence while other providers are registered is the field-case signature.</summary>
internal const string SidecarProviderName = "Sidecar";
internal const string DeviceScope = "device";
private const string TrackingPoliciesCreatedValueName = "TrackingPoliciesCreated";
private const string InstallationStateValueName = "InstallationState";
/// <summary>
/// Test seam — when non-null, <see cref="Read"/> delegates to this func instead of
/// touching the live registry. Use <see cref="ScopedOverride"/> to guarantee cleanup.
/// </summary>
internal static Func<AgentLogger, EspPolicyProviderSnapshot> TestOverride;
/// <summary>
/// Reads all registered ESP policy providers. <see cref="EspPolicyProviderSnapshot.HasData"/>
/// is <c>true</c> when the EnrollmentStatusTracking root key existed and was readable
/// (even with no providers registered); <see cref="EspPolicyProviderSnapshot.Empty"/> otherwise.
/// </summary>
/// <param name="logger">Optional logger — Debug-level trace only; no warn/error.</param>
public static EspPolicyProviderSnapshot Read(AgentLogger logger = null)
{
var probe = TestOverride;
if (probe != null) return probe(logger);
try
{
using (var rootKey = Registry.LocalMachine.OpenSubKey(TrackingRootKeyPath))
{
if (rootKey == null)
{
logger?.Debug("EspPolicyProviderProbe: EnrollmentStatusTracking registry key not found");
return EspPolicyProviderSnapshot.Empty;
}
var providers = new List<PolicyProviderState>();
foreach (var scopeName in rootKey.GetSubKeyNames())
{
var isDevice = string.Equals(scopeName, "Device", StringComparison.OrdinalIgnoreCase);
// User scope repeats the Setup structure under the signed-in user's SID.
var isUserSid = scopeName.StartsWith("S-", StringComparison.OrdinalIgnoreCase);
if (!isDevice && !isUserSid) continue;
var scopeLabel = isDevice ? DeviceScope : "user:" + scopeName;
using (var scopeKey = rootKey.OpenSubKey(scopeName))
{
if (scopeKey == null) continue;
CollectProviders(
scopeKey, @"Setup\Apps\PolicyProviders", scopeLabel, KindSetupApps,
TrackingPoliciesCreatedValueName, providers, logger);
if (isDevice)
{
CollectProviders(
scopeKey, @"DevicePreparation\PolicyProviders", scopeLabel, KindDevicePreparation,
InstallationStateValueName, providers, logger);
}
}
}
return new EspPolicyProviderSnapshot(providers);
}
}
catch (Exception ex)
{
logger?.Debug($"EspPolicyProviderProbe: read threw: {ex.Message}");
return EspPolicyProviderSnapshot.Empty;
}
}
private static void CollectProviders(
RegistryKey scopeKey,
string providersSubPath,
string scopeLabel,
string kind,
string stateValueName,
List<PolicyProviderState> providers,
AgentLogger logger)
{
try
{
using (var providersKey = scopeKey.OpenSubKey(providersSubPath))
{
if (providersKey == null) return;
foreach (var providerName in providersKey.GetSubKeyNames())
{
using (var providerKey = providersKey.OpenSubKey(providerName))
{
if (providerKey == null) continue;
var value = TryReadInt(providerKey.GetValue(stateValueName));
providers.Add(new PolicyProviderState(
name: providerName,
scope: scopeLabel,
kind: kind,
trackingPoliciesCreated: kind == KindSetupApps ? value : null,
installationState: kind == KindDevicePreparation ? value : null));
}
}
}
}
catch (Exception ex)
{
logger?.Debug($"EspPolicyProviderProbe: '{providersSubPath}' ({scopeLabel}) read threw: {ex.Message}");
}
}
/// <summary>
/// Defensive value parse: REG_DWORD expected per CSP (booleans stored as 0/1), but string
/// forms are tolerated. Anything missing or unparseable is <c>null</c> — which the
/// completion rule treats as incomplete.
/// </summary>
internal static int? TryReadInt(object rawValue)
{
switch (rawValue)
{
case null: return null;
case int i: return i;
case long l when l >= int.MinValue && l <= int.MaxValue: return (int)l;
case string s when int.TryParse(s.Trim(), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed):
return parsed;
case bool b: return b ? 1 : 0;
default: return null;
}
}
/// <summary>
/// Pure completion rule per the CSP contract: <c>setupApps</c> providers are complete when
/// <c>TrackingPoliciesCreated == 1</c>; <c>devicePreparation</c> providers when
/// <c>InstallationState</c> is 2 (NotRequired) or 3 (Completed).
/// </summary>
internal static bool IsProviderComplete(string kind, int? trackingPoliciesCreated, int? installationState)
=> kind == KindSetupApps
? trackingPoliciesCreated == 1
: installationState == 2 || installationState == 3;
/// <summary>
/// Disposable scope that sets <see cref="TestOverride"/> for the lifetime of the scope
/// and restores the previous value on Dispose. Nestable. Internal test-only helper.
/// </summary>
internal sealed class ScopedOverride : IDisposable
{
private readonly Func<AgentLogger, EspPolicyProviderSnapshot> _previous;
private int _disposed;
public ScopedOverride(Func<AgentLogger, EspPolicyProviderSnapshot> probe)
{
_previous = TestOverride;
TestOverride = probe;
}
public void Dispose()
{
if (System.Threading.Interlocked.Exchange(ref _disposed, 1) == 1) return;
TestOverride = _previous;
}
}
}
/// <summary>
/// One registered ESP policy provider with its completion-relevant state.
/// <see cref="Key"/> is the stable identity used for dwell tracking and one-shot latching.
/// </summary>
internal readonly struct PolicyProviderState
{
public PolicyProviderState(string name, string scope, string kind, int? trackingPoliciesCreated, int? installationState)
{
Name = name ?? string.Empty;
Scope = scope ?? string.Empty;
Kind = kind ?? string.Empty;
TrackingPoliciesCreated = trackingPoliciesCreated;
InstallationState = installationState;
IsComplete = EspPolicyProviderProbe.IsProviderComplete(kind, trackingPoliciesCreated, installationState);
}
/// <summary>Provider key name, e.g. "Sidecar" or "ConfigMgr".</summary>
public string Name { get; }
/// <summary><c>"device"</c> or <c>"user:S-1-5-21-…"</c>.</summary>
public string Scope { get; }
/// <summary><see cref="EspPolicyProviderProbe.KindSetupApps"/> or <see cref="EspPolicyProviderProbe.KindDevicePreparation"/>.</summary>
public string Kind { get; }
/// <summary>Raw <c>TrackingPoliciesCreated</c> value (setupApps only); <c>null</c> when missing/unparseable.</summary>
public int? TrackingPoliciesCreated { get; }
/// <summary>Raw <c>InstallationState</c> value (devicePreparation only); <c>null</c> when missing/unparseable.</summary>
public int? InstallationState { get; }
/// <summary>Completion per <see cref="EspPolicyProviderProbe.IsProviderComplete"/>.</summary>
public bool IsComplete { get; }
/// <summary>Stable dwell/latch identity: <c>kind|scope|name</c>.</summary>
public string Key => Kind + "|" + Scope + "|" + Name;
}
/// <summary>
/// Snapshot of all registered ESP policy providers. <see cref="HasData"/> is <c>false</c> only
/// when the EnrollmentStatusTracking root key was missing or unreadable — an empty provider
/// list with <see cref="HasData"/> = <c>true</c> is valid data (nothing registered yet).
/// </summary>
internal readonly struct EspPolicyProviderSnapshot
{
public static readonly EspPolicyProviderSnapshot Empty = default;
public EspPolicyProviderSnapshot(IReadOnlyList<PolicyProviderState> providers)
{
HasData = true;
Providers = providers ?? Array.Empty<PolicyProviderState>();
}
/// <summary><c>true</c> when the EnrollmentStatusTracking root key existed and was readable.</summary>
public bool HasData { get; }
/// <summary>All registered providers across scopes and kinds; empty when none registered.</summary>
public IReadOnlyList<PolicyProviderState> Providers { get; }
/// <summary>
/// <c>true</c> when a <c>Setup\Apps</c> provider named "Sidecar" (the IME) is registered in
/// any scope — its absence while other providers are registered is the co-management
/// field-case signature.
/// </summary>
public bool SidecarRegistered
{
get
{
var providers = Providers;
if (providers == null) return false;
return providers.Any(p =>
p.Kind == EspPolicyProviderProbe.KindSetupApps &&
string.Equals(p.Name, EspPolicyProviderProbe.SidecarProviderName, StringComparison.OrdinalIgnoreCase));
}
}
}
}
@@ -0,0 +1,315 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Turns the ESP policy-provider contract read by <see cref="EspPolicyProviderProbe"/> into
/// the one-shot <c>esp_policy_provider_stalled</c> Warning via two independent arms, each
/// with the same <see cref="DwellMinutes"/>-minute continuous-dwell rule and the full
/// provider table in the payload:
/// <list type="bullet">
/// <item><b>Arm 1 — <c>reason=provider_incomplete</c></b>: a registered provider stays
/// continuously incomplete (e.g. Sidecar itself never sets TrackingPoliciesCreated).</item>
/// <item><b>Arm 2 — <c>reason=sidecar_provider_missing</c></b>: at least one
/// <c>Setup\Apps</c> provider is registered but none of them is <c>Sidecar</c> (the IME).
/// This is the actual co-management field-case signature (issue #106): the foreign
/// <c>ConfigMgr</c> provider had <c>TrackingPoliciesCreated=1</c> — i.e. was "complete"
/// by the CSP value contract — yet the user ESP stayed parked at "Apps (Identifying)"
/// until the key was renamed to <c>Sidecar</c>. The ESP's Apps wait is keyed to the
/// Intune-registered provider by NAME, so provider completeness alone must not gate this
/// detection. See the probe's header for sources.</item>
/// </list>
/// <para>
/// Dwell semantics: arm 1's clock per provider starts when it is first observed incomplete
/// and RESETS when the provider completes or disappears; arm 2's clock starts when the
/// sidecar-missing condition is first observed and RESETS when Sidecar registers (covers the
/// legitimate "ConfigMgr registered before Sidecar" startup ordering) — only
/// <i>continuously</i> bad states fire. A missing root key / empty provider list is normal
/// early enrollment and clears all dwell state (probe D4). Purely observational: emission
/// goes through <see cref="InformationalEventPost"/> (InformationalEvent pass-through,
/// dispatch-guard exempt), never into decision state.
/// </para>
/// <para>
/// Duplicate suppression: in-process one-shot per provider key, plus a
/// <see cref="StartupEventGate"/> fingerprint over the sorted set of currently-stalled keys so
/// an agent restart into the same stall does not re-report it — while a NEW provider joining
/// the stall changes the fingerprint and is reported (the event carries the full list).
/// </para>
/// </summary>
internal sealed class EspPolicyProviderStallDetector
{
/// <summary>
/// Wall-clock minutes a provider must stay continuously incomplete before it is reported.
/// Mirrors the CSP's own DevicePreparation default timeout; the Setup/Apps wait this
/// tripwire mainly targets has no OS timeout at all.
/// </summary>
internal const int DwellMinutes = 15;
internal const string SourceLabel = "EspPolicyProviderStallDetector";
/// <summary>Payload <c>reason</c>: a registered provider stayed continuously incomplete (arm 1).</summary>
internal const string ReasonProviderIncomplete = "provider_incomplete";
/// <summary>Payload <c>reason</c>: Setup\Apps providers registered but Sidecar absent (arm 2 — dominant when both arms fire in one round).</summary>
internal const string ReasonSidecarMissing = "sidecar_provider_missing";
/// <summary>
/// Sentinel token representing the arm-2 condition in the cross-restart fingerprint.
/// Cannot collide with provider keys — those always contain <c>|</c> separators.
/// </summary>
private const string SidecarMissingFingerprintToken = "sidecarMissing";
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly AgentLogger _logger;
private readonly IClock _clock;
private readonly StartupEventGate? _startupGate;
private readonly TimeSpan _dwell;
private readonly Dictionary<string, DateTime> _firstSeenIncompleteUtc =
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
private readonly HashSet<string> _firedKeys =
new HashSet<string>(StringComparer.OrdinalIgnoreCase);
/// <summary>Arm-2 dwell state: first observation of "setupApps providers without Sidecar"; null = condition not active.</summary>
private DateTime? _sidecarMissingSinceUtc;
/// <summary>Arm-2 in-process one-shot latch (mirror of <see cref="_firedKeys"/>).</summary>
private bool _sidecarMissingFired;
public EspPolicyProviderStallDetector(
string sessionId,
string tenantId,
InformationalEventPost post,
AgentLogger logger,
IClock clock,
StartupEventGate? startupGate = null,
TimeSpan? dwell = null)
{
_sessionId = sessionId;
_tenantId = tenantId;
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger;
_clock = clock ?? throw new ArgumentNullException(nameof(clock));
_startupGate = startupGate;
_dwell = dwell ?? TimeSpan.FromMinutes(DwellMinutes);
}
/// <summary>Host timer callback and test seam — one probe + evaluation round.</summary>
internal void Tick()
{
var snapshot = EspPolicyProviderProbe.Read(_logger);
var eval = EvaluateStallTransition(
snapshot, _firstSeenIncompleteUtc, _firedKeys,
_sidecarMissingSinceUtc, _sidecarMissingFired, _clock.UtcNow, _dwell);
_firstSeenIncompleteUtc.Clear();
foreach (var kv in eval.UpdatedFirstSeenUtc)
_firstSeenIncompleteUtc[kv.Key] = kv.Value;
_sidecarMissingSinceUtc = eval.UpdatedSidecarMissingSinceUtc;
if (eval.NewlyStalled.Count > 0 || eval.SidecarMissingStalledForMinutes != null)
EmitStalled(snapshot, eval);
}
/// <summary>
/// Pure evaluation of one probe round — the entire detection decision, testable without
/// registry, timer, or ingress (pattern: <c>PerformanceCollector.EvaluateDiskLowTransition</c>).
/// </summary>
/// <param name="snapshot">Current provider snapshot.</param>
/// <param name="firstSeenIncompleteUtc">Arm-1 dwell state from the previous round (provider key → first-seen-incomplete).</param>
/// <param name="alreadyFired">Provider keys already reported this run (never re-report).</param>
/// <param name="sidecarMissingSinceUtc">Arm-2 dwell state from the previous round; null = condition was not active.</param>
/// <param name="sidecarMissingAlreadyFired">Arm-2 one-shot latch (never re-report).</param>
/// <param name="nowUtc">Evaluation timestamp.</param>
/// <param name="dwell">Continuous-condition threshold (shared by both arms).</param>
internal static StallEvaluation EvaluateStallTransition(
EspPolicyProviderSnapshot snapshot,
IReadOnlyDictionary<string, DateTime> firstSeenIncompleteUtc,
ISet<string> alreadyFired,
DateTime? sidecarMissingSinceUtc,
bool sidecarMissingAlreadyFired,
DateTime nowUtc,
TimeSpan dwell)
{
var result = new StallEvaluation();
// Missing root key or no registered providers = normal early enrollment: clear all
// dwell state (a condition must be continuously OBSERVED to accrue dwell).
if (!snapshot.HasData || snapshot.Providers.Count == 0)
return result;
// Arm 1 — a registered provider continuously incomplete.
foreach (var provider in snapshot.Providers)
{
if (provider.IsComplete)
continue; // completed → falls out of the dwell map (reset)
var firstSeen = firstSeenIncompleteUtc.TryGetValue(provider.Key, out var seen)
? seen
: nowUtc;
result.UpdatedFirstSeenUtc[provider.Key] = firstSeen;
var stalledFor = nowUtc - firstSeen;
if (stalledFor >= dwell && !alreadyFired.Contains(provider.Key))
result.NewlyStalled.Add(new StalledProvider(provider, stalledFor.TotalMinutes));
}
// Arm 2 — Setup\Apps providers registered, but none of them is Sidecar. Deliberately
// ignores provider completeness: the field case (issue #106) had ConfigMgr with
// TrackingPoliciesCreated=1 and the ESP still waited. Sidecar registering later
// (legitimate startup ordering) clears the condition and resets the clock.
if (HasSetupAppsProvider(snapshot) && !snapshot.SidecarRegistered)
{
var since = sidecarMissingSinceUtc ?? nowUtc;
result.UpdatedSidecarMissingSinceUtc = since;
var missingFor = nowUtc - since;
if (missingFor >= dwell && !sidecarMissingAlreadyFired)
result.SidecarMissingStalledForMinutes = missingFor.TotalMinutes;
}
return result;
}
private static bool HasSetupAppsProvider(EspPolicyProviderSnapshot snapshot)
=> snapshot.Providers.Any(p => p.Kind == EspPolicyProviderProbe.KindSetupApps);
private void EmitStalled(EspPolicyProviderSnapshot snapshot, StallEvaluation eval)
{
foreach (var stalled in eval.NewlyStalled)
_firedKeys.Add(stalled.Provider.Key);
if (eval.SidecarMissingStalledForMinutes != null)
_sidecarMissingFired = true;
// Cross-restart dedup: fingerprint = sorted set of all fired conditions that are
// still active (incomplete providers + the sidecar-missing sentinel). A restart into
// the same stall re-accrues dwell and lands on the same fingerprint → suppressed; a
// new condition joining the stall changes it → re-emitted.
var currentlyStalledKeys = snapshot.Providers
.Where(p => !p.IsComplete && _firedKeys.Contains(p.Key))
.Select(p => p.Key)
.ToList();
if (_sidecarMissingFired && HasSetupAppsProvider(snapshot) && !snapshot.SidecarRegistered)
currentlyStalledKeys.Add(SidecarMissingFingerprintToken);
currentlyStalledKeys.Sort(StringComparer.OrdinalIgnoreCase);
var fingerprint = StartupEventGate.HashString(string.Join(";", currentlyStalledKeys));
if (_startupGate != null &&
!_startupGate.ShouldEmit(Constants.EventTypes.EspPolicyProviderStalled, fingerprint))
{
_logger?.Debug(
"EspPolicyProviderStallDetector: stalled provider set already reported by a " +
"previous agent run (fingerprint match) — suppressed.");
return;
}
// Sidecar-missing is the dominant reason when both arms fire in one round: it
// explains the ESP hang regardless of the foreign providers' completeness.
var reason = eval.SidecarMissingStalledForMinutes != null
? ReasonSidecarMissing
: ReasonProviderIncomplete;
var messageParts = new List<string>();
if (eval.SidecarMissingStalledForMinutes != null)
{
var registeredNames = string.Join(", ", snapshot.Providers
.Where(p => p.Kind == EspPolicyProviderProbe.KindSetupApps)
.Select(p => $"{p.Name} ({p.Scope})"));
messageParts.Add(
$"expected IME provider '{EspPolicyProviderProbe.SidecarProviderName}' not registered for " +
$">= {DwellMinutes} min while Setup/Apps providers exist: {registeredNames}");
}
var stalledNames = string.Join(", ",
eval.NewlyStalled.Select(s => $"{s.Provider.Name} ({s.Provider.Scope}/{s.Provider.Kind})"));
if (eval.NewlyStalled.Count > 0)
messageParts.Add($"provider(s) incomplete for >= {DwellMinutes} min: {stalledNames}");
var data = new Dictionary<string, object>(StringComparer.Ordinal)
{
["reason"] = reason,
["dwellMinutes"] = DwellMinutes,
["sidecarRegistered"] = snapshot.SidecarRegistered,
["providers"] = snapshot.Providers.Select(p => new Dictionary<string, object?>(StringComparer.Ordinal)
{
["name"] = p.Name,
["scope"] = p.Scope,
["kind"] = p.Kind,
["trackingPoliciesCreated"] = p.TrackingPoliciesCreated,
["installationState"] = p.InstallationState,
["complete"] = p.IsComplete,
}).ToList(),
["stalledProviders"] = eval.NewlyStalled.Select(s => new Dictionary<string, object>(StringComparer.Ordinal)
{
["name"] = s.Provider.Name,
["scope"] = s.Provider.Scope,
["kind"] = s.Provider.Kind,
["stalledForMinutes"] = Math.Round(s.StalledForMinutes, 1),
}).ToList(),
};
if (eval.SidecarMissingStalledForMinutes != null)
data["sidecarMissingForMinutes"] = Math.Round(eval.SidecarMissingStalledForMinutes.Value, 1);
var message =
"ESP Setup/Apps tracking stalled: " + string.Join("; ", messageParts) +
" — the ESP waits on its app-tracking providers without any timeout";
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
Timestamp = _clock.UtcNow,
EventType = Constants.EventTypes.EspPolicyProviderStalled,
Severity = EventSeverity.Warning,
Source = SourceLabel,
Phase = EnrollmentPhase.Unknown,
ImmediateUpload = true,
Message = message,
Data = data
});
_startupGate?.MarkEmitted(Constants.EventTypes.EspPolicyProviderStalled);
_logger?.Warning(
$"EspPolicyProviderStallDetector: {string.Join("; ", messageParts)} " +
$"(reason={reason}, sidecarRegistered={snapshot.SidecarRegistered}).");
}
/// <summary>Result of one <see cref="EvaluateStallTransition"/> round.</summary>
internal sealed class StallEvaluation
{
/// <summary>Next round's arm-1 dwell state — only currently-incomplete providers survive.</summary>
public Dictionary<string, DateTime> UpdatedFirstSeenUtc { get; } =
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
/// <summary>Providers that crossed the dwell threshold this round and were not yet reported (arm 1).</summary>
public List<StalledProvider> NewlyStalled { get; } = new List<StalledProvider>();
/// <summary>Next round's arm-2 dwell state; null = sidecar-missing condition not active (reset).</summary>
public DateTime? UpdatedSidecarMissingSinceUtc { get; set; }
/// <summary>Non-null when the sidecar-missing condition crossed the dwell threshold this round and was not yet reported (arm 2).</summary>
public double? SidecarMissingStalledForMinutes { get; set; }
}
internal readonly struct StalledProvider
{
public StalledProvider(PolicyProviderState provider, double stalledForMinutes)
{
Provider = provider;
StalledForMinutes = stalledForMinutes;
}
public PolicyProviderState Provider { get; }
public double StalledForMinutes { get; }
}
}
}
@@ -116,6 +116,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// failure details that fire after the settle window expires.
private Dictionary<string, Timer> _provisioningFailureSettleTimers;
private Dictionary<string, EspFailureDetectedEventArgs> _provisioningFailureSettleArgs;
// Session 4910a5a5 — post-terminal recovery story. Populated when the settle window
// expires and EspFailureDetected actually fires (terminal path); consumed by the
// esp_failure_retry_detected / esp_failure_recovered emitters so a late "Try again"
// recovery (23 min after the failure in the field session) is visible on the timeline
// instead of hiding inside a generic subcategory_state_change. Keyed by category-name.
private Dictionary<string, EspFailureDetectedEventArgs> _provisioningFailureTerminalArgs;
private Dictionary<string, DateTime> _provisioningFailureTerminalUtc;
private HashSet<string> _postFailureRetryEmitted;
private HashSet<string> _postFailureRecoveredEmitted;
// Session 2bc884b6: AppX enrichment scan seams + fire-once guard. The scanner and the
// dispatcher are injectable so tests never touch the real event log or the threadpool.
private readonly IAppxDeploymentFailureScanner _appxScanner;
@@ -285,6 +294,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_lastSubcategoryStates = new Dictionary<string, Dictionary<string, string>>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureSettleTimers = new Dictionary<string, Timer>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureSettleArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureTerminalArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureTerminalUtc = new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
_postFailureRetryEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
_postFailureRecoveredEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
_deviceSetupProvisioningCompleteFired = false;
_deviceSetupFallbackTimer = null;
_accountSetupProvisioningCompleteFired = false;
@@ -665,6 +678,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// null-ing here is correctness-neutral on the start-condition side.
ResetAccountSetupFallbackTimer();
_accountSetupProvisioningCompleteFired = true;
MaybeEmitPostFailureRecovered("AccountSetupCategory.Status", "AccountSetup");
try { AccountSetupProvisioningComplete?.Invoke(this, EventArgs.Empty); }
catch (Exception ex) { _logger.Error("AccountSetupProvisioningComplete handler failed (fallback path)", ex); }
}
@@ -833,6 +847,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// null-ing here is correctness-neutral on the start-condition side.
ResetDeviceSetupFallbackTimer();
_deviceSetupProvisioningCompleteFired = true;
MaybeEmitPostFailureRecovered("DeviceSetupCategory.Status", "DeviceSetup");
try { DeviceSetupProvisioningComplete?.Invoke(this, EventArgs.Empty); }
catch (Exception ex) { _logger.Error("DeviceSetupProvisioningComplete handler failed (fallback path)", ex); }
}
@@ -946,12 +961,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
var severity = categorySucceeded == false ? EventSeverity.Warning : EventSeverity.Info;
EmitProvisioningEvent(categoryLabel, categorySucceeded, statusText, subcategories, severity);
EmitRawRegistryDump(categoryLabel, jsonValue, $"category_resolved_{(categorySucceeded == true ? "success" : "failed")}");
// Session 4910a5a5: category resolving to success after a terminally
// fired failure = the failure recovered (fallback-path successes call
// this from their own timer handlers).
if (categorySucceeded == true)
{
MaybeEmitPostFailureRecovered(categoryName, categoryLabel);
}
}
else
{
var transitions = DetectSubcategoryTransitions(categoryName, subcategories);
StoreSubcategoryStates(categoryName, subcategories);
// Session 4910a5a5: after a terminally fired failure, the failed
// subcategory leaving "failed" is the ESP re-running the step —
// consistent with the user pressing "Try again" on the failure page.
MaybeEmitPostFailureRetry(categoryName, categoryLabel, transitions);
var failureTransitions = transitions.Where(t => t.IsFailure).ToList();
if (transitions.Count > 0)
{
@@ -1025,6 +1053,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
$"escalating as {failureTypeName} (errorCode={errorCode ?? "n/a"})");
_logger.Warning($"Provisioning failure detected: {failureTypeName}");
// Session 4910a5a5: name the likely culprit app(s) directly on the failure args so
// the DecisionEngine's advisory/terminal events carry them — never-started apps
// first (they starve ESP's app gate without ever producing an IME install event),
// capped at 3 names.
IReadOnlyList<string> likelyCulpritApps = null;
if (string.Equals(failedSubcategoryName, "Apps", StringComparison.OrdinalIgnoreCase))
{
likelyCulpritApps = SnapshotTrackedAppsNotCompleted()
.OrderBy(a => a["everStartedInstalling"] is bool started && started ? 1 : 0)
.Select(a => a["appName"]?.ToString())
.Where(n => !string.IsNullOrEmpty(n))
.Take(3)
.ToList();
}
// Caller already holds _stateLock.
ArmFailureSettleTimer(
categoryName: categoryName,
@@ -1032,7 +1075,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
failureType: failureTypeName,
errorCode: errorCode,
failedSubcategory: failedSubcategoryName,
category: categoryLabel));
category: categoryLabel,
likelyCulpritApps: likelyCulpritApps));
// Settle-window owns the actual EspFailureDetected fire. The legacy "fire-after-emit"
// path in CheckProvisioningStatus is now a no-op for this code-path; it stays in place
@@ -1292,6 +1336,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_provisioningCategoriesResolved.Remove(categoryName);
}
}
else
{
// Session 4910a5a5: remember the terminally fired failure so a LATE
// recovery (outside the settle window — e.g. user pressing "Try again"
// minutes later) can be told as esp_failure_retry_detected /
// esp_failure_recovered instead of hiding in a generic state-change.
_provisioningFailureTerminalArgs[categoryName] = args;
_provisioningFailureTerminalUtc[categoryName] = DateTime.UtcNow;
}
}
if (recovered)
@@ -1382,6 +1435,133 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
});
}
/// <summary>
/// Session 4910a5a5: after a terminally fired failure (settle window expired,
/// EspFailureDetected raised), the failed subcategory leaving the <c>failed</c> state
/// means the ESP re-ran the step — consistent with the user pressing "Try again" on
/// the ESP failure page (the agent cannot observe the click itself). Emits
/// <c>esp_failure_retry_detected</c> once per terminally fired category. Caller holds
/// <see cref="_stateLock"/>.
/// </summary>
private void MaybeEmitPostFailureRetry(string categoryName, string categoryLabel, List<SubcategoryTransition> transitions)
{
if (transitions == null || transitions.Count == 0) return;
if (_postFailureRetryEmitted.Contains(categoryName)) return;
if (_provisioningFailureTerminalArgs == null
|| !_provisioningFailureTerminalArgs.TryGetValue(categoryName, out var terminalArgs))
{
return; // No terminal fire on record — in-window recovery is settle_recovered's job.
}
SubcategoryTransition? retryTransition = null;
foreach (var t in transitions)
{
if (string.Equals(t.OldState, "failed", StringComparison.OrdinalIgnoreCase)
&& !string.Equals(t.NewState, "failed", StringComparison.OrdinalIgnoreCase))
{
retryTransition = t;
break;
}
}
if (retryTransition == null) return;
var retry = retryTransition.Value;
_postFailureRetryEmitted.Add(categoryName);
var minutesSinceFailure = MinutesSinceTerminalFailure(categoryName);
var data = new Dictionary<string, object>
{
{ "category", categoryLabel },
{ "subcategory", retry.SubcategoryName },
{ "previousState", retry.OldState },
{ "newState", retry.NewState },
{ "failureType", terminalArgs.FailureType },
{ "minutesSinceFailure", minutesSinceFailure },
};
if (!string.IsNullOrEmpty(terminalArgs.ErrorCode))
data["errorCode"] = terminalArgs.ErrorCode;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.EspFailureRetryDetected,
Severity = EventSeverity.Info,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = $"ESP retry detected — {categoryLabel}/{retry.SubcategoryName} left the failed state " +
$"({retry.OldState} → {retry.NewState}) {minutesSinceFailure} min after the failure — " +
"consistent with the user pressing 'Try again' on the ESP failure page.",
Data = data,
ImmediateUpload = true
});
}
/// <summary>
/// Session 4910a5a5: the category behind a terminally fired failure reached success
/// (categorySucceeded=true or the all-subcategories-succeeded fallback) — the failure
/// demonstrably recovered. Emits <c>esp_failure_recovered</c> once per terminally
/// fired category so the timeline tells "failed → retried → recovered" instead of
/// leaving the stale failure as the last word. Caller holds <see cref="_stateLock"/>.
/// </summary>
private void MaybeEmitPostFailureRecovered(string categoryName, string categoryLabel)
{
if (_postFailureRecoveredEmitted == null || _postFailureRecoveredEmitted.Contains(categoryName)) return;
if (_provisioningFailureTerminalArgs == null
|| !_provisioningFailureTerminalArgs.TryGetValue(categoryName, out var terminalArgs))
{
return;
}
_postFailureRecoveredEmitted.Add(categoryName);
var minutesSinceFailure = MinutesSinceTerminalFailure(categoryName);
string subcategoryStatus = null;
if (terminalArgs.FailedSubcategory != null
&& _lastSubcategoryStates != null
&& _lastSubcategoryStates.TryGetValue(categoryName, out var subs)
&& subs.TryGetValue(terminalArgs.FailedSubcategory, out var currentState))
{
subcategoryStatus = currentState;
}
var data = new Dictionary<string, object>
{
{ "category", categoryLabel },
{ "failedSubcategory", terminalArgs.FailedSubcategory ?? "category-level" },
{ "failureType", terminalArgs.FailureType },
{ "minutesSinceFailure", minutesSinceFailure },
};
if (!string.IsNullOrEmpty(terminalArgs.ErrorCode))
data["errorCode"] = terminalArgs.ErrorCode;
if (subcategoryStatus != null)
data["subcategoryState"] = subcategoryStatus;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.EspFailureRecovered,
Severity = EventSeverity.Info,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = $"ESP failure recovered — {categoryLabel}/{terminalArgs.FailedSubcategory ?? "category-level"} " +
$"completed successfully {minutesSinceFailure} min after the failure; the earlier ESP failure no longer applies.",
Data = data,
ImmediateUpload = true
});
}
private int MinutesSinceTerminalFailure(string categoryName)
{
if (_provisioningFailureTerminalUtc != null
&& _provisioningFailureTerminalUtc.TryGetValue(categoryName, out var firedAt))
{
return Math.Max(0, (int)Math.Round((DateTime.UtcNow - firedAt).TotalMinutes));
}
return 0;
}
private void EmitFailureSettleStarted(EspFailureDetectedEventArgs args)
{
var data = new Dictionary<string, object>
@@ -1863,6 +2043,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_provisioningFailureSettleTimers = new Dictionary<string, Timer>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureSettleArgs == null)
_provisioningFailureSettleArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureTerminalArgs == null)
_provisioningFailureTerminalArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureTerminalUtc == null)
_provisioningFailureTerminalUtc = new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
if (_postFailureRetryEmitted == null)
_postFailureRetryEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
if (_postFailureRecoveredEmitted == null)
_postFailureRecoveredEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
}
}
@@ -0,0 +1,72 @@
using System;
using System.Reflection;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// Reads WinRT <c>Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState</c>
/// (NotStarted=0 / InProgress=1 / Completed=2) via the CLR's built-in WinRT projection —
/// no NuGet package, no winmd reference; the type is resolved at runtime with
/// <c>ContentType=WindowsRuntime</c>, the same mechanism the bootstrap script uses from
/// PowerShell 5.1.
/// <para>
/// <b>Contract:</b> observational only — never feed this into decision-engine logic.
/// Requires Windows 10 1809+ (build 17763, UniversalApiContract 7.0) and is callable
/// from the SYSTEM service context (empirically validated, probe session 9c404ae9).
/// On older builds or SKUs without the contract the type resolves to null and
/// <see cref="Read"/> returns <c>"unavailable"</c>. <see cref="Read"/> never throws.
/// </para>
/// <para>
/// Empirical timing (Win11 24H2 probe): the state flips InProgress→Completed at desktop
/// arrival — it is a corroboration signal, not an early one. WDP reseal and placeholder
/// (fooUser) flip semantics are unverified; another reason this must stay observational.
/// </para>
/// </summary>
internal static class OobeStateReader
{
public const string Unavailable = "unavailable";
// Resolved once per process. Benign race: concurrent first calls resolve the same
// PropertyInfo and the assignment is idempotent — no lock needed.
private static PropertyInfo _stateProperty;
private static bool _initTried;
/// <summary>
/// Returns <c>"not_started"</c> / <c>"in_progress"</c> / <c>"completed"</c>,
/// <c>"unknown_&lt;n&gt;"</c> for unexpected enum values, or <c>"unavailable"</c>
/// when the WinRT contract is absent or any part of the read fails. Never throws.
/// </summary>
public static string Read()
{
try
{
if (!_initTried)
{
var type = Type.GetType(
"Windows.System.Profile.SystemSetupInfo, Windows.System.Profile, ContentType=WindowsRuntime",
throwOnError: false);
_stateProperty = type?.GetProperty(
"OutOfBoxExperienceState",
BindingFlags.Public | BindingFlags.Static);
_initTried = true;
}
if (_stateProperty == null)
return Unavailable;
var value = Convert.ToInt32(_stateProperty.GetValue(null));
switch (value)
{
case 0: return "not_started";
case 1: return "in_progress";
case 2: return "completed";
default: return "unknown_" + value;
}
}
catch
{
return Unavailable;
}
}
}
}
@@ -0,0 +1,375 @@
using System;
using System.Runtime.InteropServices;
using Microsoft.Win32.SafeHandles;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// Reads the UEFI Secure Boot signature databases <c>db</c> and <c>KEK</c> directly from
/// firmware via <c>GetFirmwareEnvironmentVariableExW</c> and checks which Microsoft
/// signing certificates they contain. This is the authoritative source for the 2023
/// certificate rollout state — the <c>SecureBoot\Servicing</c> registry key only reflects
/// whether the *Windows Update* rollout ran and verified, and lags behind when the
/// certificate reached the firmware another way (OEM factory image, WinCS, manual).
/// <para>
/// The certificate subject CNs are stored as ASCII bytes inside the DER-encoded
/// signature lists, so a plain byte-substring search is sufficient — the same check
/// Microsoft documents via <c>Get-SecureBootUEFI</c> (which wraps this exact API).
/// </para>
/// <para>
/// <b>Contract:</b> observational only — never feed this into decision-engine logic.
/// <see cref="Read"/> never throws. Reading firmware variables requires
/// <c>SeSystemEnvironmentPrivilege</c>, which SYSTEM holds but must be enabled on the
/// process token first; the previous privilege state is restored after the read.
/// On legacy BIOS the API fails with <c>ERROR_INVALID_FUNCTION</c> → status
/// <c>not_uefi</c>; every failure maps to a status string instead of an exception.
/// </para>
/// </summary>
internal static class UefiSecureBootCertReader
{
public const string StatusOk = "ok";
public const string StatusNotUefi = "not_uefi";
public const string StatusVariableNotFound = "variable_not_found";
public const string StatusPrivilegeDenied = "privilege_denied";
// EFI_IMAGE_SECURITY_DATABASE_GUID — namespace of the "db" variable.
private const string DbVariableGuid = "{d719b2cb-3d3a-4596-a3bc-dad00e67656f}";
// EFI_GLOBAL_VARIABLE — namespace of the "KEK" variable.
private const string KekVariableGuid = "{8be4df61-93ca-11d2-aa0d-00e098032b8c}";
// Subject CNs searched for in db. 2023 set per Microsoft's June-2026 expiry guidance;
// 2011 set kept for context (which legacy CAs are still present).
internal const string CertWindowsUefiCa2023 = "Windows UEFI CA 2023";
internal const string CertMicrosoftUefiCa2023 = "Microsoft UEFI CA 2023";
internal const string CertMicrosoftOptionRomCa2023 = "Microsoft Option ROM UEFI CA 2023";
internal const string CertWindowsProductionPca2011 = "Microsoft Windows Production PCA 2011";
internal const string CertMicrosoftUefiCa2011 = "Microsoft Corporation UEFI CA 2011";
// Subject CNs searched for in KEK.
internal const string CertMicrosoftKek2kCa2023 = "Microsoft Corporation KEK 2K CA 2023";
internal const string CertMicrosoftKekCa2011 = "Microsoft Corporation KEK CA 2011";
private const int ErrorInvalidFunction = 1; // legacy BIOS / no UEFI
private const int ErrorInsufficientBuffer = 122;
private const int ErrorEnvvarNotFound = 203;
private const int ErrorPrivilegeNotHeld = 1314;
private const int ErrorNotAllAssigned = 1300;
private const int InitialBufferSize = 64 * 1024;
private const int RetryBufferSize = 256 * 1024;
// SeSystemEnvironmentPrivilege is adjusted on the PROCESS token, so enable → read →
// restore is a process-wide side effect and must run as one atomic sequence. Without
// this gate a second reader would observe "already enabled", correctly skip its own
// restore, and then have the privilege disabled underneath it when the first reader
// finishes — a spurious privilege_denied on a machine that actually holds it.
private static readonly object FirmwareReadGate = new object();
/// <summary>
/// Reads db and KEK and evaluates the certificate presence flags. Never throws;
/// per-variable failures surface as <see cref="UefiSecureBootCertSnapshot.DbStatus"/> /
/// <see cref="UefiSecureBootCertSnapshot.KekStatus"/>. Certificate flags are only
/// meaningful when the corresponding status is <see cref="StatusOk"/>.
/// </summary>
public static UefiSecureBootCertSnapshot Read()
{
var snapshot = new UefiSecureBootCertSnapshot();
try
{
// The critical section covers the whole privileged window, not just the
// adjustment — see FirmwareReadGate. Only synchronous kernel calls and byte
// scans run inside it, so it cannot block on anything else the agent does.
lock (FirmwareReadGate)
using (var privilege = SystemEnvironmentPrivilege.TryEnable())
{
if (!privilege.Enabled)
{
snapshot.DbStatus = StatusPrivilegeDenied;
snapshot.KekStatus = StatusPrivilegeDenied;
return snapshot;
}
byte[] db;
snapshot.DbStatus = TryReadVariable("db", DbVariableGuid, out db);
if (snapshot.DbStatus == StatusOk)
{
snapshot.DbHasWindowsUefiCa2023 = ContainsAscii(db, CertWindowsUefiCa2023);
snapshot.DbHasMicrosoftUefiCa2023 = ContainsAscii(db, CertMicrosoftUefiCa2023);
snapshot.DbHasMicrosoftOptionRomCa2023 = ContainsAscii(db, CertMicrosoftOptionRomCa2023);
snapshot.DbHasWindowsProductionPca2011 = ContainsAscii(db, CertWindowsProductionPca2011);
snapshot.DbHasMicrosoftUefiCa2011 = ContainsAscii(db, CertMicrosoftUefiCa2011);
}
byte[] kek;
snapshot.KekStatus = TryReadVariable("KEK", KekVariableGuid, out kek);
if (snapshot.KekStatus == StatusOk)
{
snapshot.KekHasMicrosoftKek2kCa2023 = ContainsAscii(kek, CertMicrosoftKek2kCa2023);
snapshot.KekHasMicrosoftKekCa2011 = ContainsAscii(kek, CertMicrosoftKekCa2011);
}
}
}
catch (Exception ex)
{
var status = "error_" + ex.GetType().Name;
if (snapshot.DbStatus == null) snapshot.DbStatus = status;
if (snapshot.KekStatus == null) snapshot.KekStatus = status;
}
return snapshot;
}
/// <summary>
/// Reads one firmware variable into <paramref name="bytes"/> (trimmed to the actual
/// length). Returns a status string; <paramref name="bytes"/> is null unless the
/// status is <see cref="StatusOk"/>.
/// </summary>
private static string TryReadVariable(string name, string guid, out byte[] bytes)
{
bytes = null;
var buffer = new byte[InitialBufferSize];
var length = NativeMethods.GetFirmwareEnvironmentVariableExW(name, guid, buffer, (uint)buffer.Length, IntPtr.Zero);
if (length == 0)
{
var error = Marshal.GetLastWin32Error();
if (error == ErrorInsufficientBuffer)
{
buffer = new byte[RetryBufferSize];
length = NativeMethods.GetFirmwareEnvironmentVariableExW(name, guid, buffer, (uint)buffer.Length, IntPtr.Zero);
if (length == 0)
error = Marshal.GetLastWin32Error();
}
if (length == 0)
{
switch (error)
{
case ErrorInvalidFunction: return StatusNotUefi;
case ErrorEnvvarNotFound: return StatusVariableNotFound;
case ErrorPrivilegeNotHeld: return StatusPrivilegeDenied;
default: return "error_" + error;
}
}
}
bytes = new byte[length];
Array.Copy(buffer, bytes, (int)length);
return StatusOk;
}
/// <summary>
/// Byte-level ASCII substring search (ordinal, case-sensitive). The signature lists
/// are at most a few hundred KB and read once per agent start — a naive scan is fine.
/// </summary>
internal static bool ContainsAscii(byte[] haystack, string needle)
{
if (haystack == null || string.IsNullOrEmpty(needle) || needle.Length > haystack.Length)
return false;
for (var i = 0; i <= haystack.Length - needle.Length; i++)
{
var j = 0;
while (j < needle.Length && haystack[i + j] == (byte)needle[j])
j++;
if (j == needle.Length)
return true;
}
return false;
}
/// <summary>
/// Enables <c>SeSystemEnvironmentPrivilege</c> on the process token for the lifetime
/// of the instance and restores the previous privilege state on dispose. SYSTEM holds
/// the privilege (default-disabled); without elevation <see cref="Enabled"/> stays
/// false and the caller reports <see cref="StatusPrivilegeDenied"/>.
/// <para>
/// <b>Crash-safety.</b> This is the repo's only privilege-adjustment code, so the
/// ownership rules are spelled out: the token is held in a <see cref="SafeHandle"/>
/// assigned by the P/Invoke marshaler, so the CLR owns it from the moment
/// <c>OpenProcessToken</c> returns — it is released by the critical finalizer even if
/// this instance never reaches the caller's <c>using</c> (async exception between the
/// call and the assignment). <see cref="Dispose"/> is idempotent, never throws, and
/// releases the handle in a <c>finally</c> so a failed restore cannot leak it. The
/// restore obligation is recorded immediately after the successful adjustment, so no
/// throw in between can strand the process with the privilege left enabled.
/// </para>
/// </summary>
private sealed class SystemEnvironmentPrivilege : IDisposable
{
public bool Enabled { get; private set; }
private SafeTokenHandle _token;
private NativeMethods.TOKEN_PRIVILEGES _previousState;
private bool _restoreNeeded;
private bool _disposed;
public static SystemEnvironmentPrivilege TryEnable()
{
var instance = new SystemEnvironmentPrivilege();
try
{
SafeTokenHandle token;
if (!NativeMethods.OpenProcessToken(NativeMethods.GetCurrentProcess(),
NativeMethods.TOKEN_ADJUST_PRIVILEGES | NativeMethods.TOKEN_QUERY, out token))
{
// Failed open still yields a (closed/invalid) SafeHandle instance.
if (token != null) token.Dispose();
return instance;
}
instance._token = token;
NativeMethods.LUID luid;
if (!NativeMethods.LookupPrivilegeValueW(null, "SeSystemEnvironmentPrivilege", out luid))
return instance;
var newState = new NativeMethods.TOKEN_PRIVILEGES
{
PrivilegeCount = 1,
Luid = luid,
Attributes = NativeMethods.SE_PRIVILEGE_ENABLED
};
NativeMethods.TOKEN_PRIVILEGES previousState;
uint returnLength;
if (!NativeMethods.AdjustTokenPrivileges(token, false, ref newState,
(uint)Marshal.SizeOf(typeof(NativeMethods.TOKEN_PRIVILEGES)),
out previousState, out returnLength))
return instance;
// AdjustTokenPrivileges returns TRUE even when the privilege is not held —
// the real verdict is ERROR_NOT_ALL_ASSIGNED in the last error. Nothing was
// adjusted in that case, so there is nothing to restore.
if (Marshal.GetLastWin32Error() == ErrorNotAllAssigned)
return instance;
// The token is now modified: record the restore obligation before anything
// else can throw. Restore only when the privilege was not already enabled.
instance._previousState = previousState;
instance._restoreNeeded = previousState.PrivilegeCount > 0
&& (previousState.Attributes & NativeMethods.SE_PRIVILEGE_ENABLED) == 0;
instance.Enabled = true;
}
catch
{
// Fail-soft: Enabled stays false. Whatever was acquired before the failure
// is already tracked on the instance and released by Dispose.
}
return instance;
}
public void Dispose()
{
if (_disposed) return;
_disposed = true;
Enabled = false;
var token = _token;
_token = null;
if (token == null) return;
try
{
if (_restoreNeeded)
{
_restoreNeeded = false;
NativeMethods.TOKEN_PRIVILEGES ignored;
uint returnLength;
NativeMethods.AdjustTokenPrivileges(token, false, ref _previousState,
(uint)Marshal.SizeOf(typeof(NativeMethods.TOKEN_PRIVILEGES)),
out ignored, out returnLength);
}
}
catch
{
// Best effort — the process token privilege state is not load-bearing for
// anything else the agent does, and Dispose must never throw.
}
finally
{
token.Dispose();
}
}
}
/// <summary>
/// Owns a process-token handle. <c>ReleaseHandle</c> runs on the finalizer thread when
/// <see cref="SystemEnvironmentPrivilege.Dispose"/> is skipped, so it must not throw —
/// a single <c>CloseHandle</c> P/Invoke satisfies that.
/// </summary>
private sealed class SafeTokenHandle : SafeHandleZeroOrMinusOneIsInvalid
{
public SafeTokenHandle() : base(true) { }
protected override bool ReleaseHandle()
{
return NativeMethods.CloseHandle(handle);
}
}
private static class NativeMethods
{
public const uint TOKEN_ADJUST_PRIVILEGES = 0x0020;
public const uint TOKEN_QUERY = 0x0008;
public const uint SE_PRIVILEGE_ENABLED = 0x0002;
[StructLayout(LayoutKind.Sequential)]
public struct LUID
{
public uint LowPart;
public int HighPart;
}
// Single-privilege variant of the variable-length native struct — sufficient here
// because exactly one privilege is adjusted per call.
[StructLayout(LayoutKind.Sequential)]
public struct TOKEN_PRIVILEGES
{
public uint PrivilegeCount;
public LUID Luid;
public uint Attributes;
}
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
public static extern uint GetFirmwareEnvironmentVariableExW(
string lpName, string lpGuid, byte[] pBuffer, uint nSize, IntPtr pdwAttributes);
[DllImport("kernel32.dll")]
public static extern IntPtr GetCurrentProcess();
[DllImport("advapi32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool OpenProcessToken(IntPtr processHandle, uint desiredAccess, out SafeTokenHandle tokenHandle);
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool LookupPrivilegeValueW(string lpSystemName, string lpName, out LUID lpLuid);
[DllImport("advapi32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool AdjustTokenPrivileges(SafeTokenHandle tokenHandle, bool disableAllPrivileges,
ref TOKEN_PRIVILEGES newState, uint bufferLength, out TOKEN_PRIVILEGES previousState, out uint returnLength);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool CloseHandle(IntPtr handle);
}
}
/// <summary>
/// Result of one firmware read. Certificate flags are only meaningful when the
/// corresponding status equals <see cref="UefiSecureBootCertReader.StatusOk"/>.
/// </summary>
internal sealed class UefiSecureBootCertSnapshot
{
public string DbStatus { get; set; }
public string KekStatus { get; set; }
public bool DbHasWindowsUefiCa2023 { get; set; }
public bool DbHasMicrosoftUefiCa2023 { get; set; }
public bool DbHasMicrosoftOptionRomCa2023 { get; set; }
public bool DbHasWindowsProductionPca2011 { get; set; }
public bool DbHasMicrosoftUefiCa2011 { get; set; }
public bool KekHasMicrosoftKek2kCa2023 { get; set; }
public bool KekHasMicrosoftKekCa2011 { get; set; }
}
}
@@ -0,0 +1,194 @@
using System;
using System.Runtime.InteropServices;
using System.Text;
using static AutopilotMonitor.Agent.V2.Core.Monitoring.Interop.WlanNativeMethods;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// Current WiFi connection details as read from the native WLAN API.
/// </summary>
public sealed class WifiConnectionInfo
{
/// <summary>SSID of the connected network; null when the AP broadcasts an empty SSID.</summary>
public string Ssid { get; internal set; }
/// <summary>Signal quality 0–100 (same semantics as the netsh "Signal" percentage).</summary>
public int SignalPercent { get; internal set; }
/// <summary>Human-readable PHY generation ("802.11ax" …); null for exotic/unknown PHYs.</summary>
public string RadioType { get; internal set; }
/// <summary>Current 802.11 channel number; null when the channel query fails.</summary>
public int? Channel { get; internal set; }
}
/// <summary>
/// Fail-soft reader over <see cref="WlanNativeMethods"/>: returns the current WiFi
/// connection or null — never throws, never blocks on a child process. Machines without
/// the WLAN AutoConfig service (VMs, wired-only devices) simply return null.
/// </summary>
public static class WifiInfoReader
{
/// <summary>
/// Reads the current WiFi connection. When <paramref name="preferredInterfaceId"/> is
/// given (the adapter GUID from <c>NetworkInterface.Id</c>), that interface is queried
/// first; otherwise — or when it is not connected — the first connected WLAN interface
/// wins. Returns null when no interface has an active connection.
/// </summary>
public static WifiConnectionInfo TryGetCurrentConnection(Guid? preferredInterfaceId = null)
{
var clientHandle = IntPtr.Zero;
try
{
if (WlanOpenHandle(WLAN_CLIENT_VERSION, IntPtr.Zero, out _, out clientHandle) != ERROR_SUCCESS)
return null;
var interfaceList = IntPtr.Zero;
try
{
if (WlanEnumInterfaces(clientHandle, IntPtr.Zero, out interfaceList) != ERROR_SUCCESS)
return null;
var count = (uint)Marshal.ReadInt32(interfaceList);
var infoSize = Marshal.SizeOf(typeof(WLAN_INTERFACE_INFO));
// List header: DWORD dwNumberOfItems + DWORD dwIndex, then the info array.
var firstEntry = new IntPtr(interfaceList.ToInt64() + 8);
WifiConnectionInfo fallback = null;
for (var i = 0; i < count; i++)
{
var entryPtr = new IntPtr(firstEntry.ToInt64() + i * infoSize);
var info = (WLAN_INTERFACE_INFO)Marshal.PtrToStructure(entryPtr, typeof(WLAN_INTERFACE_INFO));
var connection = QueryCurrentConnection(clientHandle, info.InterfaceGuid);
if (connection == null)
continue;
if (preferredInterfaceId.HasValue && info.InterfaceGuid == preferredInterfaceId.Value)
return connection;
if (fallback == null)
fallback = connection;
}
return fallback;
}
finally
{
if (interfaceList != IntPtr.Zero)
WlanFreeMemory(interfaceList);
}
}
catch
{
return null;
}
finally
{
if (clientHandle != IntPtr.Zero)
WlanCloseHandle(clientHandle, IntPtr.Zero);
}
}
private static WifiConnectionInfo QueryCurrentConnection(IntPtr clientHandle, Guid interfaceGuid)
{
var dataPtr = IntPtr.Zero;
try
{
if (WlanQueryInterface(clientHandle, ref interfaceGuid,
WLAN_INTF_OPCODE.wlan_intf_opcode_current_connection,
IntPtr.Zero, out _, out dataPtr, IntPtr.Zero) != ERROR_SUCCESS)
return null;
var attributes = (WLAN_CONNECTION_ATTRIBUTES)Marshal.PtrToStructure(
dataPtr, typeof(WLAN_CONNECTION_ATTRIBUTES));
if (attributes.isState != WLAN_INTERFACE_STATE.wlan_interface_state_connected)
return null;
var assoc = attributes.wlanAssociationAttributes;
return new WifiConnectionInfo
{
Ssid = DecodeSsid(assoc.dot11Ssid.ucSSID, assoc.dot11Ssid.uSSIDLength),
SignalPercent = (int)Math.Min(assoc.wlanSignalQuality, 100),
RadioType = PhyTypeToRadioType(assoc.dot11PhyType),
Channel = QueryChannel(clientHandle, interfaceGuid),
};
}
finally
{
if (dataPtr != IntPtr.Zero)
WlanFreeMemory(dataPtr);
}
}
private static int? QueryChannel(IntPtr clientHandle, Guid interfaceGuid)
{
var dataPtr = IntPtr.Zero;
try
{
if (WlanQueryInterface(clientHandle, ref interfaceGuid,
WLAN_INTF_OPCODE.wlan_intf_opcode_channel_number,
IntPtr.Zero, out var dataSize, out dataPtr, IntPtr.Zero) != ERROR_SUCCESS)
return null;
if (dataSize < sizeof(uint) || dataPtr == IntPtr.Zero)
return null;
return Marshal.ReadInt32(dataPtr);
}
catch
{
return null;
}
finally
{
if (dataPtr != IntPtr.Zero)
WlanFreeMemory(dataPtr);
}
}
/// <summary>
/// Decodes a raw DOT11_SSID (UTF-8 by convention) into a string; null for empty SSIDs.
/// Internal for direct unit-testing via InternalsVisibleTo.
/// </summary>
internal static string DecodeSsid(byte[] raw, uint length)
{
if (raw == null || length == 0)
return null;
var byteCount = (int)Math.Min(length, (uint)raw.Length);
try
{
var ssid = Encoding.UTF8.GetString(raw, 0, byteCount);
return string.IsNullOrEmpty(ssid) ? null : ssid;
}
catch
{
return null;
}
}
/// <summary>
/// Maps a DOT11 PHY type to the same strings netsh historically reported as
/// "Radio type" — keeps event payloads consistent across the netsh→WLAN-API switch.
/// Internal for direct unit-testing via InternalsVisibleTo.
/// </summary>
internal static string PhyTypeToRadioType(DOT11_PHY_TYPE phyType)
{
switch (phyType)
{
case DOT11_PHY_TYPE.dot11_phy_type_ofdm: return "802.11a";
case DOT11_PHY_TYPE.dot11_phy_type_hrdsss: return "802.11b";
case DOT11_PHY_TYPE.dot11_phy_type_erp: return "802.11g";
case DOT11_PHY_TYPE.dot11_phy_type_ht: return "802.11n";
case DOT11_PHY_TYPE.dot11_phy_type_vht: return "802.11ac";
case DOT11_PHY_TYPE.dot11_phy_type_dmg: return "802.11ad";
case DOT11_PHY_TYPE.dot11_phy_type_he: return "802.11ax";
case DOT11_PHY_TYPE.dot11_phy_type_eht: return "802.11be";
default: return null;
}
}
}
}
@@ -0,0 +1,142 @@
using System;
using System.Runtime.InteropServices;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// Native WLAN API interop (<c>wlanapi.dll</c>) used to read the current WiFi connection
/// (SSID, signal quality, PHY type, channel) without spawning <c>netsh</c>.
/// <para>
/// The WLAN API is language-neutral — unlike parsing <c>netsh wlan show interfaces</c>
/// output, whose labels follow the OS UI language and silently drop fields on
/// non-English images. On machines without the WLAN AutoConfig service (VMs, servers,
/// wired-only devices) <c>WlanOpenHandle</c>/<c>WlanEnumInterfaces</c> fail with a
/// non-zero error code; callers treat that as "no WiFi" (see <see cref="WifiInfoReader"/>).
/// </para>
/// </summary>
internal static class WlanNativeMethods
{
internal const uint ERROR_SUCCESS = 0;
/// <summary>Client version 2 = Vista and later (the only relevant range).</summary>
internal const uint WLAN_CLIENT_VERSION = 2;
internal enum WLAN_INTF_OPCODE : uint
{
wlan_intf_opcode_current_connection = 7,
wlan_intf_opcode_channel_number = 8,
}
internal enum WLAN_INTERFACE_STATE
{
wlan_interface_state_not_ready = 0,
wlan_interface_state_connected = 1,
wlan_interface_state_ad_hoc_network_formed = 2,
wlan_interface_state_disconnecting = 3,
wlan_interface_state_disconnected = 4,
wlan_interface_state_associating = 5,
wlan_interface_state_discovering = 6,
wlan_interface_state_authenticating = 7,
}
internal enum DOT11_PHY_TYPE : uint
{
dot11_phy_type_unknown = 0,
dot11_phy_type_fhss = 1,
dot11_phy_type_dsss = 2,
dot11_phy_type_irbaseband = 3,
dot11_phy_type_ofdm = 4, // 802.11a
dot11_phy_type_hrdsss = 5, // 802.11b
dot11_phy_type_erp = 6, // 802.11g
dot11_phy_type_ht = 7, // 802.11n
dot11_phy_type_vht = 8, // 802.11ac
dot11_phy_type_dmg = 9, // 802.11ad
dot11_phy_type_he = 10, // 802.11ax
dot11_phy_type_eht = 11, // 802.11be
}
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
internal struct WLAN_INTERFACE_INFO
{
public Guid InterfaceGuid;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 256)]
public string strInterfaceDescription;
public WLAN_INTERFACE_STATE isState;
}
[StructLayout(LayoutKind.Sequential)]
internal struct DOT11_SSID
{
public uint uSSIDLength;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 32)]
public byte[] ucSSID;
}
[StructLayout(LayoutKind.Sequential)]
internal struct WLAN_ASSOCIATION_ATTRIBUTES
{
public DOT11_SSID dot11Ssid;
public int dot11BssType;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 6)]
public byte[] dot11Bssid;
public DOT11_PHY_TYPE dot11PhyType;
public uint uDot11PhyIndex;
public uint wlanSignalQuality; // 0..100, same semantics as netsh "Signal"
public uint ulRxRate;
public uint ulTxRate;
}
[StructLayout(LayoutKind.Sequential)]
internal struct WLAN_SECURITY_ATTRIBUTES
{
[MarshalAs(UnmanagedType.Bool)]
public bool bSecurityEnabled;
[MarshalAs(UnmanagedType.Bool)]
public bool bOneXEnabled;
public int dot11AuthAlgorithm;
public int dot11CipherAlgorithm;
}
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
internal struct WLAN_CONNECTION_ATTRIBUTES
{
public WLAN_INTERFACE_STATE isState;
public int wlanConnectionMode;
[MarshalAs(UnmanagedType.ByValTStr, SizeConst = 256)]
public string strProfileName;
public WLAN_ASSOCIATION_ATTRIBUTES wlanAssociationAttributes;
public WLAN_SECURITY_ATTRIBUTES wlanSecurityAttributes;
}
[DllImport("wlanapi.dll")]
internal static extern uint WlanOpenHandle(
uint dwClientVersion,
IntPtr pReserved,
out uint pdwNegotiatedVersion,
out IntPtr phClientHandle);
[DllImport("wlanapi.dll")]
internal static extern uint WlanCloseHandle(
IntPtr hClientHandle,
IntPtr pReserved);
[DllImport("wlanapi.dll")]
internal static extern uint WlanEnumInterfaces(
IntPtr hClientHandle,
IntPtr pReserved,
out IntPtr ppInterfaceList);
[DllImport("wlanapi.dll")]
internal static extern uint WlanQueryInterface(
IntPtr hClientHandle,
[In] ref Guid pInterfaceGuid,
WLAN_INTF_OPCODE OpCode,
IntPtr pReserved,
out uint pdwDataSize,
out IntPtr ppData,
IntPtr pWlanOpcodeValueType);
[DllImport("wlanapi.dll")]
internal static extern void WlanFreeMemory(IntPtr pMemory);
}
}
@@ -31,84 +31,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
_logger.Info($"Executing FULL SELF-DESTRUCT (Scheduled Task + Files{(_configuration.RebootOnComplete ? " + Reboot" : "")})");
var currentProcessId = Process.GetCurrentProcess().Id;
var currentProcess = Process.GetCurrentProcess();
var agentBasePath = Path.GetFullPath(Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"AutopilotMonitor"
));
var logDir = Path.GetFullPath(Path.Combine(agentBasePath, "Logs"));
var keepLogs = _configuration.KeepLogFile;
// Create a self-deleting PowerShell script
var cleanupScript = $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Wait for agent process to exit
Start-Sleep -Seconds 2
Wait-Process -Id {currentProcessId} -ErrorAction SilentlyContinue -Timeout 30
Start-Sleep -Seconds 1
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan task firing
# against a deleted exe spams Task Scheduler errors every boot forever and, once the files (incl.
# the enrollment-complete marker) are gone, has no cleanup-retry path. So confirm the task is gone
# first; if it cannot be removed, SKIP the file deletion and leave everything intact for a clean
# cleanup retry on the next boot (the marker survives, so bootstrap re-runs cleanup).
$taskName = '{_configuration.ScheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
{(keepLogs ? $@"
if ($taskGone) {{
# Delete everything except Logs directory
Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' | ForEach-Object {{
$dest = $_.FullName + '.del'
try {{ Rename-Item -Path $_.FullName -NewName $dest -Force -ErrorAction SilentlyContinue }} catch {{ }}
Remove-Item -Path $dest -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
" : $@"
if ($taskGone) {{
# Rename folder then delete. Retry a few times to let the OS release handles.
$renamedPath = '{agentBasePath}.del'
$renamed = $false
for ($i = 1; $i -le 10; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$renamed = $true
break
}} catch {{
Start-Sleep -Seconds 2
}}
}}
if ($renamed) {{
Remove-Item -Path $renamedPath -Recurse -Force -ErrorAction SilentlyContinue
}} else {{
Remove-Item -Path '{agentBasePath}' -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
")}
{(_configuration.RebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
";
var cleanupScript = BuildCleanupScript(
agentProcessName: currentProcess.ProcessName,
agentBasePath: agentBasePath,
keepLogs: _configuration.KeepLogFile,
scheduledTaskName: _configuration.ScheduledTaskName,
rebootOnComplete: _configuration.RebootOnComplete);
// Write cleanup script to temp location (outside of agent folder)
var tempScriptPath = Path.Combine(Path.GetTempPath(), $"AutopilotMonitor-Cleanup-{Guid.NewGuid()}.ps1");
@@ -143,5 +77,171 @@ Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}
}
/// <summary>
/// Builds the self-deleting cleanup PowerShell script. Extracted (internal static, pure)
/// so tests can pin the hardened invariants without launching processes.
/// <para>
/// LIFE-F7 hardening (incident 2026-07-22 — exe + loaded DLLs orphaned forever):
/// </para>
/// <list type="number">
/// <item><b>Single-flight lock</b> — the enrollment-complete marker is written BEFORE
/// cleanup launches, so any agent start during an in-flight cleanup spawns a SECOND
/// cleanup script. Two unsynchronized scripts interleave their rename/delete phases
/// and leave partial trees behind. A CreateNew lock file in the SYSTEM temp dir
/// serializes them; a stale lock (&gt;10 min) is taken over.</item>
/// <item><b>Wait by process NAME, not launcher PID</b> — marker-retry instances,
/// self-update restarts and manual runs all load the same images but have PIDs the
/// old <c>Wait-Process -Id</c> never saw (and an already-freed PID can be reused by
/// an unrelated boot-time process). After the deadline any straggler is force-killed:
/// its images are about to be deleted anyway.</item>
/// <item><b>Probe-rename BEFORE task removal, never delete the original path</b> — the
/// rename is the atomic lock probe. If it fails, something still holds the tree: abort
/// with task + files + marker intact so the next boot retries (file-side symmetry to
/// the LIFE-F6 task-side rule). If the task cannot be removed afterwards, rename BACK
/// and abort for the same reason. Only a fully renamed tree with the task gone is
/// deleted — the old <c>Remove-Item</c>-on-original fallback deleted marker + state
/// around locked binaries AFTER the task was gone, converting a transient lock into a
/// permanently orphaned exe.</item>
/// </list>
/// </summary>
/// <remarks>
/// The timing parameters exist ONLY so integration tests can replay the incident
/// scenarios (locked tree, straggler process) in seconds instead of minutes —
/// production call sites always use the defaults.
/// </remarks>
internal static string BuildCleanupScript(
string agentProcessName,
string agentBasePath,
bool keepLogs,
string scheduledTaskName,
bool rebootOnComplete,
int initialDelaySeconds = 2,
int processWaitSeconds = 60,
int renameRetries = 10,
int renameRetryDelaySeconds = 2)
{
return $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Single-flight lock (LIFE-F7 #1): serialize concurrent cleanup scripts. CreateNew is
# atomic; the loser backs off and lets the winner own the deletion.
$lockPath = Join-Path ([System.IO.Path]::GetTempPath()) 'AutopilotMonitor-Cleanup.lock'
$lockStream = $null
try {{
if (Test-Path $lockPath) {{
$lockAge = (Get-Date) - (Get-Item $lockPath -ErrorAction Stop).LastWriteTime
if ($lockAge.TotalMinutes -gt 10) {{ Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue }}
}}
$lockStream = [System.IO.File]::Open($lockPath, [System.IO.FileMode]::CreateNew, [System.IO.FileAccess]::Write, [System.IO.FileShare]::None)
}} catch {{
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
exit 0
}}
try {{
# Wait for EVERY agent process to exit, by NAME (LIFE-F7 #2) — covers marker-retry
# instances, self-update restarts and manual runs, and is immune to PID reuse.
Start-Sleep -Seconds {initialDelaySeconds}
$deadline = (Get-Date).AddSeconds({processWaitSeconds})
while ((Get-Date) -lt $deadline) {{
if (-not (Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue)) {{ break }}
Start-Sleep -Seconds 2
}}
# Backstop: force-kill stragglers — their images are about to be deleted anyway.
Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
Start-Sleep -Seconds 1
{(keepLogs ? $@"
# Probe-rename all children except Logs (LIFE-F7 #3). Any failure ⇒ rename back and
# abort with the Scheduled Task intact so the next boot retries cleanly.
$renamed = @()
$probeOk = $true
foreach ($item in @(Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' -ErrorAction SilentlyContinue)) {{
$dest = $item.FullName + '.del-' + [Guid]::NewGuid().ToString('N')
$itemOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path $item.FullName -NewName $dest -Force -ErrorAction Stop
$itemOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
if ($itemOk) {{ $renamed += @{{ Original = $item.FullName; Renamed = $dest }} }}
else {{ $probeOk = $false; break }}
}}
" : $@"
# Probe-rename the whole tree (LIFE-F7 #3). Failure ⇒ abort with task + files + marker
# intact so the next boot retries cleanly. GUID suffix avoids collisions with leftovers
# of earlier attempts.
$renamedPath = '{agentBasePath}.del-' + [Guid]::NewGuid().ToString('N')
$probeOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$probeOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
$renamed = @(@{{ Original = '{agentBasePath}'; Renamed = $renamedPath }})
")}
if ($probeOk) {{
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan
# task firing against a deleted exe spams Task Scheduler errors every boot forever
# and, once the files (incl. the enrollment-complete marker) are gone, has no
# cleanup-retry path.
$taskName = '{scheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
if ($taskGone) {{
# Task gone + tree renamed ⇒ safe to delete. Never touches the original path.
foreach ($entry in $renamed) {{
Remove-Item -Path $entry.Renamed -Recurse -Force -ErrorAction SilentlyContinue
}}
}} else {{
# Task survived ⇒ rename back and leave everything for the next-boot retry
# (the marker survives, so bootstrap re-runs cleanup).
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
}} else {{
# Probe failed ⇒ something still holds the tree. Rename back whatever succeeded and
# abort WITHOUT deleting anything — task + marker stay, next boot retries.
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
{(rebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
}} finally {{
if ($lockStream) {{ $lockStream.Dispose() }}
Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}}
";
}
}
}
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Management;
using System.Text.RegularExpressions;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
@@ -28,6 +29,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
///
/// Emits a single "local_admin_analysis" event at startup and at shutdown,
/// enabling delta detection between pre- and post-enrollment state.
///
/// Tenant-supplied allowed accounts may contain glob wildcards (e.g. "adm-*");
/// see <see cref="MatchesAllowedEntry"/>.
/// </summary>
public class LocalAdminAnalyzer : IAgentAnalyzer
{
@@ -246,6 +250,43 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
}
}
// -----------------------------------------------------------------------
// Allowed-list matching
// -----------------------------------------------------------------------
/// <summary>
/// Matches an account or profile-folder name against one allowed-list entry.
/// Entries containing <c>*</c> or <c>?</c> are glob patterns (same semantics as the
/// hardware whitelist: <c>*</c> = any run, <c>?</c> = one char, case-insensitive);
/// all other entries match exactly. No ambiguity with literal names — <c>*</c> and
/// <c>?</c> are invalid characters in Windows account names.
/// Internal for direct unit-testing via InternalsVisibleTo.
/// </summary>
internal static bool MatchesAllowedEntry(string name, string entry)
{
if (entry.IndexOf('*') < 0 && entry.IndexOf('?') < 0)
return string.Equals(entry, name, StringComparison.OrdinalIgnoreCase);
var regexPattern = "^" + Regex.Escape(entry)
.Replace("\\*", ".*")
.Replace("\\?", ".") + "$";
try
{
// Timeout guards against pathological patterns (ReDoS)
return Regex.IsMatch(name, regexPattern, RegexOptions.IgnoreCase, TimeSpan.FromMilliseconds(100));
}
catch (RegexMatchTimeoutException)
{
return false;
}
}
private static bool IsAllowed(string name, List<string> allowedAccounts)
{
return allowedAccounts.Any(a => MatchesAllowedEntry(name, a));
}
// -----------------------------------------------------------------------
// Individual checks
// -----------------------------------------------------------------------
@@ -308,8 +349,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
if (disabled)
continue;
if (!allowedAccounts.Any(a =>
string.Equals(a, name, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(name, allowedAccounts))
{
unexpected.Add(name);
_logger.Debug($"{Name}: Unexpected local account: {name}");
@@ -349,8 +389,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
allFound.Add(folderName);
if (!allowedAccounts.Any(a =>
string.Equals(a, folderName, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(folderName, allowedAccounts))
{
unexpected.Add(folderName);
_logger.Debug($"{Name}: Unexpected profile folder: {folderName}");
@@ -8,6 +8,7 @@ using System.Text.Json;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Security;
using AutopilotMonitor.DecisionCore.Engine;
@@ -493,7 +494,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
}
}
EmitDeviceInfoEvent(Constants.EventTypes.SecureBootStatus, $"SecureBoot: {data["uefiSecureBootEnabled"]}, CA2023: {data["uefiCA2023Status"]}", data);
// Direct firmware evidence (UEFI db/KEK variables) — authoritative for the
// CA-2023 rollout state. The Servicing key above only mirrors whether the
// Windows Update rollout ran; it lags behind when the certificate reached
// the firmware another way (OEM factory image, WinCS, manual deployment).
var firmware = UefiSecureBootCertReader.Read();
AppendFirmwareCertFields(data, firmware);
EmitDeviceInfoEvent(Constants.EventTypes.SecureBootStatus,
$"SecureBoot: {data["uefiSecureBootEnabled"]}, CA2023: {data["uefiCA2023Status"]}, firmware: {DescribeFirmwareVerdict(firmware)}",
data);
}
catch (Exception ex)
{
@@ -501,6 +511,50 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
}
}
/// <summary>
/// Maps a <see cref="UefiSecureBootCertSnapshot"/> onto the <c>secureboot_status</c>
/// payload. Certificate booleans are only emitted when the respective variable read
/// succeeded — an absent boolean means "firmware state unknown", never "missing".
/// </summary>
internal static void AppendFirmwareCertFields(Dictionary<string, object> data, UefiSecureBootCertSnapshot firmware)
{
if (firmware == null) return;
data["uefiFirmwareReadStatus"] = firmware.DbStatus != UefiSecureBootCertReader.StatusOk
? firmware.DbStatus
: firmware.KekStatus;
if (firmware.DbStatus == UefiSecureBootCertReader.StatusOk)
{
data["uefiDbHasWindowsUefiCa2023"] = firmware.DbHasWindowsUefiCa2023;
data["uefiDbHasMicrosoftUefiCa2023"] = firmware.DbHasMicrosoftUefiCa2023;
data["uefiDbHasMicrosoftOptionRomCa2023"] = firmware.DbHasMicrosoftOptionRomCa2023;
data["uefiDbHasWindowsProductionPca2011"] = firmware.DbHasWindowsProductionPca2011;
data["uefiDbHasMicrosoftUefiCa2011"] = firmware.DbHasMicrosoftUefiCa2011;
// One-sided marker consumed by ANALYZE-SEC-001's not_exists precondition.
// The engine's not_exists passes only while the field is absent/empty, so the
// marker must be OMITTED (never emitted as false) unless the firmware db
// positively confirms the 2023 certificate — old agents and unconfirmed
// devices then keep evaluating the rule.
if (firmware.DbHasWindowsUefiCa2023)
data["uefiCa2023FirmwareConfirmed"] = true;
}
if (firmware.KekStatus == UefiSecureBootCertReader.StatusOk)
{
data["uefiKekHasMicrosoftKek2kCa2023"] = firmware.KekHasMicrosoftKek2kCa2023;
data["uefiKekHasMicrosoftKekCa2011"] = firmware.KekHasMicrosoftKekCa2011;
}
}
internal static string DescribeFirmwareVerdict(UefiSecureBootCertSnapshot firmware)
{
if (firmware == null) return "unavailable";
if (firmware.DbStatus != UefiSecureBootCertReader.StatusOk) return firmware.DbStatus ?? "unavailable";
return firmware.DbHasWindowsUefiCa2023 ? "CA2023 confirmed" : "CA2023 missing";
}
private void CollectBitLockerStatus()
{
try
@@ -652,10 +706,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
/// the desktop via "Continue anyway".
/// </para>
/// </summary>
// Serializes the ESP-config read+emit across ALL callers (CollectAll on the startup
// ThreadPool task, CollectAtEnrollmentStart and RefreshEspConfiguration on their own
// Task.Run) — without it, two concurrent collects can interleave read-before-emit and
// the LATER emission can carry the OLDER (smaller) registry snapshot. The backend keeps
// the latest list-bearing emission as positive blocking evidence, so an out-of-order
// emission would silently shrink it (Codex review). Inside the lock each collect is
// read→emit atomic: a later emission always carries an equal-or-newer snapshot.
private readonly object _espConfigCollectLock = new object();
// Internal (not private) as a test seam: unlike CollectAll/CollectAtEnrollmentStart this
// method touches no live system state once both probes are overridden, so tests can
// drive the esp_config_detected event surface in isolation (InternalsVisibleTo).
internal (bool? skipUserStatusPage, bool? skipDeviceStatusPage) CollectEspConfiguration()
{
lock (_espConfigCollectLock)
{
return CollectEspConfigurationLocked();
}
}
private (bool? skipUserStatusPage, bool? skipDeviceStatusPage) CollectEspConfigurationLocked()
{
bool? skipUser = null;
bool? skipDevice = null;
@@ -744,7 +815,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
/// Collects active network interface details (name, type, link speed, MAC) using
/// System.Net.NetworkInformation — instant, no process spawn.
/// If the active interface is WiFi, fires off a separate async task to collect
/// WiFi signal info via netsh (fire-and-forget, never blocks).
/// WiFi signal info via the native WLAN API (fire-and-forget, never blocks).
/// </summary>
private void CollectActiveNetworkInterfaceInfo()
{
@@ -787,7 +858,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
// Fire-and-forget WiFi signal collection — separate event, never blocks
if (isWifi)
{
Task.Run(() => CollectWiFiSignalInfo());
Guid? nicGuid = Guid.TryParse(activeNic.Id, out var parsedGuid) ? parsedGuid : (Guid?)null;
Task.Run(() => CollectWiFiSignalInfo(nicGuid));
}
}
catch (Exception ex)
@@ -828,78 +900,37 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
}
/// <summary>
/// Collects WiFi signal info via netsh wlan show interfaces and emits a separate
/// wifi_signal_info event. Designed to run as fire-and-forget via Task.Run —
/// never blocks other collection. VMs without WiFi service simply get no event.
/// Collects WiFi signal info via the native WLAN API (<see cref="WifiInfoReader"/>,
/// language-neutral, no process spawn) and emits a separate wifi_signal_info event.
/// Designed to run as fire-and-forget via Task.Run — never blocks other collection.
/// VMs without WiFi service simply get no event.
/// </summary>
private void CollectWiFiSignalInfo()
private void CollectWiFiSignalInfo(Guid? interfaceGuid)
{
try
{
var psi = new ProcessStartInfo
{
FileName = SystemPaths.Netsh,
Arguments = "wlan show interfaces",
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true
};
string output;
using (var process = Process.Start(psi))
{
output = process.StandardOutput.ReadToEnd();
process.WaitForExit(5000);
}
if (string.IsNullOrEmpty(output))
var wifi = WifiInfoReader.TryGetCurrentConnection(interfaceGuid);
if (wifi == null)
return;
var data = new Dictionary<string, object>();
foreach (var line in output.Split(new[] { '\r', '\n' }, StringSplitOptions.RemoveEmptyEntries))
var data = new Dictionary<string, object>
{
var trimmed = line.Trim();
var colonIndex = trimmed.IndexOf(':');
if (colonIndex < 0) continue;
["wifiSignalPercent"] = wifi.SignalPercent
};
if (wifi.Ssid != null)
data["wifiSsid"] = wifi.Ssid;
if (wifi.RadioType != null)
data["wifiRadioType"] = wifi.RadioType;
if (wifi.Channel.HasValue)
data["wifiChannel"] = wifi.Channel.Value;
var key = trimmed.Substring(0, colonIndex).Trim();
var value = trimmed.Substring(colonIndex + 1).Trim();
var message = wifi.Ssid != null
? $"WiFi: {wifi.Ssid}, Signal: {wifi.SignalPercent}%"
: $"WiFi signal info, Signal: {wifi.SignalPercent}%";
if (wifi.RadioType != null)
message += $" ({wifi.RadioType})";
if (key.Equals("SSID", StringComparison.OrdinalIgnoreCase) &&
!key.Equals("BSSID", StringComparison.OrdinalIgnoreCase))
{
data["wifiSsid"] = value;
}
else if (key.Equals("Signal", StringComparison.OrdinalIgnoreCase))
{
if (int.TryParse(value.TrimEnd('%'), out var signal))
data["wifiSignalPercent"] = signal;
}
else if (key.Equals("Radio type", StringComparison.OrdinalIgnoreCase))
{
data["wifiRadioType"] = value;
}
else if (key.Equals("Channel", StringComparison.OrdinalIgnoreCase))
{
if (int.TryParse(value, out var channel))
data["wifiChannel"] = channel;
}
}
if (data.Count > 0)
{
var message = data.ContainsKey("wifiSsid")
? $"WiFi: {data["wifiSsid"]}"
: "WiFi signal info";
if (data.ContainsKey("wifiSignalPercent"))
message += $", Signal: {data["wifiSignalPercent"]}%";
if (data.ContainsKey("wifiRadioType"))
message += $" ({data["wifiRadioType"]})";
EmitDeviceInfoEvent(Constants.EventTypes.WifiSignalInfo, message, data);
}
EmitDeviceInfoEvent(Constants.EventTypes.WifiSignalInfo, message, data);
}
catch (Exception ex)
{
@@ -114,6 +114,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
CollectTpmStatus();
}
/// <summary>
/// Re-reads the ESP configuration (skip flags + ESPTrackingInfo blocking lists) and
/// re-emits <c>esp_config_detected</c> when the payload changed. Audit Q2: the earlier
/// emissions (agent start, DeviceSetup detection) fire before the registry lists are
/// complete — the ESP writes one timestamped subkey per CSP status write, so the
/// device-scope list grows progressively and the user-scope <c>S-&lt;SID&gt;</c> lists
/// appear only after sign-in. Called by <see cref="Orchestration.DeviceInfoHost"/> as
/// phase-driven one-shots (apps sub-phase opened, AccountSetup detected) — never on a
/// timer. The StartupEventGate keeps an unchanged payload silent, so a re-collect that
/// found nothing new emits nothing.
/// </summary>
public void RefreshEspConfiguration(string trigger)
{
_logger.Info($"EnrollmentTracker: re-collecting ESP configuration (trigger={trigger})");
CollectEspConfiguration();
}
/// <summary>
/// Re-collects device info that may change during enrollment (e.g. BitLocker enabled via policy).
/// Called at enrollment complete / FinalizingSetup transition to capture final state.
@@ -77,11 +77,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
protected override void Collect()
{
// Full path writes 18 keys: agent_version + process metrics (5: cpu, ws, private,
// threads, handles) + spool stats (4) + network delta (8: requests, failures,
// bytes_up/down, avg_latency, total_up/down/requests). cap=18 → HashHelpers.GetPrime(18)=23
// buckets → no resize on the 18th key (cap=16 would land at 17 buckets and resize).
var data = new Dictionary<string, object>(capacity: 18, StringComparer.Ordinal)
// Full path writes 19 keys: agent_version + process metrics (5: cpu, ws, private,
// threads, handles) + spool stats (4) + network delta (9: requests, failures,
// bytes_up/down, avg_latency, total_up/down/requests/latency). cap=19 →
// HashHelpers.GetPrime(19)=23 buckets → no resize on the 19th key.
var data = new Dictionary<string, object>(capacity: 19, StringComparer.Ordinal)
{
{ "agent_version", _agentVersion }
};
@@ -164,6 +164,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
data["net_total_bytes_up"] = currentNet.TotalBytesUp;
data["net_total_bytes_down"] = currentNet.TotalBytesDown;
data["net_total_requests"] = currentNet.RequestCount;
// Cumulative latency sum: total/requests = session-wide average HTTP
// round-trip. The backend projects that average onto the Session row
// (AvgApiLatencyMs) — cumulative counters make the write idempotent.
data["net_total_latency_ms"] = currentNet.TotalLatencyMs;
}
catch (Exception ex)
{
@@ -8,8 +8,8 @@ using System.Net.Sockets;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Security;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
@@ -37,7 +37,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
private const int DebounceDelayMs = 3000;
private const int ConnectivityCheckDelayMs = 5000;
private const int ConnectivityCheckTimeoutMs = 8000;
private const int WifiCollectTimeoutMs = 3000;
private const string Source = "NetworkChangeDetector";
/// <summary>
@@ -374,7 +373,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
// Collect WiFi SSID synchronously (needed for before/after comparison)
if (isWifi)
{
var wifiInfo = CollectWiFiInfo();
Guid? nicGuid = Guid.TryParse(activeNic.Id, out var parsedGuid) ? parsedGuid : (Guid?)null;
var wifiInfo = CollectWiFiInfo(nicGuid);
snapshot.WifiSsid = wifiInfo.ssid;
snapshot.WifiSignalPercent = wifiInfo.signal;
snapshot.WifiRadioType = wifiInfo.radioType;
@@ -390,73 +390,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
}
/// <summary>
/// Collects WiFi SSID, signal, and radio type via netsh wlan show interfaces.
/// Synchronous with timeout — needed before emitting the event.
/// Collects WiFi SSID, signal, and radio type via the native WLAN API
/// (<see cref="WifiInfoReader"/> — language-neutral, no process spawn).
/// Synchronous — needed before emitting the event.
/// </summary>
private (string ssid, int? signal, string radioType) CollectWiFiInfo()
private (string ssid, int? signal, string radioType) CollectWiFiInfo(Guid? interfaceGuid)
{
string ssid = null;
int? signal = null;
string radioType = null;
try
{
var psi = new ProcessStartInfo
{
FileName = SystemPaths.Netsh,
Arguments = "wlan show interfaces",
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true,
CreateNoWindow = true
};
string output;
using (var process = Process.Start(psi))
{
output = process.StandardOutput.ReadToEnd();
if (!process.WaitForExit(WifiCollectTimeoutMs))
{
try { process.Kill(); } catch { }
_logger.Debug("NetworkChangeDetector: netsh timed out");
return (null, null, null);
}
}
if (string.IsNullOrEmpty(output))
return (null, null, null);
foreach (var line in output.Split(new[] { '\r', '\n' }, StringSplitOptions.RemoveEmptyEntries))
{
var trimmed = line.Trim();
var colonIndex = trimmed.IndexOf(':');
if (colonIndex < 0) continue;
var key = trimmed.Substring(0, colonIndex).Trim();
var value = trimmed.Substring(colonIndex + 1).Trim();
if (key.Equals("SSID", StringComparison.OrdinalIgnoreCase) &&
!key.Equals("BSSID", StringComparison.OrdinalIgnoreCase))
{
ssid = value;
}
else if (key.Equals("Signal", StringComparison.OrdinalIgnoreCase))
{
if (int.TryParse(value.TrimEnd('%'), out var sig))
signal = sig;
}
else if (key.Equals("Radio type", StringComparison.OrdinalIgnoreCase))
{
radioType = value;
}
}
var wifi = WifiInfoReader.TryGetCurrentConnection(interfaceGuid);
if (wifi != null)
return (wifi.Ssid, wifi.SignalPercent, wifi.RadioType);
}
catch (Exception ex)
{
_logger.Debug($"NetworkChangeDetector: WiFi info collection failed: {ex.Message}");
}
return (ssid, signal, radioType);
return (null, null, null);
}
// -------------------------------------------------------------------
@@ -169,9 +169,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// <summary>
/// Sends a critical error report to the emergency channel endpoint.
/// Fire-and-forget: swallows all exceptions so a failure here never cascades.
/// Uses a 5-second per-request timeout so it cannot block the upload loop.
/// Default 5-second per-request timeout so it cannot block the upload loop; the
/// emergency-break path passes a longer <paramref name="timeout"/> because its first
/// request of a freshly booted process pays cold TLS + DNS + a possible backend
/// cold start (observed 503s take well over 5 s).
/// Returns <c>true</c> when the HTTP round-trip completed (the endpoint always
/// answers 200), <c>false</c> on timeout/transport failure — the caller-side
/// anti-flood layer uses this to decide whether the report counts as delivered.
/// Virtual for tests.
/// </summary>
public async Task ReportAgentErrorAsync(AgentErrorReport report)
public virtual async Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
try
{
@@ -190,16 +197,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
httpRequest.Headers.Add("X-Tenant-Id", report.TenantId);
AddSecurityHeaders(httpRequest);
using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5)))
using (var cts = new CancellationTokenSource(timeout ?? TimeSpan.FromSeconds(5)))
using (var response = await _httpClient.SendAsync(httpRequest, cts.Token).ConfigureAwait(false))
{
// Response status is deliberately ignored — endpoint always returns 200.
return true;
}
}
}
catch (Exception ex)
{
_logger?.Debug($"ReportAgentErrorAsync: emergency channel failed: {ex.Message}");
return false;
}
}
@@ -57,19 +57,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// (intended to be called as fire-and-forget via <c>_ = reporter.TrySendAsync(...)</c>).
///
/// All anti-flood checks are applied synchronously before the async HTTP call,
/// so repeated calls are cheap when suppressed.
/// so repeated calls are cheap when suppressed. The anti-flood reservation is
/// taken up front (so concurrent same-key calls cannot double-send) but ROLLED
/// BACK when every attempt fails: the budget and the once-per-session dedup are
/// meant to count DELIVERED reports — the previous mark-before-send made the very
/// first transport failure permanently suppress that error key for the session
/// (found via the 48h emergency-break report, which is a single shot at boot).
///
/// <paramref name="attempts"/>/<paramref name="perAttemptTimeout"/>/<paramref name="retryDelay"/>
/// exist for callers whose process is about to exit and cannot come back later
/// (emergency break): retries stay INSIDE the one reservation, so they never
/// multiply against the per-session budget.
/// </summary>
public virtual async Task TrySendAsync(
AgentErrorType errorType,
string message,
int? httpStatusCode = null,
long? sequenceNumber = null)
long? sequenceNumber = null,
int attempts = 1,
TimeSpan? perAttemptTimeout = null,
TimeSpan? retryDelay = null)
{
// Deduplicate by error type + status code: same failure category sent only once per session.
// All anti-flood checks are protected by a lock to prevent race conditions when
// TrySendAsync is called concurrently (fire-and-forget from multiple threads).
var key = $"{errorType}:{httpStatusCode}";
int currentReportCount;
DateTime? previousLastReportTime;
lock (_antiFloodLock)
{
@@ -92,6 +106,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
return;
}
previousLastReportTime = _lastReportTime;
_reportedKeys.Add(key);
_reportCount++;
_lastReportTime = DateTime.UtcNow;
@@ -113,15 +128,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
Timestamp = DateTime.UtcNow
};
var delivered = false;
var totalAttempts = attempts < 1 ? 1 : attempts;
try
{
await _apiClient.ReportAgentErrorAsync(report);
for (var attempt = 1; attempt <= totalAttempts && !delivered; attempt++)
{
if (attempt > 1 && retryDelay.HasValue)
{
await Task.Delay(retryDelay.Value).ConfigureAwait(false);
}
delivered = await _apiClient.ReportAgentErrorAsync(report, perAttemptTimeout).ConfigureAwait(false);
if (!delivered && totalAttempts > 1)
{
_logger?.Debug($"[EmergencyChannel] Attempt {attempt}/{totalAttempts} failed: {key}");
}
}
}
catch (Exception ex)
{
// Swallow — emergency channel must never cascade failures into the caller
_logger?.Debug($"[EmergencyChannel] Failed to send report: {ex.Message}");
}
if (!delivered)
{
lock (_antiFloodLock)
{
_reportedKeys.Remove(key);
_reportCount--;
_lastReportTime = previousLastReportTime;
}
_logger?.Debug($"[EmergencyChannel] Report not delivered — reservation rolled back: {key}");
}
}
}
}
@@ -36,6 +36,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
/// <item><b>StallProbeHost</b> — <see cref="StallProbeCollector"/> +
/// <see cref="StallProbeCollectorAdapter"/>. Owns its 60-s idle-check timer (the
/// collector itself has no timer — it's a pure probe invoked from outside).</item>
/// <item><b>EspPolicyProviderStallHost</b> — always-on 60-s wall-clock tick for the
/// two-arm <c>esp_policy_provider_stalled</c> tripwire (EnrollmentStatusTracking CSP
/// provider continuously incomplete ≥ 15 min, OR Setup\Apps providers registered
/// without the expected <c>Sidecar</c>/IME provider — the no-timeout ESP wait).</item>
/// </list>
/// Optional peripheral hosts (driven by <see cref="CollectorConfiguration"/> toggles):
/// <list type="bullet">
@@ -303,6 +307,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
harmlessModernDeploymentEventIds: collectors.ModernDeploymentHarmlessEventIds));
}
// ESP policy-provider stall tripwire — always-on kernel host (no config gate, like
// disk_space_low): a provider that never completes, or co-management "ConfigMgr"
// registered without the expected "Sidecar"/IME provider (issue #106 — even with
// TrackingPoliciesCreated=1 the ESP ignores foreign provider names), parks the user
// ESP at "Apps (Identifying)" WITHOUT any OS timeout. Wall-clock dwell, deliberately
// outside the StallProbe gate: the idle clock there resets on any session activity,
// and the stall must accrue regardless of it.
hosts.Add(new EspPolicyProviderStallHost(
sessionId: sessionId,
tenantId: tenantId,
logger: logger,
ingress: ingress,
clock: clock,
startupGate: _startupEventGate));
// Deterministic update corroboration — compares the persisted OS build (CurrentBuild.UBR)
// across agent restarts and emits os_build_changed when it differs (session 7443317c:
// build jump proved an OOBE quality update the WU channel never showed). Must be added
@@ -36,6 +36,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
/// <item><c>CollectAtEnd</c> — once, on <c>FinalizingSetup</c> or desktop arrival
/// (whichever comes first; desktop arrival also covers no-ESP / WDP v2). Re-fetches
/// BitLocker (commonly enabled via policy DURING enrollment) + the active NIC.</item>
/// <item><c>RefreshEspConfiguration</c> — audit Q2, one-shot per trigger: the ESP
/// blocking lists in the registry grow progressively (device scope) and the user-scope
/// lists appear only after sign-in, so the early emissions are structurally partial.
/// Re-collected when the apps sub-phase opens (first app activity in DeviceSetup /
/// AccountSetup — the adapter's <c>phase_transition</c> declaration) and when the
/// AccountSetup ESP phase itself is detected. Phase-driven one-shots, never a timer
/// (no-heartbeat policy).</item>
/// </list>
/// <para>
/// Duplicate-event cost is zero: every re-collected event runs through the collector's
@@ -46,6 +53,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
public string Name => "DeviceInfoCollector";
// ESP-config refresh trigger labels (audit Q2) — also the values passed to
// DeviceInfoCollector.RefreshEspConfiguration for the agent-log line.
internal const string EspConfigTriggerAppsDevice = "apps_phase_device";
internal const string EspConfigTriggerAppsUser = "apps_phase_user";
internal const string EspConfigTriggerAccountSetup = "account_setup_detected";
private readonly Monitoring.Telemetry.DeviceInfo.DeviceInfoCollector _collector;
private readonly AgentLogger _logger;
@@ -58,6 +71,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
private int _endCollected;
private int _disposed;
// Audit Q2 — fire-once latches per ESP-config refresh trigger. Guarded by _espConfigLock
// instead of three Interlocked ints so the membership test + add stays one atomic step.
private readonly object _espConfigLock = new object();
private readonly HashSet<string> _espConfigRefreshesFired = new HashSet<string>(StringComparer.Ordinal);
public DeviceInfoHost(
string sessionId,
string tenantId,
@@ -100,6 +118,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
private void OnSignalPosted(DecisionSignalKind kind, IReadOnlyDictionary<string, string>? payload)
{
HandleEspConfigRefreshTrigger(kind, payload);
var startTrigger = IsEnrollmentStartTrigger(kind, payload);
var endTrigger = IsEndTrigger(kind, payload);
if (!startTrigger && !endTrigger) return;
@@ -111,10 +131,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
var runEndCollect = endTrigger && Interlocked.Exchange(ref _endCollected, 1) == 0;
if (!runStartRefresh && !runEndCollect) return;
// Both one-shots done → nothing left to observe.
if (Volatile.Read(ref _enrollmentStartCollected) == 1 && Volatile.Read(ref _endCollected) == 1)
Unsubscribe();
var trigger = kind == DecisionSignalKind.DesktopArrived ? "desktop_arrived" : "esp_phase_changed";
_logger.Info($"DeviceInfoHost: trigger '{trigger}' — scheduling re-collect (enrollmentStart={runStartRefresh}, end={runEndCollect}).");
@@ -134,6 +150,32 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
});
}
/// <summary>
/// Audit Q2 — ESP-config refresh one-shots. Each trigger label fires at most once per
/// agent run; the collector's StartupEventGate keeps an unchanged payload silent, so
/// the worst case is one silent registry re-read per trigger. The subscription stays
/// armed for the host's lifetime (unlike the start/end pair there is no "all triggers
/// seen" point that is guaranteed to arrive — SkipUser enrollments never enter
/// AccountSetup); the per-signal cost is a few dictionary lookups.
/// </summary>
private void HandleEspConfigRefreshTrigger(DecisionSignalKind kind, IReadOnlyDictionary<string, string>? payload)
{
var trigger = ClassifyEspConfigRefreshTrigger(kind, payload);
if (trigger == null) return;
lock (_espConfigLock)
{
if (!_espConfigRefreshesFired.Add(trigger)) return;
}
_logger.Info($"DeviceInfoHost: ESP-config refresh trigger '{trigger}' — scheduling re-collect.");
Task.Run(() =>
{
try { _collector.RefreshEspConfiguration(trigger); }
catch (Exception ex) { _logger.Warning($"DeviceInfoHost: ESP-config refresh threw: {ex.Message}"); }
});
}
/// <summary>First DeviceSetup phase signal — MDM enrollment has populated the registry surface.</summary>
internal static bool IsEnrollmentStartTrigger(DecisionSignalKind kind, IReadOnlyDictionary<string, string>? payload)
{
@@ -156,6 +198,45 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
&& string.Equals(phase, nameof(EnrollmentPhase.FinalizingSetup), StringComparison.Ordinal);
}
/// <summary>
/// Audit Q2 — maps a posted signal to an ESP-config refresh trigger label, or
/// <c>null</c> when the signal is not one. Two trigger families:
/// <list type="bullet">
/// <item>The adapter's apps sub-phase declaration (<c>phase_transition</c>
/// InformationalEvent with phase AppsDevice/AppsUser — fired on the first app
/// activity in an ESP phase): by then IME is actively processing the blocking set,
/// so the device-scope ESPTrackingInfo lists are as complete as they get.</item>
/// <item>The AccountSetup ESP phase (UserSetup detection): sign-in has happened, so
/// the user-scope <c>S-&lt;SID&gt;</c> lists exist now — production showed 88 of 89
/// sessions with an EMPTY user list at the last pre-fix emission.</item>
/// </list>
/// </summary>
internal static string? ClassifyEspConfigRefreshTrigger(DecisionSignalKind kind, IReadOnlyDictionary<string, string>? payload)
{
if (payload == null) return null;
if (kind == DecisionSignalKind.InformationalEvent
&& payload.TryGetValue(SignalPayloadKeys.EventType, out var eventType)
&& string.Equals(eventType, AutopilotMonitor.Shared.Constants.EventTypes.PhaseTransition, StringComparison.Ordinal)
&& payload.TryGetValue(Telemetry.Events.EventTimelineEmitter.PhaseParamKey, out var declaredPhase))
{
if (string.Equals(declaredPhase, nameof(EnrollmentPhase.AppsDevice), StringComparison.Ordinal))
return EspConfigTriggerAppsDevice;
if (string.Equals(declaredPhase, nameof(EnrollmentPhase.AppsUser), StringComparison.Ordinal))
return EspConfigTriggerAppsUser;
return null;
}
if (kind == DecisionSignalKind.EspPhaseChanged
&& payload.TryGetValue(SignalPayloadKeys.EspPhase, out var espPhase)
&& string.Equals(espPhase, nameof(EnrollmentPhase.AccountSetup), StringComparison.Ordinal))
{
return EspConfigTriggerAccountSetup;
}
return null;
}
public void Stop() => Unsubscribe();
public void Dispose()
@@ -0,0 +1,94 @@
#nullable enable
using System;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
/// <summary>
/// Drives the <see cref="EspPolicyProviderStallDetector"/> from a 60-s wall-clock tick.
/// Deliberately NOT idle-coupled (a policy-provider stall coexists with a chatty session —
/// <see cref="StallProbeHost"/>'s idle clock would keep resetting) and NOT idle-stopped
/// (<see cref="PeriodicCollectorLifecycleHost"/> would sleep through exactly the dormant
/// window the dwell must observe). Always-on kernel host, no config gate — precedent:
/// the <c>disk_space_low</c> tripwire.
/// <para>
/// First tick fires after 60 s, not at Start: the EnrollmentStatusTracking key legitimately
/// doesn't exist at agent start, and nothing can be 15 min stale at t=0.
/// </para>
/// </summary>
internal sealed class EspPolicyProviderStallHost : ICollectorHost
{
public string Name => "EspPolicyProviderStallDetector";
private static readonly TimeSpan PollInterval = TimeSpan.FromSeconds(60);
private readonly EspPolicyProviderStallDetector _detector;
private readonly AgentLogger _logger;
private Timer? _tickTimer;
private int _disposed;
public EspPolicyProviderStallHost(
string sessionId,
string tenantId,
AgentLogger logger,
ISignalIngressSink ingress,
IClock clock,
StartupEventGate? startupGate)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
_logger = logger;
var post = new InformationalEventPost(ingress, clock);
_detector = new EspPolicyProviderStallDetector(
sessionId: sessionId,
tenantId: tenantId,
post: post,
logger: logger,
clock: clock,
startupGate: startupGate);
}
public void Start()
{
_tickTimer = new Timer(
_ => SafeTick(),
state: null,
dueTime: PollInterval,
period: PollInterval);
_logger.Info($"EspPolicyProviderStallHost: started (tick every {PollInterval.TotalSeconds}s).");
}
public void Stop()
{
try
{
_tickTimer?.Dispose();
_tickTimer = null;
}
catch (Exception ex) { _logger.Warning($"EspPolicyProviderStallHost: timer dispose failed: {ex.Message}"); }
}
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
Stop();
}
private void SafeTick()
{
try
{
_detector.Tick();
}
catch (Exception ex)
{
_logger.Error("EspPolicyProviderStallHost: tick failed.", ex);
}
}
}
}
@@ -111,6 +111,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
logger: logger,
matchLogPath: expandedMatchLogPath,
stateDirectory: stateDirectory);
// Historic-replay guard runs against the agent clock, not the raw system clock —
// keeps the tracker's staleness verdicts consistent with the adapter's clamp.
_tracker.UtcNowProvider = () => clock.UtcNow;
if (simulationMode)
{
@@ -93,7 +93,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
{
using (obj)
{
manufacturer = obj["Manufacturer"]?.ToString() ?? Unknown;
manufacturer = NormalizeWmiValue(obj["Manufacturer"]?.ToString());
if (manufacturer.IndexOf("lenovo", StringComparison.OrdinalIgnoreCase) >= 0)
{
using (var lenovoSearcher = new ManagementObjectSearcher("SELECT Version FROM Win32_ComputerSystemProduct"))
@@ -103,7 +103,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
{
using (lenovoObj)
{
model = lenovoObj["Version"]?.ToString() ?? Unknown;
model = NormalizeWmiValue(lenovoObj["Version"]?.ToString());
}
break;
}
@@ -111,7 +111,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
}
else
{
model = obj["Model"]?.ToString() ?? Unknown;
model = NormalizeWmiValue(obj["Model"]?.ToString());
}
}
}
@@ -132,7 +132,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
{
using (obj)
{
serialNumber = obj["SerialNumber"]?.ToString() ?? Unknown;
serialNumber = NormalizeWmiValue(obj["SerialNumber"]?.ToString());
}
}
}
@@ -154,5 +154,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
private static bool IsResolved(string value)
=> !string.IsNullOrWhiteSpace(value)
&& !string.Equals(value, Unknown, StringComparison.OrdinalIgnoreCase);
/// <summary>
/// Trims a raw WMI string value at the source (audit Q5: BIOS serials are used as device
/// identity keys downstream — F2 device history joins on them — so padding must never
/// leave the agent). Whitespace-only and empty values collapse to the "Unknown" sentinel,
/// matching the null fallback so the retry logic treats them as unresolved.
/// </summary>
internal static string NormalizeWmiValue(string raw)
{
if (string.IsNullOrWhiteSpace(raw)) return Unknown;
return raw.Trim();
}
}
}
@@ -35,9 +35,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <summary>C:\Windows\System32\tzutil.exe</summary>
public static readonly string TzUtil = Path.Combine(System32, "tzutil.exe");
/// <summary>C:\Windows\System32\netsh.exe</summary>
public static readonly string Netsh = Path.Combine(System32, "netsh.exe");
/// <summary>C:\Windows\System32\schtasks.exe</summary>
public static readonly string Schtasks = Path.Combine(System32, "schtasks.exe");
@@ -1,6 +1,7 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Globalization;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
@@ -267,6 +268,14 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
payload["failedSubcategory"] = failedSubcategory!;
if (!string.IsNullOrEmpty(category))
payload["category"] = category!;
// Session 4910a5a5: culprit-app names ride the signal so the DecisionEngine's
// esp_failure_advisory / enrollment_failed events can name them directly.
var culprits = args?.LikelyCulpritApps;
if (culprits != null && culprits.Count > 0)
{
payload["likelyCulpritApps"] = string.Join(", ", culprits);
payload["likelyCulpritAppCount"] = culprits.Count.ToString(CultureInfo.InvariantCulture);
}
_ingress.Post(
kind: DecisionSignalKind.EspTerminalFailure,
@@ -97,12 +97,24 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
// bootstrap marker line. Lets MCP report "which device runs which bootstrap version".
private bool _bootstrapDetectedPosted;
// One-shot: historic script replay detected (a previous enrollment's IME log content
// surviving on disk) — its script events are suppressed, and a single
// historic_ime_replay_detected Info event explains the gap in the timeline.
private bool _historicReplayNoted;
private int _suppressedHistoricReplayEvents; // forensic count for the log line
// A platform script whose observed run duration reaches this is treated as having hit the
// IME script-execution timeout (default ~30 min). 25 min leaves headroom below 30 so clock
// jitter / the agent's late replay window can't miss it, while staying well above any
// legitimately long install wrapper that completes on its own.
private static readonly TimeSpan ScriptTimeoutSuspectedThreshold = TimeSpan.FromMinutes(25);
// Source log lines older than this relative to the clock are pathological — shared by
// ResolveOccurredAt's staleness clamp, the historic-replay suppression, the duration
// plausibility backstop AND the tracker's app-mutation guard (single source of truth
// in ImeLogTracker so the layers can never drift apart).
private static readonly TimeSpan StaleSourceThreshold = ImeLogTracker.HistoricReplayThreshold;
// Matches the deterministic marker line the bootstrap script writes via Write-Log,
// e.g. "Bootstrap script version: v2.0" — see scripts/Bootstrap/Install-AutopilotMonitor.ps1.
// Same shape as the web-side regex in utils/bootstrapVersion.ts; kept in sync intentionally.
@@ -120,7 +132,9 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
// Plan §5 Fix 4c — per-app install-lifecycle timing captured from state transitions.
// StartedAtUtc on first Downloading/Installing/InProgress transition (set-once);
// CompletedAtUtc on first terminal transition (Installed/Skipped/Postponed/Error).
// CompletedAtUtc on terminal transition (Installed/Skipped/Postponed/Error), re-armed
// to null on a terminal→active transition (IME retry, audit Q1) so the final terminal
// stamp reflects the last outcome instead of freezing at the first one.
// Surfaced via <see cref="AppTimings"/> to peripheral consumers (FinalStatusBuilder,
// app_tracking_summary emission in EnrollmentTerminationHandler).
private readonly Dictionary<string, (DateTime? Started, DateTime? Completed)> _appTimings =
@@ -289,17 +303,11 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
rawSourceTimestamp = sourceTsOverride ?? _tracker.LastMatchedLogTimestamp;
if (rawSourceTimestamp.HasValue)
{
// CMTrace log timestamps from IME are emitted in UTC, but DateTimeKind may
// arrive as Unspecified depending on the parser path. Normalize via
// SpecifyKind=Utc rather than ToUniversalTime() to avoid double-conversion
// when Kind is already UTC.
var asUtc = rawSourceTimestamp.Value.Kind == DateTimeKind.Utc
? rawSourceTimestamp.Value
: DateTime.SpecifyKind(rawSourceTimestamp.Value, DateTimeKind.Utc);
var asUtc = NormalizeUtc(rawSourceTimestamp.Value);
var clockNow = _clock.UtcNow;
var ageHours = (clockNow - asUtc).TotalHours;
if (ageHours > 24.0 || ageHours < -1.0)
if (ageHours > StaleSourceThreshold.TotalHours || ageHours < -1.0)
{
// Pathologically stale (>24h old) or in the future (>1h skew) — log,
// fall back, and let the caller flag the event.
@@ -325,6 +333,67 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
data["rejectedSourceTimestamp"] = rawSourceTs.Value.ToString("o", System.Globalization.CultureInfo.InvariantCulture);
}
// CMTrace Kind normalization — shared implementation lives on the tracker (used by its
// historic-replay guard too).
private static DateTime NormalizeUtc(DateTime value) => ImeLogTracker.NormalizeUtc(value);
/// <summary>
/// True when an event's source log line is a historic replay: the source timestamp
/// was rejected as stale (> <see cref="StaleSourceThreshold"/> in the PAST). Those lines
/// are script runs and app installs from a previous enrollment whose IME log survived on
/// disk (session eaf3d8c4: 156 replayed script completions + 147 replayed app events
/// shown as current activity, immediate-upload flood → ingress_backpressure). Future-skew
/// rejections (negative age — mid-enrollment clock jump) and clock-only fallbacks (no
/// source timestamp at all) are NOT replay and must still emit. Emits the one-shot
/// <c>historic_ime_replay_detected</c> summary on the first suppression; replay is
/// chronological, so that first suppressed line carries the earliest rejected source
/// timestamp. Bypassed in <see cref="ImeLogTracker.SimulationMode"/> — the
/// --replay-log-dir dev tool feeds historic lines on purpose.
/// </summary>
private bool SuppressIfHistoricReplay(
bool derivedFromClock, DateTime? rawSourceTs, string eventLabel, string entityId)
{
if (_tracker.SimulationMode) return false;
if (!derivedFromClock || !rawSourceTs.HasValue) return false;
var asUtc = NormalizeUtc(rawSourceTs.Value);
var clockNow = _clock.UtcNow;
var age = clockNow - asUtc;
if (age <= StaleSourceThreshold) return false; // future-skew or fresh → emit
bool emitSummary;
int suppressedSoFar;
lock (_lock)
{
suppressedSoFar = ++_suppressedHistoricReplayEvents;
emitSummary = !_historicReplayNoted;
_historicReplayNoted = true;
}
if (emitSummary)
{
var culture = System.Globalization.CultureInfo.InvariantCulture;
// Standard batch upload on purpose — no immediateUpload; the whole point of the
// suppression is to stop the replayed batch from flooding the ingest.
_post.Emit(
eventType: SharedEventTypes.HistoricImeReplayDetected,
source: SourceLabel,
message: "Historic IME activity from a previous enrollment detected in the log "
+ $"(earliest source timestamp {asUtc:o}) — its script and app events are suppressed for this session.",
severity: EventSeverity.Info,
data: new Dictionary<string, string>(StringComparer.Ordinal)
{
["decision"] = "historic_ime_replay_suppressed",
["earliestRejectedSourceTimestamp"] = asUtc.ToString("o", culture),
["staleThresholdHours"] = StaleSourceThreshold.TotalHours.ToString("F0", culture),
});
}
_logger?.Debug(
$"ImeAdapter: suppressed historic {eventLabel} for {entityId} "
+ $"(source ts {asUtc:o}, age {age.TotalHours:F1}h, suppressed so far: {suppressedSoFar})");
return true;
}
private void EmitEspPhase(string phase)
{
if (string.IsNullOrEmpty(phase)) return;
@@ -504,6 +573,14 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
var now = ResolveOccurredAt(out var derivedFromClock, out var rawSourceTs);
// Historic replay (a previous enrollment's log) — do not emit at all. Placed BEFORE
// the sub-phase declaration, the timing bookkeeping and the terminal-signal dedup so
// a replayed app neither consumes the fire-once flags nor poisons _appTimings (which
// feeds FinalStatusBuilder) nor posts a phantom DecisionSignal. The early return also
// covers the shadow download_progress and the app_tracking_summary snapshot below.
if (SuppressIfHistoricReplay(derivedFromClock, rawSourceTs, $"app-state {oldState}->{newState}", app.Id))
return;
// Plan §5 Fix 2: first app activity in a given ESP phase opens the sub-phase on
// the UI timeline via a `phase_transition` declaration event. Fire-once per ESP
// phase; emission happens BEFORE the actual app_* event so the UI sees the phase
@@ -616,10 +693,20 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
}
/// <summary>
/// Plan §5 Fix 4c — set-once per-app install-lifecycle timing. Called for every
/// state transition, no-op for duplicate stamps. Returns the up-to-date
/// Plan §5 Fix 4c — per-app install-lifecycle timing. Called for every state
/// transition, no-op for duplicate stamps. Returns the up-to-date
/// <see cref="AppInstallTiming"/> so the emitter can inline the values on the
/// current event's payload without a second dictionary lookup.
/// <para>
/// Audit Q1: a terminal→active transition (the IME retry path — <see
/// cref="AppPackageState.UpdateState"/> permits e.g. Error→Downloading and
/// Skipped→Downloading) re-arms <c>Completed</c> to <c>null</c>, so the NEXT terminal
/// transition stamps a fresh CompletedAt. StartedAt keeps its first value — the final
/// terminal payload then covers first-start → last-terminal (total occupancy across
/// attempts) instead of freezing at the first failure. Observed pre-fix: a user-scope
/// app Skipped early in DeviceSetup, retried after sign-in, emitted its final
/// Installed event with the stale early CompletedAt (durationSeconds=-2342).
/// </para>
/// </summary>
private AppInstallTiming UpdateAppTiming(string appId, AppInstallationState newState, DateTime now)
{
@@ -627,13 +714,20 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
{
_appTimings.TryGetValue(appId, out var current);
// Terminal→active (retry): the earlier terminal stamp no longer describes this
// app's outcome — re-arm so the next terminal transition stamps fresh.
if (current.Completed != null && IsLifecycleActive(newState))
{
current.Completed = null;
}
// First lifecycle-active transition → record StartedAt.
if (current.Started == null && IsLifecycleActive(newState))
{
current.Started = now;
}
// First terminal transition → record CompletedAt.
// Terminal transition → record CompletedAt (fresh after a re-arm).
if (current.Completed == null && IsCompletedState(newState))
{
current.Completed = now;
@@ -819,6 +913,11 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
if (!string.IsNullOrEmpty(patternId)) data["patternId"] = patternId!;
var now = ResolveOccurredAt(out var derivedFromClock, out var rawSourceTs);
// Historic replay — a week-old DO summary is not this enrollment's download.
if (SuppressIfHistoricReplay(derivedFromClock, rawSourceTs, SharedEventTypes.DoTelemetry, app.Id))
return;
TagDerivedTimestamp(data, derivedFromClock, rawSourceTs);
var label = string.IsNullOrEmpty(app.Name) ? app.Id : app.Name;
@@ -910,13 +1009,27 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
now = ResolveOccurredAt(script.ExitObservedAtUtc, out derivedFromClock, out rawSourceTs);
else
now = ResolveOccurredAt(out derivedFromClock, out rawSourceTs);
// Historic replay (a previous enrollment's log) — do not emit at all. The early
// return also skips MaybeEmitBootstrapDetected / MaybeEmitScriptTimeoutSuspected,
// which would otherwise fire on week-old evidence.
if (SuppressIfHistoricReplay(derivedFromClock, rawSourceTs, eventType, script.PolicyId))
return;
TagDerivedTimestamp(data, derivedFromClock, rawSourceTs);
// Per-script run duration (start line → this completion, `now`). Surfaced on every
// Per-script run duration (start line → this completion). Surfaced on every
// script completion so the UI can flag slow / inefficient scripts, not just timed-out
// ones. Omitted when the start line was never observed (StartedAtUtc null); clamped at 0
// against clock skew between the (possibly clock-derived) completion stamp and start.
//
// When the completion timestamp was clock-derived because the source line's timestamp
// was rejected (future-skew CMTrace clock jump — the stale case never reaches here),
// `now` lives on the clock's timeline while StartedAtUtc was captured RAW from the
// source log. Mixing the two produced absurd durations (session eaf3d8c4: 170 h), so
// the duration uses the raw source pair — both ends on the same timeline, the
// difference is the true runtime. The event stamp stays `now` (timeline position).
//
// L19 (delta review 2026-07-02): the same field name carries two different semantics —
// start→own-end-signal (≈ script runtime) vs HS-SCRIPT-START→HS-NEW-RESULT (the whole
// cycle INCLUDING IME's 30 s–3 min batched reporting latency, i.e. systematically
@@ -924,10 +1037,16 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
// consumers never compare the two as equals. The emitting handler stamps the basis on
// the state (health-script early-signal completions are script_runtime too); the
// scriptType fallback covers states from paths that predate the field.
var completionForDuration = derivedFromClock && rawSourceTs.HasValue
? NormalizeUtc(rawSourceTs.Value)
: now;
if (script.StartedAtUtc.HasValue)
{
var durationSeconds = (now - script.StartedAtUtc.Value).TotalSeconds;
if (durationSeconds >= 0)
var durationSeconds = (completionForDuration - script.StartedAtUtc.Value).TotalSeconds;
// Plausibility backstop: IME's script execution timeout is ~30 min; anything above
// 24 h is a cross-run artifact (e.g. a stale pending start slot from a previous
// enrollment paired with a fresh completion) — omit rather than lie.
if (durationSeconds >= 0 && durationSeconds <= StaleSourceThreshold.TotalSeconds)
{
data["durationSeconds"] = durationSeconds.ToString("F2", culture);
data["durationBasis"] = script.DurationBasis
@@ -979,7 +1098,7 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
$"ImeAdapter: script completed policyId={shortId} type={script.ScriptType ?? "?"} result={script.Result ?? "?"} exit={(script.ExitCode.HasValue ? script.ExitCode.Value.ToString() : "n/a")}");
MaybeEmitBootstrapDetected(script, now);
MaybeEmitScriptTimeoutSuspected(script, now);
MaybeEmitScriptTimeoutSuspected(script, now, completionForDuration);
}
/// <summary>
@@ -991,15 +1110,19 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
/// late start / pre-failed ESP. Advisory only: no state mutation, the underlying
/// <c>script_failed</c> is still emitted separately. Health (remediation) scripts are
/// excluded — they re-run on their own cadence and have a different timeout regime.
/// <paramref name="durationEndUtc"/> is the timeline-consistent completion for duration
/// math (raw source ts on rejected-timestamp emissions); <paramref name="occurredAtUtc"/>
/// stamps the emitted event.
/// </summary>
private void MaybeEmitScriptTimeoutSuspected(ScriptExecutionState script, DateTime completionUtc)
private void MaybeEmitScriptTimeoutSuspected(ScriptExecutionState script, DateTime occurredAtUtc, DateTime durationEndUtc)
{
if (IsRemediation(script.ScriptType)) return; // platform scripts only
if (!script.StartedAtUtc.HasValue) return; // no start ⇒ no duration
if (!string.Equals(script.Result, "Failed", StringComparison.OrdinalIgnoreCase)) return;
var duration = completionUtc - script.StartedAtUtc.Value;
var duration = durationEndUtc - script.StartedAtUtc.Value;
if (duration < ScriptTimeoutSuspectedThreshold) return;
if (duration > StaleSourceThreshold) return; // implausible — cross-run artifact, not a hung script
if (!_tracker.TryClaimScriptTimeoutSuspected(script.PolicyId)) return; // restart-safe dedup per policyId (persisted tracker state)
@@ -1026,7 +1149,7 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
message: $"Platform script {shortId} ran {durationMinutes:F0} min and was marked Failed — suspected IME script timeout blocking the enrollment.",
severity: EventSeverity.Warning,
data: data,
occurredAtUtc: completionUtc);
occurredAtUtc: occurredAtUtc);
_logger?.Info(
$"ImeAdapter: script_timeout_suspected policyId={shortId} duration={durationMinutes:F1}min result={script.Result} espPhase={espPhase ?? "(none)"}");
@@ -1081,6 +1204,12 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
if (!string.IsNullOrEmpty(patternId)) data["patternId"] = patternId!;
var now = ResolveOccurredAt(out var derivedFromClock, out var rawSourceTs);
// Historic replay — suppressing here also kills the immediateUpload flood the
// replayed batch caused (session eaf3d8c4: 3× ingress_backpressure).
if (SuppressIfHistoricReplay(derivedFromClock, rawSourceTs, SharedEventTypes.ScriptStarted, info.PolicyId))
return;
TagDerivedTimestamp(data, derivedFromClock, rawSourceTs);
var label = IsRemediation(info.ScriptType) ? "Health script" : "Platform script";
@@ -6,6 +6,7 @@ using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Runtime;
@@ -93,6 +94,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
// correction between agent start and termination no longer skews bootToStartSeconds.
private readonly ulong _bootAnchorTickMs;
private readonly DateTime _bootAnchorUtc;
// WinRT OOBE state at construction (handler is built at agent start) — corroborates
// the late-start heuristic: "completed" here means OOBE was already over on arrival.
private readonly string _oobeStateAtAgentStart;
private int _handled;
public EnrollmentTerminationHandler(
@@ -129,6 +133,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
_bootAnchorTickMs = ObservationCoverage.CurrentTickMs();
_bootAnchorUtc = DateTime.UtcNow;
_oobeStateAtAgentStart = OobeStateReader.Read();
}
/// <summary>
@@ -618,12 +623,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
}
/// <summary>
/// Emits a one-shot <c>agent_late_start</c> Warning when the agent had low observation
/// Emits a one-shot <c>agent_late_start</c> event when the agent had low observation
/// coverage: it started a long time after device boot AND lived only briefly before this
/// terminal outcome (gate in <see cref="ObservationCoverage"/>). Such a session looks like a
/// normal multi-minute failure in the list (StartedAt is back-dated to the earliest replayed
/// IME-log event) but the agent actually observed only the end-state — so the operator should
/// read the diagnosis as a post-mortem reconstruction, not live evidence. No state mutation.
/// IME-log event) but the agent actually observed only the end-state. Severity and phrasing
/// are outcome-calibrated via <see cref="ObservationCoverage.DescribeLateStart"/>: a failure
/// stays a Warning with the post-mortem/hung-script note, a success is an Info about
/// tail-only coverage (fleet 2026-07-28: tenant 659c3a90 collected 12 Warnings on healthy
/// enrollments whose bootstrap merely ran late). No state mutation.
/// </summary>
private void MaybeEmitAgentLateStart(EnrollmentTerminatedEventArgs args)
{
@@ -637,29 +645,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
out var bootToStartSeconds, out var uptimeSeconds))
return;
ObservationCoverage.DescribeLateStart(
args.Outcome, _oobeStateAtAgentStart, bootToStartSeconds, uptimeSeconds,
out var severity, out var message, out var note);
_logger.Info(
$"EnrollmentTerminationHandler: low observation coverage — agent started " +
$"{bootToStartSeconds / 60.0:F1} min after boot and lived {uptimeSeconds:F0}s; " +
$"emitting agent_late_start.");
$"emitting agent_late_start ({severity}, outcome {args.Outcome}).");
_post.Emit(new EnrollmentEvent
{
SessionId = _configuration.SessionId,
TenantId = _configuration.TenantId,
EventType = Constants.EventTypes.AgentLateStart,
Severity = EventSeverity.Warning,
Severity = severity,
Source = "EnrollmentTerminationHandler",
Phase = EnrollmentPhase.Unknown,
Message = $"Agent started {bootToStartSeconds / 60.0:F0} min after boot and observed only " +
$"{uptimeSeconds:F0}s before terminating ({args.Outcome}) — low coverage of the enrollment window.",
Message = message,
Data = new Dictionary<string, object>
{
{ "bootToAgentStartSeconds", Math.Round(bootToStartSeconds, 1) },
{ "agentUptimeSeconds", Math.Round(uptimeSeconds, 1) },
{ "deviceBootUtc", deviceBootUtc.ToString("o") },
{ "agentStartUtc", _agentStartTimeUtc.ToString("o") },
{ "oobeStateAtAgentStart", _oobeStateAtAgentStart },
{ "outcome", args.Outcome.ToString() },
{ "note", "The agent's bootstrap (an Intune platform script) ran only near the end of the enrollment, so the agent observed the already-decided end-state rather than the failure window. Treat the diagnosis as a post-mortem; check for a platform/remediation script that hung ahead of the bootstrap (see script_timeout_suspected)." },
{ "note", note },
},
ImmediateUpload = true,
});
@@ -1,6 +1,8 @@
#nullable enable
using System;
using System.Runtime.InteropServices;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Agent.V2.Core.Termination
{
@@ -76,5 +78,57 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
uptimeSeconds = Math.Max(0, uptime.TotalSeconds);
return bootToStart >= LateStartBootThreshold && uptime <= LowCoverageUptime;
}
/// <summary>
/// Severity + operator-facing phrasing for the <c>agent_late_start</c> event, calibrated
/// by fleet data (2026-07-28): the gate alone is outcome-blind and attached a
/// "post-mortem / check for a hung script" Warning to sessions that in fact succeeded —
/// there the late bootstrap is deployment hygiene, not an alarm. The OOBE state refines
/// the phrasing but must never gate: the canonical hung-script failure terminates on the
/// still-shown ESP screen, i.e. with OOBE reading <c>in_progress</c>.
/// </summary>
/// <param name="outcome">Terminal outcome the enrollment reached.</param>
/// <param name="oobeStateAtAgentStart"><see cref="Monitoring.Interop.OobeStateReader"/>
/// sample taken at THIS process's start. Only <c>completed</c> / <c>in_progress</c> are
/// interpreted; <c>not_started</c> flips in after mid-OOBE reboots and <c>completed</c> is
/// routine for post-desktop restarts, so no claim is phrased from the other values.</param>
/// <param name="bootToStartSeconds">From <see cref="IsLowObservationCoverage"/>.</param>
/// <param name="uptimeSeconds">From <see cref="IsLowObservationCoverage"/>.</param>
internal static void DescribeLateStart(
EnrollmentTerminationOutcome outcome,
string oobeStateAtAgentStart,
double bootToStartSeconds,
double uptimeSeconds,
out EventSeverity severity,
out string message,
out string note)
{
var succeeded = outcome == EnrollmentTerminationOutcome.Succeeded;
severity = succeeded ? EventSeverity.Info : EventSeverity.Warning;
message = $"Agent started {bootToStartSeconds / 60.0:F0} min after boot and observed only " +
$"{uptimeSeconds:F0}s before terminating ({outcome}) — low coverage of the enrollment window.";
switch (oobeStateAtAgentStart)
{
case "completed":
message += " OOBE was already completed when the agent started — the diagnosis is a " +
"post-mortem reconstruction, not live observation.";
break;
case "in_progress":
message += $" OOBE was still in progress at agent start — the agent did observe the " +
$"final {uptimeSeconds:F0}s live.";
break;
}
note = succeeded
? "The agent's bootstrap (an Intune platform script) ran late in the enrollment, so only the " +
"final stretch was observed live; earlier timeline entries are reconstructed from IME log " +
"replay. The enrollment itself succeeded. If this recurs across sessions, review which " +
"platform/remediation scripts run ahead of the bootstrap (see script_timeout_suspected)."
: "The agent's bootstrap (an Intune platform script) ran only near the end of the enrollment, " +
"so the agent observed the already-decided end-state rather than the failure window. Treat " +
"the diagnosis as a post-mortem; check for a platform/remediation script that hung ahead of " +
"the bootstrap (see script_timeout_suspected).";
}
}
}
@@ -1,4 +1,6 @@
using System;
using System.Net.NetworkInformation;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Models;
@@ -15,28 +17,78 @@ namespace AutopilotMonitor.Agent.V2.Runtime
/// classifier terminalize the session instead of waiting out the grace. Fully swallowed — a send
/// failure (no network, no cert) must never delay the cleanup/exit the break exists to perform.
/// See tasks/enrollment-status-reclassification.md.
/// <para>
/// Delivery hardening (2026-07-23): this is the FIRST request of a freshly booted process and
/// the process never comes back — a miss is gone forever. The old single shot with a 5 s cap
/// died on boot-time realities (NIC still associating, cold TLS/DNS, backend cold starts
/// observed as &gt;5 s 503s). Now: wait briefly for the NIC, then up to
/// <see cref="SendAttempts"/> attempts with a per-attempt timeout that survives a Functions
/// cold start. The session is 48 h dead — spending up to ~75 s here is irrelevant, losing the
/// only signal that the agent gave up is not.
/// </para>
/// </summary>
internal static class EmergencyBreakReporter
{
private static readonly TimeSpan NetworkWaitMax = TimeSpan.FromSeconds(15);
private const int SendAttempts = 3;
private static readonly TimeSpan PerAttemptTimeout = TimeSpan.FromSeconds(15);
private static readonly TimeSpan RetryDelay = TimeSpan.FromSeconds(5);
// Upper bound for the blocking wait: NIC wait + 3×15s attempts + 2×5s delays + slack.
private static readonly TimeSpan OverallBudget = TimeSpan.FromSeconds(75);
public static void TrySend(AgentConfiguration agentConfig, string agentVersion, AgentLogger logger)
{
try
{
WaitForNetwork(logger);
var auth = BackendClientFactory.BuildAuthClients(agentConfig, agentVersion, logger);
var message =
$"Agent absolute session-age emergency break fired (cap {agentConfig.AbsoluteMaxSessionHours}h) — cleaning up and exiting.";
// The process is about to exit, so block briefly on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP errors;
// the auth bundle's HttpClients are intentionally left for process teardown.
// The process is about to exit, so block on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP
// errors and keeps the retries inside one anti-flood reservation; the auth
// bundle's HttpClients are intentionally left for process teardown.
auth.EmergencyReporter
.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, message)
.Wait(TimeSpan.FromSeconds(5));
.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak,
message,
attempts: SendAttempts,
perAttemptTimeout: PerAttemptTimeout,
retryDelay: RetryDelay)
.Wait(OverallBudget);
}
catch (Exception ex)
{
logger?.Debug($"Emergency-break report send failed (best-effort): {ex.Message}");
}
}
/// <summary>
/// Boot-time NIC grace: the break fires during bootstrap, often seconds after boot while
/// Wi-Fi is still associating. Polls the cheap link-level signal — it cannot prove backend
/// reachability (the send attempts handle that), it only avoids burning the first attempts
/// into a known-dead link. Best-effort: probe errors abort the wait, never the send.
/// </summary>
private static void WaitForNetwork(AgentLogger logger)
{
try
{
if (NetworkInterface.GetIsNetworkAvailable()) return;
logger?.Info($"Emergency-break report: no network link yet — waiting up to {NetworkWaitMax.TotalSeconds:F0}s.");
var deadline = DateTime.UtcNow + NetworkWaitMax;
while (DateTime.UtcNow < deadline)
{
Thread.Sleep(TimeSpan.FromSeconds(1));
if (NetworkInterface.GetIsNetworkAvailable()) return;
}
}
catch (Exception ex)
{
logger?.Debug($"Emergency-break report: network probe failed ({ex.Message}) — sending anyway.");
}
}
}
}
@@ -2,6 +2,7 @@ using System;
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Security;
@@ -68,6 +69,10 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ "configVersion", configVersion },
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded },
{ "remoteConfigOutcome", outcome.ToString() },
// WinRT OOBE state sampled at this start. Together with previousExitType
// this makes late starts provable: "completed" on a first_run means OOBE
// was already over when the agent arrived (e.g. IME queue blocked).
{ "oobeStateAtAgentStart", OobeStateReader.Read() },
};
if (!string.IsNullOrEmpty(previousExit?.CrashExceptionType))
@@ -0,0 +1,382 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Serialization;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using Xunit;
namespace AutopilotMonitor.DecisionCore.Tests
{
/// <summary>
/// Session 4910a5a5 (2026-07-23) — advisory recovery. Field failure: DeviceSetup/Apps
/// failed (culprit app never started installing), the advisory defang armed the
/// <c>advisory_completion</c> window, the user pressed "Try again" ~23 min later, the apps
/// re-ran to 10/10 and <c>DeviceSetupProvisioningComplete</c> arrived — yet the window
/// fired 30 min after the advisory and un-defanged the stale failure
/// (<c>enrollment_failed: esp_terminal_failure</c>) on a live enrollment mid-AccountSetup,
/// because the advisory variant was exempt from ALL re-arm/rebase guards. Coverage:
/// <list type="bullet">
/// <item>The advisory records the failed category + culprit apps.</item>
/// <item>A ProvisioningComplete of the SAME category sets the resolved fact + emits
/// <c>esp_failure_advisory_resolved</c>; a different category does not.</item>
/// <item>Deadline fire after recovery + enforcement progress → re-arm, not Failed.</item>
/// <item>Deadline fire after recovery without progress → Failed with the truthful
/// <c>esp_recovered_without_completion_evidence</c> reason and
/// LastFailureTrigger=DeadlineFired (likely-stuck promotion off).</item>
/// <item>Unrecovered advisory keeps the legacy un-defang exactly (regression guard).</item>
/// <item>Reboot re-bases a recovered-advisory window (and still does NOT touch an
/// unrecovered one).</item>
/// <item>Snapshot round-trip + census surfacing of the new facts.</item>
/// </list>
/// </summary>
public sealed class AdvisoryRecoveryTests
{
private static readonly DateTime T0 = new DateTime(2026, 7, 23, 3, 30, 0, DateTimeKind.Utc);
private static DecisionSignal MakeSignal(
long ordinal,
DecisionSignalKind kind,
DateTime occurredAtUtc,
IReadOnlyDictionary<string, string>? payload = null)
{
return new DecisionSignal(
sessionSignalOrdinal: ordinal,
sessionTraceOrdinal: ordinal,
kind: kind,
kindSchemaVersion: 1,
occurredAtUtc: occurredAtUtc,
sourceOrigin: "test",
evidence: new Evidence(EvidenceKind.Synthetic, $"{kind}-{ordinal}", "test"),
payload: payload);
}
private static DecisionSignal DeadlineFired(long ordinal, DateTime occurredAtUtc) =>
MakeSignal(ordinal, DecisionSignalKind.DeadlineFired, occurredAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.AdvisoryCompletion });
private static DecisionEffect? FindTimelineEffect(DecisionStep step, string eventType) =>
step.Effects.FirstOrDefault(e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry
&& e.Parameters != null
&& e.Parameters.TryGetValue("eventType", out var et)
&& et == eventType);
private static ActiveDeadline? FindDeadline(DecisionState state, string name) =>
state.Deadlines.FirstOrDefault(d => d.Name == name);
/// <summary>
/// Replays the session-4910a5a5 shape up to (but excluding) the ESP terminal failure:
/// Classic UserDriven, ContinueAnyway profile, Hello disabled, DeviceSetup entered,
/// AccountSetup entered (04:18, BEFORE the 04:34 DeviceSetup/Apps failure — the IME
/// phase line landed while device apps were still at 9/10).
/// </summary>
private static DecisionState SetupSession(DecisionEngine engine)
{
var state = DecisionState.CreateInitial("sess-4910a5a5", "tenant-e46bc88e", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0)).NewState;
state = engine.Reduce(state, MakeSignal(
5, DecisionSignalKind.EspConfigDetected, T0.AddMinutes(1),
new Dictionary<string, string>
{
[SignalPayloadKeys.SkipUserEsp] = "false",
[SignalPayloadKeys.SkipDeviceEsp] = "false",
[SignalPayloadKeys.EspSyncFailureTimeoutMinutes] = "60",
[SignalPayloadKeys.EspAllowContinueAnyway] = "true",
})).NewState;
state = engine.Reduce(state, MakeSignal(
8, DecisionSignalKind.HelloPolicyDetected, T0.AddMinutes(2),
new Dictionary<string, string> { [SignalPayloadKeys.HelloEnabled] = "false" })).NewState;
state = engine.Reduce(state, MakeSignal(
10, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "DeviceSetup" })).NewState;
state = engine.Reduce(state, MakeSignal(
20, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(48),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
return state;
}
private static DecisionStep ApplyDeviceSetupAppsFailure(DecisionEngine engine, DecisionState state, long ordinal = 50) =>
engine.Reduce(state, MakeSignal(
ordinal, DecisionSignalKind.EspTerminalFailure, T0.AddMinutes(64),
new Dictionary<string, string>
{
["failureType"] = "Provisioning_DeviceSetup_Apps_Failed",
["failedSubcategory"] = "Apps",
["category"] = "DeviceSetup",
["likelyCulpritApps"] = "Oriflame Cosmectics - Teams Backgroundsv v2",
["likelyCulpritAppCount"] = "1",
}));
private static DecisionStep ApplyDeviceSetupResolved(DecisionEngine engine, DecisionState state, long ordinal = 60) =>
engine.Reduce(state, MakeSignal(
ordinal, DecisionSignalKind.DeviceSetupProvisioningComplete, T0.AddMinutes(87)));
// ================================================== advisory arming context ====
[Fact]
public void Advisory_RecordsFailedCategory_AndCulpritApps()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
var step = ApplyDeviceSetupAppsFailure(engine, state);
Assert.NotNull(step.NewState.EspAdvisoryFailureRecordedUtc);
Assert.Equal("DeviceSetup", step.NewState.EspAdvisoryFailureCategory?.Value);
Assert.Null(step.NewState.EspAdvisoryFailureResolvedUtc);
var advisory = FindTimelineEffect(step, "esp_failure_advisory");
Assert.NotNull(advisory);
Assert.Equal("Oriflame Cosmectics - Teams Backgroundsv v2", advisory!.Parameters!["likelyCulpritApps"]);
}
// ===================================================== recovery hook ====
[Fact]
public void DeviceSetupComplete_SameCategory_ResolvesAdvisory_AndEmitsStoryEvent()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
var step = ApplyDeviceSetupResolved(engine, state);
Assert.NotNull(step.NewState.EspAdvisoryFailureResolvedUtc);
Assert.Equal(60, step.NewState.EspAdvisoryFailureResolvedUtc!.SourceSignalOrdinal);
// The anchor itself stays — the advisory happened; it just no longer blocks.
Assert.NotNull(step.NewState.EspAdvisoryFailureRecordedUtc);
var resolved = FindTimelineEffect(step, "esp_failure_advisory_resolved");
Assert.NotNull(resolved);
Assert.Equal("DeviceSetup", resolved!.Parameters!["category"]);
Assert.Equal("23", resolved.Parameters["minutesSinceAdvisory"]);
}
[Fact]
public void DeviceSetupComplete_DifferentCategory_DoesNotResolve()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
// Advisory for AccountSetup — DeviceSetup resolving must not defuse it.
state = engine.Reduce(state, MakeSignal(
50, DecisionSignalKind.EspTerminalFailure, T0.AddMinutes(64),
new Dictionary<string, string>
{
["failureType"] = "Provisioning_AccountSetup_Apps_Failed",
["failedSubcategory"] = "Apps",
["category"] = "AccountSetup",
})).NewState;
var step = ApplyDeviceSetupResolved(engine, state);
Assert.Null(step.NewState.EspAdvisoryFailureResolvedUtc);
Assert.Null(FindTimelineEffect(step, "esp_failure_advisory_resolved"));
}
[Fact]
public void DeviceSetupComplete_Duplicate_ResolvesOnlyOnce()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
// Post-reboot duplicate (AnchorAlreadySet passthrough branch).
var duplicate = engine.Reduce(state, MakeSignal(
65, DecisionSignalKind.DeviceSetupProvisioningComplete, T0.AddMinutes(92)));
Assert.Equal(60, duplicate.NewState.EspAdvisoryFailureResolvedUtc!.SourceSignalOrdinal);
Assert.Null(FindTimelineEffect(duplicate, "esp_failure_advisory_resolved"));
}
// ================================================ deadline fire semantics ====
[Fact]
public void DeadlineFire_AfterRecovery_WithEnforcementProgress_ReArmsInsteadOfFailing()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
// Post-reboot IME re-assert of AccountSetup (session shape: 05:02:59, ordinal 654)
// — user-phase enforcement demonstrably progressed since the arming at ordinal 50.
state = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(92),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state, DeadlineFired(80, T0.AddMinutes(94)));
// Not failed — the window re-arms with fresh baselines instead.
Assert.Null(step.NewState.Outcome);
Assert.NotEqual(SessionStage.Failed, step.NewState.Stage);
var rearmed = FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion);
Assert.NotNull(rearmed);
Assert.Equal(T0.AddMinutes(94).AddMinutes(30), rearmed!.DueAtUtc);
Assert.Contains(step.Effects, e => e.Kind == DecisionEffectKind.ScheduleDeadline);
}
[Fact]
public void DeadlineFire_AfterRecovery_NoProgress_FailsWithRecoveredReason()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var step = engine.Reduce(state, DeadlineFired(80, T0.AddMinutes(94)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentFailed, step.NewState.Outcome);
var failed = FindTimelineEffect(step, "enrollment_failed");
Assert.NotNull(failed);
// Truthful reason: the original failure recovered — do not blame it.
Assert.Equal("esp_recovered_without_completion_evidence", failed!.Parameters!["reason"]);
Assert.Equal("true", failed.Parameters["failureRecovered"]);
Assert.Equal("advisory_recovered_window_expired_without_completion_evidence", failed.Parameters["advisoryReason"]);
// Likely-stuck app promotion stays off: apps are not stuck, they succeeded.
Assert.Equal(nameof(DecisionSignalKind.DeadlineFired), step.NewState.LastFailureTrigger?.Value);
}
[Fact]
public void DeadlineFire_WithoutRecovery_KeepsLegacyUndefang()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
// No recovery — even with enforcement progress the advisory variant must
// un-defang exactly as before (its anchor failure never un-happened).
state = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(92),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state, DeadlineFired(80, T0.AddMinutes(94)));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
var failed = FindTimelineEffect(step, "enrollment_failed");
Assert.NotNull(failed);
Assert.Equal("esp_terminal_failure", failed!.Parameters!["reason"]);
Assert.Equal(nameof(DecisionSignalKind.EspTerminalFailure), step.NewState.LastFailureTrigger?.Value);
}
// ======================================================= reboot rebase ====
[Fact]
public void Reboot_AfterRecovery_RebasesAdvisoryWindow()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var step = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.SystemRebootObserved, T0.AddMinutes(92)));
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.CancelDeadline
&& e.CancelDeadlineName == DeadlineNames.AdvisoryCompletion);
var rebased = FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion);
Assert.NotNull(rebased);
Assert.Equal(T0.AddMinutes(92).AddMinutes(30), rebased!.DueAtUtc);
}
[Fact]
public void Reboot_WithoutRecovery_DoesNotTouchAdvisoryWindow()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
var originalDue = FindDeadline(state, DeadlineNames.AdvisoryCompletion)!.DueAtUtc;
var step = engine.Reduce(state, MakeSignal(
70, DecisionSignalKind.SystemRebootObserved, T0.AddMinutes(92)));
Assert.DoesNotContain(step.Effects, e => e.Kind == DecisionEffectKind.CancelDeadline);
Assert.Equal(originalDue, FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion)!.DueAtUtc);
}
// ============================================== full session-4910a5a5 replay ====
[Fact]
public void Session4910a5a5_FailureRecoveryRebootFire_DoesNotFailTheLiveEnrollment()
{
// Field ordinals: 527 EspTerminalFailure → 611 DeviceSetupProvisioningComplete
// (post "Try again", apps 10/10) → 635 SystemRebootObserved → 654 AccountSetup
// re-assert → 671 DeadlineFired. Pre-fix outcome: enrollment_failed at the fire.
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state, ordinal: 527).NewState;
state = ApplyDeviceSetupResolved(engine, state, ordinal: 611).NewState;
state = engine.Reduce(state, MakeSignal(
635, DecisionSignalKind.SystemRebootObserved, T0.AddMinutes(92))).NewState;
state = engine.Reduce(state, MakeSignal(
654, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(93),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
var step = engine.Reduce(state, DeadlineFired(671, T0.AddMinutes(94)));
Assert.Null(step.NewState.Outcome);
Assert.NotEqual(SessionStage.Failed, step.NewState.Stage);
Assert.NotNull(FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion));
}
// =============================================== serialization + census ====
[Fact]
public void SnapshotRoundTrip_PreservesRecoveryFacts()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var roundtripped = StateSerializer.Deserialize(StateSerializer.Serialize(state));
Assert.Equal("DeviceSetup", roundtripped.EspAdvisoryFailureCategory?.Value);
Assert.Equal(state.EspAdvisoryFailureResolvedUtc!.Value, roundtripped.EspAdvisoryFailureResolvedUtc?.Value);
Assert.Equal(state.EspAdvisoryFailureResolvedUtc.SourceSignalOrdinal, roundtripped.EspAdvisoryFailureResolvedUtc?.SourceSignalOrdinal);
}
[Fact]
public void LegacySnapshot_WithoutRecoveryFacts_DeserializesToNull()
{
// Simulate a snapshot written by a pre-fix agent: strip the new properties from
// the JSON entirely — deserialization must default them to null (additive compat).
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var legacy = Newtonsoft.Json.Linq.JObject.Parse(StateSerializer.Serialize(state));
foreach (var prop in legacy.Properties()
.Where(p => p.Name.IndexOf("EspAdvisoryFailureResolved", StringComparison.OrdinalIgnoreCase) >= 0
|| p.Name.IndexOf("EspAdvisoryFailureCategory", StringComparison.OrdinalIgnoreCase) >= 0)
.ToList())
{
prop.Remove();
}
var roundtripped = StateSerializer.Deserialize(legacy.ToString());
Assert.Null(roundtripped.EspAdvisoryFailureResolvedUtc);
Assert.Null(roundtripped.EspAdvisoryFailureCategory);
// The pre-existing advisory anchor still round-trips.
Assert.NotNull(roundtripped.EspAdvisoryFailureRecordedUtc);
}
[Fact]
public void Census_SurfacesAdvisoryResolved()
{
var engine = new DecisionEngine();
var state = SetupSession(engine);
state = ApplyDeviceSetupAppsFailure(engine, state).NewState;
state = ApplyDeviceSetupResolved(engine, state).NewState;
var census = DecisionStateSignalCensus.Build(state);
Assert.Contains("esp_advisory_failure_resolved", census.SignalsSeen);
Assert.True(census.SignalTimestamps.ContainsKey("espAdvisoryFailureResolved"));
}
}
}
@@ -114,7 +114,10 @@ namespace AutopilotMonitor.DecisionCore.Tests
// state-change-only dedupe anchor for the completion_waiting event → 17.
// Session 772fe502 (2026-07-13) added `helloWizardStartedUtc` — genuine
// Shell-Core 62404 wizard launch; vetoes the policy-disabled Hello skip → 18.
Assert.Equal(18, facts.Count);
// Session 4910a5a5 (2026-07-23) added `espAdvisoryFailureResolvedUtc` (the failed
// category recovered — lifts the advisory's re-arm/rebase guard exemption) and
// `espAdvisoryFailureCategory` (which category the advisory was for) → 20.
Assert.Equal(20, facts.Count);
Assert.All(facts.Values, v => Assert.Null(v));
}
@@ -236,10 +239,12 @@ namespace AutopilotMonitor.DecisionCore.Tests
// `completionWaitingFingerprint` as the state-change-only dedupe anchor for
// the completion_waiting event → 17. Session 772fe502 (2026-07-13) added
// `helloWizardStartedUtc` — the genuine Shell-Core 62404 wizard-launch fact
// that vetoes/retracts the policy-disabled Hello skip → 18. If this number
// ever changes, both the count expectation AND the actual snapshot output
// need to evolve in lockstep.
Assert.Equal(18, expectedFactKeys.Count);
// that vetoes/retracts the policy-disabled Hello skip → 18. Session 4910a5a5
// (2026-07-23) added `espAdvisoryFailureResolvedUtc` + `espAdvisoryFailureCategory`
// — advisory-recovery facts that lift the AdvisoryCompletion guard exemption → 20.
// If this number ever changes, both the count expectation AND the actual snapshot
// output need to evolve in lockstep.
Assert.Equal(20, expectedFactKeys.Count);
var snapshot = DecisionStateSnapshotBuilder.Build(DecisionState.CreateInitial("s", "t"));
var facts = (Dictionary<string, object?>)snapshot["facts"]!;
@@ -0,0 +1,333 @@
using System.Text.Json;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Functions.Apps;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// F1 PR1 (insights spec §F1, source-data audit Q2/Q3): app identity on the name-keyed
/// summary row and the ESP-blocking join.
///
/// Ingest side: <c>appId</c> is adopted from the app events; a second appId under the same
/// name flags the row (<see cref="AppInstallSummary.AppIdCollision"/>) — in-batch and
/// cross-batch. Store side: sentinel-gated columns + sticky collision. Join side:
/// <see cref="EspBlockingSets"/> is positive evidence only, and
/// <see cref="TableStorageService.ShouldStampEspBlocking"/> never stamps collision rows,
/// id-less rows, or non-members. Aggregate side: collision rows leave every per-app fleet
/// aggregate with a disclosed exclusion count.
/// </summary>
public class AppInstallIdentityAndEspBlockingTests
{
private static readonly DateTime T0 = new(2026, 7, 26, 10, 0, 0, DateTimeKind.Utc);
private const string IdA = "aaaaaaaa-1111-2222-3333-444444444444";
private const string IdB = "bbbbbbbb-1111-2222-3333-444444444444";
// ── ingest fold: appId adoption + in-batch collision ────────────────────
private static Dictionary<string, AppInstallAggregationState> Aggregate(params EnrollmentEvent[] events)
{
var summaries = new Dictionary<string, AppInstallAggregationState>(StringComparer.OrdinalIgnoreCase);
foreach (var evt in events)
EventIngestProcessor.AggregateAppInstallEvent(evt, "tenant", "session", summaries);
return summaries;
}
private static EnrollmentEvent AppEvent(string eventType, string? appId = null, string appName = "App")
{
var evt = new EnrollmentEvent
{
EventType = eventType,
Timestamp = T0,
Data = new Dictionary<string, object> { ["appName"] = appName },
};
if (appId != null) evt.Data["appId"] = appId;
return evt;
}
[Fact]
public void Ingest_AdoptsAppId_FromFirstObservingEvent()
{
var s = Aggregate(
AppEvent("app_install_started", IdA),
AppEvent("app_install_completed", IdA))["App"].Summary;
Assert.Equal(IdA, s.AppId);
Assert.False(s.AppIdCollision);
}
[Fact]
public void Ingest_SameName_DifferentAppId_FlagsCollision_FirstSeenIdWins()
{
// Device- + user-scope assignment of the same display name in one batch (audit Q3:
// observed in production as two "Company Portal" appIds in one session).
var s = Aggregate(
AppEvent("app_install_started", IdA),
AppEvent("app_install_started", IdB))["App"].Summary;
Assert.True(s.AppIdCollision);
Assert.Equal(IdA, s.AppId);
}
[Fact]
public void Ingest_AppIdCaseDrift_IsNotACollision()
{
var s = Aggregate(
AppEvent("app_install_started", IdA),
AppEvent("app_install_completed", IdA.ToUpperInvariant()))["App"].Summary;
Assert.False(s.AppIdCollision);
}
[Fact]
public void Ingest_EventsWithoutAppId_LeaveSentinelEmpty()
{
var s = Aggregate(AppEvent("app_install_completed"))["App"].Summary;
Assert.Equal(string.Empty, s.AppId);
Assert.False(s.AppIdCollision);
}
// ── reconcile: cross-batch identity ─────────────────────────────────────
[Fact]
public void Reconcile_AdoptsStoredAppId_WhenIncomingBatchHasNone()
{
var existing = new TableEntity("tenant", "session_App") { ["AppId"] = IdA };
var summary = new AppInstallSummary { AppName = "App", StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal(IdA, summary.AppId);
Assert.False(summary.AppIdCollision);
}
[Fact]
public void Reconcile_CrossBatchDifferentAppId_FlagsCollision_StoredIdWins()
{
var existing = new TableEntity("tenant", "session_App") { ["AppId"] = IdA };
var summary = new AppInstallSummary { AppName = "App", AppId = IdB, StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.True(summary.AppIdCollision);
Assert.Equal(IdA, summary.AppId); // first-seen (stored) identity wins, matching the fold rule
}
[Fact]
public void Reconcile_CollisionIsSticky_AndEspBlockingVerdictIsAdopted()
{
var existing = new TableEntity("tenant", "session_App")
{
["AppId"] = IdA,
["AppIdCollision"] = true,
["EspBlocking"] = true,
};
var summary = new AppInstallSummary { AppName = "App", AppId = IdA, StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.True(summary.AppIdCollision); // one-way sticky
Assert.True(summary.EspBlocking); // terminal-path verdict survives late batches
}
// ── entity builder: sentinel gating ─────────────────────────────────────
[Fact]
public void EntityBuilder_IdentityColumns_AreSentinelGated()
{
var empty = TableStorageService.BuildAppInstallSummaryEntity(
new AppInstallSummary { AppName = "App" }, "rk");
// Absent columns → Merge-mode preserves prior values / stays unknown.
Assert.False(empty.ContainsKey("AppId"));
Assert.False(empty.ContainsKey("EspBlocking"));
Assert.False(empty.ContainsKey("AppIdCollision"));
var full = TableStorageService.BuildAppInstallSummaryEntity(
new AppInstallSummary { AppName = "App", AppId = IdA, EspBlocking = true, AppIdCollision = true }, "rk");
Assert.Equal(IdA, full.GetString("AppId"));
Assert.True(full.GetBoolean("EspBlocking"));
Assert.True(full.GetBoolean("AppIdCollision"));
}
// ── EspBlockingSets: positive-evidence parsing ──────────────────────────
private static Dictionary<string, object> Lists(
object? win32 = null, object? userWin32 = null, object? msi = null, object? pfn = null,
long? win32Count = null)
{
var d = new Dictionary<string, object> { ["source"] = "registry_firstsync" };
if (win32 != null) d["espTrackedWin32AppIds"] = win32;
if (userWin32 != null) d["espTrackedUserWin32AppIds"] = userWin32;
if (msi != null) d["espTrackedMsiProductCodes"] = msi;
if (pfn != null) d["espTrackedModernAppPfns"] = pfn;
if (win32Count.HasValue) d["espTrackedWin32Count"] = win32Count.Value;
return d;
}
[Fact]
public void Sets_PayloadWithoutListKeys_YieldsNull_NoListsIsNotEmptyLists()
{
Assert.Null(EspBlockingSets.FromEventData(null));
Assert.Null(EspBlockingSets.FromEventData(Lists())); // probe found no Diagnostics key
}
[Fact]
public void Sets_MembershipIsCaseInsensitive_AcrossAllFourLists()
{
// Storage rehydration shape: List<object> (EventDataNormalizer).
var sets = EspBlockingSets.FromEventData(Lists(
win32: new List<object> { IdA },
msi: new List<object> { "{11111111-2222-3333-4444-555555555555}" },
pfn: new List<object> { "Contoso.App_abc123" }))!;
Assert.True(sets.Contains(IdA.ToUpperInvariant()));
Assert.True(sets.Contains("{11111111-2222-3333-4444-555555555555}"));
Assert.True(sets.Contains("Contoso.App_abc123"));
Assert.False(sets.Contains(IdB)); // absent ⇒ unknown at the caller, never false here
Assert.False(sets.Contains(null));
Assert.Equal(3, sets.ListedCount);
Assert.False(sets.IsTruncated);
}
[Fact]
public void Sets_EmittedCountAboveListLength_MeansTruncated()
{
var sets = EspBlockingSets.FromEventData(Lists(
win32: new List<object> { IdA }, win32Count: 60))!;
Assert.True(sets.IsTruncated);
}
[Fact]
public void Sets_UserScopeSubset_IsExposedForCoverageLabeling()
{
var sets = EspBlockingSets.FromEventData(Lists(
win32: new List<object> { IdA, IdB },
userWin32: new List<object> { IdB }))!;
Assert.Contains(IdB, sets.UserWin32AppIds);
Assert.DoesNotContain(IdA, sets.UserWin32AppIds);
}
// ── terminal-path stamping predicate ────────────────────────────────────
private static TableEntity Row(string? appId = null, bool? collision = null, bool? espBlocking = null)
{
var row = new TableEntity("tenant", "session_App");
if (appId != null) row["AppId"] = appId;
if (collision.HasValue) row["AppIdCollision"] = collision.Value;
if (espBlocking.HasValue) row["EspBlocking"] = espBlocking.Value;
return row;
}
[Fact]
public void Stamp_OnlyPositiveMembers_NeverIdlessCollidedOrAlreadyStamped()
{
var sets = EspBlockingSets.FromEventData(Lists(win32: new List<object> { IdA }))!;
Assert.True(TableStorageService.ShouldStampEspBlocking(Row(IdA), sets));
Assert.False(TableStorageService.ShouldStampEspBlocking(Row(), sets)); // no identity → unknown
Assert.False(TableStorageService.ShouldStampEspBlocking(Row(IdB), sets)); // absent ⇒ unknown, never false
Assert.False(TableStorageService.ShouldStampEspBlocking(Row(IdA, collision: true), sets)); // ambiguous identity
Assert.False(TableStorageService.ShouldStampEspBlocking(Row(IdA, espBlocking: true), sets)); // idempotent
}
// ── per-app fleet aggregates: collision rows leave, disclosed ───────────
private static AppInstallSummary Summary(
string appName, string status = "Succeeded", bool collision = false, int duration = 30)
=> new()
{
TenantId = "00000000-0000-0000-0000-000000000fa1",
SessionId = "s-1",
AppName = appName,
Status = status,
TerminalState = status == "Succeeded" ? "Installed" : "Error",
DurationSeconds = duration,
StartedAt = DateTime.UtcNow.AddHours(-2),
AppIdCollision = collision,
};
[Fact]
public void AppMetricsPayload_ExcludesCollisionRowsFromGroups_DisclosesCount()
{
var summaries = new List<AppInstallSummary>
{
Summary("Clean App"),
Summary("Clean App"),
Summary("Company Portal", collision: true),
Summary("Company Portal", status: "Failed", collision: true),
};
var root = JsonSerializer.SerializeToElement(MetricsMath.BuildAppMetricsPayload(summaries));
Assert.Equal(1, root.GetProperty("totalApps").GetInt32()); // collided name forms no group
Assert.Equal(4, root.GetProperty("totalInstalls").GetInt32()); // raw row count unchanged
Assert.Equal(2, root.GetProperty("totalCollisionExcluded").GetInt32());
Assert.Empty(root.GetProperty("topFailingApps").EnumerateArray()); // the collided Failed row ranks nowhere
}
[Fact]
public void AppsList_ExcludesCollisionRows_DisclosesCount()
{
var summaries = new List<AppInstallSummary>
{
Summary("Clean App"),
Summary("Company Portal", collision: true),
};
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
var apps = root.GetProperty("apps").EnumerateArray().ToList();
Assert.Single(apps);
Assert.Equal("Clean App", apps[0].GetProperty("appName").GetString());
Assert.Equal(2, root.GetProperty("totalInstalls").GetInt32());
Assert.Equal(1, root.GetProperty("collisionExcluded").GetInt32());
}
[Fact]
public async Task AppAnalytics_ExcludesCollisionRows_DisclosesCount()
{
var repo = new Mock<ISessionRepository>();
repo.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync((SessionSummary?)null);
var summaries = new List<AppInstallSummary>
{
Summary("Company Portal"),
Summary("Company Portal", status: "Failed", collision: true), // the second identity's failure
};
var root = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, repo.Object, "Company Portal", days: 30));
Assert.Equal(1, root.GetProperty("collisionExcluded").GetInt32());
var stats = root.GetProperty("summary");
Assert.Equal(1, stats.GetProperty("totalInstalls").GetInt32());
Assert.Equal(0, stats.GetProperty("failed").GetInt32()); // ambiguous failure not attributed
Assert.Equal(0d, stats.GetProperty("failureRate").GetDouble());
}
[Fact]
public async Task AppAnalytics_AllRowsCollided_ReturnsEmptyShape_WithDisclosure()
{
var repo = new Mock<ISessionRepository>();
repo.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync((SessionSummary?)null);
var summaries = new List<AppInstallSummary> { Summary("Company Portal", collision: true) };
var root = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, repo.Object, "Company Portal", days: 30));
Assert.Equal(1, root.GetProperty("collisionExcluded").GetInt32());
Assert.Equal(0, root.GetProperty("summary").GetProperty("totalInstalls").GetInt32());
}
}
@@ -0,0 +1,201 @@
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// PR0 (2026-07-26, source-data audit §0.5 of tasks/insights-expansion-spec.md).
///
/// Ingest side: the agent emits <c>app_install_completed</c> for EVERY terminal transition —
/// including Skipped/Postponed no-ops (V1 wire parity; production: 78 % of zero-duration rows
/// were skips). The payload's <c>state</c> field must be persisted as
/// <see cref="AppInstallSummary.TerminalState"/> so metrics can exclude skips from durations
/// and rates.
///
/// Store side: <see cref="TableStorageService.ReconcileAppInstallSummaryWithExisting"/> pins the
/// cross-batch contract — earliest StartedAt wins, terminal status is sticky, and the Q4
/// out-of-order case (terminal batch first, started batch later) recomputes the duration.
/// </summary>
public class AppInstallTerminalStateAndReconcileTests
{
private static readonly DateTime T0 = new(2026, 7, 26, 10, 0, 0, DateTimeKind.Utc);
// ── ingest: TerminalState fold ──────────────────────────────────────────
private static Dictionary<string, AppInstallAggregationState> Aggregate(params EnrollmentEvent[] events)
{
var summaries = new Dictionary<string, AppInstallAggregationState>(StringComparer.OrdinalIgnoreCase);
foreach (var evt in events)
EventIngestProcessor.AggregateAppInstallEvent(evt, "tenant", "session", summaries);
return summaries;
}
private static EnrollmentEvent AppEvent(string eventType, DateTime ts, string? state = null, string appName = "App")
{
var evt = new EnrollmentEvent
{
EventType = eventType,
Timestamp = ts,
Data = new Dictionary<string, object> { ["appName"] = appName },
};
if (state != null) evt.Data["state"] = state;
return evt;
}
[Theory]
[InlineData("Installed")]
[InlineData("Skipped")]
[InlineData("Postponed")]
public void Completed_PersistsPayloadState(string state)
{
var s = Aggregate(AppEvent("app_install_completed", T0, state))["App"].Summary;
Assert.Equal("Succeeded", s.Status); // backward-compatible status
Assert.Equal(state, s.TerminalState); // new precision column
}
[Fact]
public void Completed_WithoutState_LeavesSentinelEmpty_NeverGuessed()
{
var s = Aggregate(AppEvent("app_install_completed", T0))["App"].Summary;
Assert.Equal(string.Empty, s.TerminalState);
}
[Fact]
public void Completed_WithNonTerminalOrGarbageState_LeavesSentinelEmpty()
{
// A payload state outside the terminal set (bug or drift) must not be persisted.
var s = Aggregate(AppEvent("app_install_completed", T0, "Downloading"))["App"].Summary;
Assert.Equal(string.Empty, s.TerminalState);
}
[Fact]
public void Failed_SetsError()
{
var s = Aggregate(AppEvent("app_install_failed", T0))["App"].Summary;
Assert.Equal("Failed", s.Status);
Assert.Equal("Error", s.TerminalState);
}
[Fact]
public void FailedThenCompleted_RetrySucceeds_LastTerminalWins()
{
// IME retry path (audit Q1): Error → retry → Installed. The later completed event is
// the authoritative outcome for Status AND TerminalState.
var s = Aggregate(
AppEvent("app_install_failed", T0),
AppEvent("app_install_completed", T0.AddMinutes(10), "Installed"))["App"].Summary;
Assert.Equal("Succeeded", s.Status);
Assert.Equal("Installed", s.TerminalState);
}
[Fact]
public void SkippedEvent_SetsSkipped_ButNeverOverridesStrongerTerminal()
{
var fresh = Aggregate(AppEvent("app_install_skipped", T0))["App"].Summary;
Assert.Equal("Succeeded", fresh.Status);
Assert.Equal("Skipped", fresh.TerminalState);
var afterFailure = Aggregate(
AppEvent("app_install_failed", T0),
AppEvent("app_install_skipped", T0.AddSeconds(5)))["App"].Summary;
Assert.Equal("Error", afterFailure.TerminalState);
}
// ── store: cross-batch reconcile ────────────────────────────────────────
[Fact]
public void Reconcile_Q4_OutOfOrder_RecomputesDurationFromStoredCompletedAt()
{
// Terminal batch landed first: row has CompletedAt but no usable duration.
var existing = new TableEntity("tenant", "session_App")
{
["Status"] = "Succeeded",
["StartedAt"] = new DateTimeOffset(T0.AddMinutes(5)), // late (completed-event) stamp
["CompletedAt"] = new DateTimeOffset(T0.AddMinutes(5)),
};
// Started batch arrives now with the true (earlier) start.
var summary = new AppInstallSummary
{
AppName = "App", SessionId = "session", TenantId = "tenant",
Status = "InProgress", StartedAt = T0,
};
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal(T0, summary.StartedAt); // earlier start kept
Assert.Equal(T0.AddMinutes(5), summary.CompletedAt); // endpoint adopted from the row
Assert.Equal(300, summary.DurationSeconds); // recomputed — was permanently unset before PR0
Assert.Equal("Succeeded", summary.Status); // sticky terminal (below)
}
[Fact]
public void Reconcile_TerminalStatusIsSticky_AgainstLateInProgressBatch()
{
var existing = new TableEntity("tenant", "session_App") { ["Status"] = "Failed" };
var summary = new AppInstallSummary { AppName = "App", Status = "InProgress", StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal("Failed", summary.Status);
}
[Fact]
public void Reconcile_TerminalToTerminal_StaysWithIncomingEvidence()
{
// Failed row + succeeded batch (successful retry) → the new terminal wins.
var existing = new TableEntity("tenant", "session_App") { ["Status"] = "Failed" };
var summary = new AppInstallSummary { AppName = "App", Status = "Succeeded", StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal("Succeeded", summary.Status);
}
[Fact]
public void Reconcile_Q4Guard_DoesNotFireWithoutStoredCompletedAt()
{
var existing = new TableEntity("tenant", "session_App") { ["Status"] = "InProgress" };
var summary = new AppInstallSummary { AppName = "App", Status = "InProgress", StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Null(summary.CompletedAt);
Assert.Equal(0, summary.DurationSeconds);
}
[Fact]
public void Reconcile_Q4Guard_IgnoresStoredCompletedAtBeforeIncomingStart()
{
// Negative interval (clock mess) must not produce a negative duration.
var existing = new TableEntity("tenant", "session_App")
{
["Status"] = "Succeeded",
["CompletedAt"] = new DateTimeOffset(T0.AddMinutes(-10)),
};
var summary = new AppInstallSummary { AppName = "App", Status = "InProgress", StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Null(summary.CompletedAt);
Assert.Equal(0, summary.DurationSeconds);
}
// ── entity builder: sentinel gating for the new column ─────────────────
[Fact]
public void EntityBuilder_TerminalState_IsSentinelGated()
{
var withState = new AppInstallSummary { AppName = "App", TerminalState = "Skipped" };
var entity = TableStorageService.BuildAppInstallSummaryEntity(withState, "rk");
Assert.Equal("Skipped", entity.GetString("TerminalState"));
var withoutState = new AppInstallSummary { AppName = "App" };
var empty = TableStorageService.BuildAppInstallSummaryEntity(withoutState, "rk");
// Column absent → Merge-mode preserves a prior batch's terminal state.
Assert.False(empty.ContainsKey("TerminalState"));
}
}
@@ -0,0 +1,247 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Functions.Apps;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the terminal-only failure-rate convention on the Apps dashboard aggregations
/// (<see cref="AppsAnalyticsHelper"/>): every failureRate is Failed / (Failed + Succeeded) —
/// finished installs only. "InProgress" rows (still installing, or orphaned by a session that
/// died mid-install) sit in the install counts but never dilute a rate, matching the enrollment
/// success-rate convention. Responses are anonymous projections, so tests assert on the
/// JSON wire shape like the sibling suites.
/// </summary>
public class AppsAnalyticsFailureRateTests
{
private const string TenantId = "00000000-0000-0000-0000-000000000fa1";
private static AppInstallSummary A(
string status,
string appName = "Contoso App",
DateTime? startedAt = null,
string sessionId = "s-1",
string appVersion = "")
=> new()
{
TenantId = TenantId,
SessionId = sessionId,
AppName = appName,
AppVersion = appVersion,
Status = status,
StartedAt = startedAt ?? DateTime.UtcNow.AddHours(-2),
DurationSeconds = status == "Succeeded" ? 30 : 0,
};
// ── /apps/list ──────────────────────────────────────────────────────────
[Fact]
public void List_FailureRate_IsOverFinishedInstalls_InProgressExcluded()
{
var summaries = new List<AppInstallSummary>
{
A("Failed"),
A("Succeeded"),
A("InProgress"),
A("InProgress"),
};
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
var app = root.GetProperty("apps")[0];
Assert.Equal(4, app.GetProperty("totalInstalls").GetInt32());
// 1 / (1 + 1) = 50% (over finished), NOT 1 / 4 = 25% (over all installs).
Assert.Equal(50d, app.GetProperty("failureRate").GetDouble());
}
[Fact]
public void List_FailureRate_IsZero_WhenNothingFinishedYet()
{
var summaries = new List<AppInstallSummary> { A("InProgress"), A("InProgress") };
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
Assert.Equal(0d, root.GetProperty("apps")[0].GetProperty("failureRate").GetDouble());
}
[Fact]
public void List_Trend_UsesFinishedRates_AndGatesOnFinishedSamples()
{
var now = DateTime.UtcNow;
var firstHalf = now.AddDays(-20); // days: 30 → midpoint at now - 15d
var secondHalf = now.AddDays(-5);
var summaries = new List<AppInstallSummary>();
// First half: 1 failed / 5 finished = 20%.
summaries.Add(A("Failed", startedAt: firstHalf));
for (var i = 0; i < 4; i++) summaries.Add(A("Succeeded", startedAt: firstHalf));
// Second half: 3 failed / 5 finished = 60% — plus 10 in-flight installs that
// previously dragged the half's rate down to 20% and masked the regression.
for (var i = 0; i < 3; i++) summaries.Add(A("Failed", startedAt: secondHalf));
for (var i = 0; i < 2; i++) summaries.Add(A("Succeeded", startedAt: secondHalf));
for (var i = 0; i < 10; i++) summaries.Add(A("InProgress", startedAt: secondHalf));
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
var app = root.GetProperty("apps")[0];
Assert.Equal("worsening", app.GetProperty("trend").GetString());
Assert.Equal(40d, app.GetProperty("trendDelta").GetDouble());
}
[Fact]
public void List_Trend_StaysStable_WhenAHalfLacksFiveFinishedInstalls()
{
var now = DateTime.UtcNow;
var summaries = new List<AppInstallSummary>();
// First half: 5 finished. Second half: 4 finished + many in-flight — the old
// install-count gate would have passed; the finished-sample gate must not.
for (var i = 0; i < 5; i++) summaries.Add(A("Succeeded", startedAt: now.AddDays(-20)));
for (var i = 0; i < 4; i++) summaries.Add(A("Failed", startedAt: now.AddDays(-5)));
for (var i = 0; i < 10; i++) summaries.Add(A("InProgress", startedAt: now.AddDays(-5)));
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
var app = root.GetProperty("apps")[0];
Assert.Equal("stable", app.GetProperty("trend").GetString());
Assert.Equal(JsonValueKind.Null, app.GetProperty("trendDelta").ValueKind);
}
// ── /apps/{appName}/analytics ───────────────────────────────────────────
private static async Task<JsonElement> BuildAnalyticsAsync(
List<AppInstallSummary> summaries, ISessionRepository? sessionRepo = null)
{
if (sessionRepo == null)
{
var mock = new Mock<ISessionRepository>();
mock.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync((SessionSummary?)null);
sessionRepo = mock.Object;
}
var result = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, sessionRepo, "Contoso App", days: 30);
return JsonSerializer.SerializeToElement(result);
}
[Fact]
public async Task Analytics_SummaryVersionAndTimeSeriesRates_AreOverFinishedInstalls()
{
var summaries = new List<AppInstallSummary>
{
A("Failed", appVersion: "1.0"),
A("Succeeded", appVersion: "1.0"),
A("InProgress", appVersion: "1.0"),
A("InProgress", appVersion: "1.0"),
};
var root = await BuildAnalyticsAsync(summaries);
// Summary: 1 / (1 + 1) = 50%, NOT 1 / 4 = 25%.
var summary = root.GetProperty("summary");
Assert.Equal(4, summary.GetProperty("totalInstalls").GetInt32());
Assert.Equal(50d, summary.GetProperty("failureRate").GetDouble());
var version = root.GetProperty("versionBreakdown")[0];
Assert.Equal(4, version.GetProperty("installs").GetInt32());
Assert.Equal(50d, version.GetProperty("failureRate").GetDouble());
// All four installs land in today's bucket → same terminal-only rate there.
var buckets = root.GetProperty("timeSeries").EnumerateArray().ToList();
var todayBucket = buckets.Single(b => b.GetProperty("installs").GetInt32() > 0);
Assert.Equal(50d, todayBucket.GetProperty("failureRate").GetDouble());
}
// ── PR0 (2026-07-26): skip / unmeasured classification ──────────────────
[Fact]
public void List_Skips_LeaveRateAndDurations_AndAreReportedSeparately()
{
var skip = A("Succeeded");
skip.TerminalState = "Skipped";
skip.DurationSeconds = 0;
var unmeasured = A("Succeeded");
unmeasured.TerminalState = "Installed";
unmeasured.DurationSeconds = 0;
var summaries = new List<AppInstallSummary> { A("Failed"), A("Succeeded"), skip, unmeasured };
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
var app = root.GetProperty("apps")[0];
Assert.Equal(4, app.GetProperty("totalInstalls").GetInt32());
Assert.Equal(2, app.GetProperty("succeeded").GetInt32()); // real install + unmeasured
Assert.Equal(1, app.GetProperty("skipped").GetInt32());
Assert.Equal(1, app.GetProperty("unmeasured").GetInt32());
// 1 / (1 + 2) — the skip is not an attempt.
Assert.Equal(33.3d, app.GetProperty("failureRate").GetDouble());
// avg over the single measured row (30s) — the zero must not drag it to 15.
Assert.Equal(30d, app.GetProperty("avgDurationSeconds").GetDouble());
}
[Fact]
public async Task Analytics_Summary_CarriesSkippedAndUnmeasured_AndP95OverMeasuredOnly()
{
var skip = A("Succeeded");
skip.TerminalState = "Skipped";
skip.DurationSeconds = 0;
var unmeasured = A("Succeeded");
unmeasured.TerminalState = "Installed";
unmeasured.DurationSeconds = 0;
var measured = A("Succeeded");
measured.TerminalState = "Installed";
measured.DurationSeconds = 100;
var root = await BuildAnalyticsAsync(new List<AppInstallSummary> { skip, unmeasured, measured });
var summary = root.GetProperty("summary");
Assert.Equal(2, summary.GetProperty("succeeded").GetInt32());
Assert.Equal(1, summary.GetProperty("skipped").GetInt32());
Assert.Equal(1, summary.GetProperty("unmeasured").GetInt32());
Assert.Equal(100d, summary.GetProperty("avgDurationSeconds").GetDouble());
Assert.Equal(100, summary.GetProperty("p95DurationSeconds").GetInt32());
Assert.Equal(0d, summary.GetProperty("failureRate").GetDouble());
}
[Fact]
public async Task Analytics_DeviceModelBreakdown_RatesAndSampleFloor_UseFinishedInstalls()
{
var sessionRepo = new Mock<ISessionRepository>();
sessionRepo.Setup(r => r.GetSessionAsync(TenantId, It.IsAny<string>()))
.ReturnsAsync((string _, string sessionId) => new SessionSummary
{
TenantId = TenantId,
SessionId = sessionId,
Manufacturer = "Contoso",
Model = sessionId.StartsWith("s-a") ? "ModelA" : "ModelB",
});
var summaries = new List<AppInstallSummary>();
// ModelA: 5 finished (1 failed → 20%) + 3 in-flight → ranked, rate undiluted.
summaries.Add(A("Failed", sessionId: "s-a-0"));
for (var i = 1; i <= 4; i++) summaries.Add(A("Succeeded", sessionId: $"s-a-{i}"));
for (var i = 5; i <= 7; i++) summaries.Add(A("InProgress", sessionId: $"s-a-{i}"));
// ModelB: 4 finished + 6 in-flight — 10 installs, but below the 5-finished floor.
for (var i = 0; i <= 3; i++) summaries.Add(A("Failed", sessionId: $"s-b-{i}"));
for (var i = 4; i <= 9; i++) summaries.Add(A("InProgress", sessionId: $"s-b-{i}"));
var root = await BuildAnalyticsAsync(summaries, sessionRepo.Object);
var breakdown = root.GetProperty("deviceModelBreakdown");
var model = Assert.Single(breakdown.EnumerateArray());
Assert.Equal("ModelA", model.GetProperty("model").GetString());
Assert.Equal(8, model.GetProperty("installs").GetInt32());
Assert.Equal(20d, model.GetProperty("failureRate").GetDouble());
}
}
@@ -34,6 +34,14 @@ public class AppsProjectionEquivalenceTests
["AppType"] = "Win32",
["AppVersion"] = "2.1.0",
["Status"] = "Failed",
// PR0/PR1 classification columns. The fixture MUST populate every column a Build*
// method reads: a column absent here passes the equivalence assert trivially (both
// sides map to default) — exactly how the missing TerminalState projection slipped
// through and zeroed totalSkipped in production (2026-07-27).
["TerminalState"] = "Skipped",
["AppId"] = "9e8d7c6b-5a49-4832-9182-736451234567",
["EspBlocking"] = true,
["AppIdCollision"] = true,
["StartedAt"] = new DateTimeOffset(DateTime.UtcNow.AddHours(-6)),
["CompletedAt"] = new DateTimeOffset(DateTime.UtcNow.AddHours(-5)),
["DurationSeconds"] = 340,
@@ -83,6 +91,9 @@ public class AppsProjectionEquivalenceTests
Assert.Equal(fromFull.AppName, fromProjected.AppName);
Assert.Equal(fromFull.Status, fromProjected.Status);
// IsSkipTerminalState / skip classification input — see the fixture comment.
Assert.Equal(fromFull.TerminalState, fromProjected.TerminalState);
Assert.Equal(fromFull.AppIdCollision, fromProjected.AppIdCollision);
Assert.Equal(fromFull.StartedAt, fromProjected.StartedAt);
Assert.Equal(fromFull.DurationSeconds, fromProjected.DurationSeconds);
Assert.Equal(fromFull.DownloadBytes, fromProjected.DownloadBytes);
@@ -122,6 +133,10 @@ public class AppsProjectionEquivalenceTests
Assert.Equal(fromFull.FailureMessage, fromProjected.FailureMessage);
Assert.Equal(fromFull.ExitCode, fromProjected.ExitCode);
Assert.Equal(fromFull.DetectionResult, fromProjected.DetectionResult);
Assert.Equal(fromFull.TerminalState, fromProjected.TerminalState);
Assert.Equal(fromFull.AppId, fromProjected.AppId);
Assert.Equal(fromFull.EspBlocking, fromProjected.EspBlocking);
Assert.Equal(fromFull.AppIdCollision, fromProjected.AppIdCollision);
}
[Fact]
@@ -0,0 +1,172 @@
using AutopilotMonitor.Functions.Functions.Infrastructure;
using AutopilotMonitor.Shared.DataAccess;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the pure helpers of <see cref="AzureMonitorAlertWebhookFunction"/> — the anonymous,
/// secret-gated bridge that turns Azure Monitor Common Alert Schema notifications into ops
/// events. Load-bearing bits: the secret gate is fail-closed (an unset app setting rejects
/// everything), severity mapping can never escalate on schema drift (unknown → Info), and
/// parsing is defensive (only the alert rule name is mandatory).
/// </summary>
public class AzureMonitorAlertWebhookFunctionTests
{
// ── SecretMatches ──────────────────────────────────────────────────────
[Fact]
public void SecretMatches_accepts_equal_secrets()
{
Assert.True(AzureMonitorAlertWebhookFunction.SecretMatches("s3cret-value", "s3cret-value"));
}
[Theory]
[InlineData("wrong", "right")]
[InlineData("right-but-longer", "right")]
[InlineData("", "configured")]
[InlineData(null, "configured")]
public void SecretMatches_rejects_mismatch_or_missing_provided(string? provided, string configured)
{
Assert.False(AzureMonitorAlertWebhookFunction.SecretMatches(provided, configured));
}
[Theory]
[InlineData(null)]
[InlineData("")]
public void SecretMatches_fails_closed_when_not_configured(string? configured)
{
// While the app setting is unset, NOTHING matches — not even an empty provided value.
Assert.False(AzureMonitorAlertWebhookFunction.SecretMatches("anything", configured));
Assert.False(AzureMonitorAlertWebhookFunction.SecretMatches("", configured));
Assert.False(AzureMonitorAlertWebhookFunction.SecretMatches(null, configured));
}
// ── MapSeverity ────────────────────────────────────────────────────────
[Theory]
[InlineData("Sev0", OpsEventSeverity.Critical)]
[InlineData("Sev1", OpsEventSeverity.Error)]
[InlineData("Sev2", OpsEventSeverity.Warning)]
[InlineData("Sev3", OpsEventSeverity.Info)]
[InlineData("Sev4", OpsEventSeverity.Info)]
public void MapSeverity_maps_azure_scale_to_ops_scale(string azure, string expected)
{
Assert.Equal(expected, AzureMonitorAlertWebhookFunction.MapSeverity(azure));
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData("Sev5")]
[InlineData("Critical")] // someone posting the ops scale directly must not pass through
public void MapSeverity_falls_back_to_info_on_unknown_values(string? azure)
{
Assert.Equal(OpsEventSeverity.Info, AzureMonitorAlertWebhookFunction.MapSeverity(azure));
}
// ── TryParse ───────────────────────────────────────────────────────────
private const string FullPayload = """
{
"schemaId": "azureMonitorCommonAlertSchema",
"data": {
"essentials": {
"alertId": "/subscriptions/xxx/providers/Microsoft.AlertsManagement/alerts/abc",
"alertRule": "Portal RSC prefetch tail",
"severity": "Sev2",
"signalType": "Log",
"monitorCondition": "Fired",
"monitoringService": "Log Alerts V2",
"alertTargetIDs": [
"/subscriptions/xxx/resourcegroups/rg-autopilotmonitor-prd-gwc/providers/microsoft.insights/components/autopilotmonitorweb"
],
"description": "RSC prefetches exceeding 10s",
"firedDateTime": "2026-07-27T15:30:00.000Z"
},
"alertContext": {
"condition": {
"windowSize": "PT1H",
"allOf": [
{
"searchQuery": "dependencies | where name contains '_rsc='",
"metricValue": 18.0,
"operator": "GreaterThan",
"threshold": "0"
}
]
}
}
}
}
""";
[Fact]
public void TryParse_extracts_essentials_and_metric_value()
{
var alert = AzureMonitorAlertWebhookFunction.TryParse(FullPayload);
Assert.NotNull(alert);
Assert.Equal("Portal RSC prefetch tail", alert!.AlertRule);
Assert.Equal("Sev2", alert.Severity);
Assert.Equal("Fired", alert.MonitorCondition);
Assert.Equal("RSC prefetches exceeding 10s", alert.Description);
Assert.Equal("Log Alerts V2", alert.MonitoringService);
// No targetResourceName in the payload → resource name segment of the first target ID.
Assert.Equal("autopilotmonitorweb", alert.TargetResource);
Assert.Equal(18.0, alert.MetricValue);
}
[Fact]
public void TryParse_prefers_explicit_target_resource_name()
{
const string payload = """
{"data":{"essentials":{"alertRule":"r","targetResourceName":"explicit-name",
"alertTargetIDs":["/sub/x/other-name"]}}}
""";
var alert = AzureMonitorAlertWebhookFunction.TryParse(payload);
Assert.Equal("explicit-name", alert!.TargetResource);
}
[Fact]
public void TryParse_minimal_payload_needs_only_the_alert_rule()
{
var alert = AzureMonitorAlertWebhookFunction.TryParse(
"""{"data":{"essentials":{"alertRule":"minimal"}}}""");
Assert.NotNull(alert);
Assert.Equal("minimal", alert!.AlertRule);
Assert.Null(alert.Severity);
Assert.Equal("Fired", alert.MonitorCondition); // default when absent
Assert.Null(alert.Description);
Assert.Null(alert.MetricValue);
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("not json at all")]
[InlineData("{}")] // no data
[InlineData("""{"data":{}}""")] // no essentials
[InlineData("""{"data":{"essentials":{}}}""")] // no alertRule
[InlineData("""{"data":{"essentials":{"alertRule":""}}}""")] // empty alertRule
[InlineData("""{"data":{"essentials":{"alertRule":123}}}""")] // wrong type
public void TryParse_rejects_unusable_payloads(string? body)
{
Assert.Null(AzureMonitorAlertWebhookFunction.TryParse(body!));
}
[Fact]
public void TryParse_tolerates_malformed_alert_context()
{
// alertContext with unexpected shapes must not break essentials extraction.
const string payload = """
{"data":{"essentials":{"alertRule":"r","severity":"Sev0"},
"alertContext":{"condition":{"allOf":"not-an-array"}}}}
""";
var alert = AzureMonitorAlertWebhookFunction.TryParse(payload);
Assert.NotNull(alert);
Assert.Equal("Sev0", alert!.Severity);
Assert.Null(alert.MetricValue);
}
}
@@ -183,4 +183,13 @@ public class ConfigRedactionTests
public void CanViewSecrets_PureDelegatedReader_False()
=> Assert.False(GetTenantConfigurationFunction.CanViewSecrets(
new RequestContext { IsDelegatedReader = true, TenantId = TenantA, TargetTenantId = TenantB }));
[Theory]
[InlineData(Constants.TenantRoles.Operator)]
[InlineData(Constants.TenantRoles.Viewer)]
public void CanViewSecrets_OwnTenantOperatorOrViewer_False(string role)
// Member-tier read-only Settings view (config GET is MemberRead): an own-tenant
// Operator/Viewer arrives with IsTenantAdmin=false ⇒ always the redacted copy.
=> Assert.False(GetTenantConfigurationFunction.CanViewSecrets(
new RequestContext { UserRole = role, TenantId = TenantA, TargetTenantId = TenantA }));
}
@@ -324,4 +324,42 @@ public class CrossTenantAccessTests
Assert.NotNull(entry);
Assert.Equal(TenantScoping.QueryParam, entry.TenantScoping);
}
// ── Cross-tenant rule WRITES: global/rules/{gather,analyze}/{ruleId} ──
// Bug (2026-07-24): a Global Admin editing a foreign tenant's rule via the JWT-scoped
// rules/{type}/{ruleId} route silently upserted into the GA's OWN tenant (200 → false "saved");
// the reload re-read the foreign tenant and showed the unchanged rule. Fix: dedicated cross-tenant
// write routes. They MUST be GlobalAdminOnly (a write tier → no delegated-read rescue, so a
// tenant-admin / delegated caller can never cross into another tenant's rules) + QueryParam (the
// tenant is named by ?tenantId=, not the {ruleId} segment). A downgrade to a tenant tier, or a slip
// to TenantScoping.None (which would ignore ?tenantId= and re-introduce the home-tenant misroute),
// is caught here.
[Theory]
[InlineData("PUT", "global/rules/gather/{ruleId}")]
[InlineData("DELETE", "global/rules/gather/{ruleId}")]
[InlineData("PUT", "global/rules/analyze/{ruleId}")]
[InlineData("DELETE", "global/rules/analyze/{ruleId}")]
public void GlobalRuleWriteRoutes_AreGlobalAdminOnly_QueryParam(string httpMethod, string routeTemplate)
{
var entry = EndpointAccessPolicyCatalog.Entries
.FirstOrDefault(e => e.HttpMethod == httpMethod.ToUpperInvariant() && e.RouteTemplate == routeTemplate);
Assert.NotNull(entry);
Assert.Equal(EndpointPolicy.GlobalAdminOnly, entry!.Policy);
Assert.Equal(TenantScoping.QueryParam, entry.TenantScoping);
}
[Theory]
[InlineData("PUT", "/api/global/rules/gather/rule-1")]
[InlineData("DELETE", "/api/global/rules/gather/rule-1")]
[InlineData("PUT", "/api/global/rules/analyze/rule-1")]
[InlineData("DELETE", "/api/global/rules/analyze/rule-1")]
public void GlobalRuleWrite_CrossTenant_GlobalAdmin_IsAllowed(string httpMethod, string path)
{
var (isBlocked, targetTenantId) = SimulateQueryParamCheck(httpMethod, path, TenantA, hasGlobalScope: true, queryTenantId: TenantB);
Assert.False(isBlocked, $"Global Admin should reach the target tenant on the cross-tenant write route: {httpMethod} {path}");
Assert.Equal(TenantB, targetTenantId);
}
}
@@ -75,8 +75,9 @@ public class DeletionManifestBuilderTests
Assert.Equal(2, c["classifierVerdictsByIdLevel"]);
Assert.Equal(11, c["signalsByKind"]);
// 15 cascade tables + tombstone = 16 steps; no inventory steps when no contributions row.
Assert.Equal(16, manifest.Steps.Count);
// 16 cascade tables (incl. SessionTimeBreakdowns, F1 PR2) + tombstone = 17 steps;
// no inventory steps when no contributions row.
Assert.Equal(17, manifest.Steps.Count);
Assert.Equal(DeletionStepClass.Final, manifest.Steps.Last().Class);
}
@@ -102,7 +103,8 @@ public class DeletionManifestBuilderTests
.WithSessionsByStage(1, propValue: "stage_marker")
.WithDeadEndsByReason(1, propValue: "deadend_marker")
.WithClassifierVerdictsByIdLevel(1, propValue: "classifier_marker")
.WithSignalsByKind(1, propValue: "signalkind_marker"));
.WithSignalsByKind(1, propValue: "signalkind_marker")
.WithSessionTimeBreakdown(propValue: "breakdown_marker"));
var manifest = await NewBuilder(reader).BuildAsync(
TenantId, SessionId, "admin_delete",
@@ -145,8 +147,8 @@ public class DeletionManifestBuilderTests
new DeletionRetentionContext { TenantRetentionDays = 90 });
var nonTombstoneSteps = manifest.Steps.Where(s => s.Class != DeletionStepClass.Final).ToList();
// Without a contributions row, neither inventory step is emitted → exactly 15 cascade-table steps.
Assert.Equal(15, nonTombstoneSteps.Count);
// Without a contributions row, neither inventory step is emitted → exactly 16 cascade-table steps.
Assert.Equal(16, nonTombstoneSteps.Count);
foreach (var step in nonTombstoneSteps)
{
Assert.Equal(0, step.RowCount);
@@ -175,8 +177,8 @@ public class DeletionManifestBuilderTests
new DeletionActor { Type = "admin", Actor = "alice@example.com" },
new DeletionRetentionContext { TenantRetentionDays = 90 });
// Expect: 15 cascade + 2 inventory + 1 tombstone = 18 steps.
Assert.Equal(18, manifest.Steps.Count);
// Expect: 16 cascade + 2 inventory + 1 tombstone = 19 steps.
Assert.Equal(19, manifest.Steps.Count);
var aggregate = manifest.Steps.Single(s => s.Class == DeletionStepClass.Aggregate);
Assert.Equal(DeletionStepNames.SoftwareInventoryDecrement, aggregate.Step);
@@ -640,6 +642,7 @@ public class DeletionManifestBuilderTests
private TableEntity? _vulnerabilityReport;
private TableEntity? _deviceSnapshot;
private TableEntity? _eventSessionIndex;
private TableEntity? _sessionTimeBreakdown;
private TableEntity? _sessionInventoryContributions;
private string _sessionsIndexRowKey = "INDEX_" + SessionId;
@@ -719,6 +722,16 @@ public class DeletionManifestBuilderTests
return this;
}
public ReaderSeed WithSessionTimeBreakdown(string propValue = "tab")
{
_sessionTimeBreakdown = new TableEntity(TenantId, SessionId)
{
["Marker"] = propValue,
};
_sessionTimeBreakdown.ETag = new ETag("0xTAB");
return this;
}
public ReaderSeed WithEventTypeIndex(int n, string propValue = "eti")
{
// PK = {tenantId}_{eventType}, RK ends with _{sessionId}.
@@ -796,6 +809,8 @@ public class DeletionManifestBuilderTests
.ReturnsAsync(_deviceSnapshot);
reader.Setup(r => r.GetEntityOrNullAsync(Constants.TableNames.EventSessionIndex, TenantId, SessionId, It.IsAny<CancellationToken>()))
.ReturnsAsync(_eventSessionIndex);
reader.Setup(r => r.GetEntityOrNullAsync(Constants.TableNames.SessionTimeBreakdowns, TenantId, SessionId, It.IsAny<CancellationToken>()))
.ReturnsAsync(_sessionTimeBreakdown);
reader.Setup(r => r.GetEntityOrNullAsync(Constants.TableNames.SessionInventoryContributions, TenantId, SessionId, It.IsAny<CancellationToken>()))
.ReturnsAsync(_sessionInventoryContributions);
@@ -0,0 +1,809 @@
using AutopilotMonitor.Functions.Functions.Metrics;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// F2 PR4 (insights spec §F2): device-key normalization (junk serials), terminal-only chain
/// maintenance (upsert, cap, deletion-ref pruning), journey grouping (first-success end,
/// 30-day gap, Incomplete attempts, WG-Pending stays open), the pure daily FTR aggregation
/// core (<see cref="MaintenanceService.BuildDeviceJourneyAggregates"/>) and the persistence
/// round-trips (table-storage-serialization rule — every property must survive Store→Map).
/// </summary>
public class DeviceJourneyAndFtrTests
{
private static readonly DateTime T0 = new(2026, 7, 20, 10, 0, 0, DateTimeKind.Utc);
private const string TenantA = "aaaaaaaa-0000-0000-0000-000000000001";
private const string TenantB = "bbbbbbbb-0000-0000-0000-000000000002";
private static SessionSummary Session(
string sessionId,
SessionStatus status = SessionStatus.Succeeded,
DateTime? startedAt = null,
DateTime? completedAt = null,
int? durationSeconds = 1800,
string? serial = "PF4X1ABC",
string tenantId = TenantA,
bool wg = false,
string enrollmentType = "v1",
string? adminMarkedAction = null)
=> new()
{
TenantId = tenantId,
SessionId = sessionId,
SerialNumber = serial!,
Status = status,
StartedAt = startedAt ?? T0,
CompletedAt = completedAt ?? (startedAt ?? T0).AddSeconds(durationSeconds ?? 0),
DurationSeconds = durationSeconds,
IsPreProvisioned = wg,
EnrollmentType = enrollmentType,
AdminMarkedAction = adminMarkedAction,
Manufacturer = "Contoso",
Model = "Laptop 5",
};
private static DeviceSessionRef Ref(
string sessionId, SessionStatus status, DateTime startedAt, DateTime? completedAt = null)
=> new()
{
SessionId = sessionId,
Status = status.ToString(),
StartedAt = startedAt,
CompletedAt = completedAt ?? startedAt.AddMinutes(30),
};
// ── serial normalization / junk exclusion ───────────────────────────────
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("abc")] // shorter than 4 chars
[InlineData("0")]
[InlineData("None")]
[InlineData("INVALID")]
[InlineData("Unknown")] // the agent's WMI-failure sentinel (audit Q5)
[InlineData("unknown")]
[InlineData("System Serial Number")]
[InlineData("TO BE FILLED BY O.E.M.")]
[InlineData("Default String")]
[InlineData(" Default string ")] // junk match happens after trim + case-fold
public void NormalizeSerial_JunkAndPlaceholders_YieldNull(string? raw)
{
Assert.Null(DeviceJourneyCalculator.NormalizeSerial(raw));
}
[Theory]
[InlineData("PF4X1ABC", "pf4x1abc")]
[InlineData(" PF4X1ABC ", "pf4x1abc")] // trim
[InlineData("pf4x1abc", "pf4x1abc")]
[InlineData("1234", "1234")] // exactly 4 chars is a valid identity
public void NormalizeSerial_RealSerials_TrimAndCaseFold(string raw, string expected)
{
Assert.Equal(expected, DeviceJourneyCalculator.NormalizeSerial(raw));
}
// ── terminal set / ref building ─────────────────────────────────────────
[Theory]
[InlineData(SessionStatus.Succeeded, true)]
[InlineData(SessionStatus.Failed, true)]
[InlineData(SessionStatus.Incomplete, true)]
[InlineData(SessionStatus.InProgress, false)]
[InlineData(SessionStatus.Pending, false)] // WG part 1 sealed — an OPEN session
[InlineData(SessionStatus.Stalled, false)]
[InlineData(SessionStatus.AwaitingUser, false)]
[InlineData(SessionStatus.Unknown, false)]
public void IsTerminal_MatchesTheF2TerminalSet(SessionStatus status, bool expected)
{
Assert.Equal(expected, DeviceJourneyCalculator.IsTerminal(status));
}
[Fact]
public void BuildSessionRef_MapsAllFields_DurationIsAuthoritativeDurationSeconds()
{
// CompletedAt deliberately diverges from StartedAt + DurationSeconds (25 % of terminal
// sessions in production) — the ref must carry DurationSeconds verbatim.
var session = Session("s-1", startedAt: T0, completedAt: T0.AddHours(5), durationSeconds: 1800,
wg: true, enrollmentType: "v1", adminMarkedAction: "Succeeded");
var reference = DeviceJourneyCalculator.BuildSessionRef(session)!;
Assert.Equal("s-1", reference.SessionId);
Assert.Equal(T0, reference.StartedAt);
Assert.Equal(T0.AddHours(5), reference.CompletedAt);
Assert.Equal("Succeeded", reference.Status);
Assert.Equal("v1", reference.EnrollmentType);
Assert.True(reference.IsPreProvisioned);
Assert.Equal(1800, reference.DurationSeconds);
Assert.True(reference.AdminMarked);
}
[Fact]
public void BuildSessionRef_WhiteGlovePending_YieldsNull_NeverAnAttempt()
{
var pending = Session("s-1", status: SessionStatus.Pending, wg: true, durationSeconds: null);
Assert.Null(DeviceJourneyCalculator.BuildSessionRef(pending));
}
[Fact]
public void BuildSessionRef_Incomplete_KeepsNullDuration()
{
var incomplete = Session("s-1", status: SessionStatus.Incomplete, durationSeconds: null);
var reference = DeviceJourneyCalculator.BuildSessionRef(incomplete)!;
Assert.Equal("Incomplete", reference.Status);
Assert.Null(reference.DurationSeconds);
Assert.False(reference.AdminMarked);
}
// ── chain maintenance ───────────────────────────────────────────────────
[Fact]
public void MergeChain_UpsertsBySessionId_ReclassificationReplacesTheEntry()
{
var existing = new List<DeviceSessionRef> { Ref("s-1", SessionStatus.Incomplete, T0) };
var update = Ref("s-1", SessionStatus.Succeeded, T0);
var merged = DeviceJourneyCalculator.MergeChain(existing, new[] { update });
var entry = Assert.Single(merged);
Assert.Equal("Succeeded", entry.Status);
}
[Fact]
public void MergeChain_OrdersByStartedAt_TiebreakSessionId_AndCapsToMostRecent()
{
var refs = new List<DeviceSessionRef>();
for (var i = 0; i < 25; i++)
refs.Add(Ref($"s-{i:D2}", SessionStatus.Failed, T0.AddDays(i)));
// Same StartedAt as s-24 — the ordinal sessionId tiebreak must keep the order stable.
refs.Add(Ref("s-00-twin", SessionStatus.Failed, T0.AddDays(24)));
var merged = DeviceJourneyCalculator.MergeChain(null, refs);
Assert.Equal(20, merged.Count); // cap 20 most recent
Assert.Equal("s-06", merged[0].SessionId); // s-00..s-05 dropped (oldest)
Assert.Equal(T0.AddDays(24), merged[^1].StartedAt);
Assert.Equal("s-24", merged[^1].SessionId); // "s-00-twin" < "s-24" ordinal → sorts first
Assert.Equal("s-00-twin", merged[^2].SessionId);
}
[Fact]
public void MergeChain_DropsNonTerminalRefs_Defensively()
{
var corrupt = new List<DeviceSessionRef>
{
Ref("s-1", SessionStatus.Succeeded, T0),
new() { SessionId = "s-2", Status = "Pending", StartedAt = T0.AddDays(1) },
new() { SessionId = "s-3", Status = "garbage", StartedAt = T0.AddDays(2) },
};
var merged = DeviceJourneyCalculator.MergeChain(corrupt, Array.Empty<DeviceSessionRef>());
Assert.Equal("s-1", Assert.Single(merged).SessionId);
}
[Fact]
public void RemoveSessionRefs_DropsDeletedSessions_EmptyChainSignalsRowDeletion()
{
var chain = new List<DeviceSessionRef>
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Succeeded, T0.AddDays(1)),
};
var partial = DeviceJourneyCalculator.RemoveSessionRefs(
chain, new HashSet<string>(StringComparer.Ordinal) { "s-1" });
Assert.Equal("s-2", Assert.Single(partial).SessionId);
var empty = DeviceJourneyCalculator.RemoveSessionRefs(
chain, new HashSet<string>(StringComparer.Ordinal) { "s-1", "s-2" });
Assert.Empty(empty);
}
// ── journey grouping ────────────────────────────────────────────────────
[Fact]
public void GroupJourneys_SingleSuccess_IsOneCompletedFirstTimeRightJourney()
{
var journeys = DeviceJourneyCalculator.GroupJourneys(
new[] { Ref("s-1", SessionStatus.Succeeded, T0) });
var journey = Assert.Single(journeys);
Assert.True(journey.Completed);
Assert.Single(journey.Attempts);
Assert.Equal("s-1", journey.CompletingRef!.SessionId);
}
[Fact]
public void GroupJourneys_FailAndIncompleteAreAttempts_JourneyEndsWithFirstSuccess()
{
var journeys = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Incomplete, T0.AddHours(2)),
Ref("s-3", SessionStatus.Succeeded, T0.AddHours(4)),
});
var journey = Assert.Single(journeys);
Assert.True(journey.Completed);
Assert.Equal(3, journey.Attempts.Count); // Incomplete is a non-successful attempt
}
[Fact]
public void GroupJourneys_SessionAfterSuccess_StartsANewJourney_Redeployment()
{
var journeys = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Succeeded, T0),
Ref("s-2", SessionStatus.Failed, T0.AddDays(2)),
});
Assert.Equal(2, journeys.Count);
Assert.True(journeys[0].Completed);
Assert.False(journeys[1].Completed); // open — no failed verdict, just not done yet
Assert.Null(journeys[1].CompletingRef);
}
[Fact]
public void GroupJourneys_FailuresOnly_JourneyStaysOpen_NeverCompleted()
{
var journeys = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Failed, T0.AddDays(1)),
});
var journey = Assert.Single(journeys);
Assert.False(journey.Completed);
Assert.Equal(2, journey.Attempts.Count);
}
[Fact]
public void GroupJourneys_GapOver30Days_StartsANewJourney_AbandonedStaysUncompleted()
{
var journeys = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0, completedAt: T0.AddHours(1)),
Ref("s-2", SessionStatus.Failed, T0.AddHours(1).AddDays(31)),
Ref("s-3", SessionStatus.Succeeded, T0.AddHours(1).AddDays(31).AddHours(3)),
});
Assert.Equal(2, journeys.Count);
Assert.False(journeys[0].Completed); // shelved/abandoned — never counts for FTR
Assert.Single(journeys[0].Attempts);
Assert.True(journeys[1].Completed);
Assert.Equal(2, journeys[1].Attempts.Count);
}
[Fact]
public void GroupJourneys_GapBoundary_Exactly30DaysStaysOneJourney_StrictlyMoreSplits()
{
var prevEnd = T0.AddHours(1);
var exactly30 = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0, completedAt: prevEnd),
Ref("s-2", SessionStatus.Failed, prevEnd.AddDays(30)),
});
Assert.Single(exactly30); // spec: "> 30 days" — the boundary itself does not split
var over30 = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0, completedAt: prevEnd),
Ref("s-2", SessionStatus.Failed, prevEnd.AddDays(30).AddHours(1)),
});
Assert.Equal(2, over30.Count);
}
[Fact]
public void GroupJourneys_GapMeasuredFromCompletedAt_NotStartedAt()
{
// Previous attempt STARTED 40 days before the next one but went terminal only 10 days
// before it — the gap is 10 days, one journey. StartedAt-based measuring would split.
var journeys = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0.AddDays(-40), completedAt: T0.AddDays(-10)),
Ref("s-2", SessionStatus.Succeeded, T0),
});
var journey = Assert.Single(journeys);
Assert.Equal(2, journey.Attempts.Count);
}
[Fact]
public void GroupJourneys_WhiteGloveIsOneSessionRow_OneAttempt_NoStitching()
{
// WG part 1 + part 2 share one session row (second-sweep finding): a completed WG
// enrollment is exactly one ref and one first-time-right attempt.
var wg = Session("s-1", wg: true, durationSeconds: 5400, completedAt: T0.AddDays(6));
var reference = DeviceJourneyCalculator.BuildSessionRef(wg)!;
var journeys = DeviceJourneyCalculator.GroupJourneys(new[] { reference });
var journey = Assert.Single(journeys);
Assert.True(journey.Completed);
Assert.Single(journey.Attempts);
Assert.Equal(5400, journey.Attempts[0].DurationSeconds); // pause-free combined duration, verbatim
}
[Fact]
public void Derive_CurrentJourneyAttempts_IsTheLastJourneys_OpenOrCompleted()
{
var (journeyCount, currentAttempts) = DeviceJourneyCalculator.Derive(new[]
{
Ref("s-1", SessionStatus.Succeeded, T0),
Ref("s-2", SessionStatus.Failed, T0.AddDays(1)),
Ref("s-3", SessionStatus.Failed, T0.AddDays(2)),
});
Assert.Equal(2, journeyCount);
Assert.Equal(2, currentAttempts); // the open redeployment journey has 2 attempts so far
Assert.Equal((0, 0), DeviceJourneyCalculator.Derive(Array.Empty<DeviceSessionRef>()));
}
// ── history row assembly (display precedence) ───────────────────────────
[Fact]
public void BuildDeviceHistoryRow_DisplayFields_FollowTheNewestChainEntry()
{
var newest = Session("s-2", serial: " PF4X1ABC ", startedAt: T0.AddDays(1));
var chain = new List<DeviceSessionRef>
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Succeeded, T0.AddDays(1)),
};
var row = TableStorageService.BuildDeviceHistoryRow(TenantA, "pf4x1abc", chain, existing: null, newest);
Assert.Equal("PF4X1ABC", row.SerialNumber); // trimmed original casing
Assert.Equal("Contoso", row.Manufacturer);
Assert.Equal("Laptop 5", row.Model);
Assert.Equal(2, row.CurrentJourneyAttempts);
Assert.Equal(1, row.JourneyCount);
Assert.Equal(DeviceJourneyCalculator.CurrentVersion, row.JourneyVersion);
}
[Fact]
public void BuildDeviceHistoryRow_BackfillingAnOlderSession_KeepsExistingDisplayFields()
{
var older = Session("s-1", serial: "pf4x1abc", startedAt: T0);
older.Model = "Laptop 4"; // stale model from the older enrollment
var chain = new List<DeviceSessionRef>
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Succeeded, T0.AddDays(1)),
};
var existing = new DeviceHistory
{
TenantId = TenantA, SerialKey = "pf4x1abc",
SerialNumber = "PF4X1ABC", Manufacturer = "Contoso", Model = "Laptop 5",
};
var row = TableStorageService.BuildDeviceHistoryRow(TenantA, "pf4x1abc", chain, existing, older);
Assert.Equal("PF4X1ABC", row.SerialNumber); // not regressed to the older session's data
Assert.Equal("Laptop 5", row.Model);
}
// ── persistence round-trips ─────────────────────────────────────────────
[Fact]
public void DeviceHistoryEntity_RoundTripsAllFields_IncludingChainJson()
{
var history = new DeviceHistory
{
TenantId = TenantA,
SerialKey = "pf4x1abc",
SerialNumber = "PF4X1ABC",
Manufacturer = "Contoso",
Model = "Laptop 5",
CurrentJourneyAttempts = 2,
JourneyCount = 3,
JourneyVersion = 1,
LastUpdated = T0,
Chain = new List<DeviceSessionRef>
{
new()
{
SessionId = "s-1", StartedAt = T0, CompletedAt = T0.AddHours(5),
Status = "Succeeded", EnrollmentType = "v2", IsPreProvisioned = true,
DurationSeconds = 1800, AdminMarked = true,
},
new()
{
SessionId = "s-2", StartedAt = T0.AddDays(1), CompletedAt = null,
Status = "Incomplete", EnrollmentType = "v1", DurationSeconds = null,
},
},
};
var mapped = TableStorageService.MapToDeviceHistory(
TableStorageService.BuildDeviceHistoryEntity(history));
Assert.Equal(TenantA, mapped.TenantId);
Assert.Equal("pf4x1abc", mapped.SerialKey);
Assert.Equal("PF4X1ABC", mapped.SerialNumber);
Assert.Equal("Contoso", mapped.Manufacturer);
Assert.Equal("Laptop 5", mapped.Model);
Assert.Equal(2, mapped.CurrentJourneyAttempts);
Assert.Equal(3, mapped.JourneyCount);
Assert.Equal(1, mapped.JourneyVersion);
Assert.Equal(T0, mapped.LastUpdated);
Assert.Equal(2, mapped.Chain.Count);
var first = mapped.Chain[0];
Assert.Equal("s-1", first.SessionId);
Assert.Equal(T0, first.StartedAt);
Assert.Equal(T0.AddHours(5), first.CompletedAt);
Assert.Equal("Succeeded", first.Status);
Assert.Equal("v2", first.EnrollmentType);
Assert.True(first.IsPreProvisioned);
Assert.Equal(1800, first.DurationSeconds);
Assert.True(first.AdminMarked);
var second = mapped.Chain[1];
Assert.Null(second.CompletedAt); // Incomplete: no terminal timestamp requirement
Assert.Null(second.DurationSeconds); // Incomplete: deliberately no duration — stays null
Assert.False(second.AdminMarked);
}
[Fact]
public void DeviceHistoryEntity_RowKeyEncoding_MakesForbiddenKeyCharsSafe_AndRoundTrips()
{
// '/', '\', '#', '?' are forbidden in Table Storage keys — a defensive encoding keeps a
// pathological serial writable, and the mapper restores the exact normalized serial.
var weird = "ab/cd\\ef#1?x";
var history = new DeviceHistory { TenantId = TenantA, SerialKey = weird };
var entity = TableStorageService.BuildDeviceHistoryEntity(history);
Assert.All(new[] { "/", "\\", "#", "?" }, c => Assert.DoesNotContain(c, entity.RowKey));
Assert.Equal(weird, TableStorageService.MapToDeviceHistory(entity).SerialKey);
// Ordinary serials stay human-readable.
Assert.Equal("pf4x1abc", TableStorageService.SerialRowKey("pf4x1abc"));
}
[Fact]
public void DeviceJourneyAggregateEntity_RoundTripsAllFields_RowKeyIsTheDate()
{
var aggregate = new DeviceJourneyDailyAggregate
{
TenantId = TenantA,
Date = "2026-07-20",
JourneyVersion = 1,
CompletedJourneyCount = 23,
FirstTimeRightCount = 19,
ExcludedSessionCount = 2,
ComputedAt = T0,
AttemptHistogram = new List<DeviceJourneyAttemptBucket>
{
new() { Attempts = 1, JourneyCount = 19 },
new() { Attempts = 2, JourneyCount = 3 },
new() { Attempts = 4, JourneyCount = 1 },
},
};
var entity = TableStorageService.BuildDeviceJourneyAggregateEntity(aggregate);
Assert.Equal("2026-07-20", entity.RowKey);
var mapped = TableStorageService.MapToDeviceJourneyAggregate(entity);
Assert.Equal(TenantA, mapped.TenantId);
Assert.Equal("2026-07-20", mapped.Date);
Assert.Equal(1, mapped.JourneyVersion);
Assert.Equal(23, mapped.CompletedJourneyCount);
Assert.Equal(19, mapped.FirstTimeRightCount);
Assert.Equal(2, mapped.ExcludedSessionCount);
Assert.Equal(T0, mapped.ComputedAt);
Assert.Equal(3, mapped.AttemptHistogram.Count);
Assert.Equal(1, mapped.AttemptHistogram[0].Attempts);
Assert.Equal(19, mapped.AttemptHistogram[0].JourneyCount);
Assert.Equal(4, mapped.AttemptHistogram[2].Attempts);
}
// ── daily FTR aggregation core ──────────────────────────────────────────
private static readonly DateTime WindowStart = T0.Date.AddDays(-30);
private static readonly DateTime WindowEnd = T0.Date.AddDays(1);
private static DeviceHistory History(string tenantId, params DeviceSessionRef[] chain)
=> new() { TenantId = tenantId, SerialKey = Guid.NewGuid().ToString("N"), Chain = chain.ToList() };
[Fact]
public void FtrAggregates_BucketByCompletingSessionsStartDate_AndMirrorGlobal()
{
var histories = new List<DeviceHistory>
{
// Tenant A device 1: fail → success (2 attempts, completes on T0 date).
History(TenantA,
Ref("a1-1", SessionStatus.Failed, T0.AddDays(-1)),
Ref("a1-2", SessionStatus.Succeeded, T0)),
// Tenant A device 2: first-time right on the same date.
History(TenantA, Ref("a2-1", SessionStatus.Succeeded, T0.AddHours(2))),
// Tenant B device: first-time right on a different date.
History(TenantB, Ref("b1-1", SessionStatus.Succeeded, T0.AddDays(-3))),
};
var rows = MaintenanceService.BuildDeviceJourneyAggregates(
histories, Array.Empty<SessionSummary>(), WindowStart, WindowEnd, T0);
// Tenant A (1 date) + Tenant B (1 date) + global (2 dates) = 4 rows.
Assert.Equal(4, rows.Count);
var tenantARow = rows.Single(r => r.TenantId == TenantA);
Assert.Equal(T0.ToString("yyyy-MM-dd"), tenantARow.Date);
Assert.Equal(2, tenantARow.CompletedJourneyCount);
Assert.Equal(1, tenantARow.FirstTimeRightCount);
Assert.Equal(DeviceJourneyCalculator.CurrentVersion, tenantARow.JourneyVersion);
Assert.Equal(2, tenantARow.AttemptHistogram.Count);
Assert.Equal(1, tenantARow.AttemptHistogram[0].Attempts);
Assert.Equal(1, tenantARow.AttemptHistogram[0].JourneyCount);
Assert.Equal(2, tenantARow.AttemptHistogram[1].Attempts);
var globalOnT0 = rows.Single(r => r.TenantId == "global" && r.Date == T0.ToString("yyyy-MM-dd"));
Assert.Equal(2, globalOnT0.CompletedJourneyCount); // tenant B's journey sits on its own date
}
[Fact]
public void FtrAggregates_OpenAndAbandonedJourneys_NeverCount()
{
var histories = new List<DeviceHistory>
{
// Open journey: failures only (e.g. a WG device still waiting for its user session
// has, at most, prior failed attempts in the chain — the Pending row itself never
// becomes a ref).
History(TenantA, Ref("a1-1", SessionStatus.Failed, T0)),
// Abandoned journey (31d gap, no success) followed by a completed journey.
History(TenantA,
Ref("a2-1", SessionStatus.Failed, T0.AddDays(-20), completedAt: T0.AddDays(-20).AddHours(1)),
Ref("a2-2", SessionStatus.Succeeded, T0.AddDays(11).AddHours(2))),
};
var rows = MaintenanceService.BuildDeviceJourneyAggregates(
histories, Array.Empty<SessionSummary>(), WindowStart, T0.Date.AddDays(12), T0);
// Only the completed journey produced rows: tenant + global on one date.
Assert.Equal(2, rows.Count);
var tenantRow = rows.Single(r => r.TenantId == TenantA);
Assert.Equal(1, tenantRow.CompletedJourneyCount);
Assert.Equal(1, tenantRow.FirstTimeRightCount); // the abandoned attempt belongs to a DIFFERENT journey
Assert.Equal(T0.AddDays(11).ToString("yyyy-MM-dd"), tenantRow.Date);
}
[Fact]
public void FtrAggregates_JourneysCompletedOutsideTheWindow_AreSkipped()
{
var histories = new List<DeviceHistory>
{
History(TenantA, Ref("a1-1", SessionStatus.Succeeded, WindowStart.AddDays(-1))),
};
var rows = MaintenanceService.BuildDeviceJourneyAggregates(
histories, Array.Empty<SessionSummary>(), WindowStart, WindowEnd, T0);
Assert.Empty(rows);
}
[Fact]
public void FtrAggregates_JunkSerialExclusions_ProduceDisclosureRows()
{
var excluded = new List<SessionSummary>
{
Session("x-1", serial: "Unknown", startedAt: T0),
Session("x-2", serial: "Unknown", startedAt: T0.AddHours(1)),
Session("x-3", serial: "Unknown", startedAt: WindowStart.AddDays(-2)), // outside window — ignored
};
var rows = MaintenanceService.BuildDeviceJourneyAggregates(
Array.Empty<DeviceHistory>(), excluded, WindowStart, WindowEnd, T0);
Assert.Equal(2, rows.Count); // tenant + global disclosure rows, no journey claim
var tenantRow = rows.Single(r => r.TenantId == TenantA);
Assert.Equal(0, tenantRow.CompletedJourneyCount);
Assert.Equal(0, tenantRow.FirstTimeRightCount);
Assert.Empty(tenantRow.AttemptHistogram);
Assert.Equal(2, tenantRow.ExcludedSessionCount);
Assert.Equal(2, rows.Single(r => r.TenantId == "global").ExcludedSessionCount);
}
[Fact]
public void FtrAggregates_TwoCompletedJourneysOfOneDevice_SameDay_BothCount()
{
// Redeploy on the same day: success closes journey 1, the next success closes journey 2.
var histories = new List<DeviceHistory>
{
History(TenantA,
Ref("a1-1", SessionStatus.Succeeded, T0),
Ref("a1-2", SessionStatus.Succeeded, T0.AddHours(3))),
};
var rows = MaintenanceService.BuildDeviceJourneyAggregates(
histories, Array.Empty<SessionSummary>(), WindowStart, WindowEnd, T0);
var tenantRow = rows.Single(r => r.TenantId == TenantA);
Assert.Equal(2, tenantRow.CompletedJourneyCount);
Assert.Equal(2, tenantRow.FirstTimeRightCount);
}
// ── attempt number (session banner, PR5) ────────────────────────────────
[Fact]
public void ComputeAttemptNumber_TerminalSessionInChain_ReturnsPositionWithinItsJourney()
{
var chain = new[]
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Succeeded, T0.AddHours(2)),
Ref("s-3", SessionStatus.Failed, T0.AddDays(5)), // redeployment → new journey
};
Assert.Equal(1, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-1", T0));
Assert.Equal(2, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-2", T0.AddHours(2)));
Assert.Equal(1, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-3", T0.AddDays(5)));
}
[Fact]
public void ComputeAttemptNumber_LiveSession_ContinuesTheOpenJourney()
{
var chain = new[] { Ref("s-1", SessionStatus.Failed, T0) };
// s-2 is still InProgress — not a chain ref; the virtual-attempt rule places it.
Assert.Equal(2, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-2", T0.AddDays(1)));
}
[Fact]
public void ComputeAttemptNumber_LiveSession_AfterCompletedJourney_IsAttemptOne()
{
var chain = new[] { Ref("s-1", SessionStatus.Succeeded, T0) };
Assert.Equal(1, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-2", T0.AddDays(2)));
}
[Fact]
public void ComputeAttemptNumber_LiveSession_AfterThirtyDayGap_IsAttemptOne()
{
var prevEnd = T0.AddHours(1);
var chain = new[] { Ref("s-1", SessionStatus.Failed, T0, completedAt: prevEnd) };
Assert.Equal(1, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-2", prevEnd.AddDays(31)));
// Inside the gap boundary the open journey continues instead.
Assert.Equal(2, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-3", prevEnd.AddDays(29)));
}
[Fact]
public void ComputeAttemptNumber_EmptyChain_YieldsNull_NoGuessedPosition()
{
Assert.Null(DeviceJourneyCalculator.ComputeAttemptNumber(
Array.Empty<DeviceSessionRef>(), "s-1", T0));
}
// ── fleet response helpers (PR5) ────────────────────────────────────────
[Fact]
public void SumAggregates_SumsAdditiveCounts_AndMergesHistograms()
{
var daily = new List<DeviceJourneyDailyAggregate>
{
new()
{
CompletedJourneyCount = 2, FirstTimeRightCount = 1, ExcludedSessionCount = 1,
AttemptHistogram = new List<DeviceJourneyAttemptBucket>
{
new() { Attempts = 1, JourneyCount = 1 },
new() { Attempts = 2, JourneyCount = 1 },
},
},
new()
{
CompletedJourneyCount = 1, FirstTimeRightCount = 1, ExcludedSessionCount = 0,
AttemptHistogram = new List<DeviceJourneyAttemptBucket> { new() { Attempts = 1, JourneyCount = 1 } },
},
};
var totals = DeviceJourneyMetricsResponse.SumAggregates(daily);
Assert.Equal(3, totals.CompletedJourneys);
Assert.Equal(2, totals.FirstTimeRight);
Assert.Equal(66.7, totals.FtrRatePct);
Assert.Equal(1, totals.ExcludedSessions);
Assert.Equal(2, totals.AttemptHistogram.Count);
Assert.Equal(2, totals.AttemptHistogram.Single(b => b.Attempts == 1).JourneyCount);
Assert.Equal(1, totals.AttemptHistogram.Single(b => b.Attempts == 2).JourneyCount);
}
[Fact]
public void SumAggregates_NoCompletedJourneys_RateIsNull_NeverZero()
{
var totals = DeviceJourneyMetricsResponse.SumAggregates(new List<DeviceJourneyDailyAggregate>
{
new() { CompletedJourneyCount = 0, FirstTimeRightCount = 0, ExcludedSessionCount = 3 },
});
Assert.Equal(0, totals.CompletedJourneys);
Assert.Null(totals.FtrRatePct);
Assert.Equal(3, totals.ExcludedSessions);
}
[Theory]
[InlineData(null, 30)] // default window
[InlineData("garbage", 30)]
[InlineData("7", 7)]
[InlineData("0", 1)] // clamped low
[InlineData("999", 180)] // clamped to aggregate retention
public void ClampDays_DefaultsAndClamps(string? raw, int expected)
{
Assert.Equal(expected, DeviceJourneyMetricsResponse.ClampDays(raw));
}
[Fact]
public void InclusiveWindowStart_YieldsExactlyNCalendarDays()
{
// Codex review: both range ends are inclusive, so "today - days" returned N+1 keys —
// days=1 summed yesterday AND today.
var today = new DateTime(2026, 7, 27);
Assert.Equal(today, DeviceJourneyMetricsResponse.InclusiveWindowStart(today, 1));
Assert.Equal(today.AddDays(-29), DeviceJourneyMetricsResponse.InclusiveWindowStart(today, 30));
}
private static DeviceHistory RepeaterHistory(
string serial, int attempts, params DeviceSessionRef[] chain)
=> new()
{
TenantId = TenantA,
SerialKey = serial,
SerialNumber = serial.ToUpperInvariant(),
Chain = chain.ToList(),
CurrentJourneyAttempts = attempts,
JourneyCount = 1,
};
[Fact]
public void SelectRepeatDevices_FiltersGate_WindowsOnNewestRef_AndPicksNewestFailure()
{
var histories = new List<DeviceHistory>
{
// Single attempt — not a repeat device.
RepeaterHistory("clean-1", attempts: 1, Ref("c1", SessionStatus.Succeeded, T0)),
// Repeat device, active in window: newest failed ref carries the reason lookup.
RepeaterHistory("hot-1", attempts: 3,
Ref("h1", SessionStatus.Failed, T0.AddDays(-2)),
Ref("h2", SessionStatus.Failed, T0.AddDays(-1)),
Ref("h3", SessionStatus.Incomplete, T0)),
// Repeat device, but its pain is OLDER than the window — current lists only.
RepeaterHistory("old-1", attempts: 4,
Ref("o1", SessionStatus.Failed, T0.AddDays(-40))),
// Lower attempt count → sorts after hot-1.
RepeaterHistory("warm-1", attempts: 2,
Ref("w1", SessionStatus.Failed, T0.AddDays(-3)),
Ref("w2", SessionStatus.Succeeded, T0.AddDays(-2))),
};
var selected = DeviceJourneyMetricsResponse.SelectRepeatDevices(histories, T0.AddDays(-30));
Assert.Equal(2, selected.Count);
Assert.Equal("hot-1", selected[0].History.SerialKey); // attempts desc
Assert.Equal("h3", selected[0].Newest.SessionId);
Assert.Equal("h2", selected[0].NewestFailed!.SessionId); // last FAILED, not last ref
Assert.Equal("warm-1", selected[1].History.SerialKey);
}
[Fact]
public void SelectRepeatDevices_CapsAtTen_ByAttemptsThenRecency()
{
var histories = new List<DeviceHistory>();
for (var i = 0; i < 12; i++)
{
histories.Add(RepeaterHistory($"dev-{i:D2}", attempts: 2 + i,
Ref($"r-{i}", SessionStatus.Failed, T0.AddDays(-i))));
}
var selected = DeviceJourneyMetricsResponse.SelectRepeatDevices(histories, T0.AddDays(-30));
Assert.Equal(10, selected.Count);
Assert.Equal("dev-11", selected[0].History.SerialKey); // highest attempts first
Assert.DoesNotContain(selected, c => c.History.SerialKey == "dev-00");
Assert.DoesNotContain(selected, c => c.History.SerialKey == "dev-01");
}
}
@@ -154,6 +154,80 @@ public class DiagnosticsBlobCascadeDeleterTests
Assert.Equal(0, deleter.HostedDeleteCalls);
}
// ── REAL DeleteAsync, Hosted branch — prefix sweep + belt-and-braces name delete ──
// The RecordingDeleter above replaces DeleteAsync wholesale, so these run the REAL
// base implementation against a fake storage service to pin the sweep behaviour.
private const string SessionGuid = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa";
[Fact]
public async Task RealDeleteAsync_Hosted_SweepsSessionPrefix_AndDeletesManifestBlob()
{
var storage = new RecordingHostedStorage();
var deleter = RealDeleter(storage);
var manifest = ManifestWith(blobName: $"{TenantA}/{Filename}", destination: "Hosted");
manifest.SessionId = SessionGuid;
var outcome = await deleter.DeleteAsync(manifest);
Assert.Equal(DiagnosticsBlobDeleteOutcome.HostedDeleted, outcome);
Assert.Equal((TenantA, SessionGuid), storage.SweptSession);
Assert.Equal($"{TenantA}/{Filename}", storage.DeletedBlobPath);
}
[Fact]
public async Task RealDeleteAsync_Hosted_NonGuidSessionId_SkipsSweep_StillDeletesManifestBlob()
{
// Legacy manifests may carry a malformed/empty SessionId — the sweep must degrade
// to the single-blob delete, never throw into the cascade poison path.
var storage = new RecordingHostedStorage();
var deleter = RealDeleter(storage);
var manifest = ManifestWith(blobName: $"{TenantA}/{Filename}", destination: "Hosted");
manifest.SessionId = "not-a-guid";
var outcome = await deleter.DeleteAsync(manifest);
Assert.Equal(DiagnosticsBlobDeleteOutcome.HostedDeleted, outcome);
Assert.Null(storage.SweptSession);
Assert.Equal($"{TenantA}/{Filename}", storage.DeletedBlobPath);
}
private static DiagnosticsBlobCascadeDeleter RealDeleter(RecordingHostedStorage storage)
// TenantConfigurationService is only consulted on the CustomerSas branch; the
// Hosted-branch tests never reach it, so null keeps the harness Azure-free.
=> new(
storage,
tenantConfig: null!,
Microsoft.Extensions.Logging.Abstractions.NullLogger<DiagnosticsBlobCascadeDeleter>.Instance);
private sealed class RecordingHostedStorage : HostedDiagnosticsBlobService
{
public (string TenantId, string SessionId)? SweptSession { get; private set; }
public string? DeletedBlobPath { get; private set; }
public RecordingHostedStorage()
: base(
new Azure.Storage.Blobs.BlobServiceClient("UseDevelopmentStorage=true"),
Microsoft.Extensions.Logging.Abstractions.NullLogger<HostedDiagnosticsBlobService>.Instance,
usesManagedIdentity: false)
{
}
public override Task<int> DeleteBySessionPrefixAsync(
string tenantId, string sessionId, CancellationToken cancellationToken = default)
{
SweptSession = (tenantId, sessionId);
return Task.FromResult(1);
}
public override Task<bool> DeleteIfExistsAsync(
string blobPath, CancellationToken cancellationToken = default)
{
DeletedBlobPath = blobPath;
return Task.FromResult(true);
}
}
// ── BuildBlobUrl helper (CustomerSas URL construction) ─────────────────────────
[Fact]
@@ -160,9 +160,9 @@ public class EndpointPolicyCatalogCompletenessTests
Assert.NotNull(getConfig);
Assert.NotNull(putConfig);
// GET config read is admin-tier read (own-tenant Admin / GA / read-only GlobalReader);
// PUT config write stays Admin/GA only.
Assert.Equal(EndpointPolicy.TenantAdminOrGlobalReader, getConfig.Policy);
// GET config read is member-tier (Operators/Viewers get the redacted read-only Settings
// view; the handler redacts for every non-admin caller); PUT config write stays Admin/GA only.
Assert.Equal(EndpointPolicy.MemberRead, getConfig.Policy);
Assert.Equal(EndpointPolicy.TenantAdminOrGA, putConfig.Policy);
var getRules = EndpointAccessPolicyCatalog.FindPolicy("GET", "/api/rules/gather");
@@ -338,7 +338,7 @@ public class EndpointPolicyCatalogCompletenessTests
[InlineData("POST", "/api/agent/telemetry", true)] // DeviceOrBootstrapAuth
[InlineData("GET", "/api/bootstrap/validate/ABC123", true)] // PublicAnonymous (param route)
[InlineData("GET", "/api/sessions", false)] // MemberRead
[InlineData("GET", "/api/config/00000000-0000-0000-0000-000000000001", false)] // TenantAdminOrGlobalReader
[InlineData("GET", "/api/config/00000000-0000-0000-0000-000000000001", false)] // MemberRead
[InlineData("GET", "/api/global/sessions", false)] // GlobalReadOrAdmin
[InlineData("GET", "/api/nonexistent", false)] // unregistered → fail-closed
public void SkipsJwtValidation_MatchesPolicyTier(string method, string path, bool expectedExempt)
@@ -10,9 +10,10 @@ namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Covers the server-side Fleet Health aggregation in
/// <see cref="MetricsMath.BuildFleetHealthPayload"/>, the single source of truth shared by the
/// tenant (metrics/fleet-health) and global (global/metrics/fleet-health) functions. These assert
/// the semantics ported verbatim from the previous client-side aggregation — notably success rate
/// over ALL sessions and average duration over non-in-progress sessions (failures included).
/// tenant (metrics/fleet-health) and global (global/metrics/fleet-health) functions. Success and
/// failure rates follow the SLA convention — over finished enrollments (Succeeded + Failed) only,
/// so in-flight sessions never dilute them. Average duration stays over non-in-progress sessions
/// that carry a duration (failures included).
/// </summary>
public class FleetHealthPayloadTests
{
@@ -55,7 +56,7 @@ public class FleetHealthPayloadTests
}
[Fact]
public void SuccessRate_IsOverAllSessions_NotJustTerminal()
public void SuccessRate_IsOverFinishedSessions_InFlightAndIncompleteExcluded()
{
var sessions = new List<SessionSummary>
{
@@ -63,16 +64,34 @@ public class FleetHealthPayloadTests
S(SessionStatus.Succeeded),
S(SessionStatus.Failed),
S(SessionStatus.InProgress),
S(SessionStatus.Pending),
S(SessionStatus.Incomplete), // terminal but non-failure — stays out of the rate
};
var payload = MetricsMath.BuildFleetHealthPayload(sessions, days: 7);
Assert.Equal(4, payload.Stats.Total);
Assert.Equal(6, payload.Stats.Total);
Assert.Equal(2, payload.Stats.Succeeded);
Assert.Equal(1, payload.Stats.Failed);
Assert.Equal(1, payload.Stats.InProgress);
// 2 / 4 = 50% (over total), NOT 2 / 3 = 66.7% (over terminal).
Assert.Equal(50d, payload.Stats.SuccessRate);
Assert.Equal(1, payload.Stats.Incomplete);
// 2 / (2 + 1) = 66.7% (over finished), NOT 2 / 6 = 33.3% (over total).
Assert.Equal(66.7d, payload.Stats.SuccessRate);
}
[Fact]
public void SuccessRate_IsZero_WhenNothingFinishedYet()
{
var sessions = new List<SessionSummary>
{
S(SessionStatus.InProgress),
S(SessionStatus.Pending),
};
var payload = MetricsMath.BuildFleetHealthPayload(sessions, days: 7);
// No finished enrollments → no rate; clients render "—" off Succeeded + Failed == 0.
Assert.Equal(0d, payload.Stats.SuccessRate);
}
[Fact]
@@ -111,6 +130,27 @@ public class FleetHealthPayloadTests
Assert.DoesNotContain(payload.ModelHealth, m => m.Model == "Contoso Model1");
}
[Fact]
public void ModelHealth_CountsFailedSeparately_InFlightInTotalOnly()
{
var sessions = new List<SessionSummary>
{
S(SessionStatus.Succeeded, model: "X"),
S(SessionStatus.Failed, model: "X"),
S(SessionStatus.InProgress, model: "X"),
};
var payload = MetricsMath.BuildFleetHealthPayload(sessions, days: 7);
var m = Assert.Single(payload.ModelHealth);
// Total keeps the full device count; the client derives the rate from
// Succeeded / (Succeeded + Failed), so the in-flight session must land
// in Total but in neither outcome bucket.
Assert.Equal(3, m.Total);
Assert.Equal(1, m.Succeeded);
Assert.Equal(1, m.Failed);
}
[Fact]
public void ModelKey_FallsBackToUnknown_WhenManufacturerAndModelBlank()
{
@@ -147,10 +187,13 @@ public class FleetHealthPayloadTests
{
var sessions = new List<SessionSummary>
{
// Model A: 1 failed of 2 → 50%
// Model A: 1 failed / 2 finished → 50%; the 2 in-flight sessions widen Total
// but must not dilute the rate (over-total it would read 25%).
S(SessionStatus.Failed, model: "A"),
S(SessionStatus.Succeeded, model: "A"),
// Model B: 2 failed of 2 → 100%
S(SessionStatus.InProgress, model: "A"),
S(SessionStatus.Pending, model: "A"),
// Model B: 2 failed of 2 finished → 100%
S(SessionStatus.Failed, model: "B"),
S(SessionStatus.Failed, model: "B"),
// Model C: no failures → excluded entirely
@@ -164,35 +207,37 @@ public class FleetHealthPayloadTests
Assert.Equal(2, payload.TopFailingModels[0].Failed);
Assert.Equal(100, payload.TopFailingModels[0].FailureRate);
Assert.Equal("Contoso A", payload.TopFailingModels[1].Model);
Assert.Equal(4, payload.TopFailingModels[1].Total);
Assert.Equal(50, payload.TopFailingModels[1].FailureRate);
Assert.DoesNotContain(payload.TopFailingModels, m => m.Model == "Contoso C");
}
[Fact]
public void FailureReasons_TruncateLongReasons_AndTakeTopFive()
public void FailureReasons_GroupByPrefix_KeepFullestText_AndTakeTopFive()
{
var sessions = new List<SessionSummary>();
// 6 distinct reasons with descending frequency 6..1 → only top 5 kept.
for (int freq = 6; freq >= 1; freq--)
// 5 distinct short reasons with descending frequency 5..1.
for (int freq = 5; freq >= 1; freq--)
for (int i = 0; i < freq; i++)
sessions.Add(S(SessionStatus.Failed, failureReason: $"Reason {freq}"));
// A null reason folds into "Unknown".
sessions.Add(S(SessionStatus.Failed, failureReason: null));
// An over-long reason is collapsed to a 50-char prefix + ellipsis.
var longReason = new string('x', 60);
sessions.Add(S(SessionStatus.Failed, failureReason: longReason));
// Two variants that share the first 50 characters collapse into one group;
// the longest variant is kept for display so the UI can show the full text.
var prefix = new string('x', 50);
var shortVariant = prefix + " short";
var longVariant = prefix + " a much longer tail that must survive intact";
for (int i = 0; i < 6; i++)
sessions.Add(S(SessionStatus.Failed, failureReason: shortVariant));
sessions.Add(S(SessionStatus.Failed, failureReason: longVariant));
var payload = MetricsMath.BuildFleetHealthPayload(sessions, days: 30);
Assert.Equal(5, payload.FailureReasons.Count);
Assert.Equal("Reason 6", payload.FailureReasons[0].Reason);
Assert.Equal(6, payload.FailureReasons[0].Count);
// The collapsed long reason exists in the tally (50 chars + "...").
var collapsed = new string('x', 50) + "...";
Assert.True(
sessions.Count(s => s.FailureReason == longReason) == 1,
"sanity: exactly one long-reason session");
Assert.Equal(53, collapsed.Length);
// The grouped cluster (6 + 1 = 7) outranks every "Reason N".
Assert.Equal(7, payload.FailureReasons[0].Count);
// The full, longest text survives end-to-end — no 50-char truncation.
Assert.Equal(longVariant, payload.FailureReasons[0].Reason);
Assert.DoesNotContain("...", payload.FailureReasons[0].Reason);
}
[Fact]
@@ -50,6 +50,8 @@ public class GeoMetricsProjectionEquivalenceTests
["GeoCity"] = "Frankfurt",
["GeoLoc"] = "50.11,8.68",
["IsPreProvisioned"] = false,
["AvgApiLatencyMs"] = 217.4,
["ApiRequestCount"] = 270,
// Noise the geo aggregation never reads.
["SerialNumber"] = "SN-FULL",
["Manufacturer"] = "Contoso",
@@ -76,6 +78,8 @@ public class GeoMetricsProjectionEquivalenceTests
Assert.Equal(fromFull.GeoRegion, fromProjected.GeoRegion);
Assert.Equal(fromFull.GeoCity, fromProjected.GeoCity);
Assert.Equal(fromFull.GeoLoc, fromProjected.GeoLoc);
Assert.Equal(fromFull.AvgApiLatencyMs, fromProjected.AvgApiLatencyMs);
Assert.Equal(fromFull.ApiRequestCount, fromProjected.ApiRequestCount);
// The grouping key must be identical too — it is composed from the Geo* fields.
Assert.Equal(
@@ -0,0 +1,124 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Functions.Functions.Metrics;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the API-latency aggregation of
/// <see cref="GetGeographicMetricsFunction.ComputeGeographicMetrics"/> (2026-07-28): both the
/// per-location and the global figure are REQUEST-WEIGHTED averages —
/// Σ(sessionAvg·requests)/Σ(requests) — not averages of session averages, so a long session
/// with many uploads counts proportionally. Sessions without latency data (agents predating
/// <c>AvgApiLatencyMs</c>) drop out of the weighting entirely.
/// </summary>
public class GeographicMetricsApiLatencyTests
{
private static SessionSummary S(string country, string city, double? avgLatencyMs, int? requestCount)
=> new()
{
TenantId = "00000000-0000-0000-0000-000000000fe0",
SessionId = Guid.NewGuid().ToString(),
Status = SessionStatus.Succeeded,
StartedAt = DateTime.UtcNow.AddHours(-2),
GeoCountry = country,
GeoRegion = country,
GeoCity = city,
GeoLoc = "0,0",
AvgApiLatencyMs = avgLatencyMs,
ApiRequestCount = requestCount,
};
[Fact]
public void LocationLatency_IsRequestWeighted_NotAverageOfAverages()
{
var sessions = new List<SessionSummary>
{
S("ID", "Jakarta", 1000, 300),
S("ID", "Jakarta", 500, 100),
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "country");
var loc = Assert.Single(result.Locations);
// (1000·300 + 500·100) / 400 = 875 — an unweighted mean would say 750.
Assert.Equal(875d, loc.AvgApiLatencyMs);
Assert.Equal(2, loc.ApiLatencySessionCount);
}
[Fact]
public void SessionsWithoutLatencyData_DropOutOfWeighting_ButStayInSessionCount()
{
var sessions = new List<SessionSummary>
{
S("DE", "Frankfurt", 200, 100),
S("DE", "Frankfurt", avgLatencyMs: null, requestCount: null), // pre-feature agent
S("DE", "Frankfurt", avgLatencyMs: 0, requestCount: 50), // zero = no signal
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "country");
var loc = Assert.Single(result.Locations);
Assert.Equal(3, loc.SessionCount);
Assert.Equal(1, loc.ApiLatencySessionCount);
Assert.Equal(200d, loc.AvgApiLatencyMs);
}
[Fact]
public void NoLatencyDataAnywhere_YieldsZeroes_NotDivisionByZero()
{
var sessions = new List<SessionSummary> { S("DE", "Frankfurt", null, null) };
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "country");
var loc = Assert.Single(result.Locations);
Assert.Equal(0d, loc.AvgApiLatencyMs);
Assert.Equal(0, loc.ApiLatencySessionCount);
Assert.Equal(0d, loc.ApiLatencyVsGlobalPct);
Assert.Equal(0d, result.GlobalAverages.AvgApiLatencyMs);
}
[Fact]
public void GlobalLatency_WeightsAcrossLocations_AndVsGlobalPctCompares()
{
var sessions = new List<SessionSummary>
{
S("DE", "Frankfurt", 200, 100),
S("ID", "Jakarta", 800, 100),
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "country");
// Equal request weights → global = (200 + 800) / 2 = 500.
Assert.Equal(500d, result.GlobalAverages.AvgApiLatencyMs);
var de = result.Locations.Single(l => l.Country == "DE");
var id = result.Locations.Single(l => l.Country == "ID");
Assert.Equal(-60d, de.ApiLatencyVsGlobalPct); // 200 vs 500
Assert.Equal(60d, id.ApiLatencyVsGlobalPct); // 800 vs 500
}
[Fact]
public void LocationWithoutData_KeepsZeroPct_EvenWhenGlobalHasData()
{
var sessions = new List<SessionSummary>
{
S("DE", "Frankfurt", 200, 100),
S("FR", "Paris", null, null),
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "country");
var fr = result.Locations.Single(l => l.Country == "FR");
Assert.Equal(0d, fr.AvgApiLatencyMs);
Assert.Equal(0d, fr.ApiLatencyVsGlobalPct); // no data ≠ "0 ms fast"
}
}
@@ -0,0 +1,99 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Functions.Functions.Metrics;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the success-rate semantics of the geographic aggregation
/// (<see cref="GetGeographicMetricsFunction.ComputeGeographicMetrics"/>): the rate is an outcome
/// quota over finished enrollments (Succeeded + Failed) following the SLA convention. In-flight
/// sessions (InProgress/Pending/Stalled/AwaitingUser) and Incomplete (terminal, non-failure) sit
/// in SessionCount but never dilute the rate — the customer-visible symptom this pins against was
/// a site mid-rollout showing a poor rate despite zero actual failures.
/// </summary>
public class GeographicMetricsSuccessRateTests
{
private static SessionSummary S(SessionStatus status, string city = "Frankfurt")
=> new()
{
TenantId = "00000000-0000-0000-0000-000000000fe0",
SessionId = Guid.NewGuid().ToString(),
Status = status,
StartedAt = DateTime.UtcNow.AddHours(-2),
GeoCountry = "DE",
GeoRegion = "Hessen",
GeoCity = city,
GeoLoc = "50.11,8.68",
};
[Fact]
public void SuccessRate_IsOverFinishedSessions_InFlightAndIncompleteExcluded()
{
var sessions = new List<SessionSummary>
{
S(SessionStatus.Succeeded),
S(SessionStatus.Succeeded),
S(SessionStatus.Succeeded),
S(SessionStatus.Failed),
S(SessionStatus.InProgress),
S(SessionStatus.Pending),
S(SessionStatus.AwaitingUser),
S(SessionStatus.Incomplete), // terminal but non-failure — stays out of the rate
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "city");
var loc = Assert.Single(result.Locations);
Assert.Equal(8, loc.SessionCount);
Assert.Equal(3, loc.Succeeded);
Assert.Equal(1, loc.Failed);
// 3 / (3 + 1) = 75% (over finished), NOT 3 / 8 = 37.5% (over all sessions).
Assert.Equal(75d, loc.SuccessRate);
}
[Fact]
public void SuccessRate_IsZero_WhenNothingFinishedYet()
{
var sessions = new List<SessionSummary>
{
S(SessionStatus.InProgress),
S(SessionStatus.Pending),
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "city");
var loc = Assert.Single(result.Locations);
// No finished enrollments → no rate; the frontend renders "—" off Succeeded + Failed == 0.
Assert.Equal(0d, loc.SuccessRate);
Assert.Equal(0, loc.Succeeded + loc.Failed);
Assert.Equal(2, loc.SessionCount);
}
[Fact]
public void SuccessRate_IsPerLocation_NotGlobal()
{
var sessions = new List<SessionSummary>
{
// Frankfurt: 1 / 1 finished → 100% despite the in-flight session.
S(SessionStatus.Succeeded, city: "Frankfurt"),
S(SessionStatus.InProgress, city: "Frankfurt"),
// Berlin: 1 / 2 finished → 50%.
S(SessionStatus.Succeeded, city: "Berlin"),
S(SessionStatus.Failed, city: "Berlin"),
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "city");
var frankfurt = result.Locations.Single(l => l.City == "Frankfurt");
var berlin = result.Locations.Single(l => l.City == "Berlin");
Assert.Equal(100d, frankfurt.SuccessRate);
Assert.Equal(50d, berlin.SuccessRate);
}
}
@@ -0,0 +1,189 @@
using System.Text.Json;
using AutopilotMonitor.Functions.Functions.Metrics;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Guards the cross-tenant redaction on GET metrics/ime-versions.
///
/// The route is <c>MemberRead</c> (EndpointAccessPolicyCatalog) and the archive itself is a
/// single global partition — every tenant member reads the SAME platform-wide rows. The only
/// thing standing between an ordinary tenant user and another tenant's identifiers is the
/// projection in <see cref="GetImeVersionHistoryFunction.BuildResponsePayload"/>. That made it
/// worth pinning: the redaction had no test at all, and a boundary nobody asserts is one a
/// later refactor can open without a single failure.
///
/// The JSON tests are the ones that matter — a property dropped from an anonymous type is only
/// a real redaction once it is also absent from the wire, and the wire is what the customer's
/// MCP client receives. camelCase mirrors Program.cs's PropertyNamingPolicy.
/// </summary>
public class GetImeVersionHistoryRedactionTests
{
private static readonly DateTime First = new(2026, 7, 8, 3, 18, 13, DateTimeKind.Utc);
private static readonly DateTime Last = new(2026, 7, 27, 16, 59, 34, DateTimeKind.Utc);
private static readonly JsonSerializerOptions CamelCase =
new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase };
// =========================================================================
// Non-global caller (ordinary tenant member) — the redaction
// =========================================================================
[Fact]
public void NonGlobalCaller_DropsForeignTenantAndSessionIds()
{
var json = SerializeFor(hasGlobalScope: false, MakeEntry());
Assert.DoesNotContain("firstSeenTenantId", json, StringComparison.OrdinalIgnoreCase);
Assert.DoesNotContain("firstSeenSessionId", json, StringComparison.OrdinalIgnoreCase);
}
/// <summary>
/// The values, not just the property names: a rename of the model property must not let the
/// identifier survive under a different key. Asserting on the GUIDs catches that.
/// </summary>
[Fact]
public void NonGlobalCaller_ForeignIdentifierValuesAppearNowhereInThePayload()
{
var json = SerializeFor(hasGlobalScope: false, MakeEntry());
Assert.DoesNotContain("57f34dd1-6a42-47f9-9bc0-b921fa6caa30", json, StringComparison.OrdinalIgnoreCase);
Assert.DoesNotContain("baae0453-6d19-47b5-aa58-8bed5a572fc8", json, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public void NonGlobalCaller_ExposesExactlyTheFourAllowedProperties()
{
using var doc = JsonDocument.Parse(SerializeFor(hasGlobalScope: false, MakeEntry()));
var keys = doc.RootElement[0].EnumerateObject().Select(p => p.Name).OrderBy(n => n).ToArray();
Assert.Equal(new[] { "firstSeenAt", "lastSeenAt", "sessionCount", "version" }, keys);
}
/// <summary>
/// SessionCount stays on purpose — it is a platform-wide total, and the route exists so any
/// tenant member can track Microsoft's IME rollout. Pinned so that "the counts are global"
/// stays a decision someone made rather than something that quietly drifts either way.
/// </summary>
[Fact]
public void NonGlobalCaller_KeepsVersionDatesAndGlobalSessionCount()
{
using var doc = JsonDocument.Parse(SerializeFor(hasGlobalScope: false, MakeEntry()));
var row = doc.RootElement[0];
Assert.Equal("1.103.101.0", row.GetProperty("version").GetString());
Assert.Equal(First, row.GetProperty("firstSeenAt").GetDateTime());
Assert.Equal(Last, row.GetProperty("lastSeenAt").GetDateTime());
Assert.Equal(3109, row.GetProperty("sessionCount").GetInt32());
}
// =========================================================================
// Global caller (Global Admin or Global Reader) — full archive
// =========================================================================
[Fact]
public void GlobalCaller_KeepsTenantAndSessionIds()
{
using var doc = JsonDocument.Parse(SerializeFor(hasGlobalScope: true, MakeEntry()));
var row = doc.RootElement[0];
Assert.Equal("57f34dd1-6a42-47f9-9bc0-b921fa6caa30", row.GetProperty("firstSeenTenantId").GetString());
Assert.Equal("baae0453-6d19-47b5-aa58-8bed5a572fc8", row.GetProperty("firstSeenSessionId").GetString());
}
[Fact]
public void GlobalCaller_ExposesTheFullEntry()
{
using var doc = JsonDocument.Parse(SerializeFor(hasGlobalScope: true, MakeEntry()));
var keys = doc.RootElement[0].EnumerateObject().Select(p => p.Name).OrderBy(n => n).ToArray();
Assert.Equal(
new[] { "firstSeenAt", "firstSeenSessionId", "firstSeenTenantId", "lastSeenAt", "sessionCount", "version" },
keys);
}
// =========================================================================
// Shape-preserving properties of the projection
// =========================================================================
[Fact]
public void BothScopes_EmitAJsonArray_EvenForASingleRow()
{
foreach (var scope in new[] { true, false })
{
using var doc = JsonDocument.Parse(SerializeFor(scope, MakeEntry()));
Assert.Equal(JsonValueKind.Array, doc.RootElement.ValueKind);
Assert.Equal(1, doc.RootElement.GetArrayLength());
}
}
[Fact]
public void BothScopes_EmptyArchive_SerializesAsEmptyArray()
{
foreach (var scope in new[] { true, false })
{
using var doc = JsonDocument.Parse(SerializeFor(scope));
Assert.Equal(JsonValueKind.Array, doc.RootElement.ValueKind);
Assert.Equal(0, doc.RootElement.GetArrayLength());
}
}
/// <summary>
/// The repository orders by FirstSeenAt descending; the projection must not reshuffle, or a
/// redacted caller would see a different sequence than a Global Admin for the same archive.
/// </summary>
[Fact]
public void NonGlobalCaller_PreservesRowOrderAndCount()
{
var newest = MakeEntry("1.103.101.0", First);
var older = MakeEntry("1.101.111.0", First.AddDays(-48));
var oldest = MakeEntry("1.99.101.0", First.AddDays(-131));
using var doc = JsonDocument.Parse(SerializeFor(false, newest, older, oldest));
Assert.Equal(3, doc.RootElement.GetArrayLength());
Assert.Equal("1.103.101.0", doc.RootElement[0].GetProperty("version").GetString());
Assert.Equal("1.101.111.0", doc.RootElement[1].GetProperty("version").GetString());
Assert.Equal("1.99.101.0", doc.RootElement[2].GetProperty("version").GetString());
}
/// <summary>
/// BuildResponsePayload returns `object`; the branch that keeps everything hands back the
/// entries themselves. System.Text.Json resolves an `object` declared type via the runtime
/// type, which is what keeps the wire format identical to the pre-extraction inline code —
/// assert it rather than trust it, since a regression here would silently emit "{}" rows.
/// </summary>
[Fact]
public void GlobalPayload_SerializedAsObject_StillEmitsPopulatedRows()
{
var payload = GetImeVersionHistoryFunction.BuildResponsePayload([MakeEntry()], hasGlobalScope: true);
var json = JsonSerializer.Serialize(payload, CamelCase);
Assert.Contains("1.103.101.0", json, StringComparison.Ordinal);
Assert.NotEqual("[{}]", json);
}
// =========================================================================
// Helpers
// =========================================================================
private static string SerializeFor(bool hasGlobalScope, params ImeVersionHistoryEntry[] versions) =>
JsonSerializer.Serialize(
GetImeVersionHistoryFunction.BuildResponsePayload(versions, hasGlobalScope),
CamelCase);
private static ImeVersionHistoryEntry MakeEntry(string version = "1.103.101.0", DateTime? firstSeenAt = null) =>
new()
{
Version = version,
FirstSeenAt = firstSeenAt ?? First,
FirstSeenSessionId = "baae0453-6d19-47b5-aa58-8bed5a572fc8",
FirstSeenTenantId = "57f34dd1-6a42-47f9-9bc0-b921fa6caa30",
LastSeenAt = Last,
SessionCount = 3109,
};
}
@@ -35,6 +35,7 @@ public class GetTenantFeatureFlagsPayloadTests
Assert.Equal(new[]
{
"bootstrapTokenEnabled",
"diagnosticsUploadConfigured",
"edition",
"enableIntegrityBypassAnalyzer",
"enableSoftwareInventoryAnalyzer",
@@ -114,6 +115,35 @@ public class GetTenantFeatureFlagsPayloadTests
Assert.DoesNotContain("webhookUrl", json, StringComparison.OrdinalIgnoreCase);
}
// ── diagnosticsUploadConfigured (drives the session-detail "Collect Logs" button) ──
[Theory]
// mode Off (or unset) → never configured, regardless of destination
[InlineData(null, null, null, false)]
[InlineData("Off", "https://x.blob.core.windows.net/c?sv=1", "CustomerSas", false)]
[InlineData("Off", null, "Hosted", false)]
// mode active + Hosted destination → configured without any SAS
[InlineData("OnFailure", null, "Hosted", true)]
[InlineData("Always", null, "Hosted", true)]
// mode active + CustomerSas → configured only when a SAS URL is present
[InlineData("OnFailure", "https://x.blob.core.windows.net/c?sv=1", "CustomerSas", true)]
[InlineData("OnFailure", null, "CustomerSas", false)]
[InlineData("Always", "", null, false)]
public void Payload_DiagnosticsUploadConfigured_ModeAndDestinationMatrix(
string? mode, string? sasUrl, string? destination, bool expected)
{
// Null-forgiving: the model declares these non-nullable with defaults, but table
// round-trips can produce null at runtime — the payload must tolerate that shape.
var element = Serialize(new TenantConfiguration
{
DiagnosticsUploadMode = mode!,
DiagnosticsBlobSasUrl = sasUrl!,
DiagnosticsUploadDestination = destination!,
});
Assert.Equal(expected, element.GetProperty("diagnosticsUploadConfigured").GetBoolean());
}
// ── Edition / trial surface ─────────────────────────────────────────────
[Fact]
@@ -0,0 +1,194 @@
using AutopilotMonitor.Functions.Functions.Reports;
using AutopilotMonitor.Shared.DataAccess;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Tests for the TPM-PSS-unsupported aggregation logic extracted from
/// GetTpmPssUnsupportedFunction. All tests target the pure static BuildAggregatedResult
/// method — no HTTP or DI needed. Aggregation is by serial number (the TPM incompatibility
/// is a per-device condition, not a per-model one — firmware levels differ within a model).
/// </summary>
public class GetTpmPssUnsupportedFunctionTests
{
private static readonly DateTime T0 = new(2026, 3, 1, 12, 0, 0, DateTimeKind.Utc);
// =========================================================================
// Empty / no matching reports
// =========================================================================
[Fact]
public void BuildAggregatedResult_EmptyList_ReturnsEmpty()
{
var (aggregated, totalRawReports) = GetTpmPssUnsupportedFunction.BuildAggregatedResult([]);
Assert.Empty(aggregated);
Assert.Equal(0, totalRawReports);
}
[Fact]
public void BuildAggregatedResult_FiltersTpmPssUnsupportedOnly()
{
var reports = new List<DistressReportEntry>
{
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "SN1", T0),
MakeEntry("DeviceNotRegistered", "Lenovo", "ThinkPad T14", "SN2", T0.AddMinutes(1)),
MakeEntry("AuthCertificateRejected", "HP", "EliteBook", "SN3", T0.AddMinutes(2)),
};
var (aggregated, totalRawReports) = GetTpmPssUnsupportedFunction.BuildAggregatedResult(reports);
Assert.Single(aggregated);
Assert.Equal(1, totalRawReports);
}
// =========================================================================
// Aggregation by serial number
// =========================================================================
[Fact]
public void BuildAggregatedResult_AggregatesBySerialCaseInsensitive()
{
var reports = new List<DistressReportEntry>
{
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "S4SQ8685", T0),
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "s4sq8685", T0.AddMinutes(1)),
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", " S4SQ8685 ", T0.AddMinutes(2)),
};
var (aggregated, totalRawReports) = GetTpmPssUnsupportedFunction.BuildAggregatedResult(reports);
Assert.Single(aggregated);
Assert.Equal(3, totalRawReports);
var item = ToDynamic(aggregated[0]);
Assert.Equal(3, (int)item.attemptCount);
}
[Fact]
public void BuildAggregatedResult_DistinctSerials_ProduceDistinctRows()
{
var reports = new List<DistressReportEntry>
{
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "SN1", T0),
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "SN2", T0.AddMinutes(1)),
MakeEntry("TpmPssUnsupported", "Microsoft", "Surface Book", "SN3", T0.AddMinutes(2)),
};
var (aggregated, totalRawReports) = GetTpmPssUnsupportedFunction.BuildAggregatedResult(reports);
Assert.Equal(3, aggregated.Count);
Assert.Equal(3, totalRawReports);
}
[Fact]
public void BuildAggregatedResult_UsesMostRecentManufacturerModel()
{
var reports = new List<DistressReportEntry>
{
MakeEntry("TpmPssUnsupported", "OldMfr", "OldModel", "SN1", T0),
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "SN1", T0.AddMinutes(5)),
};
var (aggregated, _) = GetTpmPssUnsupportedFunction.BuildAggregatedResult(reports);
var item = ToDynamic(aggregated[0]);
Assert.Equal("Lenovo", (string)item.manufacturer);
Assert.Equal("ThinkCentre M720q", (string)item.model);
Assert.Equal("SN1", (string)item.serialNumber);
}
// =========================================================================
// Timestamps
// =========================================================================
[Fact]
public void BuildAggregatedResult_FirstSeenLastSeen_Correct()
{
var t1 = T0;
var t2 = T0.AddHours(1);
var t3 = T0.AddHours(2);
var reports = new List<DistressReportEntry>
{
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "SN1", t2),
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "SN1", t1),
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "SN1", t3),
};
var (aggregated, _) = GetTpmPssUnsupportedFunction.BuildAggregatedResult(reports);
var item = ToDynamic(aggregated[0]);
Assert.Equal(t1, (DateTime)item.firstSeen);
Assert.Equal(t3, (DateTime)item.lastSeen);
}
[Fact]
public void BuildAggregatedResult_OrderedByLastSeenDescending()
{
var reports = new List<DistressReportEntry>
{
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "SN1", T0),
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkPad X13", "SN2", T0.AddHours(2)),
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720s", "SN3", T0.AddHours(1)),
};
var (aggregated, _) = GetTpmPssUnsupportedFunction.BuildAggregatedResult(reports);
Assert.Equal(3, aggregated.Count);
var first = ToDynamic(aggregated[0]);
var second = ToDynamic(aggregated[1]);
var third = ToDynamic(aggregated[2]);
Assert.True((DateTime)first.lastSeen > (DateTime)second.lastSeen);
Assert.True((DateTime)second.lastSeen > (DateTime)third.lastSeen);
}
// =========================================================================
// Null / empty serial number
// =========================================================================
[Fact]
public void BuildAggregatedResult_NullOrEmptySerials_GroupedIntoSingleBucket()
{
var reports = new List<DistressReportEntry>
{
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", null, T0),
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", "", T0.AddMinutes(1)),
MakeEntry("TpmPssUnsupported", "Lenovo", "ThinkCentre M720q", " ", T0.AddMinutes(2)),
};
var (aggregated, _) = GetTpmPssUnsupportedFunction.BuildAggregatedResult(reports);
Assert.Single(aggregated);
var item = ToDynamic(aggregated[0]);
Assert.Equal(3, (int)item.attemptCount);
}
// =========================================================================
// Helpers
// =========================================================================
private static DistressReportEntry MakeEntry(
string errorType, string? manufacturer, string? model, string? serial, DateTime ingestedAt) =>
new()
{
TenantId = "test-tenant",
ErrorType = errorType,
Manufacturer = manufacturer,
Model = model,
SerialNumber = serial,
IngestedAt = ingestedAt,
AgentTimestamp = ingestedAt,
};
private static dynamic ToDynamic(object obj)
{
var type = obj.GetType();
var dict = new System.Dynamic.ExpandoObject() as IDictionary<string, object?>;
foreach (var prop in type.GetProperties())
{
dict[prop.Name] = prop.GetValue(obj);
}
return dict;
}
}
@@ -0,0 +1,360 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.RegularExpressions;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Services;
using Azure;
using Azure.Data.Tables;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Behavioural coverage for <see cref="TableHardwareRejectionNotificationTracker"/> — the dedup
/// store behind the two distress-driven bell notifications (hardware rejection per model,
/// TPM-PSS incompatibility per serial). <see cref="HardwareRejectionNotificationTrackerKeyTests"/>
/// pins the RowKey shape; this file pins what the tracker DOES with those keys.
///
/// CORRECTNESS GUARD: the "one bell ever" promise rests entirely on the 409 branch of
/// <c>AddEntityAsync</c> — a rewrite to Upsert or a get-then-add would silently start ringing
/// the bell on every distress report (a device with an incompatible TPM retries forever, so
/// "every report" means a notification storm, not a second bell).
///
/// The retention sweep is covered here too because <c>DeleteOlderThanAsync</c> prunes BOTH key
/// spaces from the shared table: pruning a <c>tpmpss|</c> row re-arms that device's bell.
/// </summary>
public class HardwareRejectionNotificationTrackerDedupTests
{
private const string TenantId = "77777777-7777-7777-7777-777777777777";
private const string OtherTenantId = "88888888-8888-8888-8888-888888888888";
private const string Serial = "S4SQ8685";
// ── TPM PSS: first bell, then never again ──────────────────────────────────────
[Fact]
public async Task TpmPss_FirstReport_RegistersRowAndReturnsTrue()
{
var harness = new Harness();
var isFirst = await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial);
Assert.True(isFirst);
var entity = Assert.Single(harness.Store.Values);
Assert.Equal(TenantId.ToLowerInvariant(), entity.PartitionKey);
Assert.Equal("tpmpss|s4sq8685", entity.RowKey);
// Serialization guard: the row must carry the fields the cleanup sweep and any future
// operator query rely on — losing FirstNotifiedAt would make the row immortal.
Assert.Equal(TenantId, entity.GetString("TenantId"));
Assert.Equal(Serial, entity.GetString("SerialNumber"));
Assert.NotNull(entity.GetDateTime("FirstNotifiedAt"));
}
[Fact]
public async Task TpmPss_SecondReport_SameSerial_ReturnsFalse()
{
var harness = new Harness();
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
Assert.False(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
Assert.Single(harness.Store);
}
[Theory]
[InlineData("s4sq8685")]
[InlineData(" S4SQ8685 ")]
[InlineData("S4Sq8685")]
public async Task TpmPss_SecondReport_CasingOrWhitespaceVariant_StillDeduped(string variant)
{
var harness = new Harness();
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
Assert.False(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, variant));
}
[Fact]
public async Task TpmPss_DifferentSerial_OrDifferentTenant_RingsAgain()
{
var harness = new Harness();
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, "OTHER123"));
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(OtherTenantId, Serial));
Assert.Equal(3, harness.Store.Count);
}
[Theory]
[InlineData("", Serial)]
[InlineData(" ", Serial)]
[InlineData(TenantId, "")]
[InlineData(TenantId, " ")]
public async Task TpmPss_BlankInput_ReturnsFalse_WithoutTouchingTheTable(string tenantId, string serial)
{
var harness = new Harness();
Assert.False(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(tenantId, serial));
Assert.Empty(harness.Store);
harness.Table.Verify(
c => c.AddEntityAsync(It.IsAny<TableEntity>(), It.IsAny<CancellationToken>()),
Times.Never);
}
[Fact]
public async Task TpmPss_UnexpectedTableFailure_ReturnsFalse()
{
// Fail-soft in the SUPPRESSING direction: the write may or may not have landed, so the
// caller must not fire a bell it might fire again on the next report.
var harness = new Harness { AddFailure = new RequestFailedException(503, "ServerBusy") };
Assert.False(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
}
// ── Hardware rejection: same contract, disjoint key space ──────────────────────
[Fact]
public async Task HardwareRejection_SecondReport_SameModel_ReturnsFalse()
{
var harness = new Harness();
Assert.True(await harness.Sut.TryRegisterFirstNotificationAsync(TenantId, "Lenovo", "ThinkPad X1"));
Assert.False(await harness.Sut.TryRegisterFirstNotificationAsync(TenantId, "LENOVO", " thinkpad x1 "));
Assert.Single(harness.Store);
}
[Fact]
public async Task HardwareRejection_AndTpmPss_DoNotDedupEachOtherInTheSharedTable()
{
var harness = new Harness();
Assert.True(await harness.Sut.TryRegisterFirstNotificationAsync(TenantId, "Lenovo", Serial));
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
Assert.Equal(2, harness.Store.Count);
}
// ── F3 rule regression: one bell per episode, keyspaces stay independent ──────
[Fact]
public async Task RuleRegression_SecondRegister_SameRule_ReturnsFalse_OneBellPerEpisode()
{
var harness = new Harness();
var alert = new AutopilotMonitor.Shared.Models.RuleRegressionAlert
{
TenantId = TenantId, RuleId = "ANALYZE-NET-001",
FirstNotifiedAt = DateTime.UtcNow, LastEvaluatedAt = DateTime.UtcNow,
};
Assert.True(await harness.Sut.TryRegisterRuleRegressionAsync(TenantId, alert));
Assert.False(await harness.Sut.TryRegisterRuleRegressionAsync(TenantId, alert));
var entity = Assert.Single(harness.Store.Values);
Assert.Equal("ruleregression|analyze-net-001", entity.RowKey);
Assert.NotNull(entity.GetDateTime("FirstNotifiedAt")); // the retention sweep depends on it
}
[Fact]
public async Task RuleRegression_DeleteReArms_AndDoesNotTouchOtherKeyspaces()
{
var harness = new Harness();
var alert = new AutopilotMonitor.Shared.Models.RuleRegressionAlert
{
TenantId = TenantId, RuleId = "ANALYZE-NET-001",
FirstNotifiedAt = DateTime.UtcNow, LastEvaluatedAt = DateTime.UtcNow,
};
Assert.True(await harness.Sut.TryRegisterRuleRegressionAsync(TenantId, alert));
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
await harness.Sut.DeleteRuleRegressionAsync(TenantId, "ANALYZE-NET-001");
// The episode re-arms; the unrelated tpmpss row survives.
Assert.Equal("tpmpss|s4sq8685", Assert.Single(harness.Store.Values).RowKey);
Assert.True(await harness.Sut.TryRegisterRuleRegressionAsync(TenantId, alert));
}
// ── Retention sweep ───────────────────────────────────────────────────────────
[Fact]
public async Task DeleteOlderThanAsync_PrunesRuleRegressionRows_ReArmingLongBurningEpisodes()
{
var harness = new Harness();
Assert.True(await harness.Sut.TryRegisterRuleRegressionAsync(TenantId,
new AutopilotMonitor.Shared.Models.RuleRegressionAlert
{
TenantId = TenantId, RuleId = "ANALYZE-NET-001",
FirstNotifiedAt = DateTime.UtcNow, LastEvaluatedAt = DateTime.UtcNow,
}));
harness.AgeRows(TimeSpan.FromDays(31));
// Spec §F3: retention cleanup re-arms — a month-old still-burning regression rings again.
Assert.Equal(1, await harness.Sut.DeleteOlderThanAsync(DateTime.UtcNow.AddDays(-30)));
Assert.Empty(harness.Store);
}
[Fact]
public async Task DeleteOlderThanAsync_PrunesTpmPssRows_AndReArmsThatDevicesBell()
{
var harness = new Harness();
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
harness.AgeRows(TimeSpan.FromDays(31));
var deleted = await harness.Sut.DeleteOlderThanAsync(DateTime.UtcNow.AddDays(-30));
Assert.Equal(1, deleted);
Assert.Empty(harness.Store);
// A device whose TPM was never fixed keeps reporting; after the retention window it is
// worth one more bell rather than permanent silence.
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
}
[Fact]
public async Task DeleteOlderThanAsync_KeepsRowsInsideTheWindow_DedupSurvives()
{
var harness = new Harness();
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
Assert.True(await harness.Sut.TryRegisterFirstNotificationAsync(TenantId, "Lenovo", "ThinkPad X1"));
var deleted = await harness.Sut.DeleteOlderThanAsync(DateTime.UtcNow.AddDays(-30));
Assert.Equal(0, deleted);
Assert.Equal(2, harness.Store.Count);
Assert.False(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
Assert.False(await harness.Sut.TryRegisterFirstNotificationAsync(TenantId, "Lenovo", "ThinkPad X1"));
}
[Fact]
public async Task DeleteOlderThanAsync_FiltersOnFirstNotifiedAt_NotTimestamp()
{
// Rows are insert-once, so FirstNotifiedAt is the creation time. Timestamp would look
// equivalent but is rewritten by a table restore/copy, which would silently re-arm every
// bell in the tenant at once.
var harness = new Harness();
var cutoff = new DateTime(2026, 7, 25, 12, 0, 0, DateTimeKind.Utc);
await harness.Sut.DeleteOlderThanAsync(cutoff);
var filter = Assert.Single(harness.QueryFilters);
Assert.Equal("FirstNotifiedAt lt datetime'2026-07-25T12:00:00Z'", filter);
}
[Fact]
public async Task DeleteOlderThanAsync_ContinuesAfterASingleRowFailure()
{
var harness = new Harness();
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, "OTHER123"));
harness.AgeRows(TimeSpan.FromDays(31));
harness.FailDeleteForRowKeys.Add("tpmpss|s4sq8685");
var deleted = await harness.Sut.DeleteOlderThanAsync(DateTime.UtcNow.AddDays(-30));
Assert.Equal(1, deleted);
Assert.Equal("tpmpss|s4sq8685", Assert.Single(harness.Store.Values).RowKey);
}
[Fact]
public async Task DeleteOlderThanAsync_QueryFailure_ReturnsZero()
{
var harness = new Harness { QueryFailure = new RequestFailedException(500, "Boom") };
Assert.Equal(0, await harness.Sut.DeleteOlderThanAsync(DateTime.UtcNow.AddDays(-30)));
}
// ── Harness ───────────────────────────────────────────────────────────────────
private sealed class Harness
{
private static readonly Regex CutoffFilter =
new(@"^FirstNotifiedAt lt datetime'(?<cutoff>[^']+)'$", RegexOptions.Compiled);
public TableHardwareRejectionNotificationTracker Sut { get; }
public Mock<TableClient> Table { get; }
public Dictionary<(string Pk, string Rk), TableEntity> Store { get; } = new();
public List<string> QueryFilters { get; } = new();
public HashSet<string> FailDeleteForRowKeys { get; } = new();
/// <summary>Non-409 failure injected into AddEntityAsync (fail-soft path).</summary>
public Exception? AddFailure { get; init; }
/// <summary>Failure injected into QueryAsync enumeration (sweep fail-soft path).</summary>
public Exception? QueryFailure { get; init; }
public Harness()
{
Table = new Mock<TableClient>();
Table.Setup(c => c.AddEntityAsync(It.IsAny<TableEntity>(), It.IsAny<CancellationToken>()))
.Returns<TableEntity, CancellationToken>((e, _) =>
{
if (AddFailure != null) throw AddFailure;
if (Store.ContainsKey((e.PartitionKey, e.RowKey)))
throw new RequestFailedException(409, "Conflict", "EntityAlreadyExists", null);
Store[(e.PartitionKey, e.RowKey)] = e;
return Task.FromResult(new Mock<Response>().Object);
});
Table.Setup(c => c.QueryAsync<TableEntity>(
It.IsAny<string>(),
It.IsAny<int?>(),
It.IsAny<IEnumerable<string>>(),
It.IsAny<CancellationToken>()))
.Returns<string, int?, IEnumerable<string>, CancellationToken>((filter, _, _, _) =>
{
if (QueryFailure != null) throw QueryFailure;
QueryFilters.Add(filter);
var rows = Store.Values.Where(e => MatchesCutoff(filter, e)).ToList();
return AsyncPageable<TableEntity>.FromPages(new[]
{
Page<TableEntity>.FromValues(rows, null, new Mock<Response>().Object),
});
});
Table.Setup(c => c.DeleteEntityAsync(
It.IsAny<string>(),
It.IsAny<string>(),
It.IsAny<ETag>(),
It.IsAny<CancellationToken>()))
.Returns<string, string, ETag, CancellationToken>((pk, rk, _, _) =>
{
if (FailDeleteForRowKeys.Contains(rk))
throw new RequestFailedException(500, "DeleteFailed");
Store.Remove((pk, rk));
return Task.FromResult(new Mock<Response>().Object);
});
var serviceClient = new Mock<TableServiceClient>();
serviceClient.Setup(s => s.GetTableClient(It.IsAny<string>())).Returns(Table.Object);
Sut = new TableHardwareRejectionNotificationTracker(
new TableStorageService(serviceClient.Object, NullLogger<TableStorageService>.Instance),
NullLogger<TableHardwareRejectionNotificationTracker>.Instance);
}
/// <summary>Backdates every stored row so a retention sweep sees it as expired.</summary>
public void AgeRows(TimeSpan age)
{
foreach (var entity in Store.Values)
entity["FirstNotifiedAt"] = DateTime.UtcNow - age;
}
/// <summary>
/// Minimal server-side emulation of the OData filter the tracker builds. An unrecognised
/// filter matches nothing, so a rewrite that no longer selects expired rows fails loudly
/// here instead of leaving dead rows in production.
/// </summary>
private static bool MatchesCutoff(string filter, TableEntity entity)
{
var match = CutoffFilter.Match(filter ?? string.Empty);
if (!match.Success) return false;
var cutoff = DateTime.Parse(
match.Groups["cutoff"].Value,
null,
System.Globalization.DateTimeStyles.AdjustToUniversal | System.Globalization.DateTimeStyles.AssumeUniversal);
var firstNotified = entity.GetDateTime("FirstNotifiedAt");
return firstNotified.HasValue && firstNotified.Value.ToUniversalTime() < cutoff;
}
}
}
@@ -59,4 +59,35 @@ public class HardwareRejectionNotificationTrackerKeyTests
var key = TableHardwareRejectionNotificationTracker.BuildRowKey("", "");
Assert.Equal("|", key);
}
// =========================================================================
// TPM PSS row keys (same table, "tpmpss|" prefix — dedup per serial)
// =========================================================================
[Fact]
public void BuildTpmPssRowKey_LowercasesTrimsAndPrefixes()
{
var key = TableHardwareRejectionNotificationTracker.BuildTpmPssRowKey(" S4SQ8685 ");
Assert.Equal("tpmpss|s4sq8685", key);
}
[Theory]
[InlineData("S4SQ8685", "s4sq8685")]
[InlineData("s4Sq8685", "S4SQ8685")]
public void BuildTpmPssRowKey_IsCaseInsensitive(string serialA, string serialB)
{
Assert.Equal(
TableHardwareRejectionNotificationTracker.BuildTpmPssRowKey(serialA),
TableHardwareRejectionNotificationTracker.BuildTpmPssRowKey(serialB));
}
[Fact]
public void BuildTpmPssRowKey_DoesNotCollideWithHardwareKeyOfSameText()
{
// A hardware key is "{mfr}|{model}"; the TPM key's fixed "tpmpss|" prefix keeps the
// two key spaces disjoint within the shared table.
var hardwareKey = TableHardwareRejectionNotificationTracker.BuildRowKey("Lenovo", "S4SQ8685");
var tpmKey = TableHardwareRejectionNotificationTracker.BuildTpmPssRowKey("S4SQ8685");
Assert.NotEqual(hardwareKey, tpmKey);
}
}
@@ -4,9 +4,12 @@ using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services.Diagnostics;
using Azure;
using Azure.Storage.Blobs;
using Azure.Storage.Blobs.Models;
using Azure.Storage.Sas;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
@@ -172,6 +175,119 @@ public class HostedDiagnosticsBlobServiceTests
Assert.StartsWith(TenantB + "/", b.BlobPath);
}
// ── DeleteBySessionPrefixAsync — cascade sweep for multi-package sessions ──────
private const string SessionA = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa";
[Theory]
[InlineData("not-a-guid", SessionA)]
[InlineData(TenantA, "not-a-guid")]
public async Task DeleteBySessionPrefix_RejectsNonGuidInputs(string tenantId, string sessionId)
{
var svc = new PrefixDeleteFake(containerExists: true);
await Assert.ThrowsAsync<ArgumentException>(
() => svc.DeleteBySessionPrefixAsync(tenantId, sessionId));
}
[Fact]
public async Task DeleteBySessionPrefix_MissingContainer_IsNoOp()
{
var svc = new PrefixDeleteFake(containerExists: false);
var deleted = await svc.DeleteBySessionPrefixAsync(TenantA, SessionA);
Assert.Equal(0, deleted);
Assert.Null(svc.CapturedPrefix);
}
[Fact]
public async Task DeleteBySessionPrefix_UsesSessionScopedPrefix_AndDeletesEveryMatch()
{
// Two packages for the session: an on-demand (server-requested) one and the terminal
// one. The sweep must issue a delete for BOTH even though the Sessions row (and thus
// the deletion manifest) only references the last.
var svc = new PrefixDeleteFake(
containerExists: true,
blobNames: new[]
{
$"{TenantA}/AgentDiagnostics-{SessionA}-20260724T100000-server-requested.zip",
$"{TenantA}/AgentDiagnostics-{SessionA}-20260724T113000.zip",
});
var deleted = await svc.DeleteBySessionPrefixAsync(TenantA, SessionA);
Assert.Equal(2, deleted);
Assert.Equal($"{TenantA}/AgentDiagnostics-{SessionA}-", svc.CapturedPrefix);
Assert.Equal(2, svc.DeletedBlobNames.Count);
Assert.All(svc.DeletedBlobNames, n => Assert.StartsWith(svc.CapturedPrefix!, n));
}
[Fact]
public async Task DeleteBySessionPrefix_NoMatches_ReturnsZero()
{
var svc = new PrefixDeleteFake(containerExists: true, blobNames: Array.Empty<string>());
var deleted = await svc.DeleteBySessionPrefixAsync(TenantA, SessionA);
Assert.Equal(0, deleted);
Assert.Equal($"{TenantA}/AgentDiagnostics-{SessionA}-", svc.CapturedPrefix);
}
/// <summary>
/// Overrides <c>GetContainerClient</c> with a Moq'd <see cref="BlobContainerClient"/> whose
/// enumeration returns a canned page for the CAPTURED prefix. Prefix filtering itself is
/// the Azure SDK's contract — these tests pin what prefix we ask for and that every
/// returned item is deleted.
/// </summary>
private sealed class PrefixDeleteFake : HostedDiagnosticsBlobService
{
private readonly Mock<BlobContainerClient> _container;
public string? CapturedPrefix { get; private set; }
public List<string> DeletedBlobNames { get; } = new();
public PrefixDeleteFake(bool containerExists, string[]? blobNames = null)
: base(
new BlobServiceClient("UseDevelopmentStorage=true"),
NullLogger<HostedDiagnosticsBlobService>.Instance,
usesManagedIdentity: false)
{
_container = new Mock<BlobContainerClient>();
_container
.Setup(c => c.ExistsAsync(It.IsAny<CancellationToken>()))
.ReturnsAsync(Response.FromValue(containerExists, Mock.Of<Response>()));
_container
.Setup(c => c.GetBlobsAsync(
It.IsAny<BlobTraits>(), It.IsAny<BlobStates>(), It.IsAny<string?>(), It.IsAny<CancellationToken>()))
.Returns((BlobTraits _, BlobStates _, string? prefix, CancellationToken _) =>
{
CapturedPrefix = prefix;
var items = (blobNames ?? Array.Empty<string>())
.Select(n => BlobsModelFactory.BlobItem(name: n))
.ToList();
return AsyncPageable<BlobItem>.FromPages(new[]
{
Page<BlobItem>.FromValues(items, continuationToken: null, Mock.Of<Response>()),
});
});
_container
.Setup(c => c.GetBlobClient(It.IsAny<string>()))
.Returns((string name) =>
{
var blob = new Mock<BlobClient>();
blob
.Setup(b => b.DeleteIfExistsAsync(
It.IsAny<DeleteSnapshotsOption>(), It.IsAny<BlobRequestConditions>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(() =>
{
DeletedBlobNames.Add(name);
return Response.FromValue(true, Mock.Of<Response>());
});
return blob.Object;
});
}
protected override BlobContainerClient GetContainerClient() => _container.Object;
}
// ── Constants ───────────────────────────────────────────────────────────────────
[Fact]
@@ -0,0 +1,165 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the per-session API-latency projection (2026-07-28): the V2 agent emits cumulative
/// counters <c>net_total_latency_ms</c> / <c>net_total_requests</c> in every
/// <c>agent_metrics_snapshot</c>; ingest derives the session-wide average from the LAST
/// snapshot of the batch and persists it via <c>UpdateSessionNetworkLatencyAsync</c>
/// (last-write-wins on cumulative counters → replay-idempotent).
/// Wire-format note: the ingest body is parsed with Newtonsoft into
/// <c>Dictionary&lt;string, object&gt;</c>, so integer JSON numbers arrive as boxed
/// <c>long</c> — fixtures mirror that, not the agent's in-memory types.
/// </summary>
public class IngestApiLatencyProjectionTests
{
private static EnrollmentEvent Snapshot(object? totalLatencyMs, object? totalRequests)
{
var evt = new EnrollmentEvent
{
EventType = "agent_metrics_snapshot",
Data = new Dictionary<string, object>
{
["agent_cpu_percent"] = 3.62,
["net_requests"] = 50L,
["net_avg_latency_ms"] = 513.0,
},
};
if (totalLatencyMs != null) evt.Data["net_total_latency_ms"] = totalLatencyMs;
if (totalRequests != null) evt.Data["net_total_requests"] = totalRequests;
return evt;
}
[Fact]
public void CumulativeCounters_YieldSessionAverage()
{
var events = new List<EnrollmentEvent> { Snapshot(51_300L, 100L) };
Assert.True(EventIngestProcessor.TryComputeSessionApiLatency(events, out var avg, out var count));
Assert.Equal(513.0, avg);
Assert.Equal(100, count);
}
[Fact]
public void AverageIsRounded_ToOneDecimal()
{
var events = new List<EnrollmentEvent> { Snapshot(1_000L, 3L) };
Assert.True(EventIngestProcessor.TryComputeSessionApiLatency(events, out var avg, out _));
Assert.Equal(333.3, avg);
}
[Fact]
public void LastSnapshotOfBatch_Wins()
{
// Cumulative counters grow monotonically — the last snapshot carries the most
// complete session average.
var events = new List<EnrollmentEvent>
{
Snapshot(51_300L, 100L),
new EnrollmentEvent { EventType = "download_progress" },
Snapshot(120_000L, 200L),
};
Assert.True(EventIngestProcessor.TryComputeSessionApiLatency(events, out var avg, out var count));
Assert.Equal(600.0, avg);
Assert.Equal(200, count);
}
[Fact]
public void OldAgentSnapshot_WithoutTotalLatencyField_DoesNotMatch()
{
// Agents predating net_total_latency_ms emit net_total_requests only.
var events = new List<EnrollmentEvent> { Snapshot(totalLatencyMs: null, totalRequests: 100L) };
Assert.False(EventIngestProcessor.TryComputeSessionApiLatency(events, out _, out _));
}
[Fact]
public void SnapshotWithLatencyButMissingRequests_DoesNotMatch()
{
var events = new List<EnrollmentEvent> { Snapshot(51_300L, totalRequests: null) };
Assert.False(EventIngestProcessor.TryComputeSessionApiLatency(events, out _, out _));
}
[Fact]
public void ZeroRequests_DoesNotMatch()
{
// No requests yet (first snapshot before the first upload completes) — no division.
var events = new List<EnrollmentEvent> { Snapshot(0L, 0L) };
Assert.False(EventIngestProcessor.TryComputeSessionApiLatency(events, out _, out _));
}
[Fact]
public void MatchingLastSnapshot_SkipsOlderAgentSnapshotAfterIt()
{
// A field-less (old-schema) snapshot later in the batch must not shadow an earlier
// one that carries the counters — the predicate selects the last MATCHING snapshot.
var events = new List<EnrollmentEvent>
{
Snapshot(51_300L, 100L),
Snapshot(totalLatencyMs: null, totalRequests: 150L),
};
Assert.True(EventIngestProcessor.TryComputeSessionApiLatency(events, out var avg, out var count));
Assert.Equal(513.0, avg);
Assert.Equal(100, count);
}
[Fact]
public void BatchWithoutSnapshots_DoesNotMatch()
{
var events = new List<EnrollmentEvent>
{
new EnrollmentEvent { EventType = "esp_phase_changed" },
new EnrollmentEvent { EventType = "app_install_started" },
};
Assert.False(EventIngestProcessor.TryComputeSessionApiLatency(events, out _, out _));
}
// -------- TryGetDouble coercion ladder (mirrors PlatformMetricsService.GetDouble) --------
[Theory]
[InlineData(513.5)] // double — decimal JSON number
[InlineData(513)] // int — in-memory constructed events (tests, V1 paths)
[InlineData(513L)] // long — Newtonsoft wire form for integer JSON numbers
[InlineData(513f)] // float
public void TryGetDouble_CoercesBoxedNumericPrimitives(object boxed)
{
var data = new Dictionary<string, object> { ["k"] = boxed };
Assert.True(EventIngestProcessor.TryGetDouble(data, "k", out var value));
Assert.Equal(Convert.ToDouble(boxed), value);
}
[Fact]
public void TryGetDouble_ParsesStrings_InvariantCulture()
{
var data = new Dictionary<string, object> { ["k"] = "513.5" };
Assert.True(EventIngestProcessor.TryGetDouble(data, "k", out var value));
Assert.Equal(513.5, value);
}
[Theory]
[InlineData("not-a-number")]
[InlineData("")]
public void TryGetDouble_RejectsUnparseableStrings(string raw)
{
var data = new Dictionary<string, object> { ["k"] = raw };
Assert.False(EventIngestProcessor.TryGetDouble(data, "k", out _));
}
[Fact]
public void TryGetDouble_MissingKey_ReturnsFalse()
{
Assert.False(EventIngestProcessor.TryGetDouble(new Dictionary<string, object>(), "k", out _));
}
}
@@ -0,0 +1,107 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Session eaf3d8c4 (2026-07-23): a previous enrollment's IME log surviving on disk made the
/// agent replay a week-old script history — 156 replayed <c>script_completed</c> events
/// inflated the session's RemediationScriptCount. Newer agents suppress the replay at the
/// source; this predicate covers the fleet of not-yet-updated agents by skipping the script
/// counter increment for events whose <c>rejectedSourceTimestamp</c> (the source log line's
/// own time, preserved by the agent's staleness clamp) is more than 24 h older than the
/// event's clock-clamped stamp.
/// </summary>
public class IngestHistoricImeReplayTests
{
private static readonly DateTime EventStamp = new(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
private static EnrollmentEvent ScriptEvent(object? rejectedSourceTimestamp, bool includeKey = true)
{
var evt = new EnrollmentEvent
{
EventType = "script_completed",
Timestamp = EventStamp,
Data = new Dictionary<string, object> { ["scriptType"] = "remediation" },
};
if (includeKey)
evt.Data["rejectedSourceTimestamp"] = rejectedSourceTimestamp!;
return evt;
}
[Fact]
public void StaleRejectedSourceTimestamp_IsHistoricReplay()
{
var evt = ScriptEvent(EventStamp.AddDays(-7).ToString("o"));
Assert.True(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Fact]
public void StaleAppInstallEvent_IsHistoricReplay()
{
// The predicate is event-type-agnostic — it also guards AggregateAppInstallEvent
// (AppInstallSummaries would otherwise be overwritten with week-old runs).
var evt = new EnrollmentEvent
{
EventType = "app_install_completed",
Timestamp = EventStamp,
Data = new Dictionary<string, object>
{
["appName"] = "Contoso App",
["rejectedSourceTimestamp"] = EventStamp.AddDays(-7).ToString("o"),
},
};
Assert.True(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Fact]
public void JustOver24h_IsHistoricReplay()
{
var evt = ScriptEvent(EventStamp.AddHours(-25).ToString("o"));
Assert.True(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Fact]
public void Within24h_IsNotHistoricReplay()
{
var evt = ScriptEvent(EventStamp.AddHours(-23).ToString("o"));
Assert.False(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Fact]
public void FutureSkewRejection_IsNotHistoricReplay()
{
// A rejected source timestamp AHEAD of the event stamp is a mid-enrollment clock
// jump (WhiteGlove +1h CMTrace skew), not replayed history.
var evt = ScriptEvent(EventStamp.AddHours(2).ToString("o"));
Assert.False(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Fact]
public void MissingKey_IsNotHistoricReplay()
{
Assert.False(EventIngestProcessor.IsHistoricImeReplay(ScriptEvent(null, includeKey: false)));
}
[Fact]
public void NullData_IsNotHistoricReplay()
{
var evt = new EnrollmentEvent { EventType = "script_completed", Timestamp = EventStamp, Data = null! };
Assert.False(EventIngestProcessor.IsHistoricImeReplay(evt));
}
[Theory]
[InlineData("")]
[InlineData("not-a-date")]
public void UnparseableValue_IsNotHistoricReplay(string raw)
{
Assert.False(EventIngestProcessor.IsHistoricImeReplay(ScriptEvent(raw)));
}
[Fact]
public void NullEvent_IsNotHistoricReplay()
{
Assert.False(EventIngestProcessor.IsHistoricImeReplay(null));
}
}
@@ -0,0 +1,128 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Session 8e4cc4ae (2026-07-24): two on-demand ("Collect Logs") uploads landed in hosted
/// storage, but the portal never flipped to Download — the agent's ServerActionDispatcher
/// confirms the upload via <c>server_action_executed(actionType=request_diagnostics,
/// blobName=…)</c> and emits NO <c>diagnostics_uploaded</c> (only the terminal path does),
/// so the Sessions row was never stamped. These tests pin the classification predicate and
/// the destination inference that close that gap backend-side (no agent release needed).
/// </summary>
public class IngestOnDemandDiagnosticsTests
{
private const string TenantId = "b54dc1af-5320-4f60-b5d4-821e0cf2a359";
private const string HostedBlob =
TenantId + "/AgentDiagnostics-8e4cc8ae-1111-2222-3333-444455556666-20260724170902-server-requested.zip";
private static EnrollmentEvent Evt(
string eventType = "server_action_executed",
string? actionType = "request_diagnostics",
string? blobName = HostedBlob)
{
var data = new Dictionary<string, object>();
if (actionType != null) data["actionType"] = actionType;
if (blobName != null) data["blobName"] = blobName;
return new EnrollmentEvent { EventType = eventType, Data = data };
}
// -------- IsOnDemandDiagnosticsUploadConfirmation --------
[Fact]
public void ExecutedRequestDiagnosticsWithBlobName_IsConfirmation()
{
Assert.True(EventIngestProcessor.IsOnDemandDiagnosticsUploadConfirmation(Evt()));
}
[Fact]
public void ActionTypeIsCaseInsensitive()
{
Assert.True(EventIngestProcessor.IsOnDemandDiagnosticsUploadConfirmation(
Evt(actionType: "Request_Diagnostics")));
}
[Theory]
[InlineData("server_action_received")]
[InlineData("server_action_failed")]
[InlineData("diagnostics_uploaded")]
public void OtherEventTypes_AreNotConfirmations(string eventType)
{
Assert.False(EventIngestProcessor.IsOnDemandDiagnosticsUploadConfirmation(Evt(eventType: eventType)));
}
[Theory]
[InlineData("rotate_config")]
[InlineData("terminate_session")]
[InlineData(null)]
public void OtherActionTypes_AreNotConfirmations(string? actionType)
{
Assert.False(EventIngestProcessor.IsOnDemandDiagnosticsUploadConfirmation(Evt(actionType: actionType)));
}
[Theory]
[InlineData(null)]
[InlineData("")]
public void MissingOrEmptyBlobName_IsNotAConfirmation(string? blobName)
{
// A failed upload never reaches server_action_executed, but an executed event without
// blobName (defensive) must not wipe/stamp the row either.
Assert.False(EventIngestProcessor.IsOnDemandDiagnosticsUploadConfirmation(Evt(blobName: blobName)));
}
[Fact]
public void NullEventOrData_IsNotAConfirmation()
{
Assert.False(EventIngestProcessor.IsOnDemandDiagnosticsUploadConfirmation(null));
// Data is non-nullable on the model but the JSON deserializer can still hand us null
// for a payload without "data" — null! pins the guard the predicate actually has.
Assert.False(EventIngestProcessor.IsOnDemandDiagnosticsUploadConfirmation(
new EnrollmentEvent { EventType = "server_action_executed", Data = null! }));
}
// -------- InferDiagnosticsDestination --------
[Fact]
public void ExplicitDestination_AlwaysWins()
{
Assert.Equal("CustomerSas",
EventIngestProcessor.InferDiagnosticsDestination("CustomerSas", HostedBlob, TenantId));
}
[Fact]
public void TenantPrefixedBlobName_InfersHosted()
{
Assert.Equal("Hosted",
EventIngestProcessor.InferDiagnosticsDestination(null, HostedBlob, TenantId));
}
[Fact]
public void TenantPrefixMatch_IsCaseInsensitive()
{
Assert.Equal("Hosted",
EventIngestProcessor.InferDiagnosticsDestination(null, HostedBlob.ToUpperInvariant(), TenantId));
}
[Fact]
public void BareFilename_InfersNothing()
{
// CustomerSas uploads persist the bare zip filename — repo must leave the
// destination column unchanged (read-time default CustomerSas).
Assert.Null(EventIngestProcessor.InferDiagnosticsDestination(
null, "AgentDiagnostics-x-20260724.zip", TenantId));
}
[Fact]
public void ForeignTenantPrefix_InfersNothing()
{
Assert.Null(EventIngestProcessor.InferDiagnosticsDestination(
null, "99999999-9999-9999-9999-999999999999/x.zip", TenantId));
}
[Fact]
public void NullBlobName_InfersNothing()
{
Assert.Null(EventIngestProcessor.InferDiagnosticsDestination(null, null, TenantId));
}
}
@@ -119,6 +119,138 @@ public class MetricsMathAppPayloadTests
Assert.Equal(2, topCode.GetProperty("count").GetInt32());
}
[Fact]
public void FailureRate_IsOverFinishedInstalls_InProgressExcluded()
{
var summaries = new List<AppInstallSummary>
{
new() { AppName = "App A", Status = "Failed" },
new() { AppName = "App A", Status = "Succeeded", DurationSeconds = 10 },
new() { AppName = "App A", Status = "InProgress" },
new() { AppName = "App A", Status = "InProgress" },
};
var app = Build(summaries).GetProperty("topFailingApps")[0];
Assert.Equal(4, app.GetProperty("totalInstalls").GetInt32());
// 1 / (1 + 1) = 50% (over finished), NOT 1 / 4 = 25% (over all installs).
Assert.Equal(50d, app.GetProperty("failureRate").GetDouble());
}
// ── PR0 (2026-07-26): skip/unmeasured classification ────────────────────
[Fact]
public void Skips_LeaveTheFailureRate_AndAreReportedSeparately()
{
// 1 Failed + 1 real install + 3 skips ("Update for X" not applicable).
// Rate must be 1/(1+1) = 50%, NOT 1/(1+4) = 20% — skips are not attempts.
var summaries = new List<AppInstallSummary>
{
new() { AppName = "App A", Status = "Failed", TerminalState = "Error" },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 30 },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Skipped" },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Skipped" },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Postponed" },
};
var root = Build(summaries);
var app = root.GetProperty("topFailingApps")[0];
Assert.Equal(50d, app.GetProperty("failureRate").GetDouble());
Assert.Equal(1, app.GetProperty("succeeded").GetInt32());
Assert.Equal(3, app.GetProperty("skipped").GetInt32());
Assert.Equal(0, app.GetProperty("unmeasured").GetInt32());
Assert.Equal(3, root.GetProperty("totalSkipped").GetInt32());
}
[Fact]
public void ZeroDurations_AreExcludedFromAverages_AndCountedAsUnmeasured()
{
// Two measured installs (100s, 200s) + one whose start was never observed (audit §0.5:
// agent attach window — duration UNKNOWN). avg must be 150, not (100+200+0)/3 = 100.
var summaries = new List<AppInstallSummary>
{
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 100 },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 200 },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 0 },
};
var root = Build(summaries);
Assert.Equal(1, root.GetProperty("totalUnmeasured").GetInt32());
// Rebuild with a third measured row to clear the sample floor and read the average.
summaries.Add(new AppInstallSummary { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 150 });
var slowest = Build(summaries).GetProperty("slowestApps");
Assert.Equal(1, slowest.GetArrayLength());
Assert.Equal(150d, slowest[0].GetProperty("avgDurationSeconds").GetDouble());
Assert.Equal(1, slowest[0].GetProperty("unmeasured").GetInt32());
}
[Fact]
public void LegacyZeroDurationRows_StayInRate_ButNeverInDurations()
{
// Rows written before TerminalState existed: cannot be classified as skip vs unmeasured.
// They keep the legacy rate behavior (counted as succeeded) and are excluded from
// durations (counted as unmeasured — duration genuinely not measured).
var summaries = new List<AppInstallSummary>
{
new() { AppName = "App A", Status = "Failed" },
new() { AppName = "App A", Status = "Succeeded", DurationSeconds = 0 }, // legacy, no TerminalState
new() { AppName = "App A", Status = "Succeeded", DurationSeconds = 60, TerminalState = "Installed" },
};
var app = Build(summaries).GetProperty("topFailingApps")[0];
Assert.Equal(2, app.GetProperty("succeeded").GetInt32()); // legacy row stays in the rate
Assert.Equal(33.3d, app.GetProperty("failureRate").GetDouble());
Assert.Equal(1, app.GetProperty("unmeasured").GetInt32());
Assert.Equal(0, app.GetProperty("skipped").GetInt32());
}
[Fact]
public void ImplausibleDurations_AreExcludedFromAverages_AndCountedAsUnmeasured()
{
// A duration beyond the agent's max observation lifetime is a back-stamped emission
// artifact (CompletedAt pinned at session end — production: 60 175 s app rows inside a
// 2 697 s session), not a slow install. It must fall out of the averages exactly like
// an unobserved start. Boundary: exactly 6 h is still plausible/measured.
var summaries = new List<AppInstallSummary>
{
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 100 },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 200 },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed",
DurationSeconds = MetricsMath.MaxPlausibleInstallDurationSeconds }, // boundary: measured
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed",
DurationSeconds = MetricsMath.MaxPlausibleInstallDurationSeconds + 1 }, // artifact: unmeasured
};
var root = Build(summaries);
Assert.Equal(1, root.GetProperty("totalUnmeasured").GetInt32());
var slowest = root.GetProperty("slowestApps");
Assert.Equal(1, slowest.GetArrayLength());
Assert.Equal(3, slowest[0].GetProperty("measuredInstalls").GetInt32());
// avg over {100, 200, 21600} — the 21601 artifact contributes nothing.
Assert.Equal(7300d, slowest[0].GetProperty("avgDurationSeconds").GetDouble());
Assert.Equal(MetricsMath.MaxPlausibleInstallDurationSeconds, slowest[0].GetProperty("maxDurationSeconds").GetInt32());
}
[Fact]
public void SlowestApps_GateOnMeasuredInstalls_NotOnSucceededCount()
{
// 3 succeeded but only 2 measured → below the floor; an app must not rank "fast"
// (or at all) on a handful of unobserved zeros.
var summaries = new List<AppInstallSummary>
{
new() { AppName = "Mostly Unmeasured", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 5 },
new() { AppName = "Mostly Unmeasured", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 5 },
new() { AppName = "Mostly Unmeasured", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 0 },
};
var slowest = Build(summaries).GetProperty("slowestApps");
Assert.Equal(0, slowest.GetArrayLength());
}
[Fact]
public void SlowestApps_DropsAppsBelowSampleFloor()
{
@@ -125,5 +125,6 @@ public class OffboardingSessionEnumeratorTests
public Task<bool> IsSessionIndexEmptyAsync() => throw new NotImplementedException();
public Task<List<OrphanedEventSession>> GetOrphanedEventSessionsAsync(TimeSpan gracePeriod) => throw new NotImplementedException();
public Task DeleteEventSessionIndexEntryAsync(string tenantId, string sessionId) => throw new NotImplementedException();
public Task<List<(string TenantId, string SessionId)>> GetAllSessionTombstoneKeysAsync() => throw new NotImplementedException();
}
}
@@ -1,4 +1,5 @@
using System.Security.Claims;
using AutopilotMonitor.Functions.Functions.Config;
using AutopilotMonitor.Functions.Middleware;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Services;
@@ -205,8 +206,101 @@ public class PolicyEnforcementMiddlewareTests
Assert.Equal(403, result.StatusCode);
}
// ── config GET (TenantAdminOrGlobalReader): own-tenant admin view vs reader view ──
// GetTenantConfigurationFunction redacts only when IsGlobalReader && !(IsTenantAdmin && Target==Tenant).
// ── TenantAdminOrOperator (POST sessions/{id}/actions — "Collect Logs") ────────
private const string SessionA = "aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa";
private static void AsTenantMemberRole(Harness h, string tenantId, string upn, string role) =>
h.Repo.Setup(r => r.GetTenantMemberAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync(new TenantMember { Upn = upn, TenantId = tenantId, Role = role, IsEnabled = true });
[Fact]
public async Task SessionActions_TenantOperator_IsAllowed_WithOperatorRoleOnContext()
{
const string upn = "operator@contoso.com";
var h = BuildHarness();
AsTenantMemberRole(h, TenantA, upn, Constants.TenantRoles.Operator);
var result = await h.Middleware.DecideAsync(
"POST", $"/api/sessions/{SessionA}/actions", null, AuthedPrincipal(TenantA, upn));
Assert.True(result.Allowed);
// The function's per-type gate depends on these exact flags: an Operator must arrive
// with IsTenantAdmin=false + UserRole=Operator so terminate_session/rotate_config 403.
Assert.False(result.Context!.IsTenantAdmin);
Assert.Equal(Constants.TenantRoles.Operator, result.Context!.UserRole);
}
[Fact]
public async Task SessionActions_TenantAdmin_IsAllowed_WithAdminFlag()
{
const string upn = "admin@contoso.com";
var h = BuildHarness();
h.AsTenantAdmin(TenantA, upn);
var result = await h.Middleware.DecideAsync(
"POST", $"/api/sessions/{SessionA}/actions", null, AuthedPrincipal(TenantA, upn));
Assert.True(result.Allowed);
Assert.True(result.Context!.IsTenantAdmin);
}
[Fact]
public async Task SessionActions_TenantViewer_IsForbidden()
{
const string upn = "viewer@contoso.com";
var h = BuildHarness();
AsTenantMemberRole(h, TenantA, upn, Constants.TenantRoles.Viewer);
var result = await h.Middleware.DecideAsync(
"POST", $"/api/sessions/{SessionA}/actions", null, AuthedPrincipal(TenantA, upn));
Assert.False(result.Allowed);
Assert.Equal(403, result.StatusCode);
}
[Fact]
public async Task SessionActions_NonMember_IsForbidden()
{
var h = BuildHarness(); // defaults: no membership, no global role
var result = await h.Middleware.DecideAsync(
"POST", $"/api/sessions/{SessionA}/actions", null, AuthedPrincipal(TenantA, "outsider@contoso.com"));
Assert.False(result.Allowed);
Assert.Equal(403, result.StatusCode);
}
[Fact]
public async Task SessionActions_GlobalAdmin_IsAllowed()
{
const string upn = "ga@platform.com";
var h = BuildHarness();
h.AsGlobalRole(Constants.GlobalRoles.GlobalAdmin);
var result = await h.Middleware.DecideAsync(
"POST", $"/api/sessions/{SessionA}/actions", null, AuthedPrincipal(TenantA, upn));
Assert.True(result.Allowed);
Assert.True(result.Context!.IsGlobalAdmin);
}
[Fact]
public async Task SessionActions_PureGlobalReader_IsForbidden_WriteTier()
{
const string upn = "reader@platform.com";
var h = BuildHarness();
h.AsGlobalRole(Constants.GlobalRoles.GlobalReader);
var result = await h.Middleware.DecideAsync(
"POST", $"/api/sessions/{SessionA}/actions", null, AuthedPrincipal(TenantA, upn));
Assert.False(result.Allowed);
Assert.Equal(403, result.StatusCode);
}
// ── config GET (MemberRead): own-tenant admin view vs redacted reader/member view ──
// GetTenantConfigurationFunction redacts unless GA or own-tenant Admin (CanViewSecrets).
// These lock in the RequestContext that drives that decision.
[Fact]
@@ -243,6 +337,38 @@ public class PolicyEnforcementMiddlewareTests
Assert.Equal(TenantB, rc.TargetTenantId);
}
[Fact]
public async Task ConfigGet_OwnTenantOperator_IsAllowed_RedactedMemberView()
{
// Read-only Settings view for troubleshooting staff: an own-tenant Operator is admitted
// (MemberRead) but arrives with IsTenantAdmin=false, so CanViewSecrets ⇒ redacted copy.
const string upn = "operator@contoso.com";
var h = BuildHarness();
AsTenantMemberRole(h, TenantA, upn, Constants.TenantRoles.Operator);
var result = await h.Middleware.DecideAsync("GET", $"/api/config/{TenantA}", null, AuthedPrincipal(TenantA, upn));
Assert.True(result.Allowed);
var rc = result.Context!;
Assert.False(rc.IsTenantAdmin);
Assert.False(GetTenantConfigurationFunction.CanViewSecrets(rc));
}
[Fact]
public async Task ConfigGet_CrossTenantOperator_IsForbidden()
{
// Membership is resolved against the caller's JWT tenant — an Operator of A naming
// tenant B in the route is a cross-tenant read and must stay 403.
const string upn = "operator@contoso.com";
var h = BuildHarness();
AsTenantMemberRole(h, TenantA, upn, Constants.TenantRoles.Operator);
var result = await h.Middleware.DecideAsync("GET", $"/api/config/{TenantB}", null, AuthedPrincipal(TenantA, upn));
Assert.False(result.Allowed);
Assert.Equal(403, result.StatusCode);
}
// ── Cross-tenant read: GlobalReader crosses tenants (HasGlobalScope gate) ───────
[Fact]

Some files were not shown because too many files have changed in this diff Show More