Compare commits

...

119 Commits

Author SHA1 Message Date
Oliver Kieselbach 372f0c7c98 feat(web): source pill for non-Intune install rows + per-source row key
Customers who additionally deploy Microsoft 365 Apps as their own Win32 app saw two
rows in Install Progress (the IME view and the Office C2R lifecycle) with nothing
telling them apart, which read as a duplicate.

- New lib/installProgress.ts: buildInstallItems() extracted from the component so the
  event-folding rules are unit-testable; component is now render-only.
- InstallItem gains `source` (ime | office-c2r | realmjoin) and a source-scoped `key`.
  Rows were keyed by app name alone, so a Win32 app named exactly "Microsoft 365 Apps"
  (= OfficeInstallDetector.AppName) collided with the C2R row and overwrote its state.
- Pill only for non-default sources: "Click-to-Run" / "RealmJoin", neutral grey so it
  does not compete with the state badge, with a tooltip explaining what the row measures.
  Plain Intune app rows stay unlabelled.
- Drops the "RJ: " name prefix, which broke sorting and truncation.

Verified: tsc clean, vitest 643/643 (16 new).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 11:24:53 +02:00
Oliver Kieselbach 8057b44e48 fix(agent): serialize ESP-config collects so a later emission never carries an older snapshot
Codex review (P2): the ESP-config refresh triggers each run on their own
Task.Run (as do CollectAll's startup sweep and CollectAtEnrollmentStart),
so two concurrent collects could interleave read-before-emit - the LATER
emission then carried the OLDER (smaller) registry snapshot. The backend
uses the latest list-bearing esp_config_detected emission as positive
blocking evidence, so an out-of-order emission would silently shrink it.

CollectEspConfiguration now runs its read->emit sequence inside a
dedicated lock covering ALL callers; a later emission always carries an
equal-or-newer snapshot. The race predates the Q2 refresh triggers (they
only raised its frequency).

Deterministic concurrency test: 12 parallel refreshes against a probe
whose list grows per read - the emission sequence must be exactly 1..12.

Verified: Agent V2 Core 2059/2059.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 10:52:56 +02:00
Oliver Kieselbach 7842d121a0 fix(insights): codex review findings - WG pause clamping, late-event healing, stale buckets, date windows, banner count
All six review findings verified against code and real; five fixed here,
the agent-side race lands in the companion agent commit.

[P1] WhiteGlove pause counted as time: ClampToWindows clipped intervals
against the HULL of both WG windows, but the excluded pause lies inside
the hull - a reboot gap or app interval straddling the pause counted the
entire pause (a ~2-day "reboot" in a 30-minute enrollment was possible).
Seconds are now the per-window intersection sum (chronological endpoints
kept); an interval wholly inside the pause makes no claim and no longer
raises ClockSkewDropped. What-if savings are additionally capped at the
session's WallClockSeconds - endpoint deltas can still straddle the
pause, and no removal can save more than the enrollment took.

[P1] Late-arriving events never updated breakdowns: the terminal-seam
compute is one-shot and the sweep only healed missing/version-stale rows.
SessionTimeBreakdown now stores EventCountAtCompute (0 = pre-fix row,
recomputed once); the sweep recomputes on EventCount drift and then also
re-runs ResolveEspBlockingForSessionAsync (newly exposed on
IMetricsRepository) so late esp_config_detected re-emits and app batches
reach both the breakdown and the blocking evidence.

[P2] Stale aggregate buckets: the sweeps now reconcile every partition
they wrote - buckets that existed but were not regenerated (sessions
deleted, class left the window) are deleted via new targeted deletes;
rolling30 rows additionally get a 48h ComputedAt freshness filter at
read time, so a fully quiet tenant stops serving year-old numbers as
"last 30 days".

[P2] Date windows included an extra calendar day: both inclusive range
ends meant days=30 returned 31 keys and days=1 summed yesterday AND
today. InclusiveWindowStart(today, days) = today-(days-1) in both
response builders; the sweep's rolling30 source is filtered to the same
30 calendar days (its query window stays one day wider for backfill).

[P3] Device-history banner counted later sessions as "previous
enrollments" when viewing an older session: selectPriorSessions
(deviceHistoryPrior.ts) anchors on the session's own chain entry (live
sessions: new sessionStartedAt prop), sessionId tie-break mirroring the
server chain sort.

Verified: solution build clean, Backend 3480/3480 (+6: three WG-pause
golden fixtures, window-start facts, RK builder/delete consistency,
EventCountAtCompute roundtrip), Web tsc + vitest 633/633 (+6).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 10:52:39 +02:00
Oliver Kieselbach 4bc2435ec5 fix(agent): audit Q1/Q2/Q5 source fixes - timing re-arm, esp_config re-emits, serial trim
Q1 (AppInstallTiming): a terminal->active transition (IME retry path,
e.g. Error->Downloading or Skipped->Downloading) re-arms CompletedAtUtc
so the final terminal payload covers first-start -> last-terminal instead
of freezing at the first terminal state. Plus a negative-duration guard
in AppInstallTiming.DurationSeconds (inverted endpoints -> null, field
omitted) - observed in production as durationSeconds=-2342 on a
Skipped-then-retried user-scope app (session 66f1c5e1).

Q2 (esp_config_detected): the ESP blocking lists are read too early -
device lists grow progressively, user-scope S-<SID> lists appear only
after sign-in (88 of 89 prod sessions had an empty user list at last
emission). DeviceInfoHost now re-collects the ESP configuration as
phase-driven one-shots: apps sub-phase declaration (AppsDevice/AppsUser
phase_transition = first app activity) and AccountSetup detection.
The StartupEventGate keeps unchanged payloads silent (fingerprint covers
list contents), so only grown lists actually re-emit. F1 EspBlocking
resolution (latest list-bearing emission) benefits without backend change.

Q5 (HardwareInfo): WMI values (serial, manufacturer, model) trimmed at
the source via NormalizeWmiValue; whitespace-only collapses to the
"Unknown" sentinel the retry logic already treats as unresolved.

Attach-window investigation (11a3b67a): NO agent fix - hypothesis
disproved against two real agent logs (diag bundles db5ce007/2.0.1337,
4a9e9273/2.0.1379). "completed without started" first apps were already
installed (WinGet detection-only, zero install lines in AppWorkload.log);
pre-attach replay provably works (pattern-match log). PR0's
HasMeasuredDuration already handles the class server-side.
Duplicate app_install_completed (66f1c5e1): mechanism narrowed to an
in-process re-read of exactly one AppWorkload rotation window (<24h-old
lines, historic suppression not applicable) - documented as follow-up,
no fix without a bundle-bearing repro per lessons.md.

Verified: solution build clean, Agent V2 Core 2058/2058 (14 new tests),
DecisionCore 493/493. No ConfigVersion bump, no new event types,
decision rail untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:59:20 +02:00
Oliver Kieselbach 1bf37f6761 feat(insights): F3 regression radar - Wilson-gated rule hit-rate drift detection (PR6)
Analyze rules only (gather stats stay per-batch-deduped until Q6 is
fixed - never silently included). Per (tenant, rule) the radar compares
the trailing 7-day hit rate (FireCount/SessionsEvaluated from the daily
RuleStats rows) against the prior 28-day baseline and fires only when
the window has >=5 hit sessions and >=20 evaluated sessions, the rate
is >=2x baseline AND the Wilson 95% intervals are disjoint
(MetricsMath.WilsonInterval/RateIncreaseSeparated, hand-computed test
vectors). Suppressed for deleted rules, rules inside the CreatedAt
grace period (<35d), rules edited inside the window, empty baselines,
and the RuleRegressionRadarDisabled app-setting kill switch.

Alert episodes live as ruleregression|{ruleId} rows in the notification
tracker (third keyspace): AddEntity-409 dedup = one bell + one
RuleFrequencyRegression ops event (category Tenant, Warning, full
numbers, dual-registered in the web OPS_EVENT_TYPES) per episode;
numbers refresh while active without moving FirstNotifiedAt; episodes
re-arm under 1.5x baseline, on fire-stop, or via the tracker's 30d
retention prune (spec-intended reminder). On-fire dimension correlation
(osBuild/model/agentVersion/imeVersion; hit set from a RuleResults
range scan intersected with the window sessions) reports both shares
with the "correlated, not necessarily causal" wording contract or
honestly "no clear dimension concentration". Radar runs in the daily
maintenance pass after the rule-stats aggregation, anchored on
yesterday (whole days only).

Surfaces: regressions[] in the rule-stats response (MCP get_rule_stats
inherits it), tenant bell deep-linking to the rule card, and the
analyze-rules page gains a densified 30-day fire sparkline plus a
regression badge while an episode is active. OKF doc
backend/rule-regression-radar.md + index/log.

Backend 3474/3474 (27 new: Wilson vectors, every suppression branch,
window boundary dates, re-arm matrix, dimension gates, tracker
round-trips incl. tri-states, keyspace dedup/prune, wording pins),
web tsc + vitest 627/627, MCP tsc + 455 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:10:52 +02:00
Oliver Kieselbach 88da317bf4 feat(insights): F2 surfaces - device-history banner, fleet FTR panel, API routes, MCP tool (PR5)
Three read routes over the PR4 data: metrics/device-history
(MemberRead + tenant query param - one route serves the web banner AND
MCP, GA cross-tenant via ?tenantId= with a sessionId-based owner
fallback; history:null is a normal outcome), metrics/device-journeys
and global/metrics/device-journeys (days selector honored because FTR
counts are additive - window rate = ratio of summed daily rows; the
tenant-less global variant serves the "global" partition and discloses
repeatDevices:null). Attempt numbers are server-computed
(DeviceJourneyCalculator.ComputeAttemptNumber: chain position within
the journey; live sessions via a virtual non-successful attempt so the
redeploy and 30d-gap rules apply) - no consumer re-derives journey
semantics. Repeat-devices violator list: last journey >=2 attempts with
newest terminal ref in-window, attempts desc then recency, cap 10,
failure reason from the newest failed attempt (bounded point-reads).

Web: DeviceHistoryBanner on session detail ("Attempt N for this device
- View history", expandable terminal chain, durations strictly the
authoritative durationSeconds, WG/DevPrep/admin-marked badges) and the
Fleet Health "First-time-right" section (rate gated at >=20 completed
journeys with disclosed n, junk-serial exclusion note, Monday-UTC
weekly trend, attempt histogram, repeat devices) following the 7/30/90
selector. MCP: get_device_history (serial mode / fleet FTR mode,
role-aware routing, all roles - GA_FULL privilege snapshot extended
deliberately). Docs: OKF backend/device-journeys.md + index/log.

Backend 3447/3447 (14 new: attempt-number vectors incl. gap boundaries
and live sessions, additive sums, days clamping, repeat-device
gate/sort/cap), web tsc + vitest 627/627, MCP tsc + 455 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 08:38:17 +02:00
Oliver Kieselbach 1e6de47433 feat(insights): F2 journey grouping + DeviceHistories chains + daily FTR aggregates (PR4)
Device-key normalization (trim+casefold, junk-serial exclusion incl. the
agent's "Unknown" WMI sentinel), pure journey grouping (terminal set
Succeeded/Failed/Incomplete; journey ends at first success; >30d gap from
the previous attempt's CompletedAt starts a new journey; WG Pending stays
an open session, never an attempt), and the DeviceHistories table
(PK tenant, RK encoded normalized serial): terminal-only session-ref
chains capped at 20 with derived journey counts, written inline at the
session-terminal seam (all three terminal statuses, DeletionState-guarded)
and healed by a rolling 30d maintenance sweep. The sweep prunes chain refs
of deleted sessions tombstone-driven (SessionTombstones cover cascade AND
retention deletions), deletes emptied rows, and recomputes daily FTR
aggregates (DeviceJourneyAggregates, PK tenant/"global", RK date):
completed journeys, first-time-right count, attempt histogram and
junk-serial disclosure — additive counts, bucketed on the completing
success session's StartedAt date. Chain refs carry the authoritative
DurationSeconds verbatim (never CompletedAt-StartedAt) plus an
adminMarked flag. Both tables registered in Constants.All + offboarding
TenantPartitionTables; aggregates under the 180d retention block;
deliberately NO per-session deletion-manifest step for DeviceHistories.

Backend 3433/3433 (51 new tests: junk/normalization vectors, terminal
set, exact 30d gap boundaries, WG one-row-one-attempt, redeploy split,
deletion pruning, entity round-trips incl. row-key escaping, FTR
aggregation core), Agent 2044+493 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 08:14:13 +02:00
Oliver Kieselbach 6de21e821a feat(insights): F1 surfaces — session lane, fleet panel, API routes, MCP tool (PR3)
Insights expansion PR3 (tasks/insights-expansion-spec.md §F1 "Surfaces").
Completes F1: schema (PR1) -> persistence/aggregation (PR2) -> surfaces.

- Rolling range rows: the sweep now also writes "rolling30|{class}"
  aggregate rows — range statistics over the window's actual sessions. A
  median of per-day medians is not the range median, so the fleet panel
  reads these and is deliberately fixed to "last 30 days"; the daily rows
  stay in the payload for trends/MCP.
- Routes (all registered in EndpointAccessPolicyCatalog):
  GET sessions/{sessionId}/time-attribution (MemberRead + tenant query
  param, GA fallback via FindSessionTenantId; breakdown=null is SUCCESS —
  a missing lane is a normal state, not an error), GET
  metrics/time-attribution (MemberRead) and GET
  global/metrics/time-attribution (GlobalReadOrAdmin + tenantId filter),
  both returning { windowDays, classes (rolling), daily }.
- Web: TimeAttributionLane under the session-detail phase timeline —
  proportional stacked bar of segment totals incl. the explicit
  unattributed remainder (bar always sums to the session's authoritative
  duration), ESP-blocking apps with critical-path occupancy, reboot badge
  (cross-cutting, never a bar slice), quality-flag chips; fetched
  fail-soft for terminal sessions only. Fleet Health gains the "Time
  attribution" section: per-class stacked medians gated at n>=20
  ("insufficient data (n=...)" below), flagged/missing exclusions
  disclosed, and a top-blocking-apps table with class-scoped rows and
  "up to X (p75 Y)" what-if wording. Deliberate deviation: the spec's
  per-app trend column is omitted — per-day app rows are too sparse under
  the daily n>=5 gate; revisit once rolling history exists.
- MCP: get_time_attribution (one tool, session + fleet mode, role-aware
  routing like get_app_install_metrics, registered for all roles) — the
  privilege-leak-guard catalog snapshot extended deliberately.
- Docs: OKF concept doc backend/time-attribution.md (invariant, windows,
  segment mapping, positive-evidence blocking, Q7 reboot spans,
  persistence + surfaces) registered in index.md + log.md.

Backend 3382/3382 (+1 rolling-aggregate test), web tsc clean + vitest
627/627, MCP tsc clean + 455 passed. Post-deploy: web deploy +
MCP redeploy (new tool + docs corpus).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 18:33:16 +02:00
Oliver Kieselbach 55c898bb9c feat(insights): F1 breakdown persistence + daily attribution aggregates (PR2)
Insights expansion PR2 (tasks/insights-expansion-spec.md §F1 "Data & compute
changes"):

- SessionTimeBreakdowns table (PK=TenantId, RK=SessionId): span/reboot/
  blocking-app JSON columns, QualityFlags, AttributionVersion,
  EspAppsOccupancySeconds kept tri-state (absent column = unknown, never 0).
  Written once at the terminal single-writer seam (TableSessionRepository,
  next to the counter reconcile + EspBlocking resolution; Succeeded/Failed
  only — Incomplete deliberately stores no duration). Deleted with its
  session: new deletion-manifest step 16 (PkRkExact; tombstone moves to 19)
  plus tenant-offboarding partition wipe — no separate age sweep.
- Self-healing 30d maintenance sweep instead of a one-off backfill (rides
  the existing 2h tick + manual parity): backfills pre-deploy sessions,
  heals inline-compute misses, folds late-terminating sessions into their
  StartedAt-date aggregates, and enforces AttributionVersion purity — a
  stale-version breakdown whose events aged out counts as missing rather
  than mixing algorithm semantics (truthfulness rule 8). Skips sessions
  with a deletion cascade in flight.
- TimeAttributionAggregates table (PK=TenantId or "global",
  RK="{date}|{class}"): per enrollment class (whiteglove /
  device_preparation / self_deploying / user_driven — never mixed),
  median/p75/p90 per segment over the fixed six-segment stack (incl.
  unattributed; absent segment = honest 0), clean/flagged/missing counts
  disclosed (rule 7), rows written below the n>=20 UI gate on purpose
  (the UI needs the n — rule 4). 180d retention alongside UsageMetrics.
- What-if bound: WhatIfSavingSeconds implements the spec formula
  max(0, cpEnd - cpEndWithoutX) including the idle gap before X (the ESP
  waits for X to start, too); sole-interval fallback = own start. Per-app
  aggregate rows gate at >=5 sessions, top 20 by median interval, carry
  median/p75 of interval and saving ("up to" wording is the consumer
  contract).

Backend 3381/3381 (13 new tests: entity round-trips incl. tri-state,
aggregation bucketing/disclosure/gates, what-if vectors, class precedence,
manifest step-count pins 17/19), solution build clean, web untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 18:03:14 +02:00
Oliver Kieselbach e637317bdb feat(insights): F1 schema + time attribution calculator (PR1)
Insights expansion PR1 (tasks/insights-expansion-spec.md §F1, audit §0.5):

- AppInstallSummary + AppId / EspBlocking (bool?, tri-state) / AppIdCollision:
  sentinel-gated columns + mapper + both projections. appId adopted from app
  events (first-seen wins); a second appId under the same display name flags
  the name-keyed row as a collision (audit Q3) — in-batch and cross-batch
  (sticky in the reconcile). Collision rows leave every per-app fleet
  aggregate (app metrics payload, apps list, app analytics) with a disclosed
  exclusion count; the fleet DO rollup and per-app session drill-down keep
  them.
- EspBlocking resolution at the session-terminal single-writer seam
  (TableSessionRepository.UpdateSessionStatusAsync, next to the counter
  reconcile — covers ingest, admin marks, maintenance sweep, rule engine):
  joins the session's app rows against the LATEST list-carrying
  esp_config_detected emission as a positive-evidence set (audit Q2) —
  listed => true, absent => stays unknown, never false. EspBlockingSets
  parser handles truncation via the uncapped espTracked*Count fields.
- TimeAttributionCalculator (pure, AttributionVersion 1) + SessionTimeBreakdown
  models: exact partition of the session's authoritative DurationSeconds into
  device_prep / esp_apps / identity_hello / user_esp / desktop_handoff spans
  plus an explicit unattributed remainder — invariant sum(spans)+unattributed
  == wall clock enforced in every test. WhiteGlove = two observation windows
  in ONE session (part1 anchored on whiteglove_part1_complete, pause never
  attributed). Per-app blocking intervals from EVENT timestamps (first start
  -> LAST terminal, audit Q1 retry freeze), occupancy = merged union clipped
  to esp_apps. Reboot spans from event gaps around lastBootUtc (audit Q7),
  deduped per boot. QualityFlags: ClockSkewDropped, PartialObservation,
  BlockingSetUnknown, BlockingSetTruncated, WhiteGloveAnchorsIncomplete.
  Persistence (SessionTimeBreakdowns) and surfaces follow in PR2/PR3.

Backend 3368/3368 (35 new tests: 19 calculator golden fixtures + 16
identity/blocking/aggregate-exclusion), solution build clean, web untouched
(new response fields are additive).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 17:29:36 +02:00
Oliver Kieselbach 4eac7d042c fix(metrics): app duration/rate stats over real installs only — TerminalState column + out-of-order reconcile
Source-data audit (tasks/insights-expansion-spec.md §0.5) verified against
production: 17.9% of Succeeded AppInstallSummaries rows carry no duration —
78% are genuine skips (app_install_completed with payload state=Skipped,
"Update for X" WinGet policies not applicable) whose state the ingest
discarded, the rest are real installs whose start the agent never observed
(attach window). Averaging them as 0s understated avg install duration by
18% and distorted the slowest-apps ranking.

- AppInstallSummary.TerminalState (Installed/Skipped/Postponed/Error) from
  the closing event's payload state; sentinel-gated column + mapper +
  AppsDashboardProjection. Status stays "Succeeded" for compatibility.
- MetricsMath.IsSkipTerminalState/HasMeasuredDuration: durations computed
  over measured rows only; skips leave the failure/success rate (decision
  2026-07-26 — a skip is not an install attempt); per-app skipped/unmeasured
  counts + totals exposed; slowest ranking gates on measured installs.
  Applied consistently: app metrics payload, apps dashboard (list/analytics/
  time series/trend/versions/device models), SLA weekly/snapshot/breach.
- StoreAppInstallSummaryAsync: reconcile extracted to internal static
  ReconcileAppInstallSummaryWithExisting (test seam). Fixes Q4 (terminal
  batch before started batch left duration permanently unset) and hardens
  terminal status against late/replayed InProgress batches.
- Web: measured-only hints on app detail cards, skip suffix in apps list,
  fleet-health "measured installs" caption.

Backend 3333/3333 (20 new tests), web tsc clean, vitest 627/627.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 16:24:45 +02:00
Oliver Kieselbach c7ed3ec27d more safe SecureBootCertReader and section plan layout update 2026-07-25 12:48:27 +02:00
Oliver Kieselbach f311e8746b docs(trust): on-demand log collection in customer-facing claims + tracker dedup tests
The "Collect Logs" feature (b382e567) added a second opt-in path into hosted
diagnostics upload — the quick-config dialog on a session — and a new collection
trigger, neither of which the customer-facing pages described. The trust FAQ still
said hosted upload is enabled "in the settings UI", and the dialog itself named
hosted storage without stating that packages leave the tenant boundary, so the
"clearly marked disclosure" promise was not met on that path.

- CollectLogsButton: marked data-boundary disclosure in the quick-config dialog
  (leaves your Azure tenant, per-tenant isolation, blob-scoped write-only token,
  removal per Data Retention Days)
- privacy + about pages: both opt-in paths named; on-demand collection described
  (Admin/Operator of the own tenant, same content as the automatic package,
  delivered with the agent's next check-in, visible on the session timeline)
- docs repo counterpart in trust/security-faq.md (separate commit)

Distress-bell dedup: the TpmPssUnsupported one-shot bell had key-shape tests only.
New HardwareRejectionNotificationTrackerDedupTests pins the 409 dedup branch, the
fail-soft branch, the blank-input short-circuit, the disjoint hardware/tpmpss key
spaces, and the retention sweep — which also prunes tpmpss rows and thereby re-arms
that device's bell, now stated in the interface and implementation docs.

Backend tests 3313/3313, tsc clean, web vitest app/sessions 60/60.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 12:43:42 +02:00
Oliver Kieselbach e0b6ce83cf fix null warnings 2026-07-25 00:17:31 +02:00
Oliver Kieselbach 7f7b590db4 fix(web): mobile readability for rule cards + phase timeline overlap
Rule list cards (analyze rules, gather rules, IME log patterns) used a
single-line flex header where every badge was flex-shrink-0 and only the
title could shrink — on phone widths the title truncated to a few
letters. The header now wraps: on mobile the title drops onto its own
full-width row (order-last + w-full) under the badge row and may wrap;
on >=sm the layout is unchanged (title inline, truncated).

PhaseTimeline squeezed up to 8 flex-1 columns with 48px circles and
whitespace-nowrap labels into the viewport, so circles and label text
overlapped completely on mobile. The timeline now has a minimum width
of 80px per phase inside an overflow-x-auto container (swipe on phone,
no scrollbar on desktop) and labels wrap inside their own column
instead of overlaying neighbours.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-25 00:04:04 +02:00
Oliver Kieselbach 4c2c447a2e fix(web): dark-mode slate/hover-tint overrides + mobile wrap for script executions
Dark mode: globals.css was missing the entire slate family (the
"Incomplete" status filter chip rendered white-on-white) and all light
hover tints outside gray/indigo/cyan (hover:bg-purple-50 in the tenant
suggestion dropdown flashed near-white under light text). Added slate
bg/text/border mappings mirroring gray, plus dark equivalents for every
hover:bg-*-50/100/200 class in use (blue, purple, red, amber, yellow,
green, orange, sky, teal, emerald, slate, gray-200/300).

Mobile: ScriptExecutions rows were single-line flex with flex-shrink-0
right clusters, so badges and status text overlapped/overflowed on
narrow screens. Header counters, cycle card headers and item rows now
flex-wrap with gap spacing instead of space-x.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 23:46:03 +02:00
Oliver Kieselbach 8a794a1b52 fix(rules): cross-tenant GA rule edits no longer misroute to home tenant
A Global Admin editing a gather/analyze rule while viewing a FOREIGN
tenant (override) got a green "saved" banner, but the write landed in
the GA's OWN home tenant: the JWT-scoped rules/{type}/{ruleId} routes
resolve the tenant from the caller's token, and no cross-tenant write
route existed (reads had /api/global/rules/*, writes did not). The
reload re-read the foreign tenant and showed the unchanged rule.

Fix: symmetric cross-tenant WRITE routes mirroring the global reads.

Backend:
- GlobalRulesFunction: PUT/DELETE global/rules/gather/{ruleId} and
  global/rules/analyze/{ruleId}; tenant from ?tenantId= (400 if
  missing), same 1 MB guard + ValidateScopeAndEmitMode as the JWT
  variants, same service calls (toggle-partial/built-in-state/full-edit
  semantics are tenant-parameterized and identical).
- Catalog: 4 entries GlobalAdminOnly + TenantScoping.QueryParam — a
  write tier, so the delegated-read rescue never fires; only a platform
  admin may mutate another tenant's rules. No global CREATE route:
  create stays JWT-only and the portal hides Create in override.

Web:
- api.rules.globalGatherRule/globalAnalyzeRule helpers; both rule pages
  route save/toggle/mark-failed/notify/delete through a
  *RuleWriteUrl(ruleId) switch on isGlobalOverride.
- Analyze template-enable (a create) is blocked in override with a
  clear message instead of silently writing to the wrong tenant.

Tests: CrossTenantAccessTests guards the 4 routes are GA-only +
QueryParam and that a GA reaches the named target tenant. Backend
3295/3295, web vitest 627/627, tsc clean.

Note: the old bug planted orphan copies of foreign rules into the GA's
home tenant on each attempted cross-tenant edit; those are not
auto-cleaned.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 23:21:37 +02:00
Oliver Kieselbach 6c53dc33e7 feat(distress): tenant-visible TpmPssUnsupported devices + one-shot bell
TpmPssUnsupported distress reports (TPM cannot RSA-PSS-sign, Schannel drops
the client cert on Win11 25H2+, agent can never authenticate) were operator-
only, yet the remediation is tenant-side: TPM firmware update or device
replacement.

- New GET audit/tpm-pss-unsupported (MemberRead), aggregated per serial,
  mirroring audit/device-not-registered
- Bell notification "tpm_pss_unsupported" (Admin audience) once per
  (tenant, serial); dedup via the existing rejection tracker table with a
  disjoint "tpmpss|{serial}" row-key space; no webhook by design
- Web: TpmIncompatibleDevicesInsights panel in the Hardware Whitelist
  section; renders only when a device is affected (no permanent card)

No agent change, no ConfigVersion bump. Backend tests 3287/3287, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 20:47:06 +02:00
Oliver Kieselbach 801891d57a layout update section plan comparison 2026-07-24 20:30:28 +02:00
Oliver Kieselbach 1f6efca95b fix(backend): stamp DiagnosticsBlobName from on-demand server_action_executed events
Session 8e4cc4ae: two "Collect Logs" round trips completed on the agent (both
ZIPs landed in hosted storage) but the portal never flipped to Download and
the button hung at "Collecting..." — the ServerActionDispatcher confirms an
on-demand upload only via server_action_executed(actionType=request_diagnostics,
blobName=...) and emits NO diagnostics_uploaded (only the terminal upload path
does), so UpdateSessionDiagnosticsBlobAsync never ran.

The ingest classification now treats such an executed event as the upload
confirmation (a real diagnostics_uploaded in the same batch still wins — it
carries destination). Because the executed event has no destination field,
InferDiagnosticsDestination derives "Hosted" from the backend-built
"{tenantId}/{filename}" blob-name shape (CustomerSas blobs are bare
filenames); without it the download route would look in the customer's
container for a hosted blob.

Backend-only — works with the already-deployed agent fleet, no agent release.
Backend suite 3274/3274.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 19:23:47 +02:00
Oliver Kieselbach 9ec8644be2 feat(settings): operator read-only settings view with real values
Operators previously saw editable settings forms bound to hardcoded defaults
whose Save always 403'd. Now they see the tenant's real configuration
read-only, with no save affordances at all.

Backend:
- GET config/{tenantId}: TenantAdminOrGlobalReader -> MemberRead. Safe because
  the handler redacts by default (CanViewSecrets: only GA or own-tenant Admin
  see secrets); Operators/Viewers receive RedactedCopyForReader().
- POST diagnostics/files: TenantAdminOrGA -> TenantAdminOrOperator (submitting
  logs is a troubleshooting action; Viewer and read-only GlobalReader excluded).
- Tests: policy pins updated; new middleware tests (own-tenant Operator allowed
  + redacted, cross-tenant Operator 403); CanViewSecrets theory for
  Operator/Viewer.

Web:
- TenantConfigContext loads the full config for all settings users (403
  fallback to feature-flags minimal config for deploy-order safety) and
  exposes canEditConfig; saveConfiguration guards on it.
- New ReadOnlyFieldset wraps section content (native fieldset disable, values
  stay copyable); SaveResetBar is not rendered at all for read-only viewers.
  Applied to Contact, SLA Targets, Hardware Whitelist (+ rejection-insights
  toggle), Notifications, Agent Settings, Agent Analyzers, Diagnostics, Data
  Management.
- Hidden entirely from Operators (in-page admin-only card + sidebar sub-item
  gate): Autopilot Validation, Access Management, Offboarding, Unrestricted
  Mode. GlobalReader sidebar parity kept via hasGlobalScope.

Tests: backend 3257/3257, web vitest 627/627, tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 18:40:27 +02:00
Oliver Kieselbach 9acd10e1dc docs(okf): ServerActions & on-demand diagnostics concept doc
Covers the queue-on-session/deliver-on-ingest channel, the
TenantAdminOrOperator tier with in-function per-type re-gating, the portal
Collect Logs flow (quick-config orders rotate_config before
request_diagnostics), sequence-based progress matching, and the hosted-prefix
sweep in the offboarding cascade. Registered in index.md, logged in log.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 18:15:23 +02:00
Oliver Kieselbach b382e5678e feat(web): "Collect Logs" on-demand diagnostics collection on the session detail page
Queues a request_diagnostics ServerAction (delivered with the agent's next
ingest response) and tracks progress live: server_action_received flips the
button to "Collecting…", the DiagnosticsBlobName SignalR delta completes it
(Download Diagnostics appears), server_action_failed or a 3-min quiet timeout
surface as toasts.

- Button is ALWAYS rendered — disabled states carry tooltips (inactive
  session, cross-tenant view, missing role, diagnostics not configured) as a
  deliberate nudge towards enabling diagnostics upload.
- Tenant Admin on an unconfigured tenant gets a quick-config dialog: switches
  the tenant to Hosted + OnFailure (GET config -> patch two fields -> PUT
  verbatim), then queues rotate_config BEFORE request_diagnostics so the agent
  collects against the fresh config. Operators/Viewers never see the dialog.
- Operator may collect when configured (new TenantAdminOrOperator backend tier).
- useSessionTenantConfig picks up diagnosticsUploadConfigured from the
  member-readable feature-flags endpoint.
- Progress matching is sequence-based (backend-assigned) so agent clock skew
  and at-least-once action redelivery cannot produce false matches.

Pure state logic in collectLogsLogic.ts with vitest matrix (21 tests).
tsc clean; web suite 627/627.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 18:12:49 +02:00
Oliver Kieselbach 2fee50276e feat(backend): operator-tier session actions + diagnostics groundwork for on-demand log collection
- New EndpointPolicy.TenantAdminOrOperator (Admin/Operator/GA, no Viewer) with
  middleware evaluator; POST sessions/{sessionId}/actions moves to this tier so
  troubleshooting operators can queue request_diagnostics ("Collect Logs").
- QueueSessionActionFunction re-gates per type: terminate_session and
  rotate_config stay Admin/GA-only (403 for operators).
- GET config/{tenantId}/feature-flags exposes diagnosticsUploadConfigured so
  members below Admin can see whether an on-demand upload would work.
- Offboarding cascade: hosted diagnostics deletion now sweeps the session
  prefix {tenantId}/AgentDiagnostics-{sessionId}- (plus the manifest-named
  blob) so on-demand packages overwritten on the Sessions row by the terminal
  package no longer survive deletion. CustomerSas branch unchanged.
- Notification dismiss routes deliberately NOT migrated: dismissal is still
  tenant-shared; operators would clear admin-tier notifications they cannot see.

Tests: middleware role matrix on the actions route, per-type gate matrix,
feature-flag matrix, prefix-sweep + real hosted-branch cascade tests.
Backend suite 3253/3253.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 18:05:39 +02:00
Oliver Kieselbach 8bb02acc4a fix(web): bump postcss to >=8.5.12 (GHSA-6g55-p6wh-862q, high)
Dependabot alert #74: PostCSS <= 8.5.11 allows arbitrary file read /
information disclosure via an attacker-controlled sourceMappingURL in CSS
comments. Raise the floor (devDependency + override) from ^8.5.10 to
^8.5.12; the lockfile now resolves postcss 8.5.22 (its nanoid dep moves
3.3.11 -> 3.3.16 with it). npm audit reports 0 vulnerabilities.

Next.js pins postcss 8.4.31 transitively, but the existing root override
collapses the tree to a single postcss node, so the bump covers it too.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:54:57 +02:00
Oliver Kieselbach fd7d3973e7 fix(agent): esp_policy_provider_stalled gains sidecar-missing arm (issue #106)
Customer follow-up corrected the field-case mechanics: TrackingPoliciesCreated=1
WAS set under the foreign ConfigMgr provider - the ESP ignores a foreign
provider's completion values and waits on the Intune-registered "Sidecar"
provider by name (renaming the key unblocks instantly without creating any
tracking entries). The completeness-keyed detector would have missed exactly
the incident it was built for.

- Arm 2: >=1 Setup\Apps provider registered without Sidecar, continuously for
  the 15-min dwell -> reason=sidecar_provider_missing; clock resets when
  Sidecar registers later (legitimate startup ordering); DevicePreparation-only
  snapshots don't activate it
- Arm 1 (continuously-incomplete providers) unchanged -> reason=provider_incomplete;
  sidecar-missing wins when both arms cross in one round (single event)
- Cross-restart fingerprint gains a "sidecarMissing" sentinel; payload adds
  reason + sidecarMissingForMinutes
- Regression test: ConfigMgr complete + Sidecar absent -> fires (previously
  classified healthy); probe/doc headers and OKF doc corrected, web event-type
  description updated

Evidence (research session c7e3db08): Intune's ESP-policy WAP payload always
registers Sidecar (ConfigMgr only via co-management settings policy, per
Niehaus' documented payload); Microsoft's ESP troubleshooting models app
tracking exclusively through the Sidecar subkey.

Agent tests 2043/2043, web tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:48:50 +02:00
Oliver Kieselbach fd2b5c2da2 fix(ci): drop invalid SWA input, gate deploy on token via env guard
Azure/static-web-apps-deploy@v1 has no `skip_deploy_on_missing_secrets`
input (confirmed against its action.yml), so passing it did nothing but
emit an "Unexpected input(s)" warning — the intended "skip green on
Dependabot PRs" behaviour was never actually in effect.

Replace it with the documented mechanism: surface the deploy token as a
job-level env var and gate the deploy/close step with `if: env.DEPLOY_TOKEN
!= ''` (secrets cannot be referenced directly in an `if:`). On Dependabot
PRs the token is absent, so the step skips and the job stays green; real
deploys on main and internal PRs still run. Applied to both jobs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:38:58 +02:00
Oliver Kieselbach a44e37a7c2 fix(metrics): keep full failure-reason text for fleet-health display
BuildFleetFailureReasons truncated each reason to a 50-char prefix + "..."
and used that as both the grouping key AND the display value. The full text
never reached the UI, so its tooltip/tap-to-expand had nothing to reveal and
near-identical rows were indistinguishable past the cut.

Group by the 50-char prefix as before (variants still collapse into one row),
but carry the longest full reason as the display value. The UI truncates and
expands it on demand, so the complete text is now available end-to-end.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:25:11 +02:00
Oliver Kieselbach d8a252562d fix(web): pin engines.node to 22.x so SWA/Oryx does not grab Node 24
The prior ">=22" made Oryx build with the highest available Node (24.13),
which the Static Web Apps runtime rejects: "Node version 24.13.0 is not
supported. Please use one of the following versions 18, 20, 22." Pin to
22.x so Oryx selects the SWA-supported major. CI setup-node '22' already
resolves to latest 22.x, so build and deploy now agree.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:23:17 +02:00
Oliver Kieselbach a5d12ba1d0 chore(ci): unify Node toolchain on 22 across CI + engines pins
Node 20 reached EOL (2026-04-30); Node 22 is the current maintenance LTS
and already the runtime the MCP container ships (node:22-slim). Straighten
the version skew so per-component CI major == deploy runtime == engines floor:

- ci.yml: web, mcp and guardrails setup-node 20 -> 22
- combine-rules.yml: setup-node 20 -> 22
- mcp package.json: add engines.node ">=22" (matches Dockerfile)
- web package.json: add engines.node ">=22" (first explicit pin; SWA/Oryx
  no longer picks a Node version implicitly)

engine-strict stays off (no .npmrc), so the pins are advisory and cannot
hard-break local installs on other Node versions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 16:15:54 +02:00
Oliver Kieselbach 530c978bf8 fix(web): TruncatedLabel expands to full text, not just two lines
The expanded state used line-clamp-2, which re-truncated long values with an
ellipsis after two lines — unreadable on touch where the hover tooltip is
unavailable. Expand now wraps the entire string over as many lines as needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 15:11:54 +02:00
Oliver Kieselbach 09898d7ba0 feat(web): consistent truncate-with-tooltip + tap-to-expand across dashboard cards
Add reusable TruncatedLabel component: a single-line label that truncates
with an ellipsis, shows the full text as a native hover tooltip, and on
click/Enter/Space expands to two wrapped lines (touch-friendly). An
interactive={false} mode provides tooltip-only behaviour for use inside
already-clickable rows/links where an inner onClick would hijack navigation.

Apply it consistently to the long, dynamic label columns across:
- fleet-health: failure reasons, model + app ranked lists
- sla: top failing apps, violators failure reason
- geographic-performance: fastest/slowest location cards
- vulnerability exposure panel: CVE sample software
- agent metrics: top exception types
- progress + download/install: app-name labels
- admin software tabs: software names
- session reports (comment/adminNote) + ops events (message): tooltip-only

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-24 15:03:34 +02:00
Oliver Kieselbach 71eda93e8b fix(metrics): app-install failure rate over finished installs only
Second half of the terminal-only rate convention (follows 1cfc32c9, which
converted the enrollment success rates): app-install failureRate divided
by ALL install rows including "InProgress" ones — installs still running,
or orphaned when a session died mid-install — diluting every failure rate
customers see on the Apps dashboard and Fleet Health.

- New MetricsMath.TerminalFailureRatePct (Failed / (Failed + Succeeded),
  0 when nothing finished) shared by both aggregation sites
- MetricsMath.BuildAppMetricsPayload: per-app failureRate terminal-based
  (Fleet Health "Top Failing Apps"; subtitle now "% of N finished")
- AppsAnalyticsHelper: list, analytics summary, versionBreakdown,
  timeSeries and deviceModelBreakdown rates terminal-based; the
  duplicated trend block is deduplicated into ComputeFailureTrend, whose
  rates and >= 5-sample gate now run on finished installs per half; the
  device-model >= 5 sample floor likewise counts finished installs
- Web: InstallsTab "Avg Failure Rate" card computed over summed finished
  installs instead of totalInstalls

Tests: new AppsAnalyticsFailureRateTests (list/trend/analytics/model
cases incl. in-flight dilution and gate regressions) and a terminal-rate
case in MetricsMathAppPayloadTests. Backend 3223/3223, tsc clean,
vitest 602/602.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 10:01:28 +02:00
Oliver Kieselbach 1cfc32c9aa fix(metrics): success rate over finished enrollments only (geo + fleet health)
Customer feedback: the geographic performance pane showed a poor success
rate for a site mid-rollout because in-progress/pending sessions counted
against the rate. Success rate is an outcome quota — align Geo and Fleet
Health with the SLA convention already used by dashboard stats, usage
snapshots and the fleet rollup: Succeeded / (Succeeded + Failed).
Incomplete (terminal, non-failure) stays excluded.

- Geo: per-location SuccessRate over finished sessions; table badge and
  map tooltip render an em dash instead of a red 0% when nothing has
  finished yet
- Fleet Health: Stats.SuccessRate terminal-based; card shows "X of Y
  finished enrollments" and a neutral em dash state; per-model health
  gets a Failed count (additive wire field) so the client rate excludes
  in-flight devices; TopFailingModels FailureRate now failed/finished
- Tests: FleetHealthPayloadTests flipped to the new convention plus
  zero-finished, model-failed and in-flight-dilution cases; new
  GeographicMetricsSuccessRateTests pin the geo semantics

Backend 3216/3216, web tsc clean, vitest 602/602.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 09:47:31 +02:00
Oliver Kieselbach 11339c9868 fix(test): de-flake SessionDeletionWorkerTests via condition-gated runs
CI run 30047512186 failed Worker_drops_envelope_missing_required_fields_
without_invoking_handler: RunForAsync starts the BackgroundService poll loop
and waits a fixed 500ms wall-clock window — on a loaded CI runner (xUnit runs
test classes in parallel) the loop may not have processed the message before
the positive assertions (DeleteMessageAsync at least once) run. All ten
positive tests in the file shared that latent race; this run it hit the
malformed-envelope drop test.

Fix mirrors the watchdog event-gate de-flake: new Harness.RunUntilAsync
(condition, 10s ceiling, 25ms poll) runs the worker until the awaited side
effect is observable, then stops. Gates: ops-event recorded (last step of the
poison path, so send+delete+CAS are already done), main-queue delete for the
drop tests, >=3 visibility heartbeats for the heartbeat test. The kill-switch
test keeps the fixed window — a Times.Never expectation has no event to wait
for. OpsEventCalls capture is now lock-guarded so the condition poll can read
it while the worker thread appends.

16/16 worker tests green across 4 consecutive runs; full backend suite
3211/3211.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-24 00:03:10 +02:00
Oliver Kieselbach 540f543834 feat(agent): extend historic IME-replay suppression to app events + app aggregation
Part 2 of the eaf3d8c4 fix (follows 088c3b04). In that session 147 of 157 app
events were replayed from a previous enrollment's IME log — all 10 "installed
apps" were phantoms, polluting the timeline, AppInstallSummaries, MCP counts,
the app_tracking_summary header, culprit lists and final-status.json.

Rename (nothing deployed yet): historic_script_replay_detected →
historic_ime_replay_detected; SuppressIfHistoricReplay / IsHistoricImeReplay.

Agent:
- EmitAppState suppresses stale-replay transitions BEFORE the sub-phase
  declaration, timing bookkeeping and terminal-signal dedup — covers the
  informational event, the AppInstallCompleted/Failed DecisionSignal, the
  shadow download_progress and the summary snapshot. Behavior note: phantom
  terminal signals can no longer spuriously re-arm the AdvisoryCompletion
  window (app signals are otherwise observation-only — verified no completion
  gate reads them). EmitDoTelemetry gets the same guard.
- Tracker-level guard at the HandlePatternMatch choke point skips the 23
  app-mutating actions (incl. espphasedetected — a stale AccountSetup line
  would make fresh DeviceSetup lines bounce as "backward") for source lines
  >24h stale, so replayed apps never enter PackageStates, persistence,
  app_tracking_summary, culprit lists or final-status. New internal
  UtcNowProvider seam (no ctor change), threshold single-sourced as
  ImeLogTracker.HistoricReplayThreshold.
- SimulationMode now bypasses all replay suppression — 088c3b04 had broken
  the --replay-log-dir dev tool, which feeds historic lines on purpose.

Backend: IsHistoricImeReplay guard in AggregateAppInstallEvent keeps replayed
events out of AppInstallSummaries (SLA/dashboards/metrics inherit).

Web: generic lib/historicReplay.ts partition (scriptExecutions becomes a
wrapper); InstallProgress hides replayed installs with a muted note;
DownloadProgress filters silently.

MCP: get_session_summary filters replayed events — cleans appInstalls stats,
errorCount (replayed app_install_failed are Error-severity) and keyEvents.

Tests: +13 agent (app suppression, tracker guard), +1 backend, +7 web,
+6 MCP. Agent 2039, backend 3211, web 602, MCP 455 all green; drift test
validates the rename.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 23:43:59 +02:00
Oliver Kieselbach 088c3b0479 fix(agent): suppress historic IME-log script replay + timeline-consistent durations
Session eaf3d8c4: a previous enrollment's IME logs survived on disk (dated
2026-07-16, 7 days old) and the agent replayed them at startup — 156 phantom
script_completed events in a 7-minute session, each showing ~170h durations
(clock-clamped completion minus RAW stale start), plus an immediateUpload
flood (3x ingress_backpressure) and RemediationScriptCount inflated to 156.

Agent:
- Script events whose source line is >24h stale are suppressed entirely; a
  one-shot historic_script_replay_detected Info event (new EventType) dates
  the replayed window via earliestRejectedSourceTimestamp. Future-skew
  rejections and clock-only fallbacks still emit.
- durationSeconds now uses a timeline-consistent pair: on rejected-timestamp
  emissions the raw source completion pairs with the raw start (true runtime
  instead of cross-timeline garbage). 24h plausibility backstop omits
  implausible values; same consistency for script_timeout_suspected.
- Stale pending platform-script slots (>24h older than the current start
  line) are discarded so an old orphaned start can't pair with a fresh run
  (extracted HandlePlatformScriptStarted as internal test seam).

Web: partitionHistoricScriptEvents hides legacy-agent replay events (muted
"N historic executions hidden" note) and the reducer blanks >24h durations —
covers sessions recorded before the agent fix rolls out.

Backend: IsHistoricScriptReplay guard keeps replayed events from inflating
Platform-/RemediationScriptCount during the rollout window.

Registered historic_script_replay_detected in Constants.EventTypes, MCP
resource-catalog (script_events) + KEY_EVENT_TYPES, gather-rules eventTypes.

Tests: 8 agent (suppression/one-shot/raw-pair/backstop/slot-hardening),
8 web (partition/blanking), 10 backend (predicate). Agent 2026, backend
3210, web 595, MCP 449 all green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 18:39:11 +02:00
Oliver Kieselbach 3224774752 fix(test): EDR-safe straggler replay — no more renamed system executable
The straggler integration test copied powershell.exe under a synthetic
agent name and launched it — textbook MITRE T1036.003 ("Rename Legitimate
Utilities"). Defender for Endpoint flagged the run ("System executable
renamed and launched", initiating process testhost.net48.exe) and ISOLATED
the dev machine.

Replaced with function shadowing: a harness script defines Get-Process /
Stop-Process stub functions (PowerShell resolves functions before cmdlets)
and dot-sources the cleanup script. The stubs report a live agent-named
straggler on every probe and record the kill, pinning the identical
contract — poll by NAME until the deadline, force-kill by NAME, then
delete — with zero process spawns, zero PE copies, nothing EDR-visible.

The production routine never renames or copies executables; the flagged
technique existed only in this test. Lesson recorded in tasks/lessons.md.

22/22 cleanup tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:24:07 +02:00
Oliver Kieselbach e7ea431ccf feat(esp): recovery story for terminally reported ESP failures (session 4910a5a5)
DeviceSetup/Apps failed, the user pressed "Try again" ~23 min later, apps
recovered to 10/10 and DeviceSetup resolved - yet the advisory_completion
window un-defanged the stale failure and failed the live enrollment,
because the advisory variant was exempt from all re-arm/rebase guards.

Engine fix (DecisionCore):
- New facts EspAdvisoryFailureResolvedUtc + EspAdvisoryFailureCategory
  (state-borne; deadline payload would be lost on re-arm/rebase).
- TryResolveAdvisoryOnCategoryRecovery in both ProvisioningComplete
  handlers: category match sets the fact + emits
  esp_failure_advisory_resolved.
- Guards now (!hasAdvisoryAnchor || advisoryResolved): enforcement-progress
  re-arm, hello-never-observed promote and the reboot rebase apply again
  once the failure demonstrably un-happened; unresolved advisories keep
  the legacy un-defang exactly.
- Truthful terminal reason esp_recovered_without_completion_evidence
  (+ failureRecovered=true, LastFailureTrigger=DeadlineFired so the
  likely-stuck app promotion stays off).

Story events (agent tracker):
- esp_failure_retry_detected: failed subcategory leaves "failed" after the
  terminal fire - consistent with the user pressing "Try again".
- esp_failure_recovered: previously failed category reaches success.
- EspFailureDetectedEventArgs.LikelyCulpritApps (never-started first,
  cap 3) rides the EspTerminalFailure payload so esp_failure_advisory /
  enrollment_failed name the culprit app directly.

Surfacing: MCP resource-catalog, web timeline badges (Recovered /
User retry pills, culprit chip on advisory), gather-rule autocomplete,
ANALYZE-ESP-006 "ESP Failure Recovered After User Retry".

Tests: AdvisoryRecoveryTests (13, incl. the 527/611/635/671 field-ordinal
replay that no longer fails the session), EspFailureRecoveryStoryTests (5),
snapshot-builder guards 18 -> 20. All suites green (DecisionCore 493,
V2.Core 2018, Backend 3201, web vitest 589, MCP 449 incl. drift).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 17:03:44 +02:00
Oliver Kieselbach 9e24b92fbf feat(agent): ESP policy-provider stall detector + Autopilot registry guardrail
Customer field case: pre-provisioned devices with the ConfigMgr client as a
blocking app occasionally park the user ESP at "Apps (Identifying)" for days.
Root cause (verified against the EnrollmentStatusTracking CSP doc): only a
"ConfigMgr" provider is registered under Device\Setup\Apps\PolicyProviders
(no "Sidecar" = IME) and TrackingPoliciesCreated never becomes 1 - a wait the
CSP defines WITHOUT any timeout. Renaming the key to Sidecar unblocks
immediately. Microsoft documents that ESP providers are not aware of each
other and that pre-provisioning is unsupported with co-management.

New always-on esp_policy_provider_stalled tripwire, generalized to any
provider in both kinds (Setup/Apps TrackingPoliciesCreated != 1,
DevicePreparation InstallationState not in {2,3}):

- EspPolicyProviderProbe: fail-soft registry read of the
  EnrollmentStatusTracking mirror (device + user-SID scopes), defensive
  value parsing (missing/unparseable = incomplete), ScopedOverride test seam.
  Research and CSP contract quotes captured in the XML doc header.
- EspPolicyProviderStallDetector: pure EvaluateStallTransition on a 15-min
  continuous-incompleteness dwell (resets on completion/disappearance;
  absent key = normal early enrollment), one-shot per provider,
  StartupEventGate fingerprint over the sorted stalled key set for
  cross-restart dedup. Warning + ImmediateUpload via InformationalEventPost;
  purely observational, no decision-engine coupling, no config toggle,
  no ConfigVersion bump (disk_space_low precedent).
- EspPolicyProviderStallHost: 60-s wall-clock tick, deliberately outside the
  StallProbe idle gate (idle clock resets on session activity) and the
  idle-stopped periodic collectors.
- Registered in Constants.EventTypes, MCP resource-catalog (health_events),
  web gather-rule autocomplete (stall category).

Guardrail companion: SOFTWARE\Microsoft\Windows\Autopilot joins the
gather-rule registry allowlist ("Autopilot / OOBE / Setup") so tenants can
build custom rules on the subtree (e.g. not_exists on the Sidecar provider
key); generated web mirror regenerated via combine.js; positive + sibling-
spoof guard tests added.

Docs: OKF concept doc agent/esp-policy-provider-stall.md (+ index/log).

Tests: 22 new detector tests (pure transition matrix + emission surface incl.
the ConfigMgr-only customer scenario and gate-across-restart dedup); agent
suite 2013/2013, web vitest 589/589 + tsc clean, MCP 449 passed incl.
event-types-drift.

Both the detector and the embedded allowlist take effect with the next agent
release.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 15:20:47 +02:00
Oliver Kieselbach 45d7c03ccf fix(web): WhiteGlove per-block durations use sequence-canonical edges, not timestamp min/max
Customer report (session fcb44595): dashboard said 115 min, session detail
said 5h 2m for the same pre-provisioned enrollment. The detail's per-block
duration calc took min/max over ALL event timestamps inside each
sequence-block, which two real-world timestamp shapes break:

- an app_install_started re-emitted by the ImeLogTracker after the Part-2
  resume legitimately carries its historical Part-1 event time -> the User
  Enrollment block stretched back across the sealed pause (56m -> 3h18m)
- a do_telemetry with a +1h source timestamp (IME log line with a wrong
  timezone bias, just under the agent's 1h future-skew clamp) stretched
  Pre-Provisioning (58m -> 1h44m)

Sum: exactly the "5h 2m" the customer saw.

Block spans are now anchored on the first/last event BY SEQUENCE (sequence
is the canonical order), with Part 1 starting at session.startedAt for
backend DurationSeconds parity. Extracted as pure helper
computeWhiteGloveDurations with tests modeled on the real session; replay
against the customer's 880 events yields 1h 54m == backend 6872s.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:26:33 +02:00
Oliver Kieselbach 90012d9a66 feat(agent): verify Secure Boot CA-2023 state against firmware, not just servicing registry
Customer report: ANALYZE-SEC-001 warned on devices whose firmware already
carried the Windows UEFI CA 2023 certificate — the agent only read the
SecureBoot\Servicing registry value, which mirrors the Windows Update
rollout and lags behind reality (OEM factory image, WinCS, WU not run yet).

Agent: new UefiSecureBootCertReader reads the UEFI db/KEK variables
natively (GetFirmwareEnvironmentVariableExW; SeSystemEnvironmentPrivilege
enabled on the process token and restored — first privilege-adjustment
code in the repo). Fail-soft, never throws; legacy BIOS maps to not_uefi.
secureboot_status gains per-certificate booleans, uefiFirmwareReadStatus,
and the one-sided marker uefiCa2023FirmwareConfirmed (emitted only on
positive confirmation, never as false).

Rule v3.0.0: not_exists precondition on the marker suppresses the rule
when the firmware confirms the certificate; the marker's absence keeps
old agents on exact v2 behavior, so the rule is safe to seed before the
agent rollout. Firmware-verified-missing raises confidence (+20 db,
+10 KEK 2K CA 2023).

Verified: full solution build; backend 3201, agent 1988, web 582 tests
green; tsc/ajv clean; elevated cross-check on real firmware matches
Get-SecureBootUEFI on all seven certificate probes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 14:07:01 +02:00
Oliver Kieselbach e8ef4f53ae feat(agent): OOBE-state observability via WinRT SystemSetupInfo
Sample Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState
(build >= 17763, SYSTEM-callable, CLR-native WinRT projection — no NuGet)
as a purely observational signal, mirroring the bootstrap guard marker:

- oobeStateAtAgentStart on agent_started and agent_late_start payloads:
  makes late starts provable instead of heuristic ("completed" on a
  first_run = OOBE was already over when the agent arrived) and validates
  the bootstrap OOBE guard fleet-wide.
- One-shot oobe_state_completed trace event on the in_progress->completed
  flip, piggybacked on the DesktopArrivalDetector 30s poll: an
  owner-independent desktop corroboration for DAD-starved sessions
  (4d5a0b78 class — WTS+WMI owner resolution dead, desktop_arrived never
  fires). Often absent in healthy sessions since the DAD timer is
  disposed at desktop arrival, which is when the flip lands.

OobeStateReader is fail-soft (never throws, "unavailable" on missing
contract); the flip check has its own try/catch so it cannot pollute the
DAD wmiErrorCount liveness diagnostics, "unavailable" never overwrites
the baseline, and the one-shot flag survives ResetForRealUserSwitch.
No decision-engine involvement, no config/backend changes. Event type
registered in Constants.EventTypes, MCP resource-catalog (phase_events)
and web gather-rule autocomplete.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 12:39:10 +02:00
Oliver Kieselbach a0d10b8471 test(cleanup): end-to-end incident replays for the self-destruct script
Six integration tests execute the generated cleanup script with a real
powershell.exe against GUID-unique scratch trees — proving what the string
pins cannot:

- locked tree ⇒ complete abort, marker + binaries survive (the 2026-07-22
  regression: the old script force-deleted around the locks)
- full incident replay: a renamed second process holds a file in the tree,
  is found by NAME, force-killed after the wait deadline, cleanup then
  completes with nothing left behind
- fresh foreign lock ⇒ loser backs off without touching the tree; stale
  lock (>10min) is taken over and cleanup proceeds
- happy path + keepLogs variant

BuildCleanupScript gains test-only timing parameters (defaults unchanged
for production) so the replays run in seconds. Agent suite 1969/1969.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 10:06:29 +02:00
Oliver Kieselbach ad3632e9fb fix(emergency-break): survivable self-destruct, reliable break report, ops visibility
Incident 2026-07-22: the 48h emergency break left exe + loaded DLLs orphaned
forever — a second agent instance held the images while the cleanup script
(waiting on the launcher PID only) force-deleted marker + state around the
locks AFTER removing the Scheduled Task, destroying the retry path.

Agent — cleanup script (LIFE-F7):
- wait for ALL agent processes by name (PID-reuse-proof), then force-kill
  stragglers; single-flight lock file serializes concurrent cleanup scripts
- probe-rename (GUID target) BEFORE task removal; any failure renames back
  and aborts with task + marker + files intact for a next-boot retry; the
  destructive Remove-Item-on-original fallback is gone
- script generation extracted to BuildCleanupScript, pinned by 13 tests;
  all 4 variants parser-validated + E2E-smoked (happy path, locked tree)

Agent — break report delivery:
- EmergencyReporter counts DELIVERED reports: reservation rolls back on
  failure (mark-before-send permanently suppressed a key after its first
  transport failure); optional in-reservation retries
- break path: NIC wait <=15s, 3 attempts x 15s timeout (survives Functions
  cold starts observed as >5s 503s), 75s overall bound

Backend:
- backend-materialized events (agent_emergency_break, session_timeout) now
  upsert the cross-session EventType index — they were invisible to every
  search-by-eventType surface (portal + MCP)
- new AgentEmergencyBreak ops event (Agent/Warning, once per session) so
  operators see silently-lost agents; dual-registered in OPS_EVENT_TYPES
- materialization extracted to a static seam, covered by 4 new tests

Agent 1963/1963, Backend 3198/3198, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:56:46 +02:00
Oliver Kieselbach 05e1569503 feat(agent): wildcard support for Local Admin Analyzer allowed accounts
Allowed Local Accounts entries may now contain glob wildcards (* / ?),
matching the hardware-whitelist semantics: Regex.Escape conversion,
case-insensitive, 100ms ReDoS timeout. Applies to both the account and
the profile-folder check. Entries without wildcard chars keep exact
matching; * and ? are invalid in Windows account names, so patterns and
literals cannot collide. UI documents the syntax and rejects patterns
with fewer than 2 literal characters (a bare * would disable the
analyzer). No schema change - old agents treat patterns as literals and
keep alerting (fail-safe).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:47:13 +02:00
Oliver Kieselbach ba3581e253 bootstrapper added download stopwatch 2026-07-23 09:35:30 +02:00
Oliver Kieselbach 829ec4c969 fix(release): stamp Cache-Control no-cache on every blob upload
The first release after the download-alias migration exposed Front Door
caching as a correctness hazard: the alias kept serving the previous
version.json (X-Cache: TCP_REMOTE_HIT) after the origin had rotated to
2.0.1370, pinning the portal's version badge to the old build. The dangerous
variant is worse than a stale badge -- manifest and ZIP are independent cache
entries, so a stale manifest paired with a fresh ZIP fails the bootstrap's
SHA-256 integrity check and the install dies.

Route caching on /agent/* has been disabled (infra side); this commit is the
defense-in-depth half: every upload now carries x-ms-blob-cache-control
no-cache -- the CI Send-Blob (eu and legacy mirror alike), the local build.ps1
counterpart (gitignored, changed in lockstep), and the versions/*.json
manifest publishes. Uniform no-cache rather than an immutable/mutable split
because every blob in the agent container rotates in place, including the
per-line AutopilotMonitor-Agent-v2.zip -- nothing there is safely cacheable.
Existing blobs get the header with the next release's re-upload.

Log/docs updated in url-registry.md; no code paths changed, YAML and ps1
parse-checked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:17:02 +02:00
Oliver Kieselbach b2d2d37c6f feat(realmjoin): detect aborted RJ ESP (missing 110) and release the completion gate
An interactive logon during the RealmJoin first deployment (phase 100/101)
makes RJ reclassify the run as a secondary-user deployment, whose branch
writes 200/210 and never CompletedFirstDeployment (110) — no crash, no
reboot request. The V2 completion gate waited for exactly 110, so these
sessions hung until the 60-min hard timeout (session 224b2087; siblings
558bf067/3cc18724 delayed ~50 min the same way).

- promote realmjoin_phase_changed to a typed RealmJoinPhaseChanged signal
  (dual-emitted by the adapter) so RealmJoinFacts.LastDeploymentPhase is
  persisted across agent restarts
- abort rule: prev in {100,101} and current in {200,210} with no outcome
  yet -> emit realmjoin_first_deployment_incomplete (Warning), cancel the
  hard-timeout deadline, release the gate with outcome
  FirstDeploymentIncomplete; also evaluated on the RealmJoinDetected
  replay path (restart swallows the transition)
- deliberately NOT triggered by 101->0 (RJ service restart may still
  complete the first deployment) nor by a first observation of 210
  (session 6f1959c0: RJ already completed before agent boot)
- fix realmjoin_timeout always claiming "last phase: 0" - the persisted
  phase now feeds the message
- backend EnrollmentTimeoutClassifier counts the new event as a gate
  terminal; MCP resource catalog dual-synced
- CompleteIfDeferredOrBookkeep widened to a leading-effect list

Verified by replaying session 224b2087's signal log: gate releases at
15:05:17 (101->200) instead of hanging until the 15:54 deadline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:14:30 +02:00
Oliver Kieselbach e0ad51b965 refactor(urls): one guard-enforced URL registry per component, agent downloads move to the alias
Every well-known host -- own subdomains, both blob accounts, the API origin,
Graph, Entra login -- now lives in exactly one registry per component:
Constants.cs (*BaseUrl) for C#, utils/config.ts for the web app, src/config.ts
for the MCP server. lib/hostRouting.ts derives its hostnames from the registry
and middleware.ts imports from it, retiring what turned out to be a fourth
independent copy of the www/portal host pair.

Three new guard tests (backend xUnit, web vitest, MCP vitest) scan non-test
source and fail on any enforced host outside its registry. This is the lesson
of the EU cutover: the registry existed then too, but LatestVersionsService
(feeds the portal's version badge) and ValidateBootstrapCodeFunction carried
their own literal copies of the legacy blob host and were missed -- kept
working only by the fail-soft legacy mirror in build-agent.yml, whose silent
failure would have pinned badge and self-update to stale bits with a green
release. Both guards proved themselves on their first run, flagging the
middleware host copy and a literal in an MCP tool description. Test files stay
exempt on purpose: literals there are independent oracles, and
ValidateBootstrapCodeResponseShapeTests now pins the full alias URL against
the Constants-built one so registry changes must be acknowledged consciously.

The two legacy readers plus the agent SelfUpdater now use
Constants.AgentDownloadBaseUrl (download.autopilotmonitor.com/agent); the
alias was verified live for version.json, version-v2.json and both ZIPs
before the switch. The portal's bootstrap allow-list accepts the alias AND
the legacy host (AGENT_DOWNLOAD_HOSTNAMES) for already-deployed customer
scripts, which imposes a deploy order: web before or with backend. The
legacy keepalive probes and the release mirror stay untouched -- they exist
for exactly this transition. Also: the bootstrap short-link now targets www
directly instead of riding the registrar's apex 301, and the welcome-mail
docs button points at docs. instead of bouncing through www./docs.

Suites: backend 3194, agent core 1925, DecisionCore 474, web 582, MCP 449 --
all green; next build verified (it, not tsc, proves the next.config.ts
registry import). Docs: docs/url-registry.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 08:37:42 +02:00
Oliver Kieselbach bbb55034e3 feat(versioning): one version contract for agent, backend and MCP
All three components now report <major>.<minor>.<build> plus a 7-char commit
and a build timestamp. The backend had been pinned at a hand-written 1.0.0
since inception and starts at 1.5.1000; the MCP server's package.json version
was bumped by hand and starts at 1.5.300, and it now reports its own commit on
/health instead of only the documentation bundle's.

The build number keeps coming from a counter blob reserved before the build
with ETag-CAS, but there is now one implementation instead of two copies:
.github/scripts/Request-BuildNumber.ps1, called by all three workflows and by
the local build.ps1. It lives under .github/ because scripts/Deployment/ is
gitignored (those scripts carry SAS tokens) and would therefore be absent from
a CI checkout; scripts/Build/ and scripts/Release/ are shadowed by the
[Bb]uild/ and [Rr]elease/ artifact ignores.

The counter cannot be replaced by the published manifest, which is the obvious
simplification to reach for: version.json is written only on stable publish, so
two consecutive dev builds would derive the same number and the second would
overwrite the first's artifact. git rev-list --count fails the same way, two
builds of one commit collide.

Both deploy workflows now poll their health endpoint until it reports exactly
the version and commit just deployed, and publish versions/<component>.json
only then. A green deploy step alone means the package was accepted, not that
it is being served -- so a manifest that disagrees with the live endpoint now
means the running instance drifted (rollback, stale revision) rather than a
pipeline that stopped halfway. The agent manifest gains the same commit and
buildUtc fields additively; the bootstrap reads it by field name, so older
installers ignore them, and sha256/bootstrapVersion stay untouched.

Requires the versions container plus AZURE_BLOB_VERSIONS_SAS_TOKEN and the two
seeded counters (backend 999, mcp 299) before the first deploy run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 17:41:03 +02:00
Oliver Kieselbach 58c8cf87a4 updated command 2026-07-22 15:38:02 +02:00
Oliver Kieselbach 71d2c0e47f fix(maintenance): remove the time-window auto-block, it never measured data volume
GetExcessiveDataSendersAsync gated on LastEventAt - StartedAt alone. No event
count, no bytes, no rate — the name asserted something the code never checked.
What it actually selected was every enrollment session that spanned a night,
which for Autopilot is the ordinary case: device unboxed in the evening, ESP
runs, machine powered off, user signs in the next morning.

The per-device alerts from the previous commit made this visible. All three
devices blocked at the time of the analysis were false positives:

  CND3320949 / 64ed21e8    23 events over 15.5h  (1.5 events/hour)
  5CG4303CPT / db5751bc   192 events over 16.8h  (blocked mid Account Setup 1/5)
  PF5YANFB   / 2069de87   781 events over 31.5h

14 such blocks in 30 days across 9 customer tenants. Over the same window the
event-count detector fired 5 times and was right every time (2742, 2714, 2712,
2506 events). Each false block costs a live enrollment 8 hours of telemetry.

Repairing it rather than removing it does not pay: once a volume condition is
added, "many events AND long span" is a subset of "many events", which is what
the count detector already does. Volume is the quantity that costs anything; a
23-event session spanning 15 hours costs nothing. So the detector, its two admin
settings, the ExcessiveDataBlocked ops event and the portal section go, and the
event-count path is the only automatic block left.

MaxSessionWindowHours and MaintenanceBlockDurationHours are deleted rather than
deprecated: the config row is written with Replace, so the columns drop out on
the next portal save, and nothing reads them meanwhile. A comment on the
surviving detector records why there is no second one, so it does not get
rebuilt a year from now.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 15:28:28 +02:00
Oliver Kieselbach 3b2f5ed46d feat(runaway-detection): name the device in the time-window alert, link it in the block list
The two runaway detectors emitted alerts of unequal usefulness. The event-count
path fires per session and carries sessionId + serialNumber, so the Ops Events
detail modal can offer View session / Block / Kill. The time-window path
aggregated a whole tenant into one event with {devicesBlocked, windowHours} —
no device, no session, no shortcuts, on precisely the alert where an operator
wants to act. It now writes one event per blocked device with the same payload
shape; the aggregate count stays in the audit entry, where a sum is the useful
view.

The Active Blocks table showed the serial as plain text although the API already
returns BlockedSessionIds. Maintenance blocks now link the serial to the session
that triggered them; manual whole-device blocks have no session and stay text.

Two defects in the event-count path, found while comparing the detectors:

- No isBlocked guard, so the auto-action re-blocked an already-blocked device and
  overwrote BlockedAt/UnblockAt/Reason of the existing block — usually the
  time-window block on the same session, whose reason then vanished from the list.
- No status filter, so a session that finished days ago with a high EventCount
  still cost its device a 24h block. The fix needed a layer below the obvious one:
  the runaway query never selected Status, and SessionStatus.InProgress is
  ordinal 0, so a guard on the unpopulated field would have been silently
  always-true. ParseStatusForAutoAction is stricter than MapToSessionSummary --
  Enum.TryParse maps "0" onto InProgress and ORs comma-separated names into an
  unrelated member, both of which would open the gate.

Not addressed, and deliberately: a block is session-scoped and stops the upload,
so whichever detector fires first freezes the other for that session. With
MaxSessionWindowHours=12 the time detector wins in practice. That is correct
behaviour, not a defect; merging the detectors would be its own change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:43:00 +02:00
Oliver Kieselbach 7329f12217 ci(codeql): skip analysis on documentation-only changes
CodeQL analyzes C# and TypeScript, so a commit touching only docs or markdown
cannot change its result - it spent ~10 minutes (8:50 for C#, 1:32 for
TypeScript) re-reporting an unchanged outcome.

paths-ignore on push and pull_request only. The weekly cron stays unfiltered:
it exists to catch advisories published against code that did not change, and
it doubles as the backstop should the filter ever skip too much.

A commit touching docs AND code still runs - GitHub skips only when every
changed path matches the filter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:10:18 +02:00
Oliver Kieselbach c9a9df99f1 docs(ci): describe what the CI workflow actually does
The header claimed it "runs the full test suite" - it also typechecks the web
and MCP projects and fails on drift between rules/guardrails.json and the
generated rule files. It also called itself "the gate that decides whether a
change is allowed to be correct", which was both garbled and false: main is
deliberately unprotected, so a red run reports rather than blocks.

Comment only, no behaviour change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:10:17 +02:00
Oliver Kieselbach 757427ffea chore(build): clear the compiler and xUnit analyzer warnings CI annotates
The CI test job surfaced 10 annotations, but GitHub caps them at 10 per job —
a full rebuild showed 31 nullable warnings plus 26 xUnit analyzer warnings.

Production code:
- CertificateValidator: X509Certificate2(byte[]) is obsolete (SYSLIB0057) ->
  X509CertificateLoader.LoadCertificate.
- AuthenticationMiddleware: IsValidTenantId now carries [NotNullWhen(true)], so
  the guard above BuildTenantAuthority is visible to flow analysis.
- AgentEndpointMigrationRules: the out params are genuinely nullable on the
  false path; netstandard2.0 has no usable [NotNullWhen], so the contract is
  documented instead.
- Constants.ClassifyEspAppsFailure: errorCode is optional by design (the
  "no HRESULT" branch is the timeout fallback) - signature says so now.

Tests: nullable annotations on helper signatures, Assert.Single/DoesNotContain
overloads instead of pre-filtering with Where, and [InlineData(null)] theories
declare nullable parameters. GatherRule helpers now seed Parameters with the
model default (empty) rather than null, matching a deserialized rule.

No behaviour change. 5616 tests pass; solution builds warning-free apart from
the two pre-existing NU1510 pruning notices on the pinned security packages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:09:11 +02:00
Oliver Kieselbach 71832b9deb docs(about): list hardware rejection among the webhook alert types
The integrations list named start, success, failure and SLA alerts but omitted
hardware rejection, understating what tenants can actually subscribe to.

Verified before claiming it: WebhookNotifyOnHardwareRejection is wired
end-to-end - model, portal toggle in SectionHardwareWhitelist, and dispatch in
ReportDistressFunction. This list occurs nowhere else in the repo or in
autopilotmonitor-docs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:31:49 +02:00
Oliver Kieselbach ce4d884abc test(gather-rules): drive the collectors through their guards
The guard functions had unit tests, but nothing proved that Registry, Wmi,
Json, Xml and File actually consult them before touching the registry, WMI or
disk. These drive the collectors themselves: each blocked target must come
back empty, emit exactly one security_warning naming the target, and produce
no rule output. Prefix-spoofing included, plus a positive case through
JsonCollector so it is pinned that the guard is a gate and not a wall.

Path traversal had no explicit coverage either. The sharpest case is
...\Logs\..\..\..\..\Secret.txt - the RAW string starts with an allowed
prefix, so a naive StartsWith would admit it; the test pins that
Path.GetFullPath canonicalizes first and the allowlist sees C:\Secret.txt.
Also traversal into the C:\Users privacy block and into System32\config, both
asserted under unrestrictedMode, which lifts the allowlist but never those.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:31:49 +02:00
Oliver Kieselbach b56c774103 fix(logging): strip Unicode line separators in LogSanitizer
The sanitizer matched \p{Cc}, which covers the C0/C1 control ranges but not
U+2028 LINE SEPARATOR or U+2029 PARAGRAPH SEPARATOR - those are categories
Zl/Zp. A log viewer, a JSON-lines consumer or any JavaScript-side reader still
breaks a line on them, so they were a residual log-forging vector.

Matching \p{Zl} and \p{Zp} rather than listing escapes: the two categories
contain exactly those two characters, and it keeps the source free of
invisible literals.

Verified the test is not vacuous - the old pattern left "a<U+2028>b" as "a b",
the new one yields "ab".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:31:48 +02:00
Oliver Kieselbach 68767f436f fix(config): validate ContactEmail and seed it conditionally
The tenant contact address had no server-side validation, so a direct API
caller could store any string in the field service notices are sent to.
UpdateTenantConfiguration now rejects non-addresses, recipient lists (a comma
would silently widen who receives those notices), display-name forms and
control characters (mail-header injection), and normalizes what it stores.
Empty stays valid - having no contact address is a legitimate state.

The seed path had a worse problem than missing validation. It was a
read-modify-write of the whole model through SaveTenantConfigurationAsync,
which upserts with TableUpdateMode.Replace and no ETag: a concurrent portal
save was clobbered by the seeder's stale snapshot of EVERY field, not just
ContactEmail. It also wrote straight through IConfigRepository, bypassing the
TenantConfigurationService cache, so a seeded address stayed invisible for the
full 5-minute TTL.

TrySeedTenantContactEmailAsync now writes ETag-conditionally in Merge mode -
one property, and a 412 means someone else wrote the row so the seed abandons
rather than overwriting the winner. A lost seed is recoverable on the next
maintenance run; a lost admin edit is not.

The same seed logic existed twice (preview notification-email save and the
maintenance backfill). Both now call a single owner on
TenantConfigurationService, which holds the never-overwrite invariant, the
cache eviction and the fail-soft guarantee. Fail-soft lives in the service
rather than the repository so it holds for any implementation.

Tests cover the validator, the service contract, and the repository against a
mocked TableClient: Merge mode, the ETag actually passed, only ContactEmail in
the patch, the 412 race, 404, storage faults. Plus a regression pin that
SaveTenantConfigurationAsync is never called from the seed path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:31:30 +02:00
Oliver Kieselbach c7b06be8b7 fix(ratelimit): key the user throttle on CallerId, not the UPN
Follow-up to 479247cd. Publishing only a genuine UPN closed the fleet-wide bucket
collapse but opened a narrower hole: AuthenticationMiddleware validates issuer,
audience, lifetime and signature but requires neither upn nor scp, so an app-only
(client-credentials) token authenticates and reaches every AuthenticatedUser route
(feedback, realtime/negotiate, progress/*, health/detailed, auth/mcp, ...). Such a
token carries no upn, so it resolved to an empty identity and UserRateLimitMiddleware
skipped it entirely — previously it at least landed in the shared anonymous bucket.

Rather than reinstating a placeholder as a bucket key, separate the two concepts:

- RequestContext.UserPrincipalName keeps its meaning — a real UPN or nothing. Audit,
  presence and admin flows continue to see only genuine user identities.
- RequestContext.CallerId is the throttle identity: UPN, else the token subject
  (oid -> sub -> appid/azp), else a fail-closed shared bucket. Empty means exactly one
  thing — no JWT at all (device/anonymous routes, which carry their own limits).
- UserRateLimitMiddleware gates and buckets on CallerId.

This matches how McpQuotaEnforcementMiddleware already keys its quota (oid), so the
subject-based identity is the existing house convention for enforcement.

10 new tests: the oid/sub/appid/azp fallback chain and its precedence, the fail-closed
bucket for a token with no identifying claim (asserting it is never "anonymous"), and
that empty CallerId occurs only on unauthenticated routes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:17:34 +02:00
Oliver Kieselbach 479247cdae fix(ratelimit): stop agent fleet sharing one anonymous user bucket
Device and anonymous routes published the "anonymous" placeholder as
RequestContext.UserPrincipalName. UserRateLimitMiddleware skips only on an EMPTY UPN, so
every agent of every tenant was bucketed under user_ratelimit_anonymous and throttled
against the 120/min per-user limit. On 2026-07-21 21:00-21:30 UTC that rejected 1821
agent requests with 429 (/api/agent/telemetry, config, register-session, error); the
7-day trend shows it recurring and growing with fleet size.

- PolicyEnforcementMiddleware: RequestContext.UserPrincipalName carries a genuine JWT
  UPN or nothing. decision.UserIdentifier keeps the placeholder for log lines.
- Device/bootstrap routes already hold their own limit in SecurityValidator (per cert
  thumbprint / per bootstrap token), so they lose nothing.
- stats/platform and diagnostics/download had NO limiter of their own and were riding on
  the accidental shared bucket. Both now limit per trusted client IP (60/min and 30/min)
  via ClientIpExtractor, matching the HealthCheck pattern.
- 13 new tests pin the empty-UPN contract across all 12 device/anonymous routes, plus a
  guard that an authenticated caller still surfaces its real UPN.

Solution suite 5552 tests green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 09:48:33 +02:00
Oliver Kieselbach b6ba87f072 fix(deps): pin patched DataProtection + Cryptography.Xml to clear NuGet advisories
Microsoft.Identity.Web 4.3.0 pulls Microsoft.AspNetCore.DataProtection and
System.Security.Cryptography.Xml transitively at 10.0.0, which carry known
advisories (GHSA-9mv3-2cwr-p262 critical on DataProtection; GHSA-37gx-xxp4-5rgx
and siblings, high, on Cryptography.Xml). Add explicit references pinning both
to the patched 10.0.10 servicing build (matching the existing JwtBearer pin) —
the NuGet equivalent of the npm overrides used for the web/MCP alerts.

Pinned in the production Functions project so the fix flows to the deployed
artifact and to Functions.Tests via project reference. `dotnet list package
--vulnerable` now reports 0 across every project; build clean; 3110 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:44:44 +02:00
Oliver Kieselbach fa640ebe61 fix(security): resolve CodeQL log-forging findings and suppress by-design queries
Bucket B (real fixes) + Bucket A (config) of the CodeQL alert triage.

Log-forging (cs/log-forging, 8 sinks): add Helpers/LogSanitizer that strips
CR/LF and other control characters, and apply it to the logged copy of every
user-controlled value in AuthenticationMiddleware, PolicyEnforcementMiddleware
(Invoke + DecideAsync), McpQuotaEnforcementMiddleware and TableUserUsageRepository.
Only the logged value is sanitized — routing, authorization and storage keys are
untouched, so tracing fidelity for legitimate values (paths, UPNs, GUIDs) is
preserved. Structured logging already isolated the fields; this closes the CRLF
vector as defense-in-depth and clears the query.

Three genuine JS/TS fixes:
- guardValidation.ts: replace the no-op `.replace(/%/g, "%")` with a real
  escapeRegExp() — the author intended to escape; this also closes the latent
  metacharacter gap (js/identity-replacement).
- access-guard.ts: use a `%s` format argument instead of interpolating the UPN
  into the console.error format string (js/tainted-format-string).
- docs-corpus.ts: strip HTML tags with a loop-until-stable stripHtmlTags() so the
  sanitizer is robust and recognized as complete (js/incomplete-multi-char-sanitization).

CodeQL config: filter the two queries that fire structurally on this codebase,
where the flagged behavior is the feature — cs/exposure-of-sensitive-information
(audit/notification logs legitimately record the actor UPN / recipient email) and
cs/cleartext-storage-of-sensitive-information (rate limiters key on the trusted
client IP). Default suites stay active; genuine one-off false positives from other
queries are dismissed individually so those queries stay sharp.

Tests: Backend xUnit 3110 (+9 LogSanitizer), MCP vitest 19 (+1 tag-strip),
Web vitest 49, all tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:38:47 +02:00
Oliver Kieselbach dbc20a9f66 test(deletion-maintenance): de-flake watchdog tests via event gates
The three watchdog tests raced a fixed fanout-body sleep against sub-second
warn/severe thresholds (e.g. 150ms body vs 500ms severe). On a loaded CI
runner the whole RunCoreAsync body overran the severe threshold before the
finally-block cancelled the watchdog, so the severe OpsEvent fired and
"LongRunning past warning" flaked (3100/3101).

Replace the wall-clock race with event gates: the fanout body now blocks on
WaitForEventAsync(<watchdog OpsEvent>) instead of sleeping, so the target
watchdog is guaranteed to have fired before the body returns. The opposite
threshold is pushed to minutes (warn-only tests) or the body waits on the
severe event (both-fire test) so warn is necessarily first. OpsEvents
recording is now lock-guarded since watchdog tasks record concurrently.

No production change. Watchdog class ran 5x green (16/16); full suite 3110/3110.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:33:26 +02:00
Oliver Kieselbach cbda62d131 Merge branch 'main' of https://github.com/okieselbach/Autopilot-Monitor 2026-07-22 01:19:36 +02:00
Oliver Kieselbach 3951000b40 ci(deploy-web): skip SWA deploy when token absent instead of failing
Dependabot PRs run without repository secrets, so AZURE_SWA_DEPLOY_TOKEN is
empty and Azure/static-web-apps-deploy exits with an error, painting a red
"Build and Deploy Job" check on every web-touching Dependabot PR. Setting
skip_deploy_on_missing_secrets makes the action skip the deploy (green) when
the token is missing; real pushes to main still carry the token and deploy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:19:26 +02:00
dependabot[bot] c59842262e chore(deps): bump the web group across 1 directory with 10 updates (#104)
Bumps the web group with 10 updates in the /src/Web/autopilot-monitor-web directory:

| Package | From | To |
| --- | --- | --- |
| [@microsoft/applicationinsights-web](https://github.com/microsoft/ApplicationInsights-JS) | `3.3.11` | `3.4.3` |
| [@xyflow/react](https://github.com/xyflow/xyflow/tree/HEAD/packages/react) | `12.10.2` | `12.11.2` |
| [fflate](https://github.com/101arrowz/fflate) | `0.8.2` | `0.8.3` |
| [next](https://github.com/vercel/next.js) | `15.5.18` | `15.5.21` |
| [@next/bundle-analyzer](https://github.com/vercel/next.js/tree/HEAD/packages/next-bundle-analyzer) | `16.2.7` | `16.2.11` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `22.19.11` | `22.20.1` |
| [autoprefixer](https://github.com/postcss/autoprefixer) | `10.4.24` | `10.5.4` |
| [eslint](https://github.com/eslint/eslint) | `9.39.2` | `9.39.5` |
| [eslint-config-next](https://github.com/vercel/next.js/tree/HEAD/packages/eslint-config-next) | `15.5.12` | `15.5.21` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `4.1.2` | `4.1.10` |



Updates `@microsoft/applicationinsights-web` from 3.3.11 to 3.4.3
- [Release notes](https://github.com/microsoft/ApplicationInsights-JS/releases)
- [Changelog](https://github.com/microsoft/ApplicationInsights-JS/blob/main/RELEASES.md)
- [Commits](https://github.com/microsoft/ApplicationInsights-JS/commits)

Updates `@xyflow/react` from 12.10.2 to 12.11.2
- [Release notes](https://github.com/xyflow/xyflow/releases)
- [Changelog](https://github.com/xyflow/xyflow/blob/main/packages/react/CHANGELOG.md)
- [Commits](https://github.com/xyflow/xyflow/commits/@xyflow/react@12.11.2/packages/react)

Updates `fflate` from 0.8.2 to 0.8.3
- [Release notes](https://github.com/101arrowz/fflate/releases)
- [Changelog](https://github.com/101arrowz/fflate/blob/master/CHANGELOG.md)
- [Commits](https://github.com/101arrowz/fflate/compare/v0.8.2...v0.8.3)

Updates `next` from 15.5.18 to 15.5.21
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/compare/v15.5.18...v15.5.21)

Updates `@next/bundle-analyzer` from 16.2.7 to 16.2.11
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/commits/v16.2.11/packages/next-bundle-analyzer)

Updates `@types/node` from 22.19.11 to 22.20.1
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `autoprefixer` from 10.4.24 to 10.5.4
- [Release notes](https://github.com/postcss/autoprefixer/releases)
- [Changelog](https://github.com/postcss/autoprefixer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/autoprefixer/compare/10.4.24...10.5.4)

Updates `eslint` from 9.39.2 to 9.39.5
- [Release notes](https://github.com/eslint/eslint/releases)
- [Commits](https://github.com/eslint/eslint/compare/v9.39.2...v9.39.5)

Updates `eslint-config-next` from 15.5.12 to 15.5.21
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/commits/v15.5.21/packages/eslint-config-next)

Updates `vitest` from 4.1.2 to 4.1.10
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.10/packages/vitest)

---
updated-dependencies:
- dependency-name: "@microsoft/applicationinsights-web"
  dependency-version: 3.4.3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: web
- dependency-name: "@xyflow/react"
  dependency-version: 12.11.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: web
- dependency-name: fflate
  dependency-version: 0.8.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: next
  dependency-version: 15.5.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: "@next/bundle-analyzer"
  dependency-version: 16.2.11
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: "@types/node"
  dependency-version: 22.20.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: web
- dependency-name: autoprefixer
  dependency-version: 10.5.4
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: web
- dependency-name: eslint
  dependency-version: 9.39.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: eslint-config-next
  dependency-version: 15.5.21
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: vitest
  dependency-version: 4.1.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:15:43 +02:00
dependabot[bot] c3e9621a78 chore(deps): bump the mcp group across 1 directory with 5 updates (#103)
Bumps the mcp group with 5 updates in the /src/McpServer/autopilot-monitor-mcp directory:

| Package | From | To |
| --- | --- | --- |
| [@modelcontextprotocol/sdk](https://github.com/modelcontextprotocol/typescript-sdk) | `1.28.0` | `1.29.0` |
| [fuse.js](https://github.com/krisk/Fuse) | `7.1.0` | `7.5.0` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `22.19.15` | `22.20.1` |
| [tsx](https://github.com/privatenumber/tsx) | `4.22.4` | `4.23.1` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `4.1.2` | `4.1.10` |



Updates `@modelcontextprotocol/sdk` from 1.28.0 to 1.29.0
- [Release notes](https://github.com/modelcontextprotocol/typescript-sdk/releases)
- [Commits](https://github.com/modelcontextprotocol/typescript-sdk/compare/v1.28.0...v1.29.0)

Updates `fuse.js` from 7.1.0 to 7.5.0
- [Release notes](https://github.com/krisk/Fuse/releases)
- [Changelog](https://github.com/krisk/Fuse/blob/main/CHANGELOG.md)
- [Commits](https://github.com/krisk/Fuse/compare/v7.1.0...v7.5.0)

Updates `@types/node` from 22.19.15 to 22.20.1
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `tsx` from 4.22.4 to 4.23.1
- [Release notes](https://github.com/privatenumber/tsx/releases)
- [Changelog](https://github.com/privatenumber/tsx/blob/master/release.config.cjs)
- [Commits](https://github.com/privatenumber/tsx/compare/v4.22.4...v4.23.1)

Updates `vitest` from 4.1.2 to 4.1.10
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.10/packages/vitest)

---
updated-dependencies:
- dependency-name: "@modelcontextprotocol/sdk"
  dependency-version: 1.29.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: mcp
- dependency-name: fuse.js
  dependency-version: 7.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: mcp
- dependency-name: "@types/node"
  dependency-version: 22.20.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: mcp
- dependency-name: tsx
  dependency-version: 4.23.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: mcp
- dependency-name: vitest
  dependency-version: 4.1.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: mcp
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:15:10 +02:00
dependabot[bot] 3ed21c37d1 chore(deps): bump actions/upload-artifact from 6 to 7 (#102)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 6 to 7.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:08:23 +02:00
dependabot[bot] 4a1194569e chore(deps): bump actions/checkout from 5 to 7 (#91)
Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 7.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v5...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:07:14 +02:00
dependabot[bot] d0794fc7c5 chore(deps): bump actions/setup-node from 5 to 7 (#90)
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 5 to 7.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v5...v7)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:06:47 +02:00
dependabot[bot] f34c9d2322 chore(deps): bump github/codeql-action from 3 to 4 (#89)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3 to 4.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:06:44 +02:00
dependabot[bot] 380a8417c8 chore(deps): bump actions/setup-dotnet from 5 to 6 (#88)
Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 5 to 6.
- [Release notes](https://github.com/actions/setup-dotnet/releases)
- [Commits](https://github.com/actions/setup-dotnet/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-dotnet
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:06:40 +02:00
Oliver Kieselbach 63826a4869 chore(dependabot): stop auto-PRing npm major bumps
Group config already batches minor/patch; majors fall outside the group and
each opens its own PR (zod 4, typescript 7, transformers 4, msal-react 5,
react-leaflet 5, react 19, @types/node 26, eslint-config-next 16 — all
breaking). Ignore version-update:semver-major for both npm ecosystems so
majors are reviewed deliberately instead of flooding the PR list. Security
updates are exempt from this filter, so vulnerability fixes still flow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:03:22 +02:00
Oliver Kieselbach 7df208fac9 fix(deps): pin patched transitive deps via overrides to clear Dependabot alerts
Force the security-patched versions of transitive dependencies without
downgrading their parents (next 15, @modelcontextprotocol/sdk 1.28), which
is what `npm audit fix` would otherwise do:

- sharp ^0.35.0   (web via next, mcp via @huggingface/transformers) — libvips CVEs
- fast-uri ^3.1.4 (mcp via ajv) — host confusion
- hono ^4.12.27   (mcp via sdk) — jsx context/XSS/header advisories
- @hono/node-server ^2.0.5 (mcp via sdk)

npm audit: 0 vulnerabilities in both projects. Builds clean; MCP 447 tests
and Web 572 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 00:58:37 +02:00
Oliver Kieselbach 3f9576d4a9 fix(agent): enforce the gather-rule allowlist in every collector
LogParserCollector and EventLogCollector called no guard at all, while
File/Json/Xml did — and rules/gather-rules.md already promised customers
that log-parser paths were allowlisted. A tenant admin could author a
logparser rule against C:\Users via POST /api/rules/gather and have up to
1000 lines uploaded as telemetry, bypassing the hard C:\Users block the
same page guarantees; eventlog could read any channel, Security included.
The backend validates no collectorType, target, or parameters, so the
agent is the only place this can be enforced.

Both now guard. logparser cannot check rule.Target directly (GetFullPath
throws on a wildcard), so it checks the directory for wildcard targets and
re-checks each resolved path. Event log channels get an allowlist matched
on a "/" boundary. Hard-blocked channels live in a C# constant rather than
guardrails.json: a parse error drops the allowlists to empty, which is
fail-closed, but would have lifted a block sourced from the same file.

Also fixes an inert timeout in CommandCollector: ReadToEnd() blocks until
the pipe closes, so WaitForExit(30000) was only reached once the process
had already exited and a hung command pinned its worker indefinitely.

Adds the first real CI gate. No workflow ran tests on a pull request — the
deploy workflows build without testing and build-agent.yml covered only
DecisionCore.Tests on manual dispatch — while trust/security-faq.md
already told customers automated tests run in CI. ci.yml now runs all six
suites on every PR and merge, plus a job that fails on generated-guardrail
drift, alongside CodeQL and Dependabot. OfficeBinaryWatcherTests is marked
SerialThreading; its bounded re-probe races the shared ThreadPool and
failed once under full-suite load.

Verified: 3101 backend, 1911 agent V2, 474 decision core, 44 dialog, 572
web, 447 MCP. The timeout fix is verified by runtime, not assertion — a
Start-Sleep 120 command now returns after ~30s.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 00:28:04 +02:00
Oliver Kieselbach ee58242d33 fix(mcp): tell callers to query the English-only corpora in English
A German question returns zero results from search_docs, not poor ones: both the
documentation and the rules are English, and all-MiniLM-L6-v2 is an English-only
model, so the query vector lands nowhere near any chunk. For a German-speaking
admin driving an MCP client in their own language, the tool is effectively blind
and gives no hint why.

The caller is itself a language model, so the cheapest correct fix is to say so:
both tool descriptions now instruct it to translate the question to English before
searching and to answer the user in their own language. No infrastructure, no
model change, no ranking impact.

A multilingual embedding model would handle it natively and was considered, but it
is substantially larger — which matters on a scale-to-zero container — and it would
change English retrieval as well, so it would have to be measured against
scripts/eval-docs-search.ts rather than swapped in on the assumption that more
languages cannot hurt.

Verified: 447 tests pass, tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 22:42:56 +02:00
Oliver Kieselbach eca528b9b3 docs(mcp): attribute the cold start — the app is 0.5s of it, the platform is the rest
The boot marks answered the question on the first production cold start, and the
answer overturns the reasoning behind an infrastructure change made earlier today.

  +244ms  node started (startup + module loading)
  +431ms  corpora read from disk          (+187ms)
  +491ms  index read, validated, hydrated  (+60ms for 3.12 MB)
  +504ms  listening
  +677ms  query embedder warm             (background)

Against a 31.5s cold start, the application accounts for half a second. The other
~98% is Container Apps activation: KEDA reacting, pod scheduling, mounting the
236 MB image, starting the container. The container also boots faster than a
development laptop (504ms vs 1273ms), so nothing here is CPU-starved.

Consequences recorded rather than quietly dropped:

- Raising the container to 0.5 vCPU was justified by the embedding model load on
  the cold-start path. Prebaked, that load is 173ms and runs in the background
  where only the first search could wait on it. The justification does not survive
  measurement. The setting stays — it plausibly helps concurrent requests, and at
  scale-to-zero the cost difference is negligible — but the Bicep comment now says
  what is true instead of what was assumed, so nobody re-derives the wrong reason.
- Index size is irrelevant to startup (60ms for 3.12 MB), so the corpus can grow
  freely, and the embedding rounding earns its keep on file size alone.
- The only lever that would actually remove the cold start is minReplicas: 1, at
  roughly 7x the monthly free consumption grant.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 22:34:24 +02:00
Oliver Kieselbach 8305f5c850 feat(mcp): time the boot so the cold start can be attributed
A cold start after scale-to-zero measures 24.5s to first byte against 60ms warm,
and nothing outside the process could say why. Log Analytics stamps every console
line of one ingestion batch with the same time, so the boot sequence collapses to
a single instant however it is queried, and piping stdout locally buffers until
exit. So the server times itself: `[boot +Nms]` marks derived from
process.uptime(), which unlike an in-process timer also covers Node's own startup
and module loading.

Measuring the precomputed path locally already contradicts two assumptions worth
recording:

  +860ms   node started (startup + module loading)
  +1199ms  corpora read from disk         (+339ms)
  +1242ms  index read, validated, hydrated (+43ms for 3.12 MB)
  +1273ms  listening                       (+31ms)
  +1497ms  query embedder warm             (+224ms, background)

The index is 43ms, not seconds — its size is not the cold-start cost, and growing
the corpus is cheap. The prebaked embedder is 224ms and loads in the background,
so only the first search could ever wait on it. That was the stated reason for
raising the container to 0.5 vCPU, which now looks like a weaker justification
than it did; the production marks will settle it. If they show the app listening
after a few seconds, the rest of the 24.5s is platform activation that neither CPU
nor index changes can touch.

Verified: 447 tests pass, marks appear on both boot paths locally. Incidentally
the run also confirmed the corpus hash guard and the staleness signal both work —
the local docs repo has moved to 6333d56 while the deployed image reports d8cb7f4,
so the precomputed index was correctly rejected and recomputed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 22:14:49 +02:00
Oliver Kieselbach c2c02b929c fix(web): show gather-rule scope/emit controls only where they change behavior
The form offered trigger, phase scope and emit mode unconditionally, which
produced combinations that cannot do anything plus one silently wrong state:

- A scope on a Phase Start / Phase End rule that names a concrete phase can only
  ever suppress the single firing — the phase IS the moment. Same for emit mode:
  a one-shot has no second result to dedup against.
- Selecting "From a phase onwards" (or "Only during specific phases") and not
  filling the detail dropdown serialised to null, i.e. the rule ran everywhere
  while the form still read "From a phase onwards".

supportsPhaseScope / supportsEmitMode now drive visibility; buildScopeFields
nulls hidden-control state so a value from an earlier trigger choice cannot leak
into the payload; validateScopeSelection rejects an unfilled scope mode on both
create and save. Startup is the deliberate exception — a scope there defers the
one-shot instead of suppressing it, so the scope control stays and only emit mode
is hidden. One-shot triggers now state in-form when they collect.

Sub-dropdown hints no longer claim "No selection = all phases" (they turn amber
and name the required action instead).

Tests: app/gather-rules/__tests__/gatherRuleFormLogic.test.ts (15) pins the
applicability matrix, the payload nulling incl. the trigger-switch leak, the
unfilled-scope rejection, and the JSON-mode scopeMode derivation.
vitest 572/572; tsc silent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 22:06:50 +02:00
Oliver Kieselbach 1821af6f05 fix(mcp): fire the search_docs keyword fallback on a weak semantic head
The live test found a third recall gap. The fallback only topped up when semantic
search returned FEWER than topK results — but the more common failure is a full
list of unconvincing ones. Querying "WOW6432Node" returned three chunks scoring
~0.29, none containing the word, while the single chunk that does was never
considered at all.

The fix is a ranking change, so it was measured rather than reasoned about. Adds
scripts/eval-docs-search.ts, which scores strategies over 2465 queries in three
deliberately different classes: every chunk heading (easy by construction, so a
regression detector), rare identifiers (the class this fallback serves), and
hand-written questions (the reality anchor — not derived from the corpus, so it
shares little vocabulary with it). Embeddings are computed once and each strategy
is a pure function over the same candidates, so sweeping a threshold costs no
extra inference.

                        known-item  literal  handwritten MRR  fixed  broken
  append-only (before)       71.9%    66.2%            1.000      -       -
  weak < 0.30                71.9%    72.6%            1.000    142       0
  weak < 0.35                73.7%    81.1%            1.000    335       0
  weak < 0.40                76.3%    88.2%            0.944    502       3
  weak < 0.45                77.2%    93.0%            0.889    613       5

0.35 is strictly dominant — better in every class, worse in none — so that is
what ships. Higher thresholds keep repairing identifier queries but start
demoting hand-written questions, the class that most resembles real use, so the
gain stops being free. When the head is weak, exact matches move to the front
while at least one semantic result is kept, so a weak-but-correct hit is never
pushed out entirely.

The measurement also resized the problem: identifier queries were succeeding only
66.2% of the time, so the reported case was one of roughly 750, not an outlier.
That is the part a single anecdote could not have shown, and the reason the
harness exists before the fix rather than after it.

Verified: 447 tests pass, including new cases for the weak path and for leaving a
confident ranking untouched. On the real corpus "WOW6432Node" now returns the
chunk that contains it at rank 1, while "how do I deploy the agent with Intune"
(0.819) and "what firewall URLs do I need to allow" (0.374) are unchanged and the
negative controls stay empty.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 21:39:26 +02:00
Oliver Kieselbach 62599d972c fix(mcp): put every doc token back into search_docs ranking
Two measured recall gaps in search_docs, both silent.

1. Truncation. MiniLM's context is 256 tokens; 87 of 324 chunks exceeded it and
13% of all tokens never reached the ranking — the tokenizer drops the tail
without complaint. Concretely, in built-in-rules.md :: Apps (734 tokens) the
strings ANALYZE-APP-013, ANALYZE-OFFICE-001, ANALYZE-CORR-003 and Click-to-Run
all sit past the cutoff, so "proxy configuration blocking app downloads" did not
retrieve that section at all.

Shrinking chunks cannot fix this: the corpus ranges 2.43 to 5.81 characters per
token, so no character budget bounds tokens, and a budget safe for the worst case
(~620 chars) would shred prose and tables alike. So the chunk stays whole and the
embedding is split instead — index() cuts long text into 250-token windows with
40 overlap, a document carries embeddings[][], and search() scores it by its BEST
window. Max rather than mean: averaging would blur a long section's several
topics into one mediocre direction, the same dilution that makes large context
windows worse than chunking.

The tokenizer is a build-time dependency only (vocabulary, not ONNX weights,
~93 ms); production boots from precomputed vectors and never loads it.

2. No general fallback. Proper nouns embed poorly — "RealmJoin" occurs in 3
chunks yet scores under the threshold. Fuzzy matching was built, measured and
rejected: at a threshold low enough to surface RealmJoin (0.28), Fuse also
returns 5 hits for "SCEPman", a word this corpus contains zero times. Literal
containment has neither problem, and every provider already exposes it. So a
short result set is now topped up via lexicalMatch on distinctive query terms,
requiring two matching terms as corroboration — without that rule "how do I bake
sourdough bread" matched a page about LLM providers, because "bake" is a
substring of "baked into the image". Keyword hits are appended and flagged, never
sorted together with cosine scores.

Also sharpens the tool description with the measured reason to prefer this over
fetching the docs site (~1k tokens for topK=3 against ~9k raw / ~600k rendered,
version-matched, and matching on meaning), and replaces the growing tail of
positional docs parameters with a DocsSearchBundle.

Verified: token coverage 87% -> 100%; "Office Click-to-Run install never
finished" 0.318 -> 0.442; "proxy configuration blocking app downloads" absent ->
0.559 at rank 1; the ten reference queries keep byte-identical scores and rank 1
(short chunks are one window, so the same vector as before — no long-chunk bias).
The fallback is not a dead path: across 904 rare corpus tokens queried alone it
contributes hits for 431, while negative controls stay empty. Precompute output
is accepted by the boot validator and a mutated corpus still rejected; index
2.63 -> 3.12 MB for 324 chunks / 424 vectors. The rules corpus gains the same
treatment unplanned: 128 rules now carry 172 vectors. 445 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 21:17:42 +02:00
Oliver Kieselbach 21b751be64 changed sidebar tenant settings entries ordering 2026-07-21 20:02:28 +02:00
Oliver Kieselbach 63ea2e3cee feat(mcp): search_docs — customer documentation as a third search corpus
The MCP server could answer telemetry questions ("why did this session fail")
but not product ones ("how do I deploy the agent", "where is my data stored").
Those answers already exist in the autopilotmonitor-docs bundle; search_docs
makes them semantically searchable.

Deliberately a separate index from search_knowledge: rules and prose answer
different questions, and sharing one topK would dilute both. Registered for
every role — published documentation carries no tenant data — and skipped
entirely when no corpus was baked in, so a doc-less build advertises no
broken tool.

Corpus handling:
- Chunked at h2/h3 and <details>/<summary>, each chunk prefixed with a
  "Page > Section > Subsection" breadcrumb so a mid-page hit is self-describing.
- Newlines are normalized before any parsing. This is a correctness
  precondition, not cosmetics: the bundle ships CRLF and JS `.` does not match
  \r, so /^(#{2,3})\s+(.*)$/ matched zero headings — silently, degrading every
  page to unlabelled paragraph splits.
- Two size exceptions: tables are never split (a split table leaves `| --- |`
  rows with no header), and unbroken changelog lists split at line level (left
  whole, their tails fall outside the model window and become unsearchable).
- Short sections are kept — a one-line answer is often the best hit, and the
  breadcrumb supplies the context a short body lacks.

Build and delivery:
- Docs are embedded at image build time into an optional third
  PrecomputedSection. Optional so a checkout without the docs repo does not
  invalidate the rules and event-type vectors; the reverse (server has docs,
  file does not) is staleness and is rejected.
- The deploy workflow checks the public docs repo into the build context and
  passes its SHA as DOCS_COMMIT, which /health reports alongside the chunk
  count — editing the docs does not update a running server, so staleness is
  made checkable rather than memorable.
- Embeddings are rounded to 6 decimals on serialize: 55% smaller index (2.63
  vs 5.88 MB) for a ~1e-5 norm error against a 1e-3 tolerance. indexPrecomputed
  now asserts unit norm, since the hydrate path never checked it.

Infra: the Bicep template was two changes behind the live deployment (EU
cutover and ACR hardening, both applied directly in Azure). Reconciled against
the running resources — managed-identity image pull replaces the admin
credentials and the acr-password secret — and infra/ is no longer gitignored so
it stops drifting. Container raised to 0.5 vCPU / 1.0 GiB for the cold-start
query-embedder load. The template does not model the mcp.autopilotmonitor.com
custom domain, so a full deployment would remove it; the header warns about it.

Verified: 428 tests pass; 45 pages produce 324 chunks; sample queries hit the
right page top-1/2; all 8 derived URLs return 200 live; the precompute output
is accepted by the boot validator and a mutated corpus is rejected.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 19:48:28 +02:00
Oliver Kieselbach 93dcac44a4 fix(web): add Contact to the tenant settings sidebar
The section was registered for routing but not in the sidebar, so it was
reachable only by typing the URL. Sidebar entries come from
globalNavConfig.tsx, which is not derived from tenantNavSections.ts -
both have to be touched for a section to exist.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 18:19:44 +02:00
Oliver Kieselbach 72be8ac89f feat(config): tenant contact address, seeded once from the preview email
Enforcement and service notices need somewhere to go. Until now the only
address on file was the Private Preview notification email, which is
onboarding data in the PreviewWhitelist table and not something a tenant
can maintain.

Adds TenantConfiguration.ContactEmail, owned and edited by the tenant's
own admins under Settings -> Tenant -> Contact:

- Seeded once when a preview notification address is saved, and only
  while ContactEmail is still empty. It is never re-synced, so an edit in
  the portal cannot be overwritten later by the onboarding source.
- Backfilled for tenants onboarded before the field existed, as a
  manual-trigger maintenance task alongside the other backfills. It skips
  any tenant that already has an address, so it converges to a no-op and
  can be dropped once everyone is covered.
- Surfaced read-only in Admin -> Tenants next to the preview address, so
  an operator who needs to warn a tenant does not have to read the table.

Purpose is stated where it is entered and in the Privacy Policy: service
matters only - a technical problem, a security matter, or a change
needing an administrator's attention. Never marketing, never shared.

The Terms already say a warning is sent "where a tenant contact address
is on file"; this is the field that makes that promise meaningful.

Wired through the full tenant-config round-trip (types, context state,
hydrate, save payload, reset, provider) so a save from any other section
cannot clobber it.

Verified: solution builds clean, 3101 backend tests and 557 web tests
pass, tsc --noEmit clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 18:00:18 +02:00
Oliver Kieselbach cd570c8b3e feat(web): add IP/licence terms, enforcement ladder, security reporting
- Intellectual Property & Licensing section: the repository licence
  governs the code, not the hosted service, its infrastructure, its data,
  or the name and branding. Customer data stays with the customer.
- Acceptable Use now states the enforcement ladder (warning, reduced rate
  limits, suspension, termination) instead of an unqualified right to
  suspend, and routes security research to GitHub Security Advisories
  with explicit testing boundaries.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 17:38:52 +02:00
Oliver Kieselbach 6dfc564be6 fix(web): align privacy and terms with the DPA-owns-the-register split
Privacy: "Sub-processors and Outbound Data Flows" becomes "External
Services". It still names who receives data, which is a controller's own
duty, but drops the Art. 28 vocabulary and the promise to announce new
sub-processors - that commitment lives in the data processing agreement.
Points at the renamed docs page and notes the DPA is available on
request.

Terms: the MSP clause now says a managing organization is responsible for
having the necessary data protection agreements with its customers,
instead of classifying it as a processor or sub-processor.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 17:25:36 +02:00
Oliver Kieselbach afecd18123 fix(web): patch js-yaml and brace-expansion DoS advisories
Dependabot alerts #63/#64/#65 (all high): brace-expansion exponential-time
expansion of consecutive non-expanding {} groups (GHSA-3jxr-9vmj-r5cp,
both the 1.x and 2.x trees) and js-yaml quadratic CPU consumption via
YAML merge-key chains (GHSA-52cp-r559-cp3m).

All three are transitive dev-only dependencies of the eslint toolchain
(eslint -> @eslint/eslintrc -> js-yaml; eslint/typescript-estree ->
minimatch -> brace-expansion) and resolve within their existing semver
ranges, so only the lockfile changes.

npx tsc --noEmit clean, vitest 557 passed, npm audit reports 0
vulnerabilities.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:53:28 +02:00
Oliver Kieselbach 5a294ac4f6 fix(mcp): patch transitive protobufjs/tar/body-parser advisories
Dependabot alert #62 (protobufjs 7.6.4, DoS via infinite loop in .proto
option parsing, GHSA fixed in 7.6.5) plus the tar (critical) and
body-parser (low) advisories npm audit surfaced alongside it. All three
are transitive under @huggingface/transformers and resolve within their
existing semver ranges, so only the lockfile changes.

Lockfile also picks up the pending 1.5.0 -> 1.5.1 package version sync.

npx tsc --noEmit clean, vitest 400 passed / 41 skipped, npm audit
reports 0 vulnerabilities.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 15:50:51 +02:00
Oliver Kieselbach dbf6e50e2d feat(agent,web,shared): phase_exit gather trigger — one-shot collection at a phase's end
Follow-up to gather-rule phase scoping. One-shot collection at a phase START
already existed (phase_change dedups per (rule, phase)) and at enrollment END
(on_event + enrollment_complete); the END of a specific phase had no trigger.

New trigger value "phase_exit": fires once when TriggerPhase is LEFT (empty =
every phase transition). Together with phase_change it forms the two one-shot
bookends of a phase.

Agent (GatherRuleExecutor.OnPhaseChanged):
- Exit rules are evaluated BEFORE _currentPhase advances, so both the
  TriggerPhase match and the phase-scope gate see the phase being LEFT. A rule
  scoped to DeviceSetup therefore fires on its own exit even though the phase
  being entered is out of scope.
- A transition into Failed fires the exit rules of the failing phase — the
  snapshot at a failure boundary is the point of the trigger.
- Exit of Unknown never fires; a repeated identical phase signal is not a
  transition.
- Dedup key "exit:{ruleId}|{phase}" shares _phaseRulesExecuted with the
  phase_change keys but stays in a disjoint key space, so an enter-rule and an
  exit-rule on the same phase cannot consume each other's slot.

Portal: trigger labels are now "Phase Start" / "Phase End"; the canonical phase
dropdown and the create/save payloads cover both via PHASE_TRIGGERS; the card
renders "entering X" / "leaving X".

No ConfigVersion bump: no new field is transported. Old agents do not know the
value and simply never fire the rule (silent no-op).

Tests: GatherRuleExecutorPhaseExitTests (11) — enter-vs-exit, no refire on
re-entry, empty-triggerPhase, exit-of-Unknown, same-phase-signal, into-Failed,
Complete, scope-against-left-phase, out-of-scope slot preservation, key-space
disjointness, on_change interplay. Build 0 Errors; V2.Core.Tests 1884/1884;
Functions.Tests 3099/3099; vitest 557/557; tsc silent.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-21 15:37:48 +02:00
Oliver Kieselbach 062962ae41 docs(claude): dedupe and condense agent directives
CLAUDE.md ships with every request. Removed the example from the
operator-only-infrastructure rule - rules state the rule, not an
instance - and applied the same yardstick to the rest of the file.

Merged what said the same thing twice: "would a staff engineer approve"
and "what would a perfectionist reviewer reject"; Demand Elegance and
Code Quality and Core Principles; Task Management's lessons item and the
Self-Improvement Loop; Task Management's plan steps and Plan Mode
Default. Dropped motivational framing that steers no decision, and a
stray heading level.

Every actual rule survives: the 3+ step plan threshold, stop-and-re-plan,
subagent usage, verify-before-done with a diff against main, autonomous
bug fixing, lessons capture, 500 LOC chunked reads, no absence claims
from one grep, the structural-fix mandate, and the Codex review note.

425 words, down from 614 at the start of the session - and that now
includes the customer-facing claims section, which did not exist then.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 14:15:14 +02:00
Oliver Kieselbach be08660976 docs(claude): condense customer-facing claims and OKF sections
CLAUDE.md ships with every request, so length is a running cost. Cut the
new claims section from 12 lines to 4 and tightened the OKF section,
dropping the spec URL and restating the rules imperatively. Net effect:
the file is now smaller than before the claims section was added
(654 words vs 856).

No rules were removed - verify-before-writing, the change triggers, the
durable-phrasing rule, and the operator-only-infra caveat all remain.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 14:09:48 +02:00
Oliver Kieselbach b032005f5b docs(claude): add customer-facing claims to the release checklist
Terms, Privacy, the About page and the trust docs make statements that
customers and security reviewers rely on, but nothing opened them during
a refactor, so they drifted. The July 2026 sweep found a ".NET 8" backend
that was .NET 10, a "Next.js 18" that does not exist, ETL log collection
that never happened, and "each tenant runs its own isolated instance" for
a shared multi-tenant service.

Records which surfaces exist, which kinds of change must update them in
the same commit, and that claims are verified against the code rather
than carried forward.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 14:01:23 +02:00
Oliver Kieselbach 0ab568f479 fix(web): correct stale About claims; open Community plan to any tenant
Terms: replace the blanket business-use-only clause. It excluded exactly
the wrong group - a registered association is not a consumer under German
law anyway (only natural persons can be), while the clause did shut out
private individuals running their own lab or personal tenant, who are a
real and intended audience for a free community edition. Community is now
stated as open to anyone with an eligible Entra ID/Intune tenant;
business-only stays where it means something, on the Enterprise contract.

About page - claims verified against the code and corrected:
- "each customer tenant runs its own isolated instance" was wrong and
  contradicted the same page two sections earlier. It is one shared
  multi-tenant service with partition-key isolation and delegated MSP
  access.
- Diagnostics: no ETL tracing and no Windows event log export happens;
  the bundle is agent logs, IME logs, agent state and device info. Upload
  is tenant-configured (off/always/on failure) at the end of enrollment,
  not triggered on demand from the portal - the portal downloads it.
- .NET 8 Isolated -> .NET 10 Isolated on Flex Consumption.
- "Next.js 18" -> Next.js 15 (React 18); 18 was React's version.
- Roles: Admin/Operator -> Admin/Operator/Viewer plus delegated MSP.
- Notifications: Teams only -> Teams, Slack, generic JSON webhooks, with
  start/success/failure and SLA triggers. Telegram stays unlisted; it is
  operator-only alerting, not a tenant feature.
- Agent deployment: Win32 app -> Intune platform script; self-destruct is
  on by default, not optional-sounding.
- Dropped the "tamper-evident" audit claim - records are ordinary table
  entities with no hash chain, signing, or immutability policy.

Confirmed while verifying: no tenant-configurable email channel exists
anywhere in the codebase, so the published privacy statement that Resend
serves only the onboarding mail is accurate.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 13:58:07 +02:00
Oliver Kieselbach fced858a7a feat(legal,agent): professionalize Terms/Privacy; wire SendTraceEvents gate
Rework the public Terms of Use and Privacy Policy from their early-days
wording to match how the service actually runs today, and fix a setting
that looked like a privacy control but gated nothing.

Legal positioning:
- glueckkanja AG operates the service and is the counterparty on BOTH
  plans (Community/Enterprise is a flag in one system, one subscription).
  Oliver Kieselbach is named as creator and maintainer of the open-source
  project and the Community edition, acting on behalf of glueckkanja AG
  and explicitly not a separate contracting party.
- Community: free, "AS-IS", no availability or support commitment.
  Enterprise: written agreement incl. DPA, higher limits, MSP delegation.
- Added business-use-only clause, simplified governing law (dropped the
  CISG exclusion - it governs sale of goods, not SaaS), split liability
  per plan, added an MSP delegation section and a rules-contribution
  clause covering the community rule pool.
- Imprint links to the glueckkanja AG imprint, whose scope clause covers
  every service linking to it - no duplicated company data to go stale.
  Reachable from the landing footer and both help menus.

Privacy Policy corrections (previous claims were inaccurate):
- "No personal data is transmitted" - backend telemetry carries the UPN;
  only the browser SDK is cookie- and UPN-free.
- "Not shared with third parties" - qualified for MSP delegation,
  customer-configured notification channels, the customer's own LLM via
  MCP, and agent-side geolocation.
- Removed "data subject rights are not guaranteed"; GDPR rights are not
  optional. Added region, sub-processors, retention and IP handling.

Agent - SendTraceEvents gate:
The setting was plumbed from TenantConfiguration through to
AgentConfiguration but never consumed; Trace severity only hides an event
from the timeline view. The outbound IP was therefore always collected
when geolocation is on (which is the default). Gate is now enforced in
TelemetryEventEmitter.Emit, the single choke point before the transport
enqueue. Dropped before the sequence counter advances so suppression
leaves no gaps. Evaluated per event via Func<bool> so a mid-session
remote-config merge takes effect. Fails open: an unwired or throwing
accessor lets trace events flow, because the gate suppresses diagnostics
and must not silently blind the timeline.

Default stays on and the switch remains operator-only via the config
table - deliberately not surfaced in the UI or documentation.

Verified: solution builds clean, 1873 agent core tests pass,
tsc --noEmit clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-21 13:42:03 +02:00
Oliver Kieselbach ee947e2ca9 updated docs links 2026-07-20 20:52:56 +02:00
Oliver Kieselbach 5f08a6ec68 updated dev script 2026-07-20 19:52:54 +02:00
Oliver Kieselbach a97888ebd5 fix(agent): neutral aggregated 2800 observations; reboot claim gated on observed reboot
Rework after the first real enrollment (session b2e890c1) exposed three
defects in a654c730:

- Shell-Core "RebootCoalescing" match was a false positive by construction:
  the token appears in the ROUTINE SubcategoryProcessing_Started bootstrap
  marker ("Starting subcategory DeviceSetup.RebootCoalescing...") on EVERY
  enrollment. esp_reboot_coalescing removed entirely (branch, constant,
  catalog, tests).
- 2800 reboot-required flags fire at every policy sync; only DeviceSetup-time
  flags cause the coalesced reboot. The session's flags fired during
  AccountSetup - no reboot - yet our Warning events claimed "device will
  restart". Raw event is now a neutral Info observation; the reboot CLAIM
  moved to ANALYZE-ESP-005 with a REQUIRED system_reboot_detected >= 1
  condition (no observed reboot -> advisory silent).
- One event per 2800 record was spam: records are now buffered and debounced
  (10 s) into ONE aggregated event per burst (rebootUris[]/uriCount/
  firstRebootUri/recordCount/unparsedCount). Watermark is persisted at FLUSH,
  not claim - a pre-emit reboot kill loses nothing, the post-restart backfill
  re-emits the burst at its historical time.

Verified 2800 description shape recorded in code, tests, and concept doc:
"The following URI has triggered a reboot: (./Device/...)."; extraction
covered by a real-text regression test. Lesson added to tasks/lessons.md
(observation-level events vs outcome-gated claims; substring tokens appear
in framework bookkeeping records).

Tests: Agent.V2.Core 1867/1867, Functions 3099/3099, MCP 400 passed,
solution build 0 errors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 18:41:47 +02:00
Oliver Kieselbach a654c7309e feat(agent): attribute mid-ESP coalesced reboots to MDM policy URIs (second sign-in detection)
Device-assigned MDM policies whose CSP URIs match the OS reboot-required
catalog force a coalesced reboot at the end of ESP DeviceSetup - the user
sees an unexpected restart and a second sign-in before AccountSetup.
Invisible in the Intune console; admins simply lived with it. Now attributed
and actionable (PMPC research).

- MdmRebootPolicyTracker (+host): watches DeviceManagement-Enterprise-
  Diagnostics-Provider/Admin EventID 2800, emits one mdm_policy_reboot_required
  per reboot-forcing URI (Warning, ImmediateUpload; tolerant EventData-first
  URI extraction with regex fallback; fail-soft OMADM RebootRequired probe).
  Cross-restart RecordId watermark (mdm-reboot-watermark.json) - mandatory,
  the coalesced reboot restarts the agent and the backfill re-reads the log.
- ShellCoreTracker: RebootCoalescing branch on the already-watched 62407
  stream -> esp_reboot_coalescing corroboration event; deliberate deviation:
  emits on backfill too (live emit races the reboot kill).
- ANALYZE-ESP-005 advisory (warning): names the first URI via {{rebootUri}},
  remediation = reassign profiles device -> user groups; +20 confidence on
  coalescing corroboration, +10 on >=2 URIs.
- Config: Collectors.MdmRebootPolicyWatcherEnabled (default ON, internal
  steering only - agent-native collector, NOT an IAgentAnalyzer, no admin UI),
  MdmRebootPolicyBackfillLookbackMinutes=60; ConfigVersion 35.
- Event types deliberately distinct from system_reboot_detected so the
  backend RebootCount (exact-string match, 2 sites) stays uninflated.
- Docs: docs/agent/mdm-reboot-coalescing.md (+index/log). Exact 2800/62407
  message texts are unverified until E2E VM validation (tolerant matching).

Tests: Agent.V2.Core 1868/1868 (21 new), Functions 3099/3099, MCP 400 passed
(event-types drift), solution build 0 errors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 18:05:11 +02:00
Oliver Kieselbach 6d1ff8d7f0 fix: code-review follow-ups (watcher dispose race, revoke-enforcement tests, scan guard)
- ConsoleBypassWatcher: re-check _disposed after watcher.Start() — a Dispose()
  racing the background arm could leave an armed WMI subscription no code path
  could ever stop (leaked until process exit).
- SignalRContext: collapse rapid accessRevoked pushes into one restart via an
  in-flight guard — a second start() on a non-Disconnected connection throws
  and burns the retry budget.
- config/all: negative-cache managed ids a rescue scan proved missing (5 min
  TTL) — one stale grant (offboarded tenant) no longer makes every delegated
  config/all call pay a full table scan.
- DelegatedAdmin/TenantGroup functions: extract testable revoke-enforcement
  cores (RevokeCoreAsync, SetEnabledCoreAsync, UnassignCoreAsync,
  DeleteGroupCoreAsync, ValidateGrantRequest) and cover the security flows:
  404-on-missing-row writes no audit row and fires no disconnect; real
  revoke/disable/unassign/group-delete always cuts live streams; grant
  validation is fail-closed (GUID tenantId, recognized roles only).
- dashboard/page.tsx: drop unused next/link import left by the banner removal.

Tests: backend 3099 passed, web 557 passed, agent ConsoleBypassWatcher suite
passed, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 13:13:26 +02:00
Oliver Kieselbach 5c8b273f37 chore: bump skills submodule (system-Timestamp warning in raw-query skills)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 12:38:04 +02:00
Oliver Kieselbach da958b5398 fix(backend): business timestamps survive storage migrations (OccurredUtc + RowKey decode)
The 2026-07-18 storage migration reset every row's system Timestamp,
exposing that AuditLogs, OpsEvents and Events displayed, filtered and
retention-swept on it (custom "Timestamp" writes were silent no-ops —
reserved system property). All migrated rows read as 2026-07-18.

- New Helpers/BusinessTimestamp: RowKey decoders (audit !revtick_guid,
  ops bare revtick, event yyyyMMddHHmmssfff_seq), RowKey-range OData
  clause builders (tick-exact, with the mandatory RowKey lt '"' guard
  so no lower bound can ever match legacy bare-GUID audit rows), and
  dual DateTimeOffset/DateTime column reads.
- Writes in all three tables now persist OccurredUtc (custom column,
  copied verbatim by future migrations); no-op ["Timestamp"] removed.
- Read mappers resolve OccurredUtc -> RowKey decode -> system Timestamp;
  GetEarliestSessionEventTimestampAsync uses the same resolution so
  StartedAt/Duration reconciliation can no longer absorb the migration
  date on migrated sessions.
- Date-window filters and retention moved off the system Timestamp onto
  RowKey ranges; ops retention loses its full-table scan and the
  post-migration mass-delete cliff; audit retention runs a temporary
  second pass for legacy GUID rows (TODO: remove after 2027-03-01).
- New GlobalAdmin backfill endpoint POST /api/maintenance/
  backfill-occurred-utc (dry-run by default, batched, idempotent MERGE
  of only OccurredUtc, table=audit|ops) — Events rows are deliberately
  never mutated; their read path decodes the RowKey instead.
- Tests: decoders, clause boundary proofs incl. GUID-safety pins,
  mapper precedence, backfill row decisions (Functions.Tests 3078).
- Docs: docs/backend/business-timestamps.md (system Timestamp is never
  authoritative; migration checklist rule).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:29:53 +02:00
Oliver Kieselbach e17eb607ca docs: gather-rule phase scoping + emit-on-change concept doc (OKF)
New rules/ section in the knowledge bundle: semantics table, from-phase latch
behavior (never via Failed, sticky through it), on_change x emitOnlyIfExists
composition, restart re-emit, no-phase-signal caveat, old-agent degradation,
and the prerequisite toggle-wipe fix. Index + log updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 09:51:14 +02:00
Oliver Kieselbach 71000a5dd2 feat(web): gather-rule scope + emit-mode UI, phase dropdowns, edit tags wipe fix
Step 4: portal round-trip for the new fields.

- types.ts: GatherRule + activePhases/activeFromPhase/emitMode; NewRuleForm +
  scopeMode ("always" | "during" | "from") as UI-only view state; canonical
  GATHER_PHASES list (EnrollmentPhase names Start..Complete with display
  labels) + EMIT_MODES; new rules default to emitMode "on_change".
- withDerivedScopeMode: JSON-mode merges rule-shaped JSON (no scopeMode key)
  over form state — the helper derives scopeMode from the data in all four
  JSON paths (create apply/save, edit apply, card edit save) so pasted scope
  fields survive into the payload instead of being silently dropped.
- GatherRuleFormFields: "Active During" mode select + phase multi-select /
  from-phase dropdown (mode switch clears the other mode's fields), "Emit
  Mode" select with helper text; triggerPhase free-text replaced by the same
  canonical dropdown (placeholder used to suggest invalid tokens like
  "DeviceESP"; legacy stored values remain selectable).
- page.tsx: create/save payloads carry the three fields; startEditing maps
  them back (existing rules load as "always", the on_change default is for
  NEW rules only); handleSaveEdit now also sends tags — same wipe class as
  the toggle bug, the full-replace PUT dropped them on every edit.
- GatherRuleCard: trigger section shows the scope + emit mode, e.g.
  "Interval every 60 seconds - from Account Setup - emit on change".

npx tsc --noEmit silent; vitest 557/557.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 09:51:04 +02:00
Oliver Kieselbach a88db34af5 feat(agent): gather-rule phase scoping + on_change emit dedup in GatherRuleExecutor
Step 3: the executor honors the new GatherRule fields.

Phase scope (ActivePhases / ActiveFromPhase), applied to ALL trigger types:
- interval: timers keep running, an out-of-scope tick is a free early-return
  (no start/stop lifecycle races on phase transitions).
- startup: scoped rules not yet in scope at UpdateRules are deferred and run
  exactly once from OnPhaseChanged when their scope activates (same
  _startupRulesExecuted dedup; not part of the WaitForStartupRules latch).
- phase_change: evaluated against the NEW phase; scope gate runs BEFORE the
  (rule, phase) dedup add so an out-of-scope pass does not consume the slot.
- on_event: gated by current phase.
- From-phase latch: phase != Unknown/Failed && ordinal >= threshold; sticky for
  the session incl. through Failed; re-evaluated on config refresh so a rule
  delivered after its threshold passed latches immediately.
- Before the first phase signal, scoped rules are inactive.
- IgnorePhaseScope bypass for --run-gather-rules (no phase context; logged).

EmitMode on_change:
- Canonical hash (keys ordinal-sorted at every level, invariant formatting,
  SHA-256) of the raw collector result BEFORE ruleId/ruleTitle injection.
- First non-empty result always emits; unchanged results increment a per-rule
  suppression counter; the next emitted event carries suppressedPolls +
  suppressedSinceUtc. Empty results (emitOnlyIfExists miss) touch nothing, so
  the combo yields zero noise, one event on appearance, then only on change.
- State survives UpdateRules refreshes (agent restart => at most one re-emit).

Tests: GatherRuleExecutorPhaseScopeTests (scope matrix synchronous via internal
IsRuleInScope; trigger wiring async with positive waits),
GatherRuleExecutorEmitModeTests (state machine via internal ShouldEmitOnChange,
canonical-hash properties, ExecuteRule wiring). V2.Core.Tests 1847/1847.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 09:50:50 +02:00
Oliver Kieselbach 21dd35f787 feat(backend,shared): gather-rule phase scoping + emit-mode fields, custom-rule toggle wipe fix
Step 0-2 of the gather-rule anti-spam plan:
- GatherRuleService.UpdateRuleAsync: toggle-style partial PUTs ({enabled,
  isBuiltIn, isCommunity}) previously full-replaced the stored custom rule and
  wiped Title/Target/Trigger/Parameters. Partials now merge Enabled into the
  existing row; full edit payloads preserve the original CreatedAt.
- GatherRule: new opt-in fields ActivePhases (run only during listed phases),
  ActiveFromPhase (sticky latch from a phase onwards), EmitMode (always |
  on_change). Absent fields = legacy behavior; schema updated accordingly.
- TableStorageService.Rules: Store+Map for the three columns (absent column /
  stored-empty -> null); MapToGatherRule internal for roundtrip tests.
- ContentEquivalent covers the new fields, null-tolerant for ActivePhases so
  pre-existing rows never diff against seeds (no reseed churn).
- GatherRulesFunction.ValidateScopeAndEmitMode: canonical EnrollmentPhase name
  list Start..Complete (Unknown/Failed and numeric tokens rejected), mutual
  exclusion of the two scope fields, emitMode whitelist; 400 on violation.
- ConfigVersion 33 -> 34 (33 was taken by MigrateToApiBaseUrl).

Tests: GatherRuleUpdatePartialMergeTests (merge/replace/unknown-rule/built-in
path), GatherRuleScopeFieldsTests (Store<->Map roundtrip incl. legacy rows,
ContentEquivalent matrix, validation accept/reject). Functions.Tests 3029/3029.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 09:50:35 +02:00
Oliver Kieselbach 9933c039ed refactor(shared,backend): MigrateToApiBaseUrl nullable, drop null-forgiving operator
null = no migration is the documented contract and the agent already
assigns null when stripping the field from the on-disk cache; annotate
the property as string? so the backend assignment needs no ! suppression.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 13:33:57 +02:00
Oliver Kieselbach ef67dec263 fix(backend): system-health URLs post EU-migration + GA endpoint visibility
- Agent Binaries check + ops watcher now probe the canonical download alias
  (download.autopilotmonitor.com) AND the legacy blob keepalive; alias failure
  is unhealthy, legacy failure a warning
- McpServerBaseUrl fallback moved from the stale kindwave WEU ACA FQDN to the
  stable custom domain mcp.autopilotmonitor.com (CNAME to the new GWC instance)
- health/detailed and health/mcp include the probed endpoint URLs in check
  details for Global Admins only (storage table endpoint, WEBSITE_HOSTNAME,
  alias/legacy download URLs, MCP server URL)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 11:42:40 +02:00
Oliver Kieselbach b005c20598 feat(agent,backend,web): config-channel agent endpoint migration (MigrateToApiBaseUrl)
Re-home agents to a new API base URL through the agent/config response
(backend move or per-tenant EU->US region move) - the lesson from the
2026-07-18 WEU->GWC cutover, where the compiled-in API hostname hard-cut
every in-flight agent (no DNS indirection possible: mTLS breaks behind a
proxy, Flex custom-domain TLS is broken).

Backend (ConfigVersion 33):
- AdminConfiguration.AgentMigrateApiBaseUrl (global) +
  AgentMigrateTenantOverridesJson (tenantId->URL; empty value pins a
  tenant against the global target; override wins)
- GetAgentConfigFunction.ResolveMigrateTarget serves the validated target;
  AgentMigrateServed / AgentMigrateRejected warnings as delivery evidence
- TableConfigRepository Store+Map; admin portal Global Settings round-trip

Agent (kill-switch-mirrored hardening):
- Shared AgentEndpointMigrationRules: https-only, *.azurewebsites.net on a
  label boundary, no path/port/userinfo - enforced on BOTH sides
- Live fetch only: RemoteConfigService cache strips the field on write AND
  read; EndpointMigration.ResolveTarget re-checks the fetch outcome
- AgentRuntimeConfig.Resolve returns before merge/integrity/bootstrap
  cleanup on a pending migration (no spurious IntegrityCheckFailed against
  a stale migration stub), Program.RunAgent rebuilds Phase 3+4 against the
  new URL - single hop, kill signal wins before and after the hop

Tests: AgentEndpointMigrationTests (backend rules + resolution),
EndpointMigrationTests + cache-hardening strip (agent). Full suites green
(Functions 2992, Agent Core 1822, vitest 557, tsc silent).

Doc: docs/agent/endpoint-migration.md

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 17:48:48 +02:00
Oliver Kieselbach e2d78c6b06 chore(ci): rename SWA workflow to deploy-web.yml with neutral deploy-token secret
The brave-grass filename and secret name referred to the pre-migration
Static Web App hostname. Workflow now matches the deploy-backend/-mcp
naming scheme and uses AZURE_SWA_DEPLOY_TOKEN (already set to the new
SWA's token). The old secret is removed after the first green run.
2026-07-18 16:51:39 +02:00
Oliver Kieselbach 3d06a689cc feat(web): rename Known Issues to Service Announcements, drop maintenance banners
Menu/dashboard/tip labels now match the docs page title. The temporary
maintenance banners on landing and dashboard are removed — the completion
notice lives in Service Announcements. Also checks in the migration
follow-up plan and the runbook outcome addendum.
2026-07-18 16:43:50 +02:00
Oliver Kieselbach 2d5f87140f chore: retarget SignalR quota watcher to Standard S1 limits + CI build hygiene
- SignalR quota watcher/health card: defaults 20/20k (Free tier) -> 1000/1M
  (Standard S1 x1 unit); settings renamed SignalRFreeTier* -> SignalRConnectionLimit/
  SignalRDailyMessageLimit (no Azure overrides were set); all operator-facing
  texts tier-neutral (connections 429 hard, message overage is billed not throttled)
- web: drop obsolete edge runtime from opengraph/twitter/apple-icon image routes
  (next/og runs on Node since 14; images now statically generated at build)
- web: update caniuse-lite browser db (silences 6-months-old Browserslist warning)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 16:38:15 +02:00
Oliver Kieselbach ce1ca3dbf4 updated SignalR limits on system health 2026-07-18 16:22:49 +02:00
Oliver Kieselbach b1a8e371ad chore(build): retarget DecisionCore.Tests to net10.0, drop 8.0.x SDK from agent CI
Nothing required net8.0 -- the test runner just predated the .NET 10
migration. Agent projects stay net48 (Intune/OOBE constraint).
474/474 tests green on net10.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 16:01:39 +02:00
Oliver Kieselbach b698a87bc8 chore(build): align local and CI builds on .NET 10 SDK
- global.json pins SDK 10.0.2xx (latestFeature) for local and CI parity
- build-agent.yml: SDK 10 as build toolchain, 8.0.x kept for the
  net8.0 DecisionCore.Tests runtime
- build.ps1: stale .NET 8 hints updated to .NET 10

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-18 15:59:23 +02:00
Oliver Kieselbach ac31ee980e feat(ci): mirror agent artifacts to legacy storage (transition until download.-alias migration; fail-soft, counter stays on eu only) 2026-07-18 15:57:40 +02:00
458 changed files with 35720 additions and 3000 deletions
+25 -67
View File
@@ -1,79 +1,37 @@
## Workflow Orchestration
# Agent Directives: Mechanical Overrides
# Agent Directives
You are operating within a constrained context window and strict system prompts. To produce production-grade code, you MUST adhere to these overrides:
- **Plan first.** Enter plan mode for any non-trivial task (3+ steps or an architectural decision) and write the spec before building. If something goes sideways, stop and re-plan instead of pushing on.
- **Use subagents liberally** for research, exploration, and parallel analysis — one focus each — to keep the main context clean.
- **Verify before claiming done.** Run the tests, check the logs, show the evidence. Diff against `main` where behaviour could regress.
- **Fix bugs autonomously.** Given a report, a failing test, or an error, resolve it — no hand-holding.
- **Capture corrections.** After any correction from the user, record the pattern and a preventing rule in `tasks/lessons.md`; review it at session start.
- Read files over 500 LOC in chunks. Never treat one grep as proof of absence.
Codex reviews your output once you are done.
## Code Quality
### 1. Plan Mode Default
- Enter plan mode for ANY non-trivial task (3+ steps or architectural decisions)
- If something goes sideways, STOP and re-plan immediately — don't keep pushing
- Use plan mode for specification steps, not just building
- Write detailed specs upfront to reduce ambiguity
### 2. Subagent Strategy
- Use subagents liberally to keep main context window clean
- Offload research, exploration, and parallel analysis to subagents
- For complex problems, throw more compute at it via subagents
- One tack per subagent for focused execution
### 3. Self-Improvement Loop
- After ANY correction from the user: update `tasks/lessons.md` with the pattern
- Write rules for yourself that prevent the same mistake
- Ruthlessly iterate on these lessons until mistake rate drops
- Review lessons at session start for relevant project
### 4. Verification Before Done
- Never mark a task complete without proving it works
- Diff behavior between main and your changes when relevant
- Ask yourself: "Would a staff engineer approve this?"
- Run tests, check logs, demonstrate correctness
### 5. Demand Elegance (Balanced)
- For non-trivial changes: pause and ask "is there a more elegant way?"
- If a fix feels hacky: "Knowing everything I know now, implement the elegant solution"
- Skip this for simple, obvious fixes — don't over-engineer
- Challenge your own work before presenting it
### 6. Autonomous Bug Fixing
- When given a bug report: just fix it. Don't ask for hand-holding
- Point at logs, errors, failing tests — then resolve them
- Zero context switching required from the user
- Go fix failing CI tests without being told how
### Tools Execution
- Read any file over 500 LOC in chunks using offset and limit parameters.
- Do not assume a single grep caught everything.
### Code Quality
- If architecture is flawed, state is duplicated, or patterns are inconsistent - propose and implement structural fixes. Ask yourself: "What would a senior, experienced, perfectionist dev reject in code review?" Fix all of it.
Codex will review your output once you are done!
- Simplicity first: minimal, targeted changes. No temporary fixes — find the root cause.
- Where architecture is flawed, state is duplicated, or patterns are inconsistent, propose and implement the structural fix. Ask what a perfectionist reviewer would reject, and fix that too.
- On non-trivial changes, ask whether there is a more elegant way before presenting. Skip it for simple fixes — don't over-engineer.
## Task Management
1. **Plan First**: Write plan to `tasks/todo.md` with checkable items
2. **Verify Plan**: Check in before starting implementation
3. **Track Progress**: Mark items complete as you go
4. **Explain Changes**: High-level summary at each step
5. **Document Results**: Add review section to `tasks/todo.md`
6. **Capture Lessons**: Update `tasks/lessons.md` after corrections
Keep the plan in `tasks/todo.md` as checkable items, confirm it before implementing, tick items off as you go, and add a review section when done.
## Core Principles
## Customer-Facing Claims
- **Simplicity First**: Make every change as simple as possible. Impact minimal code.
- **No Laziness**: Find root causes. No temporary fixes. Senior developer standards.
- **Minimal Impact**: Changes should only touch what's necessary. Avoid introducing bugs.
`src/Web/autopilot-monitor-web/app/{about,terms,privacy}/page.tsx` and `autopilotmonitor-docs/trust/*` state facts customers and security reviewers rely on. They drift silently.
## Technical Docs — OKF Knowledge Bundle (`docs/`)
- Verify every claim against the code before writing it — never carry one forward because it was already on the page.
- Update them in the same change as: runtime/framework versions, roles, notification providers, agent deployment or lifecycle, diagnostics payload, isolation/delegation model, retention caps, sub-processors, or any default governing what is collected.
- Describe only what customers can actually use; operator-only infrastructure is not a feature.
- Trust pages: durable phrasing over exact figures, and bump their "Last reviewed" date.
`docs/` is an [Open Knowledge Format (OKF) v0.1](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md) knowledge bundle for contributor/AI-facing technical documentation (customer docs live in the separate autopilotmonitor-docs repo).
## Technical Docs — OKF Bundle (`docs/`)
- **Consume first**: start at `docs/index.md` for progressive disclosure before opening individual documents.
- **When new durable technical knowledge emerges** (architecture decisions, flows, non-obvious mechanisms), capture it as an OKF concept document in `docs/` — markdown with YAML frontmatter: `type` (mandatory), plus `title`, `description`, `resource`, `tags`, `timestamp` (recommended).
- **Maintain the bundle**: add every new document to `docs/index.md`, note changes in `docs/log.md` (grouped by ISO date). Use standard RELATIVE markdown links between docs — never the OKF `/`-prefixed bundle-absolute form (GitHub resolves those from the repo root and navigation breaks). `index.md` and `log.md` are reserved filenames.
- Documents are English, structural markdown over prose (`# Schema`, `# Examples`, `# Citations` sections where applicable).
Contributor/AI-facing docs (customer docs live in the `autopilotmonitor-docs` repo). Start at `docs/index.md` before opening individual files.
- New durable knowledge (architecture decisions, flows, non-obvious mechanisms) → an OKF concept doc with YAML frontmatter: `type` (mandatory), plus `title`, `description`, `resource`, `tags`, `timestamp`.
- Register it in `docs/index.md`, note the change in `docs/log.md` (ISO date). `index.md` and `log.md` are reserved names.
- English, structural markdown (`# Schema`, `# Examples`, `# Citations`). Links between docs are RELATIVE — never `/`-prefixed, that breaks GitHub navigation.
+31
View File
@@ -0,0 +1,31 @@
# CodeQL configuration.
#
# The default query suites stay active — this file only filters out two C# queries
# that fire structurally on this codebase, where the flagged behavior is the feature,
# not a defect:
#
# cs/exposure-of-sensitive-information
# Audit and operational logs record WHO performed an action (actor UPN / email)
# and notification services deliver mail to a recipient address. Writing that
# identity to the log / notification IS the purpose of the field. Every such site
# is a legitimate, reviewed use — the query cannot distinguish it from a leak.
#
# cs/cleartext-storage-of-sensitive-information
# The rate limiters and distress guard use the trusted client IP as their bucket
# key. The IP is an identifier for throttling, not a secret at rest; storing it in
# the in-memory / table rate-limit key is by design.
#
# Genuine one-off false positives from OTHER queries (e.g. an open-redirect the taint
# model can't see is allow-listed, a rate-limited route it doesn't recognize) are
# dismissed individually in the Security tab so those queries stay sharp — they are
# NOT suppressed here.
#
# Log-injection (cs/log-forging) is deliberately NOT filtered: user-controlled values
# are sanitized at the sink via Helpers/LogSanitizer, so the query stays useful for
# catching a future unsanitized sink.
query-filters:
- exclude:
id: cs/exposure-of-sensitive-information
- exclude:
id: cs/cleartext-storage-of-sensitive-information
+47
View File
@@ -0,0 +1,47 @@
version: 2
updates:
- package-ecosystem: nuget
directory: /
schedule:
interval: weekly
open-pull-requests-limit: 5
groups:
dotnet:
patterns: ['*']
update-types: [minor, patch]
- package-ecosystem: npm
directory: /src/Web/autopilot-monitor-web
schedule:
interval: weekly
open-pull-requests-limit: 5
groups:
web:
patterns: ['*']
update-types: [minor, patch]
# Major bumps are breaking and reviewed deliberately, not auto-PR'd.
# Security updates ignore this filter, so the Security tab stays clean.
ignore:
- dependency-name: '*'
update-types: ['version-update:semver-major']
- package-ecosystem: npm
directory: /src/McpServer/autopilot-monitor-mcp
schedule:
interval: weekly
open-pull-requests-limit: 5
groups:
mcp:
patterns: ['*']
update-types: [minor, patch]
# Major bumps are breaking and reviewed deliberately, not auto-PR'd.
# Security updates ignore this filter, so the Security tab stays clean.
ignore:
- dependency-name: '*'
update-types: ['version-update:semver-major']
- package-ecosystem: github-actions
directory: /
schedule:
interval: monthly
+102
View File
@@ -0,0 +1,102 @@
<#
.SYNOPSIS
Reserves a monotone build number from a counter blob (ETag compare-and-swap).
.DESCRIPTION
Single source of build numbers for every component that ships from this repo
(agent, backend, MCP). Each component owns one counter blob holding a plain
integer; this script reads it, writes value+1 back under If-Match, and returns
the reserved number.
Reserve-BEFORE-build: two concurrent builds can never mint the same number.
A failed build burns a number, which is harmless -- uniqueness matters,
density does not.
The blob is public-read; only the write needs the container SAS.
Why a counter and not the published version manifest: manifests are written
on publish, not on build (the agent writes version.json only for stable
releases), so two dev builds in a row would derive the same number and
overwrite each other's versioned artifact.
.PARAMETER CounterUrl
Full URL of the counter blob, without SAS.
.PARAMETER SasToken
Container SAS with create+write permission. A leading '?' is tolerated.
.PARAMETER Override
Explicit build number instead of counter+1. If it is not greater than the
current counter the blob is left untouched (re-build of a number that was
already reserved). 0 = no override.
.PARAMETER MaxAttempts
CAS retries before giving up.
.OUTPUTS
[int] the reserved build number. Progress goes to the host, not the pipeline.
.EXAMPLE
$n = ./Request-BuildNumber.ps1 -CounterUrl $url -SasToken $sas
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)][string]$CounterUrl,
[Parameter(Mandatory = $true)][string]$SasToken,
[int]$Override = 0,
[int]$MaxAttempts = 5
)
$ErrorActionPreference = 'Stop'
# Tolerate both secret formats ("sp=cw&..." and "?sp=cw&...").
$writeUrl = $CounterUrl + '?' + $SasToken.TrimStart('?')
function Get-HttpStatus {
param($ErrorRecord)
if ($ErrorRecord.Exception.PSObject.Properties['Response'] -and $ErrorRecord.Exception.Response) {
return [int]$ErrorRecord.Exception.Response.StatusCode
}
return 0
}
function Read-Counter {
try {
$resp = Invoke-WebRequest -Uri $CounterUrl -Method Get -UseBasicParsing
} catch {
if ((Get-HttpStatus $_) -eq 404) {
throw "Counter blob not found at $CounterUrl. Seed it once with the last used build number: PUT the plain integer with 'x-ms-blob-type: BlockBlob' and 'If-None-Match: *'. See docs/versioning.md."
}
throw
}
$raw = if ($resp.Content -is [byte[]]) { [System.Text.Encoding]::UTF8.GetString($resp.Content) } else { [string]$resp.Content }
# If-Match requires the quoted ETag form; PowerShell strips the quotes on read.
$etag = '"' + ([string]($resp.Headers['ETag'] | Select-Object -First 1)).Trim('"') + '"'
return [pscustomobject]@{ Value = [int]$raw.Trim(); ETag = $etag }
}
for ($attempt = 1; $attempt -le $MaxAttempts; $attempt++) {
$counter = Read-Counter
if ($Override -gt 0 -and $Override -le $counter.Value) {
Write-Host " Build number $Override <= counter $($counter.Value) -- already reserved, counter blob unchanged"
return $Override
}
$target = if ($Override -gt 0) { $Override } else { $counter.Value + 1 }
try {
$headers = @{ 'x-ms-blob-type' = 'BlockBlob'; 'If-Match' = $counter.ETag }
Invoke-RestMethod -Uri $writeUrl -Method Put -Headers $headers -Body "$target" -ContentType 'text/plain' | Out-Null
Write-Host " Reserved build number $target (CAS write ok, attempt $attempt)"
return $target
} catch {
if ((Get-HttpStatus $_) -eq 412) {
Write-Host " CAS conflict on attempt $attempt (another build reserved concurrently) -- retrying"
Start-Sleep -Seconds ($attempt * 2)
} else {
throw
}
}
}
throw "Could not reserve a build number after $MaxAttempts CAS attempts"
+40 -66
View File
@@ -56,77 +56,27 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v5
uses: actions/checkout@v7
# SDK 10 = single build toolchain (parity with local build.ps1 / global.json).
- name: Setup .NET
uses: actions/setup-dotnet@v5
uses: actions/setup-dotnet@v6
with:
dotnet-version: '8.0.x'
dotnet-version: '10.0.x'
# Build numbers come from the shared counter blob (single source of truth for CI
# AND local build.ps1). Reserve-before-build with ETag-CAS: concurrent builds can
# never mint the same number; a failed build burns a number, which is harmless.
# The blob is public-read (like the rest of the container); writes need the SAS.
# Build numbers come from the shared counter blob -- one counter and one
# implementation for CI and the local build (.github/scripts/Request-BuildNumber.ps1).
# Reserve-before-build with ETag-CAS: concurrent builds can never mint the same
# number; a failed build burns a number, which is harmless.
- name: Reserve build number (counter blob CAS)
id: version
shell: pwsh
run: |
$counterBlob = "buildcounter-v$env:LINE_MAJOR.txt"
$counterUrl = "https://autopilotmonitoreu.blob.core.windows.net/agent/$counterBlob"
# Tolerate both secret formats ("sp=cw&..." and "?sp=cw&...") -- the URL below adds the '?'.
$sasToken = "${{ secrets.AZURE_BLOB_AGENT_SAS_TOKEN }}".TrimStart('?')
$override = "${{ inputs.build_number }}"
$override = 0
if ('${{ inputs.build_number }}' -ne '') { $override = [int]'${{ inputs.build_number }}' }
function Get-Counter {
try {
$resp = Invoke-WebRequest -Uri $counterUrl -Method Get
$raw = if ($resp.Content -is [byte[]]) { [System.Text.Encoding]::UTF8.GetString($resp.Content) } else { [string]$resp.Content }
# If-Match requires the quoted ETag form; pwsh strips the quotes on read.
$etag = '"' + ([string]($resp.Headers['ETag'] | Select-Object -First 1)).Trim('"') + '"'
return @{ Value = [int]$raw.Trim(); ETag = $etag }
} catch {
$status = if ($_.Exception.Response) { [int]$_.Exception.Response.StatusCode } else { 0 }
if ($status -eq 404) {
Write-Error "Counter blob '$counterBlob' not found. Seed it once with the last used build number (see docs/agent/build-counter-blob.md) -- do NOT seed from the repo's buildcounter.txt, it is a stale offline fallback."
}
throw
}
}
$buildNumber = $null
for ($attempt = 1; $attempt -le 5; $attempt++) {
$counter = Get-Counter
if ($override -ne '') {
$target = [int]$override
if ($target -le $counter.Value) {
# Re-build of an already-reserved number: counter stays monotonic, no write.
Write-Host "Manual build number $target <= counter $($counter.Value) -- counter blob unchanged"
$buildNumber = $target
break
}
Write-Host "Manual build number $target > counter $($counter.Value) -- advancing counter"
} else {
$target = $counter.Value + 1
}
try {
$headers = @{ 'x-ms-blob-type' = 'BlockBlob'; 'If-Match' = $counter.ETag }
Invoke-RestMethod -Uri "$counterUrl`?$sasToken" -Method Put -Headers $headers -Body "$target" -ContentType 'text/plain' | Out-Null
$buildNumber = $target
Write-Host "Reserved build number ${target} (CAS write ok, attempt $attempt)"
break
} catch {
$status = if ($_.Exception.Response) { [int]$_.Exception.Response.StatusCode } else { 0 }
if ($status -eq 412) {
Write-Host "CAS conflict on attempt $attempt (another build reserved concurrently) -- retrying"
Start-Sleep -Seconds ($attempt * 2)
} else {
throw
}
}
}
if ($null -eq $buildNumber) {
Write-Error "Could not reserve a build number after 5 CAS attempts"
}
$counterUrl = "https://autopilotmonitoreu.blob.core.windows.net/agent/buildcounter-v$($env:LINE_MAJOR).txt"
$buildNumber = & ./.github/scripts/Request-BuildNumber.ps1 -CounterUrl $counterUrl -SasToken '${{ secrets.AZURE_BLOB_AGENT_SAS_TOKEN }}' -Override $override
$csproj = Get-Content $env:CSPROJ_PATH -Raw
if ($csproj -match '<VersionPrefix>(\d+\.\d+)</VersionPrefix>') {
@@ -227,8 +177,13 @@ jobs:
Write-Warning "Could not parse bootstrap `$ScriptVersion from $env:BOOTSTRAP_SOURCE"
}
# Enrich the per-line manifest emitted by MSBuild (version-v2.json, read above)
# Enrich the per-line manifest emitted by MSBuild (version-v2.json, read above).
# commit/buildUtc are the same two fields the backend and MCP manifests carry —
# additive, and the bootstrap reads this file by field name, so older installers
# ignore them. sha256/bootstrapVersion stay: those are the customer contract.
$manifest | Add-Member -NotePropertyName "sha256" -NotePropertyValue $sha256Hash -Force
$manifest | Add-Member -NotePropertyName "commit" -NotePropertyValue '${{ github.sha }}'.Substring(0, 7) -Force
$manifest | Add-Member -NotePropertyName "buildUtc" -NotePropertyValue ([DateTime]::UtcNow.ToString('o')) -Force
if ($bootstrapVersion) {
$manifest | Add-Member -NotePropertyName "bootstrapVersion" -NotePropertyValue $bootstrapVersion -Force
}
@@ -265,12 +220,23 @@ jobs:
run: |
$containerUrl = "https://autopilotmonitoreu.blob.core.windows.net/agent"
$sasToken = "${{ secrets.AZURE_BLOB_AGENT_SAS_TOKEN }}".TrimStart('?')
# Legacy mirror (transition until customers moved to the download.-alias):
# blobs are ALSO pushed to the old account so the bootstrap URL baked into
# customers' Intune scripts keeps serving current bits. Fail-soft -- a
# legacy hiccup must never fail a release. Counter lives ONLY on eu.
$legacyContainerUrl = "https://autopilotmonitor.blob.core.windows.net/agent"
$legacySasToken = "${{ secrets.AZURE_BLOB_AGENT_LEGACY_SAS_TOKEN }}".TrimStart('?')
function Send-Blob {
param([string]$Local, [string]$BlobName, [string]$ContentType)
$url = "$containerUrl/$BlobName`?$sasToken"
$bytes = [System.IO.File]::ReadAllBytes($Local)
$headers = @{ "x-ms-blob-type" = "BlockBlob"; "Content-Type" = $ContentType }
# no-cache on EVERY blob: they all rotate in place (even the per-line ZIP is
# overwritten each build), and the download alias fronts them with Front Door.
# Route caching is disabled there, but a stale manifest paired with a fresh ZIP
# (independent cache entries) would fail the bootstrap SHA check -- this header
# keeps that class of bug impossible even if caching is ever re-enabled.
$headers = @{ "x-ms-blob-type" = "BlockBlob"; "Content-Type" = $ContentType; "x-ms-blob-cache-control" = "no-cache" }
if ($ContentType -eq "application/zip") {
$md5 = [System.Security.Cryptography.MD5]::Create()
$headers["Content-MD5"] = [Convert]::ToBase64String($md5.ComputeHash($bytes))
@@ -279,6 +245,14 @@ jobs:
Write-Host "Uploading $BlobName ($([math]::Round($bytes.Length / 1MB, 2)) MB)..."
Invoke-RestMethod -Uri $url -Method Put -Headers $headers -Body $bytes
Write-Host " uploaded $BlobName"
if (-not [string]::IsNullOrEmpty($legacySasToken)) {
try {
Invoke-RestMethod -Uri "$legacyContainerUrl/$BlobName`?$legacySasToken" -Method Put -Headers $headers -Body $bytes
Write-Host " mirrored $BlobName to legacy storage"
} catch {
Write-Host "::warning::legacy mirror failed for $BlobName -- $($_.Exception.Message)"
}
}
}
# Versioned namespace (always)
@@ -337,13 +311,13 @@ jobs:
subject-path: ${{ steps.artifact.outputs.versioned_zip_release }}
- name: Upload build artifact
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@v7
with:
name: AutopilotMonitor-Agent-v${{ steps.version.outputs.version }}
path: ${{ steps.artifact.outputs.versioned_zip_release }}
- name: Upload version metadata
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@v7
with:
name: version-metadata
path: ${{ env.MANIFEST_NAME }}
+120
View File
@@ -0,0 +1,120 @@
name: CI
# Tests, typechecks and drift-checks every push to main and every pull request:
# the .NET suites, the web and MCP suites, and a guard that the generated rule
# files still match rules/guardrails.json.
#
# The deploy workflows build but never test, so this is the only place the suites
# actually run. It does not gate anything — main is deliberately unprotected, so a
# red run reports rather than blocks. Read the result and follow up.
on:
push:
branches: [main]
pull_request:
branches: [main]
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
jobs:
dotnet:
name: .NET tests (backend + agent)
# windows-latest is required: the agent test projects target net48.
runs-on: windows-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-dotnet@v6
with:
dotnet-version: '10.0.x'
- name: Restore
run: dotnet restore AutopilotMonitor.sln
- name: Build
run: dotnet build AutopilotMonitor.sln --configuration Release --no-restore
- name: Backend tests
run: dotnet test src/Backend/AutopilotMonitor.Functions.Tests --configuration Release --no-build --verbosity quiet
- name: Agent V2 Core tests
run: dotnet test src/Agent/AutopilotMonitor.Agent.V2.Core.Tests --configuration Release --no-build --verbosity quiet
- name: Decision Core tests
run: dotnet test src/Agent/AutopilotMonitor.DecisionCore.Tests --configuration Release --no-build --verbosity quiet
- name: Summary Dialog tests
run: dotnet test src/Agent/AutopilotMonitor.SummaryDialog.Tests --configuration Release --no-build --verbosity quiet
web:
name: Web (typecheck + tests)
runs-on: ubuntu-latest
defaults:
run:
working-directory: src/Web/autopilot-monitor-web
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: '22'
cache: npm
cache-dependency-path: src/Web/autopilot-monitor-web/package-lock.json
- run: npm ci
- name: Typecheck
run: npx tsc --noEmit
- name: Tests
run: npm test
mcp:
name: MCP server tests
runs-on: ubuntu-latest
defaults:
run:
working-directory: src/McpServer/autopilot-monitor-mcp
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: '22'
cache: npm
cache-dependency-path: src/McpServer/autopilot-monitor-mcp/package-lock.json
- run: npm ci
- name: Typecheck
run: npx tsc --noEmit
- name: Tests
run: npm test
guardrails-in-sync:
name: Generated guardrails match rules/guardrails.json
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
- uses: actions/setup-node@v7
with:
node-version: '22'
- name: Regenerate
run: node rules/scripts/combine.js
- name: Fail if the generated files drifted
run: |
if ! git diff --quiet; then
echo "::error::Generated files are out of date. Run 'node rules/scripts/combine.js' and commit the result."
git diff --stat
exit 1
fi
+73
View File
@@ -0,0 +1,73 @@
name: CodeQL
# Documentation-only changes cannot affect what CodeQL analyzes (C# and
# TypeScript), so they are filtered out rather than spending ~10 minutes to
# re-report an unchanged result. A commit that touches docs AND code still runs:
# GitHub skips only when EVERY changed path matches the filter.
on:
push:
branches: [main]
paths-ignore:
- 'docs/**'
- '**/*.md'
- 'LICENSE'
pull_request:
branches: [main]
paths-ignore:
- 'docs/**'
- '**/*.md'
- 'LICENSE'
schedule:
# Weekly, so advisories published after a merge still surface. Deliberately
# unfiltered — it re-analyzes the whole tree regardless of what changed, which
# is also the backstop if the filters above ever skip too much.
- cron: '17 3 * * 1'
permissions:
contents: read
jobs:
analyze:
name: Analyze ${{ matrix.language }}
runs-on: ${{ matrix.runner }}
permissions:
security-events: write
packages: read
actions: read
contents: read
strategy:
fail-fast: false
matrix:
include:
# csharp needs the net48 agent projects to build, so Windows.
- language: csharp
runner: windows-latest
build-mode: manual
- language: javascript-typescript
runner: ubuntu-latest
build-mode: none
steps:
- uses: actions/checkout@v7
- uses: actions/setup-dotnet@v6
if: matrix.language == 'csharp'
with:
dotnet-version: '10.0.x'
- name: Initialize CodeQL
uses: github/codeql-action/init@v4
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
config-file: .github/codeql/codeql-config.yml
- name: Build
if: matrix.build-mode == 'manual'
run: dotnet build AutopilotMonitor.sln --configuration Release
- name: Analyze
uses: github/codeql-action/analyze@v4
with:
category: /language:${{ matrix.language }}
+3 -3
View File
@@ -22,11 +22,11 @@ jobs:
permissions:
contents: write
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- uses: actions/setup-node@v5
- uses: actions/setup-node@v7
with:
node-version: '20'
node-version: '22'
- name: Install ajv-cli
run: npm install -g ajv-cli ajv-formats
+86 -6
View File
@@ -7,13 +7,19 @@ permissions:
id-token: write # OIDC federated login (secret-less)
contents: read
env:
FUNCTION_APP: autopilotmonitor-api-eu
HEALTH_URL: https://autopilotmonitor-api-eu.azurewebsites.net/api/health
VERSIONS_CONTAINER: https://autopilotmonitoreu.blob.core.windows.net/versions
CSPROJ_PATH: src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj
jobs:
build-and-deploy:
runs-on: ubuntu-latest
name: Build and Deploy Backend
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- name: Login to Azure (OIDC)
uses: azure/login@v3
@@ -23,21 +29,95 @@ jobs:
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
- name: Setup .NET 10
uses: actions/setup-dotnet@v5
uses: actions/setup-dotnet@v6
with:
dotnet-version: '10.0.x'
# Same counter mechanism as the agent line: reserve-before-build with ETag-CAS
# against a per-component counter blob. BuildTimestampUtc is pinned here rather
# than left to MSBuild's default so the published manifest and the value baked
# into the assembly (reported by /api/health) are the same instant.
- name: Reserve build number (counter blob CAS)
id: version
shell: pwsh
run: |
$buildNumber = & ./.github/scripts/Request-BuildNumber.ps1 -CounterUrl "${{ env.VERSIONS_CONTAINER }}/backend.txt" -SasToken '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'
$csproj = Get-Content ${{ env.CSPROJ_PATH }} -Raw
if ($csproj -notmatch '<VersionPrefix>(\d+\.\d+)</VersionPrefix>') {
throw "VersionPrefix not found in ${{ env.CSPROJ_PATH }}"
}
$version = "$($Matches[1]).$buildNumber"
$buildUtc = [DateTime]::UtcNow.ToString('o')
Write-Host "Backend version: $version (built $buildUtc)"
"build_number=$buildNumber" >> $env:GITHUB_OUTPUT
"version=$version" >> $env:GITHUB_OUTPUT
"build_utc=$buildUtc" >> $env:GITHUB_OUTPUT
- name: Restore dependencies
run: dotnet restore src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj
run: dotnet restore ${{ env.CSPROJ_PATH }}
- name: Build
run: dotnet build src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj --configuration Release --no-restore -p:SourceRevisionId=${{ github.sha }}
run: dotnet build ${{ env.CSPROJ_PATH }} --configuration Release --no-restore -p:SourceRevisionId=${{ github.sha }} -p:BuildNumber=${{ steps.version.outputs.build_number }} -p:BuildTimestampUtc=${{ steps.version.outputs.build_utc }}
- name: Publish
run: dotnet publish src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj --configuration Release --output ./publish --no-build -p:SourceRevisionId=${{ github.sha }}
run: dotnet publish ${{ env.CSPROJ_PATH }} --configuration Release --output ./publish --no-build -p:SourceRevisionId=${{ github.sha }} -p:BuildNumber=${{ steps.version.outputs.build_number }} -p:BuildTimestampUtc=${{ steps.version.outputs.build_utc }}
- name: Deploy to Azure Functions
uses: Azure/functions-action@v1
with:
app-name: autopilotmonitor-api-eu
app-name: ${{ env.FUNCTION_APP }}
package: ./publish
# A green deploy step only means the package was accepted. This gate makes the
# workflow assert what is actually being served, so the manifest published below
# can be trusted as "this is live" rather than "this was uploaded".
# /api/health is IP rate-limited at 30/min; one poll per 10s stays well under.
- name: Verify deployed build is live
shell: pwsh
run: |
$expectedCommit = '${{ github.sha }}'.Substring(0, 7)
$expectedVersion = '${{ steps.version.outputs.version }}'
$deadline = [DateTime]::UtcNow.AddMinutes(3)
$attempt = 0
while ([DateTime]::UtcNow -lt $deadline) {
$attempt++
try {
$health = Invoke-RestMethod -Uri '${{ env.HEALTH_URL }}' -Method Get -TimeoutSec 30
if ($health.commitHash -eq $expectedCommit -and $health.version -eq $expectedVersion) {
Write-Host "Live after $attempt attempt(s): $($health.version) @ $($health.commitHash), built $($health.buildUtc)"
exit 0
}
Write-Host "Attempt ${attempt}: serving $($health.version) @ $($health.commitHash) -- waiting for $expectedVersion @ $expectedCommit"
} catch {
Write-Host "Attempt ${attempt}: /api/health not answering yet -- $($_.Exception.Message)"
}
Start-Sleep -Seconds 10
}
throw "Backend did not report $expectedVersion @ $expectedCommit within 3 minutes -- deployment not verified, version manifest NOT published"
# The manifest is the cross-component record of what is deployed where. It is
# written last and only on a verified deploy, so a difference between the
# manifest and /api/health means the running app drifted after the fact
# (rollback, stale instance) rather than a half-finished pipeline.
- name: Publish version manifest
shell: pwsh
run: |
$manifest = [ordered]@{
component = 'backend'
version = '${{ steps.version.outputs.version }}'
commit = '${{ github.sha }}'.Substring(0, 7)
buildUtc = '${{ steps.version.outputs.build_utc }}'
deployedUtc = [DateTime]::UtcNow.ToString('o')
runId = '${{ github.run_id }}'
} | ConvertTo-Json
$sas = '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'.TrimStart('?')
$url = "${{ env.VERSIONS_CONTAINER }}/backend.json?$sas"
# no-cache: rotating pointer, must never be served stale by any fronting cache.
Invoke-RestMethod -Uri $url -Method Put -Headers @{ 'x-ms-blob-type' = 'BlockBlob'; 'x-ms-blob-cache-control' = 'no-cache' } -Body $manifest -ContentType 'application/json' | Out-Null
Write-Host "Published versions/backend.json"
Write-Host $manifest
+100 -1
View File
@@ -12,6 +12,9 @@ env:
CONTAINER_APP_NAME: autopilotmonitor-mcp
RESOURCE_GROUP: ${{ secrets.AZURE_MCP_RESOURCE_GROUP }}
IMAGE_TAG: ${{ github.sha }}
HEALTH_URL: https://mcp.autopilotmonitor.com/health
VERSIONS_CONTAINER: https://autopilotmonitoreu.blob.core.windows.net/versions
PACKAGE_JSON: src/McpServer/autopilot-monitor-mcp/package.json
jobs:
build-and-deploy:
@@ -19,7 +22,23 @@ jobs:
name: Build and Deploy MCP Server
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
# The published customer documentation lives in its own PUBLIC repo and is
# baked into the image as the search_docs corpus (see the Dockerfile). No
# token is needed. Checking it out here rather than vendoring it keeps the
# docs single-sourced — but it also means EDITING THE DOCS DOES NOT UPDATE A
# RUNNING SERVER; this workflow must be re-run. /health reports the SHA
# captured below so that staleness is checkable.
- name: Check out documentation bundle
uses: actions/checkout@v7
with:
repository: autopilotmonitor/autopilotmonitor-docs
path: mcp-docs
- name: Resolve docs commit
id: docs
run: echo "sha=$(git -C mcp-docs rev-parse HEAD)" >> "$GITHUB_OUTPUT"
- name: Login to Azure (OIDC)
uses: azure/login@v3
@@ -28,6 +47,30 @@ jobs:
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
# Same counter mechanism as agent and backend: reserve-before-build with
# ETag-CAS against the component's own counter blob. major.minor stays curated
# in package.json; only the build part is minted here.
- name: Reserve build number (counter blob CAS)
id: version
shell: pwsh
run: |
$buildNumber = & ./.github/scripts/Request-BuildNumber.ps1 -CounterUrl "${{ env.VERSIONS_CONTAINER }}/mcp.txt" -SasToken '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'
$pkgVersion = (Get-Content ${{ env.PACKAGE_JSON }} -Raw | ConvertFrom-Json).version
$prefix = ($pkgVersion -split '\.')[0..1] -join '.'
$version = "$prefix.$buildNumber"
$buildUtc = [DateTime]::UtcNow.ToString('o')
Write-Host "MCP version: $version (built $buildUtc)"
# 7 chars, matching what the backend's BackendBuildInfo reports — one commit
# shape across all components so the values can be compared as-is.
$shortSha = '${{ github.sha }}'.Substring(0, 7)
"build_number=$buildNumber" >> $env:GITHUB_OUTPUT
"version=$version" >> $env:GITHUB_OUTPUT
"build_utc=$buildUtc" >> $env:GITHUB_OUTPUT
"short_sha=$shortSha" >> $env:GITHUB_OUTPUT
- name: Login to ACR
run: az acr login --name ${{ env.ACR_NAME }}
@@ -35,6 +78,10 @@ jobs:
run: |
docker build \
-f src/McpServer/autopilot-monitor-mcp/Dockerfile \
--build-arg DOCS_COMMIT=${{ steps.docs.outputs.sha }} \
--build-arg BUILD_NUMBER=${{ steps.version.outputs.build_number }} \
--build-arg BUILD_COMMIT=${{ steps.version.outputs.short_sha }} \
--build-arg BUILD_UTC=${{ steps.version.outputs.build_utc }} \
-t ${{ env.ACR_NAME }}.azurecr.io/${{ env.CONTAINER_APP_NAME }}:${{ env.IMAGE_TAG }} \
-t ${{ env.ACR_NAME }}.azurecr.io/${{ env.CONTAINER_APP_NAME }}:latest \
.
@@ -46,3 +93,55 @@ jobs:
--name ${{ env.CONTAINER_APP_NAME }} \
--resource-group ${{ env.RESOURCE_GROUP }} \
--image ${{ env.ACR_NAME }}.azurecr.io/${{ env.CONTAINER_APP_NAME }}:${{ env.IMAGE_TAG }}
# `az containerapp update` returns once the revision is provisioned, not once
# it serves traffic — and with minReplicas=0 the first request also pays a cold
# start. Polling /health until it reports THIS build is what makes the manifest
# below mean "live"; a full 5 minutes because the cold start alone is ~25s.
- name: Verify deployed build is live
shell: pwsh
run: |
$expectedCommit = '${{ steps.version.outputs.short_sha }}'
$expectedVersion = '${{ steps.version.outputs.version }}'
$deadline = [DateTime]::UtcNow.AddMinutes(5)
$attempt = 0
while ([DateTime]::UtcNow -lt $deadline) {
$attempt++
try {
$health = Invoke-RestMethod -Uri '${{ env.HEALTH_URL }}' -Method Get -TimeoutSec 60
if ($health.commit -eq $expectedCommit -and $health.version -eq $expectedVersion) {
Write-Host "Live after $attempt attempt(s): $($health.version) @ $($health.commit), docs $($health.docs.commit) ($($health.docs.chunks) chunks)"
exit 0
}
Write-Host "Attempt ${attempt}: serving $($health.version) @ $($health.commit) -- waiting for $expectedVersion @ $expectedCommit"
} catch {
Write-Host "Attempt ${attempt}: /health not answering yet -- $($_.Exception.Message)"
}
Start-Sleep -Seconds 15
}
throw "MCP did not report $expectedVersion @ $expectedCommit within 5 minutes -- deployment not verified, version manifest NOT published"
# Cross-component record of what is deployed where; written last and only on a
# verified deploy. docsCommit rides along because for this component "which
# build" and "which documentation" are two separate questions.
- name: Publish version manifest
shell: pwsh
run: |
$manifest = [ordered]@{
component = 'mcp'
version = '${{ steps.version.outputs.version }}'
commit = '${{ steps.version.outputs.short_sha }}'
buildUtc = '${{ steps.version.outputs.build_utc }}'
deployedUtc = [DateTime]::UtcNow.ToString('o')
runId = '${{ github.run_id }}'
docsCommit = '${{ steps.docs.outputs.sha }}'.Substring(0, 7)
} | ConvertTo-Json
$sas = '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'.TrimStart('?')
$url = "${{ env.VERSIONS_CONTAINER }}/mcp.json?$sas"
# no-cache: rotating pointer, must never be served stale by any fronting cache.
Invoke-RestMethod -Uri $url -Method Put -Headers @{ 'x-ms-blob-type' = 'BlockBlob'; 'x-ms-blob-cache-control' = 'no-cache' } -Body $manifest -ContentType 'application/json' | Out-Null
Write-Host "Published versions/mcp.json"
Write-Host $manifest
@@ -1,4 +1,4 @@
name: Azure Static Web Apps CI/CD
name: Deploy Web (Static Web App)
on:
push:
@@ -6,7 +6,7 @@ on:
- main
paths:
- 'src/Web/autopilot-monitor-web/**'
- '.github/workflows/azure-static-web-apps-brave-grass-0070b5703.yml'
- '.github/workflows/deploy-web.yml'
workflow_dispatch:
pull_request:
types: [opened, synchronize, reopened, closed]
@@ -14,20 +14,28 @@ on:
- main
paths:
- 'src/Web/autopilot-monitor-web/**'
- '.github/workflows/azure-static-web-apps-brave-grass-0070b5703.yml'
- '.github/workflows/deploy-web.yml'
jobs:
build_and_deploy_job:
if: github.event_name == 'push' || github.event_name == 'workflow_dispatch' || (github.event_name == 'pull_request' && github.event.action != 'closed')
runs-on: ubuntu-latest
name: Build and Deploy Job
# Dependabot PRs run without repo secrets, so the deploy token is absent.
# Surface it as an env var so the deploy step can gate on it — secrets
# cannot be referenced directly in an `if:` expression.
env:
DEPLOY_TOKEN: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with:
lfs: false
fetch-depth: 0 # Full history needed for git-based sitemap lastmod dates
- name: Build And Deploy
id: builddeploy
# Skip (green) instead of failing when the token is missing (Dependabot
# PRs); real deploys on main and internal PRs still have it.
if: env.DEPLOY_TOKEN != ''
uses: Azure/static-web-apps-deploy@v1
env:
NEXT_PUBLIC_ENTRA_CLIENT_ID: ${{ vars.NEXT_PUBLIC_ENTRA_CLIENT_ID }}
@@ -37,7 +45,7 @@ jobs:
NEXT_PUBLIC_PLATFORM_STATS_MANIFEST_URL: ${{ vars.NEXT_PUBLIC_PLATFORM_STATS_MANIFEST_URL }}
NEXT_PUBLIC_APPINSIGHTS_CONNECTION_STRING: ${{ secrets.NEXT_PUBLIC_APPINSIGHTS_CONNECTION_STRING }}
with:
azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_BRAVE_GRASS_0070B5703 }}
azure_static_web_apps_api_token: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
repo_token: ${{ secrets.GITHUB_TOKEN }} # Used for Github integrations (i.e. PR comments)
action: "upload"
###### Repository/Build Configurations - These values can be configured to match your app requirements. ######
@@ -51,12 +59,15 @@ jobs:
if: github.event_name == 'pull_request' && github.event.action == 'closed'
runs-on: ubuntu-latest
name: Close Pull Request Job
env:
DEPLOY_TOKEN: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
steps:
- name: Close Pull Request
id: closepullrequest
if: env.DEPLOY_TOKEN != ''
uses: Azure/static-web-apps-deploy@v1
with:
azure_static_web_apps_api_token: ${{ secrets.AZURE_STATIC_WEB_APPS_API_TOKEN_BRAVE_GRASS_0070B5703 }}
azure_static_web_apps_api_token: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
action: "close"
+5 -2
View File
@@ -117,8 +117,11 @@ scripts/Deployment/
# Migration tooling (local only, vorerst) — cross-tenant storage/cert migration scripts
scripts/Migration/
# Infra (local only)
infra/
# Infra: tracked. The Bicep templates carry no credentials — every secret is a
# @secure() parameter with an empty default, injected at deploy time, and image
# pull runs on managed identity. Keep it that way: never commit a resolved secret
# value or a *.parameters.json holding one.
infra/*.parameters.json
# M2 — local raw session fixtures before anonymization (plan §4 M2)
tests/fixtures/enrollment-sessions-local/
+1 -1
View File
@@ -34,7 +34,7 @@ For detailed information about the system architecture, components, and data flo
## Documentation
Full documentation is available at **[autopilotmonitor.com/docs](https://www.autopilotmonitor.com/docs)**
Full admin documentation is available at **[docs.autopilotmonitor.com](https://docs.autopilotmonitor.com)**
## License
+2 -2
View File
@@ -53,7 +53,7 @@ try {
Write-Host " ✓ .NET SDK version: $dotnetVersion" -ForegroundColor Green
}
catch {
Write-Host " ✗ .NET SDK not found. Please install .NET 8 SDK." -ForegroundColor Red
Write-Host " ✗ .NET SDK not found. Please install .NET 10 SDK." -ForegroundColor Red
exit 1
}
@@ -145,7 +145,7 @@ Write-Host "============================================" -ForegroundColor Cyan
Write-Host ""
Write-Host "Output locations:" -ForegroundColor White
Write-Host " Agent: src\Agent\AutopilotMonitor.Agent.V2\bin\$Configuration\net48\" -ForegroundColor Gray
Write-Host " Functions: src\Backend\AutopilotMonitor.Functions\bin\$Configuration\net8.0\" -ForegroundColor Gray
Write-Host " Functions: src\Backend\AutopilotMonitor.Functions\bin\$Configuration\net10.0\" -ForegroundColor Gray
if ($IncludeWeb) {
Write-Host " Web UI: src\Web\autopilot-monitor-web\.next\" -ForegroundColor Gray
+16 -19
View File
@@ -22,26 +22,21 @@ attestation on top.
# Schema
- **Counter blob**: `https://autopilotmonitor.blob.core.windows.net/agent/buildcounter-v{LINE}.txt`
(per release line, analogous to `version-v{LINE}.json`). Content: the **last used**
build number as plain text. Public-read like the rest of the container (the number
is visible in `version-v{LINE}.json` anyway); writes require the container SAS.
- **Protocol — reserve before build, ETag-CAS**:
1. Anonymous `GET` the blob, capture value `N` and the `ETag`.
2. `PUT N+1` with `If-Match: <etag>` (quoted form — pwsh strips the quotes on read,
they must be re-added or the request is rejected client-side).
3. `412 Precondition Failed` → another build reserved concurrently → re-read, retry
(5 attempts, linear backoff).
4. Build with `BUILD_NUMBER=N+1` in the environment. The csproj honors
`BUILD_NUMBER` and then skips its `PersistBuildCounter` target, so the repo file
is not touched.
- **Failed builds burn a number.** That is intentional: uniqueness matters, density
does not. Reserve-before-build is what makes concurrent minting impossible.
The reserve-before-build CAS protocol itself is shared with the backend and the MCP
server and is documented once in [Version Contract](../versioning.md); the single
implementation is `.github/scripts/Request-BuildNumber.ps1`. Agent-specific:
- **Counter blob**: `https://autopilotmonitoreu.blob.core.windows.net/agent/buildcounter-v{LINE}.txt`
— one per release line, analogous to `version-v{LINE}.json`, and kept in the `agent`
container next to the artifacts it numbers rather than in `versions/`. Content: the
**last used** build number as plain text. Public-read like the rest of the container
(the number is visible in `version-v{LINE}.json` anyway); writes require the SAS.
- The reserved number reaches the build as `BUILD_NUMBER` in the environment. The
csproj honors it and then skips its `PersistBuildCounter` target, so the repo file
is not touched.
- **CI manual override** (`build_number` input): numbers `<=` counter are re-builds
and leave the blob unchanged; numbers `>` counter advance it (monotonic guard).
- **Seeding** (one-time per line): `PUT <last used number>` with `If-None-Match: *`
so an existing counter can never be clobbered. V2 was seeded with `1356` on
2026-07-17.
- V2 was seeded with `1356` on 2026-07-17.
# Consequences
@@ -62,7 +57,9 @@ attestation on top.
# Citations
- [Version Contract](../versioning.md) — the shared CAS protocol and manifest format
- `.github/workflows/build-agent.yml` — "Reserve build number (counter blob CAS)" step
- `scripts/Deployment/build.ps1` (gitignored, contains SAS) — `Request-BuildNumber`
- `.github/scripts/Request-BuildNumber.ps1` — the shared implementation
- `scripts/Deployment/build.ps1` (gitignored, contains SAS) — local build path
- `src/Agent/AutopilotMonitor.Agent.V2/AutopilotMonitor.Agent.V2.csproj` —
`BUILD_NUMBER` property + `PersistBuildCounter` target
+11 -2
View File
@@ -137,8 +137,17 @@ false` and an arm holds), `EspExiting`, `ImeUserSessionCompleted`,
advisory backstop. All routes converge on `CompleteThroughFinalizingOrDefer`:
1. **Completion gates** (currently only the RealmJoin AND-gate): if RealmJoin was
detected and is neither resolved nor timed-out (hard 60-min timeout), completion is
*deferred* — the session stays in stage and emits `completion_waiting`.
detected and has no terminal outcome yet, completion is *deferred* — the session
stays in stage and emits `completion_waiting`. Three terminals open the gate:
`Resolved` (DeploymentPhase 110), `FirstDeploymentIncomplete` (phase left the
first-deployment window 100/101 for 200/210 without 110 — the RJ ESP was aborted,
typically by an interactive logon reclassifying the run as secondary-user; emits
the `realmjoin_first_deployment_incomplete` Warning, session 224b2087), and
`Timeout` (hard 60-min deadline). The abort rule reads the *persisted*
`RealmJoinFacts.LastDeploymentPhase` (updated by the typed `RealmJoinPhaseChanged`
signal), so it survives agent restarts — the `RealmJoinDetected` replay path
evaluates it too. A first observation of 210 does **not** release (RJ may have
completed before agent boot and deploy afterwards).
2. `Finalizing` stage + ~5 s `FinalizingGrace` deadline.
3. Grace fires → `Completed`, `EnrollmentComplete` outcome, `enrollment_complete`
timeline event with full audit payload.
+88
View File
@@ -0,0 +1,88 @@
---
type: Concept
title: Agent Endpoint Migration — Config-Channel Re-Home
description: How the backend re-homes agents to a new API base URL (backend move or per-tenant region move) via MigrateToApiBaseUrl on the agent config response, and why HTTP redirects were rejected.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Configuration/EndpointMigration.cs
tags:
- agent
- backend
- config
- migration
- security
timestamp: 2026-07-18T00:00:00+02:00
---
# Problem
The agent's API base URL is compiled into the binary (`Constants.ApiBaseUrl`). No DNS/domain
indirection is possible in front of the API: mTLS client-cert auth does not survive a reverse
proxy (Front Door), and Flex Consumption custom-domain TLS is broken. A backend move (like the
2026-07-18 WEU→GWC subscription migration) therefore hard-cuts every deployed agent still
pointing at the old hostname. In-flight agents self-terminate safely (local MaxLifetime watchdog
+ CleanupService), but their sessions are lost and a per-tenant region move (EU→US) has no
mechanism at all.
# Mechanism
`AgentConfigResponse.MigrateToApiBaseUrl` — served on `GET /api/agent/config` (incl. the
bootstrap wrapper), the same control channel as the kill switch, fetched at every agent
process start.
Agent flow (`Program.RunAgent`, Phase 4):
1. Config fetched from the compiled-in (old) base URL.
2. Kill-switch check runs FIRST — **kill wins over migrate**.
3. `AgentRuntimeConfig.Resolve` detects the migration signal directly after the fetch and
returns early, BEFORE merge / binary-integrity / bootstrap-cleanup — none of those side
effects may run against a config from the backend being abandoned (a migration stub may
advertise stale agent hashes; a spurious `IntegrityCheckFailed` + force-update would be
harmful).
4. `Program.RunAgent` swaps `AgentConfiguration.ApiBaseUrl`, disposes the old client stack,
rebuilds Phase 3 (auth clients) and re-runs Phase 4 against the new URL —
`allowEndpointMigration: false`, so the new backend's config **cannot chain a second hop**.
5. Kill-switch check runs again on the NEW backend's config; startup continues normally
(Phase 5+ builds mTLS/telemetry clients from the updated base URL).
# Hardening (mirrors the kill switch)
* **Live fetch only** — the on-disk config cache strips `MigrateToApiBaseUrl` on write AND
read (`RemoteConfigService.CacheConfig` / `LoadCachedConfig`); `EndpointMigration.ResolveTarget`
re-checks `LastFetchOutcome == Succeeded` as defence-in-depth. A planted cache file can
never re-home an agent.
* **Allowlist validated on BOTH sides** — `AgentEndpointMigrationRules` (Shared, single
source): https only, host must end in `.azurewebsites.net` on a label boundary, no
path/query/userinfo/port. The backend refuses to serve an invalid admin value
(`AgentMigrateRejected` warning); the agent refuses to honour one even if the backend was
compromised.
* **One hop, no loops** — second Resolve pass never migrates; a target equal to the current
base URL is a no-op (steady state during the migration window).
# Operation
Global admin settings (`AdminConfiguration`, admin portal → Global Settings):
* `AgentMigrateApiBaseUrl` — global target, set on the backend being **abandoned**; clear it
after the migration window.
* `AgentMigrateTenantOverridesJson` — JSON object `tenantId → URL` for per-tenant moves.
An entry with an **empty string** pins that tenant against the global target (staged rollout).
Overrides win over the global value.
Delivery evidence: every served target logs `AgentMigrateServed` (Warning — reaches App
Insights) with tenant, serial, agent version and target. Agent-side, the re-home is logged as
`Endpoint migration: backend re-homed this agent from … to …`.
Reach limitation: agents that cannot complete an authenticated config fetch (e.g. cert-broken
devices) never see the signal — they fall back to the local MaxLifetime/emergency-break
lifecycle. During a migration window the old backend must keep running as a serving stub;
keeping the old `*.azurewebsites.net` name parked afterwards also prevents hostname
re-registration by third parties while stragglers drain.
# Citations
* `src/Shared/AutopilotMonitor.Shared/Services/AgentEndpointMigrationRules.cs` — shared validation rules
* `src/Shared/AutopilotMonitor.Shared/Models/Config/AgentConfigResponse.cs` — wire field
* `src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs` — `ResolveMigrateTarget` + serving (ConfigVersion 33)
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Configuration/EndpointMigration.cs` — agent-side evaluation
* `src/Agent/AutopilotMonitor.Agent.V2/Runtime/AgentRuntimeConfig.cs` — early return before merge/integrity
* `src/Agent/AutopilotMonitor.Agent.V2/Program.cs` — one-hop rebuild in `RunAgent`
* Tests: `AgentEndpointMigrationTests` (backend), `EndpointMigrationTests` + `RemoteConfigServiceCacheHardeningTests` (agent)
+151
View File
@@ -0,0 +1,151 @@
---
type: Concept
title: ESP Policy-Provider Stall Detection — the no-timeout Setup/Apps wait and the co-management Sidecar gap
description: Why the ESP's Apps wait is keyed to the Intune-registered "Sidecar" provider by name — a foreign provider (ConfigMgr) with TrackingPoliciesCreated=1 does not satisfy it — plus the classic never-completes stall, and how the always-on two-arm esp_policy_provider_stalled tripwire (reason=provider_incomplete | sidecar_provider_missing, 15-min wall-clock dwell, StartupEventGate cross-restart dedup) surfaces both.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/EspPolicyProviderProbe.cs
tags:
- agent
- esp
- enrollment-status-tracking
- co-management
- stall-detection
timestamp: 2026-07-24T00:00:00+02:00
---
# ESP Policy-Provider Stall Detection
## Schema
The Enrollment Status Page decides when its "Apps" category may progress from
"Identifying" using the **EnrollmentStatusTracking CSP**
(<https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>),
mirrored in the registry under
`HKLM\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\{Device|<UserSID>}\…`.
Two contracts matter:
1. **`Setup/Apps/PolicyProviders/<Name>`** — per the CSP doc: *"Existence of this node
indicates to the ESP that it shouldn't show the tracking status message until the
TrackingPoliciesCreated node has been set to true."* This wait has **no timeout**. A
registered provider that never sets `TrackingPoliciesCreated=1` parks the ESP at
"Apps (Identifying)" indefinitely — observed for days in the field. Exists in device
scope (`Device`) and user scope (`S-<SID>`, appears at sign-in).
2. **`DevicePreparation/PolicyProviders/<Name>/InstallationState`** — must reach 2
(NotRequired) or 3 (Completed); 1=NotInstalled, 4=Error. Device scope only. The ESP
applies a default 15-minute timeout here, so these stalls self-surface as ESP errors —
the detector still records them for the diagnostic trail.
Known provider names: **`Sidecar` is the Intune Management Extension** (the CSP doc names
"Intune's agents, such as SideCar"); **`ConfigMgr`** is the Configuration Manager client,
registered during Autopilot-into-co-management — the troubleshooting section of
<https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment> queries
exactly these registry paths. The same document states that ESP policy providers **are not
aware of each other** and that **pre-provisioning is unsupported** with Autopilot into
co-management.
## The field case (issue #106)
A tenant pre-provisions devices and installs the ConfigMgr client as a blocking app.
Occasionally the user-phase ESP reaches "Apps (Identifying)" and never leaves it — for
hours to days. On those devices `Device\Setup\Apps\PolicyProviders` contains **only a
`ConfigMgr` key (no `Sidecar`)** — and, per the customer's follow-up, **`TrackingPoliciesCreated=1`
IS set under it**. The provider is "complete" by the CSP value contract, yet the ESP
still waits. Manually renaming the key to `Sidecar` — which creates no tracking entries,
only satisfies the name — unblocks the ESP instantly.
The naive CSP-doc reading ("any provider with TrackingPoliciesCreated=1 satisfies the
wait") is therefore wrong for this case. What the evidence supports instead: provider
nodes are registered **by the Intune service at enrollment**, and its ESP-policy WAP
payload always registers `Sidecar` (ConfigMgr joins only via a co-management settings
policy — see the payload documented by Niehaus). The ESP waits on the provider it
expects **by name**; a foreign provider occupying the slot does not satisfy it,
regardless of its completion values. On the affected devices the CM client (installed
as a plain Win32 app, no co-management settings policy) evidently displaced the
Sidecar node — consistent with Microsoft's warning that providers "aren't currently
aware of others" and that pre-provisioning is unsupported with co-management. The exact
CloudExperienceHost mechanism is not publicly documented; the registry signature —
setupApps providers present, `Sidecar` absent — is deterministic either way.
## Detection
`EspPolicyProviderProbe` (fail-soft, static, `ScopedOverride` test seam — structural clone
of `EspTrackingInfoProbe`) enumerates all providers in both kinds and scopes.
`EspPolicyProviderStallDetector` evaluates a pure transition function on a 60-second tick
from the always-on `EspPolicyProviderStallHost` — **two arms**, same 15-minute wall-clock
dwell, one shared event type discriminated by `reason`:
* **Arm 1, `reason=provider_incomplete`** — a provider **continuously incomplete for
≥ 15 minutes**. The dwell clock starts at first incomplete observation and resets on
completion or disappearance. Missing values parse as incomplete
(fail-toward-observation). Catches e.g. Sidecar itself never reporting.
* **Arm 2, `reason=sidecar_provider_missing`** — **≥ 1 `Setup\Apps` provider registered
but none named `Sidecar`**, continuously for ≥ 15 minutes. Deliberately ignores
provider completeness (the field case had `TrackingPoliciesCreated=1`). Sidecar
registering later — the legitimate startup ordering — clears the condition and resets
the clock. DevicePreparation-only snapshots don't activate it.
* When both arms cross in the same round, ONE event is emitted and
`sidecar_provider_missing` wins as `reason` (it explains the hang regardless of the
foreign provider's state).
* Missing root key / empty provider list = normal early enrollment: clears all dwell
state (both arms), never fires.
* One `esp_policy_provider_stalled` **Warning** per newly stalled condition set
(`ImmediateUpload`), one-shot per condition in-process, cross-restart deduped via a
`StartupEventGate` fingerprint over the sorted currently-active fired conditions
(provider keys + a `sidecarMissing` sentinel) — a restart into the same stall stays
silent, a new condition joining re-reports.
* Payload: `reason`, `providers[]` (full table: name, scope, kind, raw values, complete
flag), `stalledProviders[]` with `stalledForMinutes`, `sidecarMissingForMinutes`
(arm 2 only), `sidecarRegistered`, `dwellMinutes`.
* **Purely observational**: emitted via `InformationalEventPost` (dispatch-guard exempt
pass-through) — never touches decision state. No config toggle, no ConfigVersion bump
(precedent: `disk_space_low`).
Deliberately **not** built on `StallProbeCollector` (its idle clock resets on any session
activity — a provider stall coexists with a chatty session) and not on the periodic
collectors (idle-stopped after 15 minutes — exactly the dormant window the dwell must
observe).
## Guardrail companion
`rules/guardrails.json` allow-lists `SOFTWARE\Microsoft\Windows\Autopilot` (category
"Autopilot / OOBE / Setup") so tenants can build their own gather rules on the subtree —
e.g. a `not_exists` registry rule on the `Sidecar` provider key. The subtree carries
deployment state only (provider registrations, tracking lists, app IDs, per-user SIDs);
sensitivity is at or below existing entries. Enforcement is agent-side (embedded
resource), so both the detector and the guardrail change ship with the agent release.
## Examples
Customer-case event (abbreviated) — note the provider is "complete" by CSP values, the
hang is carried entirely by `reason=sidecar_provider_missing`:
```json
{
"eventType": "esp_policy_provider_stalled",
"severity": "Warning",
"data": {
"reason": "sidecar_provider_missing",
"dwellMinutes": 15,
"sidecarRegistered": false,
"sidecarMissingForMinutes": 15.0,
"providers": [
{ "name": "ConfigMgr", "scope": "device", "kind": "setupApps",
"trackingPoliciesCreated": 1, "installationState": null, "complete": true }
],
"stalledProviders": []
}
}
```
`reason=sidecar_provider_missing` is the co-management signature;
`reason=provider_incomplete` with `sidecarRegistered=true` covers a stalled Sidecar or
third-party provider — the generalized contract violation.
# Citations
* EnrollmentStatusTracking CSP — <https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp>
* Windows Autopilot into co-management (requirements, provider-conflict warning, troubleshooting registry paths) — <https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment>
* Troubleshoot the ESP (models app tracking exclusively via the `Sidecar` subkey) — <https://learn.microsoft.com/en-us/troubleshoot/mem/intune/device-enrollment/understand-troubleshoot-esp>
* Niehaus: the Intune ESP-policy WAP payload registering the provider nodes (`Sidecar` always, `ConfigMgr` only with a co-management settings policy) — <https://oofhours.com/2023/08/25/run-an-sccm-task-sequence-during-autopilot/>
* Niehaus: ESP tracks arbitrarily named providers when they are properly registered — <https://oofhours.com/2023/09/18/track-anything-using-esp/>
* Agent code: `EspPolicyProviderProbe.cs`, `EspPolicyProviderStallDetector.cs`, `EspPolicyProviderStallHost.cs` (SystemSignals / Orchestration.Hosts)
* Related: [MDM Reboot Coalescing](mdm-reboot-coalescing.md), [Gather-Rule Guardrails](../rules/gather-rule-guardrails.md)
+1
View File
@@ -7,5 +7,6 @@ How the on-device agent works:
* [Logs, Persistence & Crash Recovery](logs-and-persistence.md) - every persisted file and why it exists; signal log vs spool; snapshot/journal; the replay-based recovery flow and quarantine.
* [Hello Wizard Un-Skip](hello-wizard-unskip.md) - why a single "WHfB disabled" policy read never decides completion: the HelloWizardStarted signal (prevention + cure) and the HelloTracker confirmation second read.
* [Autopilot ZTD Diagnostics](autopilot-ztd-diagnostics.md) - Windows' own diagnostic surfaces for the profile-download flow: ModernDeployment event IDs (807/815/164/...), the Diagnostics\Autopilot registry key, deployment-service endpoints, and the known-issue error-code map.
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated 2800 observations (neutral, flush-time watermark) + ANALYZE-ESP-005 gated on an actually-observed reboot.
Reading order for newcomers: overview → decision engine → persistence.
+100
View File
@@ -0,0 +1,100 @@
---
type: Concept
title: MDM Reboot Coalescing — attributing the mid-ESP reboot and second sign-in to policy URIs
description: How the agent observes reboot-required MDM policy flags (DM-Enterprise EventID 2800, aggregated) and how ANALYZE-ESP-005 attributes an actually-observed enrollment reboot to them — including why the raw events stay neutral and why the Shell-Core RebootCoalescing match was removed.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals
tags:
- agent
- esp
- reboot
- policy-attribution
- second-sign-in
timestamp: 2026-07-20T00:00:00+02:00
---
# MDM Reboot Coalescing
Device-assigned MDM policies whose CSP URIs match the OS reboot-required catalog
(`HKLM\SOFTWARE\Microsoft\Provisioning\SyncML\RebootRequiredURIs`) force a **coalesced
reboot at the end of ESP DeviceSetup** — the user sees an unexpected restart and a
**second sign-in screen** before AccountSetup. Invisible in the Intune console; most
admins live with it, not knowing that reassigning the profiles to user groups removes the
reboot entirely. Source research: Patch My PC,
["Avoid the second sign-in screen after Autopilot"](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/).
The session already counts reboots (`system_reboot_detected` → `session.rebootCount`);
this feature adds the **cause attribution**.
# Verified signal facts (session b2e890c1, 2026-07-20)
* **EventID 2800**, `Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin`,
one record per URI, verified description shape:
`The following URI has triggered a reboot: (./Device/Vendor/MSFT/Policy/Config/ServiceControlManager/SvchostProcessMitigation).`
Records arrive in sub-second bursts during policy sync.
* **Timing semantics (the critical nuance)**: 2800 fires at EVERY policy sync. Only URIs
applied **during ESP DeviceSetup** are coalesced into the forced mid-ESP restart. In the
verified session the flags fired during AccountSetup (after DeviceSetup completed) —
**no reboot occurred**, and `omadmRebootRequiredFlag` read `false`. A detector that
claims "reboot incoming" on every 2800 is therefore wrong.
* **Shell-Core 62407 "RebootCoalescing" — REMOVED as a signal.** The token appears in the
ROUTINE bootstrap marker on every enrollment:
`Name: 'CommercialOOBE_BootstrapStatusCategory_SubcategoryProcessing_Started', Value: '{"message":"BootstrapStatus: Starting subcategory DeviceSetup.RebootCoalescing...","errorCode":0}'`
— the ESP merely *processes* the RebootCoalescing subcategory (which checks whether a
reboot is needed). A substring match on it is a false positive by construction. If a
real coalesced-reboot record is ever captured on the E2E VM, a precise pattern can be
added then.
# Detector — `MdmRebootPolicyTracker` (`MdmRebootPolicyWatcherHost`)
Live `EventLogWatcher` on the DM-Enterprise Admin channel (EventID 2800) + startup
backfill (default 60 min lookback). Records are **buffered and debounced (10 s)**, then
flushed as **ONE aggregated `mdm_policy_reboot_required` event** per burst:
* `Severity=Info`, neutral message ("Windows flagged N MDM policy URIs as
reboot-required: … (+N more)") — the raw event never claims a reboot.
* `Data`: `rebootUris` (distinct, sorted, capped 20), `uriCount` (uncapped distinct),
`firstRebootUri` (record-time order, drives rule interpolation), `recordCount`,
`unparsedCount`/`sampleDescription` (extraction diagnostics), `backfilled` (all-backfill),
fail-soft `omadmRebootRequiredFlag` probe of
`HKLM\SOFTWARE\Microsoft\Provisioning\OMADM\SyncML\RebootRequired` (null = unknown).
* `Timestamp` = earliest record time (backfilled bursts land at their historical instant);
`ImmediateUpload=true` (one event per burst is cheap, and a DeviceSetup burst precedes a
reboot that kills the agent).
* **Watermark is persisted at FLUSH, not at claim** (`mdm-reboot-watermark.json`): if the
coalesced reboot kills the process before the debounced emit, nothing is marked — the
post-restart backfill re-reads and emits the burst. A flushed burst is never repeated.
URI extraction: EventData values starting with `./` preferred, permissive regex over the
description as fallback (trailing punctuation trimmed — the verified text wraps the URI in
parens and ends with a period). Unparsed records still count into the aggregate.
Config (internal steering only, no admin UI — agent-native collector, NOT an
`IAgentAnalyzer`): `Collectors.MdmRebootPolicyWatcherEnabled` (default true),
`Collectors.MdmRebootPolicyBackfillLookbackMinutes` (default 60). ConfigVersion 35.
# Advisory — ANALYZE-ESP-005
The reboot **claim** lives here, and only here. Required conditions (BOTH):
1. `mdm_policy_reboot_required.firstRebootUri` exists (→ `{{firstRebootUri}}` interpolation)
2. `system_reboot_detected` count ≥ 1 — **an actually-observed reboot**
Policy flags alone (e.g. AccountSetup user-policy sync) stay advisory-silent. Optional
confidence: `uriCount ≥ 2` (+10), `omadmRebootRequiredFlag == true` (+15) on base 70 /
threshold 70. Remediation: map URIs to Intune profiles, reassign device → user groups (or
exclude the Autopilot device group), or accept + communicate the restart.
# Non-goals / invariants
* Event type **deliberately distinct from `system_reboot_detected`** — the backend
RebootCount matches that string exactly in two places
(`EventIngestProcessor.Classification` + terminal recount); attribution must never
inflate the reboot count.
* No session-level field / SessionInfoCard change in this iteration (follow-up candidate).
* The `RebootRequiredURIs` catalog itself is not read; only the 2800 events (which carry
the actually-triggered URIs) are consumed.
# Citations
* [Patch My PC — Avoid the second sign-in screen after Autopilot](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/)
* [Microsoft Learn — Enrollment Status Page tracking](https://learn.microsoft.com/en-us/autopilot/enrollment-status)
+100
View File
@@ -0,0 +1,100 @@
---
type: Concept
title: Secure Boot CA-2023 Detection — firmware evidence vs. Windows servicing status
description: Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly (GetFirmwareEnvironmentVariableExW + SeSystemEnvironmentPrivilege) instead of trusting the SecureBoot\Servicing registry key, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant rule suppression across old and new agents.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Interop/UefiSecureBootCertReader.cs
tags:
- agent
- secureboot
- ca2023
- analyze-rules
- backward-compatibility
timestamp: 2026-07-23T00:00:00+02:00
---
# Secure Boot CA-2023 Detection
## Schema
Two independent evidence sources describe the Windows UEFI CA 2023 rollout state:
1. **Servicing registry** (`HKLM\SYSTEM\CurrentControlSet\Control\SecureBoot\Servicing\UEFICA2023Status`)
— reflects whether the *Windows Update* rollout ran and self-verified. Lags reality:
devices whose firmware received the certificate another way (OEM factory image, WinCS
CLI, manual deployment) or where WU simply has not run yet show `unknown`/`notfound`.
2. **Firmware itself** — the UEFI variables `db` (signature database, GUID
`{d719b2cb-3d3a-4596-a3bc-dad00e67656f}`) and `KEK` (GUID
`{8be4df61-93ca-11d2-aa0d-00e098032b8c}`). Authoritative. Certificate subject CNs are
ASCII bytes inside the DER signature lists, so a byte-substring search for
`Windows UEFI CA 2023` / `Microsoft Corporation KEK 2K CA 2023` decides presence —
the same check `Get-SecureBootUEFI` enables from PowerShell.
`UefiSecureBootCertReader` (agent, `Monitoring/Interop`) reads both variables natively via
`GetFirmwareEnvironmentVariableExW` — no PowerShell runspace during OOBE. The call needs
`SeSystemEnvironmentPrivilege`: SYSTEM holds it default-disabled, so the reader enables it
on the process token (`AdjustTokenPrivileges`) and restores the previous state afterwards.
Fail-soft contract like `OobeStateReader`: never throws; failures map to status strings
(`not_uefi` = legacy BIOS via `ERROR_INVALID_FUNCTION`, `variable_not_found`,
`privilege_denied`, `error_<code>`).
This is the repo's only privilege-adjustment code, so it also carries the ownership rules
any future one should copy:
* The process token lives in a `SafeHandle` assigned by the P/Invoke marshaler, so the CLR
owns it from the moment `OpenProcessToken` returns — the critical finalizer releases it
even if `Dispose` never runs (async exception before the instance reaches its `using`).
* The restore obligation is recorded immediately after the successful `AdjustTokenPrivileges`,
and only when the privilege was not *already* enabled; nothing that can throw sits between
the adjustment and that assignment. `Dispose` is idempotent, swallows a failed restore, and
releases the handle in a `finally`.
* The privilege is adjusted on the **process** token, so enable → read → restore is a
process-wide side effect and runs under a static gate. Without it a concurrent reader would
see "already enabled", correctly skip its own restore, and then lose the privilege mid-read
when the first reader disposes — a spurious `privilege_denied` on a machine that holds it.
Fields land on the existing `secureboot_status` event (once per agent start,
restart-deduped by the StartupEventGate): `uefiFirmwareReadStatus`, per-cert booleans
(`uefiDbHasWindowsUefiCa2023`, `uefiDbHasWindowsProductionPca2011`,
`uefiKekHasMicrosoftKek2kCa2023`, …) — emitted only when the respective read succeeded —
plus the marker below.
## The one-sided marker (absence-tolerant suppression)
Rule-engine constraints (pinned by `RuleEnginePreconditionTests`):
* **Conditions** never match on an absent field (null guard) — new fields must stay
`required: false` or the rule dies for every old agent.
* **Preconditions**: `not_exists` passes while the field is absent/empty and fails as soon
as the field is present with *any* value; comparison operators fail closed on absence.
A two-sided boolean therefore cannot drive suppression: `not_exists` on
`uefiDbHasWindowsUefiCa2023` would also suppress the rule when the field is present with
`false`. Hence the **one-sided marker** `uefiCa2023FirmwareConfirmed = true`, emitted
ONLY when the db read succeeded AND the 2023 certificate was found — never as `false`.
ANALYZE-SEC-001 v3 adds a `not_exists` precondition on it:
| Device state | Marker | Rule |
|---|---|---|
| Old agent (no firmware fields) | absent | evaluates exactly like v2 |
| New agent, cert missing or firmware unreadable | absent | evaluates; optional conditions `uefiDbHasWindowsUefiCa2023=false` (+20) / `uefiKekHasMicrosoftKek2kCa2023=false` (+10) raise confidence |
| New agent, firmware confirms cert | present | silently suppressed (the customer-reported false positive) |
Because the marker exists nowhere until the new agent ships, the v3 rule is safe to seed
**before** the agent rollout.
## Examples
Customer-equivalent verification from PowerShell (authoritative, matches the reader):
```powershell
[Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023')
[Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name KEK).Bytes).Contains('Microsoft Corporation KEK 2K CA 2023')
```
## Citations
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Interop/UefiSecureBootCertReader.cs` — native reader
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/DeviceInfo/DeviceInfoCollector.NetworkAndSecurity.cs` — `CollectSecureBootStatus` / `AppendFirmwareCertFields`
* `rules/analyze/ANALYZE-SEC-001.json` — v3.0.0 rule definition
* `src/Backend/AutopilotMonitor.Functions.Tests/RuleEnginePreconditionTests.cs` — pinned precondition semantics + v3 mirror tests
* Microsoft: [Act now — Secure Boot certificates expire June 2026](https://techcommunity.microsoft.com/blog/windows-itpro-blog/act-now-secure-boot-certificates-expire-in-june-2026/4426856)
+95
View File
@@ -0,0 +1,95 @@
---
type: Concept
title: Business Timestamps vs. the Azure Tables System Timestamp
description: Why the system Timestamp is never authoritative, how the 2026-07-18 migration exposed that, and the OccurredUtc + RowKey-decode compensation (mappers, RowKey-range date filters, retention, backfill).
resource: /src/Backend/AutopilotMonitor.Functions/Helpers/BusinessTimestamp.cs
tags:
- backend
- table-storage
- audit-log
- ops-events
- events
- migration
timestamp: 2026-07-20T00:00:00+02:00
---
# Problem
Azure Table Storage's `Timestamp` is a **system property**: the server ignores any supplied
value on write and stamps the row's *write* time; every row rewrite (storage-account
migration, MERGE) resets it. Three tables nevertheless treated it as the business event
time — writes like `["Timestamp"] = evt.Timestamp` were silent no-ops, unnoticed because
write time ≈ event time in normal operation.
The 2026-07-18 EU storage migration rewrote every row, so all migrated rows in
**AuditLogs**, **OpsEvents**, and **Events** read as "2026-07-18 ~13:58 UTC": displays were
wrong, date-window filters matched the wrong rows, and retention froze (nothing older than
the migration moment) with a mass-delete cliff one retention period later.
# Rule
**The system Timestamp is never authoritative.** Every table that needs an event time
stores it in the custom column `OccurredUtc` (`BusinessTimestamp.OccurredUtcColumn`).
Custom columns are copied verbatim by `scripts/Migration/Copy-TableStorage.ps1`, so future
migrations preserve chronology. Migration checklist: any table whose reads/filters depend
on a timestamp must carry it as a custom column — audit this before every storage move.
# Recovery sources per table
The original time survives migrations inside the RowKey:
| Table | RowKey scheme | Decodable |
|---|---|---|
| AuditLogs | `!{MaxTicks-ticks:D19}_{guid:N}` (since 2026-05-06) | Yes; legacy bare-GUID rows (pre-2026-05) are lost |
| OpsEvents | `{MaxTicks-ticks:D19}` | Yes, all rows |
| Events | `{ts:yyyyMMddHHmmssfff}_{seq:D10}` (sanitized agent time) | Yes, all rows (ms precision) |
# Mechanism
`BusinessTimestamp` (Functions/Helpers) is the single source of truth: RowKey decoders,
the OData clause builders, and `GetUtcDateTime` (datetime columns materialize as
`DateTimeOffset` *or* `DateTime` depending on payload shape — always read via this helper).
1. **Read mappers** resolve `OccurredUtc` → RowKey decode → system Timestamp (last
resort; only audit legacy GUID rows end there). Resolvers:
`TableStorageService.ResolveAuditTimestamp`, `TableOpsEventRepository.ResolveTimestamp`,
`TableStorageService.ResolveEventTimestamp` (also used by
`GetEarliestSessionEventTimestampAsync`, which feeds StartedAt/Duration reconciliation —
a system-Timestamp read there would rewrite migrated sessions' derived fields).
2. **Date-window filters and retention use RowKey ranges**, not property comparisons:
OData excludes rows missing a compared property (an `OccurredUtc` filter would drop
every not-yet-backfilled row), and RowKey ranges are index-backed. Boundary semantics
are tick-exact equivalents of the old `Timestamp ge/le` clauses.
**Trap:** audit legacy GUID RowKeys (hex first char) sort *after* all `!`-rows, so
every lower bound `RowKey ge '!…'` must be paired with `RowKey lt '"'`
(`AuditTimeEncodedUpperBound`) — without the guard, a retention sweep would delete all
legacy rows. Pinned by `BusinessTimestampFilterClauseTests`.
3. **Writes** in all three tables now set `OccurredUtc`; the no-op `["Timestamp"]`
entries were removed.
4. **Backfill** (`POST /api/maintenance/backfill-occurred-utc`, GlobalAdminOnly, dry-run
by default, `table=audit|ops`, batched via `maxRows`/`continuation`): MERGE that adds
only `OccurredUtc` decoded from the RowKey — deterministic and idempotent. **Events
rows are deliberately never backfilled** (read-path decode is sufficient; no bulk
mutation of enrollment evidence). Note: a MERGE resets the row's system Timestamp —
irrelevant once nothing depends on it.
5. **Audit retention** runs two passes until ~2027-01: RowKey-range for `!`-rows plus a
`RowKey ge '0' and Timestamp lt …` pass that ages legacy GUID rows out via their frozen
migration-date system Timestamp (remove after 2027-03-01).
# Consequences
* Audit legacy GUID rows have "unknown date" semantics: excluded from date-filtered
views, shown with the (wrong but honest) 2026-07-18 fallback in unfiltered views.
* Raw endpoints (`/api/raw/*`, `query_table`) intentionally keep showing the raw system
Timestamp — they are storage-state inspectors, not business views.
* Event timestamps for pre-cutover rows resolve to the RowKey's sanitized agent time
(ms precision); `ReceivedAt` (server ingest time, rows since 2026-03-19) and
`OriginalTimestamp` (pre-clamp agent time, clamped rows only) remain separate fields.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/BusinessTimestamp.cs` — decoders + clause builders
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Maintenance.cs` — audit write/mapper/filters/retention
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableOpsEventRepository.cs` — ops write/mapper/filters/retention
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Sessions.cs` — event writes, `ResolveEventTimestamp`, earliest-event read
* `src/Backend/AutopilotMonitor.Functions/Services/OccurredUtcBackfillService.cs` + `Functions/Admin/BackfillOccurredUtcFunction.cs` — backfill
+122
View File
@@ -0,0 +1,122 @@
---
type: Concept
title: Device Journeys & First-Time-Right (F2)
description: How a device's terminal enrollment attempts are grouped into journeys (device key, junk-serial exclusion, first-success end, 30-day gap), where the DeviceHistories chain and the daily FTR aggregates are computed, stored and pruned, and which surfaces serve them.
resource: src/Backend/AutopilotMonitor.Functions/Helpers/DeviceJourneyCalculator.cs
tags:
- backend
- metrics
- device-history
- first-time-right
- insights
timestamp: 2026-07-27T12:00:00+02:00
---
# Device Journeys & First-Time-Right (F2)
Answers "how many devices enrolled right on the first try?" and "show me every attempt
this device made". Session success rate hides wipe-and-retry loops — every retry that
eventually succeeds counts as a success; First-Time-Right (FTR) exposes the real pain.
All grouping semantics live server-side in `DeviceJourneyCalculator` (JourneyVersion 1);
no consumer re-derives them.
# Schema
## Device key
`(TenantId, normalized serial)` — normalization is trim + lower-case. Junk/placeholder
serials are excluded entirely (never a chain, never in FTR, disclosed as a per-day
exclusion count): `System Serial Number`, `To Be Filled By O.E.M.`, `Default string`,
`0`, `None`, `INVALID`, `Unknown` (the agent's WMI-failure sentinel), empty, or shorter
than 4 characters. A motherboard swap changes the serial and therefore the device key;
re-enrollment under a different tenant is a separate key by design.
## Journey
Terminal sessions of a device key ordered by StartedAt. The terminal set is
**Succeeded / Failed / Incomplete** — `Pending` (a sealed WhiteGlove part 1),
`AwaitingUser`, `Stalled` and `InProgress` are OPEN sessions and never attempts.
* A journey **ends** with the first terminal success; the next session starts a new
journey (redeployment).
* A gap of **more than 30 days** since the previous attempt's terminal moment
(`CompletedAt`, fallback StartedAt) starts a new journey even without a success
(device shelved/repurposed). Constant, not a setting — production gaps: median 4.9 h,
92 % < 7 d.
* WhiteGlove part 1 + part 2 share ONE session row, so a completed WG enrollment is ONE
attempt; a WG device still waiting for its user session leaves the journey open —
never a failed attempt.
* **Attempt count** = terminal sessions in the journey (`Failed` and `Incomplete` are
non-successful attempts).
* **FTR rate** = completed journeys with attempt count 1 ÷ all completed journeys.
Open and gap-abandoned journeys never enter numerator or denominator.
Durations shown for attempts are the sessions' authoritative `DurationSeconds` verbatim
(WhiteGlove pause excluded; `Incomplete` deliberately has none) — never
`CompletedAt − StartedAt`, which is later in ~25 % of terminal sessions.
## Persistence & compute
* `DeviceHistories` (PK = TenantId, RK = percent-encoded normalized serial): the chain of
the device's terminal session refs `{sessionId, startedAt, completedAt, status,
enrollmentType, isPreProvisioned, durationSeconds, adminMarked}` capped at the 20 most
recent, plus derived `CurrentJourneyAttempts` (the LAST journey's attempts — the
banner's "Attempt N"), `JourneyCount` (chain-scoped) and `JourneyVersion`. Written
inline at the session-terminal single-writer seam
(`TableSessionRepository.UpdateSessionStatusAsync`, all three terminal statuses,
DeletionState-guarded) so the session banner is fresh; healed by the maintenance sweep.
Deliberately NOT part of the per-session deletion manifest — the row aggregates many
sessions. Wiped on tenant offboarding.
* Sweep (`SweepDeviceJourneysAsync`, 30-day rolling window, 2h tick): (1) drops chain
refs of deleted sessions **tombstone-driven** — every session deletion (cascade and
retention both run through the cascade worker) leaves a `SessionTombstones` marker for
~7 days, far longer than the sweep cadence, and even an expired-but-unpruned marker is
valid deletion evidence; a chain left empty deletes its row. (2) Merges the window's
terminal sessions into their chains (backfill, inline-miss healing, re-terminal
reclassifications). (3) Recomputes the daily FTR aggregates idempotently.
* `DeviceJourneyAggregates` (PK = TenantId or `global`, RK = `{yyyy-MM-dd}`): per day —
completed journeys, first-time-right count, attempt histogram, junk-serial exclusion
count, `JourneyVersion`. A journey buckets on the StartedAt date of its completing
success session (the platform's StartedAt-date convention; late terminals converge via
the sweep). All counts are **additive**, so a window rate is the ratio of summed daily
rows — there is no rolling-window row (unlike the median-based time-attribution
aggregates, which cannot merge). 180-day retention. Rows exist below the ≥20 UI gate
on purpose (the UI needs the n for "insufficient data (n=…)").
# Surfaces
* `GET api/metrics/device-history?serialNumber=&sessionId=` (MemberRead + tenant query
param — the same route serves the web banner and MCP; a Global Admin passes `tenantId`)
— the chain plus, with `sessionId`, that session's server-computed attempt number
(live sessions get their would-be position via a virtual non-successful attempt, so the
redeploy and gap rules apply). `history: null` = no recorded history.
* `GET api/metrics/device-journeys?days=` (MemberRead) /
`GET api/global/metrics/device-journeys?days=&tenantId=` (GlobalReadOrAdmin) —
`{ windowDays, totals, daily, repeatDevices }`. The days selector is honored because
counts are additive. Without `tenantId` the global variant serves the cross-tenant
aggregate rows and `repeatDevices: null` — a per-device list would require scanning
every tenant's partition.
* Repeat devices (violator list, capped 10): devices whose LAST journey took ≥2 attempts
and whose newest terminal session falls inside the window, ordered by attempts then
recency; the failure reason comes from the newest failed attempt (bounded point-reads).
* Web: "Attempt N for this device · View history" banner on session detail (expandable
chain); "First-time-right" section on Fleet Health (rate + weekly trend + attempt
histogram + repeat devices). MCP: `get_device_history` (serial mode / fleet FTR mode).
# Examples
A device fails ESP twice and succeeds on the third try within one week: one journey,
three attempts, completed — counts once in the FTR denominator, not in the numerator,
and the histogram's bucket `3` gains one. The same device redeployed a month later and
succeeding immediately starts journey 2 with attempt count 1 — first-time-right. A
device with `serialNumber = "Unknown"` (WMI failure) never gets a chain; its terminal
sessions surface only as that day's disclosed exclusion count.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/DeviceJourneyCalculator.cs` — device key, junk list, grouping, attempt numbers (JourneyVersion 1).
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.DeviceHistory.cs` — persistence + inline seam entry.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.DeviceJourneys.cs` — sweep, tombstone pruning, daily aggregation.
* `src/Backend/AutopilotMonitor.Functions/Functions/Metrics/GetDeviceJourneyFunctions.cs` — read surfaces.
* `src/Backend/AutopilotMonitor.Functions.Tests/DeviceJourneyAndFtrTests.cs` — table-driven grouping/aggregation/attempt-number tests.
* `tasks/insights-expansion-spec.md` — F2 specification and source-data audit (§0.5).
+112
View File
@@ -0,0 +1,112 @@
---
type: Concept
title: Rule Regression Radar (F3)
description: How the radar detects analyze rules whose hit rate suddenly rose (7-day window vs 28-day baseline, Wilson-separated), which suppressions keep it quiet, how alert episodes live in the notification tracker, and which surfaces show them.
resource: src/Backend/AutopilotMonitor.Functions/Helpers/RuleRegressionRadar.cs
tags:
- backend
- rules
- regression-radar
- insights
timestamp: 2026-07-27T16:00:00+02:00
---
# Rule Regression Radar (F3)
Detects "this analyze rule suddenly fires much more often" before helpdesk volume does, and
says what the spike correlates with — with honest statistics and honest wording. Analyze
rules only in v1: their daily stats are audit-verified session-deduplicated
(`FireCount` ≈ distinct sessions with a hit, `SessionsEvaluated` the same-source
denominator). Gather rules are excluded until their per-batch stats dedup is fixed —
never silently included with bad math.
# Schema
## Signal & detection
Per (tenant, ruleId), daily hit rate = `FireCount ÷ SessionsEvaluated` from the
`RuleStats` rows — RATES, not counts (absolute counts spike with every rollout wave).
A regression fires only when ALL hold for the trailing 7-day window vs. the prior
28-day baseline:
1. window has ≥ 5 hit sessions AND ≥ 20 evaluated sessions;
2. window rate ≥ 2× baseline rate;
3. the Wilson 95 % intervals are disjoint in the increase direction
(`MetricsMath.WilsonInterval` / `RateIncreaseSeparated` — deterministic, unit-pinned),
so small-n noise never alerts.
## Suppression (false positives are trust damage)
* Rule entity missing (deleted) — never alerts.
* Rule younger than baseline + window (`CreatedAt` within 35 days — grace period for new
rules; the entity timestamp is the honest gate, a stats-row count would fail on
low-volume tenants whose quiet days simply have no rows).
* Rule edited inside the window (`UpdatedAt` ≥ window start) — edits legitimately change
hit rates.
* Empty baseline denominator (rule disabled throughout the baseline).
* Platform kill switch: app setting `RuleRegressionRadarDisabled=true` skips the radar
(fail-open — it only notifies, never mutates data).
## Alert episodes (tracker keyspace)
An episode lives as `ruleregression|{ruleId}` in the notification-tracker table
(the `tpmpss|` pattern): the row IS the dedup (bell + ops event fire exactly once per
episode), the badge state and the `regressions[]` payload — window/baseline counts,
rates, lift (null for a zero-baseline "new signal" — no finite lift is invented), window
dates, and the dimension concentration captured at first fire. Numbers refresh on every
radar pass while the episode stays active; `FirstNotifiedAt` never moves. The episode
closes (badge clears, dedup re-arms) when the window rate falls back under 1.5× baseline
or fires stop entirely; the tracker's 30-day retention sweep also re-arms, so a
month-old still-burning regression rings once more by design.
## Dimension correlation (on fire only)
Hit sessions (RuleResults scan: tenant partition range + RowKey = ruleId + DetectedAt in
window, intersected with the window's sessions) are compared against all window sessions
across `osBuild`, `model` (manufacturer + model), `agentVersion` and `imeVersion`. The
top value by lift is reported with BOTH shares — wording contract: "X % of affected
sessions are on … vs Y % of all sessions — correlated, not necessarily causal". No value
with ≥ 5 hit sessions at ≥ 2× lift → "no clear dimension concentration"; a correlation
failure also yields no claim. App-version correlation for `ANALYZE-APP-*` rules is a
follow-up (needs rule→app matching semantics), tracked with the gather-rule fix.
## Compute & alert path
Runs in the daily maintenance pass right after the rule-stats aggregation
(`RunRuleRegressionRadarAsync`), anchored on YESTERDAY — whole days only, a partial
today would understate the window rate. One 35-day stats read covers all tenants;
evaluation is the pure `RuleRegressionRadar` core. On a NEW episode:
1. `RuleFrequencyRegression` ops event (category Tenant, Warning) with the full numbers
— dual-registered in the web `OPS_EVENT_TYPES` catalog for alert-rule routing.
2. Tenant bell notification (type `rule_frequency_regression`, Admin audience) with the
same numbers and a deep link to the rule's card (`/analyze-rules#rule-card-{ruleId}`).
A sessions-list pre-filtered to rule hits needs a rule filter the session search does
not have yet — follow-up alongside the gather-rule work.
3. v1 channel: bell + ops event only. Provider fan-out (webhook/Telegram) follows once
alert quality is proven — a noisy first version burns trust permanently.
# Surfaces
* `GET api/metrics/rule-stats` (and the global variant with `tenantId`) carries
`regressions[]` — the active episodes. MCP `get_rule_stats` inherits the block.
* Analyze-rules page: per-rule 30-day fire-count sparkline (densified from the `trend`
rows the endpoint already served) and a red "↑ Regression" badge while an episode is
active, with the full numbers in the tooltip.
# Examples
A rule firing on 2 % of sessions for a month jumps to 15 % over a week (15/100 vs
10/500): lift 7.5×, Wilson-separated → one bell, one ops event, badge until the rate
falls back under 3 %. The same jump on 5/20 sessions with a 12 % baseline stays silent —
the intervals overlap, so the spike is not statistically real yet. A rule that never
fired in the baseline and now hits 15 % alerts as a "new signal" with no lift figure.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/RuleRegressionRadar.cs` — gates, suppression, re-arm, dimension concentration.
* `src/Backend/AutopilotMonitor.Functions/Helpers/MetricsMath.cs` — Wilson interval + separation primitive.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.RuleRegression.cs` — sweep, episode reconcile, bell/ops wiring.
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableHardwareRejectionNotificationTracker.cs` — episode keyspace.
* `src/Backend/AutopilotMonitor.Functions.Tests/RuleRegressionRadarTests.cs` — hand-computed Wilson vectors + every suppression branch.
* `tasks/insights-expansion-spec.md` — F3 specification and source-data audit (§0.5).
@@ -0,0 +1,102 @@
---
type: Concept
title: ServerActions & On-Demand Diagnostics Collection
description: The queue-on-session, deliver-on-ingest ServerAction channel (terminate_session / rotate_config / request_diagnostics), the operator-tier authorization split, the portal's Collect Logs flow including the admin quick-config path, and why the offboarding cascade sweeps the hosted diagnostics prefix.
resource: /src/Backend/AutopilotMonitor.Functions/Functions/Sessions/QueueSessionActionFunction.cs
tags:
- backend
- agent
- web
- server-actions
- diagnostics
- authorization
timestamp: 2026-07-24T00:00:00+02:00
---
# Schema
**Channel.** A `ServerAction` (`Type`, `Reason`, `RuleId`, `Params`, `QueuedAt`) is queued onto
`Sessions.PendingActionsJson` by server-side writers — the admin endpoint
`POST /api/sessions/{sessionId}/actions`, the RuleEngine, maintenance — and delivered
piggy-backed on the agent's next ingest response. No polling channel exists; delivery latency
equals the agent's telemetry cadence (seconds to ~1 minute during enrollment). Delivery is
at-least-once: the agent's `ServerActionDispatcher` dedups on `Type|RuleId|QueuedAt` and
processes a batch sequentially in queue order. Unknown types are logged and skipped, so new
types roll out server-first. Every action emits `server_action_received` and then
`server_action_executed` or `server_action_failed` into the session timeline.
**Authorization (two layers).** The route sits on `EndpointPolicy.TenantAdminOrOperator`
(tenant Admin, tenant Operator, or Global Admin — no Viewer, no Global Reader: it is a write
tier). Because the route serves multiple action kinds, `QueueSessionActionFunction`
re-gates per type via `IsTypeAllowedForCaller`: Operators may queue only
`request_diagnostics`; `terminate_session` and `rotate_config` require
`RequestContext.IsTenantAdmin || IsGlobalAdmin` (403 otherwise). Any route on this tier with
a multi-kind body MUST follow this same in-function re-gating pattern.
**On-demand collection (`request_diagnostics`).** The dispatcher calls
`DiagnosticsPackageService.CreateAndUploadAsync(enrollmentSucceeded: false, suffix:
"server-requested")` — producing `AgentDiagnostics-{sessionId}-{timestamp}-server-requested.zip`.
Passing `enrollmentSucceeded: false` is what makes on-demand collection work under
`DiagnosticsUploadMode=OnFailure`; only `Off` (or an unconfigured destination) blocks it,
surfacing as `server_action_failed`. The `diagnostics_uploaded` event stamps
`DiagnosticsBlobName` + `Destination` onto the Sessions row (latest wins — an on-demand name
is later replaced by the terminal package), and the SignalR `sessionUpdate` delta carries the
field, so the portal's Download button flips live.
**Portal flow (Collect Logs, session detail header).** The button is always rendered; the
disabled states carry tooltips (inactive session, cross-tenant view, missing role, diagnostics
not configured) as a deliberate nudge. Members below Admin cannot read the tenant config, so
the member-readable feature-flags endpoint exposes `diagnosticsUploadConfigured`
(`mode != Off && (customer SAS present || destination Hosted)`). Progress tracking is
sequence-based (backend-assigned, monotonic), never timestamp-based — agent clock skew and
at-least-once redelivery of older actions cannot produce false matches
(`collectLogsLogic.ts`). Cross-tenant views are gated off because `TenantHelper.GetTenantId`
resolves the tenant strictly from the caller's JWT.
**Admin quick-config.** For an unconfigured tenant, a Tenant Admin's click opens a dialog
that switches the tenant to `Destination=Hosted`, `Mode=OnFailure` (GET full config → patch
exactly those two fields → PUT verbatim), then queues `rotate_config` **before**
`request_diagnostics` — the dispatcher's sequential ordering guarantees the agent refetches
the now-enabled config (the `RemoteConfigMerger` applies diagnostics fields live, remote wins)
before building the package. This works against the already-deployed agent fleet; a
"one-time override" variant would not, because the `mode=Off` check is client-side in
`CreateAndUploadAsync` and the deployed handler reads no params.
**Offboarding cascade.** Latest-wins on `DiagnosticsBlobName` means a session can leave more
hosted blobs behind than the deletion manifest references. The cascade's Hosted branch
therefore sweeps the canonical prefix `{tenantId}/AgentDiagnostics-{sessionId}-`
(`HostedDiagnosticsBlobService.DeleteBySessionPrefixAsync`) plus the manifest-named blob as
belt-and-braces; a non-GUID legacy `SessionId` degrades to the single-name delete instead of
poisoning the cascade. The CustomerSas branch is unchanged (delete only with the `d`
permission; the customer's lifecycle rules stay authoritative).
# Examples
Queue an on-demand collection (Admin or Operator, own tenant):
```
POST /api/sessions/{sessionId}/actions
{ "type": "request_diagnostics", "reason": "On-demand log collection from portal" }
→ 202 { "success": true, "queuedAt": "..." }
```
Operator queueing an admin-only type:
```
POST /api/sessions/{sessionId}/actions
{ "type": "terminate_session" }
→ 403 { "message": "Action type 'terminate_session' requires the Tenant Admin role" }
```
Timeline for a successful round trip: `server_action_received` → `server_action_executed`
(with `blobName`) → `diagnostics_uploaded` → Sessions row + SignalR delta carry the new
`DiagnosticsBlobName`.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Functions/Sessions/QueueSessionActionFunction.cs` — allowed types, per-type caller gate.
* `src/Backend/AutopilotMonitor.Functions/Security/EndpointAccessPolicyCatalog.cs` — `TenantAdminOrOperator` tier definition.
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Runtime/ServerActionDispatcher.cs` — dedup, sequential dispatch, telemetry contract.
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Runtime/DiagnosticsPackageService.cs` — mode gates, archive build, suffix.
* `src/Backend/AutopilotMonitor.Functions/Services/Diagnostics/DiagnosticsBlobCascadeDeleter.cs` — hosted prefix sweep.
* `src/Web/autopilot-monitor-web/app/sessions/[sessionId]/components/collectLogsLogic.ts` — button gating matrix, sequence-based progress evaluation.
+124
View File
@@ -0,0 +1,124 @@
---
type: Concept
title: Enrollment Time Attribution (F1)
description: How a terminal session's wall clock is partitioned into named segments with an explicit unattributed remainder, how ESP-blocking apps and reboots are attributed, and where breakdowns and fleet aggregates are computed, stored and served.
resource: src/Backend/AutopilotMonitor.Functions/Helpers/TimeAttributionCalculator.cs
tags:
- backend
- metrics
- time-attribution
- insights
timestamp: 2026-07-26T20:00:00+02:00
---
# Enrollment Time Attribution (F1)
Answers "where did the 47 minutes go?" per session and per fleet. Every number is
traceable to observed events; a missing signal produces "unknown" or an explicit
unattributed remainder — never a guess, and the partition is never normalized to 100 %.
# Schema
## Invariant
For every computed breakdown:
```
sum(span seconds) + UnattributedSeconds == WallClockSeconds == session DurationSeconds
```
`DurationSeconds` is the authoritative wall clock — for WhiteGlove it is part 1 + part 2
WITHOUT the pause, and re-terminal stamps move `CompletedAt`, so `CompletedAt − StartedAt`
is never used (it diverges in ~25 % of terminal sessions). Span seconds are floored;
sub-second dust lands in the unattributed remainder.
## Observation windows
Windows are end-anchored on authoritative timestamps and constructed so their lengths sum
exactly to `DurationSeconds`:
* Non-WhiteGlove: one window `[CompletedAt − DurationSeconds, CompletedAt]`.
* WhiteGlove (ONE session row, two windows): part 2 = `[ResumedAt, CompletedAt]`; part 1 =
`[part1End − (DurationSeconds − part2), part1End]` where `part1End` is anchored on the
`whiteglove_part1_complete` event. The pause between the windows is never attributed.
Fallback anchors set the `WhiteGloveAnchorsIncomplete` quality flag.
## Segments
Spans are sliced at phase-declaration events (only declaration events carry
`Phase != Unknown`) plus `desktop_arrived`, in canonical Sequence order; anchors whose
timestamp runs backward are dropped (`ClockSkewDropped`).
| EnrollmentPhase | Segment key |
|---|---|
| Start / DevicePreparation / DeviceSetup | `device_prep` |
| AppsDevice | `esp_apps` |
| AccountSetup / FinalizingSetup | `identity_hello` |
| AppsUser | `user_esp` |
| Complete, `desktop_arrived` event | `desktop_handoff` |
| Failed | attribution ends — tail stays unattributed |
## ESP-blocking apps (positive evidence only)
Per-app intervals use EVENT timestamps (first started/download event → LAST terminal
event, covering IME retries) — never the agent payload timing, which freezes at the first
terminal transition. Blocking membership joins the app id against the session's latest
list-carrying `esp_config_detected` emission: listed ⇒ blocking; absent ⇒ unknown, never
false (`BlockingSetUnknown` / `BlockingSetTruncated` flags). Critical-path occupancy =
overlap-merged union of blocking intervals clipped to the `esp_apps` spans;
`EspAppsOccupancySeconds` is null (unknown) when no lists were observed.
The fleet what-if bound per app X is `max(0, cpEnd − cpEndWithoutX)` (latest interval end
with vs. without X; the idle gap before X counts — the ESP also waits for X to start).
It is an upper bound by construction; every consumer says "up to", never "you will save".
## Reboots
`system_reboot_detected` is detection-time-stamped by the NEXT agent run; the real boot
moment is its `lastBootUtc` payload. Reboot spans are therefore the event-stream gap
bracketing `lastBootUtc`, deduped per boot, clipped to the observation windows, and carried
as a cross-cutting annotation (they overlap the segment they started in — never a slice of
the partition).
## Persistence & compute
* `SessionTimeBreakdowns` (PK = TenantId, RK = SessionId): spans, reboot spans, blocking
intervals (top 20 + uncapped count), quality flags, `AttributionVersion`. Written once at
the session-terminal single-writer seam (`TableSessionRepository.UpdateSessionStatusAsync`,
Succeeded/Failed only — Incomplete deliberately has no duration). Deleted with its session
(deletion-manifest step) and on tenant offboarding.
* Self-healing 30-day maintenance sweep (2h tick): backfills missing breakdowns, folds
late-terminating sessions into their StartedAt-date aggregates, and enforces version
purity — a stale-version row whose events aged out counts as missing, never mixed.
* `TimeAttributionAggregates` (PK = TenantId or `global`, RK = `{yyyy-MM-dd}|{class}` and
`rolling30|{class}`): per enrollment class (`user_driven`, `whiteglove`, `self_deploying`,
`device_preparation` — never mixed) median/p75/p90 per segment over the fixed six-segment
stack (incl. `unattributed`; a session without a segment contributes 0), plus top blocking
apps (≥5 sessions per app row). Clean/flagged/missing counts are part of every row; rows
exist below the ≥20 UI gate on purpose (the UI needs the n for "insufficient data (n=…)").
Daily rows retain 180 days; `rolling30` rows are refreshed whole (a median of per-day
medians is not the range median — the rolling rows are the honest range statistics).
## Surfaces
* `GET api/sessions/{sessionId}/time-attribution` (MemberRead + tenant query param) —
breakdown row or `breakdown: null` (pre-feature / non-terminal / Incomplete = no lane).
* `GET api/metrics/time-attribution` (MemberRead) / `GET api/global/metrics/time-attribution`
(GlobalReadOrAdmin, optional `tenantId`) — `{ windowDays, classes (rolling), daily }`.
* Web: attribution lane under the session-detail phase timeline; "Time attribution" section
on Fleet Health. MCP: `get_time_attribution` (sessionId mode / fleet mode).
# Examples
A 30-minute user-driven session: `device_prep` 5m → `esp_apps` 10m (occupancy 4m by one
blocking app) → `identity_hello` 8m → `user_esp` 5m → `desktop_handoff` 2m, unattributed 0.
A WhiteGlove session with a 2-day pause reports wall clock = part1 + part2 only; the resume
gap before the first part-2 phase declaration is unattributed, not guessed.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/TimeAttributionCalculator.cs` — calculator + semantics (AttributionVersion 1).
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.TimeAttribution.cs` — persistence.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.Aggregation.cs` — sweep + aggregation.
* `src/Backend/AutopilotMonitor.Functions.Tests/TimeAttributionCalculatorTests.cs` — golden fixtures pinning the invariant.
* `tasks/insights-expansion-spec.md` — F1 specification and source-data audit (§0.5).
+23
View File
@@ -12,11 +12,34 @@ https://docs.autopilotmonitor.com (separate repository).
# Architecture
* [Architecture Guide](architecture.md) - High-level architecture overview and solution layout for contributors.
* [Version Contract](versioning.md) - One version shape for agent, backend and MCP: the ETag-CAS counter blob that mints build numbers, and the manifests published only after a deploy is verified live.
* [URL Registry](url-registry.md) - Every well-known host lives in one registry file per component (Constants.cs / utils/config.ts / src/config.ts), enforced by guard tests; includes the agent-download migration to the download alias and its deploy-order constraint.
# Agent
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait is keyed to the Intune-registered "Sidecar" provider by NAME: co-management's "ConfigMgr" parks the user ESP at "Apps (Identifying)" for days even with TrackingPoliciesCreated=1 (issue #106); the always-on two-arm esp_policy_provider_stalled tripwire (provider_incomplete | sidecar_provider_missing) and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
# Rules
* [Gather Rule Phase Scoping, One-Shot Triggers & Emit-on-Change](rules/gather-rule-phase-scoping.md) - Restricting gather rules to enrollment phases (activePhases / activeFromPhase sticky latch), one-shot collection at a phase's start/end (phase_change / phase_exit), and emitMode "on_change" result-dedup.
* [Gather Rule Guardrails](rules/gather-rule-guardrails.md) - Why the allowlists in rules/guardrails.json are enforced on the agent and nowhere else, which hard blocks survive unrestricted mode, and the rule that every collector must call a guard and emit security_warning on refusal.
# Backend
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
* [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract.
* [ServerActions & On-Demand Diagnostics](backend/server-actions-on-demand-diagnostics.md) - The queue-on-session, deliver-on-ingest action channel, the TenantAdminOrOperator tier with in-function per-type re-gating, the portal's Collect Logs flow (quick-config orders rotate_config before request_diagnostics), and the hosted-prefix sweep in the offboarding cascade.
# MCP Server
* [MCP Docs Corpus](mcp/docs-corpus.md) - How the published customer documentation becomes the `search_docs` corpus: CRLF-safe heading chunking, build-time embedding as a third precomputed section, and why a docs edit needs an MCP redeploy.
# Security & Identity
+62
View File
@@ -1,5 +1,67 @@
# Log
## 2026-07-27
* **Creation**: Added `backend/rule-regression-radar.md` — F3 regression radar (analyze rules only; gather rules wait for their per-batch stats dedup fix): per (tenant, rule) daily hit rate from `RuleStats`, firing only when the trailing 7-day window has ≥5 hit sessions and ≥20 evaluated sessions, the rate is ≥2× the prior 28-day baseline AND the Wilson 95 % intervals are disjoint (deterministic `MetricsMath` primitive with hand-computed test vectors). Suppressed for deleted rules, rules younger than baseline+window (`CreatedAt` grace), rules edited inside the window (`UpdatedAt`), empty baselines, and the `RuleRegressionRadarDisabled` app-setting kill switch. Alert episodes live as `ruleregression|{ruleId}` tracker rows (one bell + one `RuleFrequencyRegression` ops event per episode; numbers refreshed while active; re-armed under 1.5× baseline, on fire-stop, or by the 30-day tracker retention). On-fire dimension correlation over osBuild/model/agentVersion/imeVersion with the "correlated, not necessarily causal" wording contract. Surfaces: `regressions[]` in the rule-stats response (MCP inherits), the analyze-rules page's 30-day sparkline and regression badge.
* **Creation**: Added `backend/device-journeys.md` — F2 device history / First-Time-Right: the device key (trim + case-fold serial, junk/placeholder exclusion incl. the agent's `Unknown` WMI sentinel), journey grouping over the terminal set Succeeded/Failed/Incomplete (first success ends a journey, a >30-day gap from the previous attempt's CompletedAt starts one, WhiteGlove's single session row is ONE attempt, open sessions are never attempts), the `DeviceHistories` chain (cap 20, written inline at the terminal seam, healed by the sweep, deleted-session refs pruned tombstone-driven, deliberately not in the per-session deletion manifest) and the additive `DeviceJourneyAggregates` daily rows (window rate = ratio of sums — no rolling row needed, 180-day retention). Surfaces: the session-detail "Attempt N" banner with server-computed attempt numbers (virtual-attempt rule for live sessions), the Fleet Health First-time-right section (≥20-journey gate, attempt histogram, repeat-devices violator list), and the MCP `get_device_history` tool (serial mode / fleet FTR mode).
## 2026-07-25
* **Update**: `agent/secureboot-ca2023-detection.md` — review follow-up on 90012d9 hardened the privilege scope, the repo's only privilege-adjustment code and therefore the precedent for any future one. The process token moved from a raw `IntPtr` to a `SafeHandle` assigned by the P/Invoke marshaler, so the CLR owns it from the moment `OpenProcessToken` returns and the critical finalizer releases it even when `Dispose` never runs — previously an async exception between the successful open and the caller's `using` leaked the handle *with the privilege enabled and no owner left to restore it*. `Dispose` became idempotent and now releases the handle in a `finally`, so a throwing restore cannot strand it; the restore obligation is recorded immediately after the successful `AdjustTokenPrivileges` with nothing throwable in between. The whole enable → read → restore sequence runs under a static gate because the privilege is adjusted on the *process* token: a concurrent reader would see "already enabled", correctly skip its own restore, and then lose the privilege mid-read when the first reader disposed, reporting `privilege_denied` on a machine that holds it.
## 2026-07-24
* **Creation**: Added `backend/server-actions-on-demand-diagnostics.md` — the ServerAction channel (queue on `Sessions.PendingActionsJson`, deliver piggy-backed on the next ingest response, at-least-once with `Type|RuleId|QueuedAt` dedup and sequential dispatch) and the on-demand log collection built on it: the portal's always-visible **Collect Logs** button, the new `EndpointPolicy.TenantAdminOrOperator` tier with the mandatory in-function per-type re-gate (Operators queue only `request_diagnostics`), the member-readable `diagnosticsUploadConfigured` feature flag, the admin quick-config path that must queue `rotate_config` before `request_diagnostics` (works against the deployed fleet — a one-time override would not, the mode gate is agent-side), sequence-based progress matching immune to agent clock skew, and the offboarding cascade's hosted-prefix sweep closing the latest-wins orphan-blob gap.
* **Update**: `agent/esp-policy-provider-stall.md` — the customer's follow-up on issue #106 corrected the field-case mechanics: `TrackingPoliciesCreated=1` WAS set under the foreign `ConfigMgr` provider, so the ESP ignores a foreign provider's completion values and waits on the Intune-registered `Sidecar` provider by name (the rename to `Sidecar` creates no tracking entries yet unblocks instantly). The completeness-keyed detector would have missed exactly this — the incident it was built for. The tripwire is now two-armed under one event type discriminated by `reason`: `provider_incomplete` (unchanged dwell on continuously-incomplete providers) and `sidecar_provider_missing` (≥ 1 Setup/Apps provider registered without `Sidecar`, completeness deliberately ignored, clock reset when Sidecar registers later). Sidecar-missing wins the `reason` when both arms cross in one round; the cross-restart fingerprint gains a `sidecarMissing` sentinel; payload adds `reason` and `sidecarMissingForMinutes`. Evidence for the name-keyed wait: Niehaus' documented Intune WAP payload always registers `Sidecar` (ConfigMgr only via a co-management settings policy) and Microsoft's ESP troubleshooting models app tracking exclusively through the Sidecar subkey.
* **Creation**: Added `agent/esp-policy-provider-stall.md` — a customer diagnosed that pre-provisioned devices with the ConfigMgr client as a blocking app occasionally park the user ESP at "Apps (Identifying)" for days: only a `ConfigMgr` provider is registered under `EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders` (no `Sidecar` = IME) and `TrackingPoliciesCreated` never becomes 1 — a wait the CSP defines without any timeout; renaming the key to `Sidecar` unblocks immediately. The agent now ships the always-on `esp_policy_provider_stalled` tripwire (60-s tick, 15-min continuous-incompleteness dwell, one-shot per provider, `StartupEventGate` fingerprint across restarts, purely observational via the InformationalEvent rail — no toggle, no ConfigVersion bump), generalized to any provider in both `Setup/Apps` and `DevicePreparation` kinds rather than hardcoding the ConfigMgr case. Companion change: `SOFTWARE\Microsoft\Windows\Autopilot` joins the gather-rule registry allowlist so tenants can target the subtree with custom rules (e.g. `not_exists` on the Sidecar key).
* **Creation**: Added `agent/secureboot-ca2023-detection.md` — ANALYZE-SEC-001 warned on devices whose firmware already carried the 2023 certificate, because the agent only read the `SecureBoot\Servicing\UEFICA2023Status` registry value, which mirrors the Windows Update rollout rather than the firmware (customer report). The agent now reads the UEFI `db`/`KEK` variables natively (`GetFirmwareEnvironmentVariableExW` with `SeSystemEnvironmentPrivilege` enabled on the process token — first privilege-adjustment code in the repo) and emits per-certificate booleans on `secureboot_status`. Suppression rides on a one-sided marker, `uefiCa2023FirmwareConfirmed`, emitted only on positive confirmation: the engine's `not_exists` precondition fails on any present value, so a two-sided boolean could not distinguish "confirmed" from "checked and missing", while the marker's absence keeps old agents and firmware-missing devices on the v2 path. Rule v3.0.0 is therefore safe to seed before the agent release.
* **Update**: `url-registry.md` — the first release after the alias migration surfaced Front Door caching as a correctness hazard: the alias served the previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) after the origin had rotated, and since manifest and ZIP are independent cache entries, a stale manifest with a fresh ZIP fails the bootstrap SHA-256 check. Route caching on `/agent/*` is now disabled and every blob upload stamps `Cache-Control: no-cache` (CI `Send-Blob`, local `build.ps1`, `versions/*.json` manifests) — all agent-container blobs rotate in place, so nothing there is safely cacheable.
* **Update**: `agent/decision-engine.md` — the RealmJoin completion gate has a third terminal, `FirstDeploymentIncomplete`. Session 224b2087 showed why: an interactive logon 17 s after DeploymentPhase 101 makes RealmJoin reclassify the run as a secondary-user deployment, whose branch writes 200/210 and never 110 — the gate stayed closed until the 60-min hard timeout although Hello and desktop were long resolved. `realmjoin_phase_changed` is now a typed `RealmJoinPhaseChanged` signal, so `RealmJoinFacts.LastDeploymentPhase` is persisted across agent restarts (this also fixes every `realmjoin_timeout` event claiming "last phase: 0"); a transition out of 100/101 into 200/210 releases the gate with the `realmjoin_first_deployment_incomplete` Warning, and the `RealmJoinDetected` replay path evaluates the same rule for restarts that swallow the transition. A first observation of 210 deliberately does not release — session 6f1959c0 started at 210 and only deployed afterwards.
* **Creation**: Added `url-registry.md` — all well-known hosts (own subdomains, both blob accounts, API origin, Graph, Entra login) are now defined once per component (`Constants.cs`, web `utils/config.ts`, MCP `src/config.ts`) and referenced everywhere else; three guard tests fail the build on any literal outside the registries. Written because the EU cutover missed two hardcoded copies of the legacy blob host — `LatestVersionsService` (feeds the portal's version badge) and `ValidateBootstrapCodeFunction` — which kept working only through the fail-soft legacy mirror in `build-agent.yml`; those two plus the agent `SelfUpdater` now use the `download.autopilotmonitor.com` alias (verified serving all four artifacts first). The portal's bootstrap allow-list accepts alias AND legacy host during the transition, which imposes a deploy order: web before (or with) backend. The sweep also surfaced a fourth copy of the www/portal host pair in `middleware.ts` (now derived from the registry via `hostRouting.ts`) and a backend short-link built on the bare apex domain, which only worked through a registrar-level 301.
## 2026-07-22
* **Creation**: Added `versioning.md` — agent, backend and MCP now share one version shape (`<major>.<minor>.<build>` + 7-char commit + build timestamp) and one implementation of the build-number reservation (`.github/scripts/Request-BuildNumber.ps1`, previously copy-pasted between `build-agent.yml` and `build.ps1`). The backend had been pinned at a hand-written `1.0.0` since inception and now starts at `1.5.1000`; the MCP server's `package.json` version was bumped by hand and now starts at `1.5.300`, and it reports its own commit on `/health` instead of only the documentation bundle's. Both deploy workflows poll their health endpoint until it reports the build just deployed and publish `versions/<component>.json` only then, so a manifest that disagrees with the live endpoint means the instance drifted rather than the pipeline failing halfway. `agent/build-counter-blob.md` keeps the agent-specific parts and now points at the shared protocol; its counter URL was still the pre-cutover storage account.
* **Creation**: Added `rules/gather-rule-guardrails.md` — the gather-rule allowlists are enforced only on the agent (the backend validates no `collectorType`, `target`, or `parameters`), so a collector that forgets to call its guard *is* the vulnerability. Written because two had: `LogParserCollector` read any path in `rule.Target`, bypassing even the `C:\Users` hard block that every other file collector honours, and `EventLogCollector` read any channel including `Security` — while the customer-facing documentation already promised both were allowlisted. Both now guard, with `logparser` checking the directory for wildcard targets (`Path.GetFullPath` throws on `*`) and re-checking each resolved path. Added an event-log channel allowlist (`eventLogChannels`) plus hard-blocked channels held in C# rather than JSON, so a parse error cannot lift them. Also fixed the command timeout, which was inert: `ReadToEnd()` blocks until the pipe closes, so `WaitForExit(30000)` was only reached after the process had already exited and a hung command pinned its worker forever — now read asynchronously and killed at the timeout.
* **Update**: Added `.github/workflows/ci.yml` — the full suite (backend, agent V2 Core, DecisionCore, SummaryDialog, web, MCP) now runs on every pull request and every push to main, plus a job that fails if the generated guardrail files drift from `rules/guardrails.json`. Previously **no** workflow ran tests on a PR: the deploy workflows build without testing, and `build-agent.yml` tested only `DecisionCore.Tests` on manual dispatch — while `trust/security-faq.md` already told customers that automated tests run in CI. Added CodeQL (C# and TypeScript) and Dependabot (nuget, npm ×2, actions). Marked `OfficeBinaryWatcherTests` `[Collection("SerialThreading")]`: its bounded re-probe races the shared ThreadPool and failed once under full-suite load.
## 2026-07-21
* **Update**: `search_docs` and `search_knowledge` now state in their tool descriptions that queries must be in English. Both corpora and `all-MiniLM-L6-v2` are English-only, and the failure is total rather than gradual — a German question returns zero results, not weak ones. The caller is a language model, so translating the question before the call and answering in the user's language costs nothing; a multilingual embedding model would be the alternative but is much larger and would have to be measured against the eval harness first, since it changes English retrieval too.
* **Update**: Added `scripts/eval-docs-search.ts`, an offline retrieval evaluation harness (2465 queries in three classes: chunk headings as a regression detector, rare identifiers, and hand-written questions as the reality anchor), and used it to close a third `search_docs` gap found during the live test. The keyword fallback only fired when semantic search returned *fewer* than `topK`; a pasted identifier like `WOW6432Node` instead returned three chunks at ~0.29 that did not contain the word while the one chunk that did was never considered. It now also fires on a weak semantic head (<0.35), letting exact matches take the front while keeping one semantic result. The threshold is measured, not guessed: 0.35 fixes 335 queries and breaks 0 (identifier recall 66.2% → 81.1%), while 0.40 fixes more but starts demoting hand-written questions. The measurement also resized the problem — identifier queries were only succeeding 66.2% of the time, so the reported case was one of roughly 750.
* **Update**: Closed the two measured recall gaps in `search_docs` (`mcp/docs-corpus.md`). (1) **Token windows** — 87 of 324 chunks exceeded the model's 256-token context and 13% of all tokens never reached the ranking; a character budget cannot fix this (the corpus ranges 2.43–5.81 chars/token), so long text is now embedded as 250-token windows with 40 overlap and scored by its BEST window (`embeddings: number[][]`, max not mean). Token coverage 87% → 100%; "proxy configuration blocking app downloads" went from absent to 0.559 at rank 1; the ten reference queries kept byte-identical scores. Costs +31% vectors, index 2.63 → 3.12 MB. The rules corpus gained the same treatment (128 rules → 172 vectors). (2) **Keyword fallback** — when the semantic pass returns fewer than `topK`, results are topped up via literal `lexicalMatch` on distinctive query terms, requiring two matching terms for corroboration (one incidental substring match returned a page about LLM providers for a sourdough query). Fuzzy matching was measured and rejected: at a threshold low enough to surface "RealmJoin" it invents 5 hits for "SCEPman", which the corpus never mentions. Also sharpened the tool description with the measured context argument for preferring it over fetching the docs site.
* **Creation**: Added `mcp/docs-corpus.md` (new `mcp/` section) — the MCP server gains `search_docs`, a third semantic corpus built from the published customer documentation repository (`autopilotmonitor-docs`, checked out into the Docker build context by the deploy workflow). Documents the h2/h3 + `<details>` chunking with breadcrumb prefixes, the two deliberate size exceptions (tables never split, unbroken changelog lists split at line level), the CRLF precondition (`.` does not match `\r` in JS — headings silently stop matching, which produced 286 wrong-but-plausible chunks before it was found), the optional `docs` section in `search-index.json` and its validation asymmetry, embedding rounding to 6 decimals, and the cross-repo coupling: a docs edit reaches clients only after an MCP redeploy, made checkable via `DOCS_COMMIT` on `/health`. Container resources raised to 0.5 vCPU / 1.0 GiB for the cold-start query-embedder load.
## 2026-07-20
* **Update**: `rules/gather-rule-phase-scoping.md` — the gather-rule form now shows the phase-scope and emit-mode controls only where they can change behavior (`supportsPhaseScope` / `supportsEmitMode`), nulls hidden-control state out of the payload, and rejects a scope mode selected but left unfilled. Offering all three "when" controls unconditionally allowed combinations that could only suppress a one-shot rule, plus a silent no-op state.
* **Update**: Extended `rules/gather-rule-phase-scoping.md` with the `phase_exit` trigger — the closing bookend to `phase_change`, for one-shot collection when a phase is LEFT (e.g. "state at the end of Account Setup"). Documents the evaluation-before-phase-advance rule (triggerPhase match and scope gate see the phase being left), firing into `Failed`, the disjoint `exit:` dedup key space, and why no ConfigVersion bump is needed (no new field — old agents simply never fire the rule).
* **Update**: Reworked `agent/mdm-reboot-coalescing.md` after the first real enrollment (session b2e890c1): 2800 records are now **aggregated per burst** into ONE neutral Info event (`rebootUris[]`/`uriCount`/`firstRebootUri`; watermark persisted at flush so a pre-emit reboot kill loses nothing); the raw event no longer claims a reboot — ANALYZE-ESP-005 is gated on an actually-observed `system_reboot_detected`. **Removed** the Shell-Core `esp_reboot_coalescing` event entirely: the "RebootCoalescing" token appears in the routine `SubcategoryProcessing_Started` bootstrap marker on every enrollment (false positive by construction). Verified 2800 description shape recorded.
* **Creation**: Added `agent/mdm-reboot-coalescing.md` — the agent now attributes the "unexpected mid-ESP reboot + second sign-in" pattern (PMPC research) to the device-assigned policy URIs that forced it: new `MdmRebootPolicyTracker` (DM-Enterprise EventID 2800, per-URI `mdm_policy_reboot_required` events, cross-restart RecordId watermark), a `RebootCoalescing` branch in `ShellCoreTracker` (`esp_reboot_coalescing`, backfill-emitting by design), and the `ANALYZE-ESP-005` advisory (reassign device → user groups). ConfigVersion 35; event types deliberately distinct from `system_reboot_detected` so RebootCount stays uninflated.
* **Creation**: Added `backend/business-timestamps.md` (new `backend/` section) — the 2026-07-18 migration reset every row's system Timestamp, exposing that AuditLogs/OpsEvents/Events had (unknowingly) displayed and filtered on it all along; documents the `OccurredUtc` column, the RowKey decoders and RowKey-range date/retention clauses in `BusinessTimestamp`, the GUID-guard trap, the audit/ops backfill endpoint, and the "system Timestamp is never authoritative" migration rule.
* **Creation**: Added `rules/gather-rule-phase-scoping.md` (new `rules/` section) — gather rules gain `activePhases` (run only during listed phases), `activeFromPhase` (sticky latch from a phase onwards, never via Failed) and `emitMode: "on_change"` (canonical-hash result dedup with `suppressedPolls` observability), ConfigVersion 34. Includes the prerequisite backend fix: the custom-rule toggle PUT no longer wipes the stored rule definition (partial-merge in `GatherRuleService.UpdateRuleAsync`).
## 2026-07-18
* **Creation**: Added `agent/endpoint-migration.md` — config-channel agent re-home (`MigrateToApiBaseUrl` on `GET agent/config`, ConfigVersion 33): backend or per-tenant API-URL moves without DNS indirection (impossible due to mTLS/Flex-TLS). Live-fetch-only with cache strip, `AgentEndpointMigrationRules` allowlist validated on both sides, one hop, kill-switch wins. Built as the lesson from the 2026-07-18 WEU→GWC cutover, which hard-cut all in-flight agents.
## 2026-07-26
* **Creation**: Added `backend/time-attribution.md` — the F1 enrollment time attribution: exact-partition invariant against the session's authoritative DurationSeconds, two-window WhiteGlove model, segment mapping from phase-declaration events, positive-evidence ESP-blocking join with "up to" what-if bounds, reboot spans from event gaps around `lastBootUtc`, and the SessionTimeBreakdowns / TimeAttributionAggregates tables with their terminal-seam + maintenance-sweep compute paths and read surfaces (web lane, fleet panel, MCP `get_time_attribution`).
## 2026-07-17
* **Creation**: Added `agent/build-counter-blob.md` — agent build numbers now come from a shared public counter blob (`buildcounter-v{N}.txt`, reserve-before-build with ETag-CAS) so local `build.ps1` and CI `build-agent.yml` builds can never mint the same number. Documents the CI parity fixes (SummaryDialog.exe.config in the ZIP, EXE-vs-manifest version check, hash-oracle update gated on stable cutover) and the Sigstore provenance attestation on CI-built ZIPs.
+305
View File
@@ -0,0 +1,305 @@
---
type: Concept
title: MCP Docs Corpus — search_docs and the Cross-Repo Build Coupling
description: How the published customer documentation becomes a third semantic search corpus in the MCP server — CRLF-safe heading chunking, build-time embedding, the corpus hash guard, and why editing the docs does not update a running server.
resource: /src/McpServer/autopilot-monitor-mcp/src/docs-corpus.ts
tags:
- mcp
- search
- embeddings
- documentation
- build
timestamp: 2026-07-21T19:30:00+02:00
---
# Purpose
The MCP server answers two different kinds of question, and they need two
different corpora.
* *"Why did this enrollment fail?"* → analysis rules, gather rules, IME log
patterns. That is `search_knowledge`, backed by `knowledge-base.ts`.
* *"How do I deploy the agent?" / "Where is my data stored?"* → the published
customer documentation. That is `search_docs`, backed by `docs-corpus.ts`.
They are deliberately **separate indexes**. Merged into one, they would compete
for the same `topK` and dilute each other: a rules query would surface prose
pages, and a product question would surface rule definitions.
# Schema
## Corpus source
The corpus is the `autopilotmonitor-docs` repository — a public OKF bundle
published at `docs.autopilotmonitor.com`. It is **not** vendored into this repo.
The deploy workflow checks it out into `mcp-docs/` of the Docker build context;
the Dockerfile copies it to `/app/docs` in both the build and the final stage
(byte-identical, or the boot-time corpus hash rejects the precomputed index).
`SUMMARY.md` and `log.md` are skipped as navigation scaffolding. `index.md` is
kept — it is the documentation map and answers "what documentation exists" — but
gets no `url`, because the bundle deliberately keeps it out of `SUMMARY.md` and
GitBook therefore never renders it.
## Chunking
A whole page cannot be one document: `trust/security-faq.md` is 36 KB, while
MiniLM-L6-v2 truncates at ~256 tokens. An un-chunked page would silently embed
only its first paragraph.
Chunks are cut at **h2/h3 headings**, which in this bundle are also the semantic
boundaries — `security-faq.md` carries 37 h3s, one per question.
`<details><summary>Q</summary>` blocks (used by `troubleshooting/faq.md`) are
treated the same way, with the summary as the section title. Each chunk is
prefixed with a `Page Title › Section › Subsection` breadcrumb so a hit lifted
out of the middle of a page is self-describing to both the embedder and the
reader.
Two deliberate exceptions to the ~1500-character budget:
* **Markdown tables are never split.** A table broken across chunks leaves
`| --- |` rows with no header — unreadable in a result and meaningless to
embed. Tables carry real answers here (the role matrix, the settings
reference), so an over-long table is allowed to exceed the budget.
* **Unbroken lists split at line level.** The changelog pages are one bullet
list per release with no blank lines, so paragraph splitting cannot touch
them; left whole, everything past the model's window would be truncated away
and become unsearchable.
Short sections are **kept**, not filtered. A one-line answer ("Is it free?" →
"Yes, the Community plan is free") is often the most valuable hit, and the
breadcrumb supplies the context the short body lacks. Only structural residue
(under 25 alphanumeric characters) is dropped.
## CRLF is a correctness precondition
The docs repo is authored on Windows and ships CRLF. In JavaScript `.` does not
match `\r`, so a pattern like `/^(#{2,3})\s+(.*)$/` matches **nothing** on a CRLF
file: `(.*)` stops before the `\r` and `$` never reaches end-of-string.
This was observed during development, and the failure mode is what makes it
dangerous: no crash, no error — headings and frontmatter simply stop being
recognized and every page silently degrades to unlabelled paragraph splits. The
first working build produced 286 plausible-looking chunks that were almost all
wrong; after normalizing, `security-faq.md` went from 29 blind splits to 46 real
question sections.
`normalizeNewlines()` runs before any parsing. Do not remove it, and do not
"fix" a future parsing bug here by adding `\r?` to one pattern.
## Token windows: why a chunk can hold several vectors
Chunking at heading boundaries keeps sections coherent, but it does not bound the
result by *tokens* — and it cannot. Measured on this corpus the ratio ranges from
**2.43 to 5.81 characters per token**, a factor of 2.4, so no character budget maps
reliably onto the model's 256-token window. A budget safe for the worst case
(~620 characters) would shred prose and tables alike.
Left alone, 87 of 324 chunks exceeded the window and **13% of all tokens never
reached the ranking** — silently, since the tokenizer truncates without complaint.
The effect was concrete: in `built-in-rules.md :: Apps` (734 tokens), the strings
`ANALYZE-APP-013`, `ANALYZE-OFFICE-001`, `ANALYZE-CORR-003` and `Click-to-Run` all
sit past the cutoff, so *"proxy configuration blocking app downloads"* did not
retrieve that section at all.
So the *chunk* stays whole and the *embedding* is split instead:
`VectorSearchProvider.index()` cuts long text into 250-token windows with 40 tokens
of overlap and embeds each. A document therefore carries `embeddings: number[][]`,
and `search()` scores it by its **best** window.
- **Max, not mean.** Averaging window vectors would blur a long section's several
topics into one mediocre direction — the same dilution that makes large context
windows worse than chunking. Max keeps a document findable by any one thing it
discusses.
- **The tokenizer is a build-time dependency only.** `index()` runs during the
Docker build; production boots from precomputed vectors and never loads it. It is
also only the vocabulary, not the ONNX weights (~93 ms).
- Cost: 324 chunks → 424 vectors (+31%), index 2.63 → 3.12 MB, and 424 instead of
324 dot products per query — still microseconds.
- The rules corpus benefits too, unplanned: 128 rules now carry 172 vectors.
Measured effect: token coverage 87% → **100%**, *"Office Click-to-Run install never
finished"* 0.318 → **0.442**, *"proxy configuration blocking app downloads"* absent
→ **0.559 at rank 1**. The ten reference queries kept byte-identical scores — their
chunks are short, hence one window, hence the same vector as before.
## Keyword fallback
Semantic ranking alone leaves a gap: proper nouns embed poorly. `RealmJoin` occurs
in 3 chunks yet scores under the threshold.
Fuzzy matching is the wrong remedy, and the corpus says so. At a threshold low
enough to surface `RealmJoin` (0.28), Fuse also returns 5 hits for `SCEPman` — a
word this corpus contains **zero** times. Literal containment has neither problem.
So when the semantic pass returns fewer than `topK`, `search_docs` tops the list up
via `lexicalMatch`, the substring scan every provider already exposes:
- Query terms are filtered to distinctive ones (≥4 characters, non-stopword, max 6).
- A hit needs **corroboration**: at least two matching terms, unless the query has
only one. Without this rule, "how do I bake sourdough bread" matched a page about
LLM providers, because "bake" is a substring of "baked into the image".
- Hits are **appended, never merged**: they carry no cosine similarity, so semantic
results keep the head of the list and keyword hits fill the tail, flagged
`matchType: "keyword"` with `keywordFallback` in the response.
### Language
The corpus and `all-MiniLM-L6-v2` are both English-only, and the failure is total
rather than gradual: "wo werden meine Daten gespeichert" returns **zero** results,
not weak ones, because the query vector lands nowhere near any English chunk.
The fix is a sentence in the tool description telling the caller to query in English
and answer in the user's language — the caller is a language model, so translating
the question before the call costs nothing. A multilingual embedding model would
handle it natively but is substantially larger and would need to be measured against
`scripts/eval-docs-search.ts` first, since it changes English retrieval too.
`search_knowledge` carries the same note for the same reason.
### When it fires
Two triggers, and the second one exists because the first was not enough:
1. **Fewer than `topK` semantic hits** — the obvious case.
2. **A weak semantic head** (best score below `WEAK_SEMANTIC_SCORE`, 0.35). Topping
up only a short list misses the more common failure: a pasted identifier like
`WOW6432Node` returns three chunks at ~0.29 that do not contain the word, while
the single chunk that does never gets considered. When nothing semantic is
convincing, exact matches take the front — keeping at least one semantic result,
so a weak-but-correct hit is never pushed out entirely.
The threshold was chosen by measurement, not intuition. `scripts/eval-docs-search.ts`
scores strategies over 2465 queries in three classes — every chunk heading
(known-item, a regression detector), rare identifiers (the class this fallback
serves), and hand-written questions (the reality anchor, not derived from the
corpus):
| strategy | known-item | literal | handwritten MRR | fixed | broken |
| --- | --- | --- | --- | --- | --- |
| append-only (previous) | 71.9% | 66.2% | 1.000 | — | — |
| weak < 0.30 | 71.9% | 72.6% | 1.000 | 142 | 0 |
| **weak < 0.35** | **73.7%** | **81.1%** | **1.000** | **335** | **0** |
| weak < 0.40 | 76.3% | 88.2% | 0.944 | 502 | 3 |
| weak < 0.45 | 77.2% | 93.0% | 0.889 | 613 | 5 |
0.35 is strictly dominant: better in every class, worse in none. Higher thresholds
keep repairing more identifier queries but start demoting hand-written questions —
the class that most resembles real use — so the gain is not free past that point.
The measurement also corrected the size of the problem: identifier queries were
succeeding only 66.2% of the time, so `WOW6432Node` was not an outlier but one of
roughly 750 such failures.
## Build-time embedding
Like the rules corpus, docs are embedded once at Docker build time by
`precompute-embeddings.js <rulesDir> <outFile> [docsDir]`, into a third
`PrecomputedSection` (`docs`) of `search-index.json`. The container never embeds
a corpus at boot — only the incoming query.
The `docs` section is **optional** in the file format, which is what makes a
local checkout without the docs repo work: `validatePrecomputedIndex` skips it
when the booting server loaded zero doc chunks. The asymmetry is intentional —
if the server *has* a docs corpus and the file does *not*, that is staleness and
is rejected like any other mismatch.
Embeddings are rounded to 6 decimals on serialize. A float64 component
serializes to ~20 characters of JSON, so 384 dims cost ~7.8 KB per vector;
rounding roughly halves the index file, which is `JSON.parse`d on every
scale-to-zero cold start. The induced L2 norm error (~1e-5 over 384 dims) sits
two orders of magnitude inside `UNIT_NORM_EPSILON`.
## Registration
`search_docs` is registered for **every role**, unlike the platform tools: this
is public documentation carrying no tenant data, and a tenant user asking "how do
I deploy the agent" has as much business here as a Global Admin. It is skipped
entirely when the corpus is empty, so a doc-less build advertises no broken tool.
# Cold start
`minReplicas: 0` means every idle period ends in a cold start someone waits through.
Measured from outside: **24.5 s** to first byte after `ScaledToZero`, against **60 ms**
warm.
Where those seconds go could not be answered from outside the process. Log Analytics
stamps every console line of one ingestion batch with the same time, so the boot
sequence collapses to a single instant however it is queried. The server therefore
times itself — `[boot +Nms]` marks from `process.uptime()`, which includes Node's own
startup and module loading.
Measured in production on a real cold start (31.5 s to first byte that time):
```
+244ms node started (startup + module loading)
+431ms corpora read from disk (+187ms)
+491ms index read, validated, hydrated (+60ms <- for 3.12 MB)
+504ms listening
+677ms query embedder warm (background)
```
**The application accounts for half a second of it.** The other ~31 s — 98% — is
Container Apps activation: KEDA reacting to the request, scheduling a pod, mounting
the 236 MB image, starting the container. The container also boots *faster* than a
development laptop (504 ms against 1273 ms), so nothing here is CPU-starved.
Three conclusions, and two of them overturn assumptions this file previously stated:
- **Index size is irrelevant to startup.** 3.12 MB hydrates in 60 ms. Growing the
corpus is effectively free, and the embedding rounding earns its keep on file size
alone, not on boot time.
- **Raising the container to 0.5 vCPU was not justified by cold start.** The stated
reason was the embedding model load; prebaked, that is 173 ms and runs in the
background, where only the first *search* could ever wait on it. The extra vCPU may
still help concurrent requests, but the cold-start argument does not survive
measurement.
- **The only real lever is `minReplicas: 1`**, which removes activation entirely at
roughly 7× the monthly free consumption grant. Shrinking the image would shave the
mount, but that is a fraction of platform time, not the bulk of it.
# Cross-repo coupling
**Editing the documentation does not update a running MCP server.** The corpus is
baked into the image, so only re-running the *Deploy MCP Server* workflow picks
up doc changes.
That coupling is made checkable rather than memorable: the build passes the docs
checkout SHA as `--build-arg DOCS_COMMIT`, and `/health` reports
`docs: { commit, chunks, sections }`. Comparing that commit against
`git rev-parse HEAD` in `autopilotmonitor-docs` answers "is the deployed
documentation stale?" directly.
Automatic redeploy on a docs push was considered and rejected: it would need a
PAT secret in the docs repo and would turn every typo fix into a container
deployment.
# Examples
Retrieval measured against the real bundle (323 chunks, all-MiniLM-L6-v2):
```
"where is my data stored" → 0.566 troubleshooting/faq.md :: General › Where is my data stored?
"how do I deploy the agent with Intune" → 0.819 troubleshooting/faq.md :: Setup & Agent › How do I deploy the agent?
"which roles exist and what can they do" → 0.500 trust/security-faq.md :: Identity and Access › What roles exist?
"how do I set up Teams notifications" → 0.592 reference/settings.md :: Tenant › Notifications
"what firewall URLs do I need to allow" → 0.374 reference/network-endpoints.md :: (intro)
```
The 0.25 `minScore` default is tuned to that observed band — short keyword
queries on all-MiniLM score low, and a relevant-but-marginal hit lands around
0.34.
# Citations
* `/src/McpServer/autopilot-monitor-mcp/src/docs-corpus.ts` — loader, frontmatter
parser, chunker
* `/src/McpServer/autopilot-monitor-mcp/src/tools/search.ts` — `search_docs`
registration
* `/src/McpServer/autopilot-monitor-mcp/src/precomputed-index.ts` — `docs`
section and its validation asymmetry
* `/src/McpServer/autopilot-monitor-mcp/Dockerfile` — `mcp-docs/` copy,
`DOCS_COMMIT`
* `/.github/workflows/deploy-mcp.yml` — docs checkout and SHA capture
* `/src/McpServer/autopilot-monitor-mcp/scripts/eval-docs-search.ts` — retrieval
evaluation harness; run it before changing anything about ranking
* [MCP OAuth Flow](../mcp-oauth-flow.md) — the other MCP concept document
+81
View File
@@ -0,0 +1,81 @@
---
type: Concept
title: Gather Rule Guardrails
description: >-
How gather-rule collectors are fenced in — the allowlists in rules/guardrails.json,
the hard blocks that survive unrestricted mode, and why every collector must call a
guard rather than trusting rule.Target.
resource: src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Gather
tags: [gather-rules, security, guardrails, agent]
timestamp: 2026-07-22
---
# Concept
A gather rule is authored by a **tenant administrator** through `POST /api/rules/gather` and executed by the agent as **SYSTEM** on an enrolling device. The backend does not validate `collectorType`, `target`, or `parameters` at all — it only enforces scope/emit-mode fields, a 1 MB body cap, and that the rule lands in the caller's own tenant partition. Everything that decides *what a rule may reach* is therefore enforced **on the agent**, in `GatherRuleGuards`.
This is the security boundary. The portal's `ValidationIndicator` is a convenience: it does not block submit, and the API is reachable directly.
# Schema
Allowlists come from `rules/guardrails.json`, embedded into the agent assembly as
`AutopilotMonitor.Agent.V2.Core.Resources.guardrails.json` (see the `EmbeddedResource`
item in `AutopilotMonitor.Agent.V2.Core.csproj`) and regenerated into the portal's
`utils/guardrails.generated.ts` by `node rules/scripts/combine.js`.
| Guard | Source list | Match rule |
| --- | --- | --- |
| `IsRegistryPathAllowed` | `registryPrefixes` | prefix, boundary `\` or end |
| `IsFilePathAllowed` | `filePrefixes` | `Path.GetFullPath` first, then prefix + boundary |
| `IsWmiQueryAllowed` | `wmiQueryPrefixes` | prefix, boundary whitespace or end |
| `IsCommandAllowed` | `allowedCommands` | **exact** match, trimmed — never prefix |
| `IsEventLogChannelAllowed` | `eventLogChannels` | prefix, boundary `/` or end |
**Hard blocks live in code, not in the JSON**, so a parse error cannot lift them — on a
parse failure the allowlists fall back to empty and everything is blocked (fail-closed),
but the hard blocks must hold regardless of configuration:
* `BlockedUsersPrefix` (`C:\Users`) and `AdditionalHardBlockedPathPrefixes`
(`C:\Windows\System32\config`) — exception only for `%LOGGED_ON_USER_PROFILE%`
under `AppData\Local` / `AppData\Roaming`.
* `HardBlockedCommandPatterns` — download, user/group creation, boot config,
persistence, destructive operations; plus `MaxCommandLength`.
* `HardBlockedEventLogChannels` — `Security` (audit trail of user behaviour) and the
PowerShell channels (script-block logging routinely contains secrets in clear text).
`rules/guardrails.json` mirrors the blocked lists (`blockedFilePrefixes`,
`blockedEventLogChannels`) **for display only**. Enforcement is the C# constant.
# Examples
Every collector must call its guard and return `context.EmitSecurityWarning(...)` on
refusal — that emits the `security_warning` event that makes a block visible in the
timeline instead of failing silently.
```csharp
// FileCollector / JsonCollector / XmlCollector
if (!GatherRuleGuards.IsFilePathAllowed(filePath, context.UnrestrictedMode, userProfilePath))
return context.EmitSecurityWarning(rule, "file", filePath);
```
Two collectors were added to this contract on 2026-07-22 after shipping without it:
* **`LogParserCollector`** read any path in `rule.Target`, bypassing even the `C:\Users`
hard block. It cannot guard `rule.Target` directly — a wildcard filename makes
`Path.GetFullPath` throw — so it guards **the directory** when the filename contains
`*`/`?`, and re-guards **each resolved path** after wildcard expansion (a junction can
surface a file the directory check did not cover).
* **`EventLogCollector`** read any channel, including `Security`.
`ImeLogPathOverride` comes only from the local `--ime-log-path` CLI flag, never from
remote config. The operator using it is already a local admin, so it relaxes the
allowlist exactly the way unrestricted mode does — hard blocks still apply.
# Citations
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Gather/GatherRuleGuards.cs`
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Gather/GatherRuleContext.cs` — `EmitSecurityWarning`
* `src/Agent/AutopilotMonitor.Agent.V2.Core.Tests/Monitoring/Gather/GatherRuleCollectorGuardTests.cs`
* `src/Backend/AutopilotMonitor.Functions/Functions/Rules/GatherRulesFunction.cs` — what the backend does *not* validate
* `rules/guardrails.json`, `rules/scripts/combine.js`
* Customer-facing counterpart: `autopilotmonitor-docs/rules/gather-rules.md` (§ Security guardrails)
+136
View File
@@ -0,0 +1,136 @@
---
type: Concept
title: Gather Rule Phase Scoping, One-Shot Triggers & Emit-on-Change
description: How gather rules are restricted to enrollment phases (activePhases / activeFromPhase), collect once at a phase boundary (phase_change / phase_exit), and how emitMode "on_change" suppresses repeated identical results.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Gather/GatherRuleExecutor.cs
tags:
- agent
- backend
- gather-rules
- config
timestamp: 2026-07-20T00:00:00+02:00
---
# Problem
Interval gather rules (e.g. a registry poll every 60 s) ran during **all** enrollment phases
and emitted an event on **every** tick — including `exists=false` events while the target key
did not exist yet. A rule needed only from Account Setup onwards produced hundreds of
identical noise events per session and wasted collector runs.
# Schema
Three opt-in fields on `GatherRule` (Shared model, `rules/schema/gather-rule.schema.json`,
delivered to the agent via `GET agent/config`; ConfigVersion 34). Absent fields = legacy
behavior — full back-compat.
| Field | Type | Semantics |
|---|---|---|
| `activePhases` | `string[]` \| null | Rule runs only while the current phase is in the list. Null/empty = unrestricted. |
| `activeFromPhase` | `string` \| null | Sticky latch: activates when the phase first reaches the threshold, then stays active for the rest of the session — including through `Failed`. Null = unrestricted. |
| `emitMode` | `"always"` \| `"on_change"` \| null | `on_change`: poll on the trigger cadence, emit only when the collected result changes. Null/`always` = emit every collection (legacy). |
Canonical phase tokens are the `EnrollmentPhase` enum **names** `Start`(0) …`Complete`(7).
Backend validation (`GatherRulesFunction.ValidateScopeAndEmitMode`) rejects `Unknown`/`Failed`,
numeric tokens, unknown emit modes, and both scope fields set at once (400). The agent
defensively prefers `activePhases` if both arrive anyway.
# One-shot collection at phase boundaries
Scoping answers "while *may* this rule run". The trigger answers "when does it fire". For
one-shot collection the two phase triggers are the bookends of a phase:
| Trigger | Fires |
|---|---|
| `phase_change` | Once when `triggerPhase` is **entered** (dedup key `{ruleId}\|{phase}`). |
| `phase_exit` | Once when `triggerPhase` is **left** (dedup key `exit:{ruleId}\|{phase}`). |
Empty `triggerPhase` means every transition. Both keys live in the same
`_phaseRulesExecuted` set; the `exit:` prefix keeps the two key spaces disjoint, so an
enter-rule and an exit-rule on the same phase never cannibalise each other's slot.
`phase_exit` semantics that are easy to get wrong:
* It is evaluated **before** `_currentPhase` advances, so both the `triggerPhase` match and
the phase-scope gate see the phase being **left**. A rule scoped to `DeviceSetup` therefore
still fires on its own exit, even though the phase being entered is outside that scope.
* A transition into `Failed` **does** fire the exit rules of the failing phase — capturing
state at the failure boundary is the point.
* Exit of `Unknown` never fires (nothing was entered yet), and a repeated identical phase
signal is not a transition.
* An exit rule on `Complete` only fires if something transitions *away* from Complete. For
"at the end of enrollment", use `phase_change` on `Complete` or `on_event` with
`enrollment_complete`.
* Old agents do not know the trigger value and simply never fire the rule (silent no-op) —
which is why `phase_exit` needs no ConfigVersion bump: no new field is transported.
# Scope semantics (agent, `GatherRuleExecutor`)
* Scope applies to **all trigger types**:
* `interval` — timers keep running; an out-of-scope tick is a free early-return (no
start/stop lifecycle races on phase transitions).
* `startup` — a scoped startup rule not yet in scope at `UpdateRules` is **deferred**: it
runs exactly once from `OnPhaseChanged` when its scope activates (same
`_startupRulesExecuted` dedup; it is not part of the `WaitForStartupRules` latch).
* `phase_change` — evaluated against the **new** phase; the scope gate runs before the
per-(rule, phase) dedup so an out-of-scope pass does not consume the execution slot.
* `on_event` — gated by the current phase.
* **Before the first phase signal** of a session (`_currentPhase == Unknown`) scoped rules are
inactive. Sessions that never produce phase signals never run them (documented behavior).
* **From-phase latch**: activates when `phase != Unknown && phase != Failed && (int)phase >=
(int)fromPhase`. `Failed=99` would ordinal-satisfy every threshold, hence the explicit
exclusion — but once latched, a rule stays active through `Failed`. A config refresh that
delivers a from-phase rule after its threshold passed latches it immediately.
* `--run-gather-rules` (diagnostic CLI) sets `IgnorePhaseScope = true` — no phase context
exists there, scoped rules execute unconditionally (log line marks the bypass).
# Emit-on-change semantics
* The collector result dict is canonically hashed (keys ordinal-sorted at every nesting
level, invariant value formatting, SHA-256) **before** `ruleId`/`ruleTitle` injection.
* First in-scope non-empty result always emits — on an absent registry key that first
`exists=false` event *is* the visible "we're polling now" indicator.
* Unchanged results increment a per-rule suppression counter; the next emitted event carries
`suppressedPolls` + `suppressedSinceUtc` in its `Data` so the gap is observable.
* Composition with `emitOnlyIfExists=true` (collector-level): a miss returns an empty dict —
no emit AND no hash update. Result: zero noise while absent, one event on appearance, then
only on change.
* State is in-memory and survives `UpdateRules` config refreshes; an agent restart causes at
most one re-emit per rule (acceptable under the per-enrollment lifecycle).
* Volatile collectors (eventlog timestamps, logparser positions) degrade `on_change` to
`always` — every poll hashes differently. The option stays visible for them since some
outputs are stable; this is documented, not hidden.
# Portal & delivery
* Create form defaults new rules to `emitMode: "on_change"`; existing rules load as
`always`. The scope UI is a three-way mode (always / during phases / from phase) backed by
a canonical `GATHER_PHASES` list — the same list replaced the free-text `triggerPhase`
input, whose placeholder used to suggest invalid tokens.
* **Controls are shown only where they change behavior** (`supportsPhaseScope` /
`supportsEmitMode` in `types.ts`). Offering all three "when" controls unconditionally
produced meaningless combinations — a scope on a `phase_change` rule naming a concrete
phase can only ever suppress the single firing, and `on_change` has no second result to
compare against. Hidden controls are nulled by `buildScopeFields` so state from an earlier
trigger choice cannot leak into the payload, and `validateScopeSelection` rejects a scope
mode that was selected but left unfilled (which previously serialised to null and silently
ran the rule everywhere while the form still read "From a phase onwards").
Startup is the deliberate exception: a scope there defers the one-shot rather than
suppressing it, so the scope control stays while emit mode is hidden.
* The custom-rule toggle PUT (`{enabled, isBuiltIn, isCommunity}`) previously full-replaced
the stored rule and wiped Title/Target/Trigger — `GatherRuleService.UpdateRuleAsync` now
merges toggle-style partials (empty Title/CollectorType/Target) into the existing row and
preserves `CreatedAt` on full edits. This fix is a prerequisite: without it every toggle
would silently clear the new fields.
* Old agents (Newtonsoft) ignore unknown config properties: scoped rules run everywhere,
`emitMode` is ignored — graceful degradation until the fleet rolls forward.
* `ContentEquivalent` treats null and empty `activePhases` as equal, so pre-existing DB rows
(absent columns → null) never diff against seeds — no reseed churn.
# Citations
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Gather/GatherRuleExecutor.cs` — scope gates, latch, canonical hash
* `src/Backend/AutopilotMonitor.Functions/Functions/Rules/GatherRulesFunction.cs` — `ValidateScopeAndEmitMode`
* `src/Backend/AutopilotMonitor.Functions/Services/GatherRuleService.cs` — partial-PUT merge, `ContentEquivalent`
* `src/Shared/AutopilotMonitor.Shared/Models/Rules/GatherRule.cs` — field contracts
* Tests: `GatherRuleExecutorPhaseScopeTests`, `GatherRuleExecutorEmitModeTests` (agent); `GatherRuleUpdatePartialMergeTests`, `GatherRuleScopeFieldsTests` (backend)
+85
View File
@@ -0,0 +1,85 @@
---
type: Concept
title: URL Registry — One Place per Component, Guard-Enforced
description: Every well-known own or Microsoft host lives in exactly one registry file per component (Constants.cs, utils/config.ts, src/config.ts); guard tests fail the build on any literal outside them. Includes the agent-download host migration to the download alias.
resource: /src/Shared/AutopilotMonitor.Shared/Constants.cs
tags:
- architecture
- urls
- ci
- convention
timestamp: 2026-07-23T00:00:00+02:00
---
# URL Registry
Every well-known base URL — own hosts (portal, www, docs, download alias, MCP,
API, both blob accounts) and Microsoft hosts (Graph, Entra login) — is defined
in exactly one registry file per component. Code references the constant; the
literal appears nowhere else.
| Component | Registry | Guard test |
|---|---|---|
| C# (Shared/Backend/Agent) | `AutopilotMonitor.Shared.Constants` (`*BaseUrl`) | `HardcodedUrlGuardTests` |
| Web | `utils/config.ts` (`DOCS_URL`, `SITE_URL`, `PORTAL_URL`, `ENTRA_LOGIN_URL`, `API_URL_PROD`, `BLOB_URL_PROD`, `AGENT_DOWNLOAD_HOSTNAMES`) | `utils/__tests__/hardcodedUrls.guard.test.ts` |
| MCP | `src/config.ts` (`API_BASE_URL`, `ENTRA_LOGIN_BASE_URL`, `DOCS_BASE_URL`) | `src/__tests__/hardcoded-urls.guard.test.ts` |
Derived registries build on these instead of repeating hosts:
`lib/hostRouting.ts` computes `PUBLIC_HOST`/`PORTAL_HOST`/`APEX_HOST` from the
URL constants, and `middleware.ts` imports from it.
# Schema
The guards scan non-test source for the enforced host list and fail with
file:line on any occurrence outside the registry. Comment lines are ignored
(docs may cite URLs). Test files are excluded on purpose: a literal in a test
is an independent oracle — `ValidateBootstrapCodeResponseShapeTests` spells the
full agent download URL out so a registry change must be acknowledged in two
places, backend and portal allow-list, consciously.
Why enforced rather than agreed: the registry existed before (both
`Constants.AgentBlobBaseUrl` and the MCP `config.ts` header document earlier
drift), yet the EU cutover still missed two literal copies of the blob host —
`LatestVersionsService` and `ValidateBootstrapCodeFunction` kept reading the
legacy account, silently kept alive only by the fail-soft legacy mirror in
`build-agent.yml`. A convention that is not checked decays; these guards make
the next stray literal a red build instead of a production surprise.
# Agent-download host migration (2026-07-23)
The two legacy readers above plus the agent's `SelfUpdater` now use
`Constants.AgentDownloadBaseUrl` (`download.autopilotmonitor.com/agent`, Front
Door in front of the current blob origin — verified serving `version.json`,
`version-v2.json` and both ZIPs before the switch). The portal's
`bootstrapValidation.ts` allow-lists BOTH the alias and the legacy blob host
(`AGENT_DOWNLOAD_HOSTNAMES`) because bootstrap scripts already deployed in
customer Intune tenants still carry legacy URLs.
Deploy-order constraint: the web allow-list must be live BEFORE (or with) the
backend that serves alias URLs — an old portal build rejects the alias host and
the bootstrap flow dies client-side. Remove the legacy hostname from the
allow-list only after the legacy account is torn down.
**Front Door caching is a correctness hazard here, not a staleness nuisance.**
The first release after the migration exposed it: the alias kept serving the
previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) while the origin was already
updated — and because manifest and ZIP are independent cache entries, a stale
manifest paired with a fresh ZIP fails the bootstrap's SHA-256 check. Two
defenses, both in place since 2026-07-23: route caching on `/agent/*` is
DISABLED, and every blob upload (CI `Send-Blob`, local `build.ps1`, the
`versions/*.json` manifests) stamps `Cache-Control: no-cache` — all these blobs
rotate in place, even the per-line ZIP. If caching is ever re-enabled, the
headers keep the mismatch class impossible.
Deliberately still on the legacy account: the `HealthCheckService` /
`MaintenanceService` legacy-keepalive probes and the fail-soft mirror in
`build-agent.yml` — they exist precisely to keep already-deployed customer
bootstrap scripts working until the customer migration completes.
# Citations
- `src/Shared/AutopilotMonitor.Shared/Constants.cs` — C# registry
- `src/Web/autopilot-monitor-web/utils/config.ts` — web registry
- `src/McpServer/autopilot-monitor-mcp/src/config.ts` — MCP registry
- `src/Backend/AutopilotMonitor.Functions.Tests/HardcodedUrlGuardTests.cs`
- [Version Contract](versioning.md) — the release pipeline this migration protects
+108
View File
@@ -0,0 +1,108 @@
---
type: Concept
title: Version Contract — Agent, Backend, MCP
description: One version shape for all three shipping components, the ETag-CAS counter blob that mints build numbers, and the published manifests that make "what is deployed" checkable from outside.
resource: /.github/scripts/Request-BuildNumber.ps1
tags:
- build
- ci
- release
- versioning
timestamp: 2026-07-22T00:00:00+02:00
---
# Version Contract
Three components ship from this repository — the agent, the Functions backend and the
MCP server. They share one version shape, one way of minting build numbers, and one
manifest format, so that a version string means the same thing whoever emits it.
```
version = <major>.<minor>.<build> major.minor curated by hand, build minted per build
commit = 7-char short SHA of this repo
buildUtc = ISO-8601 build moment
```
| Component | major.minor from | Counter blob | Reported by |
|---|---|---|---|
| Agent V2 | `VersionPrefix` in `AutopilotMonitor.Agent.V2.csproj` | `agent/buildcounter-v2.txt` | `agent/version.json` |
| Backend | `VersionPrefix` in `AutopilotMonitor.Functions.csproj` | `versions/backend.txt` | `GET /api/health` |
| MCP | `version` in `package.json` | `versions/mcp.txt` | `GET /health`, MCP handshake |
# Schema
## Build numbers — reserve before build, ETag-CAS
`.github/scripts/Request-BuildNumber.ps1` is the only implementation; the CI
workflows and the local `build.ps1` all call it.
1. Anonymous `GET` on the counter blob → value `N` and `ETag`. The blob holds the
**last used** number as plain text and is public-read; only the write needs the
container SAS.
2. `PUT N+1` with `If-Match: <etag>`. The quoted ETag form is required — PowerShell
strips the quotes on read, and an unquoted `If-Match` is rejected client-side.
3. `412 Precondition Failed` → a concurrent build reserved first → re-read and retry
(5 attempts, linear backoff).
4. Build with the reserved number.
Reserving *before* the build is what makes concurrent minting impossible. **A failed
build burns a number** — deliberate: uniqueness matters, density does not.
Why a counter and not the published manifest: manifests are written on publish, not on
build. The agent writes `version.json` only for stable releases, so deriving the next
number from it would hand two consecutive dev builds the same number and let the second
overwrite the first's artifact. A state-free scheme such as `git rev-list --count HEAD`
fails for the same reason — two builds of one commit collide.
**Seeding** (once per component): `PUT <last used number>` with `If-None-Match: *`, so
an existing counter can never be clobbered. Agent V2 was seeded with `1356` on
2026-07-17; `versions/backend.txt` with `999` and `versions/mcp.txt` with `299` on
2026-07-22.
## Local builds
No counter is reached, the build number stays `0`, and the commit is empty or
`unknown`. A `1.5.0` with no commit hash on `/api/health` is therefore a developer
machine, not a release — the fallback is the signal, not a defect. The agent is the
exception: `build.ps1` produces uploadable artifacts and reserves a real number unless
`-SkipUpload` is passed.
## Manifests
`versions/backend.json` and `versions/mcp.json` (public-read) record what is deployed:
```json
{ "component": "backend", "version": "1.5.1000", "commit": "abc1234",
"buildUtc": "…", "deployedUtc": "…", "runId": "…" }
```
The MCP manifest adds `docsCommit`, because for that component "which build" and "which
documentation" are separate questions.
Each is written **last and only after the deploy workflow has polled the component's
health endpoint until it reports that exact version and commit**. A green deploy step
alone means the package was accepted, not that it is being served. Consequently a
manifest that disagrees with the live endpoint means the running instance drifted after
the fact (rollback, stale revision) rather than a half-finished pipeline.
The agent keeps its own manifest at `agent/version.json` — the bootstrap derives that
URL from the download URL and verifies the ZIP against its `sha256`, so it cannot move.
It carries the same `commit` and `buildUtc` fields; `sha256` and `bootstrapVersion`
remain, being the customer-facing integrity contract.
# Consequences
- Bumping `major.minor` is a human decision, made in the component's own file
(`VersionPrefix`, or `version` in `package.json`). Nothing bumps it automatically.
- A new component joins by seeding a counter blob and calling the same script; no new
mechanism.
- The counter blobs and manifests live in the `versions` container on
`autopilotmonitoreu` (writes via `AZURE_BLOB_VERSIONS_SAS_TOKEN`). The agent's
counter stays in the `agent` container next to the artifacts it numbers.
# Citations
- `.github/scripts/Request-BuildNumber.ps1` — the CAS protocol
- `.github/workflows/deploy-backend.yml`, `deploy-mcp.yml` — reserve, verify, publish
- `.github/workflows/build-agent.yml`, `scripts/Deployment/build.ps1` — agent line
- [Build Counter Blob](agent/build-counter-blob.md) — agent-specific consequences
+6
View File
@@ -0,0 +1,6 @@
{
"sdk": {
"version": "10.0.200",
"rollForward": "latestFeature"
}
}
+268
View File
@@ -0,0 +1,268 @@
// Autopilot-Monitor MCP Server — Azure Container App Infrastructure
//
// Reconciled against the live deployment on 2026-07-21 (rg-autopilotmonitor-prd-gwc,
// Germany West Central). The template had drifted behind two changes made directly
// in Azure: the EU cutover (registry/API renamed) and the ACR hardening (admin user
// disabled, image pull moved to the app's system-assigned identity).
//
// !! BEFORE RE-DEPLOYING THIS TEMPLATE, READ THIS !!
// The live app carries an ingress custom-domain binding for mcp.autopilotmonitor.com
// (SNI-enabled, with a managed certificate). Custom domains are NOT modelled below,
// because the certificate is bound out-of-band. A full template deployment would
// therefore REMOVE that binding and take the public MCP endpoint offline.
// For routine changes (CPU/memory, env vars, image) prefer the surgical CLI form:
// az containerapp update -n autopilotmonitor-mcp -g rg-autopilotmonitor-prd-gwc --cpu 0.5 --memory 1.0Gi
// Treat this file as the documented desired state and as a rebuild-from-scratch
// recipe — not as a routinely-applied deployment.
//
// On a FROM-SCRATCH deploy the AcrPull assignment below is created only after the
// Container App exists (it needs the app's principalId), so the very first revision
// can fail to pull its image. Re-run the deployment, or restart the revision, once
// the role assignment has propagated. This does not affect updates to a live app.
//
// Deploy (two-stage; required to set the OAuth public URL safely):
// 1. First deploy without mcpPublicUrl:
// az deployment group create \
// --resource-group <rg-name> \
// --template-file infra/mcp-server.bicep \
// --parameters apiUrl=https://autopilotmonitor-api.azurewebsites.net \
// entraClientSecret=<secret-value>
// This emits `containerAppUrl` as an output (e.g. https://autopilotmonitor-mcp.<env-suffix>.<region>.azurecontainerapps.io).
//
// 2. Re-deploy with that URL pinned, so the OAuth proxy never trusts caller-
// supplied X-Forwarded-Host / Host headers when building issuer / WWW-
// Authenticate / redirect_uri metadata:
// --parameters mcpPublicUrl=<containerAppUrl from step 1>
//
// When mcpPublicUrl is set, the MCP server uses it verbatim and ignores any
// forwarded-headers from the request. Without it, the server logs a startup
// warning and falls back to header-derived URLs (acceptable for local dev,
// not for production).
@description('Azure region for all resources')
param location string = resourceGroup().location
@description('Name of the Container App')
param containerAppName string = 'autopilotmonitor-mcp'
@description('Name of the Container App Environment')
param environmentName string = 'autopilotmonitor-env'
@description('Name of the Log Analytics Workspace (dedicated for Container Apps)')
param logAnalyticsName string = 'autopilotmonitor-container-logs'
@description('Name of the Azure Container Registry')
param acrName string = 'autopilotmonitoreuacr'
@description('Backend API URL')
param apiUrl string = 'https://autopilotmonitor-api-eu.azurewebsites.net'
@description('Container image tag')
param imageTag string = 'latest'
@description('Entra ID app registration client_id used for the OAuth proxy. Public value (appears in JWT aud claim); not a secret.')
param entraClientId string = '1a400946-62c1-4ab4-aa37-f730ac89704d'
@secure()
@description('Entra ID client secret for OAuth token exchange')
param entraClientSecret string = ''
@description('Comma-separated allowlist of redirect_uri hosts for the OAuth proxy. Loopback is always allowed. Empty = built-in default (Anthropic, OpenAI, Google).')
param mcpAllowedRedirectHosts string = ''
@description('Public URL of the MCP server, used verbatim for OAuth issuer / WWW-Authenticate / redirect metadata. REQUIRED in production: the container refuses to boot without it (host-spoofing defense — see config.ts). Set on the second deploy with the value of the containerAppUrl output from the first deploy. The stage-1 deploy is safe because minReplicas=0 means no container runs until this is pinned. Empty falls back to X-Forwarded-Host headers only outside production (NODE_ENV != production).')
param mcpPublicUrl string = ''
@secure()
// NOTE: avoid characters outside cp1252 here (this one used to say a Unicode
// >=). `az bicep build --stdout` writes through the Windows console codec and
// dies with a UnicodeEncodeError on anything it cannot map. Em dashes are fine.
@description('Base64-encoded HMAC key (>=32 bytes after decode) for signing the OAuth proxy state. Naming and format match the backend\'s PaginationTokenSigningKey for consistency. Empty = generate per-instance random key on startup (acceptable for single-replica deployments since state has 10 min lifetime). Generate with PowerShell: [Convert]::ToBase64String([Security.Cryptography.RandomNumberGenerator]::GetBytes(32))')
param oAuthStateSigningKey string = ''
@description('Enable structured tool-call logging to stderr (queryable via Container App Logs)')
param toolLoggingEnabled bool = false
// --- Azure Container Registry ---
// Admin user is DISABLED: a registry-wide username/password pair is a static,
// non-rotating credential that would also have to live in a Container App secret.
// The app pulls with its system-assigned identity + AcrPull instead (see below).
resource acr 'Microsoft.ContainerRegistry/registries@2023-07-01' = {
name: acrName
location: location
sku: {
name: 'Basic'
}
properties: {
adminUserEnabled: false
}
}
// AcrPull for the Container App's system-assigned identity — this is what replaces
// the admin credentials. Without it the app cannot pull its own image.
//
// NOTE: this assignment exists on the live registry but could not be enumerated
// during reconciliation (the CLI session lacks Microsoft.Authorization read), so it
// is reproduced here from the documented pattern rather than read back from Azure.
// Verify with:
// az role assignment list --scope <acr-id> --assignee <containerApp principalId>
var acrPullRoleId = '7f951dda-4ed3-4680-a7ca-43fe172d538d'
resource acrPull 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
scope: acr
name: guid(acr.id, containerAppName, acrPullRoleId)
properties: {
roleDefinitionId: subscriptionResourceId('Microsoft.Authorization/roleDefinitions', acrPullRoleId)
principalId: containerApp.identity.principalId
principalType: 'ServicePrincipal'
}
}
// --- Log Analytics Workspace (required by Container App Environment) ---
resource logAnalytics 'Microsoft.OperationalInsights/workspaces@2023-09-01' = {
name: logAnalyticsName
location: location
properties: {
sku: {
name: 'PerGB2018'
}
retentionInDays: 30
}
}
// --- Container App Environment ---
resource environment 'Microsoft.App/managedEnvironments@2024-03-01' = {
name: environmentName
location: location
properties: {
appLogsConfiguration: {
destination: 'log-analytics'
logAnalyticsConfiguration: {
customerId: logAnalytics.properties.customerId
sharedKey: logAnalytics.listKeys().primarySharedKey
}
}
}
}
// --- Container App ---
resource containerApp 'Microsoft.App/containerApps@2024-03-01' = {
name: containerAppName
location: location
// System-assigned identity: the credential the app uses to pull from the ACR
// (see the AcrPull assignment above). Replaces the former admin-user secret.
identity: {
type: 'SystemAssigned'
}
properties: {
managedEnvironmentId: environment.id
configuration: {
ingress: {
external: true
targetPort: 8080
transport: 'auto'
allowInsecure: false
}
// 'system' = pull with the app's system-assigned identity. No username, no
// password secret — nothing to rotate and nothing to leak from the app config.
registries: [
{
server: acr.properties.loginServer
identity: 'system'
}
]
secrets: [
{
name: 'entra-client-secret'
value: entraClientSecret
}
{
name: 'oauth-state-signing-key'
value: oAuthStateSigningKey
}
]
activeRevisionsMode: 'Single'
}
template: {
containers: [
{
name: 'mcp-server'
image: '${acr.properties.loginServer}/${containerAppName}:${imageTag}'
// 0.5 vCPU, but NOT for the reason originally given. It was raised from
// 0.25 to shorten the scale-to-zero cold start, on the assumption that
// loading the ONNX embedding model dominated it. Measurement disproved
// that: the app boots in 504 ms in production (see the [boot] marks in
// the container log and docs/mcp/docs-corpus.md), of which the prebaked
// embedder is 173 ms in the background. Of a 31 s cold start, ~98% is
// Container Apps activation — pod scheduling, image mount, container
// start — which no CPU setting touches.
//
// It stays at 0.5 because it plausibly helps CONCURRENT requests
// (maxReplicas is 1, so one replica serves up to 10 in-flight calls) and
// because at scale-to-zero the container bills only while active, making
// the difference negligible. If cold start ever needs to actually go
// away, the only lever is minReplicas: 1. Container Apps accepts fixed
// CPU/memory pairs, so memory moves with the CPU.
resources: {
cpu: json('0.5')
memory: '1.0Gi'
}
env: [
{ name: 'AUTOPILOT_API_URL', value: apiUrl }
{ name: 'PORT', value: '8080' }
{ name: 'AUTOPILOT_ENTRA_CLIENT_ID', value: entraClientId }
{ name: 'AUTOPILOT_ENTRA_CLIENT_SECRET', secretRef: 'entra-client-secret' }
{ name: 'MCP_ALLOWED_REDIRECT_HOSTS', value: mcpAllowedRedirectHosts }
{ name: 'MCP_PUBLIC_URL', value: mcpPublicUrl }
{ name: 'OAuthStateSigningKey', secretRef: 'oauth-state-signing-key' }
{ name: 'MCP_TOOL_LOGGING', value: toolLoggingEnabled ? 'true' : 'false' }
]
probes: [
{
type: 'Liveness'
httpGet: {
path: '/health'
port: 8080
}
periodSeconds: 30
initialDelaySeconds: 10
}
{
type: 'Readiness'
httpGet: {
path: '/health'
port: 8080
}
periodSeconds: 10
initialDelaySeconds: 5
}
]
}
]
scale: {
minReplicas: 0
maxReplicas: 1
rules: [
{
name: 'http-scaler'
http: {
metadata: {
concurrentRequests: '10'
}
}
}
]
}
}
}
}
// --- Outputs ---
output containerAppUrl string = 'https://${containerApp.properties.configuration.ingress.fqdn}'
output mcpEndpoint string = 'https://${containerApp.properties.configuration.ingress.fqdn}/mcp'
output acrLoginServer string = acr.properties.loginServer
+85
View File
@@ -0,0 +1,85 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-ESP-005",
"title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)",
"description": "Attributes an observed enrollment reboot to device-assigned MDM policies that Windows flagged as reboot-required. Fires ONLY when both facts are present in the session: at least one actually-observed reboot (system_reboot_detected) AND reboot-required policy flags (mdm_policy_reboot_required). Policy flags alone — e.g. from a user-policy sync during AccountSetup — force no reboot and stay advisory-silent.",
"severity": "warning",
"category": "esp",
"version": "1.1.0",
"author": "Autopilot Monitor",
"enabled": true,
"trigger": "single",
"baseConfidence": 70,
"conditions": [
{
"signal": "reboot_forcing_policy_uri",
"source": "event_data",
"eventType": "mdm_policy_reboot_required",
"dataField": "firstRebootUri",
"operator": "exists",
"required": true
},
{
"signal": "reboot_observed",
"source": "event_count",
"eventType": "system_reboot_detected",
"operator": "count_gte",
"value": "1",
"required": true
},
{
"signal": "multiple_reboot_uris",
"source": "event_data",
"eventType": "mdm_policy_reboot_required",
"dataField": "uriCount",
"operator": "gte",
"value": "2",
"required": false
},
{
"signal": "omadm_reboot_flag_set",
"source": "event_data",
"eventType": "mdm_policy_reboot_required",
"dataField": "omadmRebootRequiredFlag",
"operator": "equals",
"value": "true",
"required": false
}
],
"confidenceFactors": [
{ "signal": "multiple_reboot_uris", "condition": "exists", "weight": 10 },
{ "signal": "omadm_reboot_flag_set", "condition": "exists", "weight": 15 }
],
"confidenceThreshold": 70,
"explanation": "This session **rebooted during enrollment**, and Windows flagged one or more device-assigned MDM policies as **reboot-required** — the classic cause of the coalesced mid-ESP restart and the unexpected **second sign-in screen** before AccountSetup.\n\nFirst reboot-forcing policy URI: `{{firstRebootUri}}`\n\nThe complete URI list is in the `mdm_policy_reboot_required` timeline event (`rebootUris`). Policies applied DURING ESP DeviceSetup are coalesced into one forced restart at the end of DeviceSetup; the same flags raised later (e.g. user-policy sync in AccountSetup) have no immediate effect — this advisory only fires when a reboot was actually observed.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
"remediation": [
{
"title": "Reassign the reboot-forcing policies to user groups",
"steps": [
"Open the `mdm_policy_reboot_required` timeline event and note the CSP URIs in `rebootUris` (e.g. `./Device/Vendor/MSFT/...`)",
"Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area",
"Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup",
"Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment"
]
},
{
"title": "Or accept the reboot and set expectations",
"steps": [
"If the policy must be device-assigned (e.g. a security baseline), keep it and document the expected mid-ESP restart",
"Communicate to users that a restart and second sign-in during provisioning is expected for these deployments"
]
}
],
"relatedDocs": [
{
"title": "Patch My PC: Avoid the second sign-in screen after Autopilot",
"url": "https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/"
},
{
"title": "Microsoft Learn: Enrollment Status Page tracking",
"url": "https://learn.microsoft.com/en-us/autopilot/enrollment-status"
}
],
"markSessionAsFailedDefault": false,
"tags": ["esp", "reboot", "coalescing", "second-sign-in", "policy-attribution", "user-experience"]
}
+90
View File
@@ -0,0 +1,90 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-ESP-006",
"title": "ESP Failure Recovered After User Retry",
"description": "Detects the recovery story: the ESP reported a terminal failure (typically DeviceSetup/Apps), the user most likely pressed 'Try again' on the failure page, the failed step re-ran and completed successfully. The earlier failure did not stick — the enrollment continued past it.",
"severity": "info",
"category": "esp",
"version": "1.0.0",
"enabled": true,
"baseConfidence": 85,
"conditions": [
{
"signal": "advisory",
"source": "event_type",
"eventType": "esp_failure_advisory",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "recovered",
"source": "event_type",
"eventType": "esp_failure_recovered",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "retry",
"source": "event_type",
"eventType": "esp_failure_retry_detected",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "category",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "category",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "failedSubcategory",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "failedSubcategory",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "likelyCulpritApps",
"source": "event_type",
"eventType": "esp_failure_advisory",
"dataField": "likelyCulpritApps",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "minutesSinceFailure",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "minutesSinceFailure",
"operator": "exists",
"value": "",
"required": false
}
],
"confidenceFactors": [
{ "signal": "retry", "condition": "exists", "weight": 10 },
{ "signal": "likelyCulpritApps", "condition": "exists", "weight": 5 }
],
"explanation": "The Enrollment Status Page reported a **failure** in the `{{category}}` phase (`{{failedSubcategory}}` subcategory, most likely app(s): `{{likelyCulpritApps}}`) — and then **recovered**: the failed step left its failed state and completed successfully about `{{minutesSinceFailure}}` min later.\n\nThat pattern is consistent with the user pressing **'Try again'** on the ESP failure page (the profile allows failure-page actions; the agent cannot observe the click itself). The failure the user saw was real, but it did not stick — the ESP re-ran the step, it succeeded, and the enrollment continued.\n\nRead the session timeline as one arc: `esp_failure_advisory` (the failure, with the likely culprit app) → `esp_failure_retry_detected` (the retry) → `esp_failure_recovered` (the recovery). If this pattern repeats across devices for the same app, that app's install is flaky on first attempt — fix the app rather than the devices. Any literal `{{…}}` placeholder means that detail was not recorded for this session.",
"remediation": [
{
"title": "Make the first install attempt stick",
"steps": [
"Check the likely culprit app(s) (`{{likelyCulpritApps}}`) in Intune: an app that fails on the first ESP pass but succeeds on retry usually has a dependency/ordering or detection-timing problem.",
"If the app shows 'never started installing' in esp_failure_settle_started, verify its assignment and applicability rules — ESP counted it as required, but IME never processed it.",
"If the pattern repeats across the tenant, consider removing the app from the ESP blocking list until the first-attempt install is reliable — users are seeing an avoidable failure screen.",
"No device-side action is needed for this session: the failure recovered and the enrollment continued."
]
}
],
"tags": ["esp", "recovery", "try-again", "retry", "continue-anyway", "enrollment"]
}
+33 -5
View File
@@ -2,10 +2,10 @@
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-SEC-001",
"title": "Secure Boot UEFI CA 2023 certificate not deployed",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed.",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed. Since v3 the agent reads the firmware db/KEK variables directly: when the firmware itself confirms the 2023 certificate, the rule is suppressed even if the Windows servicing status has not caught up yet.",
"severity": "warning",
"category": "security",
"version": "2.0.0",
"version": "3.0.0",
"enabled": true,
"baseConfidence": 70,
"preconditions": [
@@ -16,6 +16,14 @@
"operator": "equals",
"value": "false",
"description": "Skip on virtual machines — Secure Boot DB is host-controlled (Hyper-V/VMware/etc), the inside-the-VM UEFICA2023Status registry value is not the right indicator and not actionable for the VM admin."
},
{
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiCa2023FirmwareConfirmed",
"operator": "not_exists",
"value": "",
"description": "Skip when the firmware db itself confirms the Windows UEFI CA 2023 certificate — the marker is only emitted on positive firmware confirmation, so old agents (field absent) and unconfirmed devices keep evaluating the rule. The servicing registry status lags behind actual firmware state (OEM factory image, WinCS, manual deployment, or WU simply not run yet)."
}
],
"conditions": [
@@ -54,13 +62,33 @@
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "db_ca2023_firmware_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiDbHasWindowsUefiCa2023",
"operator": "equals",
"value": "false",
"required": false
},
{
"signal": "kek_2023_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiKekHasMicrosoftKek2kCa2023",
"operator": "equals",
"value": "false",
"required": false
}
],
"confidenceFactors": [
{ "signal": "ca2023_notfound", "condition": "exists", "weight": 20 },
{ "signal": "ca2023_error", "condition": "exists", "weight": 10 }
{ "signal": "ca2023_error", "condition": "exists", "weight": 10 },
{ "signal": "db_ca2023_firmware_missing", "condition": "exists", "weight": 20 },
{ "signal": "kek_2023_missing", "condition": "exists", "weight": 10 }
],
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nCurrent UEFI CA 2023 status: `{{uefiCA2023Status}}`",
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nOn devices running a current agent this finding is verified against the firmware itself: the agent reads the UEFI `db` signature database directly, and this warning is automatically suppressed when the firmware already contains the 2023 certificate — regardless of the Windows servicing status below.\n\nCurrent UEFI CA 2023 servicing status: `{{uefiCA2023Status}}`",
"remediation": [
{
"title": "Deploy UEFI CA 2023 certificate now",
@@ -69,7 +97,7 @@
"Use Intune policy to enable Secure Boot DB update (Windows UEFI CA 2023)",
"Check with the device OEM whether a firmware update is required for the DB update to apply on this model",
"Alternatively, use the WinCS CLI tool from Microsoft for manual deployment",
"Verify deployment: check registry HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated'",
"Verify deployment against the firmware directly: [Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023') — this is authoritative; the registry value HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated' only reflects the Windows Update rollout",
"After DB update: ensure the boot manager is updated to a 2023-signed version via monthly cumulative update"
]
}
+243 -4
View File
@@ -1785,6 +1785,209 @@
"enrollment"
]
},
{
"ruleId": "ANALYZE-ESP-005",
"title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)",
"description": "Attributes an observed enrollment reboot to device-assigned MDM policies that Windows flagged as reboot-required. Fires ONLY when both facts are present in the session: at least one actually-observed reboot (system_reboot_detected) AND reboot-required policy flags (mdm_policy_reboot_required). Policy flags alone — e.g. from a user-policy sync during AccountSetup — force no reboot and stay advisory-silent.",
"severity": "warning",
"category": "esp",
"version": "1.1.0",
"author": "Autopilot Monitor",
"enabled": true,
"trigger": "single",
"baseConfidence": 70,
"conditions": [
{
"signal": "reboot_forcing_policy_uri",
"source": "event_data",
"eventType": "mdm_policy_reboot_required",
"dataField": "firstRebootUri",
"operator": "exists",
"required": true
},
{
"signal": "reboot_observed",
"source": "event_count",
"eventType": "system_reboot_detected",
"operator": "count_gte",
"value": "1",
"required": true
},
{
"signal": "multiple_reboot_uris",
"source": "event_data",
"eventType": "mdm_policy_reboot_required",
"dataField": "uriCount",
"operator": "gte",
"value": "2",
"required": false
},
{
"signal": "omadm_reboot_flag_set",
"source": "event_data",
"eventType": "mdm_policy_reboot_required",
"dataField": "omadmRebootRequiredFlag",
"operator": "equals",
"value": "true",
"required": false
}
],
"confidenceFactors": [
{
"signal": "multiple_reboot_uris",
"condition": "exists",
"weight": 10
},
{
"signal": "omadm_reboot_flag_set",
"condition": "exists",
"weight": 15
}
],
"confidenceThreshold": 70,
"explanation": "This session **rebooted during enrollment**, and Windows flagged one or more device-assigned MDM policies as **reboot-required** — the classic cause of the coalesced mid-ESP restart and the unexpected **second sign-in screen** before AccountSetup.\n\nFirst reboot-forcing policy URI: `{{firstRebootUri}}`\n\nThe complete URI list is in the `mdm_policy_reboot_required` timeline event (`rebootUris`). Policies applied DURING ESP DeviceSetup are coalesced into one forced restart at the end of DeviceSetup; the same flags raised later (e.g. user-policy sync in AccountSetup) have no immediate effect — this advisory only fires when a reboot was actually observed.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
"remediation": [
{
"title": "Reassign the reboot-forcing policies to user groups",
"steps": [
"Open the `mdm_policy_reboot_required` timeline event and note the CSP URIs in `rebootUris` (e.g. `./Device/Vendor/MSFT/...`)",
"Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area",
"Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup",
"Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment"
]
},
{
"title": "Or accept the reboot and set expectations",
"steps": [
"If the policy must be device-assigned (e.g. a security baseline), keep it and document the expected mid-ESP restart",
"Communicate to users that a restart and second sign-in during provisioning is expected for these deployments"
]
}
],
"relatedDocs": [
{
"title": "Patch My PC: Avoid the second sign-in screen after Autopilot",
"url": "https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/"
},
{
"title": "Microsoft Learn: Enrollment Status Page tracking",
"url": "https://learn.microsoft.com/en-us/autopilot/enrollment-status"
}
],
"markSessionAsFailedDefault": false,
"tags": [
"esp",
"reboot",
"coalescing",
"second-sign-in",
"policy-attribution",
"user-experience"
]
},
{
"ruleId": "ANALYZE-ESP-006",
"title": "ESP Failure Recovered After User Retry",
"description": "Detects the recovery story: the ESP reported a terminal failure (typically DeviceSetup/Apps), the user most likely pressed 'Try again' on the failure page, the failed step re-ran and completed successfully. The earlier failure did not stick — the enrollment continued past it.",
"severity": "info",
"category": "esp",
"version": "1.0.0",
"enabled": true,
"baseConfidence": 85,
"conditions": [
{
"signal": "advisory",
"source": "event_type",
"eventType": "esp_failure_advisory",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "recovered",
"source": "event_type",
"eventType": "esp_failure_recovered",
"operator": "exists",
"value": "",
"required": true
},
{
"signal": "retry",
"source": "event_type",
"eventType": "esp_failure_retry_detected",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "category",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "category",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "failedSubcategory",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "failedSubcategory",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "likelyCulpritApps",
"source": "event_type",
"eventType": "esp_failure_advisory",
"dataField": "likelyCulpritApps",
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "minutesSinceFailure",
"source": "event_type",
"eventType": "esp_failure_recovered",
"dataField": "minutesSinceFailure",
"operator": "exists",
"value": "",
"required": false
}
],
"confidenceFactors": [
{
"signal": "retry",
"condition": "exists",
"weight": 10
},
{
"signal": "likelyCulpritApps",
"condition": "exists",
"weight": 5
}
],
"explanation": "The Enrollment Status Page reported a **failure** in the `{{category}}` phase (`{{failedSubcategory}}` subcategory, most likely app(s): `{{likelyCulpritApps}}`) — and then **recovered**: the failed step left its failed state and completed successfully about `{{minutesSinceFailure}}` min later.\n\nThat pattern is consistent with the user pressing **'Try again'** on the ESP failure page (the profile allows failure-page actions; the agent cannot observe the click itself). The failure the user saw was real, but it did not stick — the ESP re-ran the step, it succeeded, and the enrollment continued.\n\nRead the session timeline as one arc: `esp_failure_advisory` (the failure, with the likely culprit app) → `esp_failure_retry_detected` (the retry) → `esp_failure_recovered` (the recovery). If this pattern repeats across devices for the same app, that app's install is flaky on first attempt — fix the app rather than the devices. Any literal `{{…}}` placeholder means that detail was not recorded for this session.",
"remediation": [
{
"title": "Make the first install attempt stick",
"steps": [
"Check the likely culprit app(s) (`{{likelyCulpritApps}}`) in Intune: an app that fails on the first ESP pass but succeeds on retry usually has a dependency/ordering or detection-timing problem.",
"If the app shows 'never started installing' in esp_failure_settle_started, verify its assignment and applicability rules — ESP counted it as required, but IME never processed it.",
"If the pattern repeats across the tenant, consider removing the app from the ESP blocking list until the first-attempt install is reliable — users are seeing an avoidable failure screen.",
"No device-side action is needed for this session: the failure recovered and the enrollment continued."
]
}
],
"tags": [
"esp",
"recovery",
"try-again",
"retry",
"continue-anyway",
"enrollment"
]
},
{
"ruleId": "ANALYZE-ID-001",
"title": "Expected Machine Certificate Not Found",
@@ -2079,10 +2282,10 @@
{
"ruleId": "ANALYZE-SEC-001",
"title": "Secure Boot UEFI CA 2023 certificate not deployed",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed.",
"description": "Detects devices where the Windows UEFI CA 2023 certificate has not been deployed to the Secure Boot DB. The 2011 certificates expired in June 2026 — affected devices no longer receive Secure Boot security updates until the 2023 certificate is deployed. Since v3 the agent reads the firmware db/KEK variables directly: when the firmware itself confirms the 2023 certificate, the rule is suppressed even if the Windows servicing status has not caught up yet.",
"severity": "warning",
"category": "security",
"version": "2.0.0",
"version": "3.0.0",
"enabled": true,
"baseConfidence": 70,
"preconditions": [
@@ -2093,6 +2296,14 @@
"operator": "equals",
"value": "false",
"description": "Skip on virtual machines — Secure Boot DB is host-controlled (Hyper-V/VMware/etc), the inside-the-VM UEFICA2023Status registry value is not the right indicator and not actionable for the VM admin."
},
{
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiCa2023FirmwareConfirmed",
"operator": "not_exists",
"value": "",
"description": "Skip when the firmware db itself confirms the Windows UEFI CA 2023 certificate — the marker is only emitted on positive firmware confirmation, so old agents (field absent) and unconfirmed devices keep evaluating the rule. The servicing registry status lags behind actual firmware state (OEM factory image, WinCS, manual deployment, or WU simply not run yet)."
}
],
"conditions": [
@@ -2131,6 +2342,24 @@
"operator": "exists",
"value": "",
"required": false
},
{
"signal": "db_ca2023_firmware_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiDbHasWindowsUefiCa2023",
"operator": "equals",
"value": "false",
"required": false
},
{
"signal": "kek_2023_missing",
"source": "event_data",
"eventType": "secureboot_status",
"dataField": "uefiKekHasMicrosoftKek2kCa2023",
"operator": "equals",
"value": "false",
"required": false
}
],
"confidenceFactors": [
@@ -2143,9 +2372,19 @@
"signal": "ca2023_error",
"condition": "exists",
"weight": 10
},
{
"signal": "db_ca2023_firmware_missing",
"condition": "exists",
"weight": 20
},
{
"signal": "kek_2023_missing",
"condition": "exists",
"weight": 10
}
],
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nCurrent UEFI CA 2023 status: `{{uefiCA2023Status}}`",
"explanation": "This device has Secure Boot enabled but the **Windows UEFI CA 2023** certificate has not been deployed to the Secure Boot DB.\n\nThe original Windows UEFI CA 2011 certificates **expired on June 27, 2026 — this deadline has passed**. The automatic pre-expiry rollout window is over: this device no longer receives Secure Boot security updates (DB/DBX) and cannot trust 2023-signed boot managers until the 2023 certificate is deployed. Deploying it now requires active remediation (Windows servicing opt-in and, on some models, an OEM firmware update) rather than waiting for an automatic update.\n\nOn devices running a current agent this finding is verified against the firmware itself: the agent reads the UEFI `db` signature database directly, and this warning is automatically suppressed when the firmware already contains the 2023 certificate — regardless of the Windows servicing status below.\n\nCurrent UEFI CA 2023 servicing status: `{{uefiCA2023Status}}`",
"remediation": [
{
"title": "Deploy UEFI CA 2023 certificate now",
@@ -2154,7 +2393,7 @@
"Use Intune policy to enable Secure Boot DB update (Windows UEFI CA 2023)",
"Check with the device OEM whether a firmware update is required for the DB update to apply on this model",
"Alternatively, use the WinCS CLI tool from Microsoft for manual deployment",
"Verify deployment: check registry HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated'",
"Verify deployment against the firmware directly: [Text.Encoding]::ASCII.GetString((Get-SecureBootUEFI -Name db).Bytes).Contains('Windows UEFI CA 2023') — this is authoritative; the registry value HKLM\\SYSTEM\\CurrentControlSet\\Control\\SecureBoot\\Servicing\\UEFICA2023Status = 'Updated' only reflects the Windows Update rollout",
"After DB update: ensure the boot manager is updated to a 2023-signed version via monthly cumulative update"
]
}
+75 -1
View File
@@ -43,7 +43,8 @@
"prefixes": [
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Setup",
"SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\OOBE",
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon"
"SOFTWARE\\Microsoft\\Windows NT\\CurrentVersion\\Winlogon",
"SOFTWARE\\Microsoft\\Windows\\Autopilot"
]
},
{
@@ -109,6 +110,79 @@
"SELECT * FROM Win32_LogicalDisk",
"SELECT * FROM SoftwareLicensingProduct"
],
"eventLogChannels": [
{
"category": "Core Windows logs",
"channels": [
"Application",
"System",
"Setup",
"Microsoft-Windows-Kernel-Boot",
"Microsoft-Windows-Diagnostics-Performance"
]
},
{
"category": "MDM / Enrollment",
"channels": [
"Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider",
"Microsoft-Windows-ModernDeployment-Diagnostics-Provider",
"Microsoft-Windows-Provisioning-Diagnostics-Provider",
"Microsoft-Windows-AAD",
"Microsoft-Windows-User Device Registration"
]
},
{
"category": "ESP / Shell / Apps",
"channels": [
"Microsoft-Windows-Shell-Core",
"Microsoft-Windows-AppXDeployment",
"Microsoft-Windows-AppXDeploymentServer",
"Microsoft-Windows-AppReadiness",
"Microsoft-Windows-Store"
]
},
{
"category": "Security posture / Crypto",
"channels": [
"Microsoft-Windows-BitLocker-API",
"Microsoft-Windows-BitLocker-DrivePreparationTool",
"Microsoft-Windows-TPM-WMI",
"Microsoft-Windows-CertificateServicesClient-Lifecycle-System"
]
},
{
"category": "Update / Servicing",
"channels": [
"Microsoft-Windows-WindowsUpdateClient"
]
},
{
"category": "Logon / Session",
"channels": [
"Microsoft-Windows-Winlogon",
"Microsoft-Windows-User Profile Service",
"Microsoft-Windows-GroupPolicy",
"Microsoft-Windows-TaskScheduler"
]
},
{
"category": "Network",
"channels": [
"Microsoft-Windows-NetworkProfile",
"Microsoft-Windows-Dhcp-Client",
"Microsoft-Windows-DNS-Client",
"Microsoft-Windows-NCSI",
"Microsoft-Windows-WLAN-AutoConfig",
"Microsoft-Windows-Time-Service"
]
}
],
"blockedEventLogChannels": [
"Security",
"Microsoft-Windows-PowerShell",
"Windows PowerShell",
"Microsoft-Windows-Sysmon"
],
"allowedCommands": [
{
"category": "TPM and Security",
+21 -3
View File
@@ -80,8 +80,8 @@
},
"trigger": {
"type": "string",
"description": "When to collect data.",
"enum": ["startup", "phase_change", "interval", "on_event"]
"description": "When to collect data. \"phase_change\" fires once when triggerPhase is entered, \"phase_exit\" once when it is left.",
"enum": ["startup", "phase_change", "phase_exit", "interval", "on_event"]
},
"intervalSeconds": {
"type": "integer",
@@ -90,12 +90,30 @@
},
"triggerPhase": {
"type": "string",
"description": "Phase to trigger on (only used when trigger = \"phase_change\")."
"description": "Phase to trigger on (used when trigger = \"phase_change\" or \"phase_exit\"). Empty = every phase transition."
},
"triggerEventType": {
"type": "string",
"description": "Event type to trigger on (only used when trigger = \"on_event\")."
},
"activePhases": {
"type": "array",
"description": "Run the rule only while the current enrollment phase is one of these phases (EnrollmentPhase enum names). Empty/absent = unrestricted. Mutually exclusive with activeFromPhase.",
"items": {
"type": "string",
"enum": ["Start", "DevicePreparation", "DeviceSetup", "AppsDevice", "AccountSetup", "AppsUser", "FinalizingSetup", "Complete"]
}
},
"activeFromPhase": {
"type": "string",
"description": "Activate the rule once the enrollment phase first reaches this phase, then keep it active for the rest of the session (sticky latch). Absent = unrestricted. Mutually exclusive with activePhases.",
"enum": ["Start", "DevicePreparation", "DeviceSetup", "AppsDevice", "AccountSetup", "AppsUser", "FinalizingSetup", "Complete"]
},
"emitMode": {
"type": "string",
"description": "Emit behavior: \"always\" (default — emit on every collection) or \"on_change\" (poll on the trigger cadence but emit only when the collected result changes; the first in-scope result always emits).",
"enum": ["always", "on_change"]
},
"outputEventType": {
"type": "string",
"description": "EventType for the emitted event (e.g., \"gather_proxy_settings\")."
+14 -1
View File
@@ -91,6 +91,7 @@ if (fs.existsSync(guardrailsPath)) {
// Flatten categorized lists for validation
const flatRegistryPrefixes = guardrails.registryPrefixes.flatMap((g) => g.prefixes);
const flatCommands = guardrails.allowedCommands.flatMap((g) => g.commands);
const flatEventLogChannels = guardrails.eventLogChannels.flatMap((g) => g.channels);
const ts = `/**
* AUTO-GENERATED from rules/guardrails.json — DO NOT EDIT.
@@ -114,6 +115,10 @@ export const COMMAND_CATEGORIES: readonly GuardrailCategory[] = [
${categorizedArray(guardrails.allowedCommands, 'commands')}
];
export const EVENT_LOG_CHANNEL_CATEGORIES: readonly GuardrailCategory[] = [
${categorizedArray(guardrails.eventLogChannels, 'channels')}
];
// ---------------------------------------------------------------------------
// Flat arrays (for validation logic)
// ---------------------------------------------------------------------------
@@ -141,10 +146,18 @@ ${flatArray(guardrails.diagnosticsPathPrefixes)}
export const BLOCKED_FILE_PREFIXES: readonly string[] = [
${flatArray(guardrails.blockedFilePrefixes)}
];
export const ALLOWED_EVENT_LOG_CHANNELS: readonly string[] = [
${flatArray(flatEventLogChannels)}
];
export const BLOCKED_EVENT_LOG_CHANNELS: readonly string[] = [
${flatArray(guardrails.blockedEventLogChannels)}
];
`;
fs.writeFileSync(guardrailsOutput, ts, 'utf8');
console.log(`guardrails.generated.ts: ${flatRegistryPrefixes.length} registry, ${flatCommands.length} commands, ${guardrails.filePrefixes.length} file prefixes`);
console.log(`guardrails.generated.ts: ${flatRegistryPrefixes.length} registry, ${flatCommands.length} commands, ${guardrails.filePrefixes.length} file prefixes, ${flatEventLogChannels.length} event log channels`);
} else {
console.warn('guardrails.json not found — skipping guardrails generation');
}
@@ -56,6 +56,10 @@
PowerShell 5.1 (IME) reads scripts without BOM as ANSI, corrupting multi-byte chars.
.CHANGELOG
2026-07-20 v2.2 Switched default download endpoint from the legacy blob
(autopilotmonitor.blob.core.windows.net) to the Front Door alias
https://download.autopilotmonitor.com/agent/ (route /agent/*).
version.json is still derived from the same container.
2026-06-25 v2.1 OOBE-relax: detect genuine OOBE via WinRT
SystemSetupInfo.OutOfBoxExperienceState and exempt guards 2+3 when a
single freshly-created profile appears in OOBE (Windows Backup for
@@ -80,7 +84,7 @@
[CmdletBinding()]
param(
[Parameter(Mandatory = $false)]
[string]$AgentDownloadUrl = "https://autopilotmonitor.blob.core.windows.net/agent/AutopilotMonitor-Agent.zip",
[string]$AgentDownloadUrl = "https://download.autopilotmonitor.com/agent/AutopilotMonitor-Agent.zip",
[Parameter(Mandatory = $false)]
[string]$VersionJsonName = "version.json",
@@ -93,7 +97,7 @@ param(
)
# Script version (bump on meaningful changes; see .CHANGELOG above)
$ScriptVersion = "2.1"
$ScriptVersion = "2.2"
# Configuration - Everything in ProgramData for easy cleanup
$AgentBasePath = "$env:ProgramData\AutopilotMonitor"
@@ -268,13 +272,16 @@ try {
$downloadAttempt++
try {
Write-Log "Download attempt ${downloadAttempt}/${maxDownloadAttempts}"
$downloadStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Invoke-WebRequest `
-Uri $AgentDownloadUrl `
-OutFile $zipPath `
-UseBasicParsing `
-TimeoutSec 30 `
-ErrorAction Stop
Write-Log "Downloaded agent to $zipPath"
$downloadStopwatch.Stop()
$downloadSeconds = [math]::Round($downloadStopwatch.Elapsed.TotalSeconds, 1)
Write-Log "Downloaded agent to $zipPath (took ${downloadSeconds}s)"
break
}
catch {
@@ -0,0 +1,99 @@
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
{
/// <summary>
/// Pins the agent-side endpoint-migration contract (<see cref="EndpointMigration"/>):
/// only a LIVE fetch is honoured, the target must pass the shared allowlist rules even
/// though the backend already validated it, and serving the agent its own current URL is
/// a no-op — the exact mirror of the kill-switch live-fetch hardening.
/// </summary>
public sealed class EndpointMigrationTests
{
private const string CurrentUrl = "https://autopilotmonitor-api-eu.azurewebsites.net";
private const string TargetUrl = "https://autopilotmonitor-api-us.azurewebsites.net";
private static AgentConfigResponse Config(string? migrateUrl) =>
new AgentConfigResponse { MigrateToApiBaseUrl = migrateUrl };
[Fact]
public void Live_fetch_with_valid_differing_target_migrates()
{
var target = EndpointMigration.ResolveTarget(
Config(TargetUrl), RemoteConfigFetchOutcome.Succeeded, CurrentUrl, logger: null);
Assert.Equal(TargetUrl, target);
}
[Fact]
public void Target_is_normalized_to_lowercase_host_without_trailing_slash()
{
var target = EndpointMigration.ResolveTarget(
Config("https://AutopilotMonitor-API-US.azurewebsites.net/"),
RemoteConfigFetchOutcome.Succeeded, CurrentUrl, logger: null);
Assert.Equal(TargetUrl, target);
}
[Theory]
[InlineData(RemoteConfigFetchOutcome.FromCache)]
[InlineData(RemoteConfigFetchOutcome.UsedDefaults)]
[InlineData(RemoteConfigFetchOutcome.NotAttempted)]
public void Non_live_fetch_is_never_honoured(RemoteConfigFetchOutcome outcome)
{
Assert.Null(EndpointMigration.ResolveTarget(
Config(TargetUrl), outcome, CurrentUrl, logger: null));
}
[Fact]
public void Null_config_and_empty_signal_are_noops()
{
Assert.Null(EndpointMigration.ResolveTarget(
null, RemoteConfigFetchOutcome.Succeeded, CurrentUrl, logger: null));
Assert.Null(EndpointMigration.ResolveTarget(
Config(null), RemoteConfigFetchOutcome.Succeeded, CurrentUrl, logger: null));
Assert.Null(EndpointMigration.ResolveTarget(
Config(" "), RemoteConfigFetchOutcome.Succeeded, CurrentUrl, logger: null));
}
[Fact]
public void Target_equal_to_current_url_is_a_noop()
{
// Steady state during a migration window: agent already runs on the target
// (case/trailing-slash differences must not cause a rebuild loop).
Assert.Null(EndpointMigration.ResolveTarget(
Config(CurrentUrl), RemoteConfigFetchOutcome.Succeeded, CurrentUrl, logger: null));
Assert.Null(EndpointMigration.ResolveTarget(
Config(CurrentUrl.ToUpperInvariant() + "/"), RemoteConfigFetchOutcome.Succeeded, CurrentUrl, logger: null));
}
[Theory]
[InlineData("http://autopilotmonitor-api-us.azurewebsites.net")] // https only
[InlineData("https://attacker.example.com")] // host not allowlisted
[InlineData("https://evil-azurewebsites.net")] // suffix must sit on a label boundary
[InlineData("https://.azurewebsites.net")] // empty host prefix
[InlineData("https://api.azurewebsites.net:8443")] // non-default port
[InlineData("https://user:pw@api.azurewebsites.net")] // userinfo
[InlineData("https://api.azurewebsites.net/some/path")] // base URL only
[InlineData("https://api.azurewebsites.net?x=1")] // no query
[InlineData("not a url")]
public void Invalid_targets_are_rejected(string candidate)
{
Assert.Null(EndpointMigration.ResolveTarget(
Config(candidate), RemoteConfigFetchOutcome.Succeeded, CurrentUrl, logger: null));
}
[Fact]
public void Dev_api_url_override_still_migrates_to_valid_target()
{
// The running base URL may be a non-allowlisted dev override (--api-url); the
// comparison falls back to plain normalization and the migration still applies.
var target = EndpointMigration.ResolveTarget(
Config(TargetUrl), RemoteConfigFetchOutcome.Succeeded, "http://localhost:7071", logger: null);
Assert.Equal(TargetUrl, target);
}
}
}
@@ -62,6 +62,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
DeviceBlocked = true,
DeviceKillSignal = true,
UnblockAt = new DateTime(2099, 1, 1, 0, 0, 0, DateTimeKind.Utc),
MigrateToApiBaseUrl = "https://attacker.azurewebsites.net",
};
probe.DoCache(config);
@@ -69,6 +70,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// Read the file the production code actually wrote and deserialize with the same lib.
var raw = File.ReadAllText(_cachePath);
var persisted = JsonConvert.DeserializeObject<AgentConfigResponse>(raw);
Assert.NotNull(persisted);
Assert.False(persisted.UnrestrictedMode);
Assert.False(persisted.AllowAgentDowngrade);
@@ -76,9 +78,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.False(persisted.DeviceBlocked);
Assert.False(persisted.DeviceKillSignal);
Assert.Null(persisted.UnblockAt);
Assert.Null(persisted.MigrateToApiBaseUrl);
// The attacker EXE hash must not appear anywhere in the persisted bytes.
// The attacker EXE hash / re-home host must not appear anywhere in the persisted bytes.
Assert.DoesNotContain("deadbeef", raw);
Assert.DoesNotContain("attacker.azurewebsites.net", raw);
// Benign fields survive the strip.
Assert.Equal(77, persisted.ConfigVersion);
@@ -92,6 +96,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.True(config.DeviceBlocked);
Assert.True(config.DeviceKillSignal);
Assert.Equal(new DateTime(2099, 1, 1, 0, 0, 0, DateTimeKind.Utc), config.UnblockAt);
Assert.Equal("https://attacker.azurewebsites.net", config.MigrateToApiBaseUrl);
}
// ── Read path: LoadCachedConfig strips a planted (attacker) cache file ─
@@ -103,6 +108,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
[InlineData("DeviceBlocked")]
[InlineData("DeviceKillSignal")]
[InlineData("UnblockAt")]
[InlineData("MigrateToApiBaseUrl")]
public void LoadCachedConfig_strips_planted_security_sensitive_field(string field)
{
var probe = NewProbe();
@@ -119,6 +125,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
DeviceBlocked = true,
DeviceKillSignal = true,
UnblockAt = new DateTime(2099, 1, 1, 0, 0, 0, DateTimeKind.Utc),
MigrateToApiBaseUrl = "https://attacker.azurewebsites.net",
};
File.WriteAllText(_cachePath, JsonConvert.SerializeObject(attacker, Formatting.Indented));
@@ -145,6 +152,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
case "UnblockAt":
Assert.Null(loaded.UnblockAt);
break;
case "MigrateToApiBaseUrl":
Assert.Null(loaded.MigrateToApiBaseUrl);
break;
default:
throw new InvalidOperationException($"unknown field {field}");
}
@@ -193,7 +193,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
case BackendBehaviour.AlwaysAuthFailure401:
throw new BackendAuthException("Unauthorized", 401);
case BackendBehaviour.AlwaysReturnNull:
return Task.FromResult<AgentConfigResponse>(null);
return Task.FromResult<AgentConfigResponse>(null!);
default:
throw new InvalidOperationException("unknown behaviour");
}
@@ -204,9 +204,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// In-memory + per-instance cache so a prior test's successful fetch can't
// bleed into a later test that expects UsedDefaults.
private AgentConfigResponse _cached;
private AgentConfigResponse? _cached;
protected override void CacheConfig(AgentConfigResponse config) { _cached = config; }
protected override AgentConfigResponse LoadCachedConfig() => _cached;
protected override AgentConfigResponse LoadCachedConfig() => _cached!;
}
}
}
@@ -0,0 +1,157 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
using SharedEventTypes = AutopilotMonitor.Shared.Constants.EventTypes;
using DeviceInfoCollector = AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo.DeviceInfoCollector;
namespace AutopilotMonitor.Agent.V2.Core.Tests.DeviceInfo
{
/// <summary>
/// Audit Q2 — <see cref="DeviceInfoCollector.RefreshEspConfiguration"/>
/// re-reads the ESP blocking lists and re-emits <c>esp_config_detected</c> exactly when the
/// payload changed: the registry lists grow progressively (one timestamped subkey per CSP
/// status write) and the user-scope lists appear only after sign-in, so the early emissions
/// are structurally partial. The StartupEventGate keeps an unchanged re-read silent.
/// Probe overrides are static slots → serialized with the other probe-driven suites.
/// </summary>
[Collection("SerialThreading")]
public sealed class DeviceInfoCollectorEspRefreshTests
{
private static readonly DateTime T0 = new DateTime(2026, 7, 27, 10, 0, 0, DateTimeKind.Utc);
private const string DeviceApp1 = "11111111-1111-1111-1111-111111111111";
private const string DeviceApp2 = "22222222-2222-2222-2222-222222222222";
private const string UserApp1 = "33333333-3333-3333-3333-333333333333";
private static EspTrackingInfoSnapshot Snapshot(string[] deviceApps, string[] userApps)
{
var all = deviceApps.Concat(userApps).Distinct().ToArray();
return new EspTrackingInfoSnapshot(
msiProductCodes: Array.Empty<string>(),
modernAppPfns: Array.Empty<string>(),
win32AppIds: all,
userWin32AppIds: userApps,
msiCount: 0,
modernCount: 0,
win32Count: all.Length);
}
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> EspConfigEvents(FakeSignalIngressSink ingress) =>
ingress.Posted
.Where(p => p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == SharedEventTypes.EspConfigDetected)
.ToList();
private static IReadOnlyList<string> TypedList(FakeSignalIngressSink.PostedSignal signal, string key)
{
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(signal.TypedPayload);
return Assert.IsAssignableFrom<IReadOnlyList<string>>(data[key]);
}
[Fact]
public void Refresh_reemits_when_lists_grew_and_stays_silent_when_unchanged()
{
using var tmp = new TempDirectory();
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
var ingress = new FakeSignalIngressSink();
var clock = new VirtualClock(T0);
var collector = new DeviceInfoCollector(
"session-1", "tenant-1",
new InformationalEventPost(ingress, clock), logger,
signalIngress: null, clock: clock,
startupGate: new StartupEventGate(tmp.Path, logger));
using var skip = new EspSkipConfigurationProbe.ScopedFullOverride(
_ => new EspFirstSyncSnapshot(
skipUser: false, skipDevice: false,
blockInStatusPage: null, syncFailureTimeoutMinutes: null));
// Early read (agent start / DeviceSetup detection): partial device list, empty user list.
using (new EspTrackingInfoProbe.ScopedOverride(_ => Snapshot(new[] { DeviceApp1 }, Array.Empty<string>())))
{
collector.CollectEspConfiguration();
}
var afterEarly = EspConfigEvents(ingress);
Assert.Single(afterEarly);
Assert.Empty(TypedList(afterEarly[0], "espTrackedUserWin32AppIds"));
// Apps sub-phase opened: the device list grew and sign-in populated the user list.
using (new EspTrackingInfoProbe.ScopedOverride(_ => Snapshot(new[] { DeviceApp1, DeviceApp2 }, new[] { UserApp1 })))
{
collector.RefreshEspConfiguration(DeviceInfoHost.EspConfigTriggerAppsUser);
}
var afterGrowth = EspConfigEvents(ingress);
Assert.Equal(2, afterGrowth.Count);
Assert.Contains(DeviceApp2, TypedList(afterGrowth[1], "espTrackedWin32AppIds"));
Assert.Equal(new[] { UserApp1 }, TypedList(afterGrowth[1], "espTrackedUserWin32AppIds"));
// A later trigger with an UNCHANGED registry state re-reads but emits nothing —
// the StartupEventGate suppresses the identical payload.
using (new EspTrackingInfoProbe.ScopedOverride(_ => Snapshot(new[] { DeviceApp1, DeviceApp2 }, new[] { UserApp1 })))
{
collector.RefreshEspConfiguration(DeviceInfoHost.EspConfigTriggerAccountSetup);
}
Assert.Equal(2, EspConfigEvents(ingress).Count);
}
[Fact]
public async Task Concurrent_refreshes_never_emit_an_older_snapshot_after_a_newer_one()
{
// Codex review: the refresh triggers each run on their own Task.Run — without the
// collector-side lock, two collects can interleave read-before-emit and the LATER
// emission carries the OLDER (smaller) list. The backend keeps the latest
// list-bearing emission as positive evidence, so that would silently shrink it.
// The probe returns a strictly growing list per read; with the read→emit critical
// section, emission order must equal read order — sizes strictly ascending.
using var tmp = new TempDirectory();
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
var ingress = new FakeSignalIngressSink();
var clock = new VirtualClock(T0);
var collector = new DeviceInfoCollector(
"session-1", "tenant-1",
new InformationalEventPost(ingress, clock), logger,
signalIngress: null, clock: clock,
startupGate: new StartupEventGate(tmp.Path, logger));
using var skip = new EspSkipConfigurationProbe.ScopedFullOverride(
_ => new EspFirstSyncSnapshot(
skipUser: false, skipDevice: false,
blockInStatusPage: null, syncFailureTimeoutMinutes: null));
var readCounter = 0;
using var tracking = new EspTrackingInfoProbe.ScopedOverride(_ =>
{
var n = Interlocked.Increment(ref readCounter);
var ids = Enumerable.Range(1, n)
.Select(i => $"{i:d8}-1111-2222-3333-444444444444")
.ToArray();
return Snapshot(ids, Array.Empty<string>());
});
var tasks = Enumerable.Range(0, 12)
.Select(i => Task.Run(() => collector.RefreshEspConfiguration($"stress-{i}")))
.ToArray();
await Task.WhenAll(tasks);
var sizes = EspConfigEvents(ingress)
.Select(e => TypedList(e, "espTrackedWin32AppIds").Count)
.ToList();
// Every refresh saw a strictly larger list (gate never suppresses) and each
// read→emit is atomic, so the emission sequence is exactly 1..12.
Assert.Equal(Enumerable.Range(1, 12), sizes);
}
}
}
@@ -39,6 +39,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Harness
logFolder: Tmp.Path,
patterns: new List<ImeLogPattern>(),
logger: Logger);
// Mirror ImeLogHost wiring: the historic-replay guard judges staleness against the
// (virtual) agent clock so tests stay deterministic.
Tracker.UtcNowProvider = () => Clock.UtcNow;
}
public IReadOnlyList<FakeSignalIngressSink.PostedSignal> DecisionSignals(DecisionSignalKind kind) =>
@@ -51,7 +51,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Integration
logger: logger,
helloTracker: null);
EspExitedEventArgs captured = null;
EspExitedEventArgs? captured = null;
tracker.EspExited += (_, args) => captured = args;
// Source time IS in the past relative to the clock — verify the tracker carries the
@@ -0,0 +1,71 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Analyzers
{
/// <summary>
/// LocalAdminAnalyzer.MatchesAllowedEntry — the allowed-list matching contract.
/// Entries without wildcard characters match exactly (case-insensitive); entries
/// containing * or ? are glob patterns with the same semantics as the hardware
/// whitelist. The same matcher is applied to local accounts and profile folders.
/// </summary>
public sealed class LocalAdminAnalyzerTests
{
// -------------------------------------------------------------- literal entries
[Theory]
[InlineData("SupportAdmin", "SupportAdmin")]
[InlineData("supportadmin", "SupportAdmin")]
[InlineData("SUPPORTADMIN", "supportadmin")]
public void Literal_entry_matches_exactly_case_insensitive(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("SupportAdmin2", "SupportAdmin")]
[InlineData("SupportAdmin", "SupportAdmin2")]
[InlineData("Support", "SupportAdmin")]
public void Literal_entry_does_not_substring_match(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
// -------------------------------------------------------------- glob patterns
[Theory]
[InlineData("adm-12345", "adm-*")]
[InlineData("adm-", "adm-*")]
[InlineData("ADM-9F3B2C", "adm-*")]
[InlineData("lapsAdmin01", "lapsAdmin??")]
[InlineData("svc-backup-01", "svc-*-01")]
public void Wildcard_entry_matches_glob(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("administrator2", "adm-*")] // no literal "adm-" prefix
[InlineData("xadm-123", "adm-*")] // pattern is anchored at start
[InlineData("lapsAdmin1", "lapsAdmin??")] // ? is exactly one char
[InlineData("adm.123", "adm-*")] // "-" is literal, not a regex class
public void Wildcard_entry_does_not_overmatch(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Fact]
public void Regex_metacharacters_in_entry_are_treated_literally()
{
// "adm.123" as an entry has no wildcard chars → exact match only,
// the dot must not act as a regex any-char.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.123"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.123"));
// Mixed: wildcard present, remaining metacharacters stay literal.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.*"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.*"));
}
}
}
@@ -405,5 +405,144 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
var started = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorStarted);
Assert.Equal(0, started.Data["noCandidateThresholdMinutes"]);
}
// ============================================================================
// OOBE-state flip (oobe_state_completed) — observational one-shot on the poll
// ============================================================================
// The WinRT OutOfBoxExperienceState flip in_progress->completed is an owner-
// independent desktop corroboration for DAD-starved sessions (4d5a0b78 class).
// Tests drive polls synchronously via PollOnceForTest after Stop() so the real
// timer cannot race, and pin the owner seams to an excluded user so a real
// explorer.exe on the test machine can never set _desktopArrived (which would
// short-circuit PollForDesktop before the flip check).
private static DesktopArrivalDetector BuildFlipDetector(TempDirectory tmp, TraceCapture trace, System.Func<string> oobeProvider)
{
var detector = new DesktopArrivalDetector(new AgentLogger(tmp.Path, AgentLogLevel.Info), noCandidateTimeoutMinutes: 0)
{
OnTraceEvent = trace.Sink,
OobeStateProvider = oobeProvider,
SessionOwnerResolver = _ => "SYSTEM",
ProcessOwnerResolver = _ => "SYSTEM",
};
return detector;
}
/// <summary>Returns the queued states in order, then repeats the last one forever.</summary>
private static System.Func<string> StateQueue(params string[] states)
{
var queue = new Queue<string>(states);
var last = states[states.Length - 1];
return () => queue.Count > 0 ? queue.Dequeue() : last;
}
[Fact]
public void OobeFlip_emits_exactly_once_with_payload()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
// Baseline sample happens synchronously inside Start()/ArmTimer.
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // reads "completed" -> flip
detector.PollOnceForTest(); // still "completed" -> no second emit
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
var flip = trace.Events.Find(e => e.EventType == SharedEventTypes.OobeStateCompleted);
Assert.Equal("in_progress", flip.Data["previousState"]);
Assert.True(flip.Data.ContainsKey("pollCount"));
Assert.True(flip.Data.ContainsKey("minutesSinceStart"));
Assert.False((bool)flip.Data["desktopArrived"]);
Assert.True(flip.Data.ContainsKey("wmiErrorCount"));
Assert.True(flip.Data.ContainsKey("wtsErrorCount"));
}
[Fact]
public void OobeFlip_does_not_emit_when_state_is_already_completed()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest();
detector.PollOnceForTest();
// Productive machines read "completed" from the very first sample — the flip
// event must never fire there (the agent_started field already records the state).
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_does_not_emit_when_in_progress_was_never_seen()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("unavailable", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest();
// Old builds without the WinRT contract report "unavailable"; a later
// "completed" without a positively observed "in_progress" is not a flip.
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_survives_transient_unavailable_between_polls()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, StateQueue("in_progress", "unavailable", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // transient reflection failure — must NOT clear the baseline
detector.PollOnceForTest(); // "completed" -> flip still detected
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
[Fact]
public void OobeFlip_provider_throwing_never_emits_and_never_pollutes_wmi_error_count()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace, () => throw new System.InvalidOperationException("boom"));
detector.Start(); // baseline sample throws -> swallowed
detector.Stop();
detector.PollOnceForTest();
detector.PollOnceForTest();
Assert.Equal(0, trace.CountOf(SharedEventTypes.OobeStateCompleted));
// The flip check has its own try/catch — a throw must not fall into
// PollForDesktop's outer catch, which increments the WMI error counter the
// session-4d5a0b78 liveness diagnostics rely on. first_poll carries that counter.
var firstPoll = trace.Events.Find(e => e.EventType == SharedEventTypes.DesktopDetectorFirstPoll);
Assert.Equal(0, firstPoll.Data["wmiErrorCount"]);
}
[Fact]
public void OobeFlip_one_shot_flag_survives_ResetForRealUserSwitch()
{
using var tmp = new TempDirectory();
var trace = new TraceCapture();
using var detector = BuildFlipDetector(tmp, trace,
StateQueue("in_progress", "completed", "in_progress", "completed"));
detector.Start();
detector.Stop();
detector.PollOnceForTest(); // flip #1
detector.ResetForRealUserSwitch(); // re-arms + re-samples baseline ("in_progress")
detector.Stop();
detector.PollOnceForTest(); // "completed" again — must NOT re-emit
Assert.Equal(1, trace.CountOf(SharedEventTypes.OobeStateCompleted));
}
}
}
@@ -0,0 +1,417 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
{
/// <summary>
/// Allowlist enforcement for the two collectors that used to bypass it:
/// <see cref="LogParserCollector"/> read any path the rule named — including
/// C:\Users, whose hard block every other file collector honours — and
/// <see cref="EventLogCollector"/> read any channel, including Security.
/// A tenant admin can author both through the rules API, so the agent is the
/// only place this can be enforced.
/// </summary>
public sealed class GatherRuleCollectorGuardTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
private readonly List<EnrollmentEvent> _events = new List<EnrollmentEvent>();
// %TEMP% itself lives under C:\Users, which is hard-blocked — the positive
// cases need a directory the guard can actually admit, so they run beside
// the test assembly instead.
private readonly string _outsideUsers = Path.Combine(
AppContext.BaseDirectory, "guard-tests-" + Guid.NewGuid().ToString("N"));
public GatherRuleCollectorGuardTests() => Directory.CreateDirectory(_outsideUsers);
public void Dispose()
{
_tmp.Dispose();
try { Directory.Delete(_outsideUsers, recursive: true); } catch { /* best effort */ }
}
private GatherRuleContext Context(bool unrestricted = false, string? imeLogPathOverride = null)
=> new GatherRuleContext(
new AgentLogger(_tmp.Path, AgentLogLevel.Info),
"sess", "tenant",
evt => _events.Add(evt),
imeLogPathOverride,
new LogFilePositionTracker())
{
UnrestrictedMode = unrestricted
};
private static GatherRule Rule(string collectorType, string target,
Dictionary<string, string>? parameters = null) => new GatherRule
{
RuleId = "GATHER-GUARD-TEST",
Title = "guard test",
CollectorType = collectorType,
Target = target,
// Match the model default (empty, never null) so the collectors see the
// same shape a deserialized rule without parameters gives them.
Parameters = parameters ?? new Dictionary<string, string>(),
Trigger = "startup",
OutputEventType = "guard_test",
};
private EnrollmentEvent SingleSecurityWarning()
=> Assert.Single(_events, e => e.EventType == Constants.EventTypes.SecurityWarning);
// -------------------------------------------------------------------
// Event log channel allowlist
// -------------------------------------------------------------------
[Theory]
[InlineData("System")]
[InlineData("Application")]
[InlineData("Setup")]
[InlineData("Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin")]
[InlineData("Microsoft-Windows-AAD/Operational")]
[InlineData("Microsoft-Windows-Shell-Core/Operational")]
public void EventLogChannel_OnAllowlist_IsAllowed(string channel)
=> Assert.True(GatherRuleGuards.IsEventLogChannelAllowed(channel));
[Theory]
[InlineData("Security")]
[InlineData("Microsoft-Windows-PowerShell/Operational")]
[InlineData("Windows PowerShell")]
[InlineData("Microsoft-Windows-Sysmon/Operational")]
public void EventLogChannel_HardBlocked_StaysBlockedEvenUnrestricted(string channel)
{
Assert.False(GatherRuleGuards.IsEventLogChannelAllowed(channel));
Assert.False(GatherRuleGuards.IsEventLogChannelAllowed(channel, unrestrictedMode: true));
}
[Theory]
[InlineData("Microsoft-Windows-AADSomethingElse/Operational")] // prefix spoofing
[InlineData("Microsoft-Windows-TerminalServices-LocalSessionManager/Operational")]
[InlineData("")]
public void EventLogChannel_OffAllowlist_IsBlocked(string channel)
=> Assert.False(GatherRuleGuards.IsEventLogChannelAllowed(channel));
[Fact]
public void EventLogChannel_Null_IsBlocked()
=> Assert.False(GatherRuleGuards.IsEventLogChannelAllowed(null));
[Fact]
public void EventLogChannel_OffAllowlist_IsAllowedInUnrestrictedMode()
=> Assert.True(GatherRuleGuards.IsEventLogChannelAllowed(
"Microsoft-Windows-TerminalServices-LocalSessionManager/Operational",
unrestrictedMode: true));
[Fact]
public void EventLogCollector_SecurityChannel_EmitsSecurityWarningAndReturnsNoData()
{
var result = new EventLogCollector().Execute(Rule("eventlog", "Security"), Context());
Assert.Empty(result);
var warning = SingleSecurityWarning();
Assert.Equal(true, warning.Data["blocked"]);
Assert.Equal("Security", warning.Data["target"]);
}
[Fact]
public void EventLogCollector_SecurityChannel_StaysBlockedInUnrestrictedMode()
{
var result = new EventLogCollector().Execute(
Rule("eventlog", "Security"), Context(unrestricted: true));
Assert.Empty(result);
SingleSecurityWarning();
}
// -------------------------------------------------------------------
// Log parser file allowlist
// -------------------------------------------------------------------
private static Dictionary<string, string> TextPattern() => new Dictionary<string, string>
{
["pattern"] = "(?<line>.+)",
["format"] = "text",
["trackPosition"] = "false",
};
[Fact]
public void LogParser_UserProfilePath_IsBlockedAndEmitsSecurityWarning()
{
const string target = @"C:\Users\Public\Documents\secret.txt";
var result = new LogParserCollector().Execute(
Rule("logparser", target, TextPattern()), Context());
Assert.Null(result);
var warning = SingleSecurityWarning();
Assert.Equal(target, warning.Data["target"]);
Assert.DoesNotContain(_events, e => e.EventType == "guard_test");
}
[Fact]
public void LogParser_UserProfilePath_StaysBlockedInUnrestrictedMode()
{
// C:\Users is a hard block — unrestricted mode must not lift it.
new LogParserCollector().Execute(
Rule("logparser", @"C:\Users\Public\Documents\secret.txt", TextPattern()),
Context(unrestricted: true));
SingleSecurityWarning();
}
[Fact]
public void LogParser_OffAllowlistPath_IsBlockedEvenWhenFileExists()
{
var path = _tmp.File("evidence.log");
File.WriteAllText(path, "line one\nline two\n");
var result = new LogParserCollector().Execute(
Rule("logparser", path, TextPattern()), Context());
Assert.Null(result);
SingleSecurityWarning();
Assert.DoesNotContain(_events, e => e.EventType == "guard_test");
}
[Fact]
public void LogParser_WildcardTarget_IsBlockedByItsDirectory()
{
// Path.GetFullPath rejects '*', so the guard has to check the directory.
File.WriteAllText(_tmp.File("evidence.log"), "line one\n");
new LogParserCollector().Execute(
Rule("logparser", Path.Combine(_tmp.Path, "*.log"), TextPattern()), Context());
SingleSecurityWarning();
Assert.DoesNotContain(_events, e => e.EventType == "guard_test");
}
[Fact]
public void LogParser_UnrestrictedMode_ReadsPathOffTheAllowlist()
{
var path = Path.Combine(_outsideUsers, "evidence.log");
File.WriteAllText(path, "line one\nline two\n");
new LogParserCollector().Execute(
Rule("logparser", path, TextPattern()), Context(unrestricted: true));
Assert.DoesNotContain(_events, e => e.EventType == Constants.EventTypes.SecurityWarning);
Assert.Equal(2, _events.Count(e => e.EventType == "guard_test"));
}
[Fact]
public void LogParser_LocalImeLogPathOverride_RelaxesTheAllowlist()
{
// --ime-log-path is a local CLI flag, never remote config: the operator
// using it is already a local admin, so it relaxes the allowlist the way
// unrestricted mode does.
File.WriteAllText(Path.Combine(_outsideUsers, "AppWorkload.log"), "line one\n");
new LogParserCollector().Execute(
Rule("logparser", @"C:\ProgramData\Microsoft\IntuneManagementExtension\Logs\AppWorkload.log",
TextPattern()),
Context(imeLogPathOverride: _outsideUsers));
Assert.DoesNotContain(_events, e => e.EventType == Constants.EventTypes.SecurityWarning);
Assert.Single(_events, e => e.EventType == "guard_test");
}
// -------------------------------------------------------------------
// Command execution
// -------------------------------------------------------------------
[Fact]
public void Command_OffAllowlist_IsBlockedAndNeverRuns()
{
var result = new CommandCollector().Execute(
Rule("command_allowlisted", "Get-Content C:\\Windows\\System32\\config\\SAM"), Context());
Assert.Equal(true, result["blocked"]);
Assert.False(result.ContainsKey("output"));
SingleSecurityWarning();
}
[Fact]
public void Command_HardBlockedPattern_StaysBlockedInUnrestrictedMode()
{
var result = new CommandCollector().Execute(
Rule("command_allowlisted", "Invoke-WebRequest https://example.invalid/x"),
Context(unrestricted: true));
Assert.Equal(true, result["blocked"]);
SingleSecurityWarning();
}
[Fact]
public void Command_ThatHangs_IsKilledAtTheTimeoutRatherThanPinningTheWorker()
{
// Regression: reading stdout synchronously blocked until the pipe closed,
// so WaitForExit's timeout was never reached and a hung command pinned the
// worker indefinitely. Unrestricted mode is needed to run an ad-hoc command.
var sw = System.Diagnostics.Stopwatch.StartNew();
var result = new CommandCollector().Execute(
Rule("command_allowlisted", "Start-Sleep -Seconds 120"), Context(unrestricted: true));
sw.Stop();
Assert.Equal(true, result["timed_out"]);
Assert.Equal(-1, result["exit_code"]);
// The 30s timeout plus drain margin — nowhere near the command's 120s.
Assert.True(sw.Elapsed < TimeSpan.FromSeconds(60),
$"expected the hung command to be killed at the timeout; took {sw.Elapsed}");
}
// -------------------------------------------------------------------
// Every remaining collector actually calls its guard
//
// The guard functions have their own unit tests, but those prove nothing about
// whether a collector consults them before touching the registry, WMI or disk.
// These drive the collectors themselves: each blocked target must come back empty,
// emit exactly one security_warning naming the target, and produce no rule output.
// -------------------------------------------------------------------
private void AssertBlocked(Dictionary<string, object> result, string target)
{
Assert.Empty(result);
var warning = SingleSecurityWarning();
Assert.Equal(true, warning.Data["blocked"]);
Assert.Equal(target, warning.Data["target"]);
Assert.DoesNotContain(_events, e => e.EventType == "guard_test");
}
[Theory]
[InlineData(@"SAM\SAM\Domains\Account\Users")]
[InlineData(@"SECURITY\Policy\Secrets")]
[InlineData(@"HKLM\SOFTWARE\Microsoft\EnrollmentsSomethingElse")] // prefix spoofing
[InlineData(@"HKLM\SOFTWARE\Microsoft\Windows\AutopilotSomethingElse")] // sibling spoof of the Autopilot prefix
public void RegistryCollector_OffAllowlistPath_IsBlockedBeforeTheHiveIsOpened(string target)
=> AssertBlocked(new RegistryCollector().Execute(Rule("registry", target), Context()), target);
// Session 5d735290: the EnrollmentStatusTracking subtree (ESP policy providers / tracking
// state) is allow-listed so tenants can build gather rules on the co-management stall
// signature the esp_policy_provider_stalled detector reports (e.g. not_exists on the
// Sidecar provider key).
[Theory]
[InlineData(@"SOFTWARE\Microsoft\Windows\Autopilot")]
[InlineData(@"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking\Device\Setup\Apps\PolicyProviders\Sidecar")]
public void RegistryGuard_AutopilotEnrollmentTrackingSubtree_IsAllowed(string subPath)
=> Assert.True(GatherRuleGuards.IsRegistryPathAllowed(subPath));
[Theory]
[InlineData("SELECT * FROM Win32_UserAccount")]
[InlineData("SELECT * FROM Win32_ShadowCopy")]
[InlineData("SELECT * FROM Win32_OperatingSystemExtra")] // prefix spoofing
public void WmiCollector_OffAllowlistQuery_IsBlockedBeforeTheQueryRuns(string query)
=> AssertBlocked(new WmiCollector().Execute(Rule("wmi", query), Context()), query);
[Fact]
public void JsonCollector_UserProfilePath_IsBlockedBeforeTheFileIsOpened()
{
const string target = @"C:\Users\Public\Documents\secrets.json";
AssertBlocked(
new JsonCollector().Execute(
Rule("json", target, new Dictionary<string, string> { ["jsonpath"] = "$.token" }),
Context()),
target);
}
[Fact]
public void XmlCollector_UserProfilePath_IsBlockedBeforeTheFileIsOpened()
{
const string target = @"C:\Users\Public\Documents\secrets.xml";
AssertBlocked(
new XmlCollector().Execute(
Rule("xml", target, new Dictionary<string, string> { ["xpath"] = "//token" }),
Context()),
target);
}
[Fact]
public void FileCollector_UserProfilePath_IsBlockedBeforeTheFileIsOpened()
{
const string target = @"C:\Users\Public\Documents\secrets.txt";
AssertBlocked(new FileCollector().Execute(Rule("file", target), Context()), target);
}
[Fact]
public void JsonCollector_AllowedPath_ReadsTheFile()
{
// Counterpart to the blocked cases: proves the guard is a gate, not a wall —
// the collector does reach the file once the path is admitted.
var path = Path.Combine(_outsideUsers, "inventory.json");
File.WriteAllText(path, "{ \"agent\": { \"version\": \"2.4.0\" } }");
var result = new JsonCollector().Execute(
Rule("json", path, new Dictionary<string, string> { ["jsonpath"] = "$.agent.version" }),
Context(unrestricted: true));
Assert.DoesNotContain(_events, e => e.EventType == Constants.EventTypes.SecurityWarning);
Assert.Contains("2.4.0", string.Join(",", result.Values));
}
// -------------------------------------------------------------------
// Path traversal
// -------------------------------------------------------------------
[Fact]
public void FilePath_TraversalOutOfAnAllowedPrefix_IsBlocked()
{
// The raw string starts with an allowed prefix, so a naive StartsWith check would
// admit it. Path.GetFullPath runs FIRST, so the allowlist sees C:\Secret.txt.
const string traversal =
@"C:\ProgramData\Microsoft\IntuneManagementExtension\Logs\..\..\..\..\Secret.txt";
Assert.Equal(@"C:\Secret.txt", Path.GetFullPath(traversal));
Assert.False(GatherRuleGuards.IsFilePathAllowed(traversal));
}
[Fact]
public void FilePath_TraversalIntoTheUsersHardBlock_StaysBlockedEvenUnrestricted()
{
// Unrestricted mode lifts the allowlist but never the C:\Users privacy block —
// and it must not be reachable by spelling the path with '..' either.
const string traversal =
@"C:\ProgramData\Microsoft\IntuneManagementExtension\Logs\..\..\..\..\Users\Public\ntuser.dat";
Assert.StartsWith(@"C:\Users\", Path.GetFullPath(traversal), StringComparison.OrdinalIgnoreCase);
Assert.False(GatherRuleGuards.IsFilePathAllowed(traversal));
Assert.False(GatherRuleGuards.IsFilePathAllowed(traversal, unrestrictedMode: true));
}
[Fact]
public void FilePath_TraversalIntoTheConfigHiveDirectory_StaysBlockedEvenUnrestricted()
{
const string traversal = @"C:\Windows\Logs\..\System32\config\SAM";
Assert.False(GatherRuleGuards.IsFilePathAllowed(traversal));
Assert.False(GatherRuleGuards.IsFilePathAllowed(traversal, unrestrictedMode: true));
}
[Fact]
public void FileCollector_TraversalTarget_IsBlockedAndReported()
{
const string traversal =
@"C:\ProgramData\Microsoft\IntuneManagementExtension\Logs\..\..\..\..\Users\Public\ntuser.dat";
AssertBlocked(new FileCollector().Execute(Rule("file", traversal), Context()), traversal);
}
[Fact]
public void LogParser_ImeLogPathOverride_StillHonoursTheUsersHardBlock()
{
new LogParserCollector().Execute(
Rule("logparser", @"C:\ProgramData\Microsoft\IntuneManagementExtension\Logs\AppWorkload.log",
TextPattern()),
Context(imeLogPathOverride: @"C:\Users\Public\Documents"));
SingleSecurityWarning();
}
}
}
@@ -0,0 +1,249 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
{
/// <summary>
/// EmitMode "on_change" on <see cref="GatherRuleExecutor"/>: poll on the trigger cadence but
/// emit only when the collected result changes. The dedup state machine is asserted
/// synchronously via the internal <c>ShouldEmitOnChange</c> / <c>ComputeCanonicalHash</c>
/// (no timer races); the ExecuteRule wiring (hash-before-injection, empty-result no-op,
/// emitOnlyIfExists composition) through the public startup path.
/// </summary>
[Collection("SerialThreading")] // startup rules execute on the shared ThreadPool
public sealed class GatherRuleExecutorEmitModeTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
private readonly List<EnrollmentEvent> _events = new List<EnrollmentEvent>();
private readonly object _eventsGate = new object();
private readonly GatherRuleExecutor _executor;
private const string AbsentPath =
"HKLM\\SOFTWARE\\AutopilotMonitorTests\\DefinitelyAbsent_7e3a1f4b-0000-0000-0000-000000000000";
public GatherRuleExecutorEmitModeTests()
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_executor = new GatherRuleExecutor(
"sess", "tenant",
evt => { lock (_eventsGate) _events.Add(evt); },
logger)
{
UnrestrictedMode = true
};
}
public void Dispose()
{
_executor.Dispose();
_tmp.Dispose();
}
private int EventCount { get { lock (_eventsGate) return _events.Count; } }
private EnrollmentEvent SingleEvent()
{
lock (_eventsGate) return Assert.Single(_events);
}
private bool WaitForEventCount(int count, int timeoutMs = 10000)
{
var sw = Stopwatch.StartNew();
while (sw.ElapsedMilliseconds < timeoutMs)
{
if (EventCount >= count) return true;
Thread.Sleep(25);
}
return EventCount >= count;
}
private static GatherRule OnChangeRule(string id = "GATHER-OC-001") => new GatherRule
{
RuleId = id,
Title = id,
CollectorType = "registry",
Target = AbsentPath,
Trigger = "interval",
IntervalSeconds = 3600,
OutputEventType = "gather_test",
Enabled = true,
EmitMode = "on_change",
};
private static Dictionary<string, object> Result(params (string Key, object Value)[] pairs)
{
var d = new Dictionary<string, object>();
foreach (var (key, value) in pairs) d[key] = value;
return d;
}
// ── on_change state machine (synchronous) ──────────────────────────
[Fact]
public void FirstResult_AlwaysEmits_ThenUnchangedIsSuppressed_ThenChangeReemits()
{
var rule = OnChangeRule();
Assert.True(_executor.ShouldEmitOnChange(rule, Result(("exists", false)))); // first
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false)))); // unchanged
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false)))); // unchanged
Assert.True(_executor.ShouldEmitOnChange(rule, Result(("exists", true), ("Version", "1")))); // changed
}
[Fact]
public void EmitAfterSuppressionStreak_CarriesSuppressedPollCount_AndResets()
{
var rule = OnChangeRule();
Assert.True(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
var changed = Result(("exists", true));
Assert.True(_executor.ShouldEmitOnChange(rule, changed));
Assert.Equal(3, changed["suppressedPolls"]);
Assert.True(changed.ContainsKey("suppressedSinceUtc"));
// Counter must reset: the next change emits without a suppression payload.
var changedAgain = Result(("exists", false));
Assert.True(_executor.ShouldEmitOnChange(rule, changedAgain));
Assert.False(changedAgain.ContainsKey("suppressedPolls"));
Assert.False(changedAgain.ContainsKey("suppressedSinceUtc"));
}
[Fact]
public void SuppressionState_IsIsolatedPerRule()
{
var ruleA = OnChangeRule("GATHER-OC-A");
var ruleB = OnChangeRule("GATHER-OC-B");
var payload = Result(("exists", false));
Assert.True(_executor.ShouldEmitOnChange(ruleA, new Dictionary<string, object>(payload)));
// Same content, different rule: B has no hash yet — must emit.
Assert.True(_executor.ShouldEmitOnChange(ruleB, new Dictionary<string, object>(payload)));
Assert.False(_executor.ShouldEmitOnChange(ruleA, new Dictionary<string, object>(payload)));
Assert.False(_executor.ShouldEmitOnChange(ruleB, new Dictionary<string, object>(payload)));
}
[Fact]
public void DedupState_SurvivesUpdateRulesRefresh()
{
var rule = OnChangeRule();
Assert.True(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
// Config refresh mid-session must not reset the last-emitted hash (that would
// re-emit an unchanged result on every refresh).
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
}
// ── Canonical hashing ──────────────────────────────────────────────
[Fact]
public void CanonicalHash_IsKeyOrderInsensitive()
{
var a = new Dictionary<string, object> { ["alpha"] = 1, ["beta"] = "x" };
var b = new Dictionary<string, object> { ["beta"] = "x", ["alpha"] = 1 };
Assert.Equal(
GatherRuleExecutor.ComputeCanonicalHash(a),
GatherRuleExecutor.ComputeCanonicalHash(b));
}
[Fact]
public void CanonicalHash_DetectsValueChanges_IncludingNested()
{
var baseline = new Dictionary<string, object>
{
["exists"] = true,
["entries"] = new List<object> { new Dictionary<string, object> { ["id"] = 1 } },
};
var changedScalar = new Dictionary<string, object>
{
["exists"] = false,
["entries"] = new List<object> { new Dictionary<string, object> { ["id"] = 1 } },
};
var changedNested = new Dictionary<string, object>
{
["exists"] = true,
["entries"] = new List<object> { new Dictionary<string, object> { ["id"] = 2 } },
};
var baseHash = GatherRuleExecutor.ComputeCanonicalHash(baseline);
Assert.NotEqual(baseHash, GatherRuleExecutor.ComputeCanonicalHash(changedScalar));
Assert.NotEqual(baseHash, GatherRuleExecutor.ComputeCanonicalHash(changedNested));
}
[Fact]
public void CanonicalHash_DistinguishesTypeShapes()
{
// "1" (string) vs 1 (int) and null vs "<null>"-like strings must not collide.
Assert.NotEqual(
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = "1" }),
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = 1 }));
Assert.NotEqual(
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = null! }),
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = "<null>" }));
}
// ── ExecuteRule wiring (async via startup path) ────────────────────
[Fact]
public void AlwaysMode_EmitsEveryCollection()
{
// EmitMode absent (legacy) — repeated identical results all emit. Two executions of
// the same collector via two rules sharing the target prove no dedup is applied.
var rule1 = OnChangeRule("GATHER-AL-1"); rule1.EmitMode = null; rule1.Trigger = "startup";
var rule2 = OnChangeRule("GATHER-AL-2"); rule2.EmitMode = null; rule2.Trigger = "startup";
_executor.UpdateRules(new List<GatherRule> { rule1, rule2 });
Assert.True(_executor.WaitForStartupRules(30));
Assert.True(WaitForEventCount(2));
}
[Fact]
public void OnChange_FirstInScopePoll_EmitsExistsFalse_AsPollingIndicator()
{
// The first result on an absent key IS the visible "we're polling now" indicator.
var rule = OnChangeRule();
rule.Trigger = "startup";
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.True(_executor.WaitForStartupRules(30));
Assert.True(WaitForEventCount(1));
var evt = SingleEvent();
Assert.Equal(false, evt.Data["exists"]);
Assert.Equal("GATHER-OC-001", evt.Data["ruleId"]); // injection happened after hashing
}
[Fact]
public void OnChange_ComposedWithEmitOnlyIfExists_EmptyResultDoesNotTouchHashState()
{
// emitOnlyIfExists miss → empty result → no emit AND no hash update: the eventual
// first real result must still count as "first emit".
var rule = OnChangeRule();
rule.Trigger = "startup";
rule.Parameters = new Dictionary<string, string> { ["emitOnlyIfExists"] = "true" };
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.True(_executor.WaitForStartupRules(30));
Thread.Sleep(300);
Assert.Equal(0, EventCount); // zero noise while the key is absent
// When the key appears (simulated via the state machine), the result emits as first.
Assert.True(_executor.ShouldEmitOnChange(rule, Result(("exists", true), ("Version", "1"))));
}
}
}
@@ -0,0 +1,246 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
{
/// <summary>
/// The <c>phase_exit</c> trigger: a one-shot collection when a phase is LEFT — the closing
/// bookend to <c>phase_change</c> (which fires on ENTER). The distinguishing invariant is that
/// both the TriggerPhase match and the phase-scope gate must evaluate against the phase being
/// left, i.e. before the executor's current-phase state moves forward.
/// Rules use the absent-registry-path pattern so no real system state is touched.
/// </summary>
[Collection("SerialThreading")] // rules execute on the shared ThreadPool
public sealed class GatherRuleExecutorPhaseExitTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
private readonly List<EnrollmentEvent> _events = new List<EnrollmentEvent>();
private readonly object _eventsGate = new object();
private readonly GatherRuleExecutor _executor;
private const string AbsentPath =
"HKLM\\SOFTWARE\\AutopilotMonitorTests\\DefinitelyAbsent_9c4d2a7f-0000-0000-0000-000000000000";
public GatherRuleExecutorPhaseExitTests()
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_executor = new GatherRuleExecutor(
"sess", "tenant",
evt => { lock (_eventsGate) _events.Add(evt); },
logger)
{
UnrestrictedMode = true
};
}
public void Dispose()
{
_executor.Dispose();
_tmp.Dispose();
}
private int EventCount { get { lock (_eventsGate) return _events.Count; } }
private bool WaitForEventCount(int count, int timeoutMs = 10000)
{
var sw = Stopwatch.StartNew();
while (sw.ElapsedMilliseconds < timeoutMs)
{
if (EventCount >= count) return true;
Thread.Sleep(25);
}
return EventCount >= count;
}
private static GatherRule ExitRule(
string id, string triggerPhase, List<string>? activePhases = null, string? fromPhase = null)
=> new GatherRule
{
RuleId = id,
Title = id,
CollectorType = "registry",
Target = AbsentPath,
Trigger = "phase_exit",
TriggerPhase = triggerPhase,
OutputEventType = "gather_test",
Enabled = true,
ActivePhases = activePhases,
ActiveFromPhase = fromPhase,
};
[Fact]
public void FiresOnceWhenTheNamedPhaseIsLeft_NotWhenItIsEntered()
{
_executor.UpdateRules(new List<GatherRule> { ExitRule("GATHER-PX-001", "AccountSetup") });
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Thread.Sleep(300);
Assert.Equal(0, EventCount);
// Entering AccountSetup must NOT fire an exit rule.
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Thread.Sleep(300);
Assert.Equal(0, EventCount);
// Leaving it does.
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser);
Assert.True(WaitForEventCount(1));
}
[Fact]
public void DoesNotRefireWhenThePhaseIsLeftTwice()
{
_executor.UpdateRules(new List<GatherRule> { ExitRule("GATHER-PX-002", "AccountSetup") });
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser);
Assert.True(WaitForEventCount(1));
// A phase can be re-entered (ESP re-reporting); the one-shot slot stays consumed.
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser);
Thread.Sleep(400);
Assert.Equal(1, EventCount);
}
[Fact]
public void EmptyTriggerPhase_FiresOnEveryExit()
{
_executor.UpdateRules(new List<GatherRule> { ExitRule("GATHER-PX-003", "") });
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup); // exit of Unknown — must NOT fire
Thread.Sleep(300);
Assert.Equal(0, EventCount);
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup); // exit of DeviceSetup
Assert.True(WaitForEventCount(1));
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser); // exit of AccountSetup
Assert.True(WaitForEventCount(2));
}
[Fact]
public void NeverFiresOnExitOfUnknown()
{
// Before the first phase signal there is no phase to leave.
_executor.UpdateRules(new List<GatherRule> { ExitRule("GATHER-PX-004", "") });
_executor.OnPhaseChanged(EnrollmentPhase.Start);
Thread.Sleep(300);
Assert.Equal(0, EventCount);
}
[Fact]
public void RepeatedSamePhaseSignal_IsNotAnExit()
{
_executor.UpdateRules(new List<GatherRule> { ExitRule("GATHER-PX-005", "") });
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup); // same phase — no transition
Thread.Sleep(300);
Assert.Equal(0, EventCount);
}
[Fact]
public void FiresOnTransitionIntoFailed_CapturingStateAtTheFailurePoint()
{
_executor.UpdateRules(new List<GatherRule> { ExitRule("GATHER-PX-006", "AccountSetup") });
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
_executor.OnPhaseChanged(EnrollmentPhase.Failed);
Assert.True(WaitForEventCount(1), "leaving a phase into Failed must still snapshot it");
}
[Fact]
public void FiresOnExitOfCompleteOnly_WhenCompleteIsActuallyLeft()
{
// "End of enrollment" is better served by phase_change on Complete / on_event; an exit
// rule on Complete only fires if a later transition happens at all.
_executor.UpdateRules(new List<GatherRule> { ExitRule("GATHER-PX-007", "Complete") });
_executor.OnPhaseChanged(EnrollmentPhase.FinalizingSetup);
_executor.OnPhaseChanged(EnrollmentPhase.Complete);
Thread.Sleep(300);
Assert.Equal(0, EventCount);
}
[Fact]
public void ScopeGate_EvaluatesAgainstThePhaseBeingLeft()
{
// Scope = only during DeviceSetup. Leaving DeviceSetup must fire (the rule is still in
// scope at that instant) even though the NEW phase is outside the scope.
_executor.UpdateRules(new List<GatherRule>
{
ExitRule("GATHER-PX-010", "DeviceSetup", activePhases: new List<string> { "DeviceSetup" })
});
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(WaitForEventCount(1),
"the scope gate must see the phase being left, not the one being entered");
}
[Fact]
public void OutOfScopeExit_DoesNotFire_AndDoesNotConsumeTheOneShotSlot()
{
// Scope = only during AccountSetup; the rule exits DeviceSetup while out of scope.
var rule = ExitRule("GATHER-PX-011", "", activePhases: new List<string> { "AccountSetup" });
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup); // exit of DeviceSetup: out of scope
Thread.Sleep(300);
Assert.Equal(0, EventCount);
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser); // exit of AccountSetup: in scope
Assert.True(WaitForEventCount(1));
}
[Fact]
public void ExitAndChangeRulesForTheSamePhase_BothFireIndependently()
{
// The dedup key spaces must not collide: same ruleId shape, same phase, different trigger.
var enter = new GatherRule
{
RuleId = "GATHER-PX-020",
Title = "enter",
CollectorType = "registry",
Target = AbsentPath,
Trigger = "phase_change",
TriggerPhase = "AccountSetup",
OutputEventType = "gather_test",
Enabled = true,
};
var exit = ExitRule("GATHER-PX-021", "AccountSetup");
_executor.UpdateRules(new List<GatherRule> { enter, exit });
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(WaitForEventCount(1)); // enter fired
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser);
Assert.True(WaitForEventCount(2)); // exit fired
}
[Fact]
public void ExitRuleWithEmitOnChange_StillEmitsItsFirstResult()
{
var rule = ExitRule("GATHER-PX-030", "AccountSetup");
rule.EmitMode = "on_change";
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser);
Assert.True(WaitForEventCount(1));
}
}
}
@@ -0,0 +1,326 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
{
/// <summary>
/// Phase scoping on <see cref="GatherRuleExecutor"/> (ActivePhases / ActiveFromPhase):
/// the scope matrix is asserted synchronously via the internal <c>IsRuleInScope</c>
/// (no timer races); the trigger-path wiring (deferred startup, phase_change / on_event
/// gating) is asserted through the public API with positive event waits.
/// Rules use the absent-registry-path pattern from
/// <see cref="RegistryCollectorEmitOnlyIfExistsTests"/> so no real system state is touched.
/// </summary>
[Collection("SerialThreading")] // deferred startup rules execute on the shared ThreadPool
public sealed class GatherRuleExecutorPhaseScopeTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
private readonly List<EnrollmentEvent> _events = new List<EnrollmentEvent>();
private readonly object _eventsGate = new object();
private readonly GatherRuleExecutor _executor;
private const string AbsentPath =
"HKLM\\SOFTWARE\\AutopilotMonitorTests\\DefinitelyAbsent_5b1c8d2e-0000-0000-0000-000000000000";
public GatherRuleExecutorPhaseScopeTests()
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_executor = new GatherRuleExecutor(
"sess", "tenant",
evt => { lock (_eventsGate) _events.Add(evt); },
logger)
{
// Bypass the registry allowlist guard for the synthetic test path.
UnrestrictedMode = true
};
}
public void Dispose()
{
_executor.Dispose();
_tmp.Dispose();
}
private int EventCount { get { lock (_eventsGate) return _events.Count; } }
private bool WaitForEventCount(int count, int timeoutMs = 10000)
{
var sw = Stopwatch.StartNew();
while (sw.ElapsedMilliseconds < timeoutMs)
{
if (EventCount >= count) return true;
Thread.Sleep(25);
}
return EventCount >= count;
}
/// <summary>Interval rule with a 1h period — the timer never ticks within a test run,
/// so these rules exist purely for synchronous scope-matrix assertions.</summary>
private static GatherRule ScopeProbeRule(string id, List<string>? activePhases = null, string? fromPhase = null)
=> new GatherRule
{
RuleId = id,
Title = id,
CollectorType = "registry",
Target = AbsentPath,
Trigger = "interval",
IntervalSeconds = 3600,
OutputEventType = "gather_test",
Enabled = true,
ActivePhases = activePhases,
ActiveFromPhase = fromPhase,
};
private static GatherRule StartupRule(string id, List<string>? activePhases = null, string? fromPhase = null)
=> new GatherRule
{
RuleId = id,
Title = id,
CollectorType = "registry",
Target = AbsentPath,
Trigger = "startup",
OutputEventType = "gather_test",
Enabled = true,
ActivePhases = activePhases,
ActiveFromPhase = fromPhase,
};
// ── Scope matrix (synchronous) ─────────────────────────────────────
[Fact]
public void UnscopedRule_IsInScope_EvenBeforeAnyPhaseSignal()
{
var rule = ScopeProbeRule("GATHER-SCOPE-000");
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.True(_executor.IsRuleInScope(rule));
}
[Fact]
public void ActivePhases_InactiveBeforeFirstPhaseSignal()
{
var rule = ScopeProbeRule("GATHER-SCOPE-001", activePhases: new List<string> { "AccountSetup" });
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.False(_executor.IsRuleInScope(rule));
}
[Fact]
public void ActivePhases_ActiveDuringListedPhase_InactiveAfterLeaving()
{
var rule = ScopeProbeRule("GATHER-SCOPE-002", activePhases: new List<string> { "AccountSetup" });
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Assert.False(_executor.IsRuleInScope(rule));
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(_executor.IsRuleInScope(rule));
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser);
Assert.False(_executor.IsRuleInScope(rule));
}
[Fact]
public void ActivePhases_MatchIsCaseInsensitive()
{
var rule = ScopeProbeRule("GATHER-SCOPE-003", activePhases: new List<string> { "accountsetup" });
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(_executor.IsRuleInScope(rule));
}
[Fact]
public void FromPhase_LatchesAtThreshold_AndStaysSticky_IncludingThroughFailed()
{
var rule = ScopeProbeRule("GATHER-SCOPE-010", fromPhase: "AccountSetup");
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Assert.False(_executor.IsRuleInScope(rule)); // threshold not reached
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(_executor.IsRuleInScope(rule)); // latched
_executor.OnPhaseChanged(EnrollmentPhase.Failed);
Assert.True(_executor.IsRuleInScope(rule)); // sticky through Failed
}
[Fact]
public void FromPhase_NoSpuriousActivationViaFailed()
{
// Failed=99 would ordinal-satisfy every threshold — it must never latch.
var rule = ScopeProbeRule("GATHER-SCOPE-011", fromPhase: "AccountSetup");
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.Failed);
Assert.False(_executor.IsRuleInScope(rule));
}
[Fact]
public void FromPhase_LatchesWhenThresholdIsSkippedOver()
{
// The session may never report the exact threshold phase — a later phase latches too.
var rule = ScopeProbeRule("GATHER-SCOPE-012", fromPhase: "AccountSetup");
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.FinalizingSetup);
Assert.True(_executor.IsRuleInScope(rule));
}
[Fact]
public void FromPhase_RuleDeliveredByRefreshAfterPhaseReached_LatchesImmediately()
{
_executor.UpdateRules(new List<GatherRule>());
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
// Config refresh delivers a new from-phase rule whose threshold already passed.
var rule = ScopeProbeRule("GATHER-SCOPE-013", fromPhase: "DeviceSetup");
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.True(_executor.IsRuleInScope(rule));
}
[Fact]
public void BothFieldsSet_ActivePhasesWins()
{
// Backend validation rejects this shape; the agent defensively prefers ActivePhases.
var rule = ScopeProbeRule("GATHER-SCOPE-020",
activePhases: new List<string> { "DeviceSetup" }, fromPhase: "Start");
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.False(_executor.IsRuleInScope(rule)); // FromPhase(Start) would say yes — list wins
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Assert.True(_executor.IsRuleInScope(rule));
}
[Fact]
public void IgnorePhaseScope_BypassesScope_WithoutPhaseContext()
{
_executor.IgnorePhaseScope = true;
var during = ScopeProbeRule("GATHER-SCOPE-030", activePhases: new List<string> { "AccountSetup" });
var from = ScopeProbeRule("GATHER-SCOPE-031", fromPhase: "Complete");
_executor.UpdateRules(new List<GatherRule> { during, from });
Assert.True(_executor.IsRuleInScope(during));
Assert.True(_executor.IsRuleInScope(from));
}
// ── Trigger-path wiring (async, positive waits) ────────────────────
[Fact]
public void UnscopedStartupRule_RunsImmediately()
{
_executor.UpdateRules(new List<GatherRule> { StartupRule("GATHER-START-000") });
Assert.True(_executor.WaitForStartupRules(30));
Assert.True(WaitForEventCount(1));
}
[Fact]
public void ScopedStartupRule_IsDeferred_ThenFiresExactlyOnce_WhenScopeActivates()
{
_executor.UpdateRules(new List<GatherRule>
{
StartupRule("GATHER-START-001", fromPhase: "AccountSetup")
});
// Not in scope at UpdateRules: nothing queued, latch trivially satisfied.
Assert.True(_executor.WaitForStartupRules(30));
Assert.Equal(0, EventCount);
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Assert.Equal(0, EventCount);
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(WaitForEventCount(1), "deferred startup rule should fire when its scope activates");
// Later phase changes must not re-fire it (dedup via _startupRulesExecuted).
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser);
_executor.OnPhaseChanged(EnrollmentPhase.FinalizingSetup);
Thread.Sleep(300);
Assert.Equal(1, EventCount);
}
[Fact]
public void PhaseChangeRule_OutOfScopePhase_DoesNotFire_InScopePhaseFires()
{
// TriggerPhase empty = fires on every phase change; scope narrows it to AccountSetup.
var rule = new GatherRule
{
RuleId = "GATHER-PC-001",
Title = "pc",
CollectorType = "registry",
Target = AbsentPath,
Trigger = "phase_change",
TriggerPhase = "",
OutputEventType = "gather_test",
Enabled = true,
ActivePhases = new List<string> { "AccountSetup" },
};
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Thread.Sleep(300);
Assert.Equal(0, EventCount);
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(WaitForEventCount(1));
// Same phase again: the (rule, phase) dedup slot is now consumed.
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Thread.Sleep(300);
Assert.Equal(1, EventCount);
}
[Fact]
public void OnEventRule_GatedByScope()
{
var rule = new GatherRule
{
RuleId = "GATHER-OE-001",
Title = "oe",
CollectorType = "registry",
Target = AbsentPath,
Trigger = "on_event",
TriggerEventType = "app_install_failed",
OutputEventType = "gather_test",
Enabled = true,
ActivePhases = new List<string> { "AccountSetup" },
};
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnEvent("app_install_failed"); // no phase signal yet → out of scope
Thread.Sleep(300);
Assert.Equal(0, EventCount);
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
_executor.OnEvent("app_install_failed");
Assert.True(WaitForEventCount(1));
}
[Fact]
public void IgnorePhaseScope_ScopedStartupRule_RunsImmediately()
{
// The --run-gather-rules diagnostic path: scoped rules must execute unconditionally.
_executor.IgnorePhaseScope = true;
_executor.UpdateRules(new List<GatherRule>
{
StartupRule("GATHER-START-002", activePhases: new List<string> { "AccountSetup" })
});
Assert.True(_executor.WaitForStartupRules(30));
Assert.True(WaitForEventCount(1));
}
}
}
@@ -27,7 +27,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
[InlineData(" ")] // whitespace only
[InlineData("<![LOG[truncated line with no closing")] // truncated / missing fields
[InlineData("garbage prefix <![LOG[msg]LOG]!><time=\"06:08:04.8834397\" date=\"2-8-2026\" component=\"C\" context=\"\" type=\"1\" thread=\"1\" file=\"\">")] // leading junk (non-ignorable) fails the StartsWith gate
public void TryParseLine_returns_false_without_throwing_for_malformed_input(string line)
public void TryParseLine_returns_false_without_throwing_for_malformed_input(string? line)
{
var ok = CmTraceLogParser.TryParseLine(line, out var entry);
@@ -24,12 +24,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
/// </summary>
public sealed class ImeLogPatternReDoSTests
{
private static ImeLogPattern Pattern(string id, string regex, string action = null) => new ImeLogPattern
private static ImeLogPattern Pattern(string id, string regex, string? action = null) => new ImeLogPattern
{
PatternId = id,
Pattern = regex,
Category = "always",
Action = action,
// Action is optional on the wire (absent = null); the model declares it non-nullable.
Action = action!,
Enabled = true,
Parameters = new Dictionary<string, string>(),
};
@@ -0,0 +1,166 @@
using System;
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
{
/// <summary>
/// Tracker-level historic-replay guard (session eaf3d8c4 part 2): source lines > 24 h older
/// than now are content from a previous enrollment whose IME log survived on disk. The guard
/// in <c>HandlePatternMatch</c> skips app-mutating actions for such lines so replayed apps
/// never enter <c>_packageStates</c> / phase snapshots / persistence — keeping
/// app_tracking_summary, culprit lists and final-status clean. Script actions pass through
/// (the adapter suppresses their emissions), and SimulationMode bypasses the guard entirely.
/// </summary>
public sealed class ImeLogTrackerHistoricReplayGuardTests
{
private static readonly DateTime Now = new DateTime(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
private static List<ImeLogPattern> TestPatterns() => new List<ImeLogPattern>
{
new ImeLogPattern
{
PatternId = "T-POLICIES", Category = "always", Enabled = true,
Pattern = @"policies discovered = (?<policies>\[.*\])",
Action = "policiesDiscovered",
Parameters = new Dictionary<string, string>(),
},
new ImeLogPattern
{
PatternId = "T-INSTALLED", Category = "always", Enabled = true,
Pattern = @"app (?<id>[\w-]+) installed",
Action = "updateStateInstalled",
Parameters = new Dictionary<string, string>(),
},
new ImeLogPattern
{
PatternId = "T-ESP-PHASE", Category = "always", Enabled = true,
Pattern = @"In EspPhase: (?<espPhase>\w+)",
Action = "espPhaseDetected",
Parameters = new Dictionary<string, string>(),
},
new ImeLogPattern
{
PatternId = "T-SCRIPT-START", Category = "always", Enabled = true,
Pattern = @"script (?<id>[\w-]+) started",
Action = "scriptStarted",
Parameters = new Dictionary<string, string> { ["scriptType"] = "platform" },
},
};
private static ImeLogTracker BuildTracker(TempDirectory tmp)
{
var tracker = new ImeLogTracker(
logFolder: tmp.Path,
patterns: TestPatterns(),
logger: new AgentLogger(tmp.Path, AgentLogLevel.Info));
tracker.UtcNowProvider = () => Now;
return tracker;
}
private const string PoliciesLine = @"policies discovered = [{""Id"":""app-1"",""Name"":""App One""}]";
[Fact]
public void Stale_policies_line_does_not_populate_package_states()
{
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddDays(-7));
Assert.Empty(tracker.GetAllKnownPackageStates());
}
[Fact]
public void Fresh_relog_of_same_policies_is_tracked_after_stale_skip()
{
// IME re-logs the full policy JSON on every app-policy check-in — skipping the
// stale copy cannot starve the current enrollment.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddDays(-7));
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddMinutes(-2));
var all = tracker.GetAllKnownPackageStates();
var pkg = Assert.Single(all);
Assert.Equal("app-1", pkg.Id);
}
[Fact]
public void Stale_state_update_against_fresh_app_is_skipped()
{
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
var stateChanges = 0;
tracker.OnAppStateChanged = (_, __, ___) => stateChanges++;
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddMinutes(-5)); // fresh discovery
tracker.ProcessLogMessageForTest("app app-1 installed", Now.AddDays(-7)); // stale mutation
Assert.Equal(0, stateChanges);
var pkg = Assert.Single(tracker.GetAllKnownPackageStates());
Assert.NotEqual(AppInstallationState.Installed, pkg.InstallationState);
}
[Fact]
public void Stale_esp_phase_line_does_not_block_fresh_phase_detection()
{
// A stale "AccountSetup" would advance _currentPhaseOrder and make the fresh
// enrollment's DeviceSetup bounce as "backward" — the guard must skip it.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
var phases = new List<string>();
tracker.OnEspPhaseChanged = p => phases.Add(p);
tracker.ProcessLogMessageForTest("In EspPhase: AccountSetup", Now.AddDays(-7));
tracker.ProcessLogMessageForTest("In EspPhase: DeviceSetup", Now.AddMinutes(-1));
Assert.Equal(new[] { "DeviceSetup" }, phases);
}
[Fact]
public void Stale_script_line_still_reaches_script_handlers()
{
// Script tracker state is harmless for stale lines (adapter suppresses the
// emissions; the stale-slot hardening covers leftovers) — the guard must not
// swallow script actions.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
var started = new List<ScriptStartedInfo>();
tracker.OnScriptStarted = i => started.Add(i);
tracker.ProcessLogMessageForTest("script aaaa1111-0000-0000-0000-000000000000 started", Now.AddDays(-7));
Assert.Single(started);
}
[Fact]
public void SimulationMode_bypasses_the_guard()
{
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.SimulationMode = true;
tracker.ProcessLogMessageForTest(PoliciesLine, Now.AddDays(-7));
Assert.Single(tracker.GetAllKnownPackageStates());
}
[Fact]
public void Lines_without_timestamp_are_not_treated_as_stale()
{
// Non-CMTrace lines carry no source timestamp (entry == null) — never stale.
using var tmp = new TempDirectory();
using var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest(PoliciesLine, sourceTimestampUtc: null);
Assert.Single(tracker.GetAllKnownPackageStates());
}
}
}
@@ -17,6 +17,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Office
/// cadence and stops at its bound, is idempotent, the initial scan raises when binaries already
/// exist, a filesystem create raises, and Dispose cancels the re-probe.
/// </summary>
[Collection("SerialThreading")] // the bounded re-probe runs on the shared ThreadPool
public sealed class OfficeBinaryWatcherTests
{
private static readonly string[] CoreBinaries = { "WINWORD.EXE", "EXCEL.EXE", "POWERPNT.EXE", "OUTLOOK.EXE" };
@@ -0,0 +1,40 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Smoke tests for the WinRT <c>SystemSetupInfo.OutOfBoxExperienceState</c> reader.
/// The actual state on the test machine is unknown (CI runners are post-OOBE; Server
/// SKUs may lack the UniversalApiContract entirely), so only the fail-soft contract
/// is pinned: never throw, always return a member of the known value set.
/// </summary>
public sealed class OobeStateReaderTests
{
[Fact]
public void Read_never_throws_and_returns_known_value()
{
var value = OobeStateReader.Read();
Assert.False(string.IsNullOrEmpty(value));
Assert.True(
value == "not_started"
|| value == "in_progress"
|| value == "completed"
|| value == OobeStateReader.Unavailable
|| value.StartsWith("unknown_"),
$"unexpected reader value '{value}'");
}
[Fact]
public void Read_is_stable_across_repeated_calls()
{
// Exercises the cached-PropertyInfo path (second call skips resolution).
var first = OobeStateReader.Read();
var second = OobeStateReader.Read();
Assert.Equal(first, second);
}
}
}
@@ -0,0 +1,208 @@
using System;
using System.Diagnostics;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Executes the generated cleanup script against REAL scratch trees with a real
/// powershell.exe — end-to-end replays of the 2026-07-22 incident mechanics that the
/// string pins in <see cref="CleanupServiceScriptTests"/> cannot prove:
/// <list type="bullet">
/// <item>a locked tree must abort WITHOUT deleting anything (old script force-deleted
/// marker + state around the locks and orphaned the binaries),</item>
/// <item>a straggler agent process (the incident's second instance) must be found by
/// NAME, killed, and the cleanup must then complete — simulated via function
/// shadowing, NEVER by launching a renamed executable (EDR isolation, see the test),</item>
/// <item>concurrent cleanup scripts must be serialized by the lock file.</item>
/// </list>
/// Timings are dialed down via the test-only BuildCleanupScript parameters so each case
/// runs in seconds. Every test uses GUID-unique process/task/tree names so nothing real
/// (a dev-box agent, a real Scheduled Task) can ever be touched.
/// </summary>
[Collection("SerialThreading")]
public class CleanupServiceScriptIntegrationTests : IDisposable
{
private readonly string _root;
private readonly string _basePath;
private readonly string _taskName = $"AmTest-NoSuchTask-{Guid.NewGuid():N}";
private readonly string _fakeProcessName = $"AmTestAgent{Guid.NewGuid().ToString("N").Substring(0, 8)}";
public CleanupServiceScriptIntegrationTests()
{
_root = Path.Combine(Path.GetTempPath(), $"am-cleanup-it-{Guid.NewGuid():N}");
_basePath = Path.Combine(_root, "AutopilotMonitor");
Directory.CreateDirectory(Path.Combine(_basePath, "Agent"));
Directory.CreateDirectory(Path.Combine(_basePath, "State"));
Directory.CreateDirectory(Path.Combine(_basePath, "Logs"));
File.WriteAllText(Path.Combine(_basePath, "Agent", "fake.exe"), "x");
File.WriteAllText(Path.Combine(_basePath, "State", "enrollment-complete.marker"), "x");
File.WriteAllText(Path.Combine(_basePath, "Logs", "agent.log"), "x");
}
public void Dispose()
{
try { Directory.Delete(_root, recursive: true); } catch { }
}
private static string LockPath => Path.Combine(Path.GetTempPath(), "AutopilotMonitor-Cleanup.lock");
private string BuildScript(bool keepLogs = false, int processWaitSeconds = 1)
=> CleanupService.BuildCleanupScript(
agentProcessName: _fakeProcessName,
agentBasePath: _basePath,
keepLogs: keepLogs,
scheduledTaskName: _taskName,
rebootOnComplete: false,
initialDelaySeconds: 0,
processWaitSeconds: processWaitSeconds,
renameRetries: 2,
renameRetryDelaySeconds: 1);
/// <summary>Runs the script synchronously in a real powershell.exe. Returns the script path.</summary>
private string RunScript(string script)
{
var scriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
File.WriteAllText(scriptPath, script);
using (var proc = Process.Start(new ProcessStartInfo
{
FileName = SystemPaths.PowerShell,
Arguments = $"-NoProfile -ExecutionPolicy Bypass -File \"{scriptPath}\"",
UseShellExecute = false,
CreateNoWindow = true,
}))
{
Assert.True(proc.WaitForExit(90_000), "cleanup script did not finish within 90s");
}
return scriptPath;
}
[Fact]
public void Happy_path_deletes_tree_and_removes_lock_and_script()
{
var scriptPath = RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "tree should be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(scriptPath), "script should self-delete");
Assert.False(File.Exists(LockPath), "cleanup lock should be released");
}
[Fact]
public void Locked_tree_aborts_completely_marker_and_files_survive()
{
// The incident regression: with a file locked inside the tree the OLD script
// deleted everything else (incl. the enrollment-complete marker) and left the
// binaries orphaned with the task already gone. The hardened script must treat
// the failed probe-rename as "someone still holds this" and walk away entirely.
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.None))
{
RunScript(BuildScript());
}
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")), "binaries must stay in place");
Assert.True(File.Exists(Path.Combine(_basePath, "State", "enrollment-complete.marker")),
"marker must survive so the next boot retries cleanup");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(LockPath));
}
[Fact]
public void Straggler_agent_process_is_waited_on_and_force_killed_via_name_backstop()
{
// Incident replay of the second-instance mechanics WITHOUT spawning a process:
// an earlier version of this test copied powershell.exe under the fake agent name
// and launched it — textbook MITRE T1036.003 ("Rename Legitimate Utilities"),
// which Defender for Endpoint answered with a machine ISOLATION on the dev box
// (see tasks/lessons.md). Instead, a harness script shadows Get-Process /
// Stop-Process with functions (PowerShell resolves functions before cmdlets, so
// the dot-sourced cleanup script binds to the stubs): Get-Process reports a live
// agent-named straggler on every probe; Stop-Process records the kill. This pins
// the same contract — the script polls by NAME until the deadline, then
// force-kills by NAME before deleting — with zero EDR-visible behavior.
var probeLog = Path.Combine(_root, "probe.log");
var cleanupScriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
File.WriteAllText(cleanupScriptPath, BuildScript(processWaitSeconds: 2));
var harness = @"
$probeLog = 'LOG_PATH'
$global:AmProbeCount = 0
function Get-Process {
[CmdletBinding()]
param([string[]]$Name)
if ($Name -contains 'FAKE_NAME') {
$global:AmProbeCount++
Add-Content -Path $probeLog -Value ('probe ' + $global:AmProbeCount)
return [pscustomobject]@{ Name = 'FAKE_NAME'; Id = 424242 }
}
Microsoft.PowerShell.Management\Get-Process @PSBoundParameters
}
function Stop-Process {
[CmdletBinding()]
param([Parameter(ValueFromPipeline = $true)]$InputObject, [switch]$Force)
process { Add-Content -Path $probeLog -Value ('kill ' + $InputObject.Id + ' force=' + $Force) }
}
. 'SCRIPT_PATH'
"
.Replace("LOG_PATH", probeLog)
.Replace("FAKE_NAME", _fakeProcessName)
.Replace("SCRIPT_PATH", cleanupScriptPath);
RunScript(harness);
var log = File.ReadAllLines(probeLog);
Assert.Contains("kill 424242 force=True", log);
Assert.True(Array.FindAll(log, l => l.StartsWith("probe")).Length >= 2,
"script must poll by name in the wait loop AND re-probe for the kill backstop");
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(LockPath));
}
[Fact]
public void Fresh_foreign_lock_backs_off_without_touching_the_tree()
{
File.WriteAllText(LockPath, "held by another cleanup");
try
{
var scriptPath = RunScript(BuildScript());
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")),
"loser script must not touch the tree — the lock holder owns the deletion");
Assert.True(File.Exists(LockPath), "foreign lock must not be deleted by the loser");
Assert.False(File.Exists(scriptPath), "loser still self-deletes its script");
}
finally
{
try { File.Delete(LockPath); } catch { }
}
}
[Fact]
public void Stale_foreign_lock_is_taken_over_and_cleanup_proceeds()
{
File.WriteAllText(LockPath, "crashed cleanup from a previous boot");
File.SetLastWriteTime(LockPath, DateTime.Now.AddMinutes(-15));
RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "stale lock must be taken over, cleanup must run");
Assert.False(File.Exists(LockPath));
}
[Fact]
public void KeepLogs_variant_deletes_everything_except_logs()
{
RunScript(BuildScript(keepLogs: true));
Assert.True(File.Exists(Path.Combine(_basePath, "Logs", "agent.log")), "Logs must survive");
Assert.False(Directory.Exists(Path.Combine(_basePath, "Agent")));
Assert.False(Directory.Exists(Path.Combine(_basePath, "State")));
Assert.Empty(Directory.GetFileSystemEntries(_basePath, "*.del-*"));
}
}
}
@@ -0,0 +1,135 @@
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Pins the LIFE-F7 hardening invariants of the self-destruct cleanup script
/// (incident 2026-07-22: a second agent instance held exe + loaded DLLs while the
/// old script — waiting on the launcher PID only — force-deleted around the locks
/// AFTER removing the Scheduled Task, orphaning the binaries with no retry path).
/// These are string-level pins on <see cref="CleanupService.BuildCleanupScript"/>:
/// weakening any of them silently re-introduces the orphaned-binaries failure mode.
/// </summary>
public class CleanupServiceScriptTests
{
private const string ProcessName = "AutopilotMonitor.Agent";
private const string BasePath = @"C:\ProgramData\AutopilotMonitor";
private const string TaskName = "AutopilotMonitorAgent";
private static string Script(bool keepLogs = false, bool reboot = false)
=> CleanupService.BuildCleanupScript(ProcessName, BasePath, keepLogs, TaskName, reboot);
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Waits_by_process_name_not_launcher_pid(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #2: name-based wait covers marker-retry / self-update / manual instances
// and is immune to PID reuse. Wait-Process -Id saw exactly one PID and nothing else.
Assert.Contains($"Get-Process -Name '{ProcessName}'", script);
Assert.DoesNotContain("Wait-Process", script);
// Backstop: stragglers are force-killed — their images are deleted right after.
Assert.Contains("Stop-Process -Force", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Never_deletes_the_original_path(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #3: deletion happens exclusively on successfully renamed targets
// ($entry.Renamed). A Remove-Item against the original tree is what deleted
// marker + state around locked binaries in the incident.
Assert.DoesNotContain($"Remove-Item -Path '{BasePath}'", script);
Assert.Contains("Remove-Item -Path $entry.Renamed", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Task_removal_is_gated_on_probe_and_deletion_on_task_gone(bool keepLogs)
{
var script = Script(keepLogs);
// Probe-rename first; only a successful probe reaches task removal, and only a
// removed task reaches deletion. Both failure branches rename BACK so the next
// boot retries with task + marker + files intact.
Assert.Contains("if ($probeOk)", script);
Assert.Contains("if ($taskGone)", script);
Assert.Contains("Rename-Item -Path $entry.Renamed -NewName $entry.Original", script);
// LIFE-F6 task-removal loop + module-independent fallback stay intact.
Assert.Contains("Unregister-ScheduledTask -TaskName $taskName", script);
Assert.Contains("schtasks.exe /Delete /TN $taskName /F", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Concurrent_cleanup_scripts_are_serialized_by_lock_file(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #1: CreateNew is the atomic single-flight gate; the loser exits without
// touching the tree. Stale locks (>10 min) are taken over.
Assert.Contains("AutopilotMonitor-Cleanup.lock", script);
Assert.Contains("[System.IO.FileMode]::CreateNew", script);
Assert.Contains("$lockAge.TotalMinutes -gt 10", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Rename_targets_carry_a_guid_suffix(bool keepLogs)
{
// A fixed '.del' target collides with leftovers of earlier attempts and with a
// concurrent script's rename — the GUID makes every attempt's target unique.
Assert.Contains(".del-' + [Guid]::NewGuid().ToString('N')", Script(keepLogs));
}
[Fact]
public void KeepLogs_variant_excludes_logs_directory()
{
var script = Script(keepLogs: true);
Assert.Contains("-Exclude 'Logs'", script);
Assert.DoesNotContain($"Rename-Item -Path '{BasePath}' -NewName", script);
}
[Fact]
public void Full_variant_renames_the_whole_tree()
{
var script = Script(keepLogs: false);
Assert.Contains($"Rename-Item -Path '{BasePath}' -NewName $renamedPath", script);
Assert.DoesNotContain("-Exclude 'Logs'", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Reboot_flag_controls_shutdown_invocation(bool reboot)
{
var script = Script(keepLogs: false, reboot: reboot);
if (reboot)
Assert.Contains("shutdown.exe /r /t 10", script);
else
Assert.DoesNotContain("shutdown.exe", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Script_and_lock_are_released_in_finally(bool keepLogs)
{
var script = Script(keepLogs);
Assert.Contains("finally", script);
Assert.Contains("$lockStream.Dispose()", script);
Assert.Contains("Remove-Item -Path $scriptPath -Force", script);
}
}
}
@@ -256,7 +256,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
[InlineData(null, false)]
[InlineData("", false)]
[InlineData("0x8007", false)]
public void IsAppxFamilyHresult_MatchesOnlyAppxDeploymentRange(string hresult, bool expected)
public void IsAppxFamilyHresult_MatchesOnlyAppxDeploymentRange(string? hresult, bool expected)
{
Assert.Equal(expected, AppxFailureAnalyzer.IsAppxFamilyHresult(hresult));
}
@@ -0,0 +1,256 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// Session 4910a5a5 (2026-07-23) — post-terminal recovery story. Field sequence:
/// DeviceSetup/Apps failed (culprit app never started installing) → settle window expired
/// → terminal EspFailureDetected → the user pressed "Try again" ~23 min later → Apps left
/// the failed state → 10/10 installed → DeviceSetup resolved. Pre-fix, that recovery only
/// appeared as a generic <c>subcategory_state_change</c>. Coverage:
/// <list type="bullet">
/// <item><c>esp_failure_retry_detected</c> fires once when the failed subcategory leaves
/// the failed state AFTER the terminal fire.</item>
/// <item><c>esp_failure_recovered</c> fires once when the category then resolves to
/// success.</item>
/// <item>An in-settle-window recovery keeps emitting only
/// <c>esp_failure_settle_recovered</c> — the post-terminal pair stays silent.</item>
/// <item>Apps failures carry <c>LikelyCulpritApps</c> on the EspFailureDetected args,
/// never-started apps ranked first.</item>
/// </list>
/// </summary>
public sealed class EspFailureRecoveryStoryTests
{
private static readonly DateTime Fixed = new DateTime(2026, 7, 23, 4, 30, 0, DateTimeKind.Utc);
private sealed class Fixture : IDisposable
{
public TempDirectory Tmp { get; } = new TempDirectory();
public FakeSignalIngressSink Sink { get; } = new FakeSignalIngressSink();
public List<EspFailureDetectedEventArgs> EspFailures { get; } = new List<EspFailureDetectedEventArgs>();
public ProvisioningStatusTracker Tracker { get; }
public Fixture(Func<IReadOnlyList<AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState>>? packageStatesProbe = null)
{
var clock = new VirtualClock(Fixed);
var post = new InformationalEventPost(Sink, clock);
var logger = new AgentLogger(Tmp.Path, AgentLogLevel.Info);
Tracker = new ProvisioningStatusTracker(
sessionId: "S1",
tenantId: "T1",
post: post,
logger: logger,
appxScanner: new FakeAppxDeploymentFailureScanner(),
backgroundDispatcher: action => action(),
packageStatesProbe: packageStatesProbe);
Tracker.EspFailureDetected += (_, args) => EspFailures.Add(args);
}
public IReadOnlyList<(string EventType, IReadOnlyDictionary<string, object> Data)> CapturedEvents()
{
return Sink.Posted
.Where(p => p.Kind == DecisionSignalKind.InformationalEvent && p.Payload != null)
.Select(p =>
{
var eventType = p.Payload!.TryGetValue(SignalPayloadKeys.EventType, out var et) ? et : "";
var data = p.TypedPayload as IReadOnlyDictionary<string, object>
?? (IReadOnlyDictionary<string, object>)new Dictionary<string, object>();
return (eventType, data);
})
.ToList();
}
/// <summary>Drive the session-4910a5a5 shape up to the terminal fire.</summary>
public void DriveToTerminalDeviceSetupAppsFailure()
{
Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (9 of 10 installed)""}
}");
Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
}
public void Dispose()
{
Tracker.Dispose();
Tmp.Dispose();
}
}
[Fact]
public void PostTerminalRetry_EmitsRetryDetected_Once()
{
using var f = new Fixture();
f.DriveToTerminalDeviceSetupAppsFailure();
Assert.Single(f.EspFailures);
// "Try again": Apps leaves the failed state well after the settle window.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
var retry = Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_retry_detected");
Assert.Equal("DeviceSetup", (string)retry.Data["category"]);
Assert.Equal("Apps", (string)retry.Data["subcategory"]);
Assert.Equal("failed", (string)retry.Data["previousState"]);
Assert.Equal("inProgress", (string)retry.Data["newState"]);
Assert.Equal("Provisioning_DeviceSetup_Apps_Failed", (string)retry.Data["failureType"]);
Assert.True(retry.Data.ContainsKey("minutesSinceFailure"));
// A later unrelated transition does not re-emit.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed)""}
}");
Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_retry_detected");
}
[Fact]
public void PostTerminalRecovery_EmitsRecovered_OnCategorySuccess()
{
using var f = new Fixture();
f.DriveToTerminalDeviceSetupAppsFailure();
// Retry → all apps installed → Windows resolves the category boolean.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed)""}
}");
var recovered = Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_recovered");
Assert.Equal("DeviceSetup", (string)recovered.Data["category"]);
Assert.Equal("Apps", (string)recovered.Data["failedSubcategory"]);
Assert.Equal("Provisioning_DeviceSetup_Apps_Failed", (string)recovered.Data["failureType"]);
Assert.True(recovered.Data.ContainsKey("minutesSinceFailure"));
// Duplicate category-success writes do not re-emit (fire-once).
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Security policies (1 of 1 applied)""},
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (10 of 10 installed, take 2)""}
}");
Assert.Single(f.CapturedEvents(), e => e.EventType == "esp_failure_recovered");
}
[Fact]
public void InSettleWindowRecovery_KeepsPostTerminalPairSilent()
{
using var f = new Fixture();
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (0 of 1 installed)""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
// Recovery INSIDE the settle window — terminal fire is suppressed.
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""AppsSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Apps (Working on it...)""}
}");
f.Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
Assert.Empty(f.EspFailures);
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": true,
""AppsSubcategory"": {""subcategoryState"":""succeeded"",""subcategoryStatusText"":""Apps (1 of 1 installed)""}
}");
var events = f.CapturedEvents();
Assert.Contains(events, e => e.EventType == "esp_failure_settle_recovered");
Assert.DoesNotContain(events, e => e.EventType == "esp_failure_retry_detected");
Assert.DoesNotContain(events, e => e.EventType == "esp_failure_recovered");
}
[Fact]
public void AppsFailure_TerminalArgs_CarryLikelyCulprits_NeverStartedFirst()
{
var neverStarted = BuildApp("app-bg", "Teams Backgrounds v2",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: false);
var startedButPending = BuildApp("app-slow", "Slow Installer",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: true);
var installed = BuildApp("app-ok", "Falcon Sensor",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Installed,
downloadingSeen: true);
using var f = new Fixture(() => new[] { startedButPending, neverStarted, installed });
f.DriveToTerminalDeviceSetupAppsFailure();
var fired = Assert.Single(f.EspFailures);
Assert.Equal(2, fired.LikelyCulpritApps.Count);
// Never-started app ranked first — it starved the ESP gate without any IME event.
Assert.Equal("Teams Backgrounds v2", fired.LikelyCulpritApps[0]);
Assert.Equal("Slow Installer", fired.LikelyCulpritApps[1]);
}
[Fact]
public void NonAppsFailure_TerminalArgs_CarryNoCulprits()
{
var neverStarted = BuildApp("app-bg", "Teams Backgrounds v2",
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown,
downloadingSeen: false);
using var f = new Fixture(() => new[] { neverStarted });
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""inProgress"",""subcategoryStatusText"":""Working""}
}");
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status", @"{
""categorySucceeded"": null,
""SecurityPoliciesSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Failed""}
}");
f.Tracker.TriggerSettleTimerForTest("DeviceSetupCategory.Status");
var fired = Assert.Single(f.EspFailures);
Assert.Empty(fired.LikelyCulpritApps);
}
private static AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState BuildApp(
string id, string name,
AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState state,
bool downloadingSeen)
{
var pkg = new AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState(id, 0);
pkg.UpdateName(name);
pkg.UpdateIntent(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppIntent.Install);
pkg.UpdateTargeted(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppTargeted.Device);
// DownloadingOrInstallingSeen is derived inside UpdateState (Downloading..Installing
// range) — drive it through a Downloading transition instead of setting it directly.
if (downloadingSeen)
pkg.UpdateState(AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Downloading);
if (state != AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Unknown
&& state != AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppInstallationState.Downloading)
pkg.UpdateState(state);
return pkg;
}
}
}
@@ -0,0 +1,465 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// Session 5d735290 — pure detection logic of <see cref="EspPolicyProviderStallDetector"/>:
/// the CSP completion rule and the dwell/one-shot transition. No registry, no timers.
/// </summary>
public sealed class EspPolicyProviderStallLogicTests
{
private static readonly DateTime T0 = new DateTime(2026, 7, 23, 10, 0, 0, DateTimeKind.Utc);
private static readonly TimeSpan Dwell = TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes);
private static PolicyProviderState SetupApps(string name, int? tracking, string scope = "device") =>
new PolicyProviderState(name, scope, EspPolicyProviderProbe.KindSetupApps, tracking, null);
private static PolicyProviderState DevicePrep(string name, int? installState) =>
new PolicyProviderState(name, "device", EspPolicyProviderProbe.KindDevicePreparation, null, installState);
private static EspPolicyProviderSnapshot Snapshot(params PolicyProviderState[] providers) =>
new EspPolicyProviderSnapshot(providers);
// ------------------------------------------------- completion rule (CSP contract)
[Theory]
[InlineData(1, true)] // TrackingPoliciesCreated=true
[InlineData(0, false)] // explicit false (the CSP default)
[InlineData(null, false)] // value missing — the field case
[InlineData(2, false)] // anything else is not "true"
public void SetupApps_provider_complete_only_at_tracking_1(int? tracking, bool expected)
=> Assert.Equal(expected, EspPolicyProviderProbe.IsProviderComplete(
EspPolicyProviderProbe.KindSetupApps, tracking, null));
[Theory]
[InlineData(2, true)] // NotRequired
[InlineData(3, true)] // Completed
[InlineData(1, false)] // NotInstalled
[InlineData(4, false)] // Error
[InlineData(null, false)]
public void DevicePreparation_provider_complete_at_state_2_or_3(int? state, bool expected)
=> Assert.Equal(expected, EspPolicyProviderProbe.IsProviderComplete(
EspPolicyProviderProbe.KindDevicePreparation, null, state));
[Fact]
public void TryReadInt_tolerates_dword_string_and_bool_forms()
{
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(1));
Assert.Equal(3, EspPolicyProviderProbe.TryReadInt(3L));
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(" 1 "));
Assert.Equal(1, EspPolicyProviderProbe.TryReadInt(true));
Assert.Equal(0, EspPolicyProviderProbe.TryReadInt(false));
Assert.Null(EspPolicyProviderProbe.TryReadInt(null));
Assert.Null(EspPolicyProviderProbe.TryReadInt("not-a-number"));
Assert.Null(EspPolicyProviderProbe.TryReadInt(new byte[] { 1 }));
}
// ------------------------------------------------- dwell transition
[Fact]
public void Empty_or_missing_snapshot_clears_dwell_state_and_never_stalls()
{
var previous = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var missing = EspPolicyProviderStallDetector.EvaluateStallTransition(
EspPolicyProviderSnapshot.Empty, previous, new HashSet<string>(), T0, false, T0.AddHours(2), Dwell);
Assert.Empty(missing.UpdatedFirstSeenUtc);
Assert.Empty(missing.NewlyStalled);
Assert.Null(missing.UpdatedSidecarMissingSinceUtc); // arm 2 resets too
Assert.Null(missing.SidecarMissingStalledForMinutes);
var registeredButEmpty = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(), previous, new HashSet<string>(), T0, false, T0.AddHours(2), Dwell);
Assert.Empty(registeredButEmpty.UpdatedFirstSeenUtc);
Assert.Empty(registeredButEmpty.NewlyStalled);
Assert.Null(registeredButEmpty.UpdatedSidecarMissingSinceUtc);
}
[Fact]
public void Incomplete_provider_accrues_dwell_but_does_not_stall_before_threshold()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Equal(T0, round1.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]);
Assert.Empty(round1.NewlyStalled);
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(),
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(14), Dwell);
Assert.Equal(T0, round2.UpdatedFirstSeenUtc["setupApps|device|ConfigMgr"]); // first-seen preserved
Assert.Empty(round2.NewlyStalled);
Assert.Null(round2.SidecarMissingStalledForMinutes); // 14 min < dwell on arm 2 as well
}
[Fact]
public void Provider_stalls_at_dwell_threshold_with_correct_minutes()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: 0));
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, seen, new HashSet<string>(), null, false, T0.AddMinutes(15), Dwell);
var stalled = Assert.Single(eval.NewlyStalled);
Assert.Equal("ConfigMgr", stalled.Provider.Name);
Assert.Equal(15.0, stalled.StalledForMinutes, precision: 3);
}
[Fact]
public void Completion_or_disappearance_resets_the_dwell_clock()
{
var key = "setupApps|device|ConfigMgr";
var seen = new Dictionary<string, DateTime> { [key] = T0 };
// Completes mid-dwell → dropped from the map.
var completed = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)), seen, new HashSet<string>(), null, false, T0.AddMinutes(10), Dwell);
Assert.Empty(completed.UpdatedFirstSeenUtc);
// Re-appears incomplete much later → dwell restarts from the new now, no stall yet.
var reappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 0)), completed.UpdatedFirstSeenUtc,
new HashSet<string>(), null, false, T0.AddHours(3), Dwell);
Assert.Equal(T0.AddHours(3), reappeared.UpdatedFirstSeenUtc[key]);
Assert.Empty(reappeared.NewlyStalled);
// Disappearance behaves the same as completion.
var disappeared = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("Sidecar", tracking: 1)), new Dictionary<string, DateTime> { [key] = T0 },
new HashSet<string>(), null, false, T0.AddMinutes(20), Dwell);
Assert.False(disappeared.UpdatedFirstSeenUtc.ContainsKey(key));
}
[Fact]
public void Already_fired_provider_is_never_reported_again()
{
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: null));
var seen = new Dictionary<string, DateTime> { ["setupApps|device|ConfigMgr"] = T0 };
var fired = new HashSet<string>(StringComparer.OrdinalIgnoreCase) { "setupApps|device|ConfigMgr" };
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, seen, fired, null, false, T0.AddHours(5), Dwell);
Assert.Empty(eval.NewlyStalled);
Assert.True(eval.UpdatedFirstSeenUtc.ContainsKey("setupApps|device|ConfigMgr")); // still tracked
}
[Fact]
public void Mixed_kinds_and_scopes_are_tracked_under_distinct_keys()
{
var snapshot = Snapshot(
SetupApps("ConfigMgr", tracking: null),
SetupApps("ConfigMgr", tracking: null, scope: "user:S-1-5-21-1111"),
DevicePrep("ConfigMgr", installState: 1));
var eval = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Equal(3, eval.UpdatedFirstSeenUtc.Count);
Assert.Contains("setupApps|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
Assert.Contains("setupApps|user:S-1-5-21-1111|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
Assert.Contains("devicePreparation|device|ConfigMgr", eval.UpdatedFirstSeenUtc.Keys);
}
[Fact]
public void SidecarRegistered_matches_setupApps_sidecar_only()
{
Assert.True(Snapshot(SetupApps("Sidecar", tracking: 0)).SidecarRegistered);
Assert.True(Snapshot(SetupApps("sidecar", tracking: 1, scope: "user:S-1-5-21-1")).SidecarRegistered);
Assert.False(Snapshot(SetupApps("ConfigMgr", tracking: 1)).SidecarRegistered);
Assert.False(Snapshot(DevicePrep("Sidecar", installState: 3)).SidecarRegistered); // wrong kind
Assert.False(EspPolicyProviderSnapshot.Empty.SidecarRegistered);
}
// ------------------------------------------------- arm 2: sidecar-missing transition
[Fact]
public void Issue106_foreign_provider_complete_but_sidecar_missing_stalls_after_dwell()
{
// The actual field case: ConfigMgr with TrackingPoliciesCreated=1 — "complete" by the
// CSP value contract, arm 1 never fires — yet the ESP waits on the absent Sidecar.
var snapshot = Snapshot(SetupApps("ConfigMgr", tracking: 1));
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Empty(round1.NewlyStalled); // arm 1 stays silent...
Assert.Equal(T0, round1.UpdatedSidecarMissingSinceUtc); // ...but arm 2 starts its clock
Assert.Null(round1.SidecarMissingStalledForMinutes);
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
snapshot, round1.UpdatedFirstSeenUtc, new HashSet<string>(),
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(15), Dwell);
Assert.Empty(round2.NewlyStalled);
Assert.Equal(15.0, round2.SidecarMissingStalledForMinutes!.Value, precision: 3);
}
[Fact]
public void Sidecar_registering_resets_the_sidecar_missing_clock()
{
// ConfigMgr first (legitimate startup ordering) → clock runs...
var round1 = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Equal(T0, round1.UpdatedSidecarMissingSinceUtc);
// ...Sidecar joins → condition clears, state resets.
var round2 = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1), SetupApps("Sidecar", tracking: 0)),
round1.UpdatedFirstSeenUtc, new HashSet<string>(),
round1.UpdatedSidecarMissingSinceUtc, false, T0.AddMinutes(10), Dwell);
Assert.Null(round2.UpdatedSidecarMissingSinceUtc);
Assert.Null(round2.SidecarMissingStalledForMinutes);
// Sidecar gone again much later → clock restarts from the new now, no instant stall.
var round3 = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
round2.UpdatedFirstSeenUtc, new HashSet<string>(),
round2.UpdatedSidecarMissingSinceUtc, false, T0.AddHours(3), Dwell);
Assert.Equal(T0.AddHours(3), round3.UpdatedSidecarMissingSinceUtc);
Assert.Null(round3.SidecarMissingStalledForMinutes);
}
[Fact]
public void Sidecar_missing_one_shot_and_deviceprep_only_scoping()
{
// Already fired → never re-reported, but the clock keeps running.
var latched = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(SetupApps("ConfigMgr", tracking: 1)),
new Dictionary<string, DateTime>(), new HashSet<string>(), T0, true, T0.AddHours(5), Dwell);
Assert.Null(latched.SidecarMissingStalledForMinutes);
Assert.Equal(T0, latched.UpdatedSidecarMissingSinceUtc);
// DevicePreparation providers alone don't constitute the Setup/Apps wait — arm 2 off.
var devicePrepOnly = EspPolicyProviderStallDetector.EvaluateStallTransition(
Snapshot(DevicePrep("ConfigMgr", installState: 1)),
new Dictionary<string, DateTime>(), new HashSet<string>(), null, false, T0, Dwell);
Assert.Null(devicePrepOnly.UpdatedSidecarMissingSinceUtc);
}
}
/// <summary>
/// Emission surface of the detector via the real <see cref="InformationalEventPost"/> rail.
/// Static probe override → serial collection (same contract as
/// <c>EspConfigDetectedTrackingFieldsTests</c>).
/// </summary>
[Collection("SerialThreading")]
public sealed class EspPolicyProviderStallDetectorEmissionTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 7, 23, 12, 0, 0, DateTimeKind.Utc);
private readonly TempDirectory _tmp = new TempDirectory();
public void Dispose() => _tmp.Dispose();
private (EspPolicyProviderStallDetector sut, FakeSignalIngressSink sink, VirtualClock clock)
BuildDetector(StartupEventGate? gate = null)
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
var sink = new FakeSignalIngressSink();
var clock = new VirtualClock(At);
var post = new InformationalEventPost(sink, clock);
var sut = new EspPolicyProviderStallDetector("S1", "T1", post, logger, clock, gate);
return (sut, sink, clock);
}
private static EspPolicyProviderSnapshot ConfigMgrOnly() => new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps,
trackingPoliciesCreated: null, installationState: null),
});
private static EspPolicyProviderSnapshot SidecarCompleteConfigMgrIncomplete() => new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("Sidecar", "device", EspPolicyProviderProbe.KindSetupApps, 1, null),
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, 0, null),
});
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> StalledEvents(FakeSignalIngressSink sink) =>
sink.Posted.Where(p =>
p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == "esp_policy_provider_stalled").ToList();
[Fact]
public void CustomerScenario_configmgr_without_sidecar_emits_one_warning_after_dwell()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly());
var (sut, sink, clock) = BuildDetector();
sut.Tick(); // first sighting — dwell starts
Assert.Empty(StalledEvents(sink));
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick(); // threshold crossed → emit
var evt = Assert.Single(StalledEvents(sink));
Assert.Equal(DecisionSignalKind.InformationalEvent, evt.Kind);
Assert.Equal("Warning", evt.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", evt.Payload[SignalPayloadKeys.ImmediateUpload]);
Assert.Equal(EspPolicyProviderStallDetector.SourceLabel, evt.Payload[SignalPayloadKeys.Source]);
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
// Both arms cross in the same round (ConfigMgr incomplete AND Sidecar absent) →
// ONE event; sidecar-missing is the dominant reason.
Assert.Equal(EspPolicyProviderStallDetector.ReasonSidecarMissing, data["reason"]);
Assert.Equal(false, data["sidecarRegistered"]);
Assert.Equal(15.0, data["sidecarMissingForMinutes"]);
Assert.Equal(EspPolicyProviderStallDetector.DwellMinutes, data["dwellMinutes"]);
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
var entry = Assert.Single(stalled);
Assert.Equal("ConfigMgr", entry["name"]);
Assert.Equal(EspPolicyProviderProbe.KindSetupApps, entry["kind"]);
Assert.Equal(15.0, entry["stalledForMinutes"]);
}
[Fact]
public void Issue106_configmgr_complete_without_sidecar_still_emits_sidecar_missing()
{
// Regression test for the customer's follow-up on issue #106: TrackingPoliciesCreated=1
// WAS present under ConfigMgr — the provider looks "complete", arm 1 never fires, yet
// the ESP hangs because Sidecar is absent. Arm 2 must catch exactly this.
var configMgrComplete = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps,
trackingPoliciesCreated: 1, installationState: null),
});
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => configMgrComplete);
var (sut, sink, clock) = BuildDetector();
sut.Tick();
Assert.Empty(StalledEvents(sink));
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
var evt = Assert.Single(StalledEvents(sink));
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
Assert.Equal(EspPolicyProviderStallDetector.ReasonSidecarMissing, data["reason"]);
Assert.Equal(false, data["sidecarRegistered"]);
Assert.Equal(15.0, data["sidecarMissingForMinutes"]);
Assert.Empty(Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]));
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
Assert.Equal(true, Assert.Single(providers)["complete"]); // the misleading "complete" is visible
}
[Fact]
public void Sidecar_complete_configmgr_incomplete_reports_only_configmgr_with_sidecar_flag()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => SidecarCompleteConfigMgrIncomplete());
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
var evt = Assert.Single(StalledEvents(sink));
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(evt.TypedPayload);
Assert.Equal(EspPolicyProviderStallDetector.ReasonProviderIncomplete, data["reason"]); // arm 2 off — Sidecar is there
Assert.Equal(true, data["sidecarRegistered"]);
Assert.False(data.ContainsKey("sidecarMissingForMinutes"));
var stalled = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object>>>(data["stalledProviders"]);
Assert.Equal("ConfigMgr", Assert.Single(stalled)["name"]);
var providers = Assert.IsAssignableFrom<IReadOnlyList<Dictionary<string, object?>>>(data["providers"]);
Assert.Equal(2, providers.Count); // full table always attached
}
[Fact]
public void Latch_prevents_second_emission_for_the_same_provider()
{
using var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly());
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(30));
sut.Tick();
sut.Tick();
Assert.Single(StalledEvents(sink));
}
[Fact]
public void Startup_gate_suppresses_same_stall_across_restart_but_reports_new_provider_set()
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
// Run 1: fires and persists the fingerprint.
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly()))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Single(StalledEvents(sink));
}
// Run 2 (simulated restart — fresh detector, gate reloaded from the same state dir):
// identical stall re-accrues dwell but lands on the same fingerprint → suppressed.
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => ConfigMgrOnly()))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
// Run 3: a SECOND provider joins the stall → fingerprint changes → reported.
var widened = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindSetupApps, null, null),
new PolicyProviderState("Contoso", "device", EspPolicyProviderProbe.KindSetupApps, 0, null),
});
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => widened))
{
var (sut, sink, clock) = BuildDetector(new StartupEventGate(_tmp.Path, logger));
sut.Tick();
clock.Advance(TimeSpan.FromMinutes(EspPolicyProviderStallDetector.DwellMinutes));
sut.Tick();
Assert.Single(StalledEvents(sink));
}
}
[Fact]
public void Healthy_or_absent_tracking_never_emits()
{
var healthy = new EspPolicyProviderSnapshot(new[]
{
new PolicyProviderState("Sidecar", "device", EspPolicyProviderProbe.KindSetupApps, 1, null),
new PolicyProviderState("ConfigMgr", "device", EspPolicyProviderProbe.KindDevicePreparation, null, 3),
});
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => healthy))
{
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromHours(4));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
using (var _probe = new EspPolicyProviderProbe.ScopedOverride(_ => EspPolicyProviderSnapshot.Empty))
{
var (sut, sink, clock) = BuildDetector();
sut.Tick();
clock.Advance(TimeSpan.FromHours(4));
sut.Tick();
Assert.Empty(StalledEvents(sink));
}
}
}
}
@@ -0,0 +1,336 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// MdmRebootPolicyTracker — aggregates DM-Enterprise EventID 2800 bursts into ONE neutral
/// <c>mdm_policy_reboot_required</c> event (session b2e890c1 rework: per-record events were
/// spam, and the raw event must not CLAIM a reboot — ANALYZE-ESP-005 does, gated on an
/// observed system_reboot_detected). Drives the primitive ProcessEvent seam + explicit
/// FlushPending (debounce 0 = manual-flush mode; no timers, no SerialThreading needed).
/// The registry probe is always overridden — tests never touch the live registry.
/// </summary>
public sealed class MdmRebootPolicyTrackerTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 7, 20, 16, 20, 57, DateTimeKind.Utc);
private const string UriSvchost = "./Device/Vendor/MSFT/Policy/Config/ServiceControlManager/SvchostProcessMitigation";
private const string UriSystemGuard = "./Device/Vendor/MSFT/Policy/Config/DeviceGuard/ConfigureSystemGuardLaunch";
// Verified real 2800 description shape (session b2e890c1, 2026-07-20).
private const string RealDescription =
"The following URI has triggered a reboot: (./Device/Vendor/MSFT/Policy/Config/ServiceControlManager/SvchostProcessMitigation).";
private readonly TempDirectory _tmp = new TempDirectory();
private readonly FakeSignalIngressSink _sink;
private readonly MdmRebootPolicyTracker _tracker;
private long _nextRecordId = 1;
public MdmRebootPolicyTrackerTests()
{
_sink = new FakeSignalIngressSink();
_tracker = MakeTracker(stateDirectory: null);
}
public void Dispose() => _tmp.Dispose();
private MdmRebootPolicyTracker MakeTracker(string? stateDirectory)
{
var post = new InformationalEventPost(_sink, new VirtualClock(At));
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
var tracker = new MdmRebootPolicyTracker(
sessionId: "sess-mdm",
tenantId: "tenant-mdm",
post: post,
logger: logger,
backfillEnabled: false,
stateDirectory: stateDirectory,
debounceMilliseconds: 0); // manual-flush mode — tests call FlushPending explicitly
tracker.RebootRequiredFlagProbe = () => null; // registry untouched by default
return tracker;
}
private void Buffer(string? rebootUri, long? recordId = null, bool isBackfill = false, DateTime? timeCreatedUtc = null, string? description = null)
{
_tracker.ProcessEvent(
eventId: MdmRebootPolicyTracker.EventId_PolicyRebootRequired,
recordId: recordId ?? _nextRecordId++,
timeCreatedUtc: timeCreatedUtc ?? At,
rebootUri: rebootUri,
formattedDescription: description,
isBackfill: isBackfill);
}
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> Emitted() =>
_sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == Constants.EventTypes.MdmPolicyRebootRequired).ToList();
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
[Fact]
public void Burst_FlushesAsOneAggregatedEvent_InfoSeverity_ImmediateUpload()
{
Buffer(UriSvchost, timeCreatedUtc: At);
Buffer(UriSystemGuard, timeCreatedUtc: At.AddMilliseconds(120));
_tracker.FlushPending();
var s = Assert.Single(Emitted());
Assert.Equal("Info", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
var data = Data(s);
Assert.Equal(2, data["uriCount"]);
Assert.Equal(2, data["recordCount"]);
Assert.Equal(UriSvchost, data["firstRebootUri"]);
var uris = Assert.IsType<List<string>>(data["rebootUris"]);
Assert.Equal(2, uris.Count);
Assert.Contains(UriSvchost, uris);
Assert.Contains(UriSystemGuard, uris);
}
[Fact]
public void Message_IsNeutralObservation_NoRebootClaim()
{
// Session b2e890c1: flags raised during AccountSetup have no effect — the raw event
// must state the observation only, never a future consequence.
Buffer(UriSvchost);
_tracker.FlushPending();
var s = Assert.Single(Emitted());
var message = s.Payload![SignalPayloadKeys.Message];
Assert.Contains("flagged", message);
Assert.DoesNotContain("will restart", message);
Assert.DoesNotContain("second sign-in", message);
}
[Fact]
public void FlushPending_Empty_EmitsNothing()
{
_tracker.FlushPending();
Assert.Empty(Emitted());
}
[Fact]
public void AggregateTimestamp_IsEarliestRecordTime()
{
var early = At.AddMinutes(-30);
Buffer(UriSystemGuard, timeCreatedUtc: At, isBackfill: true);
Buffer(UriSvchost, timeCreatedUtc: early, isBackfill: true);
_tracker.FlushPending();
var s = Assert.Single(Emitted());
Assert.Equal(early, s.OccurredAtUtc);
Assert.Equal(true, Data(s)["backfilled"]);
// firstRebootUri follows record-time order, not buffer order.
Assert.Equal(UriSvchost, Data(s)["firstRebootUri"]);
}
[Fact]
public void Backfilled_False_WhenAnyRecordIsLive()
{
Buffer(UriSvchost, isBackfill: true);
Buffer(UriSystemGuard, isBackfill: false);
_tracker.FlushPending();
Assert.Equal(false, Data(Assert.Single(Emitted()))["backfilled"]);
}
[Fact]
public void DuplicateUris_AreDistinctInAggregate()
{
Buffer(UriSvchost);
Buffer(UriSvchost);
_tracker.FlushPending();
var data = Data(Assert.Single(Emitted()));
Assert.Equal(1, data["uriCount"]);
Assert.Equal(2, data["recordCount"]);
}
[Fact]
public void UnparsedRecords_CountedWithSampleDescription_StillEmitted()
{
Buffer(rebootUri: null, description: "Some unexpected 2800 wording");
_tracker.FlushPending();
var s = Assert.Single(Emitted());
var data = Data(s);
Assert.False(data.ContainsKey("rebootUris"));
Assert.False(data.ContainsKey("firstRebootUri"));
Assert.Equal(0, data["uriCount"]);
Assert.Equal(1, data["unparsedCount"]);
Assert.Equal("Some unexpected 2800 wording", data["sampleDescription"]);
}
[Fact]
public void SameRecordId_BufferedTwice_CountsOnce()
{
Buffer(UriSvchost, recordId: 42);
Buffer(UriSvchost, recordId: 42);
_tracker.FlushPending();
Assert.Equal(1, Data(Assert.Single(Emitted()))["recordCount"]);
}
[Fact]
public void RecordAfterFlush_GoesIntoNextAggregate()
{
Buffer(UriSvchost, recordId: 10);
_tracker.FlushPending();
Buffer(UriSystemGuard, recordId: 11);
_tracker.FlushPending();
Assert.Equal(2, Emitted().Count);
}
[Fact]
public void BackfillRecordWithLowerRecordId_AfterLiveRecord_IsStillBuffered()
{
// Live watcher is armed BEFORE the backfill scan — an older, never-seen backfill
// record must not be suppressed by a high-water mark.
Buffer(UriSvchost, recordId: 100);
Buffer(UriSystemGuard, recordId: 50, isBackfill: true);
_tracker.FlushPending();
Assert.Equal(2, Data(Assert.Single(Emitted()))["recordCount"]);
}
[Fact]
public void NegativeRecordId_IsNeverDeduped()
{
Buffer(UriSvchost, recordId: -1);
Buffer(UriSvchost, recordId: -1);
_tracker.FlushPending();
Assert.Equal(2, Data(Assert.Single(Emitted()))["recordCount"]);
}
[Fact]
public void Watermark_PersistedAtFlush_SkipsReReadAcrossInstances()
{
// The coalesced reboot restarts the agent; the post-reboot backfill re-reads the same
// records — a FLUSHED record must be skipped by the next instance.
var first = MakeTracker(stateDirectory: _tmp.Path);
first.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
timeCreatedUtc: At, rebootUri: UriSvchost, formattedDescription: null, isBackfill: false);
first.FlushPending();
var second = MakeTracker(stateDirectory: _tmp.Path);
second.LoadWatermark();
second.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
timeCreatedUtc: At, rebootUri: UriSvchost, formattedDescription: null, isBackfill: true);
second.FlushPending();
Assert.Single(Emitted());
}
[Fact]
public void Watermark_NotPersisted_WhenBufferedButNeverFlushed()
{
// Process killed by the very reboot the records announce: nothing was flushed, so the
// next instance must re-read and emit the records.
var first = MakeTracker(stateDirectory: _tmp.Path);
first.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
timeCreatedUtc: At, rebootUri: UriSvchost, formattedDescription: null, isBackfill: false);
// no FlushPending — simulated reboot kill
var second = MakeTracker(stateDirectory: _tmp.Path);
second.LoadWatermark();
second.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
timeCreatedUtc: At, rebootUri: UriSvchost, formattedDescription: null, isBackfill: true);
second.FlushPending();
var s = Assert.Single(Emitted());
Assert.Equal(true, Data(s)["backfilled"]);
}
[Fact]
public void RebootRequiredFlagProbe_True_AddsField_Null_OmitsIt()
{
_tracker.RebootRequiredFlagProbe = () => true;
Buffer(UriSvchost, recordId: 1);
_tracker.FlushPending();
_tracker.RebootRequiredFlagProbe = () => null;
Buffer(UriSystemGuard, recordId: 2);
_tracker.FlushPending();
var emitted = Emitted();
Assert.Equal(2, emitted.Count);
Assert.Equal(true, Data(emitted[0])["omadmRebootRequiredFlag"]);
Assert.False(Data(emitted[1]).ContainsKey("omadmRebootRequiredFlag"));
}
[Fact]
public void RebootRequiredFlagProbe_Throwing_IsFailSoft()
{
_tracker.RebootRequiredFlagProbe = () => throw new InvalidOperationException("boom");
Buffer(UriSvchost);
_tracker.FlushPending();
Assert.False(Data(Assert.Single(Emitted())).ContainsKey("omadmRebootRequiredFlag"));
}
[Fact]
public void BuildXPath_Targets2800()
{
Assert.Equal("*[System[(EventID=2800)]]", MdmRebootPolicyTracker.BuildXPath());
}
// ------------------------------------------------------------------ URI extraction
[Fact]
public void ExtractRebootUri_FromVerifiedRealDescription()
{
// Real 2800 text wraps the URI in parentheses and ends with a period — both must be
// excluded from the extracted URI.
var uri = MdmRebootPolicyTracker.ExtractRebootUri(null, RealDescription);
Assert.Equal(UriSvchost, uri);
}
[Fact]
public void ExtractRebootUri_PrefersEventDataValue_OverDescription()
{
var eventData = new Dictionary<string, string> { ["Data0"] = " " + UriSystemGuard + " " };
var uri = MdmRebootPolicyTracker.ExtractRebootUri(eventData, RealDescription);
Assert.Equal(UriSystemGuard, uri);
}
[Fact]
public void ExtractRebootUri_ReturnsNull_WhenNothingUriLike()
{
Assert.Null(MdmRebootPolicyTracker.ExtractRebootUri(
new Dictionary<string, string> { ["Data0"] = "plain" }, "no uri here"));
Assert.Null(MdmRebootPolicyTracker.ExtractRebootUri(null, null));
}
[Fact]
public void ParseEventData_CollectsNamedAndPositionalData()
{
const string xml =
"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>" +
"<System><EventID>2800</EventID></System>" +
"<EventData><Data Name='Uri'>./Device/Vendor/MSFT/X</Data><Data>positional</Data></EventData>" +
"</Event>";
var parsed = MdmRebootPolicyTracker.ParseEventData(xml);
Assert.Equal("./Device/Vendor/MSFT/X", parsed["Uri"]);
Assert.Equal("positional", parsed["Data0"]);
}
}
}
@@ -39,7 +39,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
public List<EspFailureDetectedEventArgs> EspFailures { get; } = new List<EspFailureDetectedEventArgs>();
public ProvisioningStatusTracker Tracker { get; }
public Fixture(Func<IReadOnlyList<AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState>> packageStatesProbe = null)
public Fixture(Func<IReadOnlyList<AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState>>? packageStatesProbe = null)
{
var clock = new VirtualClock(Fixed);
var post = new InformationalEventPost(Sink, clock);
@@ -179,7 +179,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
}");
var captured = f.CapturedEvents();
Assert.Single(captured.Where(e => e.EventType == "esp_failure_settle_started"));
Assert.Single(captured, e => e.EventType == "esp_failure_settle_started");
}
// =====================================================================
@@ -214,8 +214,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
// No terminal fire — the failure was retracted.
Assert.Empty(f.EspFailures);
var recovered = Assert.Single(f.CapturedEvents()
.Where(e => e.EventType == "esp_failure_settle_recovered"));
var recovered = Assert.Single(
f.CapturedEvents(), e => e.EventType == "esp_failure_settle_recovered");
Assert.Equal("AccountSetup", (string)recovered.Data["category"]);
Assert.Equal("Apps", (string)recovered.Data["failedSubcategory"]);
Assert.Equal("inProgress", (string)recovered.Data["observedState"]);
@@ -280,7 +280,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
f.Tracker.TriggerSettleTimerForTest("AccountSetupCategory.Status");
Assert.Single(f.EspFailures);
Assert.Empty(f.CapturedEvents().Where(e => e.EventType == "esp_failure_settle_recovered"));
Assert.DoesNotContain(f.CapturedEvents(), e => e.EventType == "esp_failure_settle_recovered");
}
// =====================================================================
@@ -326,8 +326,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
var settleStarted = Assert.Single(f.CapturedEvents()
.Where(e => e.EventType == "esp_failure_settle_started"));
var settleStarted = Assert.Single(
f.CapturedEvents(), e => e.EventType == "esp_failure_settle_started");
Assert.Equal(1, (int)settleStarted.Data["trackedAppsNotCompletedCount"]);
var apps = (List<Dictionary<string, object>>)settleStarted.Data["trackedAppsNotCompleted"];
var app = Assert.Single(apps);
@@ -354,8 +354,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
""SecurityPoliciesSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Failed""}
}");
var settleStarted = Assert.Single(f.CapturedEvents()
.Where(e => e.EventType == "esp_failure_settle_started"));
var settleStarted = Assert.Single(
f.CapturedEvents(), e => e.EventType == "esp_failure_settle_started");
Assert.False(settleStarted.Data.ContainsKey("trackedAppsNotCompleted"));
Assert.False(settleStarted.Data.ContainsKey("trackedAppsNotCompletedCount"));
}
@@ -0,0 +1,143 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
{
/// <summary>
/// Pins the anti-flood semantics of <see cref="EmergencyReporter.TrySendAsync"/> after the
/// delivery-hardening change (2026-07-23): the once-per-session dedup and the per-session
/// budget count DELIVERED reports. The previous mark-before-send permanently suppressed an
/// error key after its first transport failure — fatal for single-shot callers like the 48h
/// emergency break, whose process exits right after the attempt.
/// </summary>
public class EmergencyReporterTests
{
private sealed class FakeApiClient : BackendApiClient
{
public readonly List<TimeSpan?> Timeouts = new List<TimeSpan?>();
public readonly Queue<bool> Results = new Queue<bool>();
public FakeApiClient() : base(
httpClient: new HttpClient(),
baseUrl: "http://localhost",
manufacturer: string.Empty,
model: string.Empty,
serialNumber: string.Empty,
useBootstrapTokenAuth: false,
bootstrapToken: null,
agentVersion: "0.0.0",
logger: null)
{
}
public int Calls => Timeouts.Count;
public override Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
Timeouts.Add(timeout);
return Task.FromResult(Results.Count > 0 && Results.Dequeue());
}
}
private static EmergencyReporter Reporter(FakeApiClient client)
=> new EmergencyReporter(client, "session", "tenant", "0.0.0", logger: null);
[Fact]
public async Task Failed_send_rolls_back_reservation_so_a_later_call_retries()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // first call fails
client.Results.Enqueue(true); // second call succeeds
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
// Old behaviour: second call suppressed as "already reported" ⇒ 1 call.
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Delivered_send_is_deduplicated_per_key()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(1, client.Calls);
}
[Fact]
public async Task Retries_stay_inside_one_reservation_and_stop_on_success()
{
var client = new FakeApiClient();
client.Results.Enqueue(false);
client.Results.Enqueue(true); // second attempt delivers — third must not fire
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, perAttemptTimeout: TimeSpan.FromMilliseconds(50), retryDelay: TimeSpan.Zero);
Assert.Equal(2, client.Calls);
// Delivered ⇒ dedup holds for the session.
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Per_attempt_timeout_is_forwarded_to_the_api_client()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
var timeout = TimeSpan.FromSeconds(15);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 1, perAttemptTimeout: timeout);
Assert.Equal(timeout, Assert.Single(client.Timeouts));
}
[Fact]
public async Task All_attempts_failing_rolls_back_after_exhausting_attempts()
{
var client = new FakeApiClient(); // queue empty ⇒ every attempt fails
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, retryDelay: TimeSpan.Zero);
Assert.Equal(3, client.Calls);
// Rolled back ⇒ a later call may try again (budget + dedup + cooldown restored).
client.Results.Enqueue(true);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(4, client.Calls);
}
[Fact]
public async Task Different_keys_do_not_suppress_each_other_after_rollback()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // break report fails → rollback (incl. cooldown restore)
client.Results.Enqueue(true); // unrelated report must still go out
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.ConfigFetchFailed, "m");
Assert.Equal(2, client.Calls);
}
}
}
@@ -0,0 +1,212 @@
#nullable enable
using System.Collections.Generic;
using System.Text;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Pins the pure parts of the UEFI Secure Boot certificate probe: the ASCII substring
/// search over raw signature-list bytes and the <c>secureboot_status</c> payload shaping
/// (in particular the one-sided <c>uefiCa2023FirmwareConfirmed</c> marker whose ABSENCE
/// is load-bearing for ANALYZE-SEC-001's not_exists precondition). The actual firmware
/// read is only smoke-tested: test machines may be VMs, non-elevated, or legacy BIOS,
/// so only the fail-soft contract is asserted.
/// </summary>
public sealed class UefiSecureBootCertReaderTests
{
// --- ContainsAscii ---
[Fact]
public void ContainsAscii_finds_needle_embedded_in_binary_noise()
{
var bytes = Concat(new byte[] { 0x00, 0xA1, 0x30, 0x82 },
Encoding.ASCII.GetBytes("Windows UEFI CA 2023"),
new byte[] { 0x00, 0xFF });
Assert.True(UefiSecureBootCertReader.ContainsAscii(bytes, "Windows UEFI CA 2023"));
}
[Fact]
public void ContainsAscii_finds_needle_at_the_very_end()
{
var bytes = Concat(new byte[] { 0x01, 0x02 }, Encoding.ASCII.GetBytes("KEK 2K CA 2023"));
Assert.True(UefiSecureBootCertReader.ContainsAscii(bytes, "KEK 2K CA 2023"));
}
[Fact]
public void ContainsAscii_rejects_partial_overlap_and_absent_needle()
{
var bytes = Encoding.ASCII.GetBytes("Windows UEFI CA 2011 something else");
Assert.False(UefiSecureBootCertReader.ContainsAscii(bytes, "Windows UEFI CA 2023"));
// Repeated prefix bytes must not confuse the scan.
var tricky = Encoding.ASCII.GetBytes("ababac");
Assert.True(UefiSecureBootCertReader.ContainsAscii(tricky, "abac"));
}
[Fact]
public void ContainsAscii_handles_null_empty_and_oversized_needle()
{
Assert.False(UefiSecureBootCertReader.ContainsAscii(null, "x"));
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[0], "x"));
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[] { 1, 2 }, ""));
Assert.False(UefiSecureBootCertReader.ContainsAscii(new byte[] { 1, 2 }, "abc"));
}
// --- Payload shaping ---
[Fact]
public void Marker_emitted_only_when_db_ok_and_ca2023_present()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusOk,
KekStatus = UefiSecureBootCertReader.StatusOk,
DbHasWindowsUefiCa2023 = true,
KekHasMicrosoftKek2kCa2023 = true
});
Assert.Equal(true, data["uefiCa2023FirmwareConfirmed"]);
Assert.Equal("ok", data["uefiFirmwareReadStatus"]);
Assert.Equal(true, data["uefiDbHasWindowsUefiCa2023"]);
Assert.Equal(true, data["uefiKekHasMicrosoftKek2kCa2023"]);
}
[Fact]
public void Marker_absent_when_ca2023_missing_even_though_read_succeeded()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusOk,
KekStatus = UefiSecureBootCertReader.StatusOk,
DbHasWindowsUefiCa2023 = false,
DbHasWindowsProductionPca2011 = true
});
// The marker must be OMITTED, not false — not_exists fails on any present value.
Assert.False(data.ContainsKey("uefiCa2023FirmwareConfirmed"));
Assert.Equal(false, data["uefiDbHasWindowsUefiCa2023"]);
Assert.Equal(true, data["uefiDbHasWindowsProductionPca2011"]);
}
[Fact]
public void Failed_db_read_emits_status_but_no_cert_booleans_and_no_marker()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusPrivilegeDenied,
KekStatus = UefiSecureBootCertReader.StatusPrivilegeDenied
});
Assert.Equal(UefiSecureBootCertReader.StatusPrivilegeDenied, data["uefiFirmwareReadStatus"]);
Assert.False(data.ContainsKey("uefiDbHasWindowsUefiCa2023"));
Assert.False(data.ContainsKey("uefiKekHasMicrosoftKek2kCa2023"));
Assert.False(data.ContainsKey("uefiCa2023FirmwareConfirmed"));
}
[Fact]
public void Db_ok_but_kek_failed_keeps_db_booleans_and_reports_kek_status()
{
var data = new Dictionary<string, object>();
DeviceInfoCollector.AppendFirmwareCertFields(data, new UefiSecureBootCertSnapshot
{
DbStatus = UefiSecureBootCertReader.StatusOk,
KekStatus = UefiSecureBootCertReader.StatusVariableNotFound,
DbHasWindowsUefiCa2023 = true
});
Assert.Equal(UefiSecureBootCertReader.StatusVariableNotFound, data["uefiFirmwareReadStatus"]);
Assert.Equal(true, data["uefiDbHasWindowsUefiCa2023"]);
Assert.Equal(true, data["uefiCa2023FirmwareConfirmed"]);
Assert.False(data.ContainsKey("uefiKekHasMicrosoftKek2kCa2023"));
}
[Fact]
public void Null_snapshot_leaves_payload_untouched()
{
var data = new Dictionary<string, object> { { "uefiSecureBootEnabled", true } };
DeviceInfoCollector.AppendFirmwareCertFields(data, null);
Assert.Single(data);
}
[Fact]
public void DescribeFirmwareVerdict_covers_all_shapes()
{
Assert.Equal("unavailable", DeviceInfoCollector.DescribeFirmwareVerdict(null));
Assert.Equal(UefiSecureBootCertReader.StatusNotUefi, DeviceInfoCollector.DescribeFirmwareVerdict(
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusNotUefi }));
Assert.Equal("CA2023 confirmed", DeviceInfoCollector.DescribeFirmwareVerdict(
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusOk, DbHasWindowsUefiCa2023 = true }));
Assert.Equal("CA2023 missing", DeviceInfoCollector.DescribeFirmwareVerdict(
new UefiSecureBootCertSnapshot { DbStatus = UefiSecureBootCertReader.StatusOk, DbHasWindowsUefiCa2023 = false }));
}
// --- Reader smoke (real firmware; machine state unknown) ---
[Fact]
public void Read_never_throws_and_reports_a_status_for_both_variables()
{
var snapshot = UefiSecureBootCertReader.Read();
Assert.NotNull(snapshot);
Assert.False(string.IsNullOrEmpty(snapshot.DbStatus));
Assert.False(string.IsNullOrEmpty(snapshot.KekStatus));
}
/// <summary>
/// SeSystemEnvironmentPrivilege is adjusted on the PROCESS token, so the enable →
/// read → restore sequence is a process-wide side effect. Unserialized, a second
/// reader observes "already enabled", skips its own restore, and then loses the
/// privilege mid-read when the first reader disposes — surfacing as a privilege_denied
/// that disagrees with the other readers. The machine's actual privilege state does
/// not matter here: whatever it is, every reader must report the SAME status.
/// </summary>
[Fact]
public void Read_is_serialized_across_concurrent_callers()
{
const int readers = 8;
var snapshots = new UefiSecureBootCertSnapshot[readers];
var failures = new System.Exception[readers];
var threads = new System.Threading.Thread[readers];
for (var i = 0; i < readers; i++)
{
var index = i;
threads[i] = new System.Threading.Thread(() =>
{
try { snapshots[index] = UefiSecureBootCertReader.Read(); }
catch (System.Exception ex) { failures[index] = ex; }
});
threads[i].Start();
}
foreach (var thread in threads) thread.Join();
Assert.All(failures, Assert.Null);
Assert.All(snapshots, s => Assert.False(string.IsNullOrEmpty(s.DbStatus)));
Assert.All(snapshots, s => Assert.Equal(snapshots[0].DbStatus, s.DbStatus));
Assert.All(snapshots, s => Assert.Equal(snapshots[0].KekStatus, s.KekStatus));
}
private static byte[] Concat(params byte[][] parts)
{
var total = 0;
foreach (var part in parts) total += part.Length;
var result = new byte[total];
var offset = 0;
foreach (var part in parts)
{
System.Array.Copy(part, 0, result, offset, part.Length);
offset += part.Length;
}
return result;
}
}
}
@@ -44,7 +44,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
private readonly FakeQuarantineSink _quarantine = new FakeQuarantineSink();
private readonly FakeSignalIngressSink _sink = new FakeSignalIngressSink();
private readonly DecisionEngine _engine = new DecisionEngine();
private DecisionStepProcessor _sut;
private DecisionStepProcessor? _sut;
public DecisionStepProcessorParkedTripwireTests()
{
@@ -107,7 +107,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
return (step, signal);
}
private FakeSignalIngressSink.PostedSignal FindTripwirePost()
private FakeSignalIngressSink.PostedSignal? FindTripwirePost()
{
foreach (var posted in _sink.Posted)
{
@@ -121,6 +121,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
return null;
}
/// <summary>The tripwire post, asserted present — for the assertions that only run
/// after <c>WaitForTripwire()</c> has already confirmed one was emitted.</summary>
private FakeSignalIngressSink.PostedSignal RequireTripwirePost()
{
var post = FindTripwirePost();
Assert.NotNull(post);
return post!;
}
private int CountTripwirePosts()
{
var count = 0;
@@ -165,9 +174,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
Thread.Sleep(SuppressWait);
Assert.Equal(1, CountTripwirePosts());
var post = FindTripwirePost();
var post = RequireTripwirePost();
Assert.Equal(DecisionSignalKind.InformationalEvent, post.Kind);
Assert.Equal(nameof(SessionStage.EspAccountSetup), post.Payload["stage"]);
Assert.Equal(nameof(SessionStage.EspAccountSetup), post.Payload!["stage"]);
Assert.Equal("Warning", post.Payload[SignalPayloadKeys.Severity]);
Assert.Equal("true", post.Payload[SignalPayloadKeys.ImmediateUpload]);
Assert.Contains("esp_final_exit", post.Payload["signalsSeen"]);
@@ -222,7 +231,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
sut.ApplyStep(step, signal);
Assert.True(WaitForTripwire());
Assert.Equal(DeadlineNames.ClassifierTick, FindTripwirePost().Payload["armedDeadlines"]);
Assert.Equal(DeadlineNames.ClassifierTick, RequireTripwirePost().Payload!["armedDeadlines"]);
}
[Fact]
@@ -237,7 +246,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
sut.ApplyStep(step, signal);
Assert.True(WaitForTripwire());
Assert.Equal(nameof(SessionStage.AwaitingDesktop), FindTripwirePost().Payload["stage"]);
Assert.Equal(nameof(SessionStage.AwaitingDesktop), RequireTripwirePost().Payload!["stage"]);
}
// ===================================================================== Dwell semantics
@@ -37,5 +37,41 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
Assert.False(DeviceInfoHost.IsEndTrigger(DecisionSignalKind.EspPhaseChanged, null));
Assert.False(DeviceInfoHost.IsEndTrigger(DecisionSignalKind.InformationalEvent, Phase("FinalizingSetup")));
}
// Audit Q2 — the payload shape the adapter's apps sub-phase declaration produces on the
// ingress rail (InformationalEventPost: eventType + declared phase enum name).
private static IReadOnlyDictionary<string, string> SubPhaseDeclaration(string declaredPhase) =>
new Dictionary<string, string>
{
{ SignalPayloadKeys.EventType, AutopilotMonitor.Shared.Constants.EventTypes.PhaseTransition },
{ AutopilotMonitor.Agent.V2.Core.Telemetry.Events.EventTimelineEmitter.PhaseParamKey, declaredPhase },
};
[Fact]
public void Apps_subphase_declarations_classify_as_esp_config_refresh_triggers()
{
Assert.Equal(DeviceInfoHost.EspConfigTriggerAppsDevice,
DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent, SubPhaseDeclaration("AppsDevice")));
Assert.Equal(DeviceInfoHost.EspConfigTriggerAppsUser,
DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent, SubPhaseDeclaration("AppsUser")));
// Non-apps phase declarations (e.g. the reducer's Finalizing entry) never classify.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent, SubPhaseDeclaration("FinalizingSetup")));
// A phase_transition without a declared phase never classifies.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent,
new Dictionary<string, string> { { SignalPayloadKeys.EventType, AutopilotMonitor.Shared.Constants.EventTypes.PhaseTransition } }));
}
[Fact]
public void AccountSetup_phase_classifies_as_the_user_setup_refresh_trigger()
{
Assert.Equal(DeviceInfoHost.EspConfigTriggerAccountSetup,
DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.EspPhaseChanged, Phase("AccountSetup")));
// DeviceSetup is the enrollment-start trigger, not an ESP-config refresh.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.EspPhaseChanged, Phase("DeviceSetup")));
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.EspPhaseChanged, null));
// Unrelated informational events never classify.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent,
new Dictionary<string, string> { { SignalPayloadKeys.EventType, "esp_config_detected" } }));
}
}
}
@@ -256,8 +256,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
RemoteConfigFetchOutcome outcome,
int configVersion = 0,
int attempts = 1,
string failureType = null,
string failureMessage = null,
string? failureType = null,
string? failureMessage = null,
int? authStatusCode = null)
{
// CurrentConfig is set via the protected SetConfig path — easiest:
@@ -0,0 +1,33 @@
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
{
/// <summary>
/// Audit Q5 — WMI hardware values are trimmed at the source. BIOS serials become device
/// identity keys downstream (F2 device history joins on TenantId + serial), so padding
/// must never leave the agent; whitespace-only values collapse to the "Unknown" sentinel
/// the retry logic already treats as unresolved.
/// </summary>
public sealed class HardwareInfoNormalizeTests
{
[Theory]
[InlineData(" PF60H941 ", "PF60H941")]
[InlineData("PF60H941", "PF60H941")]
[InlineData("\tLENOVO \r\n", "LENOVO")]
[InlineData("Unknown", "Unknown")]
public void Trims_whitespace_at_the_source(string raw, string expected)
=> Assert.Equal(expected, HardwareInfo.NormalizeWmiValue(raw));
[Theory]
[InlineData("")]
[InlineData(" ")]
[InlineData("\t\r\n")]
public void Empty_and_whitespace_only_collapse_to_the_Unknown_sentinel(string raw)
=> Assert.Equal("Unknown", HardwareInfo.NormalizeWmiValue(raw));
[Fact]
public void Null_collapses_to_the_Unknown_sentinel()
=> Assert.Equal("Unknown", HardwareInfo.NormalizeWmiValue(null));
}
}
@@ -33,7 +33,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
[InlineData("Network connections (1 of 1 added)", null)]
[InlineData("", null)]
[InlineData(null, null)]
public void TryExtractErrorCode_extracts_parenthesised_hex_tail(string statusText, string expected)
public void TryExtractErrorCode_extracts_parenthesised_hex_tail(string? statusText, string? expected)
{
var actual = ProvisioningStatusTracker.TryExtractErrorCode(statusText);
Assert.Equal(expected, actual);
@@ -0,0 +1,157 @@
using System;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.SignalAdapters;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared.Models;
using SharedEventTypes = AutopilotMonitor.Shared.Constants.EventTypes;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
{
/// <summary>
/// Historic-replay suppression for APP events (session eaf3d8c4 part 2): 147 of 157 app
/// events in that session were replayed from a previous enrollment's IME log — all 10
/// "installed apps" were phantoms. Stale app-state transitions and DO telemetry must be
/// suppressed BEFORE the sub-phase declaration, the timing bookkeeping and the terminal
/// DecisionSignal dedup, so fire-once flags stay unconsumed and no phantom
/// AppInstallCompleted/Failed reaches the engine (which could spuriously re-arm the
/// AdvisoryCompletion window). The script twin lives in
/// <see cref="ImeLogTrackerAdapterHistoricReplayTests"/>.
/// </summary>
public sealed class ImeLogTrackerAdapterHistoricAppReplayTests
{
private static readonly DateTime ClockNow = new DateTime(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
[Fact]
public void Stale_terminal_app_transition_is_fully_suppressed()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-old", 0);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Installed);
Assert.Empty(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Empty(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted));
// Shadow download_progress (status=completed) and the summary snapshot ride the
// same emit — both must be gone too.
Assert.Empty(f.InfoEvents(SharedEventTypes.DownloadProgress));
Assert.Empty(f.InfoEvents(SharedEventTypes.AppTrackingSummary));
// Timing bookkeeping must not be poisoned (feeds FinalStatusBuilder).
Assert.False(adapter.AppTimings.ContainsKey("app-old"));
Assert.Single(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void Fresh_run_after_suppressed_stale_run_fires_signal_timing_and_subphase()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-X", 0);
// Fresh ESP phase so the sub-phase declaration is armed.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-5);
adapter.TriggerEspPhaseFromTest("DeviceSetup");
// Stale replay of the same app — suppressed, must consume NEITHER the sub-phase
// fire-once NOR the terminal-signal dedup for this appId.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Installed);
Assert.Empty(f.InfoEvents(SharedEventTypes.PhaseTransition));
Assert.Empty(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted));
// The app genuinely runs in THIS enrollment.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-2);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Installing);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-1);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Installed);
Assert.Single(f.InfoEvents(SharedEventTypes.PhaseTransition)); // sub-phase fired fresh
Assert.Single(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted)); // signal fired fresh
var timing = adapter.AppTimings["app-X"];
Assert.Equal(ClockNow.AddMinutes(-2), timing.StartedAtUtc); // fresh stamps, not stale
Assert.Equal(ClockNow.AddMinutes(-1), timing.CompletedAtUtc);
}
[Fact]
public void Stale_do_telemetry_is_suppressed_fresh_emits()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-do", 0);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerDoTelemetryFromTest(app);
Assert.Empty(f.InfoEvents(SharedEventTypes.DoTelemetry));
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-1);
adapter.TriggerDoTelemetryFromTest(app);
Assert.Single(f.InfoEvents(SharedEventTypes.DoTelemetry));
}
[Fact]
public void Oneshot_summary_is_shared_between_script_and_app_suppressions()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "aaa11111",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
adapter.TriggerAppStateFromTest(new AppPackageState("app-old", 0),
AppInstallationState.Installing, AppInstallationState.Installed);
Assert.Single(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void FutureSkew_app_transition_still_emits()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-skew", 0);
// Mid-enrollment clock jump — genuine current activity, stamped to the clock.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddHours(2);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Installed);
var info = Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Equal(ClockNow, info.OccurredAtUtc);
Assert.Equal("true", info.Payload!["derivedTimestamp"]);
Assert.Single(f.DecisionSignals(DecisionSignalKind.AppInstallCompleted));
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void SimulationMode_bypasses_suppression_for_apps_and_scripts()
{
// The --replay-log-dir dev tool intentionally feeds historic lines.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.SimulationMode = true;
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerAppStateFromTest(new AppPackageState("app-sim", 0),
AppInstallationState.Installing, AppInstallationState.Installed);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "bbb22222",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
}
}
@@ -0,0 +1,257 @@
using System;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.SignalAdapters;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using SharedEventTypes = AutopilotMonitor.Shared.Constants.EventTypes;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
{
/// <summary>
/// Historic-replay handling (session eaf3d8c4): a previous enrollment's IME log surviving
/// on disk made the agent replay week-old script activity — 156 phantom script_completed
/// events with ~170 h durations (clock-clamped completion minus RAW stale start) and an
/// immediate-upload flood. Script events whose source line is &gt; 24 h stale are now
/// suppressed entirely (one-shot <c>historic_ime_replay_detected</c> summary instead),
/// and durations are computed from a timeline-consistent timestamp pair with a 24 h
/// plausibility backstop.
/// </summary>
public sealed class ImeLogTrackerAdapterHistoricReplayTests
{
private static readonly DateTime ClockNow = new DateTime(2026, 7, 23, 15, 42, 0, DateTimeKind.Utc);
// ---------------------------------------------------------------------
// Suppression: stale (> 24 h past) source lines are a previous enrollment
// ---------------------------------------------------------------------
[Fact]
public void Stale_completion_is_suppressed_and_emits_oneshot_summary()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var ancient = ClockNow.AddDays(-7);
f.Tracker.LastMatchedLogTimestamp = ancient;
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "d78c1822",
ScriptType = "platform",
ExitCode = 0,
Result = "Failed",
StartedAtUtc = ancient.AddMinutes(-30),
// Bootstrap marker in stdout — week-old evidence must not drive the
// bootstrap-detected trace either.
Stdout = "Bootstrap script version: v2.0",
});
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptFailed));
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptTimeoutSuspected));
Assert.Empty(f.InfoEvents(SharedEventTypes.AgentTrace));
var summary = f.InfoEvent(SharedEventTypes.HistoricImeReplayDetected);
Assert.Equal(ancient.ToString("o"), summary.Payload!["earliestRejectedSourceTimestamp"]);
}
[Fact]
public void Summary_is_oneshot_across_started_and_completed_suppressions()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerScriptStartedFromTest(new ScriptStartedInfo { PolicyId = "aaa11111", ScriptType = "platform" });
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7).AddMinutes(1);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "aaa11111",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7).AddMinutes(2);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "bbb22222",
ScriptType = "remediation",
ScriptPart = "detection",
ExitCode = 0,
ComplianceResult = "True",
});
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptStarted));
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptCompleted));
// Exactly ONE summary regardless of how many replayed lines followed — and it dates
// the replay window from the FIRST (earliest) suppressed line.
var summary = Assert.Single(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
Assert.Equal(ClockNow.AddDays(-7).ToString("o"), summary.Payload!["earliestRejectedSourceTimestamp"]);
}
[Fact]
public void Fresh_events_after_suppression_emit_normally()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddDays(-7);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "aaa11111",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptCompleted));
// The agent catches up to current log content — normal emission resumes.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-2);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "ccc33333",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
StartedAtUtc = ClockNow.AddMinutes(-3),
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Equal("ccc33333", info.Payload!["policyId"]);
Assert.Equal("60.00", info.Payload["durationSeconds"]);
}
[Fact]
public void Clock_fallback_without_source_timestamp_still_emits()
{
// derivedFromClock with NO rejected source ts (synthetic/callback path) is not a
// replay — must emit, and must not trigger the summary.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "ddd44444",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Equal("true", info.Payload!["derivedTimestamp"]);
Assert.False(info.Payload.ContainsKey("rejectedSourceTimestamp"));
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
// ---------------------------------------------------------------------
// Duration: timeline-consistent pair + plausibility backstop
// ---------------------------------------------------------------------
[Fact]
public void FutureSkew_completion_emits_with_clamped_stamp_and_raw_pair_duration()
{
// Mid-enrollment clock jump (WhiteGlove +1h CMTrace skew case): the source ts is
// rejected as future-skewed — NOT suppressed — and the duration must come from the
// raw source pair (both ends on the same skewed timeline), not clamped-now − raw-start.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var skewed = ClockNow.AddHours(2);
f.Tracker.LastMatchedLogTimestamp = skewed;
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "eee55555",
ScriptType = "platform",
ExitCode = 0,
Result = "Success",
StartedAtUtc = skewed.AddSeconds(-360),
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptCompleted));
Assert.Equal(ClockNow, info.OccurredAtUtc); // stamp: clamped to clock
Assert.Equal("true", info.Payload!["derivedTimestamp"]);
Assert.Equal(skewed.ToString("o"), info.Payload["rejectedSourceTimestamp"]);
Assert.Equal("360.00", info.Payload["durationSeconds"]); // duration: raw pair
Assert.Empty(f.InfoEvents(SharedEventTypes.HistoricImeReplayDetected));
}
[Fact]
public void FutureSkew_failed_platform_script_timeout_heuristic_uses_raw_pair()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var skewed = ClockNow.AddHours(2);
f.Tracker.LastMatchedLogTimestamp = skewed;
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "fff66666",
ScriptType = "platform",
ExitCode = 0,
Result = "Failed",
StartedAtUtc = skewed.AddMinutes(-26), // past the 25-min suspicion threshold
});
var suspected = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptTimeoutSuspected));
Assert.Equal("1560.00", suspected.Payload!["durationSeconds"]);
Assert.Equal(ClockNow, suspected.OccurredAtUtc);
}
[Fact]
public void Implausible_duration_above_24h_is_omitted()
{
// Fresh (accepted) completion paired with an ancient start — e.g. a stale pending
// slot a legacy path left behind. The duration would be a cross-run lie; omit it.
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddMinutes(-2);
adapter.TriggerScriptCompletedFromTest(new ScriptExecutionState
{
PolicyId = "abc12345",
ScriptType = "platform",
ExitCode = 0,
Result = "Failed",
StartedAtUtc = ClockNow.AddHours(-30),
});
var info = Assert.Single(f.InfoEvents(SharedEventTypes.ScriptFailed));
Assert.False(info.Payload!.ContainsKey("durationSeconds"));
Assert.False(info.Payload.ContainsKey("durationBasis"));
// The 30 h "duration" must not read as a hung script either.
Assert.Empty(f.InfoEvents(SharedEventTypes.ScriptTimeoutSuspected));
}
// ---------------------------------------------------------------------
// Tracker slot hardening: stale pending start must not pair with a fresh run
// ---------------------------------------------------------------------
[Fact]
public void Fresh_start_discards_stale_pending_slot_from_previous_enrollment()
{
using var f = new ImeLogTrackerAdapterFixture(ClockNow);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
// Previous enrollment: replayed start line whose matching result never appeared
// (the script was mid-run when that enrollment ended).
f.Tracker.SeedPendingPlatformScriptForTesting("beef7777", exitCode: null, exitObservedAtUtc: null,
startedAtUtc: ClockNow.AddDays(-7));
// Same policy genuinely re-runs now — the fresh start must begin a NEW execution.
f.Tracker.LastMatchedLogTimestamp = ClockNow.AddSeconds(-90);
f.Tracker.HandlePlatformScriptStarted("beef7777");
f.Tracker.LastMatchedLogTimestamp = ClockNow;
f.Tracker.CompletePlatformScriptFromImeResultForTesting("beef7777", "Success", ClockNow);
var info = f.InfoEvent(SharedEventTypes.ScriptCompleted);
// Without the hardening the slot kept the 7-day-old start: the duration would have
// been dropped by the 24 h backstop (or, before that, shown as ~168 h).
Assert.Equal("90.00", info.Payload!["durationSeconds"]);
}
}
}
@@ -195,7 +195,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
var all = f.InfoEvents(SharedEventTypes.ScriptCompleted);
Assert.Equal(2, all.Count);
Assert.Equal("65.00", all[1].Payload!["durationSeconds"]);
Assert.Equal("cycle_including_reporting_latency", all[1].Payload["durationBasis"]);
Assert.Equal("cycle_including_reporting_latency", all[1].Payload!["durationBasis"]);
}
[Fact]
@@ -243,5 +243,87 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
Assert.Equal(firstStamp, adapter.AppTimings["app-P"].StartedAtUtc);
}
[Fact]
public void Terminal_to_active_retry_rearms_CompletedAt_and_final_terminal_covers_total_occupancy()
{
// Audit Q1: the IME retry path (Error -> Downloading -> Installed is permitted by
// AppPackageState.UpdateState). Pre-fix, CompletedAt froze at the first Error, so
// the final Installed payload carried the first failure's completedAt/duration.
using var f = new ImeLogTrackerAdapterFixture(T0);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-R", 0);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Installing);
f.Clock.Advance(TimeSpan.FromSeconds(5));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Error);
f.Clock.Advance(TimeSpan.FromSeconds(10));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Error, AppInstallationState.Downloading);
// Re-armed while active again: the retry's download event carries no completedAt,
// and the FIRST start stamp is kept.
Assert.Null(adapter.AppTimings["app-R"].CompletedAtUtc);
var retryDownload = f.InfoEvent(SharedEventTypes.AppDownloadStarted);
Assert.False(retryDownload.Payload!.ContainsKey("completedAt"));
Assert.False(retryDownload.Payload.ContainsKey("durationSeconds"));
Assert.Equal(T0.ToString("o"), retryDownload.Payload["startedAt"]);
f.Clock.Advance(TimeSpan.FromSeconds(20));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Downloading, AppInstallationState.Installed);
var timing = adapter.AppTimings["app-R"];
Assert.Equal(T0, timing.StartedAtUtc);
Assert.Equal(T0.AddSeconds(35), timing.CompletedAtUtc);
Assert.Equal(35.0, timing.DurationSeconds);
var completed = Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Equal("35.00", completed.Payload!["durationSeconds"]);
Assert.Equal(T0.AddSeconds(35).ToString("o"), completed.Payload["completedAt"]);
}
[Fact]
public void Skipped_then_retried_app_emits_positive_duration_from_retry_start()
{
// Production regression (session 66f1c5e1, audit Q1): a user-scope app evaluated
// Skipped during DeviceSetup (terminal, no start observed), retried after sign-in
// (Skipped -> Downloading is permitted), Installed later. Pre-fix payload:
// completedAt frozen at the early Skip + startedAt from the retry ->
// durationSeconds=-2342. Post-fix: retry re-arms, duration covers retry-start ->
// final terminal.
using var f = new ImeLogTrackerAdapterFixture(T0);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-S", 0);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Skipped);
Assert.Null(adapter.AppTimings["app-S"].StartedAtUtc);
Assert.Equal(T0, adapter.AppTimings["app-S"].CompletedAtUtc);
f.Clock.Advance(TimeSpan.FromSeconds(60));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Skipped, AppInstallationState.Downloading);
f.Clock.Advance(TimeSpan.FromSeconds(30));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Downloading, AppInstallationState.Installed);
var timing = adapter.AppTimings["app-S"];
Assert.Equal(T0.AddSeconds(60), timing.StartedAtUtc);
Assert.Equal(T0.AddSeconds(90), timing.CompletedAtUtc);
Assert.Equal(30.0, timing.DurationSeconds);
var completedEvents = f.InfoEvents(SharedEventTypes.AppInstallComplete);
Assert.Equal(2, completedEvents.Count);
Assert.Equal("30.00", completedEvents[1].Payload!["durationSeconds"]);
}
[Fact]
public void Inverted_endpoints_yield_null_DurationSeconds()
{
// Negative guard (audit Q1): endpoints stamped from out-of-order source lines must
// never surface as a negative durationSeconds (observed -2342s in production). The
// payload emitter omits the field when DurationSeconds is null.
var inverted = new AppInstallTiming(startedAtUtc: T0.AddSeconds(100), completedAtUtc: T0);
Assert.Null(inverted.DurationSeconds);
var wellOrdered = new AppInstallTiming(startedAtUtc: T0, completedAtUtc: T0.AddSeconds(100));
Assert.Equal(100.0, wellOrdered.DurationSeconds);
}
}
}
@@ -151,17 +151,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
}
[Fact]
public void TriggerPhaseChangedFromTest_emits_informational_only_no_decision_signal()
public void TriggerPhaseChangedFromTest_emits_typed_signal_and_informational_event()
{
using var f = new Fixture();
using var adapter = new RealmJoinWatcherAdapter(f.Watcher, f.Ingress, f.Clock);
adapter.TriggerPhaseChangedFromTest(prev: 100, curr: 200);
// Phase change is observability-only. There is no DecisionSignalKind.RealmJoinPhaseChanged.
// 1) Typed signal — persists LastDeploymentPhase and drives the aborted-RJ-ESP
// detection in the reducer (session 224b2087). Not Detected/Resolved.
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.RealmJoinDetected);
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.RealmJoinResolved);
var decision = f.Ingress.Posted.Single(p => p.Kind == DecisionSignalKind.RealmJoinPhaseChanged);
Assert.Equal("RealmJoinWatcher", decision.SourceOrigin);
Assert.Equal("200", decision.Payload![DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase]);
Assert.Equal("100", decision.Payload[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase]);
// 2) Dual-emit informational event keeps the existing timeline entry.
var info = f.Ingress.Posted.Single(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
@@ -1,4 +1,4 @@
using System;
using System;
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic;
using Xunit;
@@ -19,7 +19,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
private static List<BandwidthJobSample> Jobs(params BandwidthJobSample[] jobs)
=> new List<BandwidthJobSample>(jobs);
private static BandwidthJobSample Job(string fileId, long wan, long lan = 0)
private static BandwidthJobSample Job(string? fileId, long wan, long lan = 0)
=> new BandwidthJobSample { FileId = fileId, WanBytes = wan, LanBytes = lan };
[Fact]
@@ -43,8 +43,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.NotNull(estimate);
Assert.Equal(1, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsMax.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsMax!.Value, 3);
Assert.Equal(6_000_000, estimate.WanBytesObserved);
Assert.Equal("10-50", estimate.Bucket);
}
@@ -58,7 +58,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
estimator.AddSnapshot(T0.AddSeconds(3), Jobs(Job("a", 3_000_000), Job("b", 3_000_000)));
var estimate = estimator.TryBuildEstimate();
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
}
[Fact]
@@ -74,7 +74,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.Equal(2, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsMax.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsMax!.Value, 3);
}
[Fact]
@@ -91,7 +91,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
estimator.AddSnapshot(T0.AddSeconds(63), Jobs(Job("a", 106_000_000)));
var estimate = estimator.TryBuildEstimate();
Assert.Equal(1, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
}
[Fact]
@@ -130,8 +130,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
estimator.AddSnapshot(T0.AddSeconds(3), Jobs(Job("a", wan: 6_000_000, lan: 375_000_000)));
var estimate = estimator.TryBuildEstimate();
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(1000.0, estimate.LanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
Assert.Equal(1000.0, estimate.LanMbpsP90!.Value, 3);
Assert.Equal("10-50", estimate.Bucket); // bucket keyed to WAN, not LAN
}
@@ -182,8 +182,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.Equal(20, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3); // p90 stays on the steady plateau
Assert.Equal(160.0, estimate.WanMbpsMax.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3); // p90 stays on the steady plateau
Assert.Equal(160.0, estimate.WanMbpsMax!.Value, 3);
}
[Fact]
@@ -265,7 +265,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.Equal(1, estimate.WanSampleCount); // only the finite positive sample survived
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
Assert.Equal(0, estimate.WanBytesObserved);
}
@@ -17,13 +17,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry.Events
FakeTelemetryTransport transport,
out EventSequenceCounter counter,
string sessionId = "sess-1",
string tenantId = "tenant-1")
string tenantId = "tenant-1",
Func<bool>? traceEventsEnabled = null)
{
var tmp = new TempDirectory();
counter = new EventSequenceCounter(new EventSequencePersistence(tmp.File("seq.json")));
return new TelemetryEventEmitter(transport, counter, sessionId, tenantId);
return new TelemetryEventEmitter(transport, counter, sessionId, tenantId, traceEventsEnabled);
}
private static EnrollmentEvent NewTraceEvent(string eventType = "outbound_ip") =>
new EnrollmentEvent
{
EventType = eventType,
Severity = EventSeverity.Trace,
Source = "test",
Timestamp = At,
Phase = EnrollmentPhase.Unknown,
Message = "trace message",
};
private static EnrollmentEvent NewTestEvent(string eventType = "enrollment_complete") =>
new EnrollmentEvent
{
@@ -49,6 +61,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry.Events
Assert.Equal("sess-1", evt.SessionId);
Assert.Equal("tenant-1", evt.TenantId);
Assert.NotNull(item);
Assert.Equal(TelemetryItemKind.Event, item.Kind);
Assert.Equal("tenant-1_sess-1", item.PartitionKey);
Assert.Equal(evt.RowKey, item.RowKey);
@@ -174,5 +187,86 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry.Events
Assert.Equal("test-reason", (string?)data["reason"]);
Assert.Equal("Strong", (string?)data["wgConfidence"]);
}
// ============================================================ Trace-event gate
[Fact]
public void Trace_event_is_dropped_when_gate_is_off()
{
var transport = new FakeTelemetryTransport();
var emitter = Build(transport, out _, traceEventsEnabled: () => false);
var item = emitter.Emit(NewTraceEvent());
Assert.Null(item);
Assert.Equal(0, transport.EnqueueCount);
}
[Fact]
public void Trace_event_flows_when_gate_is_on()
{
var transport = new FakeTelemetryTransport();
var emitter = Build(transport, out _, traceEventsEnabled: () => true);
Assert.NotNull(emitter.Emit(NewTraceEvent()));
Assert.Equal(1, transport.EnqueueCount);
}
[Fact]
public void Trace_event_flows_when_no_gate_is_wired()
{
var transport = new FakeTelemetryTransport();
var emitter = Build(transport, out _);
Assert.NotNull(emitter.Emit(NewTraceEvent()));
Assert.Equal(1, transport.EnqueueCount);
}
[Fact]
public void Gate_off_does_not_suppress_non_trace_events()
{
var transport = new FakeTelemetryTransport();
var emitter = Build(transport, out _, traceEventsEnabled: () => false);
foreach (var severity in new[] { EventSeverity.Info, EventSeverity.Warning, EventSeverity.Error })
{
var evt = NewTestEvent();
evt.Severity = severity;
Assert.NotNull(emitter.Emit(evt));
}
Assert.Equal(3, transport.EnqueueCount);
}
[Fact]
public void Dropped_trace_event_does_not_consume_a_sequence_number()
{
var transport = new FakeTelemetryTransport();
var traceAllowed = true;
// ReSharper disable once AccessToModifiedClosure — deliberate: the gate is read
// per event, mirroring a mid-session remote-config merge.
var emitter = Build(transport, out _, traceEventsEnabled: () => traceAllowed);
emitter.Emit(NewTestEvent()); // seq 1
traceAllowed = false;
emitter.Emit(NewTraceEvent()); // dropped — must not burn seq 2
emitter.Emit(NewTestEvent()); // seq 2
Assert.Equal(2, transport.EnqueueCount);
var first = JObject.Parse(transport.Enqueued[0].PayloadJson);
var second = JObject.Parse(transport.Enqueued[1].PayloadJson);
Assert.Equal(1, (long)first["Sequence"]!);
Assert.Equal(2, (long)second["Sequence"]!);
}
[Fact]
public void Throwing_gate_fails_open_so_diagnostics_are_never_silently_lost()
{
var transport = new FakeTelemetryTransport();
var emitter = Build(transport, out _, traceEventsEnabled: () => throw new InvalidOperationException("boom"));
Assert.NotNull(emitter.Emit(NewTraceEvent()));
Assert.Equal(1, transport.EnqueueCount);
}
}
}
@@ -0,0 +1,57 @@
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Models;
using AutopilotMonitor.Shared.Services;
namespace AutopilotMonitor.Agent.V2.Core.Configuration
{
/// <summary>
/// Agent-side evaluation of the config-channel endpoint migration signal
/// (<see cref="AgentConfigResponse.MigrateToApiBaseUrl"/>). Mirrors the kill-switch
/// hardening: only a LIVE fetch is honoured (the on-disk config cache strips the field on
/// write and read, and this check re-validates the outcome as defence-in-depth), and the
/// target must pass <see cref="AgentEndpointMigrationRules"/> even though the backend
/// already validated it — a compromised or misconfigured backend must not be able to
/// re-home agents to an arbitrary host.
/// </summary>
public static class EndpointMigration
{
/// <summary>
/// Returns the normalized migration target when the config carries a valid, live-fetched
/// re-home signal that differs from <paramref name="currentBaseUrl"/>; otherwise null.
/// Never throws — every rejection is logged and degrades to "no migration".
/// </summary>
public static string ResolveTarget(
AgentConfigResponse remoteConfig,
RemoteConfigFetchOutcome fetchOutcome,
string currentBaseUrl,
AgentLogger logger)
{
if (remoteConfig == null || string.IsNullOrWhiteSpace(remoteConfig.MigrateToApiBaseUrl))
return null;
if (fetchOutcome != RemoteConfigFetchOutcome.Succeeded)
{
logger?.Warning(
$"Endpoint migration signal ignored — config source is {fetchOutcome}, not a live fetch.");
return null;
}
if (!AgentEndpointMigrationRules.IsEffectiveMigration(
remoteConfig.MigrateToApiBaseUrl, currentBaseUrl, out var target))
{
// Either the target failed validation (scheme/host allowlist) or it equals the
// URL the agent is already using (steady state during a migration window once
// the new backend's own config echoes nothing).
if (target == null && !AgentEndpointMigrationRules.TryNormalizeTarget(
remoteConfig.MigrateToApiBaseUrl, out _))
{
logger?.Warning(
$"Endpoint migration signal REJECTED — target failed validation: '{remoteConfig.MigrateToApiBaseUrl}'");
}
return null;
}
return target;
}
}
}
@@ -249,18 +249,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
// - DeviceBlocked/DeviceKillSignal/UnblockAt: kill is only honoured from a
// live fetch — a planted cached kill would self-destruct every future
// session, a stale cached kill would outlive the admin removing the rule
// - MigrateToApiBaseUrl: endpoint re-home is only honoured from a live
// fetch — a planted cached URL would silently redirect every future
// session's telemetry to an attacker-chosen host
var liveUnrestricted = config.UnrestrictedMode;
var liveAllowDowngrade = config.AllowAgentDowngrade;
var liveExeHash = config.LatestAgentExeSha256;
var liveBlocked = config.DeviceBlocked;
var liveKill = config.DeviceKillSignal;
var liveUnblockAt = config.UnblockAt;
var liveMigrateUrl = config.MigrateToApiBaseUrl;
config.UnrestrictedMode = false;
config.AllowAgentDowngrade = false;
config.LatestAgentExeSha256 = null;
config.DeviceBlocked = false;
config.DeviceKillSignal = false;
config.UnblockAt = null;
config.MigrateToApiBaseUrl = null;
var json = JsonConvert.SerializeObject(config, Formatting.Indented);
config.UnrestrictedMode = liveUnrestricted;
config.AllowAgentDowngrade = liveAllowDowngrade;
@@ -268,6 +273,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
config.DeviceBlocked = liveBlocked;
config.DeviceKillSignal = liveKill;
config.UnblockAt = liveUnblockAt;
config.MigrateToApiBaseUrl = liveMigrateUrl;
File.WriteAllText(_cacheFilePath, json);
_logger.Debug("Remote config cached to disk");
@@ -299,6 +305,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
config.DeviceBlocked = false;
config.DeviceKillSignal = false;
config.UnblockAt = null;
config.MigrateToApiBaseUrl = null;
}
return config;
}
@@ -10,9 +10,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// the <c>app_tracking_summary</c> terminal event.
/// <para>
/// <b>StartedAtUtc</b> is recorded on the first lifecycle transition into Downloading /
/// Installing / InProgress. <b>CompletedAtUtc</b> is recorded on the first transition to a
/// terminal state (Installed / Skipped / Postponed / Error). Both are "set-once" — subsequent
/// identical-state events do not overwrite an earlier stamp.
/// Installing / InProgress and never overwritten. <b>CompletedAtUtc</b> is recorded on a
/// transition to a terminal state (Installed / Skipped / Postponed / Error) and re-armed
/// to <c>null</c> when the app goes terminal → active again (IME retry / re-evaluation,
/// audit Q1) — so the final terminal event carries first-start → last-terminal.
/// </para>
/// </summary>
public sealed class AppInstallTiming
@@ -30,10 +31,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// <summary>
/// Elapsed time from first Downloading/Installing transition to terminal state, in
/// seconds. <c>null</c> when either endpoint is missing (e.g. app still installing,
/// or the adapter missed the start event because it came online mid-install).
/// or the adapter missed the start event because it came online mid-install) — and
/// <c>null</c> when the endpoints are inverted (CompletedAt before StartedAt: log lines
/// from different source files can be processed out of chronological order; observed in
/// production as durationSeconds=-2342 on a Skipped-then-retried user-scope app,
/// audit Q1). A negative duration is never a real measurement, so it is not emitted.
/// </summary>
public double? DurationSeconds =>
StartedAtUtc.HasValue && CompletedAtUtc.HasValue
StartedAtUtc.HasValue && CompletedAtUtc.HasValue && CompletedAtUtc.Value >= StartedAtUtc.Value
? (CompletedAtUtc.Value - StartedAtUtc.Value).TotalSeconds
: (double?)null;
}
@@ -208,39 +208,60 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
}
else
{
// Platform script: AgentExecutor.log entries create/enrich, IME log entries also create
if (!_pendingPlatformScripts.ContainsKey(id))
{
// A fresh start for this policy (no pending entry) begins a NEW execution.
// Clear any emitted-marker from a prior run so IME re-evaluations / retries of
// the same platform-script policy within one agent lifetime are not silently
// deduped away. Within a single run the start always precedes exit/result, so
// this never clears the current run's own marker.
_platformScriptResultEmitted.Remove(id);
_pendingPlatformScripts[id] = new ScriptExecutionState
{
PolicyId = id,
ScriptType = "platform",
// Prefer the source CMTrace timestamp so a script that started before the
// agent launched (replayed log content) is dated to its real start, not now.
// Set only at slot creation: the start line fires twice (agentexecutor + ime
// source) but the earliest wins.
StartedAtUtc = LastMatchedLogTimestamp ?? DateTime.UtcNow,
};
// Live "running" indicator — same signal health scripts emit. Gated on slot
// creation so the duplicate start line (agentexecutor + ime source) doesn't
// double-emit for the same execution.
try { OnScriptStarted?.Invoke(new ScriptStartedInfo { PolicyId = id, ScriptType = "platform" }); }
catch (Exception ex) { _logger.Warning($"ImeLogTracker: OnScriptStarted handler threw: {ex.Message}"); }
}
_lastPlatformScriptPolicyId = id;
// Started lines fire twice per script (agentexecutor + ime source) and carry no
// outcome — the matching `platform script completed` line below carries result+exit
// and stays on Info. Keep starts on Debug so Info reflects script outcomes only.
_logger.Debug($"ImeLogTracker: platform script started: {id} (source: {source ?? "ime"})");
HandlePlatformScriptStarted(id, source);
}
}
// Platform script: AgentExecutor.log entries create/enrich, IME log entries also create.
// Internal so the stale-slot hardening is directly testable (the public path arrives via
// a regex Match).
internal void HandlePlatformScriptStarted(string id, string source = null)
{
// Stale-slot hardening (session eaf3d8c4): a replayed start line from a previous
// enrollment whose matching result never appeared (script was mid-run when that
// enrollment ended) leaves a pending slot with a days-old StartedAtUtc. When the
// same policy genuinely re-runs now, the fresh start must begin a NEW execution —
// otherwise the ancient start pairs with the fresh completion and the duration
// lies. > 24 h mirrors the adapter's source-timestamp staleness clamp.
var currentStartTs = LastMatchedLogTimestamp ?? DateTime.UtcNow;
if (_pendingPlatformScripts.TryGetValue(id, out var pendingSlot)
&& pendingSlot.StartedAtUtc.HasValue
&& currentStartTs - pendingSlot.StartedAtUtc.Value > TimeSpan.FromHours(24))
{
_logger.Debug($"ImeLogTracker: discarding stale pending platform-script slot for {id} (started {pendingSlot.StartedAtUtc:o})");
_pendingPlatformScripts.Remove(id);
}
if (!_pendingPlatformScripts.ContainsKey(id))
{
// A fresh start for this policy (no pending entry) begins a NEW execution.
// Clear any emitted-marker from a prior run so IME re-evaluations / retries of
// the same platform-script policy within one agent lifetime are not silently
// deduped away. Within a single run the start always precedes exit/result, so
// this never clears the current run's own marker.
_platformScriptResultEmitted.Remove(id);
_pendingPlatformScripts[id] = new ScriptExecutionState
{
PolicyId = id,
ScriptType = "platform",
// Prefer the source CMTrace timestamp so a script that started before the
// agent launched (replayed log content) is dated to its real start, not now.
// Set only at slot creation: the start line fires twice (agentexecutor + ime
// source) but the earliest wins.
StartedAtUtc = currentStartTs,
};
// Live "running" indicator — same signal health scripts emit. Gated on slot
// creation so the duplicate start line (agentexecutor + ime source) doesn't
// double-emit for the same execution.
try { OnScriptStarted?.Invoke(new ScriptStartedInfo { PolicyId = id, ScriptType = "platform" }); }
catch (Exception ex) { _logger.Warning($"ImeLogTracker: OnScriptStarted handler threw: {ex.Message}"); }
}
_lastPlatformScriptPolicyId = id;
// Started lines fire twice per script (agentexecutor + ime source) and carry no
// outcome — the matching `platform script completed` line below carries result+exit
// and stays on Info. Keep starts on Debug so Info reflects script outcomes only.
_logger.Debug($"ImeLogTracker: platform script started: {id} (source: {source ?? "ime"})");
}
// Routes line-by-line script-data handlers (context / exitCode / output) to the right
// accumulator based on the pattern's scriptType parameter. Health-script lines feed the
// single-slot _pendingHealthScript; platform-script lines feed _pendingPlatformScripts.
@@ -194,6 +194,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
}
}
// Actions that mutate app/phase tracking state (_packageStates, _phasePackageSnapshots,
// _currentPhaseOrder, DO telemetry) — the historic-replay guard skips these for source
// lines from a previous enrollment so replayed apps never enter tracked state,
// persistence, app_tracking_summary, culprit lists or final-status. Script actions are
// deliberately NOT here: their tracker state is harmless (stale-slot hardening covers
// leftovers) and the adapter suppresses their emissions. espphasedetected IS here — a
// stale "In EspPhase: AccountSetup" would advance _currentPhaseOrder and make the fresh
// enrollment's DeviceSetup lines bounce as "backward"; IME re-logs the phase constantly,
// so fresh lines re-deliver it immediately.
private static readonly HashSet<string> AppMutatingActions = new HashSet<string>(StringComparer.Ordinal)
{
"imeshutdown", "espphasedetected", "setcurrentapp",
"updatestateinstalled", "updatestatedownloading", "updatestateinstalling",
"updatestateskipped", "updatestateerror", "updatestatepostponed",
"captureexitcode", "capturehresult", "captureappversion",
"captureapptypewinget", "captureapptypemsi", "captureattemptnumber",
"capturedetectionresult", "esptrackstatus", "policiesdiscovered",
"ignorecompletedapp", "updatename", "updatewin32appstate",
"cancelstuckandsetcurrent", "updatedotelemetry",
};
private void HandlePatternMatch(CompiledPattern pattern, Match match, string message, CmTraceLogEntry entry)
{
LastMatchedPatternId = pattern.PatternId;
@@ -206,6 +227,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
try { OnPatternMatched?.Invoke(pattern.PatternId); }
catch (Exception ex) { _logger?.Warning($"OnPatternMatched handler threw: {ex.Message}"); }
// Historic-replay guard (session eaf3d8c4): a source line > 24 h older than now is
// content from a previous enrollment whose IME log survived on disk. App-mutating
// actions are skipped BEFORE _seenAppIds so replayed apps poison neither the tracked
// state nor the phase-change ignore list. SimulationMode (--replay-log-dir dev tool)
// replays historic logs on purpose and bypasses the guard.
var isStaleReplayLine = !SimulationMode && entry != null
&& (UtcNowProvider() - NormalizeUtc(entry.Timestamp)) > HistoricReplayThreshold;
if (isStaleReplayLine && AppMutatingActions.Contains(pattern.Action?.ToLower() ?? string.Empty))
{
_logger.Debug($"ImeLogTracker: skipped app action '{pattern.Action}' for historic line ({entry.Timestamp:o})");
return;
}
try
{
var id = match.Groups["id"]?.Value;
@@ -71,6 +71,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
public double SpeedFactor { get; set; } = 50;
private DateTime _lastLogTimestamp = DateTime.MinValue;
// Source log lines older than this relative to now are content from a previous
// enrollment (IME logs surviving a re-enrollment, session eaf3d8c4). Single source of
// truth shared with the adapter's ResolveOccurredAt clamp and replay suppression.
internal static readonly TimeSpan HistoricReplayThreshold = TimeSpan.FromHours(24);
// Deterministic-time seam for the historic-replay guard; production default is the
// system clock, ImeLogHost rewires it to the agent IClock, tests to their VirtualClock.
internal Func<DateTime> UtcNowProvider { get; set; } = () => DateTime.UtcNow;
/// <summary>
/// CMTrace timestamps from IME are emitted in UTC, but DateTimeKind may arrive as
/// Unspecified depending on the parser path. Normalize via SpecifyKind=Utc rather than
/// ToUniversalTime() to avoid double-conversion when Kind is already UTC.
/// </summary>
internal static DateTime NormalizeUtc(DateTime value) =>
value.Kind == DateTimeKind.Utc ? value : DateTime.SpecifyKind(value, DateTimeKind.Utc);
// Background task
private Task _pollingTask;
private CancellationTokenSource _cts;
@@ -61,6 +61,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
internal Func<int, string> SessionOwnerResolver { get; set; }
internal Func<int, string> ProcessOwnerResolver { get; set; }
// Test seam for the WinRT OOBE-state sample (null = OobeStateReader.Read).
internal Func<string> OobeStateProvider { get; set; }
// OOBE-state flip tracking (observational only — see OobeStateReader contract).
// _oobeCompletedEmitted is one-shot per agent lifetime and deliberately NOT reset
// by ResetForRealUserSwitch: the OOBE flip is a global OS transition, not per-user.
private string _lastOobeState;
private bool _oobeCompletedEmitted;
/// <summary>
/// System/service account names that should NOT be considered real users.
/// </summary>
@@ -161,6 +170,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
_startedAtUtc = DateTime.UtcNow;
// Baseline OOBE-state sample so the first poll can already detect a flip
// (self-contained try/catch inside, like the Emit* helpers).
CheckOobeStateFlip();
_pollingTimer?.Dispose();
_pollingTimer = new Timer(
PollForDesktop,
@@ -186,6 +199,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_pollCount++;
CheckOobeStateFlip();
try
{
var explorerProcesses = Process.GetProcessesByName("explorer");
@@ -381,6 +396,66 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
catch (Exception ex) { _logger.Verbose($"DesktopArrivalDetector: detector-started emit failed: {ex.Message}"); }
}
/// <summary>
/// Samples the WinRT OOBE state (<see cref="OobeStateReader"/>) and emits the one-shot
/// <c>oobe_state_completed</c> trace event on the in_progress-&gt;completed flip.
/// Observational only — an owner-independent desktop corroboration for sessions where
/// WTS+WMI owner resolution starves and <c>desktop_arrived</c> never fires. Often absent
/// in healthy sessions: the polling timer is disposed at desktop arrival, which is when
/// the flip lands (empirical probe, session 9c404ae9).
/// <para>
/// Self-contained try/catch: a throw here must NOT reach <see cref="PollForDesktop"/>'s
/// outer catch, which would inflate <see cref="_wmiErrorCountSinceStart"/> and skew the
/// session-4d5a0b78 liveness diagnostics. "unavailable" readings never overwrite the last
/// known state, so a transient reflection failure between two polls cannot mask the flip.
/// One-shot flag survives <see cref="ResetForRealUserSwitch"/> — the flip is a global OS
/// transition, not per-user.
/// </para>
/// </summary>
private void CheckOobeStateFlip()
{
try
{
var current = (OobeStateProvider ?? OobeStateReader.Read)();
if (string.IsNullOrEmpty(current) || current == OobeStateReader.Unavailable)
return;
if (!_oobeCompletedEmitted
&& _lastOobeState == "in_progress"
&& current == "completed")
{
_oobeCompletedEmitted = true;
var minutesSinceStart = Math.Round((DateTime.UtcNow - _startedAtUtc).TotalMinutes, 1);
_logger.Info("DesktopArrivalDetector: OOBE state flipped in_progress->completed (WinRT SystemSetupInfo)");
OnTraceEvent?.Invoke(
SharedEventTypes.OobeStateCompleted,
$"OOBE state flipped in_progress->completed (WinRT SystemSetupInfo, poll {_pollCount}) — owner-independent desktop corroboration.",
new Dictionary<string, object>
{
{ "previousState", "in_progress" },
{ "pollCount", _pollCount },
{ "minutesSinceStart", minutesSinceStart },
{ "desktopArrived", _desktopArrived },
{ "wmiErrorCount", _wmiErrorCountSinceStart },
{ "wtsErrorCount", _wtsErrorCountSinceStart },
});
}
_lastOobeState = current;
}
catch (Exception ex)
{
_logger.Verbose($"DesktopArrivalDetector: OOBE state check failed: {ex.Message}");
}
}
/// <summary>
/// Drives a single synchronous poll for unit tests — the production poll is private and
/// timer-driven with a fixed 30 s interval (precedent: <see cref="ResolveOwner"/> is
/// likewise internal for direct testing).
/// </summary>
internal void PollOnceForTest() => PollForDesktop(null);
/// <summary>
/// Resolves the owning user of an explorer.exe candidate. Session 4d5a0b78 fix
/// (2026-06-11): the WMI <c>Win32_Process.GetOwner</c> call failed on every poll on
@@ -1,4 +1,5 @@
using System;
using System.Collections.Generic;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
@@ -38,16 +39,30 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// </summary>
public string Category { get; }
/// <summary>
/// Session 4910a5a5: names of the tracked apps most likely behind an Apps-subcategory
/// failure (capped, never-started apps ranked first — those starve ESP's app gate
/// without ever producing an IME install event). Snapshot from
/// <c>ProvisioningStatusTracker.SnapshotTrackedAppsNotCompleted</c> at failure-arming
/// time; empty for non-Apps failures and event-log-derived failures. Propagated onto
/// the EspTerminalFailure signal payload so the DecisionEngine's
/// <c>esp_failure_advisory</c> / <c>enrollment_failed</c> events can name the culprit
/// directly.
/// </summary>
public IReadOnlyList<string> LikelyCulpritApps { get; }
public EspFailureDetectedEventArgs(
string failureType,
string errorCode = null,
string failedSubcategory = null,
string category = null)
string category = null,
IReadOnlyList<string> likelyCulpritApps = null)
{
FailureType = string.IsNullOrEmpty(failureType) ? "unknown" : failureType;
ErrorCode = string.IsNullOrEmpty(errorCode) ? null : errorCode;
FailedSubcategory = string.IsNullOrEmpty(failedSubcategory) ? null : failedSubcategory;
Category = string.IsNullOrEmpty(category) ? null : category;
LikelyCulpritApps = likelyCulpritApps ?? Array.Empty<string>();
}
}
}
@@ -0,0 +1,307 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using Microsoft.Win32;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Session 5d735290 (2026-07-23) — reads the ESP's registered policy providers from
/// <c>HKLM\SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking</c>, the registry
/// mirror of the EnrollmentStatusTracking CSP
/// (https://learn.microsoft.com/en-us/windows/client-management/mdm/enrollmentstatustracking-csp).
/// <para>
/// CSP contract the ESP enforces (quotes from the doc above):
/// <list type="bullet">
/// <item><c>Setup/Apps/PolicyProviders/&lt;Name&gt;</c> — "Existence of this node indicates
/// to the ESP that it shouldn't show the tracking status message until the
/// TrackingPoliciesCreated node has been set to true." This wait has NO timeout: a
/// registered provider that never sets <c>TrackingPoliciesCreated=1</c> parks the ESP at
/// "Apps (Identifying)" indefinitely (observed for days in the field).</item>
/// <item><c>DevicePreparation/PolicyProviders/&lt;Name&gt;/InstallationState</c> — must reach
/// 2 (NotRequired) or 3 (Completed); 1=NotInstalled, 4=Error. ESP applies a default
/// 15-minute timeout here, so stalls in this stage are self-surfacing — still reported
/// for the diagnostic trail.</item>
/// </list>
/// Registry layout: <c>Device</c> subkey for device scope, <c>S-&lt;SID&gt;</c> subkeys for user
/// scope (appear at sign-in), each containing <c>Setup\Apps\PolicyProviders\&lt;Name&gt;</c>;
/// <c>DevicePreparation\PolicyProviders</c> exists in device scope only.
/// </para>
/// <para>
/// Known provider names: <c>Sidecar</c> is the Intune Management Extension ("Intune's agents,
/// such as SideCar" per the CSP doc); <c>ConfigMgr</c> is the Configuration Manager client
/// (registered during Autopilot-into-co-management, see the troubleshooting registry paths in
/// https://learn.microsoft.com/en-us/intune/configmgr/comanage/autopilot-enrollment). Microsoft
/// warns that ESP policy providers are not aware of each other and that pre-provisioning is
/// unsupported with co-management. Field case that motivated this probe (issue #106): a
/// pre-provisioned device with the ConfigMgr client installed as a blocking app ended up with
/// ONLY <c>ConfigMgr</c> registered under <c>Device\Setup\Apps\PolicyProviders</c> (no
/// <c>Sidecar</c>) — and although <c>TrackingPoliciesCreated=1</c> WAS set under it, the user
/// ESP hung at "Apps (Identifying)" for hours to days; manually renaming the key to
/// <c>Sidecar</c> unblocked it immediately. So the ESP's Apps wait is keyed to the provider
/// the Intune service registered at enrollment (always <c>Sidecar</c>; ConfigMgr only joins
/// via a co-management settings policy — see the WAP payload documented in
/// https://oofhours.com/2023/08/25/run-an-sccm-task-sequence-during-autopilot/), and a
/// foreign provider's completion values do not satisfy it. Detection therefore keys on the
/// provider NAME topology, not only on completeness:
/// <see cref="EspPolicyProviderStallDetector"/> turns this contract into the two-arm
/// <c>esp_policy_provider_stalled</c> tripwire.
/// </para>
/// <para>
/// Fail-soft like <see cref="EspTrackingInfoProbe"/>: any failure is Debug-logged and yields
/// <see cref="EspPolicyProviderSnapshot.Empty"/>; a missing root key (early enrollment,
/// non-Autopilot device) is not an error. Values are parsed defensively — a missing or
/// unparseable value counts as incomplete (fail-toward-observation).
/// </para>
/// </summary>
internal static class EspPolicyProviderProbe
{
internal const string TrackingRootKeyPath =
@"SOFTWARE\Microsoft\Windows\Autopilot\EnrollmentStatusTracking";
/// <summary>Provider kind: <c>Setup\Apps\PolicyProviders</c> (TrackingPoliciesCreated wait, no timeout).</summary>
internal const string KindSetupApps = "setupApps";
/// <summary>Provider kind: <c>DevicePreparation\PolicyProviders</c> (InstallationState wait, 15-min OS timeout).</summary>
internal const string KindDevicePreparation = "devicePreparation";
/// <summary>The IME's provider name — its absence while other providers are registered is the field-case signature.</summary>
internal const string SidecarProviderName = "Sidecar";
internal const string DeviceScope = "device";
private const string TrackingPoliciesCreatedValueName = "TrackingPoliciesCreated";
private const string InstallationStateValueName = "InstallationState";
/// <summary>
/// Test seam — when non-null, <see cref="Read"/> delegates to this func instead of
/// touching the live registry. Use <see cref="ScopedOverride"/> to guarantee cleanup.
/// </summary>
internal static Func<AgentLogger, EspPolicyProviderSnapshot> TestOverride;
/// <summary>
/// Reads all registered ESP policy providers. <see cref="EspPolicyProviderSnapshot.HasData"/>
/// is <c>true</c> when the EnrollmentStatusTracking root key existed and was readable
/// (even with no providers registered); <see cref="EspPolicyProviderSnapshot.Empty"/> otherwise.
/// </summary>
/// <param name="logger">Optional logger — Debug-level trace only; no warn/error.</param>
public static EspPolicyProviderSnapshot Read(AgentLogger logger = null)
{
var probe = TestOverride;
if (probe != null) return probe(logger);
try
{
using (var rootKey = Registry.LocalMachine.OpenSubKey(TrackingRootKeyPath))
{
if (rootKey == null)
{
logger?.Debug("EspPolicyProviderProbe: EnrollmentStatusTracking registry key not found");
return EspPolicyProviderSnapshot.Empty;
}
var providers = new List<PolicyProviderState>();
foreach (var scopeName in rootKey.GetSubKeyNames())
{
var isDevice = string.Equals(scopeName, "Device", StringComparison.OrdinalIgnoreCase);
// User scope repeats the Setup structure under the signed-in user's SID.
var isUserSid = scopeName.StartsWith("S-", StringComparison.OrdinalIgnoreCase);
if (!isDevice && !isUserSid) continue;
var scopeLabel = isDevice ? DeviceScope : "user:" + scopeName;
using (var scopeKey = rootKey.OpenSubKey(scopeName))
{
if (scopeKey == null) continue;
CollectProviders(
scopeKey, @"Setup\Apps\PolicyProviders", scopeLabel, KindSetupApps,
TrackingPoliciesCreatedValueName, providers, logger);
if (isDevice)
{
CollectProviders(
scopeKey, @"DevicePreparation\PolicyProviders", scopeLabel, KindDevicePreparation,
InstallationStateValueName, providers, logger);
}
}
}
return new EspPolicyProviderSnapshot(providers);
}
}
catch (Exception ex)
{
logger?.Debug($"EspPolicyProviderProbe: read threw: {ex.Message}");
return EspPolicyProviderSnapshot.Empty;
}
}
private static void CollectProviders(
RegistryKey scopeKey,
string providersSubPath,
string scopeLabel,
string kind,
string stateValueName,
List<PolicyProviderState> providers,
AgentLogger logger)
{
try
{
using (var providersKey = scopeKey.OpenSubKey(providersSubPath))
{
if (providersKey == null) return;
foreach (var providerName in providersKey.GetSubKeyNames())
{
using (var providerKey = providersKey.OpenSubKey(providerName))
{
if (providerKey == null) continue;
var value = TryReadInt(providerKey.GetValue(stateValueName));
providers.Add(new PolicyProviderState(
name: providerName,
scope: scopeLabel,
kind: kind,
trackingPoliciesCreated: kind == KindSetupApps ? value : null,
installationState: kind == KindDevicePreparation ? value : null));
}
}
}
}
catch (Exception ex)
{
logger?.Debug($"EspPolicyProviderProbe: '{providersSubPath}' ({scopeLabel}) read threw: {ex.Message}");
}
}
/// <summary>
/// Defensive value parse: REG_DWORD expected per CSP (booleans stored as 0/1), but string
/// forms are tolerated. Anything missing or unparseable is <c>null</c> — which the
/// completion rule treats as incomplete.
/// </summary>
internal static int? TryReadInt(object rawValue)
{
switch (rawValue)
{
case null: return null;
case int i: return i;
case long l when l >= int.MinValue && l <= int.MaxValue: return (int)l;
case string s when int.TryParse(s.Trim(), NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsed):
return parsed;
case bool b: return b ? 1 : 0;
default: return null;
}
}
/// <summary>
/// Pure completion rule per the CSP contract: <c>setupApps</c> providers are complete when
/// <c>TrackingPoliciesCreated == 1</c>; <c>devicePreparation</c> providers when
/// <c>InstallationState</c> is 2 (NotRequired) or 3 (Completed).
/// </summary>
internal static bool IsProviderComplete(string kind, int? trackingPoliciesCreated, int? installationState)
=> kind == KindSetupApps
? trackingPoliciesCreated == 1
: installationState == 2 || installationState == 3;
/// <summary>
/// Disposable scope that sets <see cref="TestOverride"/> for the lifetime of the scope
/// and restores the previous value on Dispose. Nestable. Internal test-only helper.
/// </summary>
internal sealed class ScopedOverride : IDisposable
{
private readonly Func<AgentLogger, EspPolicyProviderSnapshot> _previous;
private int _disposed;
public ScopedOverride(Func<AgentLogger, EspPolicyProviderSnapshot> probe)
{
_previous = TestOverride;
TestOverride = probe;
}
public void Dispose()
{
if (System.Threading.Interlocked.Exchange(ref _disposed, 1) == 1) return;
TestOverride = _previous;
}
}
}
/// <summary>
/// One registered ESP policy provider with its completion-relevant state.
/// <see cref="Key"/> is the stable identity used for dwell tracking and one-shot latching.
/// </summary>
internal readonly struct PolicyProviderState
{
public PolicyProviderState(string name, string scope, string kind, int? trackingPoliciesCreated, int? installationState)
{
Name = name ?? string.Empty;
Scope = scope ?? string.Empty;
Kind = kind ?? string.Empty;
TrackingPoliciesCreated = trackingPoliciesCreated;
InstallationState = installationState;
IsComplete = EspPolicyProviderProbe.IsProviderComplete(kind, trackingPoliciesCreated, installationState);
}
/// <summary>Provider key name, e.g. "Sidecar" or "ConfigMgr".</summary>
public string Name { get; }
/// <summary><c>"device"</c> or <c>"user:S-1-5-21-…"</c>.</summary>
public string Scope { get; }
/// <summary><see cref="EspPolicyProviderProbe.KindSetupApps"/> or <see cref="EspPolicyProviderProbe.KindDevicePreparation"/>.</summary>
public string Kind { get; }
/// <summary>Raw <c>TrackingPoliciesCreated</c> value (setupApps only); <c>null</c> when missing/unparseable.</summary>
public int? TrackingPoliciesCreated { get; }
/// <summary>Raw <c>InstallationState</c> value (devicePreparation only); <c>null</c> when missing/unparseable.</summary>
public int? InstallationState { get; }
/// <summary>Completion per <see cref="EspPolicyProviderProbe.IsProviderComplete"/>.</summary>
public bool IsComplete { get; }
/// <summary>Stable dwell/latch identity: <c>kind|scope|name</c>.</summary>
public string Key => Kind + "|" + Scope + "|" + Name;
}
/// <summary>
/// Snapshot of all registered ESP policy providers. <see cref="HasData"/> is <c>false</c> only
/// when the EnrollmentStatusTracking root key was missing or unreadable — an empty provider
/// list with <see cref="HasData"/> = <c>true</c> is valid data (nothing registered yet).
/// </summary>
internal readonly struct EspPolicyProviderSnapshot
{
public static readonly EspPolicyProviderSnapshot Empty = default;
public EspPolicyProviderSnapshot(IReadOnlyList<PolicyProviderState> providers)
{
HasData = true;
Providers = providers ?? Array.Empty<PolicyProviderState>();
}
/// <summary><c>true</c> when the EnrollmentStatusTracking root key existed and was readable.</summary>
public bool HasData { get; }
/// <summary>All registered providers across scopes and kinds; empty when none registered.</summary>
public IReadOnlyList<PolicyProviderState> Providers { get; }
/// <summary>
/// <c>true</c> when a <c>Setup\Apps</c> provider named "Sidecar" (the IME) is registered in
/// any scope — its absence while other providers are registered is the co-management
/// field-case signature.
/// </summary>
public bool SidecarRegistered
{
get
{
var providers = Providers;
if (providers == null) return false;
return providers.Any(p =>
p.Kind == EspPolicyProviderProbe.KindSetupApps &&
string.Equals(p.Name, EspPolicyProviderProbe.SidecarProviderName, StringComparison.OrdinalIgnoreCase));
}
}
}
}
@@ -0,0 +1,315 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Turns the ESP policy-provider contract read by <see cref="EspPolicyProviderProbe"/> into
/// the one-shot <c>esp_policy_provider_stalled</c> Warning via two independent arms, each
/// with the same <see cref="DwellMinutes"/>-minute continuous-dwell rule and the full
/// provider table in the payload:
/// <list type="bullet">
/// <item><b>Arm 1 — <c>reason=provider_incomplete</c></b>: a registered provider stays
/// continuously incomplete (e.g. Sidecar itself never sets TrackingPoliciesCreated).</item>
/// <item><b>Arm 2 — <c>reason=sidecar_provider_missing</c></b>: at least one
/// <c>Setup\Apps</c> provider is registered but none of them is <c>Sidecar</c> (the IME).
/// This is the actual co-management field-case signature (issue #106): the foreign
/// <c>ConfigMgr</c> provider had <c>TrackingPoliciesCreated=1</c> — i.e. was "complete"
/// by the CSP value contract — yet the user ESP stayed parked at "Apps (Identifying)"
/// until the key was renamed to <c>Sidecar</c>. The ESP's Apps wait is keyed to the
/// Intune-registered provider by NAME, so provider completeness alone must not gate this
/// detection. See the probe's header for sources.</item>
/// </list>
/// <para>
/// Dwell semantics: arm 1's clock per provider starts when it is first observed incomplete
/// and RESETS when the provider completes or disappears; arm 2's clock starts when the
/// sidecar-missing condition is first observed and RESETS when Sidecar registers (covers the
/// legitimate "ConfigMgr registered before Sidecar" startup ordering) — only
/// <i>continuously</i> bad states fire. A missing root key / empty provider list is normal
/// early enrollment and clears all dwell state (probe D4). Purely observational: emission
/// goes through <see cref="InformationalEventPost"/> (InformationalEvent pass-through,
/// dispatch-guard exempt), never into decision state.
/// </para>
/// <para>
/// Duplicate suppression: in-process one-shot per provider key, plus a
/// <see cref="StartupEventGate"/> fingerprint over the sorted set of currently-stalled keys so
/// an agent restart into the same stall does not re-report it — while a NEW provider joining
/// the stall changes the fingerprint and is reported (the event carries the full list).
/// </para>
/// </summary>
internal sealed class EspPolicyProviderStallDetector
{
/// <summary>
/// Wall-clock minutes a provider must stay continuously incomplete before it is reported.
/// Mirrors the CSP's own DevicePreparation default timeout; the Setup/Apps wait this
/// tripwire mainly targets has no OS timeout at all.
/// </summary>
internal const int DwellMinutes = 15;
internal const string SourceLabel = "EspPolicyProviderStallDetector";
/// <summary>Payload <c>reason</c>: a registered provider stayed continuously incomplete (arm 1).</summary>
internal const string ReasonProviderIncomplete = "provider_incomplete";
/// <summary>Payload <c>reason</c>: Setup\Apps providers registered but Sidecar absent (arm 2 — dominant when both arms fire in one round).</summary>
internal const string ReasonSidecarMissing = "sidecar_provider_missing";
/// <summary>
/// Sentinel token representing the arm-2 condition in the cross-restart fingerprint.
/// Cannot collide with provider keys — those always contain <c>|</c> separators.
/// </summary>
private const string SidecarMissingFingerprintToken = "sidecarMissing";
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly AgentLogger _logger;
private readonly IClock _clock;
private readonly StartupEventGate? _startupGate;
private readonly TimeSpan _dwell;
private readonly Dictionary<string, DateTime> _firstSeenIncompleteUtc =
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
private readonly HashSet<string> _firedKeys =
new HashSet<string>(StringComparer.OrdinalIgnoreCase);
/// <summary>Arm-2 dwell state: first observation of "setupApps providers without Sidecar"; null = condition not active.</summary>
private DateTime? _sidecarMissingSinceUtc;
/// <summary>Arm-2 in-process one-shot latch (mirror of <see cref="_firedKeys"/>).</summary>
private bool _sidecarMissingFired;
public EspPolicyProviderStallDetector(
string sessionId,
string tenantId,
InformationalEventPost post,
AgentLogger logger,
IClock clock,
StartupEventGate? startupGate = null,
TimeSpan? dwell = null)
{
_sessionId = sessionId;
_tenantId = tenantId;
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger;
_clock = clock ?? throw new ArgumentNullException(nameof(clock));
_startupGate = startupGate;
_dwell = dwell ?? TimeSpan.FromMinutes(DwellMinutes);
}
/// <summary>Host timer callback and test seam — one probe + evaluation round.</summary>
internal void Tick()
{
var snapshot = EspPolicyProviderProbe.Read(_logger);
var eval = EvaluateStallTransition(
snapshot, _firstSeenIncompleteUtc, _firedKeys,
_sidecarMissingSinceUtc, _sidecarMissingFired, _clock.UtcNow, _dwell);
_firstSeenIncompleteUtc.Clear();
foreach (var kv in eval.UpdatedFirstSeenUtc)
_firstSeenIncompleteUtc[kv.Key] = kv.Value;
_sidecarMissingSinceUtc = eval.UpdatedSidecarMissingSinceUtc;
if (eval.NewlyStalled.Count > 0 || eval.SidecarMissingStalledForMinutes != null)
EmitStalled(snapshot, eval);
}
/// <summary>
/// Pure evaluation of one probe round — the entire detection decision, testable without
/// registry, timer, or ingress (pattern: <c>PerformanceCollector.EvaluateDiskLowTransition</c>).
/// </summary>
/// <param name="snapshot">Current provider snapshot.</param>
/// <param name="firstSeenIncompleteUtc">Arm-1 dwell state from the previous round (provider key → first-seen-incomplete).</param>
/// <param name="alreadyFired">Provider keys already reported this run (never re-report).</param>
/// <param name="sidecarMissingSinceUtc">Arm-2 dwell state from the previous round; null = condition was not active.</param>
/// <param name="sidecarMissingAlreadyFired">Arm-2 one-shot latch (never re-report).</param>
/// <param name="nowUtc">Evaluation timestamp.</param>
/// <param name="dwell">Continuous-condition threshold (shared by both arms).</param>
internal static StallEvaluation EvaluateStallTransition(
EspPolicyProviderSnapshot snapshot,
IReadOnlyDictionary<string, DateTime> firstSeenIncompleteUtc,
ISet<string> alreadyFired,
DateTime? sidecarMissingSinceUtc,
bool sidecarMissingAlreadyFired,
DateTime nowUtc,
TimeSpan dwell)
{
var result = new StallEvaluation();
// Missing root key or no registered providers = normal early enrollment: clear all
// dwell state (a condition must be continuously OBSERVED to accrue dwell).
if (!snapshot.HasData || snapshot.Providers.Count == 0)
return result;
// Arm 1 — a registered provider continuously incomplete.
foreach (var provider in snapshot.Providers)
{
if (provider.IsComplete)
continue; // completed → falls out of the dwell map (reset)
var firstSeen = firstSeenIncompleteUtc.TryGetValue(provider.Key, out var seen)
? seen
: nowUtc;
result.UpdatedFirstSeenUtc[provider.Key] = firstSeen;
var stalledFor = nowUtc - firstSeen;
if (stalledFor >= dwell && !alreadyFired.Contains(provider.Key))
result.NewlyStalled.Add(new StalledProvider(provider, stalledFor.TotalMinutes));
}
// Arm 2 — Setup\Apps providers registered, but none of them is Sidecar. Deliberately
// ignores provider completeness: the field case (issue #106) had ConfigMgr with
// TrackingPoliciesCreated=1 and the ESP still waited. Sidecar registering later
// (legitimate startup ordering) clears the condition and resets the clock.
if (HasSetupAppsProvider(snapshot) && !snapshot.SidecarRegistered)
{
var since = sidecarMissingSinceUtc ?? nowUtc;
result.UpdatedSidecarMissingSinceUtc = since;
var missingFor = nowUtc - since;
if (missingFor >= dwell && !sidecarMissingAlreadyFired)
result.SidecarMissingStalledForMinutes = missingFor.TotalMinutes;
}
return result;
}
private static bool HasSetupAppsProvider(EspPolicyProviderSnapshot snapshot)
=> snapshot.Providers.Any(p => p.Kind == EspPolicyProviderProbe.KindSetupApps);
private void EmitStalled(EspPolicyProviderSnapshot snapshot, StallEvaluation eval)
{
foreach (var stalled in eval.NewlyStalled)
_firedKeys.Add(stalled.Provider.Key);
if (eval.SidecarMissingStalledForMinutes != null)
_sidecarMissingFired = true;
// Cross-restart dedup: fingerprint = sorted set of all fired conditions that are
// still active (incomplete providers + the sidecar-missing sentinel). A restart into
// the same stall re-accrues dwell and lands on the same fingerprint → suppressed; a
// new condition joining the stall changes it → re-emitted.
var currentlyStalledKeys = snapshot.Providers
.Where(p => !p.IsComplete && _firedKeys.Contains(p.Key))
.Select(p => p.Key)
.ToList();
if (_sidecarMissingFired && HasSetupAppsProvider(snapshot) && !snapshot.SidecarRegistered)
currentlyStalledKeys.Add(SidecarMissingFingerprintToken);
currentlyStalledKeys.Sort(StringComparer.OrdinalIgnoreCase);
var fingerprint = StartupEventGate.HashString(string.Join(";", currentlyStalledKeys));
if (_startupGate != null &&
!_startupGate.ShouldEmit(Constants.EventTypes.EspPolicyProviderStalled, fingerprint))
{
_logger?.Debug(
"EspPolicyProviderStallDetector: stalled provider set already reported by a " +
"previous agent run (fingerprint match) — suppressed.");
return;
}
// Sidecar-missing is the dominant reason when both arms fire in one round: it
// explains the ESP hang regardless of the foreign providers' completeness.
var reason = eval.SidecarMissingStalledForMinutes != null
? ReasonSidecarMissing
: ReasonProviderIncomplete;
var messageParts = new List<string>();
if (eval.SidecarMissingStalledForMinutes != null)
{
var registeredNames = string.Join(", ", snapshot.Providers
.Where(p => p.Kind == EspPolicyProviderProbe.KindSetupApps)
.Select(p => $"{p.Name} ({p.Scope})"));
messageParts.Add(
$"expected IME provider '{EspPolicyProviderProbe.SidecarProviderName}' not registered for " +
$">= {DwellMinutes} min while Setup/Apps providers exist: {registeredNames}");
}
var stalledNames = string.Join(", ",
eval.NewlyStalled.Select(s => $"{s.Provider.Name} ({s.Provider.Scope}/{s.Provider.Kind})"));
if (eval.NewlyStalled.Count > 0)
messageParts.Add($"provider(s) incomplete for >= {DwellMinutes} min: {stalledNames}");
var data = new Dictionary<string, object>(StringComparer.Ordinal)
{
["reason"] = reason,
["dwellMinutes"] = DwellMinutes,
["sidecarRegistered"] = snapshot.SidecarRegistered,
["providers"] = snapshot.Providers.Select(p => new Dictionary<string, object?>(StringComparer.Ordinal)
{
["name"] = p.Name,
["scope"] = p.Scope,
["kind"] = p.Kind,
["trackingPoliciesCreated"] = p.TrackingPoliciesCreated,
["installationState"] = p.InstallationState,
["complete"] = p.IsComplete,
}).ToList(),
["stalledProviders"] = eval.NewlyStalled.Select(s => new Dictionary<string, object>(StringComparer.Ordinal)
{
["name"] = s.Provider.Name,
["scope"] = s.Provider.Scope,
["kind"] = s.Provider.Kind,
["stalledForMinutes"] = Math.Round(s.StalledForMinutes, 1),
}).ToList(),
};
if (eval.SidecarMissingStalledForMinutes != null)
data["sidecarMissingForMinutes"] = Math.Round(eval.SidecarMissingStalledForMinutes.Value, 1);
var message =
"ESP Setup/Apps tracking stalled: " + string.Join("; ", messageParts) +
" — the ESP waits on its app-tracking providers without any timeout";
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
Timestamp = _clock.UtcNow,
EventType = Constants.EventTypes.EspPolicyProviderStalled,
Severity = EventSeverity.Warning,
Source = SourceLabel,
Phase = EnrollmentPhase.Unknown,
ImmediateUpload = true,
Message = message,
Data = data
});
_startupGate?.MarkEmitted(Constants.EventTypes.EspPolicyProviderStalled);
_logger?.Warning(
$"EspPolicyProviderStallDetector: {string.Join("; ", messageParts)} " +
$"(reason={reason}, sidecarRegistered={snapshot.SidecarRegistered}).");
}
/// <summary>Result of one <see cref="EvaluateStallTransition"/> round.</summary>
internal sealed class StallEvaluation
{
/// <summary>Next round's arm-1 dwell state — only currently-incomplete providers survive.</summary>
public Dictionary<string, DateTime> UpdatedFirstSeenUtc { get; } =
new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
/// <summary>Providers that crossed the dwell threshold this round and were not yet reported (arm 1).</summary>
public List<StalledProvider> NewlyStalled { get; } = new List<StalledProvider>();
/// <summary>Next round's arm-2 dwell state; null = sidecar-missing condition not active (reset).</summary>
public DateTime? UpdatedSidecarMissingSinceUtc { get; set; }
/// <summary>Non-null when the sidecar-missing condition crossed the dwell threshold this round and was not yet reported (arm 2).</summary>
public double? SidecarMissingStalledForMinutes { get; set; }
}
internal readonly struct StalledProvider
{
public StalledProvider(PolicyProviderState provider, double stalledForMinutes)
{
Provider = provider;
StalledForMinutes = stalledForMinutes;
}
public PolicyProviderState Provider { get; }
public double StalledForMinutes { get; }
}
}
}
@@ -0,0 +1,628 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.Eventing.Reader;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using System.Threading;
using System.Xml.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using Newtonsoft.Json;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Watches <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> for
/// EventID 2800 — logged once per MDM policy URI that matches the OS reboot-required catalog —
/// and emits ONE aggregated <c>mdm_policy_reboot_required</c> event per burst carrying the URI
/// list (<c>rebootUris</c>, <c>uriCount</c>, <c>firstRebootUri</c>).
/// <para>
/// Verified against session b2e890c1 (2026-07-20): the 2800 description reads
/// <c>The following URI has triggered a reboot: (./Device/...).</c> and records arrive in
/// sub-second bursts during policy sync — hence the debounce aggregation instead of per-record
/// events. CRITICAL semantics: only URIs applied DURING ESP DeviceSetup cause the coalesced
/// mid-ESP reboot ("second sign-in", PMPC research); later syncs (AccountSetup and beyond)
/// merely flag the requirement with no forced reboot. The event is therefore a NEUTRAL
/// observation (Info) — the reboot claim lives in ANALYZE-ESP-005, which requires an
/// actually-observed <c>system_reboot_detected</c> before attributing anything.
/// </para>
/// <para>
/// <b>Backfill + watermark.</b> Device policies can apply before the agent starts, and a
/// genuine coalesced reboot kills the agent BEFORE the debounced emit — in both cases the
/// post-(re)start backfill re-reads the .evtx and emits the aggregate then (historical
/// timestamps). The cross-restart RecordId watermark is persisted only at successful flush, so
/// a pre-emit death loses nothing and a completed emit is never repeated.
/// </para>
/// </summary>
internal sealed class MdmRebootPolicyTracker : IDisposable
{
internal const string Channel = "Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin";
internal const int EventId_PolicyRebootRequired = 2800;
internal const string WatermarkStateFileName = "mdm-reboot-watermark.json";
/// <summary>Cap for the emitted <c>rebootUris</c> list — <c>uriCount</c> always carries the uncapped distinct total.</summary>
internal const int MaxUrisPerEvent = 20;
/// <summary>Default quiet-period before a burst of 2800 records is flushed as one event.</summary>
internal const int DefaultDebounceMilliseconds = 10000;
/// <summary>Registry key whose <c>RebootRequired</c> value/subkey flags a pending OMA-DM coalesced reboot.</summary>
internal const string OmadmSyncMlKeyPath = @"SOFTWARE\Microsoft\Provisioning\OMADM\SyncML";
// Permissive OMA-DM URI shape ("./Device/Vendor/MSFT/...", "./User/..."). The verified 2800
// text wraps the URI in parentheses — ')' is deliberately outside the class so the match
// stops cleanly; trailing sentence punctuation is trimmed after the match.
private static readonly Regex UriPattern = new Regex(
@"\./[A-Za-z0-9_\-./\[\]{}%~+]+", RegexOptions.Compiled);
private readonly AgentLogger _logger;
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly bool _backfillEnabled;
private readonly int _backfillLookbackMinutes;
private readonly string _stateDirectory;
private readonly int _debounceMilliseconds;
private EventLogWatcher _watcher;
private Timer _debounceTimer;
private sealed class PendingRecord
{
public long RecordId;
public DateTime TimeUtc;
public string Uri; // null when extraction failed
public string Description; // may be null
public bool IsBackfill;
}
// Pending burst buffer + dedup state, all guarded by _stateLock. Same claim contract as
// WindowsUpdateTracker (immutable cross-restart boundary + intra-run HashSet), with ONE
// deliberate difference: the watermark is persisted at FLUSH time, not at claim time —
// records buffered but never flushed (process killed by the very reboot they announce)
// must be re-read and emitted by the post-restart backfill.
private readonly List<PendingRecord> _pending = new List<PendingRecord>();
private long _restartWatermark = -1;
private readonly HashSet<long> _seenThisRun = new HashSet<long>();
private long _maxPersistedRecordId = -1;
private readonly object _stateLock = new object();
/// <summary>
/// Fail-soft corroboration probe for the OMA-DM RebootRequired flag — null = unknown
/// (key absent or unreadable). Settable test seam; the default reads the live registry.
/// </summary>
internal Func<bool?> RebootRequiredFlagProbe { get; set; } = ReadOmadmRebootRequiredFlag;
public MdmRebootPolicyTracker(
string sessionId,
string tenantId,
InformationalEventPost post,
AgentLogger logger,
bool backfillEnabled = true,
int backfillLookbackMinutes = 60,
string stateDirectory = null,
int debounceMilliseconds = DefaultDebounceMilliseconds)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_backfillEnabled = backfillEnabled;
_backfillLookbackMinutes = backfillLookbackMinutes;
_stateDirectory = stateDirectory != null ? Environment.ExpandEnvironmentVariables(stateDirectory) : null;
_debounceMilliseconds = debounceMilliseconds;
}
public void Start()
{
LoadWatermark();
StartWatcher();
if (_backfillEnabled && _backfillLookbackMinutes > 0)
{
BackfillRecentEvents();
// The backfill batch is complete — flush immediately instead of waiting out the
// debounce (live records that raced in are simply included).
FlushPending();
}
else
{
_logger.Info("MdmRebootPolicy backfill disabled by config");
}
}
public void Stop()
{
// Flush any un-emitted burst before tearing down (normal agent shutdown path).
try { FlushPending(); }
catch (Exception ex) { _logger.Warning($"MdmRebootPolicy flush on stop failed: {ex.Message}"); }
var timer = Interlocked.Exchange(ref _debounceTimer, null);
if (timer != null)
{
try { timer.Dispose(); } catch { }
}
if (_watcher == null) return;
try
{
_watcher.Enabled = false;
_watcher.Dispose();
_logger.Info("MdmRebootPolicy watcher stopped");
}
catch (Exception ex)
{
_logger.Error("Error stopping MdmRebootPolicy watcher", ex);
}
finally
{
_watcher = null;
}
}
public void Dispose() => Stop();
// -----------------------------------------------------------------------
// Watcher lifecycle
// -----------------------------------------------------------------------
private void StartWatcher()
{
try
{
var query = new EventLogQuery(Channel, PathType.LogName, BuildXPath());
_watcher = new EventLogWatcher(query);
_watcher.EventRecordWritten += OnEventRecordWritten;
_watcher.Enabled = true;
_logger.Info($"MdmRebootPolicy watcher started: {Channel} (EventID {EventId_PolicyRebootRequired})");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"MdmRebootPolicy event log not found: {Channel} (normal on non-MDM-enrolled test environments)");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"MdmRebootPolicy watcher access denied for {Channel}: {ex.Message}");
}
catch (Exception ex)
{
_logger.Error($"Failed to start MdmRebootPolicy watcher for {Channel}", ex);
// MON-D1: surface a dead watcher as one-shot telemetry, not just a local log line.
CollectorDegradationReporter.Report(_post, _sessionId, _tenantId,
collectorName: "MdmRebootPolicyTracker", reason: $"watcher_arm_failed:{Channel}", ex: ex);
}
}
/// <summary>Targeted-EventID XPath filter. Exposed as internal for tests.</summary>
internal static string BuildXPath() =>
$"*[System[(EventID={EventId_PolicyRebootRequired})]]";
// -----------------------------------------------------------------------
// Event processing — buffer + debounce
// -----------------------------------------------------------------------
private void OnEventRecordWritten(object sender, EventRecordWrittenEventArgs e)
{
if (e.EventRecord == null) return;
try
{
ProcessRecord(e.EventRecord, isBackfill: false);
}
catch (Exception ex)
{
_logger.Error("Error processing MdmRebootPolicy event", ex);
}
}
private void ProcessRecord(EventRecord record, bool isBackfill)
{
var recordId = record.RecordId ?? -1;
// Cheap dedup gate BEFORE the expensive XML/FormatDescription work.
if (IsAlreadyProcessed(recordId))
return;
string xml = null;
try { xml = record.ToXml(); }
catch { /* fall back to description-only parse below */ }
string description = null;
try { description = record.FormatDescription(); }
catch { /* some events lack formatting resources */ }
var rebootUri = ExtractRebootUri(ParseEventData(xml), description);
ProcessEvent(
eventId: record.Id,
recordId: recordId,
timeCreatedUtc: record.TimeCreated?.ToUniversalTime(),
rebootUri: rebootUri,
formattedDescription: description,
isBackfill: isBackfill);
}
/// <summary>
/// Buffers one 2800 record into the pending burst (primitive-input test seam, mirroring the
/// WindowsUpdateTracker pattern). The aggregated event is emitted by <see cref="FlushPending"/>
/// — after the debounce quiet-period, at backfill completion, or at <see cref="Stop"/>.
/// </summary>
internal void ProcessEvent(
int eventId,
long recordId,
DateTime? timeCreatedUtc,
string rebootUri,
string formattedDescription,
bool isBackfill)
{
lock (_stateLock)
{
if (recordId >= 0)
{
// Already emitted in a prior run, or already buffered/emitted this run.
if (recordId <= _restartWatermark || !_seenThisRun.Add(recordId))
return;
}
_pending.Add(new PendingRecord
{
RecordId = recordId,
TimeUtc = timeCreatedUtc ?? DateTime.UtcNow,
Uri = string.IsNullOrEmpty(rebootUri) ? null : rebootUri,
Description = formattedDescription,
IsBackfill = isBackfill,
});
}
ArmDebounce();
}
private void ArmDebounce()
{
if (_debounceMilliseconds <= 0) return; // manual-flush mode (tests; backfill/Stop flush explicitly)
var timer = _debounceTimer;
if (timer == null)
{
var created = new Timer(_ =>
{
try { FlushPending(); }
catch (Exception ex) { _logger.Warning($"MdmRebootPolicy debounce flush failed: {ex.Message}"); }
}, null, Timeout.Infinite, Timeout.Infinite);
timer = Interlocked.CompareExchange(ref _debounceTimer, created, null) ?? created;
if (!ReferenceEquals(timer, created))
{
try { created.Dispose(); } catch { }
}
}
try { timer.Change(_debounceMilliseconds, Timeout.Infinite); }
catch (ObjectDisposedException) { /* raced Stop() — Stop's flush covers the buffer */ }
}
/// <summary>
/// Emits the pending burst as ONE aggregated <c>mdm_policy_reboot_required</c> event and
/// persists the watermark. No-op when nothing is pending. Internal for tests.
/// </summary>
internal void FlushPending()
{
List<PendingRecord> batch;
long watermarkToPersist = -1;
lock (_stateLock)
{
if (_pending.Count == 0) return;
batch = new List<PendingRecord>(_pending);
_pending.Clear();
foreach (var r in batch)
{
if (r.RecordId > _maxPersistedRecordId)
_maxPersistedRecordId = r.RecordId;
}
watermarkToPersist = _maxPersistedRecordId;
}
if (watermarkToPersist >= 0) PersistWatermark(watermarkToPersist);
batch.Sort((a, b) => a.TimeUtc.CompareTo(b.TimeUtc));
var distinctUris = batch
.Where(r => r.Uri != null)
.Select(r => r.Uri)
.Distinct(StringComparer.OrdinalIgnoreCase)
.ToList();
var firstUri = batch.FirstOrDefault(r => r.Uri != null)?.Uri;
var unparsedCount = batch.Count(r => r.Uri == null);
var allBackfill = batch.All(r => r.IsBackfill);
var earliest = batch[0].TimeUtc;
var data = new Dictionary<string, object>
{
{ "windowsEventId", EventId_PolicyRebootRequired },
{ "eventLogChannel", Channel },
{ "recordCount", batch.Count },
{ "uriCount", distinctUris.Count },
{ "backfilled", allBackfill },
};
if (distinctUris.Count > 0)
{
data["rebootUris"] = distinctUris
.OrderBy(u => u, StringComparer.OrdinalIgnoreCase)
.Take(MaxUrisPerEvent)
.ToList();
data["firstRebootUri"] = firstUri;
}
if (unparsedCount > 0)
{
data["unparsedCount"] = unparsedCount;
var sample = batch.FirstOrDefault(r => r.Uri == null && !string.IsNullOrEmpty(r.Description))?.Description;
if (sample != null)
data["sampleDescription"] = sample.Length > 500 ? sample.Substring(0, 500) + "…" : sample;
}
var flag = SafeProbeRebootRequiredFlag();
if (flag.HasValue) data["omadmRebootRequiredFlag"] = flag.Value;
string message;
if (distinctUris.Count == 1)
message = $"Windows flagged an MDM policy URI as reboot-required: {firstUri}";
else if (distinctUris.Count > 1)
message = $"Windows flagged {distinctUris.Count} MDM policy URIs as reboot-required: {firstUri} (+{distinctUris.Count - 1} more)";
else
message = $"Windows flagged {batch.Count} MDM policy record(s) as reboot-required (URI not parsed — see sampleDescription)";
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
// Earliest record time, not UtcNow: backfilled (and post-reboot re-read) bursts
// must land on the timeline at when the policies were actually flagged.
Timestamp = earliest,
EventType = Constants.EventTypes.MdmPolicyRebootRequired,
// Neutral observation — reboot-required flags outside ESP DeviceSetup have no
// immediate effect. The reboot CLAIM is ANALYZE-ESP-005's job, gated on an
// actually-observed system_reboot_detected.
Severity = EventSeverity.Info,
Source = "MdmRebootPolicyWatcher",
Phase = EnrollmentPhase.Unknown,
Message = message,
Data = data,
// Cheap (one event per burst) and still time-critical: a burst DURING DeviceSetup
// precedes a coalesced reboot that kills the agent.
ImmediateUpload = true,
});
_logger.Info($"MdmRebootPolicy: flushed {batch.Count} record(s) as one event " +
$"(uris={distinctUris.Count}, unparsed={unparsedCount}, backfilled={allBackfill})");
}
/// <summary>
/// Extracts the reboot-forcing OMA-DM URI. Preference order: (1) a structured EventData
/// value that starts with <c>./</c>, (2) a permissive regex over the formatted description
/// (verified shape: <c>The following URI has triggered a reboot: (./Device/...).</c>).
/// Returns null when nothing URI-like is found — the record still counts into the
/// aggregate (<c>unparsedCount</c> + <c>sampleDescription</c>).
/// </summary>
internal static string ExtractRebootUri(Dictionary<string, string> eventData, string description)
{
if (eventData != null)
{
foreach (var value in eventData.Values)
{
var trimmed = value?.Trim();
if (!string.IsNullOrEmpty(trimmed) && trimmed.StartsWith("./", StringComparison.Ordinal))
return trimmed;
}
}
if (!string.IsNullOrEmpty(description))
{
var match = UriPattern.Match(description);
if (match.Success)
{
// The permissive pattern can swallow sentence punctuation trailing the URI.
// OMA-DM URI segments never end in punctuation — trim it.
var candidate = match.Value.TrimEnd('.', ',', ';', ':', '\'', '"', ')');
if (candidate.Length > 2) return candidate;
}
}
return null;
}
/// <summary>
/// Extracts <c>&lt;Data Name="..."&gt;value&lt;/Data&gt;</c> pairs from an event's rendered
/// XML, keyed case-insensitively. Also collects UNNAMED Data elements under positional keys
/// (<c>Data0</c>, <c>Data1</c>, …) — DM-Enterprise events frequently carry positional-only
/// payloads. Namespace-agnostic; returns an empty map on null/malformed XML — never throws.
/// </summary>
internal static Dictionary<string, string> ParseEventData(string xml)
{
var result = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
if (string.IsNullOrEmpty(xml)) return result;
try
{
var doc = XDocument.Parse(xml);
var positionalIndex = 0;
foreach (var el in doc.Descendants().Where(e => e.Name.LocalName == "Data"))
{
var nameAttr = el.Attribute("Name");
var key = nameAttr != null && !string.IsNullOrEmpty(nameAttr.Value)
? nameAttr.Value
: "Data" + positionalIndex++;
if (!result.ContainsKey(key))
result[key] = el.Value;
}
}
catch
{
// Malformed / unexpected XML — best effort, return what we have.
}
return result;
}
// -----------------------------------------------------------------------
// OMA-DM RebootRequired flag probe (fail-soft corroboration)
// -----------------------------------------------------------------------
private bool? SafeProbeRebootRequiredFlag()
{
try
{
return RebootRequiredFlagProbe?.Invoke();
}
catch
{
return null; // corroboration only — never let the probe break the emit
}
}
/// <summary>
/// Reads the pending-reboot flag under <see cref="OmadmSyncMlKeyPath"/>. Both a value and a
/// subkey named <c>RebootRequired</c> count as "pending". Fail-soft: null = unknown
/// (key absent / unreadable), matching the EspSkipConfigurationProbe convention.
/// </summary>
internal static bool? ReadOmadmRebootRequiredFlag()
{
try
{
using (var key = Microsoft.Win32.Registry.LocalMachine.OpenSubKey(OmadmSyncMlKeyPath))
{
if (key == null) return null;
if (key.GetValue("RebootRequired") != null) return true;
using (var sub = key.OpenSubKey("RebootRequired"))
{
return sub != null ? (bool?)true : false;
}
}
}
catch
{
return null;
}
}
// -----------------------------------------------------------------------
// Backfill
// -----------------------------------------------------------------------
private void BackfillRecentEvents()
{
try
{
var lookbackMs = (long)_backfillLookbackMinutes * 60 * 1000;
var xpath = $"*[System[(EventID={EventId_PolicyRebootRequired}) and TimeCreated[timediff(@SystemTime) <= {lookbackMs}]]]";
_logger.Info($"MdmRebootPolicy backfill: scanning {Channel} " +
$"(lookback={_backfillLookbackMinutes}min, restartWatermark={_restartWatermark})");
var query = new EventLogQuery(Channel, PathType.LogName, xpath);
int processed = 0;
using (var reader = new EventLogReader(query))
{
EventRecord record;
while ((record = reader.ReadEvent()) != null)
{
using (record)
{
ProcessRecord(record, isBackfill: true);
processed++;
}
}
}
if (processed == 0)
_logger.Debug($"MdmRebootPolicy backfill: no EventID {EventId_PolicyRebootRequired} records in last {_backfillLookbackMinutes} minutes");
else
_logger.Info($"MdmRebootPolicy backfill: scanned {processed} record(s)");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"MdmRebootPolicy event log not found during backfill: {Channel} (normal on non-MDM-enrolled test environments)");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"MdmRebootPolicy backfill access denied: {ex.Message}");
}
catch (Exception ex)
{
_logger.Warning($"MdmRebootPolicy backfill failed: {ex.Message}");
}
}
// -----------------------------------------------------------------------
// Watermark dedup (cross-restart) — persisted at flush, not at claim
// -----------------------------------------------------------------------
private bool IsAlreadyProcessed(long recordId)
{
if (recordId < 0) return false; // no RecordId → cannot dedup, let it through
lock (_stateLock)
{
return recordId <= _restartWatermark || _seenThisRun.Contains(recordId);
}
}
internal void LoadWatermark()
{
if (string.IsNullOrEmpty(_stateDirectory)) return;
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
if (!File.Exists(filePath)) return;
try
{
var json = File.ReadAllText(filePath);
var state = JsonConvert.DeserializeObject<WatermarkState>(json);
if (state != null)
{
lock (_stateLock)
{
_restartWatermark = state.LastRecordId;
_maxPersistedRecordId = state.LastRecordId;
}
_logger.Info($"MdmRebootPolicy watermark loaded: lastRecordId={state.LastRecordId} (persisted {state.PersistedUtc:O})");
}
}
catch (Exception ex)
{
_logger.Warning($"Failed to load MdmRebootPolicy watermark: {ex.Message}");
}
}
private void PersistWatermark(long recordId)
{
if (string.IsNullOrEmpty(_stateDirectory)) return;
try
{
Directory.CreateDirectory(_stateDirectory);
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
var state = new WatermarkState { LastRecordId = recordId, PersistedUtc = DateTime.UtcNow };
var json = JsonConvert.SerializeObject(state, Formatting.Indented);
var tempPath = filePath + ".tmp";
File.WriteAllText(tempPath, json);
File.Copy(tempPath, filePath, overwrite: true);
try { File.Delete(tempPath); } catch { }
}
catch (Exception ex)
{
_logger.Warning($"Failed to persist MdmRebootPolicy watermark: {ex.Message}");
}
}
internal sealed class WatermarkState
{
public long LastRecordId { get; set; }
public DateTime PersistedUtc { get; set; }
}
}
}
@@ -116,6 +116,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// failure details that fire after the settle window expires.
private Dictionary<string, Timer> _provisioningFailureSettleTimers;
private Dictionary<string, EspFailureDetectedEventArgs> _provisioningFailureSettleArgs;
// Session 4910a5a5 — post-terminal recovery story. Populated when the settle window
// expires and EspFailureDetected actually fires (terminal path); consumed by the
// esp_failure_retry_detected / esp_failure_recovered emitters so a late "Try again"
// recovery (23 min after the failure in the field session) is visible on the timeline
// instead of hiding inside a generic subcategory_state_change. Keyed by category-name.
private Dictionary<string, EspFailureDetectedEventArgs> _provisioningFailureTerminalArgs;
private Dictionary<string, DateTime> _provisioningFailureTerminalUtc;
private HashSet<string> _postFailureRetryEmitted;
private HashSet<string> _postFailureRecoveredEmitted;
// Session 2bc884b6: AppX enrichment scan seams + fire-once guard. The scanner and the
// dispatcher are injectable so tests never touch the real event log or the threadpool.
private readonly IAppxDeploymentFailureScanner _appxScanner;
@@ -285,6 +294,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_lastSubcategoryStates = new Dictionary<string, Dictionary<string, string>>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureSettleTimers = new Dictionary<string, Timer>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureSettleArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureTerminalArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
_provisioningFailureTerminalUtc = new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
_postFailureRetryEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
_postFailureRecoveredEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
_deviceSetupProvisioningCompleteFired = false;
_deviceSetupFallbackTimer = null;
_accountSetupProvisioningCompleteFired = false;
@@ -665,6 +678,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// null-ing here is correctness-neutral on the start-condition side.
ResetAccountSetupFallbackTimer();
_accountSetupProvisioningCompleteFired = true;
MaybeEmitPostFailureRecovered("AccountSetupCategory.Status", "AccountSetup");
try { AccountSetupProvisioningComplete?.Invoke(this, EventArgs.Empty); }
catch (Exception ex) { _logger.Error("AccountSetupProvisioningComplete handler failed (fallback path)", ex); }
}
@@ -833,6 +847,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// null-ing here is correctness-neutral on the start-condition side.
ResetDeviceSetupFallbackTimer();
_deviceSetupProvisioningCompleteFired = true;
MaybeEmitPostFailureRecovered("DeviceSetupCategory.Status", "DeviceSetup");
try { DeviceSetupProvisioningComplete?.Invoke(this, EventArgs.Empty); }
catch (Exception ex) { _logger.Error("DeviceSetupProvisioningComplete handler failed (fallback path)", ex); }
}
@@ -946,12 +961,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
var severity = categorySucceeded == false ? EventSeverity.Warning : EventSeverity.Info;
EmitProvisioningEvent(categoryLabel, categorySucceeded, statusText, subcategories, severity);
EmitRawRegistryDump(categoryLabel, jsonValue, $"category_resolved_{(categorySucceeded == true ? "success" : "failed")}");
// Session 4910a5a5: category resolving to success after a terminally
// fired failure = the failure recovered (fallback-path successes call
// this from their own timer handlers).
if (categorySucceeded == true)
{
MaybeEmitPostFailureRecovered(categoryName, categoryLabel);
}
}
else
{
var transitions = DetectSubcategoryTransitions(categoryName, subcategories);
StoreSubcategoryStates(categoryName, subcategories);
// Session 4910a5a5: after a terminally fired failure, the failed
// subcategory leaving "failed" is the ESP re-running the step —
// consistent with the user pressing "Try again" on the failure page.
MaybeEmitPostFailureRetry(categoryName, categoryLabel, transitions);
var failureTransitions = transitions.Where(t => t.IsFailure).ToList();
if (transitions.Count > 0)
{
@@ -1025,6 +1053,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
$"escalating as {failureTypeName} (errorCode={errorCode ?? "n/a"})");
_logger.Warning($"Provisioning failure detected: {failureTypeName}");
// Session 4910a5a5: name the likely culprit app(s) directly on the failure args so
// the DecisionEngine's advisory/terminal events carry them — never-started apps
// first (they starve ESP's app gate without ever producing an IME install event),
// capped at 3 names.
IReadOnlyList<string> likelyCulpritApps = null;
if (string.Equals(failedSubcategoryName, "Apps", StringComparison.OrdinalIgnoreCase))
{
likelyCulpritApps = SnapshotTrackedAppsNotCompleted()
.OrderBy(a => a["everStartedInstalling"] is bool started && started ? 1 : 0)
.Select(a => a["appName"]?.ToString())
.Where(n => !string.IsNullOrEmpty(n))
.Take(3)
.ToList();
}
// Caller already holds _stateLock.
ArmFailureSettleTimer(
categoryName: categoryName,
@@ -1032,7 +1075,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
failureType: failureTypeName,
errorCode: errorCode,
failedSubcategory: failedSubcategoryName,
category: categoryLabel));
category: categoryLabel,
likelyCulpritApps: likelyCulpritApps));
// Settle-window owns the actual EspFailureDetected fire. The legacy "fire-after-emit"
// path in CheckProvisioningStatus is now a no-op for this code-path; it stays in place
@@ -1292,6 +1336,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_provisioningCategoriesResolved.Remove(categoryName);
}
}
else
{
// Session 4910a5a5: remember the terminally fired failure so a LATE
// recovery (outside the settle window — e.g. user pressing "Try again"
// minutes later) can be told as esp_failure_retry_detected /
// esp_failure_recovered instead of hiding in a generic state-change.
_provisioningFailureTerminalArgs[categoryName] = args;
_provisioningFailureTerminalUtc[categoryName] = DateTime.UtcNow;
}
}
if (recovered)
@@ -1382,6 +1435,133 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
});
}
/// <summary>
/// Session 4910a5a5: after a terminally fired failure (settle window expired,
/// EspFailureDetected raised), the failed subcategory leaving the <c>failed</c> state
/// means the ESP re-ran the step — consistent with the user pressing "Try again" on
/// the ESP failure page (the agent cannot observe the click itself). Emits
/// <c>esp_failure_retry_detected</c> once per terminally fired category. Caller holds
/// <see cref="_stateLock"/>.
/// </summary>
private void MaybeEmitPostFailureRetry(string categoryName, string categoryLabel, List<SubcategoryTransition> transitions)
{
if (transitions == null || transitions.Count == 0) return;
if (_postFailureRetryEmitted.Contains(categoryName)) return;
if (_provisioningFailureTerminalArgs == null
|| !_provisioningFailureTerminalArgs.TryGetValue(categoryName, out var terminalArgs))
{
return; // No terminal fire on record — in-window recovery is settle_recovered's job.
}
SubcategoryTransition? retryTransition = null;
foreach (var t in transitions)
{
if (string.Equals(t.OldState, "failed", StringComparison.OrdinalIgnoreCase)
&& !string.Equals(t.NewState, "failed", StringComparison.OrdinalIgnoreCase))
{
retryTransition = t;
break;
}
}
if (retryTransition == null) return;
var retry = retryTransition.Value;
_postFailureRetryEmitted.Add(categoryName);
var minutesSinceFailure = MinutesSinceTerminalFailure(categoryName);
var data = new Dictionary<string, object>
{
{ "category", categoryLabel },
{ "subcategory", retry.SubcategoryName },
{ "previousState", retry.OldState },
{ "newState", retry.NewState },
{ "failureType", terminalArgs.FailureType },
{ "minutesSinceFailure", minutesSinceFailure },
};
if (!string.IsNullOrEmpty(terminalArgs.ErrorCode))
data["errorCode"] = terminalArgs.ErrorCode;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.EspFailureRetryDetected,
Severity = EventSeverity.Info,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = $"ESP retry detected — {categoryLabel}/{retry.SubcategoryName} left the failed state " +
$"({retry.OldState} → {retry.NewState}) {minutesSinceFailure} min after the failure — " +
"consistent with the user pressing 'Try again' on the ESP failure page.",
Data = data,
ImmediateUpload = true
});
}
/// <summary>
/// Session 4910a5a5: the category behind a terminally fired failure reached success
/// (categorySucceeded=true or the all-subcategories-succeeded fallback) — the failure
/// demonstrably recovered. Emits <c>esp_failure_recovered</c> once per terminally
/// fired category so the timeline tells "failed → retried → recovered" instead of
/// leaving the stale failure as the last word. Caller holds <see cref="_stateLock"/>.
/// </summary>
private void MaybeEmitPostFailureRecovered(string categoryName, string categoryLabel)
{
if (_postFailureRecoveredEmitted == null || _postFailureRecoveredEmitted.Contains(categoryName)) return;
if (_provisioningFailureTerminalArgs == null
|| !_provisioningFailureTerminalArgs.TryGetValue(categoryName, out var terminalArgs))
{
return;
}
_postFailureRecoveredEmitted.Add(categoryName);
var minutesSinceFailure = MinutesSinceTerminalFailure(categoryName);
string subcategoryStatus = null;
if (terminalArgs.FailedSubcategory != null
&& _lastSubcategoryStates != null
&& _lastSubcategoryStates.TryGetValue(categoryName, out var subs)
&& subs.TryGetValue(terminalArgs.FailedSubcategory, out var currentState))
{
subcategoryStatus = currentState;
}
var data = new Dictionary<string, object>
{
{ "category", categoryLabel },
{ "failedSubcategory", terminalArgs.FailedSubcategory ?? "category-level" },
{ "failureType", terminalArgs.FailureType },
{ "minutesSinceFailure", minutesSinceFailure },
};
if (!string.IsNullOrEmpty(terminalArgs.ErrorCode))
data["errorCode"] = terminalArgs.ErrorCode;
if (subcategoryStatus != null)
data["subcategoryState"] = subcategoryStatus;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.EspFailureRecovered,
Severity = EventSeverity.Info,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = $"ESP failure recovered — {categoryLabel}/{terminalArgs.FailedSubcategory ?? "category-level"} " +
$"completed successfully {minutesSinceFailure} min after the failure; the earlier ESP failure no longer applies.",
Data = data,
ImmediateUpload = true
});
}
private int MinutesSinceTerminalFailure(string categoryName)
{
if (_provisioningFailureTerminalUtc != null
&& _provisioningFailureTerminalUtc.TryGetValue(categoryName, out var firedAt))
{
return Math.Max(0, (int)Math.Round((DateTime.UtcNow - firedAt).TotalMinutes));
}
return 0;
}
private void EmitFailureSettleStarted(EspFailureDetectedEventArgs args)
{
var data = new Dictionary<string, object>
@@ -1863,6 +2043,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_provisioningFailureSettleTimers = new Dictionary<string, Timer>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureSettleArgs == null)
_provisioningFailureSettleArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureTerminalArgs == null)
_provisioningFailureTerminalArgs = new Dictionary<string, EspFailureDetectedEventArgs>(StringComparer.OrdinalIgnoreCase);
if (_provisioningFailureTerminalUtc == null)
_provisioningFailureTerminalUtc = new Dictionary<string, DateTime>(StringComparer.OrdinalIgnoreCase);
if (_postFailureRetryEmitted == null)
_postFailureRetryEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
if (_postFailureRecoveredEmitted == null)
_postFailureRecoveredEmitted = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
}
}
@@ -17,6 +17,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// CommercialOOBE_ESPProgress_WhiteGlove_Success — WhiteGlove complete
/// CommercialOOBE_ESPProgress_Failure/_Timeout/_Abort/WhiteGlove_Failed — ESP failure
///
/// NOTE (session b2e890c1, 2026-07-20): a bare "RebootCoalescing" substring match on 62407 was
/// tried and REMOVED — the token appears in the ROUTINE SubcategoryProcessing_Started marker
/// ("Starting subcategory DeviceSetup.RebootCoalescing...") on every enrollment, so it proves
/// nothing about an actual coalesced reboot. Policy-reboot attribution lives entirely in
/// MdmRebootPolicyTracker (EventID 2800) + the ANALYZE-ESP-005 rule gated on an observed reboot.
///
/// Raises <see cref="FinalizingSetupPhaseTriggered"/>, <see cref="WhiteGloveCompleted"/>,
/// and <see cref="EspFailureDetected"/>. Cross-notifies the <see cref="HelloTracker"/>
/// on Hello wizard start and ESP exit so Hello timers can react.
@@ -0,0 +1,72 @@
using System;
using System.Reflection;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// Reads WinRT <c>Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState</c>
/// (NotStarted=0 / InProgress=1 / Completed=2) via the CLR's built-in WinRT projection —
/// no NuGet package, no winmd reference; the type is resolved at runtime with
/// <c>ContentType=WindowsRuntime</c>, the same mechanism the bootstrap script uses from
/// PowerShell 5.1.
/// <para>
/// <b>Contract:</b> observational only — never feed this into decision-engine logic.
/// Requires Windows 10 1809+ (build 17763, UniversalApiContract 7.0) and is callable
/// from the SYSTEM service context (empirically validated, probe session 9c404ae9).
/// On older builds or SKUs without the contract the type resolves to null and
/// <see cref="Read"/> returns <c>"unavailable"</c>. <see cref="Read"/> never throws.
/// </para>
/// <para>
/// Empirical timing (Win11 24H2 probe): the state flips InProgress→Completed at desktop
/// arrival — it is a corroboration signal, not an early one. WDP reseal and placeholder
/// (fooUser) flip semantics are unverified; another reason this must stay observational.
/// </para>
/// </summary>
internal static class OobeStateReader
{
public const string Unavailable = "unavailable";
// Resolved once per process. Benign race: concurrent first calls resolve the same
// PropertyInfo and the assignment is idempotent — no lock needed.
private static PropertyInfo _stateProperty;
private static bool _initTried;
/// <summary>
/// Returns <c>"not_started"</c> / <c>"in_progress"</c> / <c>"completed"</c>,
/// <c>"unknown_&lt;n&gt;"</c> for unexpected enum values, or <c>"unavailable"</c>
/// when the WinRT contract is absent or any part of the read fails. Never throws.
/// </summary>
public static string Read()
{
try
{
if (!_initTried)
{
var type = Type.GetType(
"Windows.System.Profile.SystemSetupInfo, Windows.System.Profile, ContentType=WindowsRuntime",
throwOnError: false);
_stateProperty = type?.GetProperty(
"OutOfBoxExperienceState",
BindingFlags.Public | BindingFlags.Static);
_initTried = true;
}
if (_stateProperty == null)
return Unavailable;
var value = Convert.ToInt32(_stateProperty.GetValue(null));
switch (value)
{
case 0: return "not_started";
case 1: return "in_progress";
case 2: return "completed";
default: return "unknown_" + value;
}
}
catch
{
return Unavailable;
}
}
}
}
@@ -0,0 +1,375 @@
using System;
using System.Runtime.InteropServices;
using Microsoft.Win32.SafeHandles;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// Reads the UEFI Secure Boot signature databases <c>db</c> and <c>KEK</c> directly from
/// firmware via <c>GetFirmwareEnvironmentVariableExW</c> and checks which Microsoft
/// signing certificates they contain. This is the authoritative source for the 2023
/// certificate rollout state — the <c>SecureBoot\Servicing</c> registry key only reflects
/// whether the *Windows Update* rollout ran and verified, and lags behind when the
/// certificate reached the firmware another way (OEM factory image, WinCS, manual).
/// <para>
/// The certificate subject CNs are stored as ASCII bytes inside the DER-encoded
/// signature lists, so a plain byte-substring search is sufficient — the same check
/// Microsoft documents via <c>Get-SecureBootUEFI</c> (which wraps this exact API).
/// </para>
/// <para>
/// <b>Contract:</b> observational only — never feed this into decision-engine logic.
/// <see cref="Read"/> never throws. Reading firmware variables requires
/// <c>SeSystemEnvironmentPrivilege</c>, which SYSTEM holds but must be enabled on the
/// process token first; the previous privilege state is restored after the read.
/// On legacy BIOS the API fails with <c>ERROR_INVALID_FUNCTION</c> → status
/// <c>not_uefi</c>; every failure maps to a status string instead of an exception.
/// </para>
/// </summary>
internal static class UefiSecureBootCertReader
{
public const string StatusOk = "ok";
public const string StatusNotUefi = "not_uefi";
public const string StatusVariableNotFound = "variable_not_found";
public const string StatusPrivilegeDenied = "privilege_denied";
// EFI_IMAGE_SECURITY_DATABASE_GUID — namespace of the "db" variable.
private const string DbVariableGuid = "{d719b2cb-3d3a-4596-a3bc-dad00e67656f}";
// EFI_GLOBAL_VARIABLE — namespace of the "KEK" variable.
private const string KekVariableGuid = "{8be4df61-93ca-11d2-aa0d-00e098032b8c}";
// Subject CNs searched for in db. 2023 set per Microsoft's June-2026 expiry guidance;
// 2011 set kept for context (which legacy CAs are still present).
internal const string CertWindowsUefiCa2023 = "Windows UEFI CA 2023";
internal const string CertMicrosoftUefiCa2023 = "Microsoft UEFI CA 2023";
internal const string CertMicrosoftOptionRomCa2023 = "Microsoft Option ROM UEFI CA 2023";
internal const string CertWindowsProductionPca2011 = "Microsoft Windows Production PCA 2011";
internal const string CertMicrosoftUefiCa2011 = "Microsoft Corporation UEFI CA 2011";
// Subject CNs searched for in KEK.
internal const string CertMicrosoftKek2kCa2023 = "Microsoft Corporation KEK 2K CA 2023";
internal const string CertMicrosoftKekCa2011 = "Microsoft Corporation KEK CA 2011";
private const int ErrorInvalidFunction = 1; // legacy BIOS / no UEFI
private const int ErrorInsufficientBuffer = 122;
private const int ErrorEnvvarNotFound = 203;
private const int ErrorPrivilegeNotHeld = 1314;
private const int ErrorNotAllAssigned = 1300;
private const int InitialBufferSize = 64 * 1024;
private const int RetryBufferSize = 256 * 1024;
// SeSystemEnvironmentPrivilege is adjusted on the PROCESS token, so enable → read →
// restore is a process-wide side effect and must run as one atomic sequence. Without
// this gate a second reader would observe "already enabled", correctly skip its own
// restore, and then have the privilege disabled underneath it when the first reader
// finishes — a spurious privilege_denied on a machine that actually holds it.
private static readonly object FirmwareReadGate = new object();
/// <summary>
/// Reads db and KEK and evaluates the certificate presence flags. Never throws;
/// per-variable failures surface as <see cref="UefiSecureBootCertSnapshot.DbStatus"/> /
/// <see cref="UefiSecureBootCertSnapshot.KekStatus"/>. Certificate flags are only
/// meaningful when the corresponding status is <see cref="StatusOk"/>.
/// </summary>
public static UefiSecureBootCertSnapshot Read()
{
var snapshot = new UefiSecureBootCertSnapshot();
try
{
// The critical section covers the whole privileged window, not just the
// adjustment — see FirmwareReadGate. Only synchronous kernel calls and byte
// scans run inside it, so it cannot block on anything else the agent does.
lock (FirmwareReadGate)
using (var privilege = SystemEnvironmentPrivilege.TryEnable())
{
if (!privilege.Enabled)
{
snapshot.DbStatus = StatusPrivilegeDenied;
snapshot.KekStatus = StatusPrivilegeDenied;
return snapshot;
}
byte[] db;
snapshot.DbStatus = TryReadVariable("db", DbVariableGuid, out db);
if (snapshot.DbStatus == StatusOk)
{
snapshot.DbHasWindowsUefiCa2023 = ContainsAscii(db, CertWindowsUefiCa2023);
snapshot.DbHasMicrosoftUefiCa2023 = ContainsAscii(db, CertMicrosoftUefiCa2023);
snapshot.DbHasMicrosoftOptionRomCa2023 = ContainsAscii(db, CertMicrosoftOptionRomCa2023);
snapshot.DbHasWindowsProductionPca2011 = ContainsAscii(db, CertWindowsProductionPca2011);
snapshot.DbHasMicrosoftUefiCa2011 = ContainsAscii(db, CertMicrosoftUefiCa2011);
}
byte[] kek;
snapshot.KekStatus = TryReadVariable("KEK", KekVariableGuid, out kek);
if (snapshot.KekStatus == StatusOk)
{
snapshot.KekHasMicrosoftKek2kCa2023 = ContainsAscii(kek, CertMicrosoftKek2kCa2023);
snapshot.KekHasMicrosoftKekCa2011 = ContainsAscii(kek, CertMicrosoftKekCa2011);
}
}
}
catch (Exception ex)
{
var status = "error_" + ex.GetType().Name;
if (snapshot.DbStatus == null) snapshot.DbStatus = status;
if (snapshot.KekStatus == null) snapshot.KekStatus = status;
}
return snapshot;
}
/// <summary>
/// Reads one firmware variable into <paramref name="bytes"/> (trimmed to the actual
/// length). Returns a status string; <paramref name="bytes"/> is null unless the
/// status is <see cref="StatusOk"/>.
/// </summary>
private static string TryReadVariable(string name, string guid, out byte[] bytes)
{
bytes = null;
var buffer = new byte[InitialBufferSize];
var length = NativeMethods.GetFirmwareEnvironmentVariableExW(name, guid, buffer, (uint)buffer.Length, IntPtr.Zero);
if (length == 0)
{
var error = Marshal.GetLastWin32Error();
if (error == ErrorInsufficientBuffer)
{
buffer = new byte[RetryBufferSize];
length = NativeMethods.GetFirmwareEnvironmentVariableExW(name, guid, buffer, (uint)buffer.Length, IntPtr.Zero);
if (length == 0)
error = Marshal.GetLastWin32Error();
}
if (length == 0)
{
switch (error)
{
case ErrorInvalidFunction: return StatusNotUefi;
case ErrorEnvvarNotFound: return StatusVariableNotFound;
case ErrorPrivilegeNotHeld: return StatusPrivilegeDenied;
default: return "error_" + error;
}
}
}
bytes = new byte[length];
Array.Copy(buffer, bytes, (int)length);
return StatusOk;
}
/// <summary>
/// Byte-level ASCII substring search (ordinal, case-sensitive). The signature lists
/// are at most a few hundred KB and read once per agent start — a naive scan is fine.
/// </summary>
internal static bool ContainsAscii(byte[] haystack, string needle)
{
if (haystack == null || string.IsNullOrEmpty(needle) || needle.Length > haystack.Length)
return false;
for (var i = 0; i <= haystack.Length - needle.Length; i++)
{
var j = 0;
while (j < needle.Length && haystack[i + j] == (byte)needle[j])
j++;
if (j == needle.Length)
return true;
}
return false;
}
/// <summary>
/// Enables <c>SeSystemEnvironmentPrivilege</c> on the process token for the lifetime
/// of the instance and restores the previous privilege state on dispose. SYSTEM holds
/// the privilege (default-disabled); without elevation <see cref="Enabled"/> stays
/// false and the caller reports <see cref="StatusPrivilegeDenied"/>.
/// <para>
/// <b>Crash-safety.</b> This is the repo's only privilege-adjustment code, so the
/// ownership rules are spelled out: the token is held in a <see cref="SafeHandle"/>
/// assigned by the P/Invoke marshaler, so the CLR owns it from the moment
/// <c>OpenProcessToken</c> returns — it is released by the critical finalizer even if
/// this instance never reaches the caller's <c>using</c> (async exception between the
/// call and the assignment). <see cref="Dispose"/> is idempotent, never throws, and
/// releases the handle in a <c>finally</c> so a failed restore cannot leak it. The
/// restore obligation is recorded immediately after the successful adjustment, so no
/// throw in between can strand the process with the privilege left enabled.
/// </para>
/// </summary>
private sealed class SystemEnvironmentPrivilege : IDisposable
{
public bool Enabled { get; private set; }
private SafeTokenHandle _token;
private NativeMethods.TOKEN_PRIVILEGES _previousState;
private bool _restoreNeeded;
private bool _disposed;
public static SystemEnvironmentPrivilege TryEnable()
{
var instance = new SystemEnvironmentPrivilege();
try
{
SafeTokenHandle token;
if (!NativeMethods.OpenProcessToken(NativeMethods.GetCurrentProcess(),
NativeMethods.TOKEN_ADJUST_PRIVILEGES | NativeMethods.TOKEN_QUERY, out token))
{
// Failed open still yields a (closed/invalid) SafeHandle instance.
if (token != null) token.Dispose();
return instance;
}
instance._token = token;
NativeMethods.LUID luid;
if (!NativeMethods.LookupPrivilegeValueW(null, "SeSystemEnvironmentPrivilege", out luid))
return instance;
var newState = new NativeMethods.TOKEN_PRIVILEGES
{
PrivilegeCount = 1,
Luid = luid,
Attributes = NativeMethods.SE_PRIVILEGE_ENABLED
};
NativeMethods.TOKEN_PRIVILEGES previousState;
uint returnLength;
if (!NativeMethods.AdjustTokenPrivileges(token, false, ref newState,
(uint)Marshal.SizeOf(typeof(NativeMethods.TOKEN_PRIVILEGES)),
out previousState, out returnLength))
return instance;
// AdjustTokenPrivileges returns TRUE even when the privilege is not held —
// the real verdict is ERROR_NOT_ALL_ASSIGNED in the last error. Nothing was
// adjusted in that case, so there is nothing to restore.
if (Marshal.GetLastWin32Error() == ErrorNotAllAssigned)
return instance;
// The token is now modified: record the restore obligation before anything
// else can throw. Restore only when the privilege was not already enabled.
instance._previousState = previousState;
instance._restoreNeeded = previousState.PrivilegeCount > 0
&& (previousState.Attributes & NativeMethods.SE_PRIVILEGE_ENABLED) == 0;
instance.Enabled = true;
}
catch
{
// Fail-soft: Enabled stays false. Whatever was acquired before the failure
// is already tracked on the instance and released by Dispose.
}
return instance;
}
public void Dispose()
{
if (_disposed) return;
_disposed = true;
Enabled = false;
var token = _token;
_token = null;
if (token == null) return;
try
{
if (_restoreNeeded)
{
_restoreNeeded = false;
NativeMethods.TOKEN_PRIVILEGES ignored;
uint returnLength;
NativeMethods.AdjustTokenPrivileges(token, false, ref _previousState,
(uint)Marshal.SizeOf(typeof(NativeMethods.TOKEN_PRIVILEGES)),
out ignored, out returnLength);
}
}
catch
{
// Best effort — the process token privilege state is not load-bearing for
// anything else the agent does, and Dispose must never throw.
}
finally
{
token.Dispose();
}
}
}
/// <summary>
/// Owns a process-token handle. <c>ReleaseHandle</c> runs on the finalizer thread when
/// <see cref="SystemEnvironmentPrivilege.Dispose"/> is skipped, so it must not throw —
/// a single <c>CloseHandle</c> P/Invoke satisfies that.
/// </summary>
private sealed class SafeTokenHandle : SafeHandleZeroOrMinusOneIsInvalid
{
public SafeTokenHandle() : base(true) { }
protected override bool ReleaseHandle()
{
return NativeMethods.CloseHandle(handle);
}
}
private static class NativeMethods
{
public const uint TOKEN_ADJUST_PRIVILEGES = 0x0020;
public const uint TOKEN_QUERY = 0x0008;
public const uint SE_PRIVILEGE_ENABLED = 0x0002;
[StructLayout(LayoutKind.Sequential)]
public struct LUID
{
public uint LowPart;
public int HighPart;
}
// Single-privilege variant of the variable-length native struct — sufficient here
// because exactly one privilege is adjusted per call.
[StructLayout(LayoutKind.Sequential)]
public struct TOKEN_PRIVILEGES
{
public uint PrivilegeCount;
public LUID Luid;
public uint Attributes;
}
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
public static extern uint GetFirmwareEnvironmentVariableExW(
string lpName, string lpGuid, byte[] pBuffer, uint nSize, IntPtr pdwAttributes);
[DllImport("kernel32.dll")]
public static extern IntPtr GetCurrentProcess();
[DllImport("advapi32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool OpenProcessToken(IntPtr processHandle, uint desiredAccess, out SafeTokenHandle tokenHandle);
[DllImport("advapi32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool LookupPrivilegeValueW(string lpSystemName, string lpName, out LUID lpLuid);
[DllImport("advapi32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool AdjustTokenPrivileges(SafeTokenHandle tokenHandle, bool disableAllPrivileges,
ref TOKEN_PRIVILEGES newState, uint bufferLength, out TOKEN_PRIVILEGES previousState, out uint returnLength);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
public static extern bool CloseHandle(IntPtr handle);
}
}
/// <summary>
/// Result of one firmware read. Certificate flags are only meaningful when the
/// corresponding status equals <see cref="UefiSecureBootCertReader.StatusOk"/>.
/// </summary>
internal sealed class UefiSecureBootCertSnapshot
{
public string DbStatus { get; set; }
public string KekStatus { get; set; }
public bool DbHasWindowsUefiCa2023 { get; set; }
public bool DbHasMicrosoftUefiCa2023 { get; set; }
public bool DbHasMicrosoftOptionRomCa2023 { get; set; }
public bool DbHasWindowsProductionPca2011 { get; set; }
public bool DbHasMicrosoftUefiCa2011 { get; set; }
public bool KekHasMicrosoftKek2kCa2023 { get; set; }
public bool KekHasMicrosoftKekCa2011 { get; set; }
}
}
@@ -31,84 +31,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
_logger.Info($"Executing FULL SELF-DESTRUCT (Scheduled Task + Files{(_configuration.RebootOnComplete ? " + Reboot" : "")})");
var currentProcessId = Process.GetCurrentProcess().Id;
var currentProcess = Process.GetCurrentProcess();
var agentBasePath = Path.GetFullPath(Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"AutopilotMonitor"
));
var logDir = Path.GetFullPath(Path.Combine(agentBasePath, "Logs"));
var keepLogs = _configuration.KeepLogFile;
// Create a self-deleting PowerShell script
var cleanupScript = $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Wait for agent process to exit
Start-Sleep -Seconds 2
Wait-Process -Id {currentProcessId} -ErrorAction SilentlyContinue -Timeout 30
Start-Sleep -Seconds 1
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan task firing
# against a deleted exe spams Task Scheduler errors every boot forever and, once the files (incl.
# the enrollment-complete marker) are gone, has no cleanup-retry path. So confirm the task is gone
# first; if it cannot be removed, SKIP the file deletion and leave everything intact for a clean
# cleanup retry on the next boot (the marker survives, so bootstrap re-runs cleanup).
$taskName = '{_configuration.ScheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
{(keepLogs ? $@"
if ($taskGone) {{
# Delete everything except Logs directory
Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' | ForEach-Object {{
$dest = $_.FullName + '.del'
try {{ Rename-Item -Path $_.FullName -NewName $dest -Force -ErrorAction SilentlyContinue }} catch {{ }}
Remove-Item -Path $dest -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
" : $@"
if ($taskGone) {{
# Rename folder then delete. Retry a few times to let the OS release handles.
$renamedPath = '{agentBasePath}.del'
$renamed = $false
for ($i = 1; $i -le 10; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$renamed = $true
break
}} catch {{
Start-Sleep -Seconds 2
}}
}}
if ($renamed) {{
Remove-Item -Path $renamedPath -Recurse -Force -ErrorAction SilentlyContinue
}} else {{
Remove-Item -Path '{agentBasePath}' -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
")}
{(_configuration.RebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
";
var cleanupScript = BuildCleanupScript(
agentProcessName: currentProcess.ProcessName,
agentBasePath: agentBasePath,
keepLogs: _configuration.KeepLogFile,
scheduledTaskName: _configuration.ScheduledTaskName,
rebootOnComplete: _configuration.RebootOnComplete);
// Write cleanup script to temp location (outside of agent folder)
var tempScriptPath = Path.Combine(Path.GetTempPath(), $"AutopilotMonitor-Cleanup-{Guid.NewGuid()}.ps1");
@@ -143,5 +77,171 @@ Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}
}
/// <summary>
/// Builds the self-deleting cleanup PowerShell script. Extracted (internal static, pure)
/// so tests can pin the hardened invariants without launching processes.
/// <para>
/// LIFE-F7 hardening (incident 2026-07-22 — exe + loaded DLLs orphaned forever):
/// </para>
/// <list type="number">
/// <item><b>Single-flight lock</b> — the enrollment-complete marker is written BEFORE
/// cleanup launches, so any agent start during an in-flight cleanup spawns a SECOND
/// cleanup script. Two unsynchronized scripts interleave their rename/delete phases
/// and leave partial trees behind. A CreateNew lock file in the SYSTEM temp dir
/// serializes them; a stale lock (&gt;10 min) is taken over.</item>
/// <item><b>Wait by process NAME, not launcher PID</b> — marker-retry instances,
/// self-update restarts and manual runs all load the same images but have PIDs the
/// old <c>Wait-Process -Id</c> never saw (and an already-freed PID can be reused by
/// an unrelated boot-time process). After the deadline any straggler is force-killed:
/// its images are about to be deleted anyway.</item>
/// <item><b>Probe-rename BEFORE task removal, never delete the original path</b> — the
/// rename is the atomic lock probe. If it fails, something still holds the tree: abort
/// with task + files + marker intact so the next boot retries (file-side symmetry to
/// the LIFE-F6 task-side rule). If the task cannot be removed afterwards, rename BACK
/// and abort for the same reason. Only a fully renamed tree with the task gone is
/// deleted — the old <c>Remove-Item</c>-on-original fallback deleted marker + state
/// around locked binaries AFTER the task was gone, converting a transient lock into a
/// permanently orphaned exe.</item>
/// </list>
/// </summary>
/// <remarks>
/// The timing parameters exist ONLY so integration tests can replay the incident
/// scenarios (locked tree, straggler process) in seconds instead of minutes —
/// production call sites always use the defaults.
/// </remarks>
internal static string BuildCleanupScript(
string agentProcessName,
string agentBasePath,
bool keepLogs,
string scheduledTaskName,
bool rebootOnComplete,
int initialDelaySeconds = 2,
int processWaitSeconds = 60,
int renameRetries = 10,
int renameRetryDelaySeconds = 2)
{
return $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Single-flight lock (LIFE-F7 #1): serialize concurrent cleanup scripts. CreateNew is
# atomic; the loser backs off and lets the winner own the deletion.
$lockPath = Join-Path ([System.IO.Path]::GetTempPath()) 'AutopilotMonitor-Cleanup.lock'
$lockStream = $null
try {{
if (Test-Path $lockPath) {{
$lockAge = (Get-Date) - (Get-Item $lockPath -ErrorAction Stop).LastWriteTime
if ($lockAge.TotalMinutes -gt 10) {{ Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue }}
}}
$lockStream = [System.IO.File]::Open($lockPath, [System.IO.FileMode]::CreateNew, [System.IO.FileAccess]::Write, [System.IO.FileShare]::None)
}} catch {{
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
exit 0
}}
try {{
# Wait for EVERY agent process to exit, by NAME (LIFE-F7 #2) — covers marker-retry
# instances, self-update restarts and manual runs, and is immune to PID reuse.
Start-Sleep -Seconds {initialDelaySeconds}
$deadline = (Get-Date).AddSeconds({processWaitSeconds})
while ((Get-Date) -lt $deadline) {{
if (-not (Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue)) {{ break }}
Start-Sleep -Seconds 2
}}
# Backstop: force-kill stragglers — their images are about to be deleted anyway.
Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
Start-Sleep -Seconds 1
{(keepLogs ? $@"
# Probe-rename all children except Logs (LIFE-F7 #3). Any failure ⇒ rename back and
# abort with the Scheduled Task intact so the next boot retries cleanly.
$renamed = @()
$probeOk = $true
foreach ($item in @(Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' -ErrorAction SilentlyContinue)) {{
$dest = $item.FullName + '.del-' + [Guid]::NewGuid().ToString('N')
$itemOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path $item.FullName -NewName $dest -Force -ErrorAction Stop
$itemOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
if ($itemOk) {{ $renamed += @{{ Original = $item.FullName; Renamed = $dest }} }}
else {{ $probeOk = $false; break }}
}}
" : $@"
# Probe-rename the whole tree (LIFE-F7 #3). Failure ⇒ abort with task + files + marker
# intact so the next boot retries cleanly. GUID suffix avoids collisions with leftovers
# of earlier attempts.
$renamedPath = '{agentBasePath}.del-' + [Guid]::NewGuid().ToString('N')
$probeOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$probeOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
$renamed = @(@{{ Original = '{agentBasePath}'; Renamed = $renamedPath }})
")}
if ($probeOk) {{
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan
# task firing against a deleted exe spams Task Scheduler errors every boot forever
# and, once the files (incl. the enrollment-complete marker) are gone, has no
# cleanup-retry path.
$taskName = '{scheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
if ($taskGone) {{
# Task gone + tree renamed ⇒ safe to delete. Never touches the original path.
foreach ($entry in $renamed) {{
Remove-Item -Path $entry.Renamed -Recurse -Force -ErrorAction SilentlyContinue
}}
}} else {{
# Task survived ⇒ rename back and leave everything for the next-boot retry
# (the marker survives, so bootstrap re-runs cleanup).
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
}} else {{
# Probe failed ⇒ something still holds the tree. Rename back whatever succeeded and
# abort WITHOUT deleting anything — task + marker stay, next boot retries.
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
{(rebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
}} finally {{
if ($lockStream) {{ $lockStream.Dispose() }}
Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}}
";
}
}
}
@@ -370,7 +370,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
try
{
var versionUrl = $"{Constants.AgentBlobBaseUrl}/{Constants.AgentVersionFileNameForLine(2)}";
// Download alias, not the legacy blob account: the legacy mirror in the
// release pipeline is fail-soft, so it can silently lag behind eu.
var versionUrl = $"{Constants.AgentDownloadBaseUrl}/{Constants.AgentVersionFileNameForLine(2)}";
using (var handler = new HttpClientHandler())
using (var client = new HttpClient(handler) { Timeout = TimeSpan.FromMilliseconds(VersionCheckTimeoutMs) })
@@ -497,7 +499,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
try
{
var zipUrl = $"{Constants.AgentBlobBaseUrl}/{Constants.AgentZipFileNameForLine(2)}";
var zipUrl = $"{Constants.AgentDownloadBaseUrl}/{Constants.AgentZipFileNameForLine(2)}";
// Clean up any previous download (fragment from aborted run, or symlink/junction).
// Pre-delete + FileMode.CreateNew below: CreateNew refuses to open an existing
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Management;
using System.Text.RegularExpressions;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
@@ -28,6 +29,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
///
/// Emits a single "local_admin_analysis" event at startup and at shutdown,
/// enabling delta detection between pre- and post-enrollment state.
///
/// Tenant-supplied allowed accounts may contain glob wildcards (e.g. "adm-*");
/// see <see cref="MatchesAllowedEntry"/>.
/// </summary>
public class LocalAdminAnalyzer : IAgentAnalyzer
{
@@ -246,6 +250,43 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
}
}
// -----------------------------------------------------------------------
// Allowed-list matching
// -----------------------------------------------------------------------
/// <summary>
/// Matches an account or profile-folder name against one allowed-list entry.
/// Entries containing <c>*</c> or <c>?</c> are glob patterns (same semantics as the
/// hardware whitelist: <c>*</c> = any run, <c>?</c> = one char, case-insensitive);
/// all other entries match exactly. No ambiguity with literal names — <c>*</c> and
/// <c>?</c> are invalid characters in Windows account names.
/// Internal for direct unit-testing via InternalsVisibleTo.
/// </summary>
internal static bool MatchesAllowedEntry(string name, string entry)
{
if (entry.IndexOf('*') < 0 && entry.IndexOf('?') < 0)
return string.Equals(entry, name, StringComparison.OrdinalIgnoreCase);
var regexPattern = "^" + Regex.Escape(entry)
.Replace("\\*", ".*")
.Replace("\\?", ".") + "$";
try
{
// Timeout guards against pathological patterns (ReDoS)
return Regex.IsMatch(name, regexPattern, RegexOptions.IgnoreCase, TimeSpan.FromMilliseconds(100));
}
catch (RegexMatchTimeoutException)
{
return false;
}
}
private static bool IsAllowed(string name, List<string> allowedAccounts)
{
return allowedAccounts.Any(a => MatchesAllowedEntry(name, a));
}
// -----------------------------------------------------------------------
// Individual checks
// -----------------------------------------------------------------------
@@ -308,8 +349,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
if (disabled)
continue;
if (!allowedAccounts.Any(a =>
string.Equals(a, name, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(name, allowedAccounts))
{
unexpected.Add(name);
_logger.Debug($"{Name}: Unexpected local account: {name}");
@@ -349,8 +389,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
allFound.Add(folderName);
if (!allowedAccounts.Any(a =>
string.Equals(a, folderName, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(folderName, allowedAccounts))
{
unexpected.Add(folderName);
_logger.Debug($"{Name}: Unexpected profile folder: {folderName}");
@@ -8,6 +8,7 @@ using System.Text.Json;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Security;
using AutopilotMonitor.DecisionCore.Engine;
@@ -493,7 +494,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
}
}
EmitDeviceInfoEvent(Constants.EventTypes.SecureBootStatus, $"SecureBoot: {data["uefiSecureBootEnabled"]}, CA2023: {data["uefiCA2023Status"]}", data);
// Direct firmware evidence (UEFI db/KEK variables) — authoritative for the
// CA-2023 rollout state. The Servicing key above only mirrors whether the
// Windows Update rollout ran; it lags behind when the certificate reached
// the firmware another way (OEM factory image, WinCS, manual deployment).
var firmware = UefiSecureBootCertReader.Read();
AppendFirmwareCertFields(data, firmware);
EmitDeviceInfoEvent(Constants.EventTypes.SecureBootStatus,
$"SecureBoot: {data["uefiSecureBootEnabled"]}, CA2023: {data["uefiCA2023Status"]}, firmware: {DescribeFirmwareVerdict(firmware)}",
data);
}
catch (Exception ex)
{
@@ -501,6 +511,50 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
}
}
/// <summary>
/// Maps a <see cref="UefiSecureBootCertSnapshot"/> onto the <c>secureboot_status</c>
/// payload. Certificate booleans are only emitted when the respective variable read
/// succeeded — an absent boolean means "firmware state unknown", never "missing".
/// </summary>
internal static void AppendFirmwareCertFields(Dictionary<string, object> data, UefiSecureBootCertSnapshot firmware)
{
if (firmware == null) return;
data["uefiFirmwareReadStatus"] = firmware.DbStatus != UefiSecureBootCertReader.StatusOk
? firmware.DbStatus
: firmware.KekStatus;
if (firmware.DbStatus == UefiSecureBootCertReader.StatusOk)
{
data["uefiDbHasWindowsUefiCa2023"] = firmware.DbHasWindowsUefiCa2023;
data["uefiDbHasMicrosoftUefiCa2023"] = firmware.DbHasMicrosoftUefiCa2023;
data["uefiDbHasMicrosoftOptionRomCa2023"] = firmware.DbHasMicrosoftOptionRomCa2023;
data["uefiDbHasWindowsProductionPca2011"] = firmware.DbHasWindowsProductionPca2011;
data["uefiDbHasMicrosoftUefiCa2011"] = firmware.DbHasMicrosoftUefiCa2011;
// One-sided marker consumed by ANALYZE-SEC-001's not_exists precondition.
// The engine's not_exists passes only while the field is absent/empty, so the
// marker must be OMITTED (never emitted as false) unless the firmware db
// positively confirms the 2023 certificate — old agents and unconfirmed
// devices then keep evaluating the rule.
if (firmware.DbHasWindowsUefiCa2023)
data["uefiCa2023FirmwareConfirmed"] = true;
}
if (firmware.KekStatus == UefiSecureBootCertReader.StatusOk)
{
data["uefiKekHasMicrosoftKek2kCa2023"] = firmware.KekHasMicrosoftKek2kCa2023;
data["uefiKekHasMicrosoftKekCa2011"] = firmware.KekHasMicrosoftKekCa2011;
}
}
internal static string DescribeFirmwareVerdict(UefiSecureBootCertSnapshot firmware)
{
if (firmware == null) return "unavailable";
if (firmware.DbStatus != UefiSecureBootCertReader.StatusOk) return firmware.DbStatus ?? "unavailable";
return firmware.DbHasWindowsUefiCa2023 ? "CA2023 confirmed" : "CA2023 missing";
}
private void CollectBitLockerStatus()
{
try
@@ -652,10 +706,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
/// the desktop via "Continue anyway".
/// </para>
/// </summary>
// Serializes the ESP-config read+emit across ALL callers (CollectAll on the startup
// ThreadPool task, CollectAtEnrollmentStart and RefreshEspConfiguration on their own
// Task.Run) — without it, two concurrent collects can interleave read-before-emit and
// the LATER emission can carry the OLDER (smaller) registry snapshot. The backend keeps
// the latest list-bearing emission as positive blocking evidence, so an out-of-order
// emission would silently shrink it (Codex review). Inside the lock each collect is
// read→emit atomic: a later emission always carries an equal-or-newer snapshot.
private readonly object _espConfigCollectLock = new object();
// Internal (not private) as a test seam: unlike CollectAll/CollectAtEnrollmentStart this
// method touches no live system state once both probes are overridden, so tests can
// drive the esp_config_detected event surface in isolation (InternalsVisibleTo).
internal (bool? skipUserStatusPage, bool? skipDeviceStatusPage) CollectEspConfiguration()
{
lock (_espConfigCollectLock)
{
return CollectEspConfigurationLocked();
}
}
private (bool? skipUserStatusPage, bool? skipDeviceStatusPage) CollectEspConfigurationLocked()
{
bool? skipUser = null;
bool? skipDevice = null;

Some files were not shown because too many files have changed in this diff Show More