Compare commits

..

94 Commits

Author SHA1 Message Date
dependabot[bot] beac51720f build(deps): bump the web group (#135)
Bumps the web group in /src/Web/autopilot-monitor-web with 4 updates: [next](https://github.com/vercel/next.js), [@next/bundle-analyzer](https://github.com/vercel/next.js/tree/HEAD/packages/next-bundle-analyzer), [eslint-config-next](https://github.com/vercel/next.js/tree/HEAD/packages/eslint-config-next) and [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest).


Updates `next` from 16.3.1 to 16.3.2
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/compare/v16.3.1...v16.3.2)

Updates `@next/bundle-analyzer` from 16.3.1 to 16.3.2
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/commits/v16.3.2/packages/next-bundle-analyzer)

Updates `eslint-config-next` from 16.3.1 to 16.3.2
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/commits/v16.3.2/packages/eslint-config-next)

Updates `vitest` from 4.1.10 to 4.1.11
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.11/packages/vitest)

---
updated-dependencies:
- dependency-name: next
  dependency-version: 16.3.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: "@next/bundle-analyzer"
  dependency-version: 16.3.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: eslint-config-next
  dependency-version: 16.3.2
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: vitest
  dependency-version: 4.1.11
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-25 12:32:44 +02:00
dependabot[bot] 32268ba245 build(deps-dev): bump vitest (#134)
Bumps the mcp group in /src/McpServer/autopilot-monitor-mcp with 1 update: [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest).


Updates `vitest` from 4.1.10 to 4.1.11
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.11/packages/vitest)

---
updated-dependencies:
- dependency-name: vitest
  dependency-version: 4.1.11
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: mcp
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-25 12:32:17 +02:00
Oliver Kieselbach c91d8d4501 fix(agent): complete enrollments with zero user-targeted apps instead of failing them
IME never writes its "Completed user session" line when the user has no
assigned Win32 apps - its zero-app branch returns before that statement
(ApplicationPoller.cs:713-721, identical in 1.80/1.83/1.97/1.104). So
ImeUserSessionCompletedUtc stayed null, every arm of
ShouldTransitionToAwaitingHello was unsatisfiable, and the 30-min
advisory_completion backstop resolved healthy enrollments as
esp_exit_without_completion_evidence. One tenant: 10 of 30 sessions in a week,
all false positives - DAD-validated desktop, Hello provisioned, RealmJoin still
installing. Every Succeeded session there carries ime_user_session_completed;
not one of the failed ones does.

IME-USER-SESSION-ZERO-APPS reads the line IME does write there and routes it
through the same OnUserSessionCompleted callback. That is IME's own verdict
rather than a weakened gate: the zero-app branch locks the ESP in with an empty
application list, and the sibling emit site returns ProvisioningComplete right
after logging it. Guarded to the AccountSetup phase so it cannot burn the
adapter fire-once flag on a pre-sign-in timestamp, and replayed on the phase
transition when the observation predates the phase marker. AreUserEspAppsSettled
accepts an empty app list only with that explicit observation, never on
emptiness alone.

Second, independent defect: advisory_completion (30 min) overran the RealmJoin
completion gate, which deliberately holds the session and carries its own
bounded resolution (60 min from detection, activity-extended, 4 h cap). One
session installed 25 RealmJoin packages inside the window, the last 2 min before
the failure, while completion_waiting still read "waiting on:
realmjoin_resolution". The handler now re-arms while that gate is closed and its
own deadline is still armed; convergence comes from the gate itself, which
always ends with an Outcome set.

ime_user_session_completed gains an evidence field (zero_user_apps_enumerated /
user_session_completed) - no new event type, so the Constants ->
resource-catalog -> shared-manifests triple stays untouched. The pattern ships
via the config API and reaches existing agents, where an unknown action is a
single Debug line; the behaviour needs the agent release.

Build 0 warnings. DecisionCore 558/558, V2.Core 2530/2530, Backend 4634/4634,
MCP event-types-drift 6/6.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-25 12:27:42 +02:00
Oliver Kieselbach bf6850dedc perf(security): log DevPrep association shadow once per lookup, not per request
Same gate the cert-device-binding shadow got in 35a3ed0c, applied to its
sibling. Measured on the live enrollment: 263 identical shadow lines for one
device off a handful of Graph calls, because the cache answered nearly every
request and the log sat outside it.

DeviceAssociationResult.AsCacheHit() copies rather than marking the shared
cached instance, so the first real lookup is never mistaken for a cache hit
and dropped. Transient outcomes are not cached and keep logging every time.

Backend 4633/4633, 0 warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 20:18:57 +02:00
Oliver Kieselbach 35a3ed0cfb perf(security): log device binding once per Graph lookup, not per request
Measured on the first live enrollment: 136 agent requests produced exactly
ONE Graph call, because the 30-minute positive cache absorbed the rest. So
the check is not a throttling risk (the per-request Autopilot serial
validator made 25 calls in the same window) and the positive TTL needs no
change — raising it would only delay noticing a wiped or removed device.

What the 136 requests did cost was 136 log lines all saying the same thing.
Gate the shadow log on ServedFromCache so it emits one line per real lookup.
Cached repeats carry no new information: the reported age just counts up
mechanically from the same enrolledDateTime. Outcomes that are never cached
(Transient, NoDeviceIdInCert) keep logging every time, which is what an
anomaly warrants.

AsCacheHit() copies rather than marking the shared cached instance, so the
first real lookup is never mistaken for a cache hit and dropped.

Backend 4631/4631, 0 warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 19:18:37 +02:00
Oliver Kieselbach ea6a375c4b feat(security): cert-to-device binding, stage 3 GA-gated shadow
Tenant binding (stage 1) proves the client certificate was issued to this
tenant. It does not prove the certificate belongs to a device the tenant
still has: a cert lifted from a decommissioned machine keeps working.

IntuneDeviceBindingValidator resolves the Intune managedDevice id from the
certificate's Subject CN against the tenant's own managedDevices inventory.
Point-GET on managedDevices/{id} rather than an OData filter, so there is no
filter literal to inject into and a 404 is unambiguous. Cache/retry/transient
contract mirrors CloudPcDeviceValidator.

Shadow only, and visible only to Global Admins (same gate as the DevPrep
association preview) because the open question is a race, not an attack: a
device object can appear after the agent's first call. Every result carries
enrolledDateTime and the log line records enrolledAgeSeconds, so a device
created seconds before the request is distinguishable from a foreign cert.
Grep marker: CERT-DEVICE-BINDING-SHADOW.

The lookup is fire-and-forget like the DevPrep shadow so a cold Graph
round-trip never sits in the agent request path. That rules out stage 1's
request-row dimension (the row is written before the task finishes), hence
the Warning log; widening beyond preview should move to inline + dimension.

Needs DeviceManagementManagedDevices.Read.All, granted per tenant via the
customer script as feature IntuneDeviceBinding. The script's catalog,
ValidateSet and help block stay pinned to GraphFeatureCatalog by
GraphFeatureCatalogSyncTests (extended with the help-block case).

Build 0 warnings, backend 4629/4629, web 874/874, tsc clean, eslint 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 18:26:33 +02:00
Oliver Kieselbach fb79c7711c fix(security): carry cert tenant binding on the request row, not a dead log line
Post-deploy verification of 249e39bc found zero AgentCertTenantBinding rows
across 32k agent requests. Cause: the Match outcome was emitted with
LogInformation, and worker-side ILogger below Warning never reaches App
Insights (provider default rule is Warning+, host forwarding of worker logs
is off). The bulk outcome was therefore invisible, which is indistinguishable
from the code never running — the shadow telemetry had no denominator.

Stamp every outcome onto the request row instead (FunctionContext.Items ->
RequestTelemetryMiddleware -> CertTenantBinding dimension). That row already
exists per request and is unsampled, so the denominator costs no extra
telemetry. Non-Match outcomes keep their Warning trace with full context.

FromCache existed only to sample the dead Match line and is removed with it.

Backend 4606/4606, 0 warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 16:40:04 +02:00
Oliver Kieselbach 249e39bc68 feat(security): cert tenant binding, stage 1 shadow mode
A chain-valid Intune MDM certificate only proves "issued to some tenant" —
the pinned roots are shared by every Intune tenant. Tenant scoping rested
entirely on the device serial in the downstream validators, which is not a
secret. Bind the tenant stamped into the auth cert to the requested tenant.

The Entra tenant id sits in OID 1.2.840.113556.5.14 of the cert the agent
already presents for mTLS, so this needs no Graph call, no consent and no
agent release. Not 5.6 (Intune Account id, a different GUID) and not 5.4
(device id, stored as 16 bare bytes) — the decoder handles both encodings
and now lives once in Shared, used by agent and backend.

Stage 1 observes only: ObserveCertTenantBinding returns void and swallows
its own exceptions, so it cannot fail an otherwise valid request. Match is
logged only on cert-cache miss, keeping the denominator without per-request
volume. Grep marker for stage 2: CERT-TENANT-BINDING-SHADOW.

Backend 4604/4604, Agent V2.Core 2519/2519, 0 warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 16:03:05 +02:00
Oliver Kieselbach 6bd9139869 fix(agent): cert GUID parser accepts bare 16-byte extension values
Intune's MDM device cert stores the DeviceId in OID 1.2.840.113556.5.4 as a
bare 16-byte GUID with no ASN.1 wrapper, so the strict OCTET-STRING parser
bailed on the tag check and returned null. Verified against a live Intune
device cert: 5.4 is 16 raw bytes, while 5.6 / 5.11 / 5.14 carry the usual
04 10 wrapper.

- ParseGuidFromAsn1OctetString -> ParseGuidFromExtension (the old name no
  longer describes what it parses); both encodings now handled, with the
  shared LE-GUID construction pulled into ToGuid(). The forms cannot
  collide: a wrapped 16-byte GUID is always >= 18 bytes.
- Document the Intune MDM cert OID arc on the class: 5.4 DeviceId,
  5.6 Intune AccountId (NOT the Entra tenant), 5.14 Entra TenantId,
  5.11 undocumented GUID, 5.15/.16/.17/.18/.19/.23 INTEGER enums.
- 13 tests pinning both encodings plus the malformed-input rejections,
  including a bare GUID whose first byte is the 0x04 OCTET STRING tag.

No behaviour change for the MS-Organization-Access probe the agent uses;
TryGetTenantIdFromCert is untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 14:41:00 +02:00
Oliver Kieselbach 73373a2c9e feat(agent): compiled-in fallback rule catalogs for the no-config default
The built-in default config shipped empty ImeLogPatterns/GatherRules lists, so a
session that got neither a live fetch nor a cached config ran with the entire IME
tracking pipeline dead — no app installs, no script results, no health-script
compliance, no IME ESP-phase signals — although the agent binary carries all 37
action handlers. The scalar knobs never drift (shared AgentConfigResponse property
initializers are also the backend's fallbacks), but the catalogs cannot come from
initializers.

BuiltInConfigCatalog embeds the SAME CI-combined rules/dist artifacts the backend
embeds and serves (ime-log-patterns.json, gather-rules.json) and feeds the enabled
entries into CreateDefaultConfig. Single source by construction: each agent release
snapshots the catalog state of its own build, so the offline fallback can never
drift further than the running agent version itself. A live fetch (startup,
post-registration recovery, rotate_config) always wins. Fail-soft: a load error
degrades to the previous empty-list behaviour.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 12:14:30 +02:00
Oliver Kieselbach 6d607c9045 fix(agent): network-link gate before initial config fetch + post-registration recovery
Root cause (tenant aebdce78 audit 2026-08-24, session ba828038): on boot-time
relaunches the agent starts before the Wi-Fi link is up. The Phase 4 config fetch
had no network gate (98f8114d only guarded Phase 6 registration), burned its
10/30/60s retry budget into a dead link and stranded the whole session on built-in
defaults (ConfigVersion=0) — up to ~50% of that tenant's sessions, with no
self-heal short of a manual rotate_config action. Not a version regression: rates
varied wildly per day within the same agent build; the failed requests never
reached the backend.

Two complementary layers:
- RemoteConfigService gates the initial fetch on the shared NetworkLinkWait poll
  (60s cap — longer than registration's 15s because the fetch runs first), so the
  retry budget starts on a live link. rotate_config stays ungated (latency-bound).
- Phase 6.5: after registration proves the backend reachable, TryRecoverConfigAsync
  refetches once and AgentRuntimeConfig.TryRecoverAfterRegistration re-applies the
  live config (same merge/integrity pipeline as Phase 4). LastFetch* keeps the
  original failure record; remote_config_fetch_failed reports it with a recovered
  flag at Info severity, agent_started flags remoteConfigRecovered.

Deliberately not re-run in recovery: kill-signal check (telemetry-channel control
signals cover post-registration kills) and endpoint migration (next start).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 11:53:04 +02:00
Oliver Kieselbach 3af94d9dcd infra(telemetry): DCR also drops host-emitted successful SignalR dependencies
Live-verified after 0d5e2572 deploy: the worker filter removes every 404/412/409
and worker SignalR row, but the SignalR output-binding calls are emitted by the
host process (SDK rdddsc) and only reachable at ingestion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 19:15:41 +02:00
Oliver Kieselbach 0d5e2572aa perf(telemetry): one canonical request record; drop expected storage outcomes and SignalR success deps
- RequestTelemetryMiddleware: SamplingPercentage=100 per item (worker copy was
  adaptively sampled, ItemCount 2-5 on weekdays, despite host.json excludedTypes)
- StorageDependencyFilterProcessor: also drop 404/412/409 storage outcomes
  (InProc + HTTP shape) and successful *.service.signalr.net REST calls
- infra/appinsights-workspace-transforms.bicep: DCR dropping the host HTTP
  request duplicate at ingestion (timer/queue rows kept)
- docs/backend/telemetry-ingest-shaping.md; skills updated

Measured baseline 7d: ~1.3 GB/week removed, requests now 100% with tenant context.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 18:31:32 +02:00
Oliver Kieselbach 793e907e60 fix(cveindex): merge findings per CVE before upsert; bounded parallelism
One CVE commonly appears under many findings of the same report (e.g. one
RD CVE across 14 installed RD components). Each occurrence produced its own
concurrent upsert on the same PK/RK, so the stored SoftwareName/OverallRisk
was race-dependent. Collapse to one entity per CVE (highest CVSS wins,
IsKev OR-ed, OverallRisk max) and write with Parallel.ForEachAsync(8) and
per-entity fail-soft logging instead of Task.WhenAll + one outer catch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 17:52:02 +02:00
Oliver Kieselbach 8e8bb66371 test(agent): thread-safe FakeSignalIngressSink (parked-tripwire CI flake)
Timer-thread Post() appended to a bare List while the test polled it in a
SpinWait foreach -> 'Collection was modified' (run 32644425162, no assert
text). Posted now returns a locked snapshot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 16:28:58 +02:00
Oliver Kieselbach 7e1389cf32 fix(ratelimit): keep history and lock in one cache entry; drop leaking per-key lock map
Both sliding-window limiters re-Set the cache entry without its eviction callback.
The replaced entry fired the callback asynchronously, removing the per-key lock while
a new one was already handed out, so two threads could mutate the same List under
different locks; the replacement entry never cleaned its lock, leaking one object per
key for the process lifetime.

The history now lives in a RequestBucket together with its own lock: same list ⇒ same
lock, no callback, no dictionary. Bucket creation is serialized (GetOrCreate is not
atomic) behind a lock-free TryGetValue fast path. The redundant re-Set is gone — the
lookup already refreshes the sliding expiration.

Concurrency tests (16 threads × 500 requests, one key) fail 3/3 on the old code and
pass on the new one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 16:26:44 +02:00
Oliver Kieselbach 3a89319472 feat(telemetry): startup duration metrics for backend and MCP
StartupTelemetryService emits BackendStartupMs (process start ->
ApplicationStarted) and BackendTableInitMs as App Insights custom
metrics with version/commit/tableInitFullPass/instance dimensions;
TableStorageService exposes LastInitialization. MCP server writes one
{"type":"boot","readyMs"} stderr line per start next to the
tool_call lines so cold-start time is queryable as a trend.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 16:04:31 +02:00
Oliver Kieselbach b37e6f975c perf(startup): table schema sentinel gates CreateTableIfNotExists pass
Cold start point-reads a SHA-256 over TableNames.All stored in
AdminConfiguration/SchemaSentinel/tables instead of issuing 66
CreateTableIfNotExists calls. Missing/stale sentinel runs the full
idempotent pass once and rewrites it. SessionsIndex startup backfill
only after a full pass, behind a conditional-insert claim (one instance
under scale-out); IsSessionIndexEmptyAsync no longer treats errors as
empty. Daily maintenance runs IStorageInitializer.EnsureAllAsync to
repair out-of-band table deletions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 15:33:45 +02:00
Oliver Kieselbach cacbd24667 feat(mcp): get_verdict_calibration response shaping (explicit nulls, minSharePct/top, omitted report)
Drop per-row window7/baseline28.sessions (duplicated trend denominators), make
reEnrollRatePct/lift always present with null = withheld, add minSharePct/top
tail trimming that never drops override-carrying rows and reports what was
omitted. Backend wire shape and admin page untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 12:51:17 +02:00
Oliver Kieselbach a157d98a40 fix(radar): verdict share regression only for backend-decided paths
First production pass alerted on agent:whiteglove_pending (two tenants in a
WhiteGlove rollout week) and agent:complete_soft (a Continue-Anyway tenant) —
customer workflow mix, not classifier drift. agent:* and register:* are now
ineligible for share_regression (still in the matrix and the silence group);
episodes opened by the broader gate re-arm on the next pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 12:32:00 +02:00
Oliver Kieselbach 7fa50d09c1 perf(maintenance): one projected 35d Sessions scan shared by all rolling sweeps
The cross-tenant StartedAt-only date-range query is a full-table drain in
Table Storage; time attribution, device journeys, verdict calibration and the
calibration radar's tenant discovery each ran their own — four identical reads
per 2h tick. RunAllAsync / RunManualAsync now load the window once through
GetMaintenanceWindowSessionsAsync (MaintenanceSweepSessionProjection: the union
of the sweeps' columns, drops FailureSnapshotJson/geo/OS/diagnostics) and the
sweeps slice it by StartedAt. MaintenanceSweepProjectionEquivalenceTests pins
that every consumed field survives the projection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 12:19:41 +02:00
Oliver Kieselbach 94afc02e62 docs(okf): verdict calibration concept (VerdictPath, prior verdicts, aggregate, radar)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:40:34 +02:00
Oliver Kieselbach 5e4c5e80be feat: verdict calibration drift radar (share / silence / evidence-gap) with ops events
Applies the rule-regression statistics (7d vs 28d, lift >=2, Wilson-separated,
small-n gates) to the verdict-calibration rows: per-path share regression,
sweep+maxlife silence-share regression (agent liveness signal) and an
absolute r6 evidence-gap gate. Episodes live in the notification tracker
(verdictcalibration|kind|path|status), fire one VerdictCalibrationDrift ops
event (operator-only, no tenant bell), re-arm under 1.5x / 15%, and surface as
alerts[] + a Drift pill in the calibration matrix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:39:06 +02:00
Oliver Kieselbach b535973531 feat: verdict calibration aggregate, GA endpoint, MCP tool and operator matrix
Daily (tenant x StartedAt date) buckets per verdict path: count, derived
attribution, 7-day re-enrollment proxy (eligible/re-enrolled), and the
correction stream attributed to the PRIOR path (admin / late completion /
other). Written by the 2h maintenance sweep after the device-journey pass,
180d retention, regenerable. Read via GET global/metrics/verdict-calibration
(GlobalReadOrAdmin, 7d-vs-28d share trend, rates withheld below n=20),
the get_verdict_calibration MCP tool, and the admin metrics section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:22:17 +02:00
Oliver Kieselbach 7e0c578a59 feat(backend): VerdictPath + PriorStatus/PriorVerdictPath on every session status write
Every status writer now declares the code path that produced the status
(VerdictPaths vocabulary, stamped for all statuses — unlike FailureSource
which is Failed-only), the timeout classifier returns its rule id, and a
write that overrides a prior verdict preserves it as PriorStatus/
PriorVerdictPath (admin mark, late completion, retro). Pre-instrumentation
rows derive a path read-side (VerdictPathDerivation, legacy:* when
ambiguous). Foundation for the verdict calibration aggregate + radar.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:06:37 +02:00
Oliver Kieselbach 1257e0d195 Merge branch 'main' of https://github.com/okieselbach/Autopilot-Monitor 2026-08-23 09:57:39 +02:00
Oliver Kieselbach 98f8114d90 fix(classification): self-deploying profiles reconcile to Succeeded after Device Setup, never AwaitingUser
Kiosk tenant audit 2026-08-23: 836 of ~970 self-deploying sessions in one week
were parked AwaitingUser ("Account Setup 0/5") and graduated to Incomplete 51h
later although the devices were finished — the profile has no user phase, so the
classifier's rule 5 was asking for evidence that can never arrive.

- EnrollmentTimeoutClassifier: new rule 1c IsSelfDeployingProvisioned (profile flag +
  Device Setup all-succeeded + no explicit failure) => Succeeded reconcile with an
  honest reason; isSelfDeployingProfile passed by every caller (sweep stage 1+2,
  max-lifetime verdict, late-telemetry heal, legacy retro mode).
- MaintenanceService stage 1: TryReconcileSelfDeployingAsync gate before the Stalled
  marker (mirrors the WhiteGlove Part-2 gate), audit counter SessionsSelfDeployingReconciled.
- LegacyReclassificationService: new admin mode self_deploying_silent (heal-only,
  dry-run by default) + GetSelfDeployingSilentSessionsAsync for the existing backlog.
- Agent SessionRegistrationHelper: wait up to 15s for a network link and retry 6x
  (~62s) instead of 5x (~30s) so the BootTrigger relaunch on Wi-Fi no longer exits 7
  before the link is back (the mechanism that left these sessions silent).
- Tests: 6 classifier cases (gate, within/past grace, user-driven guard, failure,
  pre-Device-Setup), 2 registration tests; docs/backend/silent-session-classification.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 09:55:06 +02:00
Oliver Kieselbach 7b3230d507 fix(web): mobile wrapping for App Details header, Software tables and SLA banner (#133) 2026-08-23 05:19:20 +02:00
Oliver Kieselbach 87b7a20223 fix(web): diagnostics path rows wrap the full path on narrow screens
Below `sm` the path owns the first line(s) and wraps in full (flex-wrap,
basis-full, break-all) with badges and controls flowing onto the next line —
a path truncated to "C:\…" is useless on a phone and a tooltip cannot be
hovered there. From `sm` up the row stays a single truncated line. The
"N file types" pill is shown on every width again now that the badge line
has room.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 04:30:10 +02:00
Oliver Kieselbach 4c19be1de4 feat(web): drop Experimental/Pre-Release badges from Software Inventory & Vulnerability Analyzer
Settings section loses both pills; the vulnerability report footer keeps the
best-effort CVE matching note without the Experimental prefix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 04:26:55 +02:00
Oliver Kieselbach 93de130cd1 feat(diagnostics): built-in collection catalog visible in the portal, RealmJoin + Device Preparation sections gated
The diagnostics package's built-in collection is now ONE catalog in Shared
(DiagnosticsBuiltInSections.All): the agent iterates it when building the ZIP and
the backend serves the same list to the portal (GET /api/diagnostics/paths,
MemberRead), so what administrators see is exactly what the agent collects.

Agent
- New conditional sections: RealmJoin client/package/Chocolatey/per-user logs
  behind the tenant's RealmJoin Watcher toggle (mirrored onto
  AgentConfiguration.EnableRealmJoinWatcher by RemoteConfigMerger) and the Device
  Preparation bootstrapper event log behind the deterministic WDP marker (probe
  once per build, fail toward Classic). ZIP layout mirrors the disk
  (RealmJoinLogs/Windows|Choco|User). Every skip is a package-manifest line.
- Built-ins bypass DiagnosticsPathGuards by design (reviewed code); caps,
  reparse-point skip and evtx export still apply. No ConfigVersion bump.

Backend
- GET diagnostics/paths (MemberRead, JWT-scoped) with the catalog + global paths.
- MergeDiagnosticsLogPaths: global ∪ tenant dedup (trimmed, case-insensitive) —
  duplicates used to produce duplicate ZIP entry names.

Web
- Collapsed read-only built-in list (RealmJoin on/off in the tenant header,
  neutral in the Global-Admin view), one-line DiagnosticsPathRow for every entry,
  add-row above the lists; tenant context no longer fetches /api/global/config.

Tests: agent RealmJoin on/off, no user session, bootstrapper evtx, probe failure,
merger mirror; backend catalog invariants, payload shape, dedup, policy; web
builtInSectionDisplay. Docs: docs/agent/diagnostics-package.md (+index/log).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 04:09:50 +02:00
Oliver Kieselbach b8b95c7e76 refactor(web): move RealmJoin Watcher and Keep Awake from Agent Analyzers to Agent Settings
Both toggles gate collector hosts / device behavior, not IAgentAnalyzers, and
tenants looked for them under Agent Settings. RealmJoin Watcher now sits in the
Agent Collectors card, Keep Awake During User-ESP in Agent Parameters; OOBE
console detection stays with the security analyzers. Section ownership in
SECTION_FIELD_MAP and the per-section reset handlers follow, the Agent Settings
Save/Reset bar moves below both cards it saves, and the Analyzers sidebar
description names all four checks. Wire contract (AnalyzerConfiguration) and
agent are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 04:06:39 +02:00
Oliver Kieselbach e57e081cec docs(web): Discord in notification-provider copy on about, privacy, settings and rule card
Privacy page last-updated bumped to 23 August 2026.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 02:47:52 +02:00
Oliver Kieselbach 3a60186824 revert(email): drop the gradient slant row and corner cell — solid fallbacks made non-gradient clients worse
Back to the 838da4cc header: one facet gradient, three-row brand mark, 113px
band, clip-path edge for clients that support it. Clients without gradient
support get a flat band instead of band-coloured blocks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 02:30:40 +02:00
Oliver Kieselbach c35595503f feat(email): header slant and light corner facet without clip-path — gradients Outlook can draw
Row 1 keeps the single facet gradient and adds a top-right cell with its own
dark/light split (the corner facet); row 2 draws the slanted bottom edge as a
green/white gradient in three band-coloured cells. Pixel-measured against the
reference render: band boundaries identical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 02:22:44 +02:00
Oliver Kieselbach 838da4cc8a fix(email): single Outlook-safe facet gradient, no bgcolor inside the brand mark
Outlook web/iOS render exactly one linear-gradient with hex colours and % stops
(multiple layers / rgba stops are dropped), and dark mode inverts cell
backgrounds but not border colours — the white cell in the mark became a black
bar. Facets are now one gradient (top-right light corner dropped), the mark's
middle row is a white-bordered div.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 02:11:17 +02:00
Oliver Kieselbach a9937e0f5d fix(email): brand mark as three non-overlapping rows — Outlook ignores negative margins
Triangle / white cell + triangle / triangle, no overlays, no transparent td
borders; matches the reference mark in a side-by-side zoom render.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 02:02:14 +02:00
Oliver Kieselbach 1dc2cc9a54 fix(email): header rebuilt 1:1 from the approved banner — faceted gradients and four-edge brand mark
Verified side-by-side against the reference render (scratch header-render.html):
three layered half-plane gradients reproduce the light/dark facets, the mark is
a CSS trapezoid plus two green wedges for the slanted right edge and the
descending bottom edge. Still no images in the mail.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 01:44:26 +02:00
Oliver Kieselbach dd7ae299a8 fix(email): header restored to the approved rendering (e1de5a89 geometry)
112deg stripe gradient, 13/20/2 border-drawn mark, clip-path edge — the version
the user signed off on; later 'improvements' drifted from it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 01:23:22 +02:00
Oliver Kieselbach d96d10ccec fix(email): header matches the reference banner — mark 12/21/0 trapezoid, stripes thin/wide/thin in the proven 8-stop gradient form
Verified against the original SVG mark in a Chromium render; the gradient keeps
the exact stop count that rendered in Outlook earlier.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 01:05:12 +02:00
Oliver Kieselbach 457fff65f5 fix(email): brand mark back to the div border trick (Outlook paints transparent td borders white)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:56:09 +02:00
Oliver Kieselbach dc45694611 docs(okf): email template overrides and image-free mail header
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:50:41 +02:00
Oliver Kieselbach 7569061983 feat(admin): editable email templates — preview, test send and HTML override in Settings
- EmailTemplateService: override-or-built-in resolution per send (PreviewConfig
  table, partition EmailTemplates, 5-min cache, {{domainName}} placeholder,
  30k limit); send path fail-soft, admin read path strict
- GA endpoints global/email-templates/{kind} (GET/PUT/DELETE) and /test, which
  mails the effective template or an unsaved draft to the caller's tenant
  contact address
- portal: Settings → Email Templates with preview frame, test send, HTML editor
- mail header: brand mark drawn with CSS borders, no images left in the mails

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:50:10 +02:00
Oliver Kieselbach 0f5e36db94 fix(email): exact brand mark as PNG again, stripes direction and widths match the reference
The CSS border trapezoid did not reproduce the mark's shape across clients; the
mark is brand identity, so it is the one image in the mail (empty alt). Stripes
now run top-left to bottom-right, thin/wide/thin as in the reference banner.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:40:15 +02:00
Oliver Kieselbach e1de5a8971 feat(email): image-free header — CSS stripes, slanted edge and border-drawn brand mark
Progressive enhancement: solid bgcolor band + text wordmark everywhere; the
brand mark is a CSS border trapezoid; diagonal stripes (linear-gradient) and the
slanted bottom edge (clip-path) render where supported and degrade to a flat
band elsewhere. No remote images left in either mail.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:22:26 +02:00
Oliver Kieselbach a2d93e884c fix(email): native HTML header that survives blocked images
Solid bgcolor band with the wordmark as real text; only the 36px brand mark is
an image (empty alt). Replaces the full-width header PNG, which collapsed to a
bare green bar on clients that block remote images (iOS Mail).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:15:45 +02:00
Oliver Kieselbach 425229deee feat(email): branded header image, sign-in link, drop preview banner in welcome/farewell mails
- header is a hosted PNG (www/email/header.png, rendered from the brand mark +
  wordmark) on a solid bgcolor band; mail clients drop CSS gradients, which left
  the old header white
- welcome mail links "sign in" to the portal, active-development banner removed
- em dashes replaced with en dashes

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 23:55:53 +02:00
Oliver Kieselbach 8675e7ee06 feat(backend): provider-neutral EmailService on Mailchimp Transactional (Mandrill), replaces Resend
- EmailService + IEmailService: Mandrill messages/send via typed HttpClient with the
  Notification resilience policy; provider confined to one transport method and the
  Email:* settings (ApiKey, Endpoint, FromAddress, FromName). Resend NuGet dropped.
- open/click tracking explicitly off; sent/queued/scheduled = ok, rejected/invalid = warn
- callers (TenantApprovalService, PreviewWhitelistFunction) depend on IEmailService
- EmailServiceTests pin wire format, result semantics and fail-soft contract
- privacy page + OKF welcome-mail doc updated for the new sub-processor

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 23:26:32 +02:00
Oliver Kieselbach e857e99436 fix(web): list Generic JSON provider last in channel provider dropdown
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 21:54:34 +02:00
Oliver Kieselbach 6435d5f290 feat(notifications): Discord webhook provider + HMAC signing for generic webhooks
Discord (provider 30) renders alerts as a single embed with Discord's
field/length caps; actions become markdown links since webhooks cannot
post buttons. Generic JSON channels gain an optional per-channel signing
secret: requests carry X-AutopilotMonitor-Timestamp and
X-AutopilotMonitor-Signature (sha256 hex HMAC over "{timestamp}.{body}"),
computed over the exact posted JSON with same-named custom headers
stripped first. Secret is redacted for readers and restored by channel
id on save; validation enforces 16-128 printable ASCII characters.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 21:09:12 +02:00
Oliver Kieselbach 445ce4e75b feat(backend): 30-day retention grace period on Pro downgrade
Retention was the only enforcement point that destroyed state on a
Pro -> Community downgrade: the 12-hourly sweep clamps to the Community
cap at read time and would immediately start hard-deleting the
90-365-day band (trial expiry identical, silently).

New policy: for 30 days after losing Pro
(FeatureEntitlementCatalog.RetentionDowngradeGraceDays) the Pro cap
keeps applying. Anchor model: ProDowngradedUtc (backend-only field,
patch-denied + revert-protected) stamped/cleared by the plan endpoint's
new pure ApplyPlanChanges core on EFFECTIVE-edition transitions (covers
explicitly ended trials; a planTier downgrade under an active trial
deliberately does not stamp); trial expiry anchors read-time via
TrialExpiresUtc. days<=0 (infinite) stays untouched.

Ops visibility: TenantPlanDowngraded (PATCH handler, new OpsEventService
dep) plus TenantRetentionGraceExpiring/Ended from the daily
TrialExpirySweep with the same stateless window mechanics, emitted only
when stored retention exceeds the Community cap; all dual-registered in
the web OPS_EVENT_TYPES catalog. Fanout now uses its _utcNow seam
instead of inline DateTime.UtcNow. OKF doc
backend/plan-downgrade-retention-grace.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 21:00:53 +02:00
Oliver Kieselbach 4e54ffe4b4 feat(web): shared plan cards, public /plans and /buy pages
Community/Pro comparison cards extracted into a shared module
(components/plans: planData.ts as single copy source, PlanCards.tsx with
marketing/portal surfaces) so the portal Plan section and the new public
/plans page can never drift. Pro price is one constant (null renders
"Pricing - announced soon").

New public pages following the /help pattern: /plans (free-first
messaging, nav item "Plans") and /buy (direct purchase + Microsoft
Marketplace as coming-soon channels). Landing nav, footer, FinalCta and
get-started now link to /plans instead of the docs plans page; portal
Plan section links to /buy via absolute www URL. Purple utilities
neutralized inside .dark .landing-v2; sitemap/robots/lastmod cover
/plans, /buy and the previously missing /help.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 19:46:44 +02:00
Oliver Kieselbach 65d45a616e feat(web): Community Edition tag under navbar wordmark
Community tenants see a small "Community Edition" line under the brand
wordmark as a standing reminder (and nudge toward upgrade options); Pro
tenants (incl. trial) keep clean chrome. Label renders only after the
feature-flags fetch confirms the edition - no flash for Pro while loading.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 19:38:19 +02:00
Oliver Kieselbach 4aa1f07fbb feat(web): green welcome accent + session-dismissable dev banner with telemetry
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 19:17:53 +02:00
Oliver Kieselbach ede616444a fix(web): honest skip/uninstall/pending semantics in Download/Install Progress
IME reports skips and uninstall enforcements through the same completed
events, so both panels counted them as successful installs/downloads and
their pills contradicted the app_tracking_summary headers (session 502274b4:
"2 of 8 installed" vs "6 completed").

- Install fold: app_install_completed with state Skipped -> Skipped;
  successful uninstall intent -> new terminal state Uninstalled with its own
  pill and neutral row badge; completed now counts only real installs.
- Download fold: rows are classified by observed byte activity, never by
  intent (Win32 uninstalls re-download the full package and stay visible
  with an Uninstall badge); 0-byte rows sit behind a "no download" toggle.
- Both panels render tracker-known pending apps (summary pendingNames)
  behind a "pending" toggle, so every "X of Y" header fully adds up.
- Headers wrap on narrow screens (flex-wrap + gap instead of space-x).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 18:42:23 +02:00
Oliver Kieselbach 61a359ee54 feat(web): public /help page for support channels
Adds www.autopilotmonitor.com/help as a mail-linkable page describing how
to get help: Report Session in the portal for session-specific problems,
GitHub Issues for everything else, LinkedIn for private contact, plus
self-service links (docs, FAQ, announcements, changelog) and the private
security-advisory path. Registered as public path in hostRouting (guard
test pinned) and linked from the site footer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 18:12:09 +02:00
Oliver Kieselbach babc89c776 fix(backend): park silent WhiteGlove Part-2 sessions as AwaitingUser instead of Stalled
fairstone.ca analysis 2026-08-21: after the reseal-reboot technicians routinely
power devices off at the logon screen to box them. Both silence paths (2h
maintenance sweep, agent stall probe) then labeled the expected parking state
"Stalled - Agent silent", and stage 2 later reconciled it to Succeeded on the
Part-1 whiteglove_complete with a reason claiming Account Setup completed.

- EnrollmentTimeoutClassifier: shared pure gate IsWhiteGloveAwaitingUser
  (pre-provisioned + ResumedAt + Device Setup provisioned + zero user evidence
  since resume) with one canonical customer-facing reason; new rollup field
  HasEnrollmentComplete separates real completions from Part-1 whiteglove_complete
- ClassifyTimedOutSession: gated sessions are AwaitingUser within grace and
  reconcile past grace with an honest WhiteGlove reason; on resumed sessions
  whiteglove_complete no longer counts as terminal-complete evidence (rule 2),
  so partial user progress falls through to the honest AwaitingUser/Incomplete rules
- MaintenanceService stage 1 + session_stalled ingest mapping apply the same
  gate (fail-soft to Stalled); sweep audit now carries SessionsAwaitingUser
- Callsites (stage 2, max-lifetime verdict, late-telemetry reconcile, legacy
  reclassification) pass IsPreProvisioned/ResumedAt through

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 22:30:29 +02:00
Oliver Kieselbach a2f448cad0 fix(web): day-scale duration formatting in network timeline
fmtDuration rolls hours into days above 24h ("768d 8h" instead of
"18440h 58m") - minutes are noise at day scale. Surfaced by the first
fleet session with a CMOS clock years off (0bd4af1c, 2024 -> 2026 step);
also keeps the slanted clock-marker label inside its headroom. Test
pinned on the real fleet delta.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 21:05:04 +02:00
Oliver Kieselbach f15e115dea fix(sessions): backfilled system-timeline events no longer anchor StartedAt/duration
Field case d0c5b672: a pre-agent system_clock_changed event (backfilled from
the Windows event log with its original timestamp, ~14min before the first
real activity) became the partition's earliest row and re-anchored the
session's StartedAt, inflating the duration toward pre-enrollment OOBE idle
time. With the watcher's 24h backfill lookback, the 2h backward-shift guard
only caps this effect, it does not prevent it.

SystemTimelineWatcher rows (clock steps, sleep episodes) are environment
observation, not enrollment activity, and are now excluded from all three
anchor derivations: the Events-table earliest-row probe (new
IsSessionAnchorEligible, probe scans past ineligible rows), the ingest
batch's EarliestEventTimestamp fold, and the web's client-side
enrollment-duration derivation. The events themselves stay on the timeline
unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 20:18:57 +02:00
Oliver Kieselbach 977c5fb447 fix(web): headroom + symmetric insets for network-timeline marker labels
A left-edge "Clock -3h 0m" label collided with the HTML legend above the
band (first live fleet session d0c5b672). Marker-label headroom grows to
48 units so long labels stay inside the SVG, and the drawing content is
now inset 64 units on BOTH sides - the left inset mirrors the existing
right one, giving edge labels room and a calmer, symmetric frame.

Verified against the live session carrying the first system_clock_changed
event (backfilled step -> marker only, no rebase, as designed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 20:14:29 +02:00
Oliver Kieselbach 27fdd19568 feat(agent): optional Delivery Optimization GroupId from network fingerprint
EnableDoGroupIdAutoSet (ConfigVersion 40, default off): the agent seeds the
DOGroupId policy value with a deterministic GUID derived from the default
gateway's IPv4 + SendARP MAC, RealmJoin-compatible byte layout so mixed
fleets peer together. Foreign DOGroupId/DOGroupIdSource policies always win
(ownership tracked via StartupEventGate fingerprint); no random-GUID
fallback. New do_group_id_auto_set event, portal toggle, admin report row,
manifest/catalog sync, OKF doc. Gateway heuristic deduplicated into
NetworkInterfaceLocator.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 20:09:45 +02:00
Oliver Kieselbach 72f7adcb3d feat(web): clock-era axis + sleep segments in network timeline
Consume the two new agent events:

- system_clock_changed: live-observed steps (timeDeltaMs authoritative,
  >=30s; sub-30s NTP nudges ignored) split the stream into clock eras and
  rebase every later timestamp by the cumulative delta, so the axis shows
  true elapsed time. Backfilled steps (pre-agent, replayed from the event
  log) are marked but never rebased. Each step gets a slanted "Clock +1h"
  lifecycle marker with old->new time and reason in the tooltip.
- system_sleep_episode: episodes (emitted retroactively at wake) carve an
  "Asleep" segment out of the active network instead of attributing the
  quiet period to it; tooltip carries kind (Modern Standby/sleep), wake
  source and AC/battery. Episode boundaries get the same era correction
  as the event that carried them.

Stats row gains "Asleep total" and "Clock changes"; the event list
includes both new types (now titled "Network & system events").

Layout: drawing content stops 64 units short of the right viewBox edge
(CONTENT_W), so right-edge lifecycle labels slant up-right like all
others instead of flipping left.

7 new vitest cases pin era rebasing (live/backfilled/nudge), sleep
overlay incl. era-corrected boundaries, and hotspot confidence.

Live verification against fleet data pending - the emitting agent was
released minutes ago; no session carries the events yet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 19:57:19 +02:00
Oliver Kieselbach 06cf1c4c12 feat(agent): system timeline watcher — clock steps + sleep episodes as ground truth
Two enrollment blind spots become timeline ground truth from the System event
log via one EventLogWatcher (WindowsUpdateTracker pattern, 24h backfill with
RecordId watermark recovering pre-agent records):

- system_clock_changed (Kernel-General 1): oldTime/newTime/signed timeDeltaMs,
  reason and the setting process. 2s floor suppresses w32time 1ms micro-slews;
  steps >= 5min are Warning + immediate upload. Payload instants are the
  authoritative source (event timestamps can be clamped server-side).
- system_sleep_episode (Power-Troubleshooter 1 classic sleep/hibernate,
  Kernel-Power 507 Modern Standby; 506 deliberately unconsumed): kind,
  enteredAt/exitedAt, durationSeconds, wake source / AC state. 60s real-sleep
  floor filters Modern Standby screen-off scenarios.

ConfigVersion 39 (Collectors.EnableSystemTimelineWatcher +
SystemEventBackfillLookbackMinutes). Both types are non-activity on agent and
backend (idle/stall clocks unaffected; a sleeping device stays Stalled).

Consumers: TimeJumpBadge gains the ground-truth cause "clock-set"
(classifyTimeJump matches backward display steps against recorded clock sets);
EventTimeline badge blocks for both events plus the previously bare
power_state_change; SessionInfoCard keeps the wall-clock duration and adds a
standby note; TimeAttributionCalculator builds SleepSpans (RebootSpan clone,
AttributionVersion 3) persisted as SleepSeconds/SleepSpansJson and rendered in
TimeAttributionLane; ANALYZE-DEV-012 (info, single) fires once per session for
episodes >= 5min - deliberately the only new rule, clock findings stay with
DEV-008 to avoid duplicate-finding flood.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 19:41:56 +02:00
Oliver Kieselbach e612c69178 feat(web): feature-form default for the grant command + docs link
The generated PowerShell command now uses the readable -Features form
(first catalog feature as the example) instead of raw permission strings;
a note below lists all available values incl. the All meta-feature and
how to fall back to -Permissions via Copy permissions. The section links
to the Optional Graph Permissions customer docs (via DOCS_URL registry),
and the "What this script does" bullet reflects the Cloud Shell
ambient-identity sign-in. Script help no longer claims the admin UI
generates the -Permissions form.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 19:27:51 +02:00
Oliver Kieselbach 829a841d9b feat(web): slanted inline labels for network-timeline lifecycle markers
The gray lifecycle dots (Reboot / Desktop / Completed) were hard to hit
and mute without hovering. Their label is now written directly at the dot,
slanted up-right (flipping up-left near the right edge so it stays inside
the drawing), with the band shifted down to make headroom. The whole
marker group (line, dot, label) shares an enlarged invisible hit target,
and a label is suppressed when its neighbor sits too close - the tooltip
still carries the details.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 19:09:03 +02:00
Oliver Kieselbach 10eedded13 feat(web): confidence tier for network-timeline hotspot badge
An SSID matching the typical iPhone hotspot name (".. iPhone") is strong
evidence on its own — such networks now get a plain "Hotspot" badge, while
subnet-only evidence keeps the hedged "Hotspot?". Tooltip and footnote
explain the distinction; detection also fires on iPhone-named SSIDs
outside the Apple hotspot subnet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 19:00:56 +02:00
Oliver Kieselbach a9f6ccd86d feat(web): power + network status chips on the progress portal
Taskbar-style chips in the progress page status header showing the
device's last-reported power state (AC/battery with percent, low-battery
warning) and network state (WiFi with SSID tooltip + signal tiers,
Ethernet, offline). Derived purely client-side by folding the event
stream the portal already downloads (power_state_check/change,
network_interface_info, network_state_change, wifi_signal_info) - no
backend, API, or auth change. Chips render only while InProgress and
disappear gracefully for sessions from agents without these events;
devices without a battery show no power chip (Windows-tray parity).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 18:56:04 +02:00
Oliver Kieselbach 7ba67df1a1 Agent instructions 2026-08-21 18:39:35 +02:00
Oliver Kieselbach e214e8dc1a feat(scripts): -Features All meta-value grants every optional permission at once
All is a script-side convenience that expands to every GraphFeatureCatalog
feature (works with -Revoke too). The catalog sync test now pins All in the
ValidateSet while keeping it out of the catalog comparison, and the admin UI
mentions the shorthand next to the generated grant command.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 18:32:48 +02:00
Oliver Kieselbach f8d0ccd2f5 feat(web): per-session network timeline page + device-details deep link
New /sessions/network-timeline?id=&tenantId= page visualizing every network
the device was on during enrollment: one colored segment per network
(Ethernet / WiFi SSIDs, fixed-order validated palette), hatched offline
gaps, ESP phase context lane, lifecycle + connectivity-check markers with
hover/tap tooltips, stats row and the raw network event list.

Segment model derives from network_interface_info + network_state_change
(before/after), enriches SSIDs from wifi_signal_info, infers missing
post-reconnect SSIDs via matching gateway, and flags probable smartphone
hotspots from vendor-default subnets (Apple 172.20.10.x, Windows
192.168.137.x, legacy Android 192.168.43.x) - always as a heuristic badge.
Axis is honest: linear per segment, minimum width for short segments,
boundary ticks show real times. Mobile scrolls horizontally.

Entry point: "Network timeline" deep link in DeviceDetailsCard's Network
section via new networkTimelineUrl() builder. Route registered in the SWA
config (exact routes before the /sessions/* wildcard, incl. the index.txt
RSC route) and excluded from LegacyPathRedirect's /sessions/{id} rewrite
via the new static-siblings set (test pinned).

Planned follow-up once the agent emits them: clock-change events (rebase
axis into clock eras) and standby/Modern-Standby events (render sleep as
its own segment kind).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 18:26:49 +02:00
Oliver Kieselbach c9701301ea feat(scripts): Cloud Shell ambient-identity sign-in for Grant-AutopilotMonitorAddOn
Try Connect-MgGraph -Identity first (Azure Cloud Shell issues a delegated
token for the signed-in admin, avoiding the device-code flow that
Conditional Access commonly blocks), probe Graph readability, and fall
back to interactive sign-in with explicit scopes otherwise. The
AppRoleAssignment.ReadWrite.All scope guard now applies only to the
interactive path, since ambient-identity tokens carry a fixed scope set.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 18:24:31 +02:00
Oliver Kieselbach ba2a390a83 fix(agent): battery-instance fallback for the dead Win32_PowerManagementEvent provider
On Windows 11 24H2/25H2 (observed on builds 26200 and 26220, session
161b838c) the MS_Power_Management_Event_Provider is still registered in the
WMI repository but activating the extrinsic push query fails with
WBEM_E_NOT_FOUND - the legacy provider no longer loads on exactly the OS
builds new devices enroll with, so the power watcher degraded fleet-wide.

Arm chain: try the push event first (still alive on older builds), then
fall back to the intrinsic __InstanceModificationEvent WITHIN 30 on
Win32_Battery. The WITHIN sampling runs inside WinMgmt, not in the agent;
the handler path is unchanged (payload ignored, re-probe + diff), worst-case
detection latency ~35s incl. debounce. collector_degraded now fires only
when BOTH attempts fail; the arm log line names the active mode.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 15:49:03 +02:00
Oliver Kieselbach dbd7d2ac6f Page-Size-Option „5" 2026-08-21 15:26:03 +02:00
Oliver Kieselbach 8f24876682 feat(agent): live power-state watcher + battery analyze rules DEV-009/010
Event-driven AC/battery observation during enrollment (no polling): a WMI
Win32_PowerManagementEvent (EventCode 10) push subscription re-probes
GetSystemPowerStatus and diffs snapshots through a pure tracker into
power_state_change events - transition ac_to_battery (Warning, immediate) /
battery_to_ac (Info) / threshold_crossed at 50/30/15% (per-level lifetime
latch, lowest-crossed-only on jumps, arm-time baseline covers sessions
starting low, 5s trailing-edge debounce, emission cap 20). Never armed on
devices without a battery; arm failure surfaces as collector_degraded.
Kill-switch Collectors.EnablePowerStateWatcher (ConfigVersion 38).

power_state_change is classified non-activity on both sides
(SignalActivityClassifier + IsPeriodicOrStallEvent) so battery drain can
neither keep idle collectors alive nor heal a Stalled session.

New analyze rules, both with on_event:power_state_change interim triggers
and deliberately single-required-condition (event_data conditions scan
events independently - no same-instance join):
- ANALYZE-DEV-009 (high): thresholdPercent equals 15
- ANALYZE-DEV-010 (warning): transition equals ac_to_battery

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 15:24:41 +02:00
Oliver Kieselbach 2d5ee887d7 fix(web): make rule JSON paste-import lossless in both rule editors
The JSON editors serialize the FORM object, but what users paste is usually a
rule EXPORT (e.g. received from another tenant). The create/edit whitelists
rebuilt collector parameters and evaluateOn from form fields the export does
not carry, so a pasted gather rule silently lost every collector parameter and
a pasted analyze rule silently reverted to the enrollment-end default.

Both rules pages get a single shape-tolerant mapper in types.ts:

- gatherRuleToForm unflattens the nested parameters object (authoritative when
  present, so edit-merge leftovers cannot leak), maps nullable rule fields to
  form defaults, imports a missing emitMode as 'always' (the rule's actual
  behavior) instead of the empty-form on_change default, and now also backs
  startEditing so edit and paste share one mapping.
- analyze jsonToForm maps evaluateOn onto the eval* checkboxes (an explicit
  evaluateOn wins over stale form flags in edit-mode merges); terminal-only
  exports fall back to the enrollment-end default.
- tags and the folded effective markSessionAsFailed now survive the import:
  both forms carry tags into the create/save payloads, analyze creates carry
  markSessionAsFailedDefault. notify/notifyChannelIds stay excluded — channel
  ids are tenant-specific.

9 new vitest cases pin both shapes, the merge precedence, and the round-trips;
full web suite 824/824, tsc and eslint clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 14:29:08 +02:00
Oliver Kieselbach 81c14bb3ed fix(rules): close the rule-ID namespace gaps around the shared global keyspace
Custom rules are tenant-partitioned and functionally collision-free, but three
surfaces still keyed on the bare non-unique ruleId string:

- RuleStats global aggregate (global_{ruleId}) summed same-ID custom rules of
  unrelated tenants (title/severity last-writer-wins). The global row is now
  catalog-only: analyze gates on IsBuiltIn/IsCommunity, gather gates on the
  reserved-namespace pattern (agent events carry no flag; custom rules can
  never occupy the reserved pattern).
- The 'never ship a built-in in the CUSTOM namespace' commitment from
  RuleIdPolicy was documented but unenforced: combine.js now fails on
  duplicate or non-reserved built-in IDs, BuiltInRuleCatalogPolicyTests pins
  the same invariants on the embedded catalog, and the GitHub reseed rejects
  non-reserved IDs before writing to the global partition.
- DeleteRuleStatesForRuleIdAcrossTenantsAsync (cross-partition delete by bare
  RowKey) now refuses IDs outside the reserved namespace with failed=-1 so
  callers abort the follow-up global catalog delete.

Verified: prod global partitions (45 analyze + 11 gather) are fully inside the
reserved namespace, so the guards touch no existing data. New OKF doc
docs/rules/rule-id-namespace.md captures the identity/collision model.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 14:13:46 +02:00
Oliver Kieselbach 3a0bd74b12 fix(backend): sequence the synthetic vulnerability_report event instead of defaulting to 0
The async queue correlation stored the event with Sequence 0, sorting it to
the top of the timeline (and tripping the time-jump badge) even though it
arrives last. Follow the BuildSessionTimeoutEvent/BuildAgentEmergencyBreakEvent
pattern: max+1 so it sorts last; WhiteGlove Part 1 anchors at the Part-1
terminal markers so a late device_setup report can never cross the web
timeline's sequence-based Part-1/Part-2 split. Also upsert the EventType
index the queue path never wrote, making the event visible to cross-session
search.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 13:57:16 +02:00
Oliver Kieselbach 50e7590060 feat: app install durations measure the status-defining attempt
The IME processes the ESP app list in multiple passes (device-ESP evaluation
ending Skipped, real install up to an hour later), and AppInstallSummary
folded first observation -> last terminal across them: a 23-second app late
in the queue reported 4002 s (session 44862cd3, "Set TimeZone").

Backend ingest: new max-folded LastAttemptStartedAt anchors DurationSeconds
(CompletedAt - last attempt start; legacy rows without the anchor keep the
span fallback), replay-guarded additive InstallPassCount, attempt-scoped
DownloadDurationSeconds pairing, and a weaker-terminal guard (completed with
state Skipped/Postponed never overrides Installed/Error, in-batch and in the
cross-batch reconcile). StartedAt stays the earliest-wins window/version-
ordering key. AppsDashboardProjection carries the new columns.

AppVersionRegressionRadar: cutover gate — only attempt-anchored rows enter
the duration population on BOTH comparison sides, so mixed-semantics medians
can neither mask real regressions nor falsely re-arm active episodes.

Time attribution: per-pass segment pairing replaces the first-start ->
last-terminal hull. An app's seconds = sum of its ACTIVE segments (failed
attempts and evaluation passes count; inter-pass idle gaps don't), occupancy
merges the segments, hull endpoints remain for display and what-if bounds.

Agent: a terminal->active transition re-arms BOTH timing endpoints, so event
payload durationSeconds also describes the attempt.

Web/PDF/MCP: labels and tooltips say "install time (final attempt)", the
apps sessions table gets a xN multi-pass chip, the time-series chart carries
a cutover footnote, MCP tool descriptions document the semantics split
between get_app_install_metrics and get_time_attribution.

No backfill: pre-cutover rows keep span values (documented in the public
changelog). OKF doc: docs/backend/app-install-duration-attempt.md.

Suites: Functions 4383, Agent 2407, MCP 564, Web 814 - all green, build 0/0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 13:40:47 +02:00
Oliver Kieselbach 61e53a2e9b feat: lead duration cards with median + P90 instead of outlier-skewed average
Enrollment durations are heavily right-skewed; a handful of overnight/WhiteGlove
outliers made the Fleet Health and Dashboard average cards read like hours
(541 min) and answer no real question. Both cards now lead with the median
(robust 'typical enrollment' answer) and keep the tail visible via a P90
subtext. AvgDurationMinutes stays in both DTOs for API compatibility.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 12:49:42 +02:00
Oliver Kieselbach 7e7b6f8086 fix(backend): declare LastIngestAt in the SessionsIndex manifest
bbc7a1d8 stamped LastIngestAt into IncrementSessionEventCountAsync's merge
entity intending it as Sessions-only ("deliberately not mirrored"), but that
same entity flows through SyncSessionIndexAsync into the index merge — the
field was already landing in SessionsIndex on every ingest batch, just
undeclared. The fail-soft drift guard has been warning ~1,200x/h in prod
since the 08-20 deploy, and a StartedAt-shift full rebuild would silently
drop the column, making index-served session reads lose lastIngestAt that
Sessions-served reads carry.

Declare it properly instead of suppressing: ConditionallyProjected in
SessionIndexFieldManifest, projected in BuildSessionIndexEntity (read side
MapToSessionSummary already handled it), sample value pinned in the
bidirectional manifest tests.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 01:54:34 +02:00
Oliver Kieselbach 2e20476559 feat(agent): extend RealmJoin timeout while first deployment is demonstrably active
The 60-min RealmJoinTimeout deadline arms at RJ detection (the RJ agent MSI
install during DeviceSetup) and used to cut off monitoring mid-install: report
55e6afd61c9d (session 46a96078) lost the tail of a large package catalog whose
last observed package completed 3 seconds before the deadline, with the first
deployment only starting 16 minutes into the window.

When the deadline now fires with LastDeploymentPhase 100/101 and deployment
activity (phase change or package observation, persisted restart-safe in
RealmJoinFacts.LastActivityUtc; the detection-time phase capture deliberately
does not count) inside the last 60 min, the handler re-arms to
lastActivity + 60 min instead of timing out — Info event
realmjoin_timeout_extended — clamped at an absolute cap of detection + 4 h.
realmjoin_timeout carries a reason payload (hard_timeout / inactivity /
absolute_cap); a fire of a deadline incarnation superseded by a re-arm
dead-ends via the OccurredAtUtc == DueAtUtc scheduler contract. The idle case
(no activity ever) still times out at 60 min, and phase-200/210 activity never
extends. Packages are only registry-visible at completion, so the inactivity
window keeps the full original 60 min.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 01:11:48 +02:00
Oliver Kieselbach 990641b19e feat: P14 - per-upload-batch send time separates spool delay from clock offset
Agent stamps X-Send-Time-Utc (device clock, ISO-8601 round-trip) per upload
ATTEMPT; ingest parses it (year>=2000 guard, deliberately no upper clamp) and
server-stamps SentAt onto every event of the request. SentAt - OccurredUtc is
pure spool delay, ReceivedAt - SentAt is network + device-vs-server clock
offset - previously indistinguishable and only separable via median statistics.

clock_skew condition: batches carrying SentAt measure the frame directly
(spool-immune by construction - no spread cap, IME/clamped exclusions
irrelevant, IME-only sessions become measurable; evidence sentAtBatchCount);
legacy batches keep the per-event median path unchanged. CmTraceSkewTripwire
deliberately untouched: its differential median math cancels clock offset as
common mode.

Full persistence chain: Store x2 + Map, fields= projection (sentAt), MCP
get_session_events field list + rule-authoring-guide. New OKF doc
backend/send-time-frame-separation.md.

Suites: Functions 4370 (+16), Agent 2407 (+1), MCP vitest 564 + tsc 0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 00:44:29 +02:00
Oliver Kieselbach 3ca2f3b499 refactor(web): collapse time provenance behind an inline toggle
The labeled provenance rows duplicated values already visible in the
raw JSON dump and inflated the metadata block for every IME event. The
section is now collapsed by default behind a small "time provenance"
toggle sitting inline on the Created row (whose value it explains) -
zero extra height when collapsed, and the base label column returns to
its original width, so the default detail view is pixel-identical to
before P13. Expanding reveals the same rows, indented under Created.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 00:02:26 +02:00
Oliver Kieselbach b9fa760404 feat(web): P13 - CMTrace time provenance visible in the session timeline
The per-line self-anchoring provenance (sourceLocalTs, applied offset +
origin, observational writer offset, derived/rejected markers) reaches
the browser but was buried unlabeled in the raw JSON dump. Sequence
stays THE sort order - without exception (now pinned by a regression
test); the corrected UTC is display-only, and a backwards step of the
displayed time within sequence order is marked as information instead
of confusing the reader.

- lib/timeProvenance.ts: readTimeProvenance (camelCase/snake_case,
  tolerant of truncated payloads) + classifyTimeJump with cause
  precedence derived > rejected-source > era-offset; era-offset
  requires applied offsets on BOTH rows - it never guesses.
- TimeJumpBadge on the row AFTER the jump, computed against the
  previous RENDERED row (filters/phase boundaries respected); sky for
  known causes, amber otherwise; tooltip carries both times, the delta
  and the "true write order (sequence)" sentence. Threshold 5 min,
  field-calibrated on 20 sessions: 12/12 normal sessions show zero
  badges, the era-mixed regression session shows exactly its 6.
- Clamped chip (rare) + TIME PROVENANCE detail section: only the
  applied offset says "(applied)"; the writer offset is explicitly
  observational (sticky after era flip-backs); origin chips with
  forward-compatible fallback for unknown values.
- EnrollmentEvent gains timestampClamped/originalTimestamp (wire
  already carried them); formatUtcOffset joins lib/formatting.

Verified: 814 web tests (19 new), tsc/eslint/build clean; Playwright
against e9753578 (badges match the pre-measurement), 9f475bb2 (no
badge on line-anchored rows, detail section correct, raw mode shows
exactly the one predicted jump).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 23:54:39 +02:00
Oliver Kieselbach ab61839685 feat(backend): P6 split - CMTrace skew tripwire + customer clock_skew rule
Operator side: CmTraceSkewTripwire piggybacks on the terminal counter
reconcile's partition scan (zero extra reads) and compares median
ReceivedAt-OccurredUtc deltas of IME-derived vs other events. A clean
15-min-grid divergence (residual < 2 min, >=20 samples / >=3 distinct
batches per side) emits the CmTraceTimeSkewRegression ops event
(Agent/Warning, goal: never fires). Bias-dominated sessions are
suppressed; measuredWriterOffsetMinutes is never consulted (sticky
after era flip-backs). Kill switch: CmTraceSkewTripwireDisabled.

Customer side: new clock_skew analyze condition source (skewMetric
clock_jump | sustained_offset, threshold seconds, gt/gte magnitude)
behind built-in ANALYZE-DEV-008 (300 s = the Kerberos/Entra 5-min skew
boundary). Batch-median frames with spool-spread filter, plateau step
detection (a gradual backlog-drain ramp cannot fake a jump) and
end-state persistence; IME-derived events are excluded so a CMTrace
anchoring regression can never fire the customer rule. Evidence carries
field=skewSummary/value for {{skewSummary}} interpolation.

Full mirroring chain: schema enum + skewMetric property, shared
manifest + combine.js regeneration, MCP authoring guide + semantic
lint, web SOURCES/form/card, ops-event dual-register, docs. P5 is
subsumed by the tripwire. PercentileMath is now the single nearest-rank
implementation (Apps/PlatformMetrics delegate).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 22:30:10 +02:00
Oliver Kieselbach 34fd40c3ad fix(agent): refuse cross-file calibration anchors, make rejections visible
Field forensics from the 1412 verification run (session 067797d8): the
label/anchor crossing recurred on a provably genuine binary, and the
new anchor-timestamp tripwire delivered — the "+02:00 measured for
IntuneManagementExtension.log" anchor (local 19:51:26.948) does not
exist in ANY log file on the device, and IME.log never produced its
expected first measurement during 24 s of growing passes beforehand.
Mechanism still unexplained; per-line anchoring keeps resolution
immune (all 58 line-anchored events of the session were correct), but
the observational layer must convict the culprit next time:

- CalibrateFrom now receives the iteration's ACTUAL file name next to
  the calibration key. A mismatch — the smoking gun the two field
  occurrences imply but static analysis says is impossible — logs a
  Warning with both names plus the anchor, and refuses the anchor, so
  even the unexplained path can no longer poison the measurement store.
- The first calibrator rejection per file logs at Debug (anchor local
  time + clock). The 24 s of silent non-measurement is exactly what
  made the phenomenon undiagnosable from the agent log.

232 IME tracker tests green against a verified-clean build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 20:18:35 +02:00
Oliver Kieselbach a0eb1b9e35 Merge branch 'main' of https://github.com/okieselbach/Autopilot-Monitor 2026-08-20 19:45:40 +02:00
Oliver Kieselbach 9d677db2a8 docs(okf): CMTrace time resolution — per-line self-anchoring concept
Why the reader-zone assumption and the per-file offset application both
failed, the per-line self-anchoring rule with its strict freshness
contract (LastCheckedUtc on every poll look, 30 s window, no freshness
across restarts), the provenance vocabulary that keeps every event
re-derivable, and the documented limits: the bounded grid-replay edge,
snapshot readers staying on the marked fallback, and the still-open
e9753578 label/anchor crossing with its tripwire.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 19:28:54 +02:00
Oliver Kieselbach c728a0bd41 feat(backend): surface genuine agent binary-integrity mismatches as ops events
IntegrityCheckFailed emergency reports used to land in App Insights
only — session e9753578's agent reported a hash mismatch and no product
surface showed it, which cost a day of mis-attributed root-causing.
They now materialize as an AgentBinaryIntegrityMismatch ops event
(Warning, Agent category, dual-registered in the web alert-rule
catalog per feedback_ops_event_types_dual_register).

With suppression for the KNOWN, accepted release race the operator
correctly called out: around every release, agents compare against
hash snapshots that do not belong to their own binary — an older agent
sees the new release's hash on its next config fetch, and a freshly
installed latest agent can receive the PREVIOUS release's hashes from
the 5-minute AdminConfiguration cache (e9753578: genuine 1410 exe
checked against the cached 1409 hash — proven by the 1409 archive
logging "verified OK" with exactly that hash). Both self-heal via the
forced self-update and are suppressed at materialization time against
the CURRENT config: reported running hash == published exe, or
reported version != latest. What remains — an agent claiming the
latest version whose binary is NOT the published one — is the real
signal: tamper, stale blob, or a build outside the release pipeline.
Missing config fails open to recording; visibility beats suppression.

Ops event only, deliberately no timeline event: the report is about
the binary, not the enrollment, and may precede session registration.

Backend suite 4320 green, web tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 19:28:43 +02:00
Oliver Kieselbach 64336c0c89 feat(agent): resolve fresh CMTrace lines by per-line self-anchoring
The era-aware successor to the reverted per-file offset application
(04b1a7c6). At a 100 ms poll, every line of a provably fresh pass was
written essentially "now", so its own distance to the agent clock,
rounded to the 15-minute offset grid, IS its writer's UTC offset — no
cross-line state, which is what interleaved writer eras break. The line
timestamp still contributes sub-poll precision and ordering; the grid
absorbs poll latency. Origin: LineAnchored → "line-anchored".

Freshness is stricter than "the file grew": every poll look stamps
LastCheckedUtc (empty first sights included, so AgentExecutor.log
anchors from its very first content), a pass anchors only when the gap
since the last look is ≤ 30 s (FreshLineMaxAge — far above the poll,
far below the 13-minute zone where a line's AGE rounds onto the grid),
and a restart-restored bookmark deliberately carries no freshness at
all — downtime backlog must never anchor. Everything not provably
fresh keeps the uniform reader-zone fallback: wrong in absolute terms,
self-consistent for durations, and marked as such (the revert's core
lesson — partially corrected is strictly worse than uniformly wrong).

Accepted, bounded edge (characterization test): a freshly WRITTEN line
carrying a replayed timestamp aged exactly N×15 min ± 2 min anchors to
≈now — an anchored line lands at now ± 2 min by construction, and
sub-24h replays are treated as current by the historic-replay guard.

The per-file measurement stays observational (Info log +
measuredWriterOffsetMinutes) and now logs the anchor's local timestamp
as a tripwire: session e9753578 logged "+02:00 measured for
IntuneManagementExtension.log" although that file provably held no
+2-era line — a label/anchor pairing the committed code could not be
shown to produce (iteration-local anchor, single sequential call site,
green repro of the exact field scenario, no +2-capable chunk in the
window). The mechanism remains open; the resolution path is immune to
it by design, and CmTraceCalibratorFileIsolationTests pins the file
isolation so a real regression can never ship silently.

Tests: 13 new — resolves-to-T across five writer beliefs, interleaved
eras inside one chunk, first-sight/empty-first-sight/poll-gap/restart
freshness boundaries (including the nastiest backlog: exactly 30 min
old, on-grid), the grid-replay edge, and the committed two-writer-era
fixture replayed at production poll cadence across its write silences.
Agent suite 2406 green against a verified-clean build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-20 19:28:27 +02:00
Oliver Kieselbach 59c944618b test(agent): move CMTrace fixture to tests/fixtures and anonymize it
Two corrections to 3c25172d, both found by asking why the repo suddenly had two
fixture locations.

Location: committed test data lives under tests/fixtures/. I had put this one
inside the agent test project without checking, creating a second parallel home.
It does NOT belong in tests/fixtures/enrollment-sessions/ either — that is a
governed corpus of anonymized DecisionSignal JSONL for the Replay Harness, with a
schema, category prefixes and a paired-scenario-test rule that raw log text cannot
satisfy. So: a sibling corpus, tests/fixtures/cmtrace-logs/, with its own README
stating why it is separate.

Anonymization: the fixture carried three real Intune policy/script GUIDs from a
customer tenant. Point 1 of the existing fixture policy requires tenant-scoped
identifiers to be replaced, and I committed without applying it — the check
belonged BEFORE the commit, not after. Replaced with
policy-anon-NNNN-0000-0000-0000-000000000000, consistent with the tenant-anon /
device-anon / user-anon convention already in use. The all-zero GUID is a literal
in IME's own output and stays; the session ID stays as our diagnostic anchor and
is already referenced from the commit history and tasks/todo.md.

The new README carries the commit policy for this corpus, including the rules that
were implicit before: verbatim content (the parser is tested against real
formatting), a header stating what the fixture pins, versioned filenames so a
pinned test cannot silently change meaning, and .cmtrace rather than .log because
*.log is globally gitignored.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 17:54:55 +02:00
Oliver Kieselbach 3c25172dfb test(agent): real two-writer-era CMTrace fixture from session e9753578
Captured from ImeLogs/AgentExecutor.log of the session that exposed the regression
(agent 2.0.1410; device booted on the Pacific OOBE default, agent set W. Europe at
12:13:22Z).

One file, two writer eras: 371 lines rendered at local hour 05 by a writer still
on PDT (UTC-7) and 195 at hour 14 by one already on CEST (UTC+2) - nine hours
apart, interleaved, and no line carries a bias suffix, so nothing in the file
itself says which is which. Exactly the case the per-file calibration could not
express, and the test that was missing.

Analysing the archive also turned up a SECOND, independent defect the revert
commit did not know about: the agent log reports "IntuneManagementExtension.log
writer offset measured 02:00:00" while that file contains ONLY hour-05 lines
(2022 of them, zero at hour 14). A +2 measurement is impossible from its own
content, so the calibration anchor leaked across file boundaries - files are
walked in sorted order and AgentExecutor.log precedes it. Two-eras-per-file and
anchor-leak are separate bugs; both must be fixed.

Extension is .cmtrace.txt, not .log: *.log is globally gitignored, and a fixture
has no reason to claim that extension.

Fixture only, nothing consumes it yet - the era-aware calibrator is next.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 17:47:15 +02:00
Oliver Kieselbach eda61eaade Fix Tenant Groups layout overflowing on phones (#132) 2026-08-20 15:27:08 +02:00
Oliver Kieselbach 04b1a7c6e7 revert(agent): stop applying the measured CMTrace offset — one offset per file is wrong
Field regression, session e9753578 (agent 2.0.1410, PST-defaulted device in DE).
Five script_started events resolved 9 h early while their script_completed
counterparts resolved correctly, inflating every platform-script duration to
~32,400 s. Reverts the application introduced in e9dba11b.

Root cause: calibration keeps ONE offset per log file, but a single file can hold
lines from two writer eras. IME restarting across a timezone change leaves older
lines in the old zone and newer ones in the new zone, in the same file. The anchor
is the newest line, so its offset was also applied to the older lines, for which
it does not hold. The class doc even claimed "self-healing on writer restart" —
that reasoning only ever covered FUTURE lines, never the ones already on disk.

The more expensive mistake was the second one: correcting ABSOLUTE timestamps
without checking what depends on their DIFFERENCES. Applying the reader zone
uniformly is wrong in absolute terms but self-consistent — both ends of a duration
are wrong by the same amount, so the duration is right. Correcting only some
timestamps produced a mixed frame inside one session, which is strictly worse than
being uniformly wrong. This is the same asymmetry the original analysis described
for AppInstallSummaries (68 min reported against 8 min real) and then reproduced.

What stays, deliberately:
- Bias-carrying lines still resolve via the writer-declared offset; that path was
  never implicated.
- The measurement keeps running. It costs nothing, feeds the Info log, and is the
  field data the era-aware fix has to be built on — the first anlauf got the era
  question wrong from assumption rather than observation.
- Provenance now separates measured from applied (measuredWriterOffsetMinutes vs
  sourceOffsetMinutes). The broken event was tagged "calibrated" while carrying an
  offset that did not hold for its line; that confusion must not be expressible.
- The fallback path reports the offset it actually applied. An event stating no
  offset cannot be recomputed later, which defeats the point of provenance.

Provenance earned its keep here: 57 events measured -420 (the PDT writer) against
5 at +120 made the diagnosis a two-minute query instead of median statistics over
a session.

Next: era-aware calibration keyed on the IME-STARTED marker, which we already
track, plus a test over a log file containing two writer eras — the case that was
missing.

Agent 2393 green against a verified-clean build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-20 14:31:50 +02:00
410 changed files with 28556 additions and 2662 deletions
+127
View File
@@ -0,0 +1,127 @@
---
type: concept
title: CMTrace Time Resolution — Per-Line Self-Anchoring
description: How the agent converts bias-less CMTrace local times to UTC without trusting any process's timezone belief — each provably fresh line anchors its own offset against the agent clock on the 15-minute grid; everything not provably fresh falls back uniformly and says so.
resource: src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/Ime
tags: [agent, ime, cmtrace, timestamps, timezone, calibration, provenance]
timestamp: 2026-08-20
---
# Problem
A CMTrace line carries local time and no offset — IME 1.104 writes `DateTime.Now.TimeOfDay`
in both of its trace listeners. Converting such a line with the reader's own
`TimeZoneInfo.Local` silently assumes the writing and reading processes agree on the zone.
They do not: each caches its zone at process start and follows neither a later `tzutil` nor
a Windows auto-timezone change. The error is `Offset_writer − Offset_reader` — zero only
when both believe the same thing, right or wrong. Field measurement over 11,068 sessions
found 26 sessions with a real error, from +1 h to −17 h.
Two designs failed before this one:
* **Fix the reader's belief** (`TimeZoneInfo.ClearCachedData`, live OS offset): makes the
agent isolated-correct and thereby breaks the majority of sessions, where two equally
stale beliefs cancel.
* **One measured offset per file** (agent 2.0.1410, reverted in `04b1a7c6`): a single log
file holds lines from multiple writer *eras*. `AgentExecutor.log` is written by
short-lived child processes whose zone belief flips per process — interleaved in one
file, and flipping *back* (fixture:
`tests/fixtures/cmtrace-logs/agentexecutor-two-writer-eras-v1.cmtrace`). Any cross-line
anchor — per file, or nearest-in-write-order — applies one era's offset to the other
era's lines. In the field this shifted `script_started` events by −9 h against their
correct completions.
# Schema
The resolution rule, in order:
1. **Writer-declared bias** (`+480` suffix): authoritative, used as-is. Never a calibration
anchor — it needs no measurement and must not overwrite one.
2. **Per-line self-anchoring** — only for lines read in a *provably fresh* pass:
```
offset_line = round((local − agentUtcNow) / 15 min) · 15 min
lineUtc = local − offset_line
```
Every real UTC offset is a whole multiple of 15 minutes, so the rounding absorbs poll
and flush latency. No cross-line state exists, so interleaved eras resolve line by
line — the property the fixture test pins. The line's own timestamp still contributes
sub-poll precision and ordering (two lines 6 ms apart stay 6 ms apart; a plain
`occurredAt = now` could not do that). Guards: grid residual ≤ 2 min, offset within
UTC−12 … UTC+14.
3. **Uniform reader-zone fallback** for everything else: wrong in absolute terms whenever
the writer held a different belief, but *self-consistent* — both ends of a duration are
wrong by the same amount, so derived durations stay right. The revert's core lesson:
partially corrected is strictly worse than uniformly wrong.
**Freshness** is what makes rule 2 sound, and it is strict:
* Every poll look at a file stamps `LastCheckedUtc` (`LogFilePositionTracker.MarkChecked`),
including "no new data" and an empty first sight — so a file first seen empty
(`AgentExecutor.log` appearing mid-enrollment) anchors from its very first content.
* A pass's lines are fresh only if the file was previously observed *in this process* and
the gap since the last look is ≤ `FreshLineMaxAge` (30 s — far above the 100 ms poll,
far below the 13-minute zone where a line's *age* could round onto the offset grid).
* A restart-restored position bookmark deliberately carries **no** freshness: the first
pass after a restart reads downtime backlog and must never anchor.
**Provenance** (`sourceOffsetOrigin` in event DataJson): `bias` | `line-anchored` |
`reader-zone-fallback` (plus retired `calibrated` from 2.0.1410 events). The applied
offset is always recorded (`sourceOffsetMinutes`), the raw local time too
(`sourceLocalTs`), so any event can be re-derived later. The per-*file* measurement still
runs, purely observationally (`measuredWriterOffsetMinutes`, Info log when it disagrees
with the process zone).
# Constraints
* An anchored line resolves to `now ± 2 min` *by construction*. The accepted edge: a
freshly written line carrying a **replayed** old timestamp whose age is exactly
N×15 min ± 2 min anchors to ≈now instead of its replayed past — bounded error, and
sub-24 h replays are treated as current by the historic-replay guard anyway.
* Snapshot readers (`StallProbeCollector`, `LogParserCollector`) cannot tail and therefore
cannot prove freshness — they stay on the marked fallback. This is a documented limit,
not an omission; sharing the per-file measurement with them would re-apply exactly what
the revert removed.
* Mixed pairs remain possible across a restart (start resolved from backlog fallback,
completion line-anchored) and are recognizable via provenance.
# Backend regression tripwire
The backend watches for the failure mode this design eliminates. On every terminal ingest
batch, the counter reconcile's existing event-partition scan additionally collects
`Δ = ReceivedAt − OccurredUtc` per event, split by `Source == "ImeLogTracker"` versus the
rest (`CmTraceSkewTripwire`, hooked in `EventIngestProcessor`). If
`median(Δ_IME) − median(Δ_other)` is a clean 15-minute multiple (residual < 2 min — the
same constants as the anchoring grid guard above) with ≥ 20 samples over ≥ 3 distinct
upload batches per side, a `CmTraceTimeSkewRegression` ops event fires (category Agent,
warning), carrying all numbers plus the session's `sourceOffsetOrigin` histogram.
Bias-dominated sessions are suppressed (writer-declared offsets cannot be an anchoring
regression); `measuredWriterOffsetMinutes` is never consulted (sticky after era
flip-backs). Kill switch: app setting `CmTraceSkewTripwireDisabled`. Goal state: the event
never fires — any hit is an anchoring case the per-line design misses, or a detector bug.
Device-clock problems (the customer-actionable cousin) are covered separately by the
`clock_skew` analyze condition behind `ANALYZE-DEV-008`, which excludes IME-derived events
precisely so an anchoring regression can never surface as a customer finding.
# Citations
* `CmTraceOffsetCalibrator.TryMeasureOffset` — the pure grid measurement, shared by
anchoring and the observational per-file path.
* `CmTraceSkewTripwire` + `CmTraceSkewTripwireTests` — the backend tripwire above.
* `RuleEngine.ConditionEvaluators.EvaluateClockSkewCondition` + `RuleEngineClockSkewTests`
— the customer-facing device-clock metrics (batch-median frames, spool-spread filter,
plateau step detection, end-state persistence).
* `ImeLogTracker.LogProcessing.cs` `ResolveEntryUtc` — the resolution order above.
* `ImeLogTrackerLineAnchoringTests` — resolves-to-T across writer beliefs, interleaved
eras in one chunk, freshness boundaries (first sight, empty first sight, poll gap,
restart restore), the grid-replay edge, and the field fixture replayed at production
poll cadence.
* `CmTraceCalibratorFileIsolationTests` — pins per-file isolation of the observational
measurement (session e9753578 logged a cross-file pairing the committed code could not
be shown to produce; the mechanism remains open, the resolution path is immune by
design, and `CalibrateFrom` logs the anchor timestamp as a tripwire).
* `tasks/todo.md` (2026-08-20) — full evidence trail: field measurements, the revert,
the two-writer-era fixture, and the disproven foreign-binary hypothesis (the SHA
mismatch was the known release race against the 5-minute AdminConfiguration cache).
+33 -1
View File
@@ -128,6 +128,15 @@ user-ESP Install-intent apps are still pending (IME logs "Completed user session
*pass*, not once per session — sessions 14690fc2/6cb01530). The reducer stays free of
app-pending guards; adapters sanitize evidence before it becomes a signal.
**Zero user-targeted apps** (sessions 81daa77f/75d6ae8e): IME never writes its
"Completed user session" line at all when the user has no assigned apps — its zero-app
branch returns first — so fact 3 was unreachable by construction and every such device
fell through to the backstop. `IME-USER-SESSION-ZERO-APPS` reads the line IME *does*
write there and posts the same signal, guarded to the AccountSetup phase. The event
carries an `evidence` field (`zero_user_apps_enumerated` vs `user_session_completed`)
naming which observation carried the completion. See
[zero-user-apps-completion-gate.md](zero-user-apps-completion-gate.md).
## Completion attempt sites and sequence
Completion can be attempted from whichever fact arrives last: `HelloResolved`,
@@ -143,11 +152,34 @@ advisory backstop. All routes converge on `CompleteThroughFinalizingOrDefer`:
first-deployment window 100/101 for 200/210 without 110 — the RJ ESP was aborted,
typically by an interactive logon reclassifying the run as secondary-user; emits
the `realmjoin_first_deployment_incomplete` Warning, session 224b2087), and
`Timeout` (hard 60-min deadline). The abort rule reads the *persisted*
`Timeout` (60-min deadline from detection). The abort rule reads the *persisted*
`RealmJoinFacts.LastDeploymentPhase` (updated by the typed `RealmJoinPhaseChanged`
signal), so it survives agent restarts — the `RealmJoinDetected` replay path
evaluates it too. A first observation of 210 does **not** release (RJ may have
completed before agent boot and deploy afterwards).
**Activity-based timeout extension** (report 55e6afd61c9d — a large RJ package
catalog outlived the 60-min window mid-install): when the timeout deadline fires
while the first deployment is demonstrably active — `LastDeploymentPhase` 100/101
*and* deployment activity (`RealmJoinPhaseChanged` or a package observation, tracked
in `RealmJoinFacts.LastActivityUtc`; the phase captured at detection deliberately
does not count) within the last 60 min — the handler re-arms the deadline to
`lastActivity + 60 min` instead of timing out, emitting an Info
`realmjoin_timeout_extended` event. Re-arms clamp to an absolute cap of
**detection + 4 h**. Package activity is only observable at package *completion*
(RJ writes the registry subkey when the install finishes), so a single install
that is registry-silent for over 60 min still times out — the eventual
`realmjoin_timeout` carries a `reason` payload (`hard_timeout` / `inactivity` /
`absolute_cap`) telling the three shapes apart. A stale fire of an older deadline
incarnation (armed due later than the fire time; `OccurredAtUtc = DueAtUtc` per
scheduler contract) dead-ends as `realmjoin_timeout_stale_superseded_by_rearm`.
The idle case (detected, no deployment activity ever) times out at 60 min
unchanged, and phase 200/210 activity never extends.
A closed gate also outranks the 30-minute `AdvisoryCompletion` backstop: that
handler re-arms (trigger suffix `:CompletionGateHolding`) instead of failing while
the RealmJoin gate is closed *and* its own deadline is still armed, because the gate
carries a bounded resolution of its own. Before this, the shorter timer always won —
session 75d6ae8e was failed 2 min after its 25th successful RealmJoin package.
2. `Finalizing` stage + ~5 s `FinalizingGrace` deadline.
3. Grace fires → `Completed`, `EnrollmentComplete` outcome, `enrollment_complete`
timeline event with full audit payload.
+147
View File
@@ -0,0 +1,147 @@
---
type: concept
title: Diagnostics Package
description: What the agent's diagnostics ZIP contains and why — the built-in section catalog shared with the portal, the RealmJoin and Device Preparation collection gates, the configured-path guards, caps, and the package manifest.
resource: agent
tags: [diagnostics, realmjoin, device-preparation, guardrails, portal]
timestamp: 2026-08-23
---
# Schema
The diagnostics package is the agent's last-resort support artifact: a ZIP built in memory
by `DiagnosticsPackageService.BuildArchiveBytes` and uploaded through a short-lived SAS
URL at the end of a session, at the WhiteGlove seal (`Always` mode) or on a server-requested
on-demand collection ([server actions](../backend/server-actions-on-demand-diagnostics.md)).
## Archive layout
```
sessioninfo.txt session/tenant/device/outcome, hardware via WMI
<built-in sections> catalog order, see below
AdditionalLogs/<last folder>/ admin-configured paths (global ∪ tenant)
package-manifest.txt every packaging decision (always written)
_TRUNCATED.txt only when a cap skipped something
```
## The built-in catalog is the single source of truth
Everything the package collects **before** any configured path is data, not code paths:
`DiagnosticsBuiltInSections.All` in Shared (`Models/Diagnostics/DiagnosticsBuiltInSections.cs`).
The agent iterates it in order; the backend serves the same list to the portal
(`GET /api/diagnostics/paths`, MemberRead) so what administrators see is exactly what the
agent does. There is no second copy to drift.
| Id | ZIP folder | Source (unexpanded) | Patterns | Recursive | Condition |
| --- | --- | --- | --- | --- | --- |
| AgentLogs | `AgentLogs` | `%ProgramData%\AutopilotMonitor\Logs` | log patterns | no | Always |
| ImeLogs | `ImeLogs` | `%ProgramData%\Microsoft\IntuneManagementExtension\Logs` | log patterns | no | Always |
| ImeBootstrapperEventLog | `ImeLogs` | `C:\Windows\System32\winevt\Logs` | `BootstrapperAgentServiceLogProvider.evtx` | no | DevicePreparation |
| AgentState | `AgentState` | `%ProgramData%\AutopilotMonitor\State` | log patterns + `*.complete`, `*.marker` | yes | Always |
| AgentSpool | `AgentSpool` | `%ProgramData%\AutopilotMonitor\Spool` | `*.jsonl`, `*.json` | no | Always |
| AgentMarkers | `AgentMarkers` | `%ProgramData%\AutopilotMonitor` | `*.complete`, `*.marker` | no | Always |
| RealmJoinWindows | `RealmJoinLogs/Windows` | `C:\Windows\Logs` | `realmjoin*.log` | no | RealmJoinWatcher |
| RealmJoinPackages | `RealmJoinLogs/Windows/RealmJoin` | `C:\Windows\Logs\RealmJoin` | `*.log` | yes | RealmJoinWatcher |
| RealmJoinChoco | `RealmJoinLogs/Choco` | `%ProgramData%\RealmJoin\choco\logs` | `*.log` | yes | RealmJoinWatcher |
| RealmJoinUserTray | `RealmJoinLogs/User` | `%LOGGED_ON_USER_PROFILE%\AppData\Local\RealmJoin` | `tray*.log` | no | RealmJoinWatcher |
| RealmJoinUserLogs | `RealmJoinLogs/User/Logs` | `%LOGGED_ON_USER_PROFILE%\AppData\Local\RealmJoin\Logs` | `*.log` | yes | RealmJoinWatcher |
"Log patterns" = `*.log *.txt *.json *.jsonl *.etl *.evtx *.xml *.csv *.cab`. The five
`Always` sections are the historical layout (PR1-B forensics) and are lock-tested in order;
the RealmJoin ZIP layout mirrors the disk (`C:\Windows\Logs\{realmjoin*.log, RealmJoin\…}` →
`RealmJoinLogs/Windows/…`, `%LOCALAPPDATA%\RealmJoin\{tray*.log, Logs\…}` →
`RealmJoinLogs/User/…`), so a flat section and a tree section never produce the same entry
name. Two sections may share a ZIP folder only when they read different source folders
(`ImeLogs` + the bootstrapper evtx) — `(ZipFolder, SourceFolder)` pairs are unique.
### Conditions
`DiagnosticsSectionCondition` is evaluated by the agent (`IsSectionActive`, pure) against its
effective configuration and the enrollment scenario; unknown values fail closed.
* **RealmJoinWatcher** — the tenant's RealmJoin Watcher toggle
(`AnalyzerConfiguration.EnableRealmJoinWatcher`). The package service only sees
`AgentConfiguration`, so `RemoteConfigMerger` mirrors that one Analyzers knob onto
`AgentConfiguration.EnableRealmJoinWatcher` (the RealmJoin host keeps reading the remote
block). Deliberately NOT additionally gated on the compile-time `RealmJoinTrackingEnabled`
kill switch: if the live watcher is ever disabled, the logs become more useful, not less.
No ConfigVersion bump — no new wire field.
* **DevicePreparation** — `EnrollmentRegistryDetector.IsDeterministicDevicePreparation()`,
the same deterministic `DevicePreparation\BootstrapperAgent\ExecutionContext` marker every
other WDP gate keys on ([WDP scenario gates](wdp-scenario-gates.md)); CloudAssigned*
fallback rules count as Classic. The probe is a registry read, evaluated once per build
(`SCENARIO: devicePreparation=… realmJoinWatcher=…` in the manifest); a probe failure is
treated as Classic and logged — the evtx export is skipped, never the package. The channel
is exclusively locked while active, so the existing `wevtutil epl` export path
(`ResolveChannelForEvtxFile`, field case 2026-08-17) handles it.
Inactive sections are never silent: `BUILT-IN SKIPPED (RealmJoin Watcher disabled): <Id>`,
`BUILT-IN SKIPPED (not a Device Preparation enrollment): <Id>`,
`BUILT-IN SKIPPED (no user session for token): <Id> '<source>'` (the
`%LOGGED_ON_USER_PROFILE%` token resolves via `UserProfileResolver` only once an
interactive user exists — identical to configured paths). Active ones log
`BUILT-IN: <Id> -> folder='…' zip='…' recursive=…` followed by the per-pattern
`ADDED / NO MATCH / FOLDER MISSING` lines.
## Built-ins bypass the configured-path guards — on purpose
`DiagnosticsPathGuards` ([gather-rule guardrails](../rules/gather-rule-guardrails.md))
validates **admin-typed** paths: allowlisted prefixes, wildcard only in the last segment,
`C:\Users` blocked except the token's `AppData\Local|Roaming`. The catalog is reviewed code,
exactly like the five sections that never went through the guard before — and
`%ProgramData%\RealmJoin` is not on the allowlist at all. What does apply to every section:
the reparse-point skip (files and directories), the `.evtx` channel export, and the caps
(100 MB per file, 500 MB total, 5000 files) tracked globally across sections by
`BudgetTracker`, with every skip recorded in the manifest and `_TRUNCATED.txt`.
## Configured paths: global ∪ tenant
`GetAgentConfigFunction.MergeDiagnosticsLogPaths` sends global paths (Global Admin,
`AdminConfiguration.DiagnosticsGlobalLogPathsJson`) first, then tenant paths
(`TenantConfiguration.DiagnosticsLogPathsJson`), dropping blank entries and collapsing
duplicates (trimmed, case-insensitive) onto the first occurrence — a path present in both
lists used to produce duplicate ZIP entry names. The agent keys each entry's ZIP folder on
the last directory name (`AdditionalLogs/<folder>`), so two configured paths ending in a
same-named folder share one ZIP folder.
## Portal surfaces
* Global Admin → Settings → Diagnostics Log Paths: the built-in catalog (collapsed, read-only,
neutral condition pills) above the editable global list; add-row first.
* Tenant → Settings → Agent → Diagnostics Package: built-in catalog (header carries the
tenant's persisted RealmJoin Watcher state, RealmJoin rows show on/off), the global paths
read-only, then the tenant's own entries; every role with a tenant membership can read the
catalog (the route is MemberRead, JWT-scoped). One line per entry (`DiagnosticsPathRow`),
full values in tooltips.
## Test seams
* `DiagnosticsPackageService` internal ctor: the five historical folder overrides map onto
catalog ids; `sectionFolderOverrides` covers any other section; `devicePreparationProbe`
replaces the registry read (seam callers default to "not WDP", so existing rigs never touch
the registry). `UserProfileResolver.SetForTesting(path|null)` / `Reset()` drive the token.
* Catalog invariants live in the backend test project (`DiagnosticsBuiltInSectionsTests`),
because Shared has no test project.
# Examples
A RealmJoin tenant (watcher on) with a signed-in user: `RealmJoinLogs/Windows/realmjoin.log`,
`RealmJoinLogs/Windows/RealmJoin/Packages/<pkg>/<timestamp>_install.log`,
`RealmJoinLogs/Choco/<pkg>/<timestamp>_install.log`, `RealmJoinLogs/User/tray.log`,
`RealmJoinLogs/User/Logs/RjImeHost.log`. The same tenant's package collected before the
desktop: the two `User` sections appear as `BUILT-IN SKIPPED (no user session for token)`.
A Classic enrollment: `BUILT-IN SKIPPED (not a Device Preparation enrollment):
ImeBootstrapperEventLog`. A tenant that had added
`%LOGGED_ON_USER_PROFILE%\AppData\Local\RealmJoin\Logs\*.log` manually keeps working — the
files simply appear twice (`AdditionalLogs/Logs/…` and `RealmJoinLogs/User/Logs/…`) until the
entry is removed.
# Citations
* `src/Shared/AutopilotMonitor.Shared/Models/Diagnostics/DiagnosticsBuiltInSections.cs` — the catalog
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Runtime/DiagnosticsPackageService.cs` — `BuildArchiveBytes`, `AddBuiltInSection`, `IsSectionActive`, caps, evtx export
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Configuration/RemoteConfigMerger.cs` — `EnableRealmJoinWatcher` mirror
* `src/Backend/AutopilotMonitor.Functions/Functions/Diagnostics/GetDiagnosticsPathsFunction.cs` — `GET diagnostics/paths`
* `src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs` — `MergeDiagnosticsLogPaths`
* `src/Web/autopilot-monitor-web/components/diagnostics/` — `DiagnosticsPathRow`, `BuiltInSectionsList`, `builtInSectionDisplay`
* Tests: `DiagnosticsPackageServiceTests` (RealmJoin on/off, no user session, bootstrapper evtx, probe failure), `DiagnosticsBuiltInSectionsTests`, `GetDiagnosticsPathsPayloadTests`, `GetAgentConfigFunctionTests.MergeDiagnosticsLogPaths_*`
+113
View File
@@ -0,0 +1,113 @@
---
type: Concept
title: DO GroupId Auto-Set — Delivery Optimization peering group from a network fingerprint
description: EnableDoGroupIdAutoSet (ConfigVersion 40) makes the agent set the DOGroupId policy value to a deterministic GUID derived from the default gateway's IPv4 address and MAC (RealmJoin-compatible byte layout), so devices on the same local network form one DO peering group during enrollment; foreign DOGroupId/DOGroupIdSource policies always win.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Transport/DoGroupIdService.cs
tags:
- agent
- delivery-optimization
- network
- tenant-configuration
timestamp: 2026-08-21T00:00:00+02:00
---
# DO GroupId Auto-Set
## Schema
### The problem
With Delivery Optimization download mode Group (2), peering is scoped by GroupId. When no
GroupId is configured, Windows derives it from AD site → authenticated domain SID → Entra
tenant ID — for a cloud-native tenant that means **one tenant-wide group**, so "peers on
the same network" degrades to "peers anywhere in the tenant" and DO's cloud service has to
match them by public IP. A GroupId derived from the local network makes the group exactly
"devices behind the same gateway" — including branch offices behind NAT. RealmJoin ships
this as its network-fingerprint GroupId; our agent is on the device even earlier (during
ESP, when the big app downloads actually happen), so it can seed the same value for
Autopilot enrollments.
### The setting
`TenantConfiguration.EnableDoGroupIdAutoSet` (`bool?`, default null→false), served as
`AgentConfigResponse.EnableDoGroupIdAutoSet` — **ConfigVersion 40**. Portal: Settings →
Agent → Settings, toggle "Set Delivery Optimization Group ID", stored via the usual
tenant-config roundtrip (`TableConfigRepository` Store+Map). Also shown in the admin
tenant-config report. `RemoteConfigMerger` maps the bool 1:1; it survives the offline
config cache like every plain feature flag.
The agent does NOT configure `DODownloadMode` — the fingerprint GroupId only has an
effect when the tenant already deploys download mode Group (2) via Intune/GPO.
### The fingerprint
`DoGroupIdService.ComputeGroupId(ipv4Bytes, macBytes)` builds a 16-byte GUID:
```
{101,48,6} + {0,0,0} + {ip[1], 0, ip[2], ip[3]} + mac[0..5] → new Guid(byte[16])
```
Gateway discovery reuses the shared `NetworkInterfaceLocator` heuristic (Up, not
Loopback/Tunnel, non-0.0.0.0 gateway; IPv4 gateway required), the gateway MAC comes from
`ArpNativeMethods` (`iphlpapi!SendARP`). The byte layout — including the IP shuffle and
the zero padding — is **RealmJoin-compatible and must never change**: devices managed by
RealmJoin on the same network compute the identical GUID and peer with ours. The registry
value is written as `Guid.ToString()` (braceless lowercase), identical to RealmJoin's
`SetValue(name, Guid)` string form. Deliberate divergence from RealmJoin: **no
random-GUID fallback** — a random per-device ID would isolate the device in a one-member
group, which is worse than the OS default grouping, so no gateway/no MAC means skip.
### Ownership + conflict rules (never fight a foreign policy)
Target: `HKLM\SOFTWARE\Policies\Microsoft\Windows\DeliveryOptimization\DOGroupId`
(64-bit registry view — the Policies store is WOW64-redirected; key paths reused from
`DoThrottlePolicyReader`). Only this GPO-store value is written; the legacy pre-1607
`GroupId` value name is intentionally not.
Skip conditions (Info event with `skipReason`, nothing written):
| skipReason | Meaning |
|---|---|
| `no_ipv4_gateway` | No active interface with an IPv4 default gateway |
| `arp_failed` | Gateway MAC could not be resolved via SendARP |
| `group_id_source_policy` | `DOGroupIdSource` configured in GPO or MDM store (dynamic group source wins) |
| `mdm_policy_present` | `DOGroupId` managed via the Policy CSP store (Intune owns it) |
| `foreign_value_present` | Existing GPO value neither matches the computed GUID nor the one we last wrote |
Ownership is tracked through the `StartupEventGate` fingerprint for the
`do_group_id_auto_set` key: `written:<guid>` records the GUID we last wrote, so after a
gateway change the agent recognizes its own stale value and updates it — while a value
written by anyone else is never overwritten (`DecideAction`: empty→write, equal→no-op,
ours→update, else→skip; unparseable existing values count as foreign). Every write is
verified by readback.
### The event
One `do_group_id_auto_set` event per outcome change (Source `StartupEnvironmentProbes`,
gated on the fingerprint — unchanged reruns across reboots stay silent; errors never
latch, so each failing start keeps its retry trail). Severity: Info for
success/skip, Warning for errors. Data: `groupId`, `previousValue`, `gatewayIp`,
`gatewayMac`, `written`, `skipped`, `skipReason`, `success`, `error`.
## Examples
First boot on a network with gateway `192.168.178.1` / MAC `00-11-22-33-44-55`:
```
do_group_id_auto_set (Info): Delivery Optimization GroupId set to
00063065-0000-00a8-b201-001122334455 (gateway 192.168.178.1)
```
Tenant already deploys a DO GroupId via Intune:
```
do_group_id_auto_set (Info): Delivery Optimization GroupId not set: mdm_policy_present
```
## Citations
* Service (fingerprint, ownership, write+readback): `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Transport/DoGroupIdService.cs`
* ARP interop: `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Interop/ArpNativeMethods.cs`
* Probe wiring + event + fingerprint dedup: `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Runtime/StartupEnvironmentProbes.cs`
* Backend serve + ConfigVersion: `../../src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs`
* DO policy stores (key paths, Registry64 rationale): `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Periodic/DoThrottlePolicyReader.cs`
+5
View File
@@ -5,8 +5,13 @@ How the on-device agent works:
* [Runtime Overview](overview.md) - lifecycle (install → boot guards → runtime → termination), collector hosts, the single-rail event pipeline to the backend, config & kill switch, on-disk layout.
* [Decision Engine](decision-engine.md) - the pure-reducer state machine (DecisionCore): signals, state, effects, the A/B/C completion arms, invariants, threading.
* [Logs, Persistence & Crash Recovery](logs-and-persistence.md) - every persisted file and why it exists; signal log vs spool; snapshot/journal; the replay-based recovery flow and quarantine.
* [Diagnostics Package](diagnostics-package.md) - what the diagnostics ZIP contains: the built-in section catalog shared with the portal (Shared `DiagnosticsBuiltInSections`, served by GET diagnostics/paths), the RealmJoin Watcher and Device Preparation collection gates, why built-ins bypass the configured-path guards, global ∪ tenant path dedup, caps and the package manifest.
* [Hello Wizard Un-Skip](hello-wizard-unskip.md) - why a single "WHfB disabled" policy read never decides completion: the HelloWizardStarted signal (prevention + cure) and the HelloTracker confirmation second read.
* [Zero User-Targeted Apps](zero-user-apps-completion-gate.md) - why a user with no targeted Intune apps starved every user-phase completion path (IME never writes its "Completed user session" line in the zero-app branch), and why a completion gate that is still holding the session must outrank the 30-minute backstop.
* [Autopilot ZTD Diagnostics](autopilot-ztd-diagnostics.md) - Windows' own diagnostic surfaces for the profile-download flow: ModernDeployment event IDs (807/815/164/...), the Diagnostics\Autopilot registry key, deployment-service endpoints, and the known-issue error-code map.
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated 2800 observations (neutral, flush-time watermark) + ANALYZE-ESP-005 gated on an actually-observed reboot.
* [Power-State Watcher](power-state-watcher.md) - live AC↔battery transitions and 50/30/15% downward threshold crossings via a WMI push subscription (no polling): power_state_change semantics (latch/debounce/cap, never armed without a battery) and the single-condition analyze rules ANALYZE-DEV-009/-010 on top.
* [DO GroupId Auto-Set](do-group-id-auto-set.md) - the optional Delivery Optimization peering-group seed: deterministic GUID from default gateway IPv4 + MAC (RealmJoin-compatible), written to the DOGroupId policy value with readback, ownership-tracked so foreign Intune/GPO values are never overwritten; skip reasons and the do_group_id_auto_set event.
* [System Timeline Watcher](system-timeline-watcher.md) - clock steps (system_clock_changed: oldTime/newTime/timeDeltaMs/process, micro-slew floor) and completed sleep episodes (system_sleep_episode: sleep/hibernate/Modern Standby with authoritative enteredAt/exitedAt) from one System-channel EventLogWatcher with pre-agent backfill; feeds the web clock-set jump cause, the standby duration note, time-attribution SleepSpans and ANALYZE-DEV-012.
Reading order for newcomers: overview → decision engine → persistence.
+106
View File
@@ -0,0 +1,106 @@
---
type: Concept
title: Power-State Watcher — live AC/battery transitions and threshold events during enrollment
description: How the agent observes AC↔battery switches and downward battery-threshold crossings mid-enrollment via a WMI push subscription (no polling), the power_state_change event semantics (transition field, 50/30/15 ladder, latch/debounce/cap), and the two analyze rules on top (ANALYZE-DEV-009/-010).
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/PowerStateWatcherHost.cs
tags:
- agent
- power
- battery
- collector
- wmi
- analyze-rules
timestamp: 2026-08-21T00:00:00+02:00
---
# Power-State Watcher
The startup probe (`power_state_check`, one-shot from `StartupEnvironmentProbes` via
`PowerStateProbe`/`GetSystemPowerStatus`) only captures the power state at agent start. A
device unplugged mid-enrollment — or draining its battery across a multi-hour ESP — was
invisible. The watcher closes that gap **event-driven, with zero polling**.
# Mechanism
- `PowerStateWatcherHost` (`Orchestration/Hosts`) arms a WMI event subscription with a
two-step chain:
1. `SELECT * FROM Win32_PowerManagementEvent WHERE EventCode = 10` (PBT_APMPOWERSTATUSCHANGE:
true kernel push, fires on AC↔DC switches and battery-percentage steps).
2. **Fallback** `SELECT * FROM __InstanceModificationEvent WITHIN 30 WHERE TargetInstance
ISA 'Win32_Battery'` — required because on Windows 11 24H2/25H2 (observed on builds
26200 and 26220, session 161b838c) the `MS_Power_Management_Event_Provider` is still
registered in the repository but its activation fails with `WBEM_E_NOT_FOUND`: the
legacy push provider no longer loads on exactly the OS builds new devices enroll with.
The WITHIN sampling runs inside WinMgmt (one Win32_Battery snapshot every 30 s), not in
the agent; worst-case detection latency ≈ 35 s incl. debounce. `collector_degraded`
(`watcher_arm_failed`) is emitted only when BOTH attempts fail; the agent log's arm
line names the active mode.
Either way the arriving event's payload is ignored — each arrival re-probes via the
existing `PowerStateProbe.Probe()` (cheap P/Invoke).
- Snapshots are diffed by the pure `PowerStateTransitionTracker`
(`Monitoring/Enrollment/SystemSignals`), which decides what to emit. The host only owns
WMI, a 5 s trailing-edge debounce timer (dock-flap AC→battery→AC inside the window
collapses to a no-diff) and the emit path (`InformationalEventPost`, single rail).
- **No battery / probe error ⇒ WMI is never armed** — desktops and VMs pay nothing. WMI
arming runs on a background thread (WinMgmt can hang in OOBE; `ConsoleBypassWatcher`
precedent); an arm failure emits one `collector_degraded` (`watcher_arm_failed`).
- Registered as a plain always-on peripheral host after `NetworkChangeHost`, deliberately
NOT under `PeriodicCollectorLifecycleHost`: it is push-based (zero idle cost) and must be
alive precisely when the device idles on battery. Kill-switch:
`Collectors.EnablePowerStateWatcher` (default true, compile-time default like the other
collector toggles; ConfigVersion 38).
# Event: power_state_change
`data.transition` discriminates three shapes (payload always carries `onAcPower`,
`batteryPercent`, `isCharging`, `batteryLifeMinutes`; unknowns as `"unknown"`):
| transition | Severity | Immediate | Notes |
|---|---|---|---|
| `ac_to_battery` | Warning | yes | one per accepted edge |
| `battery_to_ac` | Info | no | recovery |
| `threshold_crossed` | Info (50) / Warning (30) / Error (15) | 30+15 only | extra field `thresholdPercent` |
Threshold semantics (tracker):
- Ladder 50/30/15, evaluated only while ON battery; **each level latches once per agent
lifetime** (charge-up + re-drain does not re-emit — no plug-cycle alarm spam).
- A multi-level jump emits only the LOWEST newly-crossed level (60%→10% ⇒ one `15` event).
- Entering battery power already below a level counts as crossing it; the arm-time baseline
does too — an enrollment STARTING at 12% battery emits its `thresholdPercent: 15` event
immediately (the analyze rule would otherwise never see sessions that began low).
- Lifetime emission cap 20 (storm backstop against a flapping dock / dying battery
controller); the host logs one warning when it engages.
Backend/agent hygiene: `power_state_change` is classified as non-activity on both sides —
`SignalActivityClassifier.NonActivityEventTypes` (agent: must not reset the
PeriodicCollectorLifecycleHost/StallProbeHost idle clocks) and
`EventIngestProcessor.IsPeriodicOrStallEvent` (backend: must not heal a Stalled session —
battery drain is not enrollment progress).
# Analyze Rules
- **ANALYZE-DEV-009** (high): battery crossed 15% while on battery — matches the
agent-stamped `thresholdPercent equals 15`.
- **ANALYZE-DEV-010** (warning): AC→battery switch — matches `transition equals
ac_to_battery`.
- Both: `evaluateOn: ["enrollment_end", "whiteglove_sealed", "on_event:power_state_change"]`
(live interim finding + terminal finalization), `markSessionAsFailedDefault: false`.
- Both are deliberately **single-required-condition**: `event_data` conditions scan events
independently (no same-instance join), so a two-condition rule (e.g. `transition ==
threshold_crossed AND batteryPercent < 15`) could false-positive across two different
power events. `thresholdPercent` exists only on threshold events and its value pins the
ladder level, so one condition fully identifies the instance.
- `{{token}}` caveat: `AddDataFieldsToEvidence` whitelists only generic identifiers
(appId/errorCode/…), so `{{batteryPercent}}` would render literally. DEV-009 interpolates
`{{thresholdPercent}}` (the matched condition's own dataField); the exact percentage lives
on the linked event in the timeline.
# Citations
- `src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/PowerStateWatcherHost.cs`
- `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/PowerStateTransitionTracker.cs`
- `src/Agent/AutopilotMonitor.Agent.V2.Core/Runtime/PowerStateProbe.cs` (probe, unchanged)
- `rules/analyze/ANALYZE-DEV-009.json`, `rules/analyze/ANALYZE-DEV-010.json`
- [analyze-rule-triggers](../rules/analyze-rule-triggers.md) (interim-trigger design; the WG battery rule was its motivating case)
+110
View File
@@ -0,0 +1,110 @@
---
type: Concept
title: System Timeline Watcher — clock steps and sleep episodes from the System event log
description: How the agent turns two enrollment blind spots into timeline ground truth via one EventLogWatcher on the System channel — system_clock_changed (Kernel-General 1, oldTime/newTime/timeDeltaMs/process) and system_sleep_episode (Power-Troubleshooter 1 classic sleep/hibernate, Kernel-Power 507 Modern Standby) — including backfill of pre-agent records, the payload-times-are-authoritative rule, SleepSpans in time attribution, the web clock-set jump cause, and ANALYZE-DEV-012.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/SystemTimelineTracker.cs
tags:
- agent
- eventlog
- time
- clock-skew
- standby
- collector
- time-attribution
- analyze-rules
timestamp: 2026-08-21T00:00:00+02:00
---
# System Timeline Watcher
Two things silently distort every session timeline and its wall-clock duration:
1. **The OS clock gets stepped** — classically w32time correcting a wrong BIOS clock early
in OOBE, but also tzutil/MDM actions. Before this watcher the jump was only *inferred*
(TimeJumpBadge, `clock_skew`/DEV-008); Windows records the ground truth.
2. **The device sleeps mid-enrollment** — the timeline shows an unexplained gap, the
duration looks inflated, and nothing says "the device was simply asleep for 56 minutes".
`SystemTimelineTracker` (hosted by `SystemTimelineWatcherHost`, config flag
`Collectors.EnableSystemTimelineWatcher`, ConfigVersion 39) subscribes to the **System**
channel with one provider-qualified XPath and forwards both facts as events.
# Sources and events
| Source (System channel) | Emitted event | Key payload |
|---|---|---|
| `Microsoft-Windows-Kernel-General` EventID 1 | `system_clock_changed` | `oldTime`, `newTime` (ISO-8601), signed `timeDeltaMs`, `reason`/`reasonText` (1=`application_set`, 2=`hardware_clock_sync`), `processName`/`processId` |
| `Microsoft-Windows-Power-Troubleshooter` EventID 1 | `system_sleep_episode` | `kind`=`sleep`/`hibernate` (EffectiveState 5 ⇒ hibernate), `enteredAt`=SleepTime, `exitedAt`=WakeTime, `durationSeconds`, `wakeSourceType`/`wakeSourceText` |
| `Microsoft-Windows-Kernel-Power` EventID 507 | `system_sleep_episode` | `kind`=`modern_standby`, `exitedAt`=TimeCreated, `enteredAt`=exit − DurationInUs, `durationSeconds` (scenario), `sleepDurationSeconds` (real sleep), `reason`, `onAcPower`, `batteryRemainingCapacityOnExit` |
**506 is deliberately not consumed.** 507 carries the full episode; pairing 506/507 would
need an orphan-tolerant state machine across agent restarts for zero information gain — an
episode without a 507 never completed and would not be emitted under any design.
# Thresholds (constants, not knobs)
- `MinClockDeltaMs = 2000` — w32time logs Kernel-General 1 for routine **1 ms micro-slews**
(verified live, several per hour); only genuine steps are emitted. Suppressed records are
still claimed by the dedup watermark.
- `ClockDeltaWarningMs = 5 min` — larger steps are Warning + immediate upload (can break
token/cert validation windows; matches the web TimeJumpBadge threshold).
- `MinSleepDurationSeconds = 60` — **Modern Standby only**: every screen-off scenario logs a
506/507 pair, including seconds-long ones with `SleepDurationInUs 0` (`SleepEntered=false`);
the single duration gate suppresses both. Classic Power-Troubleshooter episodes have no
floor — they only exist for genuinely completed S3/S4 transitions.
# Payload times are authoritative
The decisive clock correction happens exactly when the device clock is untrustworthy, and
the backend clamps implausible event timestamps (±168 h/24 h) to server time. Every semantic
instant therefore travels as an explicit ISO-8601 payload field (`oldTime`/`newTime`,
`enteredAt`/`exitedAt`); consumers must never read the event's own `Timestamp` for these.
The event `Timestamp` is set to the record's TimeCreated (WindowsUpdateTracker backfill
convention) so rows land where the wake/step happened.
# Backfill and dedup
Template: `WindowsUpdateTracker`. `LoadWatermark → arm watcher → backfill` with lookback
`SystemEventBackfillLookbackMinutes` (default 1440 = 24 h; 0 disables). The System .evtx
persists across OOBE reboots, so the pre-agent w32time correction and pre-agent standby
episodes are recovered with `backfilled=true`. Cross-restart dedup via RecordId watermark
(`system-timeline-watermark.json` in the state directory) plus an intra-run `HashSet` —
deliberately **not** a high-water mark, which would drop exactly the older, never-emitted
pre-agent records (the live watcher arms before the backfill scan).
# Non-activity classification (both sides)
Both event types are environment observation, not enrollment progress:
`SignalActivityClassifier.NonActivityEventTypes` (agent — must not reset idle/stall clocks)
and `EventIngestProcessor.IsPeriodicOrStallEvent` (backend — a device that slept through its
stall must stay Stalled).
# Consumers
- **Web timeline** (`EventTimeline.tsx`): badge blocks for both events; the TimeJumpBadge
gains the ground-truth cause `clock-set` — `classifyTimeJump` receives the session's
`system_clock_changed` deltas and matches a backward display step against a backward clock
set within max(60 s, 20 %) (`lib/timeProvenance.ts`).
- **Duration story** (`SessionInfoCard` via `useSessionDerivedData.standbySeconds`): the
wall-clock duration deliberately keeps the pause (the enrollment really took that long);
a `· 🌙 56m standby` note explains it. Episodes are dedup'd on `enteredAt`.
- **Time attribution** (`TimeAttributionCalculator.BuildSleepSpans`, AttributionVersion 3):
`SleepSpan`s mirror `RebootSpan`s — cross-cutting annotations clipped to the observation
windows (a WhiteGlove-pause episode contributes only its in-window flanks), never a slice
of the wall-clock partition. Persisted as `SleepSeconds`/`SleepSpansJson`; rendered as the
standby chip in `TimeAttributionLane`.
- **ANALYZE-DEV-012** (info, `trigger: single`, `on_event:system_sleep_episode`): fires once
per session for episodes ≥ 5 min. Deliberately the ONLY new rule: clock issues already
have DEV-008 (`clock_skew`) — a second clock rule would double-report the same action item
(user decision 2026-08-21); clock steps stay timeline ground truth with badges instead.
# Citations
- `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/SystemTimelineTracker.cs`
- `src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/SystemTimelineWatcherHost.cs`
- `src/Backend/AutopilotMonitor.Functions/Helpers/TimeAttributionCalculator.cs` (`BuildSleepSpans`)
- `src/Web/autopilot-monitor-web/lib/timeProvenance.ts` (`clock-set`, `findExplainingClockChange`)
- `rules/analyze/ANALYZE-DEV-012.json`
- Live payload evidence (2026-08-21, Win11 26220): Kernel-General 1 with `TimeDeltaInMs=1`
micro-slews; Kernel-Power 507 with `SleepDurationInUs=2324095807`; Power-Troubleshooter 1
with `SleepTime`/`WakeTime` across a hibernate.
@@ -0,0 +1,136 @@
---
type: Concept
title: Zero User-Targeted Apps — the starved user-phase completion gate
description: Why a device with no user-targeted Intune apps could never satisfy the IME user-session gate, how that turned into a 30-minute esp_exit_without_completion_evidence false positive on a third of one tenant's enrollments, and the two fixes — reading the line IME actually writes in its zero-app branch, and refusing to fail a session while a completion gate is deliberately holding it.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/Ime
tags:
- agent
- ime
- esp
- completion
- realmjoin
- false-positive
timestamp: 2026-08-25
---
# Summary
A Classic user-driven enrollment whose user has **zero targeted Intune Win32 apps** could not
produce the evidence every user-phase completion path required. The session parked, and the
30-minute `advisory_completion` backstop resolved it as
`enrollment_failed: esp_exit_without_completion_evidence` — while the user was at a working
desktop, Windows Hello was provisioned, and RealmJoin was still installing packages.
One tenant hit this on **10 of 30 enrollments in a week**. Every one of them was a false
positive; the devices were healthy in Intune and RealmJoin.
# The starvation chain
The IME writes `Completed user session N, userId: …` at the end of a user-session processing
pass. `IME-USER-SESSION-COMPLETED` matches it and the adapter posts
`ImeUserSessionCompleted`, which sets `DecisionState.ImeUserSessionCompletedUtc`.
But the IME never reaches that statement when the user has no assigned apps. Decompiled IME
`1.104.102.0`, `Win32AppPlugIn/ApplicationPoller.cs`:
```csharp
if (!appPolicies.Any())
{
AppWorkloadLog.TraceInformation($"[Win32App] Get 0 apps for user session {user.RDSSessionId}, user id = {user.IntuneId}");
RegisterApplicationsInIfInESP(sender, user, delegate {
espLockInProcessor.AttemptToLockInApplications(user, new List<ProcessorSubgraph>());
});
continue; // skips the "Completed user session" statement below
}
```
Identical shape in `1.97.107.0`, `1.83.107.0`, `1.80.132.0`. With that fact missing, every door
out of the user phase is shut at once:
| Path | Requirement | Why it is dead |
|---|---|---|
| `ShouldTransitionToAwaitingHello` arm A | `AccountSetupProvisioningSucceededUtc` | ESP registry froze at `AccountSetup: 1 of 5`; neither `categorySucceeded` nor the all-subcategories fallback can ever hold |
| arm B | `SkipUserEsp` | Full ESP, not a skip-user flow |
| arm C | `IsImeUserSessionGenuine` | needs the missing IME fact |
| arm D | `EnrollmentMode.DevicePreparation` | Autopilot v1, not WDP |
| `MaybeSynthesizeAccountSetupCompleteFromSettledUserApps` | `AreUserEspAppsSettled` | returned `false` on an empty app list |
| `HandleAdvisoryCompletionDeadlineFired` conjunction | `imeUserSessionGenuine \|\| isObservationAdvisory` | both false |
The last row is the one that produced the verdict: the backstop that exists to resolve a
dead-end resolved it as a failure.
`AreUserEspAppsSettled` deserves its own note. Its "at least one required app" rule is
deliberately conservative — an empty list normally means *the phase just cleared* or *apps have
not surfaced yet*. But it made **zero apps**, the most settled state possible, indistinguishable
from *unknown*, and there was no evidence available to tell them apart.
# Fix 1 — read the line IME does write
`IME-USER-SESSION-ZERO-APPS` matches the zero-app line and routes it through the **same**
`OnUserSessionCompleted` callback. Downstream this genuinely *is* a completed user session, so
the adapter's pending-apps deferral and its fire-once flag apply unchanged.
This is not a weakened gate — it is IME's own verdict. The zero-app branch locks the ESP in with
an *empty* application list, and the sibling emit site in `AppWorkloadAbstraction.ExecuteUserCheckIn`
returns `ProviderResult.ProvisioningComplete` straight after logging it. All three emit sites are
user-session-scoped (`user.RDSSessionId`, `user.IntuneId`); none is device-scoped.
Guards that stay in force:
* **Phase guard** — the observation only fires while the tracker is in `AccountSetup`. Firing
earlier would burn the adapter's fire-once flag on a pre-sign-in (defaultuser0) timestamp that
`IsImeUserSessionGenuine` (`>= AccountSetupEnteredUtc`) can never accept. An observation made
before the phase was known is remembered and replayed on the transition into it, because IME
writes its check-in lines independently of when the agent first parses the phase marker.
* **Engine guards unchanged** — arm C still requires a post-AccountSetup final ESP exit *and* a
DAD-validated real-user desktop.
* `AreUserEspAppsSettled` returns `true` for an empty list **only** with this explicit
observation — never for a merely empty list.
# Fix 2 — do not fail while a completion gate is holding
An independent defect, and the decisive one for the affected tenant. `advisory_completion` runs
for 30 minutes. The RealmJoin completion gate blocks completion until phase 110 and carries its
*own* bounded resolution: 60 minutes from detection, re-armed on deployment activity, hard-capped
at 4 hours. The shorter timer always won.
Session `75d6ae8e` installed **25 RealmJoin packages successfully inside the window**, the last
one 2 minutes before the failure, while the engine's own `completion_waiting` still read
`waiting on: realmjoin_resolution`.
`HandleAdvisoryCompletionDeadlineFired` now re-arms instead of failing when the RealmJoin gate is
closed *and* its deadline is still armed, emitting `completion_waiting` with trigger
`DeadlineFired:advisory_completion:CompletionGateHolding`.
Convergence comes from the gate itself: `Resolved`, `FirstDeploymentIncomplete` and `Timeout` all
set `RealmJoinFacts.Outcome`, which opens `RealmJoinGateOpen` and stops the guard applying.
Requiring the gate's deadline to still be armed keeps a gate that can no longer resolve itself
from parking the session; `AgentMaxLifetimeMinutes` bounds it regardless.
# Timeline evidence
`ime_user_session_completed` carries an `evidence` field so the basis of the verdict is legible
without the diagnostics archive:
* `user_session_completed` — IME wrote its completion line and no required app was pending.
* `zero_user_apps_enumerated` — IME reported zero targeted apps; the user phase had nothing to
enforce.
No new event type: the fact is the same, only the observation behind it differs.
# Deployment
The pattern ships through the config API and reaches existing agents without a rebuild; an agent
that does not know the `userSessionZeroApps` action logs one Debug line and ignores it. The
*behaviour* needs an agent release. Sessions already stamped `Failed` are not reclassified — the
backend mirrors the agent verdict (`VerdictPaths.AgentFailed`) and has no independent judgement
here by design.
# Citations
* `rules/ime-log-patterns/IME-USER-SESSION-ZERO-APPS.json`
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/Ime/ImeLogTracker.Handlers.cs` — `HandleUserSessionZeroApps`, the phase replay in `HandleEspPhaseDetected`
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/Ime/ImeLogTracker.cs` — `AreUserEspAppsSettled`
* `src/Agent/AutopilotMonitor.Agent.V2.Core/SignalAdapters/ImeLogTrackerAdapter.cs` — `EmitUserSessionCompleted`, evidence labels
* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.Edge.cs` — `HandleAdvisoryCompletionDeadlineFired`
* [decision-engine.md](decision-engine.md), [post-reboot-completion-recovery.md](post-reboot-completion-recovery.md)
@@ -0,0 +1,82 @@
---
type: concept
title: Attempt-Scoped App Install Durations
description: AppInstallSummary.DurationSeconds measures the status-defining (last) install attempt via a LastAttemptStartedAt max-fold anchor — replacing the first-observation-to-last-terminal span that billed multi-pass IME processing gaps to trivial apps; time attribution counts the union of active attempt segments.
resource: src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.AppInstall.cs
tags: [backend, agent, ingest, app-installs, metrics, time-attribution, durations]
timestamp: 2026-08-21
---
# Problem
The IME processes the ESP app list in multiple passes: a device-ESP evaluation pass can
touch an app (often ending `Skipped`) long before the real install runs in a later pass.
`AppInstallSummary` is keyed by `{SessionId}_{AppName}` and folded from ALL of the app's
events, so `DurationSeconds = last terminal − first observation` spanned those passes.
Field case (session `44862cd3`, "Set TimeZone"): evaluation pass 09:16, real install
10:22:58 → 10:23:21 — reported 4002 s for a 23-second install. Fleet averages, the
Slowest-Apps ranking, p95s, and the version-regression radar all inherited the inflation;
the rule engine's `app_install_duration` (already attempt-scoped) contradicted the
dashboard for the same install.
# Schema
Three durable anchors on the row — each with a distinct, non-interchangeable job:
| Column | Fold | Role |
|---|---|---|
| `StartedAt` | min (earliest wins) | First observation. Server-side window/bucket filter of every AppInstallSummaries scan and the radar's version-ordering key. Never re-anchored. |
| `LastAttemptStartedAt` | **max** (in-batch and in `ReconcileAppInstallSummaryWithExisting`) | Anchor of `DurationSeconds`. Null = sentinel (row predates 2026-08, or no start observed) → duration falls back to the historical `StartedAt` span. Replay of an older batch can never regress it. |
| `InstallPassCount` | additive, guarded by `LastAttemptStartedAt` monotonicity (a batch whose newest start is not newer than the stored anchor adds 0) | Explains "finished long after first seen" (web ×N chip); 2+ is normal. |
`DurationSeconds = CompletedAt − LastAttemptStartedAt`, computed at the terminal event and
re-derived in one central reconcile site covering in-order and Q4 out-of-order batches.
A `LastAttemptStartedAt` newer than `CompletedAt` means a fresh pass is in flight — the
completed attempt's stored duration stands (the 0 value omits the column; Merge preserves).
`DownloadDurationSeconds` pairs the latest `app_download_started` with the NEXT
`app_install_started` (attempt-scoped, in-batch). The full span stays derivable as
`CompletedAt − StartedAt` — deliberately not a column.
Weaker-terminal guard (both in-batch and cross-batch): `app_install_completed` with
state `Skipped`/`Postponed` never overrides an `Installed`/`Error` terminal, nor its
`CompletedAt`/`DurationSeconds` — mirroring the long-pinned `app_install_skipped` rule.
# Consumers
* **Fleet metrics / apps dashboard / PDF** (`MetricsMath`, `AppsAnalyticsHelper`): values
drop to attempt scale; labels say "install time (final attempt)". The 6 h
`MaxPlausibleInstallDurationSeconds` gate remains (an attempt longer than the agent's
lifetime is still unobservable).
* **AppVersionRegressionRadar**: cutover gate in `MeasuredAppGroups` — only rows with
`LastAttemptStartedAt` enter the duration population, on BOTH comparison sides. A mixed
population would mask real regressions and falsely re-arm active episodes for up to the
35-day horizon. `MinMeasuredInstalls` keeps it silent until enough post-cutover rows
exist. Thresholds (2.0× / 300 s) were tuned on span data — re-evaluate ~4 weeks after
rollout.
* **Time attribution** (`TimeAttributionCalculator.BuildBlockingAppIntervals`):
sequence-ordered start→terminal pairing yields one segment per pass/attempt. Per-app
seconds = SUM of clamped segments (failed attempts and evaluation passes count — they
occupied the path); occupancy merges the segments, never the per-app hulls; hull
endpoints remain first-start→last-terminal for display and what-if. Deliberately
BROADER than the metrics duration: active time across all passes vs. final attempt.
* **Agent payloads** (`AppInstallTiming` / `ImeLogTrackerAdapter.UpdateAppTiming`): a
terminal→active transition re-arms BOTH endpoints, so event `durationSeconds` also
describes the attempt (needs the agent release carrying this change).
* **Rule engine** `app_install_duration`: was already attempt-scoped — now consistent.
# Rollout
No backfill (user decision, analog to the CMTrace legacy): pre-cutover rows keep span
values, so time-series charts step down visibly at the changeover (documented in the
public platform changelog). Sentinel-gated columns follow the standard chain:
`BuildAppInstallSummaryEntity` + `MapToAppInstallSummary` + `AppsDashboardProjection`
(the only projection whose consumers read the new columns).
# Citations
* `src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.AppInstall.cs` — fold + weaker-terminal guard + attempt pairing
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Metrics.cs` — reconcile max-fold, pass-count guard, central duration recompute
* `src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs` — cutover gate
* `src/Backend/AutopilotMonitor.Functions/Helpers/TimeAttributionCalculator.cs` — per-pass segments + occupancy
* `src/Agent/AutopilotMonitor.Agent.V2.Core/SignalAdapters/ImeLogTrackerAdapter.cs` — payload timing re-arm
* Tests: `AppInstallTerminalStateAndReconcileTests`, `EventTimestampValidationTests`, `AppVersionRegressionRadarTests`, `TimeAttributionCalculatorTests`
+216
View File
@@ -0,0 +1,216 @@
---
type: Concept
title: Client-Certificate Tenant Binding — Closing the Cross-Tenant Gap
description: Why a chain-valid Intune MDM certificate does not prove the caller belongs to the tenant it claims, which certificate extension carries the Entra tenant id (and which two look like it but do not), and the shadow-then-enforce rollout that measures field coverage before it can lock devices out.
resource: src/Backend/AutopilotMonitor.Functions/Security/CertTenantBinding.cs
tags:
- backend
- security
- authentication
- certificates
- intune
- multi-tenant
timestamp: 2026-08-24T00:00:00+02:00
---
# Problem
Agent requests authenticate with mTLS. `CertificateValidator` pins the chain to the embedded
Intune roots (`X509ChainTrustMode.CustomRootTrust`), checks the validity window and requires the
Client-Authentication EKU. All of that together proves exactly one thing: **the certificate was
issued by the Microsoft Intune MDM Device CA to some tenant**.
Those roots are shared by every Intune tenant on the planet. Nothing in the certificate stage ties
the caller to the tenant in the request. The tenant scoping came entirely from the downstream
device validators — the serial number must appear in that tenant's Autopilot / imported-device /
Cloud PC inventory. Device serial numbers are not secrets; they are printed on the chassis.
So the pre-existing attack was: enroll a device in your own Intune tenant, obtain a legitimate
device certificate, learn one serial number of the victim tenant, and submit telemetry as that
tenant. Certificate validation passes, device validation passes.
# Mechanism
The Intune MDM Device CA stamps the customer's identifiers into the leaf certificate. Since the
agent presents this certificate in the TLS handshake, its contents are proof of possession and the
backend can compare them against the requested tenant — no Graph call, no additional consent, and
nothing the agent has to send.
## Which extension carries what
Verified by decoding a real field certificate (issued 2026-05-08, `Functions.Tests/device-cert-sample.pem`):
| OID | Encoding | Content |
| --- | --- | --- |
| `1.2.840.113556.5.14` | nested OCTET STRING (`04 10` + 16 bytes) | **Entra tenant id** — the one to compare |
| `1.2.840.113556.5.6` | nested OCTET STRING (`04 10` + 16 bytes) | Intune **Account** id — a *different* GUID, never equal to the tenant id |
| `1.2.840.113556.5.4` | **16 raw bytes, no wrapper** | Intune device id, identical to the certificate's Subject CN |
Two traps are worth stating explicitly, because both cost real debugging time:
1. `…5.6` is widely described as "the tenant id (AccountID)". It is not the Entra tenant id.
Comparing it against the requested tenant would reject every request in production.
2. The encodings are not uniform. `…5.4` holds the GUID bare while the other two nest it in a
second OCTET STRING, so a decoder written against one shape silently fails on the other.
`MsDeviceCertificateOids.TryParseGuid` accepts both, and all GUIDs are Microsoft
little-endian (what `new Guid(byte[])` expects).
A 16-byte input is inherently ambiguous — it is indistinguishable from a truncated wrapper and is
always read as a bare GUID. That is harmless here: the value is only ever compared for equality
against a known tenant id, never trusted as a standalone claim.
## Where it is NOT
`EntraDeviceCertHelper` reads tenant and device ids from the **MS-Organization-Access** certificate
(`1.2.840.113556.1.5.284.*`). That certificate is not part of the authentication path at all — the
agent selects the Intune MDM certificate by exact issuer match (`CN=Microsoft Intune MDM Device CA`,
see `CertificateHelper.SelectMdmCertificate`) and the backend pins only the MDM chain. The Entra
certificate is used purely as a local fallback for `TenantIdResolver`; it never leaves the device.
Sending it in a header instead would prove nothing: only the certificate used in the TLS handshake
demonstrates possession of a private key. A certificate copied into a custom header can be replayed
by anyone who has seen it once.
# Rollout
Enforcing immediately would be reckless: any device population whose certificates predate the
extension, or whose CA behaves differently (sovereign clouds), would be locked out at once. The
rollout therefore has staged consequences.
**Stage 1 — shadow (current).** `SecurityValidator.ObserveCertTenantBinding` evaluates the binding
after certificate validation and logs the outcome. It returns `void`, changes no state, and swallows
its own exceptions, so it cannot fail a request that would otherwise be accepted. The pure comparison
lives in `CertTenantBinding.Evaluate`; the outcome codes (`Match`, `Mismatch`, `ExtensionMissing`,
`Unparseable`, `RequestTenantNotAGuid`) are stable and queried by exact match in KQL.
`WouldRejectUnderEnforcement` records what stage 2 *would* have done, so the telemetry can be read
without re-deriving the rule at query time.
The telemetry uses two carriers, because the numerator and the denominator have very different
costs and answer different questions:
* **Every** outcome, `Match` included, is stamped onto the request row as the `CertTenantBinding`
dimension — `SecurityValidator` writes it into `FunctionContext.Items`
(`CertTenantBinding.RequestItemKey`) and `RequestTelemetryMiddleware` copies it onto the
`RequestTelemetry` it already emits. That row exists per request and is unsampled, so the
denominator costs no additional telemetry at all.
* Outcomes that need an enforcement decision (mismatch, missing, undecodable) are **additionally**
logged at Warning with the full context — thumbprint, certificate tenant, session.
`Match` is deliberately not a trace line. Worker-side `LogInformation` never reaches App Insights
(the ApplicationInsightsLoggerProvider default rule is Warning+ and host-forwarding of worker logs
is disabled), so a `Match` trace is dead code that silently costs the denominator. The first
deployment of this feature made exactly that mistake: it emitted zero rows across 32k requests,
which is indistinguishable from the code never running.
**Stage 2 — enforce.** Once telemetry shows the `ExtensionMissing` rate is negligible, `Mismatch`
becomes a rejection. The decision that still has to be made is the policy for certificates without
the extension: skip with a warning, or reject. Grep marker for every stage-2 site:
`CERT-TENANT-BINDING-SHADOW`.
The bootstrap-token path (`X-Bootstrap-Token`) bypasses certificate validation by design and is
unaffected by either stage.
**Stage 3 — device binding (Global-Admin-only preview).** Tenant binding proves the certificate was
issued to this tenant; it does not prove the certificate belongs to a device the tenant still has.
`IntuneDeviceBindingValidator` closes that by resolving the Intune managedDevice id from the
certificate's Subject CN against the tenant's own `deviceManagement/managedDevices` inventory
(point-GET on `managedDevices/{id}` — no OData filter to inject into, and a 404 is unambiguous).
It is modelled on `CloudPcDeviceValidator` and shares its cache/retry/transient contract, with one
important difference in how it is wired: the lookup is fire-and-forget, like the DevPrep
association shadow, because a cold Graph round-trip can take seconds and must never sit in the
agent's request path. That rules out the request-row dimension stage 1 uses — the row is written
the moment the function returns, before the detached task finishes — so the outcome is logged at
Warning instead. Volume stays bounded because only Global Admins can see the toggle
(`user?.isGlobalAdmin === true`, the same gate as the DevPrep preview). Widening it beyond preview
should move to an inline call plus a request dimension.
Outcomes: `Match`, `NotFound`, `NoDeviceIdInCert`, `PermissionMissing`, `Transient`. Only
`Transient` is retried and left uncached — a missing grant or a foreign device is a state, not an
outage.
Graph cost is not a concern, and that is measured rather than assumed: the first live enrollment
produced **136 requests and exactly one Graph call**, because the 30-minute positive cache absorbs
the rest. For comparison, the per-request Autopilot serial validator made 25 calls in the same
window. Lengthening the positive TTL would buy nothing and would delay noticing a device that has
been wiped or removed; shortening the 5-minute negative TTL matters more, since that is how quickly
a device object appearing late is picked up.
What the 136 requests *did* cost was log lines saying the same thing 136 times, so the shadow log is
gated on `IntuneDeviceBindingResult.ServedFromCache`: one line per real Graph lookup. Cached repeats
add nothing — the reported age just counts up mechanically from the same `enrolledDateTime`.
Outcomes that are never cached (`Transient`, `NoDeviceIdInCert`) still log every time, which is the
right behaviour for an anomaly.
The open question this preview exists to answer is a race, not an attack: a device object can in
principle appear in Intune *after* the agent's first call, and enforcing before that is understood
would reject legitimate enrollments. Every result therefore carries `enrolledDateTime`, and the log
line records `enrolledAgeSeconds` — a device whose object was created seconds before the request
looks completely different from a certificate that never belonged to the tenant. Grep marker:
`CERT-DEVICE-BINDING-SHADOW`.
Enabling it needs the optional Graph permission `DeviceManagementManagedDevices.Read.All`, granted
per tenant with the customer script — it is not in the default-consent set:
```powershell
.\Grant-AutopilotMonitorAddOn.ps1 -ClientId "<client-id>" -Features IntuneDeviceBinding
```
`GraphFeatureCatalog` (feature `IntuneDeviceBinding`) and the script's catalog, `ValidateSet` and
help block are pinned to each other by `GraphFeatureCatalogSyncTests`.
# Examples
Coverage — the question stage 2 depends on ("would enforcing lock anyone out?"):
```kusto
requests
| where isnotempty(customDimensions.CertTenantBinding)
| summarize Requests = sum(itemCount)
by Outcome = tostring(customDimensions.CertTenantBinding),
Tenant = tostring(customDimensions.TenantId)
| order by Requests desc
```
Detail on the individually actionable ones:
```kusto
traces
| where message has "AgentCertTenantBinding"
| extend Outcome = tostring(customDimensions.Outcome),
Tenant = tostring(customDimensions.TenantId)
| summarize count() by Outcome, Tenant
```
Anything in `Mismatch` is a cross-tenant certificate; anything in `ExtensionMissing` is a device
that stage-2 enforcement would have to make a policy decision about. Note that the trace query
alone can never show `Match`, so it cannot tell "everything matched" apart from "nothing ran" —
use the request-row query for that.
Stage 3, including the enrollment-race measurement:
```kusto
traces
| where message has "AgentCertDeviceBinding"
| extend Outcome = tostring(customDimensions.Outcome),
AgeSecs = toint(customDimensions.EnrolledAgeSeconds)
| summarize Requests = count(),
MinAgeSecs = min(AgeSecs),
P05AgeSecs = percentile(AgeSecs, 5)
by Outcome
```
`NotFound` is the row that decides whether this can ever become a gate. A small `MinAgeSecs` on
`Match` means device objects are appearing only just in time — the race is real and enforcement
would need a grace path.
# Citations
* `src/Shared/AutopilotMonitor.Shared/Security/MsDeviceCertificateOids.cs` — OID catalog and the
shared ASN.1 GUID decoder (one implementation for agent and backend).
* `src/Backend/AutopilotMonitor.Functions/Security/CertificateValidator.cs` — extension extraction
alongside chain validation, cached per thumbprint.
* `src/Backend/AutopilotMonitor.Functions/Security/CertTenantBinding.cs` — the pure comparison.
* `src/Backend/AutopilotMonitor.Functions/Security/SecurityValidator.cs` — shadow observation.
* [W365 Cloud PC Device Validation](cloudpc-device-validation.md) — the other consumer of this
certificate's identity fields (Subject CN = Intune device id).
+2 -1
View File
@@ -78,7 +78,8 @@ Durations shown for attempts are the sessions' authoritative `DurationSeconds` v
completed journeys, first-time-right count, attempt histogram, junk-serial exclusion
count, `JourneyVersion`. A journey buckets on the StartedAt date of its completing
success session (the platform's StartedAt-date convention; late terminals converge via
the sweep). All counts are **additive**, so a window rate is the ratio of summed daily
the sweep, which slices the tick's shared projected window scan — see
`time-attribution.md`). All counts are **additive**, so a window rate is the ratio of summed daily
rows — there is no rolling-window row (unlike the median-based time-attribution
aggregates, which cannot merge). 180-day retention. Rows exist below the ≥20 UI gate
on purpose (the UI needs the n for "insufficient data (n=…)").
@@ -0,0 +1,93 @@
---
type: concept
title: Plan Downgrade Policy & the Retention Grace Period
description: What happens when a tenant loses Pro (explicit downgrade or trial expiry) — every entitlement gates read-time and reversibly, except retention, which gets a 30-day grace before the Community cap starts hard-deleting; the anchor model, the ops events, and the deliberate non-grace of everything else.
resource: backend
tags: [plan, edition, downgrade, retention, grace-period, trial, entitlements]
timestamp: 2026-08-22
---
# Schema
## The downgrade model: gate, don't destroy
Edition is resolved at READ time (`FeatureEntitlementCatalog.ResolveEdition`): Pro ⇔
permanent tier `pro`/legacy `enterprise` OR an active trial. A plan mutation is a single
field write (`PATCH config/{tenantId}/plan`, GA-only) — there is no downgrade event
handler, no cascade. On Pro → Community every entitlement simply gates on the next read
(≤ ~5 min config cache):
* **Unrestricted Mode** — effective value flips false; the GA on-request flag and the
tenant toggle survive untouched (inert). Re-upgrade restores it verbatim.
* **Delegated (MSP) admin** — scope suppressed at the home-tenant gate
(`DelegatedAdminService.ApplyHomeTenantGateAsync`); grant rows / tenant groups /
assignments are never touched. Resurrects on re-upgrade.
* **OOBE bootstrap** — endpoints 403/404 unless the additive GA per-tenant flag
(`BootstrapTokenEnabled`) is set; `BootstrapSessions` rows are kept.
* **Rate limits / MCP quota** — pure computed-at-read floors/plan names.
**Retention is the one exception**: the retention sweep (12-hourly
`SessionDeletionMaintenance` → `SessionRetentionFanoutService`) hard-deletes sessions
older than `GetEffectiveRetentionDays` = `min(stored DataRetentionDays, edition cap)`.
Without protection, a downgrade (or trial expiry) with stored retention 365 would start
irreversibly cascade-deleting the 90–365-day band on the next tick.
## The retention grace period (30 days)
`FeatureEntitlementCatalog.RetentionDowngradeGraceDays = 30`. For 30 days after losing
Pro, `GetEffectiveRetentionDays` keeps clamping to the **Pro** cap (365); only after the
grace does the Community cap (90) bite. The stored `DataRetentionDays` is never rewritten.
`days <= 0` (GA-only "infinite") short-circuits before any cap — those tenants are never
swept, grace or not.
**Anchor** — "when was Pro lost" (`TenantEntitlementService.GetRetentionGraceEndUtc`,
returns null while effectively Pro; grace end = latest anchor + 30d):
* **Explicit downgrade**: `TenantConfiguration.ProDowngradedUtc` (backend-only field, no
agent ConfigVersion impact), maintained exclusively by the plan endpoint's pure mutation
core `PlanManagementFunction.ApplyPlanChanges`. Stamped when the EFFECTIVE edition
(before vs. after, same `nowUtc`) transitions Pro → Community — this covers both a
planTier downgrade and a GA explicitly ending a trial (`trialExpiresUtc: null`, where
the trial timestamp disappears and could not anchor). Cleared on any transition back to
effectively Pro (PATCH pro, trial grant, self-service trial start), so a re-upgrade also
resets the grace clock.
* **Trial expiry**: no write happens — the stored `TrialExpiresUtc` itself is the anchor
(read-time, like edition resolution). A planTier downgrade under a still-active trial
deliberately does NOT stamp: the edition is still Pro, and the later trial expiry
anchors the grace then.
The field is on the patch-endpoint `BaseDeniedFields` and `RevertProtectedFields` lists
(a config revert must not time-travel the anchor) and roundtrips the table-serialization
contract (Store+Map + roundtrip tests). Legacy rows read null → no anchor → no grace,
which is correct: no tenant had been downgraded before the feature existed.
## Ops visibility (dual-registered event types)
* `TenantPlanDowngraded` (Warning) — emitted by the PATCH handler on an effective
Pro→Community transition; carries `retentionGraceEndsUtc` + stored retention days. The
response body also returns `retentionGraceEndsUtc`.
* `TenantRetentionGraceExpiring` / `TenantRetentionGraceEnded` (Warning) — emitted by the
daily `TrialExpirySweepFunction` (03:30 UTC) with the same stateless window mechanics as
the trial events (≤3-day heads-up re-fired daily; ended once via the 24h look-back), and
ONLY when data is actually at risk: stored `DataRetentionDays` above the Community cap
(0 = infinite ⇒ never). Re-upgraded tenants go silent automatically (grace resolves null).
All three are registered in the web `OPS_EVENT_TYPES` catalog (OpsAlertRulesSection.tsx)
so Telegram alert rules can route them.
## Deliberate non-goals
* No grace for Unrestricted/MSP/bootstrap/limits — reversible gates need none.
* The retention WRITE cap (`TenantConfigValidation`) does not honor the grace: during the
grace a tenant admin cannot SET a value above the Community cap. The grace protects
existing data; new writes follow the new plan.
* The feature-flags endpoint keeps publishing the plain edition cap
(`retentionCapDays`) — it feeds the settings input maximum, which is the write cap.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Services/TenantEntitlementService.cs` — grace + effective retention
* `src/Backend/AutopilotMonitor.Functions/Functions/Config/PlanManagementFunction.cs` — `ApplyPlanChanges` anchor lifecycle
* `src/Backend/AutopilotMonitor.Functions/Functions/Maintenance/TrialExpirySweepFunction.cs` — grace event windows
* `src/Backend/AutopilotMonitor.Functions/Services/Deletion/SessionRetentionFanoutService.cs` — enforcement point
* Tests: `TenantEntitlementServiceTests`, `PlanManagementTransitionTests`, `TrialExpirySweepFunctionTests`, `SessionRetentionFanoutServiceTests`, `TenantConfigTableSerializationTests`
@@ -0,0 +1,84 @@
---
type: concept
title: Send-Time Frame Separation (X-Send-Time-Utc)
description: One device-clock send timestamp per upload batch splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset — turning clock diagnostics from median statistics into direct per-request measurement.
resource: src/Backend/AutopilotMonitor.Functions/Functions/Ingest/IngestTelemetryFunction.cs
tags: [backend, agent, ingest, timestamps, clock-skew, telemetry, provenance]
timestamp: 2026-08-21
---
# Problem
Events carry two time anchors: `OccurredUtc` (device clock at emission) and `ReceivedAt`
(server clock at ingest). Their difference conflates two physically distinct phenomena:
* **Spool delay** — the event waited offline or in retry backoff before upload. Expected,
harmless, minutes to hours.
* **Device-vs-server clock offset** — the actual diagnostic signal.
A uniform spool backlog (events emitted close together, uploaded after an outage) is
indistinguishable from a clock offset in that delta; a draining backlog fakes a clock jump
ramp. Every consumer had to filter this statistically (spread caps, plateau guards,
persistence tests) — and the first field clock-jump measurement failed exactly on this
ambiguity.
# Schema
The agent stamps one header per upload attempt (`BackendTelemetryUploader`):
```
X-Send-Time-Utc: <DateTime.UtcNow, ISO-8601 round-trip>
```
Per **attempt**, not per batch content: a retry after backoff is a new send moment. The
ingest function parses it (`ParseSendTimeHeader`: round-trip parse, UTC-normalized, values
before year 2000 rejected as garbage — deliberately **no upper clamp**, a future-dated
send time IS the clock-error measurement), and `StampServerFields` writes it as `SentAt`
onto every event of the request — server-stamped, request-level, never trusted per-event
from the wire. Stored as the nullable `SentAt` column on Events (absent for pre-header
agents), mapped back everywhere events are read, exposed as `sentAt` in the `fields=`
projection and verbatim on the raw endpoints.
The separation is then exact:
```
SentAt − OccurredUtc = pure spool delay (both device frame; offset cancels)
ReceivedAt − SentAt = network latency + clock offset (the only frame crossing)
```
Network latency is seconds; anything beyond is device clock error, per request, no
statistics required.
# Consumers
* **`clock_skew` analyze condition** (ANALYZE-DEV-008): batches whose events carry
`SentAt` measure the batch offset directly as `SentAt − ReceivedAt` — spool-immune by
construction, so the spread cap and the IME/clamped exclusions are unnecessary there
(no event timestamps are read at all; IME-only sessions become measurable). Batches
without `SentAt` keep the legacy per-event median + spread-cap path. Downstream logic
(stability check, plateau step detection, end-state persistence) is shared; evidence
carries `sentAtBatchCount`.
* **`CmTraceSkewTripwire` deliberately unchanged**: `median(Δ_IME) − median(Δ_other)`
cancels the device clock offset as common mode — send time adds nothing to that
differential measurement.
# Constraints
* `SentAt` comes from the same device wall clock as the event timestamps — a clock jump
between emission and send lands in the spool-delay term. That is the intended place for
it to become visible, not a defect.
* Signals and DecisionTransitions do not carry `SentAt` (no consumer); the header is
request-level, so extending them later is a storage-only change.
* Fleet coverage grows with agent rollout; all analyses must treat `SentAt == null`
(legacy rows, old agents) as "legacy path", never as zero.
# Citations
* `BackendTelemetryUploader.UploadBatchAsync` — header stamping per attempt.
* `IngestTelemetryFunction.ParseSendTimeHeader` + `IngestTelemetryFunctionTests` — parse
contract incl. the no-upper-clamp decision.
* `EventIngestProcessor.StampServerFields` + `IngestCriticalPathTests` — server-side
stamping, forged per-event values overridden.
* `RuleEngine.ConditionEvaluators.EvaluateClockSkewCondition` + `RuleEngineClockSkewTests`
— the two measurement paths (the backlog-alias test pair pins why SentAt exists).
* `tasks/todo.md` P14 (2026-08-21) — decision trail incl. why the tripwire stays as is.
@@ -0,0 +1,110 @@
---
type: Concept
title: Silent-Session Classification (Sweep, Max-Lifetime, Retro-Reconcile)
description: How a session whose agent went silent gets its honest status — the two-stage maintenance sweep (2h Stalled marker, 5h timeout classification), the EnrollmentTimeoutClassifier rule order incl. the WhiteGlove Part-2 and self-deploying gates, which ingest paths share the classifier, and the admin retro-reconcile modes.
resource: src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs
tags:
- backend
- maintenance
- session-status
- classification
- whiteglove
- self-deploying
timestamp: 2026-08-23T18:00:00+02:00
---
# Silent-Session Classification
A session only ever becomes terminal through evidence. When the agent stops reporting
(power-off, reboot without relaunch, Wi-Fi never re-associating, max-lifetime watchdog),
the backend must decide the status from what it last saw — it never hard-fails a session
for being silent. One pure function holds that decision,
`EnrollmentTimeoutClassifier.ClassifyTimedOutSession`, and every silent-session path calls it
so a session gets the same verdict regardless of which path noticed the silence first.
# Schema
## Inputs (`EspProvisioningRollup` + session flags)
`ExtractRollup(events)` distills the event stream into facts: Device/Account Setup
subcategory counts (incl. the 30s fallback-confirmed messages), `HasExplicitFailure`,
`HasEnrollmentComplete` vs. `HasTerminalComplete` (the Part-1 `whiteglove_complete`),
`HasAgentEmergencyBreak`, `DesktopArrived`, `HelloResolved`, `HelloPolicyDisabled`,
`SkipUserEsp`, RealmJoin detection/resolution. Session-row flags are passed alongside:
`IsPreProvisioned` + `ResumedAt` (WhiteGlove Part 2) and `IsSelfDeployingProfile`
(registry-confirmed `CloudAssignedOobeConfig 0x20|0x40`, set at registration).
## Rule order (first match wins)
| # | Condition | Verdict |
| --- | --- | --- |
| 1 | `HasExplicitFailure` | Failed |
| 1b | `IsWhiteGloveAwaitingUser`: pre-provisioned + resumed + Device Setup done + **no** user evidence since resume + no explicit failure/enrollment_complete | AwaitingUser within grace; Succeeded (honest WG reason) past grace |
| 1c | `IsSelfDeployingProvisioned`: self-deploying profile + Device Setup done + no explicit failure | **Succeeded** (reconcile) — no grace, no AwaitingUser: the profile has no user phase |
| 2 | Account Setup all-succeeded, or `enrollment_complete`, or `whiteglove_complete` on a **non-resumed** session | Succeeded |
| 3 | `HasAgentEmergencyBreak` | Incomplete immediately (agent self-destructed, nothing can arrive) |
| 4 | `DesktopArrived` + (Hello terminal, or Hello disabled + SkipUserEsp) | Succeeded (user provably finished) |
| 5 | Device Setup done | AwaitingUser within grace, Incomplete past grace |
| 6 | otherwise | Incomplete |
Grace = `ResolveGraceHours(SessionGraceHours, AbsoluteMaxSessionHours)` (~51h default),
measured from `ResumedAt ?? StartedAt`. Every Succeeded-reconcile reason carries the
silence-transparency clause (`AppendReconcileTiming`: last agent contact, silence, verdict time).
## Why rule 1c exists (kiosk tenant audit 2026-08-23)
On a self-deploying profile Device ESP all-succeeded IS the enrollment's end; the agent's
own SelfDeploying terminal only adds a 5-min confirmation window
(`DecisionEngine.SelfDeploying.s_deviceOnlyEspDetectionWindow`). Tenant `aebdce78` (school,
Wi-Fi ThinkBooks, a language-change app forcing a reboot right after Device ESP) lost that
window on 836 of ~970 sessions in one week: the agent died 0–4 min after
`DeviceSetup fallback_confirmed`, the reboot relaunch never registered (30s registration
budget on a Wi-Fi link still associating), and rule 5 labeled finished kiosks
"awaiting user / Account Setup 0/5" → Incomplete 51h later. Nobody signs in on these devices
(prepared during the holidays, boxed, in service) — AwaitingUser was factually wrong and
Incomplete turned a working rollout into a red wall. Other self-deploying tenants completed
normally in the same window, so this is a classification gap, not a profile-wide agent bug.
Agent-side companion fix: `SessionRegistrationHelper` now waits up to 15s for a network link
and retries 6× (~62s backoff) instead of 5× (~30s).
## Callers (all pass both gate inputs)
| Path | File | Notes |
| --- | --- | --- |
| Sweep stage 1 (2h agent-silent marker) | `MaintenanceService.MarkStalledSessionsAsTimedOutAsync` | Default: InProgress → Stalled. Gates run BEFORE the Stalled write: `TryMarkWhiteGloveAwaitingUserAsync` (AwaitingUser) and `TryReconcileSelfDeployingAsync` (Succeeded). Only flagged sessions pay the event read. |
| Sweep stage 2 (SessionTimeoutHours, 5h default) | same | InProgress/Stalled/AwaitingUser past the window and past the 2h silence cutoff → full classification; AwaitingUser within grace is skipped without an event read. |
| Agent max-lifetime (`enrollment_failed`/max_lifetime, `agent_shutting_down`/max_lifetime) | `EventIngestProcessor.ApplyMaxLifetimeVerdictAsync` | Watchdog = "agent stopped waiting", not a verdict; classified from evidence. Pending sessions exempt. |
| Late-telemetry reconcile | `EventIngestProcessor.TryLateTelemetryReconcileAsync` | Heal-only: applies a Succeeded verdict to Incomplete/AwaitingUser rows when straggler events arrive. |
| Agent stall probe (`session_stalled`) | `EventIngestProcessor.TryMapStallToWhiteGloveAwaitingUserAsync` | Only the WG gate (1b); a stall probe on a self-deploying device with Device Setup done means the agent is alive and something else blocks — stays Stalled. |
| Admin retro-reconcile | `LegacyReclassificationService` | Modes `legacy_timeouts`, `pending_orphans`, `self_deploying_silent` (Incomplete/AwaitingUser/Stalled rows with the profile flag, heal-only to Succeeded, admin-marked rows skipped). `POST /api/maintenance/reclassify-legacy?mode=…&dryRun=false&tenantId=…&maxSessions=…`, dry-run by default. |
## Known gap (decided, not built)
After max-lifetime the agent writes `enrollment-complete.marker` and self-destructs, so an
AwaitingUser verdict from that path can never heal — it is a 51h countdown to Incomplete.
The emergency-break path (rule 3) already terminalizes immediately; max-lifetime still goes
through the grace window by design ("user simply absent"). Revisit if customers read
AwaitingUser as "still possible".
# Examples
* Session `195593e2` (tenant aebdce78): `DeviceSetup 4/4` + `fallback_confirmed`, `AccountSetup 0/5`
(IME false positive), Hello disabled, SkipUser=True, silent after 13:01 → rule 1c →
Succeeded "self-deploying profile — Device Setup fully provisioned … agent went silent
before confirming completion. Agent last reported 2026-08-21 13:01 UTC …".
* Same event shape on a user-driven profile → rule 5 → AwaitingUser (within grace) /
Incomplete (past grace). Guarded by
`Classify_same_stream_without_self_deploying_flag_keeps_user_phase_verdicts`.
* fairstone WG Part 2 reseal-reboot, technician powers off at logon screen → rule 1b →
AwaitingUser; user unboxes days later → ingest completion heals to Succeeded.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs` — rollup + rule order, `IsWhiteGloveAwaitingUser`, `IsSelfDeployingProvisioned`, reason helpers.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs` — `MarkStalledSessionsAsTimedOutAsync`, `TryMarkWhiteGloveAwaitingUserAsync`, `TryReconcileSelfDeployingAsync`.
* `src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs` — max-lifetime verdict, late-telemetry reconcile, stall mapping.
* `src/Backend/AutopilotMonitor.Functions/Services/LegacyReclassificationService.cs`, `Functions/Admin/ReclassifyLegacySessionsFunction.cs` — retro modes.
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Maintenance.cs` — `GetAgentSilentSessionsAsync`, `GetStalledSessionsAsync`, `GetSelfDeployingSilentSessionsAsync`.
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Security/SessionRegistrationHelper.cs` — link wait + 6-attempt registration budget.
* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.SelfDeploying.cs` — the agent-side 5-min SelfDeploying terminal window.
* `src/Backend/AutopilotMonitor.Functions.Tests/EnrollmentTimeoutClassifierTests.cs` — rule coverage incl. the 2026-08-23 self-deploying block.
+67
View File
@@ -0,0 +1,67 @@
---
type: Concept
title: Table Schema Sentinel (startup table initialization)
description: Why a cold start no longer issues one CreateTableIfNotExists per registered table - a hash over TableNames.All stored in AdminConfiguration gates the full pass, the SessionsIndex backfill only runs after a full pass and behind a conditional-insert claim, and daily maintenance repairs out-of-band deletions.
resource: /src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.cs
tags:
- backend
- table-storage
- startup
- maintenance
timestamp: 2026-08-23
---
# Schema
## Problem
`TableInitializerService` (hosted service) called `InitializeTablesAsync` on every process start, which issued `CreateTableIfNotExistsAsync` for every entry of `Constants.TableNames.All` (66 at the time of writing, DoP 8). Each call is a Create POST whose 409 is swallowed - on Flex Consumption with frequent cold starts that was ~9 serial round-trip waves before the first request. It was followed by a SessionsIndex emptiness query whose *error* path returned "empty", so a transient storage error triggered a full Sessions scan with serial upserts - on every scaled-out instance at once.
## Sentinel
* Row `AdminConfiguration` / PK `SchemaSentinel` / RK `tables`, property `TableSchemaHash` (+ `TableCount`, `UpdatedUtc`).
* Value = `ComputeTableSchemaHash(TableNames.All)`: SHA-256 over the ordinal-sorted, newline-joined table names. Derived, never hand-maintained: adding a table to `TableNames.All` changes the hash by construction (`TableLifecycleBucketTests` already forces every constant into `All`).
* `InitializeTablesAsync` (returns `bool fullPassRan`):
1. Point-read the sentinel. Match -> mark initialized, **zero** table calls, return `false`.
2. Missing / stale / unreadable (fresh storage has no AdminConfiguration table yet) -> `EnsureAllTablesAsync` (the old full pass), then upsert the sentinel **only if no table failed**, return `true`. A failed pass leaves `_tablesInitialized = false`, so the next call retries.
* Scale-out: two instances that both see a stale sentinel both run the idempotent pass - harmless.
## SessionsIndex backfill
* Only evaluated when the full pass ran (`fullPassRan == true`): an empty index can only exist on fresh storage, which is exactly the case without a sentinel.
* `IsSessionIndexEmptyAsync` now returns `false` on error (skip), not `true`.
* `TryClaimSessionIndexBackfillAsync`: conditional `AddEntityAsync` on `SchemaSentinel` / `sessionIndexBackfill` - only the winner scans; 409 or any error -> skip. The manual maintenance backfill (`RunManualAsync`) remains the safety net.
## Known gap and repair
A table deleted out-of-band (cleanup skill, portal, tests) is not noticed at startup anymore. `IStorageInitializer.EnsureAllAsync` runs as the first step of the daily timer maintenance (`MaintenanceService.RunAllAsync`), recreating it within 24h. Storage helpers are fail-soft in the meantime.
## Measuring it
`StartupTelemetryService` (hosted service) emits two App Insights metrics on `IHostApplicationLifetime.ApplicationStarted` - i.e. after every hosted service finished `StartAsync`:
* `BackendStartupMs` - process start (`Process.StartTime`) to host ready; includes runtime, DI and all hosted services.
* `BackendTableInitMs` - the `InitializeTablesAsync` slice alone (`TableStorageService.LastInitialization`).
Dimensions: `version`, `commit`, `tableInitFullPass`, `instance`. Metrics bypass adaptive sampling and the worker log level, so they arrive even though Information traces do not. The MCP server logs the same idea as one `{"type":"boot","readyMs":…}` stderr line per start (ContainerAppConsoleLogs_CL, see `.claude/commands/mcp-telemetry.md`).
```kusto
customMetrics
| where name in ("BackendStartupMs", "BackendTableInitMs")
| summarize p50 = percentile(value, 50), p95 = percentile(value, 95), n = count()
by name, version = tostring(customDimensions.version), fullPass = tostring(customDimensions.tableInitFullPass)
```
# Examples
* Deploy that adds a table: first cold start after the deploy runs the full pass once (log: "Table schema sentinel missing or stale"), every later start logs "Table schema sentinel matches (...)".
* Fresh storage account: sentinel read throws TableNotFound -> full pass -> index empty -> one instance claims and backfills.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.cs` - sentinel, `InitializeTablesAsync`, `EnsureAllTablesAsync`, `TryClaimSessionIndexBackfillAsync`
* `src/Backend/AutopilotMonitor.Functions/Services/TableInitializerService.cs` - startup gating
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs` - daily `EnsureAllTablesAsync`
* `src/Backend/AutopilotMonitor.Functions/Services/StartupTelemetryService.cs` - `BackendStartupMs` / `BackendTableInitMs`
* `src/McpServer/autopilot-monitor-mcp/src/index.ts` - `{"type":"boot"}` line
* `src/Backend/AutopilotMonitor.Functions.Tests/TableSchemaSentinelTests.cs`, `StartupTelemetryServiceTests.cs`
+89
View File
@@ -0,0 +1,89 @@
---
type: Concept
title: Telemetry Ingest Shaping (App Insights / Log Analytics)
description: What the backend deliberately keeps out of Application Insights and why - one canonical request record (worker copy, per-item sampling bypass; host HTTP duplicate dropped by a workspace transformation), expected storage outcomes (404/412/409) and successful SignalR REST calls filtered in the worker, and the query rules that follow.
resource: /src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs
tags:
- backend
- telemetry
- application-insights
- cost
timestamp: 2026-08-23
---
# Telemetry Ingest Shaping
The backend is storage-I/O heavy and runs on the Functions isolated worker, which emits telemetry from TWO processes (host and worker). Left alone, Log Analytics ingestion is dominated by rows nobody reads. This document records what is removed, where, and what that means for queries.
# Schema
## Request telemetry — one canonical record
| Source | Emitted by | Content | Fate |
|---|---|---|---|
| Host HTTP request (`Properties.Source` empty, `Url` set) | Functions host process | Name, duration, status — no tenant/user/correlation | **Dropped at ingestion** by the workspace transformation on `AppRequests` (`infra/appinsights-workspace-transforms.bicep`) |
| Host non-HTTP invocation (`Url` empty) | Functions host process | Timer / queue executions | Kept (the worker middleware never sees them) |
| Worker request (`Properties.Source == 'WorkerMiddleware'`) | `RequestTelemetryMiddleware` | TenantId, UserId, UserRole, CorrelationId, ClientSource, McpToolName, HttpPath | **Kept, unsampled** — the canonical record |
Why two mechanisms:
* The host request item cannot be suppressed from worker code or from `host.json` (`Host.Results` controls log traces, not the request item; verified 2026-06-09). The only lever is a Log Analytics workspace transformation: `source | where tostring(Properties.Source) == 'WorkerMiddleware' or isempty(Url)`.
* `host.json` `samplingSettings.excludedTypes: "Request;Event"` only applies to the host process. The worker's `AddApplicationInsightsTelemetryWorkerService` runs its own adaptive sampling; before 2026-08-23 worker request rows carried `ItemCount` 2–5 on weekdays (≈30 % of individual requests missing, `count()` undercounting by the same share — `sum(ItemCount)` matched the host copy to 1.00 per day). Re-configuring the worker sampling chain did not take effect in the isolated worker (see the L4 note in `Program.cs`). The per-item opt-out does: the middleware sets `SamplingPercentage = 100` on its own `RequestTelemetry`, and the SDK sampling processor passes through every item whose sampling percentage is already set.
## Dependency telemetry — keep only signal
`StorageDependencyFilterProcessor` (worker pipeline) drops:
* successful Azure Storage calls (Table/Queue/Blob) in both shapes — the HTTP shape (`Azure table`, target `*.table.core.windows.net`) and the Azure SDK ActivitySource shape (`InProc | Microsoft.Tables`, `InProc | Microsoft.Storage`);
* storage calls with an **expected outcome**: `404` (point-read miss — EventTypeIndex, BlockedDevices lookups), `412` (ETag precondition), `409` (idempotent insert conflict). HTTP-shape rows carry the code in `ResultCode`; InProc rows only in `Properties["Error"]` (`Status: 404 (`). Before this filter these were 100 % of the remaining "failed" storage rows (~540 MB/week, two rows per call);
* successful Azure SignalR Service REST calls the worker makes itself (`*.service.signalr.net`).
The bulk of SignalR REST traffic (group add/remove per connection, ~470 MB/week) is emitted by the **host** process for the SignalR output binding (`SDKVersion rdddsc:*`) and never passes the worker pipeline — live-verified after the 2026-08-23 deploy (worker rows: 0, host rows: ~560/h). Those are removed by the second data flow of the same workspace transformation: `AppDependencies | where not(Success == true and Target endswith '.service.signalr.net')`.
Everything else passes: storage 429/5xx/auth/timeouts, failed SignalR calls, all HTTP/Graph/NVD/MSRC dependencies, the worker's own `InProc Invoke` span, and every non-dependency item.
## Measured baseline (7 days before the change, `_BilledSize`, `autopilotmonitor-api-eu`)
| Table | Rows | MB | After |
|---|---|---|---|
| AppDependencies | 976k | 1248 | ≈ 240 (SignalR + expected-outcome rows gone) |
| AppRequests host copy | 274k | 323 | ≈ 0 (only timer/queue rows) |
| AppRequests worker copy | 189k | 208 | ≈ 300 (now 100 % of requests) |
| AppTraces | 142k | 137 | unchanged |
| AppEvents | 6k | 4 | unchanged — not worth touching |
Net ≈ −1.3 GB/week with MORE usable data (every request now has tenant context).
# Examples
Counting requests (exact since 2026-08-23; for older data replace `count()` with `sum(itemCount)`):
```kql
requests
| where customDimensions.Source == 'WorkerMiddleware'
| summarize total=count(), failed=countif(success == false) by name
```
Real storage trouble only:
```kql
dependencies
| where type startswith 'InProc | Microsoft.Tables' or target endswith '.table.core.windows.net'
| summarize count() by resultCode, tostring(customDimensions.Error)
```
Post-deploy verification of the per-item sampling bypass (must be `ItemCount == 1` only):
```kql
AppRequests
| where TimeGenerated > ago(1h) and tostring(Properties.Source) == 'WorkerMiddleware'
| summarize rows=count() by ItemCount
```
# Citations
* `src/Backend/AutopilotMonitor.Functions/Middleware/RequestTelemetryMiddleware.cs` — `SamplingPercentage = 100` before `TrackRequest`.
* `src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs` + `StorageDependencyFilterProcessorTests.cs` — the filter contract.
* `src/Backend/AutopilotMonitor.Functions/Program.cs` — the failed L4 worker-sampling attempt and why it must not be retried.
* `infra/appinsights-workspace-transforms.bicep` — the AppRequests transformation DCR, apply/verify/rollback commands.
* `.claude/commands/backend-logs.md`, `session-debug.md` — query conventions.
+46 -1
View File
@@ -4,7 +4,7 @@ title: Tenant Activation & the Once-Only Welcome Mail
description: How a signup becomes an activated tenant (manual approve or auto-approve queue), and how the welcome mail is sent exactly once even though activation and the user-provided address race each other.
resource: backend
tags: [activation, auto-approve, welcome-email, preview-whitelist, race]
timestamp: 2026-08-02
timestamp: 2026-08-23
---
# Schema
@@ -58,6 +58,46 @@ The Global-Admin resend endpoint (`POST /api/preview/send-welcome-email/{tenantI
always sends (explicit intent) and consumes the marker best-effort so the automatic
paths never duplicate afterwards.
## Mail transport
Both transactional mails (welcome, post-offboarding farewell) go through the
provider-neutral `EmailService` (`IEmailService` for the welcome path,
`IOffboardFarewellEmailSender` for the farewell path). The provider is an implementation
detail confined to one private transport method plus the `Email:*` configuration
section — a provider swap must not rename classes, DI registrations or callers.
* Current provider: **Mailchimp Transactional (Mandrill)**, `POST {Email:Endpoint}`
(`messages/send`), API key in the JSON body, typed `HttpClient` with the shared
`ResiliencePolicies.Notification` retry policy (15 s timeout).
* Settings (App Settings use `Email__Key`): `Email:ApiKey` (required — empty means every
send is a logged no-op, never an error), `Email:Endpoint`, `Email:FromAddress`
(default `noreply@autopilotmonitor.com`), `Email:FromName` (default `Autopilot Monitor`).
* Result handling: per-recipient status `sent`/`queued`/`scheduled` = success;
`rejected`/`invalid` (with `reject_reason`), non-2xx or malformed bodies = warning.
Never throws — both callers rely on that fail-soft contract.
* `track_opens`/`track_clicks` are explicitly `false` and `auto_text` is `true`: the provider
receives only the recipient address and the tenant domain, which is exactly the claim on
the privacy page and the trust data-flows page. Changing tracking = changing those pages.
## Template overrides (operator-editable)
Global Admins can replace either HTML body without a deployment (portal: Settings →
Email Templates). `EmailTemplateService` resolves the body for every send: the stored
override wins, otherwise the built-in `EmailTemplates.*`. Overrides live in the
`PreviewConfig` table (partition `EmailTemplates`, row key `welcome` | `farewell`,
properties `Html`, `UpdatedBy`, `UpdatedUtc`), cached 5 minutes, invalidated on save/reset.
* The only placeholder is `{{domainName}}`; an empty domain renders as "your organization",
matching the built-ins. Subjects are not editable (pinned constants).
* Limit 30 000 characters (Table string property cap is 32K UTF-16 chars).
* The send path is fail-soft: a storage error falls back to the built-in body. The admin
read path is not — a broken store surfaces as an error instead of silently showing the
built-in.
* Endpoints (all GlobalAdminOnly): `GET|PUT|DELETE /api/global/email-templates/{kind}` and
`POST /api/global/email-templates/{kind}/test`. The test send goes to the caller's tenant
contact address (activation notification email → `ContactEmail` → caller UPN) with the
caller's tenant domain, optionally with an unsaved draft body; provider tag `test`.
# Examples
* Auto-approve wins the race: approve at T+64 s finds no address → logs "deferred to the
@@ -73,3 +113,8 @@ paths never duplicate afterwards.
* `src/Backend/AutopilotMonitor.Functions/Functions/Rules/PreviewWhitelistFunction.cs` — save-path trigger, GA resend
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableConfigRepository.cs` — conditional inserts (approved / welcome-email-sent)
* `src/Backend/AutopilotMonitor.Functions.Tests/TenantApprovalServiceTests.cs` — ordering + dedup pins
* `src/Backend/AutopilotMonitor.Functions/Services/EmailService.cs` — provider-neutral transport (Mandrill wire format, result handling, tracking off)
* `src/Backend/AutopilotMonitor.Functions.Tests/EmailServiceTests.cs` — wire-format, fail-soft and gate-ordering pins
* `src/Backend/AutopilotMonitor.Functions/Services/EmailTemplateService.cs` — override resolution, placeholder rendering, cache
* `src/Backend/AutopilotMonitor.Functions/Functions/Admin/EmailTemplatesFunction.cs` — GA endpoints (read / override / reset / test send)
* `src/Web/autopilot-monitor-web/app/admin/settings/sections/SectionEmailTemplates.tsx` — portal editor, preview frame, test send
+5
View File
@@ -105,6 +105,11 @@ the partition).
* Self-healing 30-day maintenance sweep (2h tick): backfills missing breakdowns, folds
late-terminating sessions into their StartedAt-date aggregates, and enforces version
purity — a stale-version row whose events aged out counts as missing, never mixed.
Reads the tick's **shared window scan** (`MaintenanceService.LoadSweepWindowSessionsAsync`,
one projected 35-day cross-tenant drain via `MaintenanceSweepSessionProjection` that
also feeds the device-journey and verdict-calibration sweeps) — the StartedAt-only
filter is a full-table read in Table Storage, so it is done once per tick, not once per
sweep; `MaintenanceSweepProjectionEquivalenceTests` pins the column set.
* `TimeAttributionAggregates` (PK = TenantId or `global`, RK = `{yyyy-MM-dd}|{class}` and
`rolling30|{class}`): per enrollment class (`user_driven`, `whiteglove`, `self_deploying`,
`device_preparation` — never mixed) median/p75/p90 per segment over the fixed six-segment
+164
View File
@@ -0,0 +1,164 @@
---
type: Concept
title: Verdict Calibration (classifier thermometer)
description: How every session status write declares its VerdictPath, how overriding a verdict preserves it as PriorStatus/PriorVerdictPath, how the daily calibration aggregate counts paths with the re-enrollment proxy and the correction stream, and how the drift radar (share / silence / evidence-gap) turns that into operator-only ops events.
resource: src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.VerdictCalibration.cs
tags:
- backend
- sessions
- classifier
- calibration
- regression-radar
timestamp: 2026-08-23T18:00:00+02:00
---
# Verdict Calibration (classifier thermometer)
A session's terminal status is written by ~20 code paths: the agent's own terminal events,
the silence classifier (`EnrollmentTimeoutClassifier`, rules 1–6) reached via the
maintenance sweep / the agent's max-lifetime shutdown / late-telemetry heal / retro
reclassification, the analyze-rule engine, admin marks and session superseding. Verdict
calibration makes every one of those paths **countable**, tracks how often a path's verdict
was later **overridden**, attaches the delayed **re-enrollment proxy**, and raises an
operator-only alarm when a path's share **drifts**. It is deterministic, fact-based tuning
input for the rules — explicitly not a learned model, and never a customer-facing feature.
# Schema
## VerdictPath on the session row
`SessionSummary.VerdictPath` (`VerdictPaths` vocabulary, `origin:detail`, append-only) is
stamped by **every** status write — `UpdateSessionStatusAsync` takes it as a required
parameter, `StoreSessionAsync` stamps `register:new` / `register:whiteglove_resume` and
preserves an existing path through the re-registration Replace, `SetSessionPreProvisionedAsync`
requires it whenever it writes a status. Unlike `FailureSource` (persisted only on `Failed`),
the path survives for Incomplete, AwaitingUser, Stalled and Pending.
| Origin | Paths |
| --- | --- |
| `agent` | `complete`, `complete_soft` (Continue-Anyway), `failed`, `esp_failure_fallback`, `gather_complete`, `whiteglove_pending`, `whiteglove_resumed`, `stall_probe`, `stall_heal` |
| `ingest` | `wg_awaiting` (stall probe mapped to WhiteGlove AwaitingUser) |
| `sweep` | `stalled`, `wg_awaiting`, `sd_reconcile`, `r{rule}` (stage-2 classifier) |
| `maxlife` / `late` / `retro` | `r{rule}` (classifier via max-lifetime shutdown / late-telemetry heal / retro reclassification); `retro:sd_reconcile`, `retro:superseded` |
| `register` | `new`, `whiteglove_resume`, `superseded` |
| `rule` | `rule:{ruleId}` (analyze rule with MarkSessionAsFailed) |
| `manual` | `failed`, `succeeded` |
| `legacy` | read-side derivation only (`legacy:{rule}` when the rule is recoverable from the reason literal, `legacy:unknown` otherwise) — never written |
Classifier rule ids (`ClassifierRules`): `r1`, `r1b_awaiting`, `r1b_succeeded`, `r1c`, `r2`,
`r3`, `r4`, `r5_awaiting`, `r5_incomplete`, `r6`. `ClassifyTimedOutSession` returns
`(Status, Reason, Rule)` so each `return` names its rule.
## PriorStatus / PriorVerdictPath — the correction stream
When a write replaces a prior **verdict** (existing status ∈ Succeeded/Failed/Incomplete/
AwaitingUser with a stamped path, incoming status different), the storage seam preserves the
pair (`TableStorageService.ComputePriorVerdict`, both the ETag path and the force path). A
stall marker healing to InProgress, a same-status refresh, or an unstamped legacy row never
sets it. Admin marks, late agent completions and retro reclassifications all flow through it.
## Read-side derivation for pre-instrumentation rows
`VerdictPathDerivation.Derive(session)` returns the stamped path verbatim, otherwise derives
from `AdminMarkedAction`, `FailureSource`, `ReconcileReason` and `FailureReason` using the
writers' fixed literals (classifier rule prefixes, the `max-lifetime watchdog` / `Retro-
reclassified` suffixes, `Rule: `, `Superseded by session`, `Late completion report received`).
Unambiguous rows get the real path (`agent:complete`, `manual:failed`, `rule:…`,
`maxlife:r6`), ambiguous-but-rule-known rows `legacy:{rule}`, the rest `legacy:unknown`. No
mass backfill: the aggregate counts derived rows with `DerivedCount` so the matrix shows
history while flagging its weaker evidence. Tests pin the literals against the classifier
output so a wording change fails loudly.
## Daily aggregate — `VerdictCalibrationAggregates`
PK = `TenantId` | `"global"`, RK = `yyyy-MM-dd` (session **StartedAt** date, like every daily
aggregate). `VerdictCalibrationDailyAggregate { Version, SessionCount, TerminalSessionCount,
Buckets[], ComputedAt }` with one bucket per (VerdictPath, Status):
| Field | Meaning |
| --- | --- |
| `Count` / `DerivedCount` | sessions currently on the path; of those, derived read-side |
| `Eligible7d` | terminal sessions whose end (`CompletedAt ?? LastEventAt ?? StartedAt`) lies ≥ 7 d before compute time |
| `ReEnrolled7d` | of the eligible: the device's `DeviceHistories` chain shows the next terminal session starting < 7 d after this one's end |
| `OverriddenByAdmin` / `OverriddenByLateCompletion` / `OverriddenOther` | sessions whose **PriorVerdictPath** is this bucket — attributed to the path that was overridden, so a bucket can carry overrides with `Count = 0` |
`MaintenanceService.SweepVerdictCalibrationAsync` (2-hourly, after the device-journey sweep
so the chains are fresh; manual maintenance parity; kill switch
`VerdictCalibrationSweepDisabled`) slices the tick's shared projected window scan
(`LoadSweepWindowSessionsAsync` — one 35-day cross-tenant drain feeds the time-attribution,
device-journey and calibration sweeps plus the radar's tenant discovery; the StartedAt-only
filter is a full-table read, so never per sweep), recomputes the rolling 30-day window whole (Replace),
deletes stale date rows it did not regenerate, and the unbounded-tables cleanup applies the
180-day retention. Sessions inside a deletion cascade are skipped; the table is regenerable
and is wiped with the tenant on offboarding.
## Read surfaces
* `GET global/metrics/verdict-calibration?days=&tenantId=` (GlobalReadOrAdmin,
TenantScoping.QueryParam; `VerdictCalibrationResponse.Build`): per path — count, share of
all window sessions, derived count, eligible/re-enrolled with `reEnrollRatePct` **null below
20 eligible**, the three override counters, a today-anchored 7d-vs-28d share trend
(`window7`, `baseline28`, `lift` null without baseline) — plus totals, the trend basis and
the active `alerts[]`. Empty but well-formed before the first sweep.
* MCP `get_verdict_calibration` (platform scope: GA + Global Reader). Same endpoint, LLM-shaped in
`verdict-calibration-shape.ts`: per-row `window7.sessions` / `baseline28.sessions` are dropped (the
denominators live once in `trend`), `reEnrollRatePct` / `lift` are always present (`null` = withheld),
and `minSharePct` / `top` trim the one-session tail — rows carrying overrides are never trimmed and
everything dropped is reported in `omitted`. The wire shape the admin page consumes is untouched.
* Portal: Global Admin → Metrics → Verdict Calibration (`/admin/metrics/verdict-calibration`):
matrix grouped by origin, status pill, `derived n` gray pill, trend glyph (arrow only from
5 window hits and lift ≥ 2 / ≤ 0.5), re-enrollment rate or `— (n=…)`, overrides
`admin / late / other`, `↑ Drift` pill on rows with an active episode, and the active
episodes list.
## Drift radar — `VerdictCalibrationRadar`
Same statistics as the [rule regression radar](rule-regression-radar.md) (7-day window vs the
prior 28 days, ≥ 5 window hits, ≥ 20 window sessions, lift ≥ 2, `MetricsMath.
RateIncreaseSeparated`), anchored on yesterday, over every partition with rows (each tenant
and `global`):
| Kind | Signal | Re-arm |
| --- | --- | --- |
| `share_regression` | one **backend-decided** path's share of all sessions doubled (`sweep`/`maxlife`/`late`/`retro`/`rule`/`manual`/`ingest`; `agent:*` and `register:*` mirror customer workflow mix — first prod pass fired on `agent:whiteglove_pending` rollout weeks — and `legacy:*` is derived, all excluded) | share < 1.5× baseline, the path stops occurring, or the path is no longer eligible |
| `silence_share_regression` | the `sweep:*` + `maxlife:*` share doubled — the backend had to decide more often because the agent went silent (a liveness signal, not a classifier one) | same |
| `evidence_gap` | absolute: `r6` (pure fallthrough) decides ≥ 20 % of the window's classifier verdicts, ≥ 20 verdicts | < 15 % or < 20 verdicts |
Episodes are tracker rows `verdictcalibration|{kind}|{path}|{status}` (register = `AddEntity`,
409 = already burning, fail-closed; refresh carries `FirstNotifiedAt`; 30-day tracker
retention re-arms long burners). A new episode fires **one** `VerdictCalibrationDrift` ops
event (category Maintenance, Warning; dual-registered in the web `OPS_EVENT_TYPES`) — there
is deliberately **no tenant bell**. Per-path findings on a tenant partition get the on-fire
dimension correlation (`RuleRegressionRadar.ComputeDimensionConcentration` over the window's
sessions whose derived path matches), reported with the shared wording contract "correlated,
not necessarily causal"; group kinds and the global partition carry no dimension claim.
Kill switch `VerdictCalibrationRadarDisabled`.
# Examples
* **Reading the thermometer (prod, 2026-08-23 hand analysis that motivated this):**
re-enrollment within 7 days after Succeeded 8.9 % (lab background), after Failed 24.3 %,
after Incomplete 10.6 % — Incomplete behaves like Succeeded, so rules 5/6 are probably too
cautious; the matrix now shows this per rule instead of per status.
* **Correction stream:** a session swept to `sweep:r5_incomplete`, later completed by the
agent → row `agent:complete` with `PriorVerdictPath = sweep:r5_incomplete`; the aggregate
increments `OverriddenByLateCompletion` on the `sweep:r5_incomplete / Incomplete` bucket.
Four admin marks since launch were invisible before — now each one counts against the path
it corrected.
* **Evidence gap:** a tenant whose agent build stopped emitting the Device Setup rollup line
drives its sweep verdicts into `r6`; the absolute gate fires at 20 % and names the
tenant — the fix is agent-side, the classifier is only the messenger.
# Citations
* `src/Shared/AutopilotMonitor.Shared/Models/VerdictPaths.cs` — vocabulary + `ClassifierRules`
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Sessions.cs` — `ComputePriorVerdict`, `ApplyVerdictPath`, re-registration preserve
* `src/Backend/AutopilotMonitor.Functions/Helpers/VerdictPathDerivation.cs`
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.VerdictCalibration.cs` — sweep + `BuildVerdictCalibrationAggregates`
* `src/Backend/AutopilotMonitor.Functions/Functions/Metrics/GetVerdictCalibrationFunction.cs`
* `src/Backend/AutopilotMonitor.Functions/Helpers/VerdictCalibrationRadar.cs`, `Services/MaintenanceService.VerdictCalibrationRadar.cs`
* `src/McpServer/autopilot-monitor-mcp/src/tools/admin.ts` — `get_verdict_calibration`
* `src/McpServer/autopilot-monitor-mcp/src/verdict-calibration-shape.ts` — LLM-facing response shaping (`minSharePct` / `top`, explicit nulls)
* `src/Web/autopilot-monitor-web/app/admin/metrics/sections/SectionVerdictCalibration.tsx`
* Tests: `VerdictPathTests`, `VerdictCalibrationTests`, `VerdictCalibrationRadarTests`
* Related: [Silent-Session Classification](silent-session-classification.md), [Device Journeys](device-journeys.md), [Rule Regression Radar](rule-regression-radar.md)
+15 -1
View File
@@ -19,16 +19,21 @@ https://docs.autopilotmonitor.com (separate repository).
# Agent
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [diagnostics package](agent/diagnostics-package.md) (built-in section catalog shared with the portal, RealmJoin/Device Preparation collection gates, caps, manifest), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [Gather Rule Debug Log](agent/gather-rule-debug-log.md) - EnableGatherRuleDebugLog (ConfigVersion 36): the per-tenant local trace file that explains every gather-rule evaluation decision — scope skips, on_change suppression, empty collector results, logparser position/parse-mode outcomes — for rules that produce no timeline events.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
* [ESP Completion Starvation — the sits-d Cloud-PC complex](agent/post-reboot-completion-recovery.md) - Why a SkipUser=true flow (no user ESP page → no Shell-Core 62407, no AccountSetup registry) left the completion gate satisfied but never evaluated — five Cloud PCs idled six hours at "waiting on: hello_resolution" — and the fix: the observed skip stands in for the final-exit evidence in the IME user-session completion attempt. Plus the two adjacent defects fixed alongside: the edge-only user-apps-settled synthesis (now level-triggered) and the never-wired Shell-Core restart replay (now Hello-wizard-only, with the reasoning why a replayed 62407 must never reach the reducer).
* [Zero User-Targeted Apps — the starved user-phase completion gate](agent/zero-user-apps-completion-gate.md) - Why a device with no user-targeted Intune apps could never satisfy the IME user-session gate — IME skips its "Completed user session" line entirely in the zero-app branch — turning the 30-minute completion backstop into `esp_exit_without_completion_evidence` on a third of one tenant's enrollments, with the user at the desktop and RealmJoin still installing. The two fixes: read the line IME does write (IME-USER-SESSION-ZERO-APPS), and never fail a session while a completion gate with its own armed deadline is deliberately holding it.
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
* [Continue-Anyway Observation Mode](agent/continue-anyway-observation.md) - EnableEspContinueAnywayObservation (ConfigVersion 37, operator-set): Device-phase ESP terminal failures on Continue-Anyway profiles get a bounded 60-min observation instead of an immediate hard fail; the DAD-validated real-user desktop + Hello gate detect the true end, the session completes Succeeded + EspSoftFailure (amber "with issues" badge), expiry un-defangs to the original esp_terminal_failure.
* [CMTrace Time Resolution](agent/cmtrace-time-resolution.md) - Converting bias-less CMTrace local times to UTC without trusting any process's timezone belief: per-line self-anchoring on the 15-minute offset grid for provably fresh lines, uniform marked reader-zone fallback for backlog, and the provenance vocabulary (bias / line-anchored / reader-zone-fallback) that keeps every event re-derivable.
* [Registry App-State Second Pillar](agent/registry-app-state-pillar.md) - IME's authoritative per-app state read straight from the registry (Win32Apps EnforcementStateMessage, EspTrackingWin32Apps, StatusServiceReports) via a snapshot-and-diff watcher; registry/log reconciliation doubles as the built-in IME log-pattern drift alarm.
* [WDP Scenario Gates & Completion Backstop](agent/wdp-scenario-gates.md) - How the agent gates Autopilot/ESP-only machinery off on Windows Device Preparation sessions (deterministic ExecutionContext marker, fail-toward-Classic), which noise it suppresses (ZTD probe, esp_config unknown, ModernDeployment 100/417/418/1005, starved sweep), and the WDP completion model: High-confidence seed, desktop-based arm D, and the 30-min device_prep_completion backstop.
* [Power-State Watcher](agent/power-state-watcher.md) - Live AC↔battery transitions and 50/30/15% downward battery-threshold crossings during enrollment via a WMI Win32_PowerManagementEvent push subscription (zero polling, never armed on battery-less devices): the power_state_change event (transition field, per-level lifetime latch, 5s debounce, emission cap) and the analyze rules ANALYZE-DEV-009 (battery <15%, high) / ANALYZE-DEV-010 (AC→battery, warning) with on_event interim triggers.
* [System Timeline Watcher](agent/system-timeline-watcher.md) - Ground truth for two timeline blind spots from one System-channel EventLogWatcher with pre-agent backfill: system_clock_changed (Kernel-General 1 — oldTime/newTime/signed timeDeltaMs/reason/setting process; 2 s micro-slew floor, ≥5 min Warning+immediate) and system_sleep_episode (Power-Troubleshooter 1 sleep/hibernate + Kernel-Power 507 Modern Standby; authoritative enteredAt/exitedAt payload instants, 60 s Modern-Standby floor). Consumed by the TimeJumpBadge clock-set cause, the SessionInfoCard standby note, time-attribution SleepSpans (AttributionVersion 3) and ANALYZE-DEV-012.
* [DO GroupId Auto-Set](agent/do-group-id-auto-set.md) - EnableDoGroupIdAutoSet (ConfigVersion 40): the agent seeds the Delivery Optimization DOGroupId policy with a deterministic network-fingerprint GUID (default gateway IPv4 + SendARP MAC, RealmJoin-compatible byte layout) so same-network devices form one DO peering group during ESP; ownership-tracked via the StartupEventGate fingerprint, foreign DOGroupId/DOGroupIdSource policies always win, no random-GUID fallback.
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait is keyed to the Intune-registered "Sidecar" provider by NAME: co-management's "ConfigMgr" parks the user ESP at "Apps (Identifying)" for days even with TrackingPoliciesCreated=1 (issue #106); the always-on two-arm esp_policy_provider_stalled tripwire (provider_incomplete | sidecar_provider_missing) and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
# Rules
@@ -36,16 +41,24 @@ https://docs.autopilotmonitor.com (separate repository).
* [Gather Rule Phase Scoping, One-Shot Triggers & Emit-on-Change](rules/gather-rule-phase-scoping.md) - Restricting gather rules to enrollment phases (activePhases / activeFromPhase sticky latch), one-shot collection at a phase's start/end (phase_change / phase_exit), and emitMode "on_change" result-dedup.
* [Gather Rule Guardrails](rules/gather-rule-guardrails.md) - Why the allowlists in rules/guardrails.json are enforced on the agent and nowhere else, which hard blocks survive unrestricted mode, and the rule that every collector must call a guard and emit security_warning on refusal.
* [Rule Authoring Surface](rules/rule-authoring-surface.md) - How customer AIs author rules against the real contract: the baked MCP resources (guide/schemas/guardrails via combine.js), the MCP-local validate_rule pre-flight, and the side-effect-free POST rules/analyze/dryrun whose trace twin shares the production evaluators (parity + side-effect-freedom pinned by tests).
* [Rule ID Namespace & Collision Guards](rules/rule-id-namespace.md) - Why rule identity is the human-readable ruleId (no GUIDs) with tenant isolation via Table Storage partitioning, the reserved built-in namespace (RuleIdPolicy), which collisions are possible (custom-vs-custom across tenants: allowed and harmless), and the guards keeping the global RuleStats aggregate catalog-only and the global partition inside the reserved namespace (combine.js, reseed, cross-tenant RuleState GC).
* [Analyze Rule Evaluation Triggers](rules/analyze-rule-triggers.md) - Opt-in `evaluateOn` interim triggers (`whiteglove_sealed`, `on_event:<type>`) so rules can fire before a session is terminal, with RuleResults update-semantics (interim → finalize/resolve, NotifiedAt notification dedupe) replacing the permanent dedupe freeze, KO/stat suppression on interim runs, the ingest-side cached trigger registry, and the authoring lints that keep interim rules monotonic.
# Backend
* [Telemetry Ingest Shaping](backend/telemetry-ingest-shaping.md) - One canonical App Insights request record (worker copy with tenant context, per-item sampling bypass; host HTTP duplicate dropped by a workspace transformation DCR), expected storage outcomes 404/412/409 and successful SignalR REST calls filtered in the worker, and the count()/sum(itemCount) query rules that follow.
* [Table Schema Sentinel](backend/table-schema-sentinel.md) - Startup no longer issues one CreateTableIfNotExists per table: a SHA-256 over TableNames.All in AdminConfiguration gates the full pass (derived, never hand-maintained), the SessionsIndex backfill runs only after a full pass behind a conditional-insert claim, and daily maintenance repairs out-of-band deletions.
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
* [Send-Time Frame Separation](backend/send-time-frame-separation.md) - One device-clock send timestamp per upload batch (X-Send-Time-Utc → SentAt on Events) splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset; the clock_skew condition measures SentAt batches directly (spool-immune), the CMTrace tripwire deliberately stays on its differential math.
* [OOBE Bootstrap go-Code Flow](backend/oobe-bootstrap-go-flow.md) - The bootstrap short-code lifecycle (portal-created session → `irm go.autopilotmonitor.com/CODE | iex` at Shift+F10 → GetBootstrapScriptFunction), the always-HTTP-200 text/plain contract, the two-line injection defense (GUID/URL/expiry validator + single-quoted PS literals, golden-pinned against the legacy TS template), and the per-IP rate-limit story behind Front Door.
* [Silent-Session Classification](backend/silent-session-classification.md) - How a session whose agent went silent gets its honest status: the two-stage maintenance sweep (2h Stalled marker with the WhiteGlove Part-2 AwaitingUser gate and the self-deploying Succeeded gate, 5h timeout classification), the EnrollmentTimeoutClassifier rule order (explicit failure → WG Part 2 → self-deploying → completion evidence → emergency break → desktop+Hello → Device Setup/grace), the ingest paths sharing it (max-lifetime, late-telemetry heal, stall probe) and the admin retro-reconcile modes incl. self_deploying_silent.
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
* [Attempt-Scoped App Install Durations](backend/app-install-duration-attempt.md) - AppInstallSummary.DurationSeconds = status-defining (last) install attempt via the LastAttemptStartedAt max-fold anchor instead of the first-observation→last-terminal span that billed multi-pass IME gaps to trivial apps; InstallPassCount, the weaker-terminal guard, the radar's cutover gate, and time attribution's union-of-attempt-segments occupancy.
* [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract.
* [Verdict Calibration](backend/verdict-calibration.md) - How every session status write declares its VerdictPath (origin:detail vocabulary incl. classifier rule ids), how overriding a verdict preserves PriorStatus/PriorVerdictPath as the correction stream, the daily per-path calibration aggregate with the 7-day re-enrollment proxy and read-side legacy derivation, the GA endpoint / MCP tool / admin matrix, and the drift radar (share, silence-share, r6 evidence gap) with its operator-only ops events.
* [App Version Duration Regression Radar](backend/app-version-regression-radar.md) - How app versions whose median install duration regressed vs. the previous version are detected (first-seen-ordered versions, median lift ≥2× plus a 5-minute absolute floor over measured installs only), how alert episodes live in the appversionregression tracker keyspace with drain/median re-arm semantics, and the bell/ops-event/analytics surfaces.
* [Plan Downgrade Policy & the Retention Grace Period](backend/plan-downgrade-retention-grace.md) - What happens when a tenant loses Pro: every entitlement gates read-time and reversibly ("gate, don't destroy"), except retention — a 30-day grace keeps the Pro cap before the Community cap starts hard-deleting; the ProDowngradedUtc/trial-expiry anchor model, the TenantPlanDowngraded/RetentionGrace* ops events, and the deliberate non-grace of everything else.
* [Tenant Activation & the Once-Only Welcome Mail](backend/tenant-activation-welcome-mail.md) - How a signup becomes an activated tenant (manual approve or the auto-approve queue), and the write-then-read + conditional-marker arbitration that sends the welcome mail exactly once even though activation and the user-typed notification address race each other.
* [W365 Cloud PC Device Validation](backend/cloudpc-device-validation.md) - Why Cloud PCs can never pass the Autopilot serial lookup, and the cert-CN-bound CloudPc fallback validator: Subject CN (= Intune device id) matched against the tenant's Graph virtualEndpoint/cloudPCs inventory — admits exactly the service-provisioned Cloud PCs, nothing else.
* [ServerActions & On-Demand Diagnostics](backend/server-actions-on-demand-diagnostics.md) - The queue-on-session, deliver-on-ingest action channel, the TenantAdminOrOperator tier with in-function per-type re-gating, the portal's Collect Logs flow (quick-config orders rotate_config before request_diagnostics), and the hosted-prefix sweep in the offboarding cascade.
@@ -64,6 +77,7 @@ https://docs.autopilotmonitor.com (separate repository).
# Security & Identity
* [MCP OAuth Flow](mcp-oauth-flow.md) - Who authenticates where when connecting an AI client to the MCP server; two identities, three parties plus Entra ID.
* [Client-Certificate Tenant Binding](backend/client-cert-tenant-binding.md) - Why a chain-valid Intune MDM certificate only proves "issued to some tenant", which extension actually carries the Entra tenant id (`…5.14`) versus the two that look like it (`…5.6` Intune Account id, `…5.4` device id, different encodings), why the Entra join certificate is not part of the auth path, and the shadow-then-enforce rollout with its volume-controlled telemetry.
# Conventions for this bundle
+57
View File
@@ -1,7 +1,64 @@
# Log
## 2026-08-25
* **New**: `agent/zero-user-apps-completion-gate.md` — a Classic enrollment whose user has zero targeted Intune Win32 apps could not produce the evidence any user-phase completion path required: IME's zero-app branch (`ApplicationPoller`, verified against decompiled 1.104.102.0 and three older builds) `continue`s before its `Completed user session` statement, so `ImeUserSessionCompletedUtc` stayed null, all four arms of `ShouldTransitionToAwaitingHello` were dead, and the 30-min `advisory_completion` backstop resolved the session as `esp_exit_without_completion_evidence` — with a DAD-validated desktop, Hello provisioned and RealmJoin still installing. One tenant: 10 of 30 enrollments in a week, all false positives. Two fixes. (1) New pattern `IME-USER-SESSION-ZERO-APPS` reads the line IME *does* write and routes it through the same `OnUserSessionCompleted` callback, guarded to the AccountSetup phase and replayed if the observation predates the phase marker; `AreUserEspAppsSettled` now accepts an empty app list only with that explicit observation. (2) `HandleAdvisoryCompletionDeadlineFired` re-arms instead of failing while the RealmJoin gate is closed and its own deadline is still armed — session 75d6ae8e installed 25 RealmJoin packages inside the window, the last 2 min before the failure. `ime_user_session_completed` gains an `evidence` field (`zero_user_apps_enumerated` / `user_session_completed`); no new event type. The pattern reaches existing agents via the config API (an unknown action is a Debug line), the behaviour needs an agent release.
## 2026-08-24
* **Change**: `DeviceAssociationValidator` / `SecurityValidator` — the DevPrep association shadow now logs one line per real Graph lookup instead of one per request (`DeviceAssociationResult.ServedFromCache`, `AsCacheHit()` copies rather than marking the shared cached instance), the same gate the cert-device-binding shadow uses. Measured: one enrollment produced 263 identical lines off a handful of calls. Status of that preview, for the record: the `/beta/deviceManagement/tenantAssociatedDevices` endpoint answers (6 calls, all HTTP 200, `transient=False`), but every result so far is `matched=False` from a single Autopilot **v1** device — which is the correct answer for a device that was never part of a Device Preparation policy. The matching itself is therefore still unproven, and the endpoint remains undocumented in both v1.0 and beta Graph reference, so it is not a basis for enforcement.
* **Update**: `backend/client-cert-tenant-binding.md` — stage 3 verified live (`ea6a375`, gktatooine): 136 requests, all `Match`, device object ~72 s older than the session start, so no enrollment race in that run. Measured Graph cost: **1 call for 136 requests** (30 min positive cache), versus 25 for the per-request Autopilot validator — the positive TTL needs no change. The shadow log is now gated on `ServedFromCache` (one line per real lookup instead of per request); never-cached outcomes still log every time.
* **Update**: `backend/client-cert-tenant-binding.md` — stage 3, the cert-to-device binding: new `IntuneDeviceBindingValidator` resolves the Subject-CN managedDevice id against the tenant's own `deviceManagement/managedDevices` (point-GET, no filter literal), modelled on `CloudPcDeviceValidator` and wired fire-and-forget like the DevPrep shadow so a cold Graph round-trip never enters the agent request path. New GA-only tenant toggle `ValidateIntuneDeviceBinding` (shadow — never touches `deviceValidated`), new optional Graph feature `IntuneDeviceBinding` → `DeviceManagementManagedDevices.Read.All` in `GraphFeatureCatalog` + the customer grant script, and `enrolledAgeSeconds` on every log line so the enrollment race can be measured before this could ever become a gate. Grep marker: `CERT-DEVICE-BINDING-SHADOW`.
* **Update**: `backend/client-cert-tenant-binding.md` — shadow telemetry moved off the dead `LogInformation` path: the `Match` outcome now rides on the request row as the `CertTenantBinding` dimension (`SecurityValidator` → `FunctionContext.Items` → `RequestTelemetryMiddleware`), which is unsampled and already exists per request, so the denominator costs nothing. Non-Match outcomes keep their Warning trace. Post-deploy verification of 249e39bc found zero rows across 32k requests — worker `LogInformation` never reaches App Insights (provider default rule Warning+), making "everything matched" indistinguishable from "never ran". `CertificateValidationResult.FromCache` removed with the sampling path it existed for.
* **New**: `backend/client-cert-tenant-binding.md` — the agent's mTLS certificate chains to Microsoft's shared Intune roots, so certificate validation alone never proved tenant membership; tenant scoping rested entirely on the (non-secret) device serial in the downstream validators. The Intune MDM leaf carries the Entra tenant id in OID `1.2.840.113556.5.14` (nested OCTET STRING), which is NOT `…5.6` (Intune Account id, a different GUID) and NOT `…5.4` (device id, stored as 16 bare bytes) — all decoded by one shared `MsDeviceCertificateOids.TryParseGuid` now used by agent and backend. Stage 1 ships as shadow only: `CertTenantBinding.Evaluate` (pure) plus `SecurityValidator.ObserveCertTenantBinding` (returns void, swallows its own exceptions, cannot change an authorization outcome), with `Match` sampled via `CertificateValidationResult.FromCache` so the denominator survives without per-request volume. Grep marker for the stage-2 enforcement change: `CERT-TENANT-BINDING-SHADOW`.
## 2026-08-23
* **New**: `backend/telemetry-ingest-shaping.md` — `RequestTelemetryMiddleware` sets `SamplingPercentage=100` on its own item (worker request rows were adaptively sampled, ItemCount 2–5 on weekdays, despite host.json `excludedTypes`), the redundant host HTTP request copy is dropped by a workspace transformation DCR (`infra/appinsights-workspace-transforms.bicep`, timer/queue rows kept), and `StorageDependencyFilterProcessor` now also drops expected storage outcomes (404/412/409, both InProc and HTTP shapes) and successful SignalR REST calls; the host-emitted SignalR binding calls (bulk of the volume, SDK `rdddsc`) are dropped by a second `AppDependencies` flow in the same DCR. Measured ≈ −1.3 GB/week.
* **Update**: `backend/table-schema-sentinel.md` — "Measuring it": `StartupTelemetryService` emits `BackendStartupMs` (process start → ApplicationStarted) and `BackendTableInitMs` as App Insights custom metrics with version/commit/fullPass dimensions; the MCP server writes one `{"type":"boot","readyMs"}` stderr line per start (query in `.claude/commands/mcp-telemetry.md` §A2).
* **New**: `backend/table-schema-sentinel.md` — startup table initialization gated by a schema sentinel (SHA-256 over `TableNames.All` in `AdminConfiguration`/`SchemaSentinel`/`tables`): matching sentinel ⇒ zero `CreateTableIfNotExists` calls on a cold start, missing/stale ⇒ one full pass + sentinel rewrite. SessionsIndex startup backfill only after a full pass, `IsSessionIndexEmptyAsync` no longer treats errors as "empty", conditional-insert claim so one scaled-out instance backfills. Daily maintenance runs `IStorageInitializer.EnsureAllAsync` to repair out-of-band table deletions.
* **Update**: `backend/verdict-calibration.md` — MCP `get_verdict_calibration` response shaping: per-row trend denominators dropped, explicit nulls, `minSharePct` / `top` with an `omitted` report.
* **Update**: `backend/verdict-calibration.md` — share-regression radar restricted to backend-decided paths; agent-declared/registration paths (customer workflow mix) and derived legacy paths no longer alert, stale episodes re-arm themselves.
* **Update**: `backend/time-attribution.md`, `backend/device-journeys.md`, `backend/verdict-calibration.md` — the rolling maintenance sweeps (time attribution, device journeys, verdict calibration, calibration-radar discovery) now share ONE projected 35-day cross-tenant Sessions scan per tick (`LoadSweepWindowSessionsAsync` + `MaintenanceSweepSessionProjection`, pinned by `MaintenanceSweepProjectionEquivalenceTests`) instead of four full-table drains; each sweep slices its own window by StartedAt.
* **New**: `backend/verdict-calibration.md` — the classifier thermometer: every status write now stamps `VerdictPath` (required parameter on `UpdateSessionStatusAsync`; `EnrollmentTimeoutClassifier` returns its rule id), overriding a verdict preserves `PriorStatus`/`PriorVerdictPath`, `VerdictPathDerivation` attributes pre-instrumentation rows read-side (`legacy:*` when ambiguous). New daily aggregate `VerdictCalibrationAggregates` (per-path count/derived/eligible7d/reEnrolled7d/overridden-by-admin|late|other, sweep after the device-journey pass, 180d retention), `GET global/metrics/verdict-calibration`, MCP `get_verdict_calibration`, admin metrics section, and the `VerdictCalibrationRadar` (share / silence-share / r6 evidence-gap episodes in the tracker keyspace `verdictcalibration|`, `VerdictCalibrationDrift` ops event, no tenant bell).
* **New**: `backend/silent-session-classification.md` — one place for the silent-session verdict: `EnrollmentTimeoutClassifier` rule order incl. the new rule 1c `IsSelfDeployingProvisioned` (self-deploying profile + Device Setup all-succeeded + no explicit failure ⇒ Succeeded reconcile, no grace/AwaitingUser — kiosk tenant aebdce78 audit: 836 finished devices parked AwaitingUser/Incomplete), the sweep stage-1 gates (`TryMarkWhiteGloveAwaitingUserAsync`, new `TryReconcileSelfDeployingAsync`), the ingest callers (max-lifetime, late-telemetry heal, stall probe) and the new admin retro mode `self_deploying_silent` (`GetSelfDeployingSilentSessionsAsync`). Agent companion: `SessionRegistrationHelper` waits up to 15s for a network link and retries 6× (~62s) instead of 5× (~30s) so the reboot relaunch on Wi-Fi no longer exits 7 before the link is back.
* **New**: `agent/diagnostics-package.md` — the diagnostics ZIP's built-in collection is now ONE catalog in Shared (`DiagnosticsBuiltInSections.All`): the agent iterates it in order and the backend serves the same list to the portal (`GET diagnostics/paths`, MemberRead), so what administrators see is exactly what the agent collects. New conditional sections: RealmJoin client/package/Chocolatey/per-user logs behind the tenant's RealmJoin Watcher toggle (mirrored onto `AgentConfiguration.EnableRealmJoinWatcher` by the merger) and the Device Preparation bootstrapper event log behind the deterministic WDP marker (probe once per build, fail toward Classic); every skip is a manifest line. Built-ins bypass `DiagnosticsPathGuards` by design (reviewed code, `%ProgramData%\RealmJoin` is not allowlisted); caps/reparse/evtx export still apply. `MergeDiagnosticsLogPaths` dedups global ∪ tenant paths. Portal: collapsed read-only catalog + one-line rows in both cards, tenant context dropped the GA-only global fetch.
* **Update**: `rules/gather-rule-guardrails.md` — `DiagnosticsPathGuards` scope clarified: configured diagnostics paths only, never the built-in catalog.
* **Update**: `backend/tenant-activation-welcome-mail.md` — new "Template overrides" section: Global Admins can replace the welcome/farewell HTML in the portal (Settings → Email Templates); `EmailTemplateService` resolves override-or-built-in per send (PreviewConfig table, partition `EmailTemplates`, 5-min cache, `{{domainName}}` placeholder, 30k limit, fail-soft send path), GA endpoints `global/email-templates/{kind}` (+`/test` to the caller's tenant contact address). Also: the mail header is image-free (CSS-drawn brand mark, gradient stripes and clip-path edge as progressive enhancement).
## 2026-08-22
* **Update**: `backend/tenant-activation-welcome-mail.md` — new "Mail transport" section: transactional mail now goes through the provider-neutral `EmailService` (`IEmailService` + `IOffboardFarewellEmailSender`) with the provider confined to one private transport method and the `Email:*` settings (`ApiKey`, `Endpoint`, `FromAddress`, `FromName`); current provider Mailchimp Transactional (Mandrill) replacing Resend (NuGet dropped, typed `HttpClient` + `ResiliencePolicies.Notification`). Result-array semantics (`sent`/`queued`/`scheduled` ok, `rejected`/`invalid` warn), open/click tracking explicitly off because the privacy/trust claim depends on it.
* **New**: `backend/plan-downgrade-retention-grace.md` — plan downgrade policy: every Pro entitlement gates read-time and reversibly on downgrade ("gate, don't destroy"), except retention, which now gets a 30-day grace (`FeatureEntitlementCatalog.RetentionDowngradeGraceDays`) before the Community cap starts hard-deleting the 90–365-day band. Anchor model: `ProDowngradedUtc` stamped/cleared by the plan endpoint's pure `ApplyPlanChanges` core on EFFECTIVE-edition transitions (covers explicitly ended trials; planTier downgrade under an active trial deliberately doesn't stamp), trial expiry anchors read-time via `TrialExpiresUtc`. New Warning ops events `TenantPlanDowngraded` (PATCH handler) and `TenantRetentionGraceExpiring`/`TenantRetentionGraceEnded` (TrialExpirySweep window mechanics, only when stored retention exceeds the Community cap). Field is patch-denied + revert-protected; retention WRITE cap deliberately ignores the grace.
## 2026-08-21
* **New**: `agent/do-group-id-auto-set.md` — optional DO peering-group seed (`EnableDoGroupIdAutoSet`, default off, ConfigVersion 40): `DoGroupIdService` derives a deterministic GUID from the default gateway's IPv4 + SendARP MAC in RealmJoin's exact byte layout (cross-product peering) and writes `DOGroupId` in the 64-bit GPO Policies store with readback verification. Conservative by design: no random-GUID fallback (skip on missing gateway/MAC), `DOGroupIdSource`/MDM-managed values/foreign GPO values always win (skip reasons in the `do_group_id_auto_set` event), own stale values are recognized via the `written:<guid>` StartupEventGate fingerprint and updated after a gateway change. Gateway heuristic extracted into the shared `NetworkInterfaceLocator` (dedup of NetworkChangeDetector + DeviceInfoCollector copies).
* **New**: `agent/system-timeline-watcher.md` — clock-step and sleep-episode ground truth from the System event log: one `EventLogWatcher` (Kernel-General 1, Power-Troubleshooter 1, Kernel-Power 507; 506 deliberately unconsumed) emits `system_clock_changed` (oldTime/newTime/signed timeDeltaMs/reason/process; 2 s micro-slew floor, ≥5 min = Warning+immediate) and `system_sleep_episode` (kind sleep/hibernate/modern_standby, authoritative `enteredAt`/`exitedAt` payload instants, 60 s Modern-Standby floor). WindowsUpdateTracker-style backfill (24 h default, `system-timeline-watermark.json`) recovers pre-agent records; ConfigVersion 39 (`Collectors.EnableSystemTimelineWatcher`). Consumers: TimeJumpBadge cause `clock-set` (ground truth beats inference), SessionInfoCard standby note next to the unchanged wall-clock duration, `SleepSpan`s in time attribution (AttributionVersion 3, RebootSpan clone), and ANALYZE-DEV-012 (info, single, ≥5 min episode; deliberately the only new rule — clock issues stay with DEV-008).
* **New**: `agent/power-state-watcher.md` — live AC/battery observation during enrollment: `PowerStateWatcherHost` subscribes to the WMI push `Win32_PowerManagementEvent EventCode 10` (no polling; re-probes `GetSystemPowerStatus` per push, 5 s trailing-edge debounce, never armed on battery-less devices, `Collectors.EnablePowerStateWatcher` kill-switch, ConfigVersion 38). The pure `PowerStateTransitionTracker` emits `power_state_change` with `transition` = ac_to_battery (Warning, immediate) / battery_to_ac (Info) / threshold_crossed at 50/30/15% (each level latched once per lifetime, lowest-crossed-only on jumps, baseline-below-level covered, emission cap 20). Classified non-activity on both sides (SignalActivityClassifier + IsPeriodicOrStallEvent). New analyze rules ANALYZE-DEV-009 (thresholdPercent=15, high) and ANALYZE-DEV-010 (transition=ac_to_battery, warning), both with `on_event:power_state_change` interim triggers and deliberately single-required-condition (event_data conditions have no same-instance join).
* **Update**: `rules/rule-id-namespace.md` — the export→paste round-trip is no longer lossy: the JSON editors now map rule-shaped exports into the form (`gatherRuleToForm` unflattens the nested gather `parameters` object, analyze `jsonToForm` maps `evaluateOn` onto the eval checkboxes; `tags` and the effective `markSessionAsFailed` carry into the created rule). Both editors still accept the serialized-form shape; edit-mode pastes merge, with an explicit `parameters`/`evaluateOn` winning over stale flat fields.
* **New**: `rules/rule-id-namespace.md` — rule identity model: human-readable `ruleId` strings (no GUIDs), tenant isolation via Table Storage partitioning (`global` = built-in/community, tenantId = custom), the reserved built-in namespace from `RuleIdPolicy`, and the four collision guards: catalog-only global `RuleStats` aggregate (same-ID custom rules of unrelated tenants no longer sum into one `global_{ruleId}` row), combine.js duplicate/namespace fail, GitHub-reseed rejection of non-reserved IDs into the global partition, and the namespace gate on the cross-tenant RuleState GC.
* **New**: `backend/app-install-duration-attempt.md` — attempt-scoped app install durations: `DurationSeconds` = status-defining (last) attempt anchored on the new max-folded `LastAttemptStartedAt` (StartedAt stays the earliest-wins window/version-ordering key; legacy rows without the anchor keep the span fallback); `InstallPassCount` with replay-guarded additive fold; weaker-terminal guard (completed-with-Skipped never overrides Installed/Error, in-batch and cross-batch); attempt-scoped `DownloadDurationSeconds` pairing; the version-regression radar's both-sides cutover gate; time attribution counts the union of per-pass active segments (occupancy merges segments, hulls stay for display/what-if); agent payload timing re-arms both endpoints on terminal→active. No backfill — pre-cutover rows keep span semantics.
* **Update**: `agent/decision-engine.md` — activity-based RealmJoin timeout extension (report 55e6afd61c9d: a large RJ package catalog outlived the 60-min window mid-install — the timer arms at RJ-agent-MSI detection, so the first deployment had effectively ~44 min). When the `RealmJoinTimeout` deadline fires with `LastDeploymentPhase` 100/101 and deployment activity (phase change or package observation, persisted in `RealmJoinFacts.LastActivityUtc`; the detection-time phase capture deliberately excluded) inside the last 60 min, the deadline re-arms to `lastActivity + 60 min` (Info event `realmjoin_timeout_extended`), clamped at detection + 4 h. `realmjoin_timeout` now carries `reason` = `hard_timeout` / `inactivity` / `absolute_cap`; a fire of a superseded deadline incarnation dead-ends. Idle detection and phase-200/210 sessions keep the original 60-min behavior.
* **New**: `backend/send-time-frame-separation.md` — P14: the agent stamps one device-clock send timestamp per upload attempt (`X-Send-Time-Utc`), the ingest function server-stamps it as `SentAt` on every event of the request. `SentAt − OccurredUtc` is pure spool delay, `ReceivedAt − SentAt` is network + device-vs-server clock offset — the two were indistinguishable before and had to be separated statistically. The `clock_skew` condition measures SentAt-carrying batches directly (spool-immune, no spread cap, IME-only sessions become measurable; evidence carries `sentAtBatchCount`), legacy batches keep the median path; the CMTrace tripwire deliberately stays on its differential math (common-mode clock offset cancels there).
## 2026-08-20
* **Update**: `agent/cmtrace-time-resolution.md` — new "Backend regression tripwire" section: the P6 split. Operator side: `CmTraceSkewTripwire` piggybacks on the terminal counter reconcile's partition scan (`Δ = ReceivedAt − OccurredUtc` per source, 15-min grid + 2-min residual, ≥20 samples / ≥3 distinct batches per side, bias-dominated sessions suppressed, `measuredWriterOffsetMinutes` never consulted) and emits the `CmTraceTimeSkewRegression` ops event — goal state: never fires; kill switch `CmTraceSkewTripwireDisabled`. Customer side: new `clock_skew` analyze condition source (`skewMetric: clock_jump | sustained_offset`, threshold in seconds, batch-median frames with spool-spread filter, plateau step detection, end-state persistence, IME-derived events excluded) behind the new built-in `ANALYZE-DEV-008` (300 s = the Kerberos/Entra 5-minute skew boundary) — the during-enrollment complement of ANALYZE-DEV-007's one-shot NTP check. P5 (agent-side observability event) is subsumed by the tripwire.
* **New**: `agent/cmtrace-time-resolution.md` — the era-aware answer to the CMTrace timezone skew: a bias-less line read in a provably fresh pass anchors its own UTC offset against the agent clock (15-minute grid, 2-minute residual), so interleaved writer eras in one file resolve line by line; everything not provably fresh keeps the uniform reader-zone fallback and says so via `sourceOffsetOrigin`. Freshness is a per-file `LastCheckedUtc` stamped on every poll look (empty first sights included) with a 30-second window, never restored across restarts. Replaces the reverted per-file offset application (04b1a7c6); the per-file measurement stays observational, now with the anchor timestamp logged as a tripwire for the still-unexplained e9753578 label/anchor crossing.
* **Update**: `agent/post-reboot-completion-recovery.md` — retitled to "ESP Completion Starvation — the sits-d Cloud-PC complex" and reframed on the evidence: the reboot was a red herring. The five stuck Cloud PCs carried `SkipUserStatusPage=True` (read 10+ times across three agent processes, `True` every time) and Windows honoured it — no user ESP page, hence no Shell-Core 62407 and no AccountSetup provisioning registry, verified over a six-hour uninterrupted post-reboot run with the watcher armed. Arm B of `ShouldTransitionToAwaitingHello` was open from the bootstrap onward, but every evaluation site hangs off a signal a skip-user flow cannot produce, and the `AdvisoryCompletion` backstop arms exclusively from an `EspExiting` signal — an open gate nobody knocks on. Fix: `HandleImeUserSessionCompletedV1` accepts the observed skip in place of the final-exit evidence (`skipUserExitEquivalent`; anchor + genuine IME completion + real-user desktop stay mandatory, WDP excluded). The knock deliberately lives on the IME edge, not desktop arrival — ordering-robust against Fix 10's AccountSetup bounce-back and semantically "user-phase work finished" rather than mid-install. `ClassicSkipUserEspUserSessionCompletionTests` replays the real 8110e262 ordering end to end. Docs and comments no longer claim an ESP-exit recovery that does not exist (stray duplicate summary in `ShellCoreTracker`, `ImeLogHost` restore-callback comment); the residual `SkipUser=false` exit-in-downtime gap is named honestly and surfaced by `ANALYZE-ESP-005` via `session_stalled` instead of being patched with an unorderable replay.
## 2026-08-19
+1 -1
View File
@@ -98,7 +98,7 @@ Interim evaluation changes what a rule's conditions mean mid-flight. Three new l
# Application to the Motivating Cases
- WG battery rule (sits-d): `evaluateOn: ["enrollment_end", "whiteglove_sealed"]` — technician gets the channel alert at the bench; Part-2 terminal pass finalizes or resolves the finding.
- WG battery rule (sits-d): shipped 2026-08-21 as ANALYZE-DEV-009 (battery below 15% while on battery) and ANALYZE-DEV-010 (AC→battery switch), both `evaluateOn: ["enrollment_end", "whiteglove_sealed", "on_event:power_state_change"]` on the live power watcher's `power_state_change` events (see `../agent/power-state-watcher.md`) — technician gets the channel alert at the bench; the terminal pass finalizes the finding. Both are deliberately single-required-condition rules: `event_data` conditions scan events independently (no same-instance join), so DEV-009 matches the agent-stamped `thresholdPercent` and DEV-010 matches `transition`.
- Hybrid-affinity stuck (esa): ANALYZE-ID-004 gets `evaluateOn: ["enrollment_end", "on_event:hybrid_login_pending"]` — the second overdue-login event (threshold 65) produces a live finding on the still-InProgress session. ANALYZE-APP-015 stays `enrollment_end`-only until its conditions are hardened (lint 1).
- Stuck after a forced mid-ESP reboot (sits-d Cloud PCs, 2026-08-19): `on_event:session_stalled` is the established trigger for the never-terminal stuck session. The agent emits `session_stalled` exactly once per session (60-minute stall probe), so the trigger is rare by construction and needs no throttle. Five Cloud PCs sat at `InProgress` with **zero findings** for hours — a mid-ESP reboot had killed the agent in `Stage=EspAccountSetup` — before the max-lifetime watchdog eventually terminalised them as `Incomplete`. ANALYZE-ESP-005 carries the trigger and fires on that stream at confidence 95 (verified against session `8110e262`): both its conditions (`mdm_policy_reboot_required.firstRebootUri` exists, `system_reboot_detected` count ≥ 1) are **monotonic** and it has no terminal preconditions, so lint 1 does not apply. ANALYZE-APP-016 is interim-safe for the same reason (`app_install_starved` count ≥ 1 only ever grows).
+6
View File
@@ -31,6 +31,12 @@ item in `AutopilotMonitor.Agent.V2.Core.csproj`) and regenerated into the portal
| `IsCommandAllowed` | `allowedCommands` | **exact** match, trimmed — never prefix |
| `IsEventLogChannelAllowed` | `eventLogChannels` | prefix, boundary `/` or end |
`DiagnosticsPathGuards.IsDiagnosticsPathAllowed` (allowlist `diagnosticsPathPrefixes`, a C#
literal mirrored in the JSON) applies the same hard blocks and the same user-profile
exception to **admin-configured diagnostics paths** only. The diagnostics package's built-in
sections are reviewed code and never pass through it — see
[Diagnostics Package](../agent/diagnostics-package.md).
**Hard blocks live in code, not in the JSON**, so a parse error cannot lift them — on a
parse failure the allowlists fall back to empty and everything is blocked (fail-closed),
but the hard blocks must hold regardless of configuration:
+47
View File
@@ -0,0 +1,47 @@
---
type: concept
title: Rule ID Namespace & Collision Guards
description: How gather/analyze rule identity is scoped (tenant partition vs global partition), why there are no GUIDs, which collisions are possible, and the guards that keep the reserved built-in namespace and the global RuleStats aggregate clean.
resource: backend, rules
tags: [rules, ruleid, namespace, collision, rulestats, tenant-isolation]
timestamp: 2026-08-21
---
# Schema
Rule identity is the human-readable `ruleId` string — there are no GUIDs, deliberately: the ID appears in logs, mails, notifications, RuleResults, the agent config payload, and docs, and tenant isolation is provided by Table Storage partitioning instead.
| Store | PartitionKey | RowKey | Content |
|---|---|---|---|
| `GatherRules` / `AnalyzeRules` | `"global"` | ruleId | Built-in + community rules (shared, all tenants) |
| `GatherRules` / `AnalyzeRules` | tenantId | ruleId | Custom rules (visible only to that tenant) |
| `RuleStates` | tenantId | ruleId | Per-tenant enable/notify overrides — built-in/community rules ONLY (custom rules fold state into their own row) |
| `RuleStats` | date | `{tenantId}_{ruleId}` and `global_{ruleId}` | Daily fire/eval telemetry; the global row is catalog-only (see guards) |
Namespace split (`RuleIdPolicy.cs`, shared model):
- **Reserved built-in namespace**: `(ANALYZE|GATHER)-<CATEGORY>-<NUMBER>` with `CATEGORY != CUSTOM`, case-insensitive, including currently unused numbers (gaps = retired rules that may return).
- **Tenant namespace**: everything else — `ANALYZE-CUSTOM-NNN` (portal suggestion), `-CUSTOM` suffixes, org prefixes (`CONTOSO-WIFI-001`), freeform.
# Collision model
- **Custom vs built-in**: impossible via API — `CreateRuleAsync`/`UpdateRuleAsync` reject reserved IDs (409) and check existence against `global` ∪ own tenant partition. Legacy debris is neutralized at merge time: global wins, tenant copy dropped with a warning (pinned by `RuleMergeCollisionGuardTests`, motivated by the 5ca2b350 duplicate-key incident).
- **Custom vs custom across tenants**: allowed and harmless — separate partitions, separate evaluation, separate RuleResults/tenant stats. Two tenants both using `ANALYZE-CUSTOM-101` never interact.
- **JSON copy between tenants** (export → paste into create editor): goes through the same create path, so an ID already taken in the receiving tenant surfaces as 409 — the user renames. There is no bulk import endpoint. The JSON editors accept both shapes — a rule export and the serialized form — via `gatherRuleToForm` / `jsonToForm` (web `types.ts` of each rules page): nested gather `parameters` are unflattened, analyze `evaluateOn` maps onto the eval checkboxes, and `tags` plus the effective `markSessionAsFailed` survive the import.
# Guards (all enforced, 2026-08-21)
1. **Global RuleStats aggregate is catalog-only.** The `global_{ruleId}` row would otherwise sum same-ID custom rules of unrelated tenants (title/severity last-writer-wins). Analyze gates on `rule.IsBuiltIn || rule.IsCommunity` (`AnalyzeOnEnrollmentEndHandler`), gather gates on `RuleIdPolicy.IsReservedBuiltInId` (`EventIngestProcessor.RuleStats`) because agent events carry no flag — valid because custom rules can never occupy the reserved pattern.
2. **Catalog build fails on namespace violations.** `rules/scripts/combine.js` rejects duplicate IDs (case-insensitive) and any gather/analyze built-in outside the reserved pattern; `BuiltInRuleCatalogPolicyTests` pins the same invariants on the embedded resource actually deployed.
3. **GitHub reseed rejects non-reserved IDs** before writing to the `global` partition (`ReseedFromGitHubFunction`) — a seeded `*-CUSTOM-*` rule would silently shadow every tenant's same-ID custom rule at merge time.
4. **Cross-tenant RuleState GC is namespace-gated.** `DeleteRuleStatesForRuleIdAcrossTenantsAsync` deletes by bare RowKey across all partitions; it now refuses non-reserved IDs with `failed=-1` (callers then skip the global catalog delete, same contract as enumeration failure).
Invariant chain: schema pattern + combine.js + reseed guard ⇒ the `global` partition only ever contains reserved-pattern IDs ⇒ the pattern is a safe built-in/custom discriminator everywhere no `IsBuiltIn` flag is available.
# Citations
- `src/Shared/AutopilotMonitor.Shared/Models/Rules/RuleIdPolicy.cs` — reserved pattern + rationale
- `src/Backend/AutopilotMonitor.Functions/Services/AnalyzeRuleService.cs`, `GatherRuleService.cs` — create/update guards, merge collision guard
- `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Rules.cs` — key schemes, cross-tenant GC guard
- `src/Backend/AutopilotMonitor.Functions.Tests/BuiltInRuleCatalogPolicyTests.cs`, `RuleIdReservationTests.cs`, `RuleMergeCollisionGuardTests.cs`
- `rules/scripts/combine.js` — build-time namespace/duplicate guard
@@ -0,0 +1,78 @@
// Autopilot-Monitor — Log Analytics workspace transformation (ingestion-time filter)
//
// Two ingestion-time filters. (2) Successful Azure SignalR REST dependencies emitted by the
// HOST process for the SignalR output binding (SDKVersion rdddsc:*) — the worker-side
// StorageDependencyFilterProcessor never sees those. Failed SignalR calls are kept.
// (1) Drops the redundant HOST-side HTTP request record from AppRequests so that exactly one
// canonical request row remains: the worker-side copy emitted by RequestTelemetryMiddleware
// (Properties.Source == 'WorkerMiddleware', carries TenantId/UserId/CorrelationId and is
// sampling-exempt per item). Non-HTTP host invocations (timer/queue triggers) have an empty
// Url and are kept — the worker middleware never sees them.
//
// Why ingestion-time: the host request item is emitted by the Functions host process and
// cannot be suppressed from worker code or host.json (verified 2026-06-09, see Program.cs).
// Live numbers 2026-08-23 (7d): host copy 274k rows / 323 MB, worker copy 189k rows / 208 MB.
//
// Cost note: transformation filtering is free up to 50% of the table's incoming volume; the
// portion beyond that (here ~11% of AppRequests) is billed at the (cheap) data-processing rate.
//
// Apply (two steps; the link is a CLI update = GET+PUT, which keeps retention/SKU intact):
// az deployment group create -g rg-autopilotmonitor-prd-gwc \
// --template-file infra/appinsights-workspace-transforms.bicep \
// --query properties.outputs.dcrId.value -o tsv
// az monitor log-analytics workspace update -g rg-autopilotmonitor-prd-gwc -n AutopilotMonitor \
// --data-collection-rule <dcrId-from-above>
//
// Verify after ~20 min (ingestion latency):
// AppRequests | where TimeGenerated > ago(15m) | where AppRoleName == 'autopilotmonitor-api-eu'
// | summarize rows=count() by source=tostring(Properties.Source), hasUrl=isnotempty(Url)
// → only (WorkerMiddleware, true) and ('', false) buckets remain.
//
// Rollback: az monitor log-analytics workspace update -g <rg> -n AutopilotMonitor
// --set properties.defaultDataCollectionRuleResourceId=null (then delete the DCR)
@description('Log Analytics workspace that backs the workspace-based Application Insights resources.')
param workspaceName string = 'AutopilotMonitor'
param location string = resourceGroup().location
resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' existing = {
name: workspaceName
}
resource transforms 'Microsoft.Insights/dataCollectionRules@2023-03-11' = {
name: 'dcr-autopilotmonitor-ws-transforms'
location: location
kind: 'WorkspaceTransforms'
properties: {
description: 'AppRequests: keep the enriched worker request record (Properties.Source == WorkerMiddleware) and non-HTTP host invocations (empty Url); drop the host HTTP duplicate.'
destinations: {
logAnalytics: [
{
workspaceResourceId: workspace.id
name: 'ws'
}
]
}
dataFlows: [
{
streams: [ 'Microsoft-Table-AppRequests' ]
destinations: [ 'ws' ]
transformKql: 'source | where tostring(Properties.Source) == \'WorkerMiddleware\' or isempty(Url)'
}
{
// Successful Azure SignalR REST calls (group add/remove per connection) are emitted by
// the HOST process for the SignalR output binding (SDKVersion rdddsc:*), so the worker's
// StorageDependencyFilterProcessor never sees them. Failed SignalR calls are kept.
streams: [ 'Microsoft-Table-AppDependencies' ]
destinations: [ 'ws' ]
transformKql: 'source | where not(Success == true and Target endswith \'.service.signalr.net\')'
}
]
}
}
// A workspace has exactly one transformation DCR, activated by linking it as the workspace's
// defaultDataCollectionRuleResourceId — deliberately NOT modelled here (see Apply above).
output dcrId string = transforms.id
+69
View File
@@ -0,0 +1,69 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-DEV-008",
"title": "Device Clock Wrong During Enrollment — Jump or Sustained Offset",
"description": "The device's clock changed by five or more minutes mid-session (clock jump) or ran five or more minutes off server time for the whole session (sustained offset), measured against the server receive frame of every uploaded event. Complements ANALYZE-DEV-007, which is a one-shot NTP check at agent startup and misses devices where NTP was unreachable or the clock changed later. IME-log-derived events are excluded from the measurement.",
"severity": "warning",
"category": "device",
"version": "1.0.0",
"author": "Autopilot Monitor",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed"
],
"trigger": "single",
"baseConfidence": 60,
"confidenceThreshold": 50,
"conditions": [
{
"signal": "clock_jump",
"source": "clock_skew",
"skewMetric": "clock_jump",
"operator": "gte",
"value": "300",
"required": false
},
{
"signal": "sustained_offset",
"source": "clock_skew",
"skewMetric": "sustained_offset",
"operator": "gte",
"value": "300",
"required": false
}
],
"confidenceFactors": [],
"explanation": "This device's {{skewSummary}} (five-minute threshold). A clock that is off by five or more minutes breaks time-sensitive enrollment mechanisms: Kerberos tolerates at most 5 minutes of skew (Hybrid Join), Entra ID token validation allows about 5 minutes on token lifetimes, and freshly issued SCEP/PKCS certificates can be rejected as \"not yet valid\" when the clock is more than ~10 minutes behind the CA.\n\nUnlike ANALYZE-DEV-007 (a one-shot NTP comparison at agent startup), this finding is measured across the whole session against the server's receive time — it also catches devices where the NTP check could not run (UDP 123 blocked) or where the clock changed after startup (time sync correcting a wrong RTC, Windows auto-timezone, manual change).",
"remediation": [
{
"title": "Fix the device clock and time source",
"steps": [
"Check the BIOS/RTC clock and the CMOS battery if the wrong time recurs on the same hardware.",
"At OOBE, run 'w32tm /resync /force' (Shift+F10 command prompt) to sync with time.windows.com.",
"Ensure UDP port 123 to time.windows.com is open in the deployment network.",
"Review 'Set time automatically' and 'Set time zone automatically' behavior on the deployment image — automatic timezone changes mid-enrollment show up as clock jumps."
]
}
],
"relatedDocs": [
{
"title": "Microsoft Learn: Windows Autopilot known issues — device provisioning can fail (clock skew)",
"url": "https://learn.microsoft.com/en-us/autopilot/known-issues"
},
{
"title": "Microsoft Learn: Windows Autopilot requirements — Networking (NTP)",
"url": "https://learn.microsoft.com/en-us/autopilot/requirements?tabs=networking"
}
],
"markSessionAsFailedDefault": false,
"tags": [
"device",
"clock-skew",
"clock-jump",
"time-sync",
"kerberos",
"certificates",
"known-issue"
]
}
+48
View File
@@ -0,0 +1,48 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-DEV-009",
"title": "Battery Critically Low During Enrollment",
"description": "Detects that the device's battery dropped to 15% or less while enrolling on battery power (power_state_change threshold event from the agent's live power watcher). A device dying mid-enrollment leaves a half-provisioned state that typically needs a full reset; this rule surfaces the risk while there is still time to plug the device in. Uses a single required condition on the agent-stamped thresholdPercent field (the data field 'transition' on the event is unrelated to this rule's 'trigger' property).",
"severity": "high",
"category": "device",
"version": "1.0.0",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed",
"on_event:power_state_change"
],
"trigger": "single",
"baseConfidence": 90,
"confidenceThreshold": 60,
"conditions": [
{
"signal": "battery_threshold_15",
"source": "event_data",
"eventType": "power_state_change",
"dataField": "thresholdPercent",
"operator": "equals",
"value": "15",
"required": true
}
],
"explanation": "The device's battery fell below **{{thresholdPercent}}%** while the enrollment was running on battery power. If the battery drains completely mid-enrollment, the device is left in a half-provisioned state — apps, policies, or the user profile may be incomplete, and in most cases the device has to be reset and re-enrolled. The agent detected the threshold crossing live via a Windows power-status notification; the exact percentage at that moment is recorded on the matched power_state_change event in the session timeline.",
"remediation": [
{
"title": "Secure power for the running enrollment",
"steps": [
"Connect the device to AC power immediately — the enrollment continues normally once power is stable.",
"If the device shut down before completing, check whether the session reached the desktop; if not, reset the device (Autopilot reset or reimage) and re-enroll it plugged in.",
"For technician/White Glove flows, make AC power part of the bench checklist — pre-provisioning on battery is the most common source of this alert.",
"If this fires across many devices of one model, review that model's battery health (the hardware_spec event carries design vs. full-charge capacity)."
]
}
],
"markSessionAsFailedDefault": false,
"tags": [
"device",
"power",
"battery",
"environment"
]
}
+48
View File
@@ -0,0 +1,48 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-DEV-010",
"title": "Enrollment Switched From AC to Battery Power",
"description": "Detects that the device lost AC power and switched to battery while the enrollment was running (power_state_change event with transition ac_to_battery from the agent's live power watcher). Uses a single required condition so the match is pinned to one event instance; the event payload carries the battery percentage at the moment of the switch. The agent debounces power flapping (5s) so a brief unplug/replug does not produce this event.",
"severity": "warning",
"category": "device",
"version": "1.0.0",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed",
"on_event:power_state_change"
],
"trigger": "single",
"baseConfidence": 85,
"confidenceThreshold": 60,
"conditions": [
{
"signal": "switched_to_battery",
"source": "event_data",
"eventType": "power_state_change",
"dataField": "transition",
"operator": "equals",
"value": "ac_to_battery",
"required": true
}
],
"explanation": "The device switched from AC to battery power mid-enrollment. Enrollments are power-hungry — app downloads, Windows Update and disk encryption drain a battery quickly, and Windows may also throttle CPU and network on battery, slowing the enrollment down. The battery percentage at the moment of the switch is recorded on the matched power_state_change event in the session timeline; if the device stayed on battery, look there for subsequent threshold events (50/30/15%) — a later battery_to_ac event means power was restored.",
"remediation": [
{
"title": "Restore AC power",
"steps": [
"Reconnect the device to AC power; the enrollment continues normally.",
"Check the session timeline for a subsequent battery_to_ac event (power restored) or threshold_crossed events (battery draining) to judge urgency.",
"If this alert is frequent in a deployment area, check docking stations/USB-C hubs for loose power delivery — a flapping dock is a common cause.",
"Make 'plugged in during setup' part of the end-user/technician enrollment instructions."
]
}
],
"markSessionAsFailedDefault": false,
"tags": [
"device",
"power",
"battery",
"environment"
]
}
+49
View File
@@ -0,0 +1,49 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-DEV-012",
"title": "Device Entered Standby During Enrollment",
"description": "Detects that the device completed a sleep, hibernate or Modern Standby episode of at least 5 minutes while the enrollment was running (system_sleep_episode event from the agent's system timeline watcher, recorded retroactively at wake from the Windows event log — pre-agent episodes are backfilled). Uses a single required condition on the episode's durationSeconds so the match is pinned to one event instance and the interpolated duration is the matched value. Short Modern Standby screen-off scenarios (under 60 s of real sleep) never emit the event, and this rule adds a 5-minute floor so brief lid-close moments do not fire it.",
"severity": "info",
"category": "device",
"version": "1.0.0",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed",
"on_event:system_sleep_episode"
],
"trigger": "single",
"baseConfidence": 90,
"confidenceThreshold": 60,
"conditions": [
{
"signal": "slept_during_enrollment",
"source": "event_data",
"eventType": "system_sleep_episode",
"dataField": "durationSeconds",
"operator": "gte",
"value": "300",
"required": true
}
],
"explanation": "The device was asleep for {{durationSeconds}} seconds during the enrollment window. Nothing progresses while the device sleeps — downloads pause, the IME stops processing and the ESP simply waits — so the wall-clock enrollment duration includes this pause and can look much slower than the enrollment actually was. The matched system_sleep_episode event in the session timeline carries the exact enter/exit times, the episode kind (sleep, hibernate or Modern Standby) and, for Modern Standby, whether the device was on AC power. A long unexplained gap in the timeline right before that event is this pause, not missing telemetry.",
"remediation": [
{
"title": "Keep the device awake during enrollment",
"steps": [
"Check the matched system_sleep_episode event in the timeline for the episode bounds and kind — the enrollment duration minus the episode duration is the real active time.",
"If the device slept on battery, connect AC power during setup; Windows sleeps far more aggressively on battery.",
"Instruct users/technicians to keep the lid open and the device powered until the enrollment finishes.",
"For recurring cases, review the power plan applied during OOBE/ESP (e.g. via provisioning or Intune policy) — consider disabling sleep on AC for the enrollment window."
]
}
],
"markSessionAsFailedDefault": false,
"tags": [
"device",
"power",
"standby",
"sleep",
"environment"
]
}
+210
View File
@@ -1516,6 +1516,216 @@
"known-issue"
]
},
{
"ruleId": "ANALYZE-DEV-008",
"title": "Device Clock Wrong During Enrollment — Jump or Sustained Offset",
"description": "The device's clock changed by five or more minutes mid-session (clock jump) or ran five or more minutes off server time for the whole session (sustained offset), measured against the server receive frame of every uploaded event. Complements ANALYZE-DEV-007, which is a one-shot NTP check at agent startup and misses devices where NTP was unreachable or the clock changed later. IME-log-derived events are excluded from the measurement.",
"severity": "warning",
"category": "device",
"version": "1.0.0",
"author": "Autopilot Monitor",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed"
],
"trigger": "single",
"baseConfidence": 60,
"confidenceThreshold": 50,
"conditions": [
{
"signal": "clock_jump",
"source": "clock_skew",
"skewMetric": "clock_jump",
"operator": "gte",
"value": "300",
"required": false
},
{
"signal": "sustained_offset",
"source": "clock_skew",
"skewMetric": "sustained_offset",
"operator": "gte",
"value": "300",
"required": false
}
],
"confidenceFactors": [],
"explanation": "This device's {{skewSummary}} (five-minute threshold). A clock that is off by five or more minutes breaks time-sensitive enrollment mechanisms: Kerberos tolerates at most 5 minutes of skew (Hybrid Join), Entra ID token validation allows about 5 minutes on token lifetimes, and freshly issued SCEP/PKCS certificates can be rejected as \"not yet valid\" when the clock is more than ~10 minutes behind the CA.\n\nUnlike ANALYZE-DEV-007 (a one-shot NTP comparison at agent startup), this finding is measured across the whole session against the server's receive time — it also catches devices where the NTP check could not run (UDP 123 blocked) or where the clock changed after startup (time sync correcting a wrong RTC, Windows auto-timezone, manual change).",
"remediation": [
{
"title": "Fix the device clock and time source",
"steps": [
"Check the BIOS/RTC clock and the CMOS battery if the wrong time recurs on the same hardware.",
"At OOBE, run 'w32tm /resync /force' (Shift+F10 command prompt) to sync with time.windows.com.",
"Ensure UDP port 123 to time.windows.com is open in the deployment network.",
"Review 'Set time automatically' and 'Set time zone automatically' behavior on the deployment image — automatic timezone changes mid-enrollment show up as clock jumps."
]
}
],
"relatedDocs": [
{
"title": "Microsoft Learn: Windows Autopilot known issues — device provisioning can fail (clock skew)",
"url": "https://learn.microsoft.com/en-us/autopilot/known-issues"
},
{
"title": "Microsoft Learn: Windows Autopilot requirements — Networking (NTP)",
"url": "https://learn.microsoft.com/en-us/autopilot/requirements?tabs=networking"
}
],
"markSessionAsFailedDefault": false,
"tags": [
"device",
"clock-skew",
"clock-jump",
"time-sync",
"kerberos",
"certificates",
"known-issue"
]
},
{
"ruleId": "ANALYZE-DEV-009",
"title": "Battery Critically Low During Enrollment",
"description": "Detects that the device's battery dropped to 15% or less while enrolling on battery power (power_state_change threshold event from the agent's live power watcher). A device dying mid-enrollment leaves a half-provisioned state that typically needs a full reset; this rule surfaces the risk while there is still time to plug the device in. Uses a single required condition on the agent-stamped thresholdPercent field (the data field 'transition' on the event is unrelated to this rule's 'trigger' property).",
"severity": "high",
"category": "device",
"version": "1.0.0",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed",
"on_event:power_state_change"
],
"trigger": "single",
"baseConfidence": 90,
"confidenceThreshold": 60,
"conditions": [
{
"signal": "battery_threshold_15",
"source": "event_data",
"eventType": "power_state_change",
"dataField": "thresholdPercent",
"operator": "equals",
"value": "15",
"required": true
}
],
"explanation": "The device's battery fell below **{{thresholdPercent}}%** while the enrollment was running on battery power. If the battery drains completely mid-enrollment, the device is left in a half-provisioned state — apps, policies, or the user profile may be incomplete, and in most cases the device has to be reset and re-enrolled. The agent detected the threshold crossing live via a Windows power-status notification; the exact percentage at that moment is recorded on the matched power_state_change event in the session timeline.",
"remediation": [
{
"title": "Secure power for the running enrollment",
"steps": [
"Connect the device to AC power immediately — the enrollment continues normally once power is stable.",
"If the device shut down before completing, check whether the session reached the desktop; if not, reset the device (Autopilot reset or reimage) and re-enroll it plugged in.",
"For technician/White Glove flows, make AC power part of the bench checklist — pre-provisioning on battery is the most common source of this alert.",
"If this fires across many devices of one model, review that model's battery health (the hardware_spec event carries design vs. full-charge capacity)."
]
}
],
"markSessionAsFailedDefault": false,
"tags": [
"device",
"power",
"battery",
"environment"
]
},
{
"ruleId": "ANALYZE-DEV-010",
"title": "Enrollment Switched From AC to Battery Power",
"description": "Detects that the device lost AC power and switched to battery while the enrollment was running (power_state_change event with transition ac_to_battery from the agent's live power watcher). Uses a single required condition so the match is pinned to one event instance; the event payload carries the battery percentage at the moment of the switch. The agent debounces power flapping (5s) so a brief unplug/replug does not produce this event.",
"severity": "warning",
"category": "device",
"version": "1.0.0",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed",
"on_event:power_state_change"
],
"trigger": "single",
"baseConfidence": 85,
"confidenceThreshold": 60,
"conditions": [
{
"signal": "switched_to_battery",
"source": "event_data",
"eventType": "power_state_change",
"dataField": "transition",
"operator": "equals",
"value": "ac_to_battery",
"required": true
}
],
"explanation": "The device switched from AC to battery power mid-enrollment. Enrollments are power-hungry — app downloads, Windows Update and disk encryption drain a battery quickly, and Windows may also throttle CPU and network on battery, slowing the enrollment down. The battery percentage at the moment of the switch is recorded on the matched power_state_change event in the session timeline; if the device stayed on battery, look there for subsequent threshold events (50/30/15%) — a later battery_to_ac event means power was restored.",
"remediation": [
{
"title": "Restore AC power",
"steps": [
"Reconnect the device to AC power; the enrollment continues normally.",
"Check the session timeline for a subsequent battery_to_ac event (power restored) or threshold_crossed events (battery draining) to judge urgency.",
"If this alert is frequent in a deployment area, check docking stations/USB-C hubs for loose power delivery — a flapping dock is a common cause.",
"Make 'plugged in during setup' part of the end-user/technician enrollment instructions."
]
}
],
"markSessionAsFailedDefault": false,
"tags": [
"device",
"power",
"battery",
"environment"
]
},
{
"ruleId": "ANALYZE-DEV-012",
"title": "Device Entered Standby During Enrollment",
"description": "Detects that the device completed a sleep, hibernate or Modern Standby episode of at least 5 minutes while the enrollment was running (system_sleep_episode event from the agent's system timeline watcher, recorded retroactively at wake from the Windows event log — pre-agent episodes are backfilled). Uses a single required condition on the episode's durationSeconds so the match is pinned to one event instance and the interpolated duration is the matched value. Short Modern Standby screen-off scenarios (under 60 s of real sleep) never emit the event, and this rule adds a 5-minute floor so brief lid-close moments do not fire it.",
"severity": "info",
"category": "device",
"version": "1.0.0",
"enabled": true,
"evaluateOn": [
"enrollment_end",
"whiteglove_sealed",
"on_event:system_sleep_episode"
],
"trigger": "single",
"baseConfidence": 90,
"confidenceThreshold": 60,
"conditions": [
{
"signal": "slept_during_enrollment",
"source": "event_data",
"eventType": "system_sleep_episode",
"dataField": "durationSeconds",
"operator": "gte",
"value": "300",
"required": true
}
],
"explanation": "The device was asleep for {{durationSeconds}} seconds during the enrollment window. Nothing progresses while the device sleeps — downloads pause, the IME stops processing and the ESP simply waits — so the wall-clock enrollment duration includes this pause and can look much slower than the enrollment actually was. The matched system_sleep_episode event in the session timeline carries the exact enter/exit times, the episode kind (sleep, hibernate or Modern Standby) and, for Modern Standby, whether the device was on AC power. A long unexplained gap in the timeline right before that event is this pause, not missing telemetry.",
"remediation": [
{
"title": "Keep the device awake during enrollment",
"steps": [
"Check the matched system_sleep_episode event in the timeline for the episode bounds and kind — the enrollment duration minus the episode duration is the real active time.",
"If the device slept on battery, connect AC power during setup; Windows sleeps far more aggressively on battery.",
"Instruct users/technicians to keep the lid open and the device powered until the enrollment finishes.",
"For recurring cases, review the power plan applied during OOBE/ESP (e.g. via provisioning or Intune policy) — consider disabling sleep on AC for the enrollment window."
]
}
],
"markSessionAsFailedDefault": false,
"tags": [
"device",
"power",
"standby",
"sleep",
"environment"
]
},
{
"ruleId": "ANALYZE-ENRL-001",
"title": "Enrollment Failed",
+8
View File
@@ -675,6 +675,14 @@
"description": "Detects when IME completes a user session, indicating enrollment has finished for that user.",
"enabled": true
},
{
"patternId": "IME-USER-SESSION-ZERO-APPS",
"category": "always",
"pattern": "\\[Win32App\\] Get 0 apps for user session (?<sessionId>\\d+), user id = (?<userId>[a-f0-9\\-]+)",
"action": "userSessionZeroApps",
"description": "Detects when IME enumerated a user session and found zero targeted Win32 apps. IME then skips its 'Completed user session' line entirely (the zero-app branch in ApplicationPoller continues before that log statement), so this line is the only evidence that user-phase enforcement finished on a device with no user-targeted apps. Treated as equivalent to a completed user session.",
"enabled": true
},
{
"patternId": "IME-WIN32-STATE-1",
"category": "currentPhase",
@@ -0,0 +1,9 @@
{
"$schema": "../schema/ime-log-pattern.schema.json",
"patternId": "IME-USER-SESSION-ZERO-APPS",
"category": "always",
"pattern": "\\[Win32App\\] Get 0 apps for user session (?<sessionId>\\d+), user id = (?<userId>[a-f0-9\\-]+)",
"action": "userSessionZeroApps",
"description": "Detects when IME enumerated a user session and found zero targeted Win32 apps. IME then skips its 'Completed user session' line entirely (the zero-app branch in ApplicationPoller continues before that log statement), so this line is the only evidence that user-phase enforcement finished on a device with no user-targeted apps. Treated as equivalent to a completed user session.",
"enabled": true
}
+6 -1
View File
@@ -169,7 +169,12 @@
"source": {
"type": "string",
"description": "Source of the signal.",
"enum": ["event_type", "event_data", "event_data_array", "phase_duration", "event_count", "app_install_duration", "event_correlation"]
"enum": ["event_type", "event_data", "event_data_array", "phase_duration", "event_count", "app_install_duration", "event_correlation", "clock_skew"]
},
"skewMetric": {
"type": "string",
"description": "For clock_skew only: which device-clock metric to evaluate. clock_jump = persistent step in the device's clock frame mid-session; sustained_offset = whole session ran on a clock off by at least the threshold. value is the threshold in seconds; operator is limited to gt/gte on the magnitude. IME-log-derived events are excluded from the measurement.",
"enum": ["clock_jump", "sustained_offset"]
},
"eventType": {
"type": "string",
+1
View File
@@ -63,6 +63,7 @@
"imeImpersonation",
"imeTokenFailure",
"enrollmentCompleted",
"userSessionZeroApps",
"updateDoTelemetry",
"scriptStarted",
"scriptContext",
+25 -2
View File
@@ -13,18 +13,26 @@ const rulesRoot = path.resolve(__dirname, '..');
// ── Rule combination ────────────────────────────────────────────────────────
// Shipped gather/analyze rules must stay inside the reserved built-in namespace:
// (ANALYZE|GATHER)-<CATEGORY>-<NNN> with CATEGORY !== CUSTOM. The CUSTOM category is
// the sanctioned tenant namespace (RuleIdPolicy.cs) — a built-in shipped there would
// silently shadow every tenant's same-ID custom rule at merge time (global wins).
const RESERVED_BUILTIN_ID = /^(ANALYZE|GATHER)-(?!CUSTOM-)[A-Z]+-\d{3}$/;
const configs = [
{
dir: path.join(rulesRoot, 'gather'),
output: path.join(rulesRoot, 'dist', 'gather-rules.json'),
schema: '../schema/gather-rule.schema.json',
idField: 'ruleId'
idField: 'ruleId',
reservedNamespace: RESERVED_BUILTIN_ID
},
{
dir: path.join(rulesRoot, 'analyze'),
output: path.join(rulesRoot, 'dist', 'analyze-rules.json'),
schema: '../schema/analyze-rule.schema.json',
idField: 'ruleId'
idField: 'ruleId',
reservedNamespace: RESERVED_BUILTIN_ID
},
{
dir: path.join(rulesRoot, 'ime-log-patterns'),
@@ -37,12 +45,27 @@ const configs = [
for (const config of configs) {
const files = fs.readdirSync(config.dir).filter(f => f.endsWith('.json')).sort();
const rules = [];
const seenIds = new Map(); // lowercased id -> filename (case variants collide for humans and RuleIdPolicy)
for (const file of files) {
const content = fs.readFileSync(path.join(config.dir, file), 'utf8');
const rule = JSON.parse(content);
// Remove $schema from individual entries (it's on the wrapper)
delete rule['$schema'];
const id = rule[config.idField] || '';
const idKey = id.toLowerCase();
if (seenIds.has(idKey)) {
console.error(`ERROR: duplicate ${config.idField} "${id}" in ${file} (already defined in ${seenIds.get(idKey)})`);
process.exit(1);
}
seenIds.set(idKey, file);
if (config.reservedNamespace && !config.reservedNamespace.test(id)) {
console.error(`ERROR: ${file}: ${config.idField} "${id}" is outside the reserved built-in namespace ${config.reservedNamespace} — built-ins must never use the tenant CUSTOM namespace`);
process.exit(1);
}
rules.push(rule);
}
@@ -1,4 +1,4 @@
#Requires -Version 5.1
#Requires -Version 5.1
<#
.SYNOPSIS
Grants additional Microsoft Graph application permissions to the Autopilot Monitor
@@ -28,10 +28,18 @@
(validates Windows 365 Cloud PCs against the tenant's Cloud PC inventory --
Cloud PCs are never Autopilot-registered, so this enables agent monitoring
of Cloud PC first-connect enrollment)
- IntuneDeviceBinding -> DeviceManagementManagedDevices.Read.All
(cert-to-device binding: verifies that the Intune device id carried in the
agent's client certificate is a device THIS tenant actually enrolled --
currently a Global-Admin-only preview that records telemetry and does not
block enrollment)
- All -> every optional permission above, in one run
(also works with -Revoke to remove all optional grants at once)
.PARAMETER Permissions
Microsoft Graph application permission names to grant directly. Use when the admin UI
rendered a copy-paste command for you, or when you have a specific list in mind.
Microsoft Graph application permission names to grant directly. Use when you copied the
permission list from the admin UI ("Copy permissions"), or when you have a specific list
in mind.
.PARAMETER TenantId
Tenant ID (GUID) or verified domain. Optional -- defaults to the signed-in context.
@@ -47,12 +55,16 @@
.\Grant-AutopilotMonitorAddOn.ps1 -ClientId "<your-client-id>" -Features ScriptDisplayNames
.EXAMPLE
# Same effect, using the explicit-permission form (what the admin UI generates):
# Same effect, using the explicit-permission form:
.\Grant-AutopilotMonitorAddOn.ps1 `
-ClientId "<your-client-id>" `
-Permissions "DeviceManagementScripts.Read.All" `
-TenantId "contoso.onmicrosoft.com"
.EXAMPLE
# Grant every optional capability in one run:
.\Grant-AutopilotMonitorAddOn.ps1 -ClientId "<your-client-id>" -Features All
.EXAMPLE
# Verify current state only, no changes:
.\Grant-AutopilotMonitorAddOn.ps1 -ClientId "<your-client-id>" -Features ScriptDisplayNames -VerifyOnly
@@ -65,6 +77,12 @@
Requires:
- Microsoft.Graph.Authentication PowerShell module (auto-installs if missing).
- Sign-in as Global Administrator OR Privileged Role Administrator OR Cloud Application Administrator.
Sign-in behaviour:
In Azure Cloud Shell the script signs in via the ambient identity endpoint (a delegated
token for the already-signed-in user) - no device-code prompt, which Conditional Access
policies commonly block. Outside Cloud Shell it falls back to a normal interactive
Microsoft Graph sign-in. -TenantId only applies to the interactive fallback.
- The Autopilot Monitor app must be admin-consented in your tenant first
(its service principal must exist before grants can be added).
@@ -80,7 +98,7 @@ param(
[string]$ClientId,
[Parameter(Mandatory = $true, ParameterSetName = 'ByFeatures')]
[ValidateSet('ScriptDisplayNames', 'W365CloudPcValidation')]
[ValidateSet('All', 'ScriptDisplayNames', 'W365CloudPcValidation', 'IntuneDeviceBinding')]
[string[]]$Features,
[Parameter(Mandatory = $true, ParameterSetName = 'ByPermissions')]
@@ -103,10 +121,15 @@ $ErrorActionPreference = 'Stop'
$FeatureCatalog = @{
'ScriptDisplayNames' = @('DeviceManagementScripts.Read.All')
'W365CloudPcValidation' = @('CloudPC.Read.All')
'IntuneDeviceBinding' = @('DeviceManagementManagedDevices.Read.All')
}
# Translate Features -> Permissions when caller used the high-level form.
# 'All' is a script-side meta-value: it expands to every catalog feature.
if ($PSCmdlet.ParameterSetName -eq 'ByFeatures') {
if ($Features -contains 'All') {
$Features = @($FeatureCatalog.Keys)
}
$Permissions = @()
foreach ($f in $Features) {
$Permissions += $FeatureCatalog[$f]
@@ -133,7 +156,25 @@ if ($TenantId) { $connectParams['TenantId'] = $TenantId }
Write-Host ""
Write-Host "[Auth] Connecting to Microsoft Graph..." -ForegroundColor Cyan
Connect-MgGraph @connectParams | Out-Null
# Preferred path: ambient identity. In Azure Cloud Shell the local identity endpoint issues a
# delegated Graph token for the signed-in user, which avoids the device-code flow that
# Conditional Access policies commonly block. Outside Azure this throws and we fall back to
# an interactive sign-in with explicit scopes.
$identityConnected = $false
try {
Connect-MgGraph -Identity -NoWelcome -ErrorAction Stop | Out-Null
# An ambient identity can exist yet hold no Graph permissions (e.g. a plain VM managed
# identity) - probe with a read before committing to this session.
Invoke-MgGraphRequest -Method GET -Uri "https://graph.microsoft.com/v1.0/servicePrincipals?`$select=id&`$top=1" -ErrorAction Stop | Out-Null
$identityConnected = $true
Write-Host "[Auth] Connected via ambient identity (Azure Cloud Shell / managed identity)." -ForegroundColor Green
}
catch {
if (Get-MgContext) { Disconnect-MgGraph -ErrorAction SilentlyContinue | Out-Null }
Write-Host "[Auth] No usable ambient identity - falling back to interactive sign-in..." -ForegroundColor Yellow
Connect-MgGraph @connectParams | Out-Null
}
$ctx = Get-MgContext
if (-not $ctx) {
@@ -144,7 +185,10 @@ Write-Host "[Auth] Signed in as : $($ctx.Account)"
Write-Host "[Auth] Tenant : $($ctx.TenantId)"
Write-Host "[Auth] Granted scopes: $($ctx.Scopes -join ', ')"
if (-not ($ctx.Scopes -contains 'AppRoleAssignment.ReadWrite.All')) {
# Scope check only applies to the interactive path: an ambient-identity (Cloud Shell) token
# carries a fixed scope set (incl. Directory.AccessAsUser.All) instead of the requested scopes,
# and its effective rights follow the signed-in user's directory role.
if (-not $identityConnected -and -not ($ctx.Scopes -contains 'AppRoleAssignment.ReadWrite.All')) {
Write-Warning "Scope 'AppRoleAssignment.ReadWrite.All' was NOT granted. The signed-in user likely lacks the required role (Global Admin / Privileged Role Admin / Cloud App Admin)."
if (-not $VerifyOnly) { return }
}
@@ -0,0 +1,62 @@
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
{
/// <summary>
/// Pins the compiled-in fallback rule catalogs (<see cref="BuiltInConfigCatalog"/>):
/// the agent embeds the SAME CI-combined rules/dist artifacts the backend serves, so a
/// no-config/no-cache session still runs full IME tracking. These tests guard the
/// embedding + deserialization end-to-end — a broken LogicalName, a moved resource or a
/// shape change in the combined JSON must fail HERE, not silently strip the fallback
/// back down to zero patterns in the field.
/// </summary>
public class BuiltInConfigCatalogTests
{
[Fact]
public void ImeLogPatterns_LoadFromEmbeddedCatalog_EnabledOnly()
{
var patterns = BuiltInConfigCatalog.GetEnabledImeLogPatterns(logger: null);
Assert.NotEmpty(patterns);
Assert.All(patterns, p =>
{
Assert.True(p.Enabled);
Assert.False(string.IsNullOrWhiteSpace(p.PatternId));
Assert.False(string.IsNullOrWhiteSpace(p.Pattern));
Assert.False(string.IsNullOrWhiteSpace(p.Action));
Assert.False(string.IsNullOrWhiteSpace(p.Category));
});
Assert.Equal(patterns.Count, patterns.Select(p => p.PatternId).Distinct().Count());
}
[Fact]
public void ImeLogPatterns_CarryCoreTrackingActions()
{
// The app-tracking backbone: without these two actions the ImeLogTracker cannot
// attribute installs to apps or follow ESP phases. Their presence proves the
// deserialization actually populated the action field (a shape drift in the
// combined JSON would otherwise still pass the structural checks above).
var actions = BuiltInConfigCatalog.GetEnabledImeLogPatterns(logger: null)
.Select(p => p.Action).ToHashSet();
Assert.Contains("setCurrentApp", actions);
Assert.Contains("espPhaseDetected", actions);
}
[Fact]
public void GatherRules_LoadFromEmbeddedCatalog_EnabledOnly()
{
// The built-in gather-rule catalog ships mostly disabled rules — only the
// default-enabled ones may reach the fallback config.
var rules = BuiltInConfigCatalog.GetEnabledGatherRules(logger: null);
Assert.All(rules, r =>
{
Assert.True(r.Enabled);
Assert.False(string.IsNullOrWhiteSpace(r.RuleId));
});
}
}
}
@@ -37,6 +37,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
nameof(AgentConfigResponse.EnrollmentSummaryLaunchRetrySeconds),
nameof(AgentConfigResponse.NtpServer),
nameof(AgentConfigResponse.EnableTimezoneAutoSet),
nameof(AgentConfigResponse.EnableDoGroupIdAutoSet),
nameof(AgentConfigResponse.DiagnosticsUploadEnabled),
nameof(AgentConfigResponse.DiagnosticsUploadMode),
nameof(AgentConfigResponse.DiagnosticsLogPaths),
@@ -52,6 +53,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// Merge reads two Collectors knobs (AgentMaxLifetimeMinutes, HelloWaitTimeoutSeconds);
// the remaining CollectorConfiguration flags flow untouched to DefaultComponentFactory.
nameof(AgentConfigResponse.Collectors),
// Merge reads one Analyzers knob (EnableRealmJoinWatcher → diagnostics RealmJoin sections);
// the remaining flags flow untouched to DefaultComponentFactory + AgentAnalyzerManager.
nameof(AgentConfigResponse.Analyzers),
};
/// <summary>
@@ -69,8 +73,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
nameof(AgentConfigResponse.ImeLogPatterns),
// Passed by AgentRuntimeHost into IComponentFactory.CreateCollectorHosts → ImeLogHost sealing re-emit.
nameof(AgentConfigResponse.WhiteGloveSealingPatternIds),
// Consumed by DefaultComponentFactory (RealmJoin/KeepAwake/ConsoleBypass host gates) + AgentAnalyzerManager.
nameof(AgentConfigResponse.Analyzers),
// Self-update ZIP integrity second trust channel — BinaryIntegrityVerifier / Program update path.
nameof(AgentConfigResponse.LatestAgentSha256),
// Running-EXE integrity verification — BinaryIntegrityVerifier post-config check.
@@ -43,6 +43,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
EnrollmentSummaryLaunchRetrySeconds = 300,
NtpServer = "pool.ntp.org",
EnableTimezoneAutoSet = true,
EnableDoGroupIdAutoSet = true,
DiagnosticsUploadEnabled = true,
DiagnosticsUploadMode = "OnFailure",
DiagnosticsLogPaths = new List<DiagnosticsLogPath>
@@ -83,6 +84,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.Equal(300, agent.EnrollmentSummaryLaunchRetrySeconds);
Assert.Equal("pool.ntp.org", agent.NtpServer);
Assert.True(agent.EnableTimezoneAutoSet);
Assert.True(agent.EnableDoGroupIdAutoSet);
Assert.True(agent.DiagnosticsUploadEnabled);
Assert.Equal("OnFailure", agent.DiagnosticsUploadMode);
Assert.Single(agent.DiagnosticsLogPaths);
@@ -366,6 +368,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.Equal(30, agent.HelloWaitTimeoutSeconds);
}
[Fact]
public void Merge_mirrors_analyzers_realmjoin_watcher_onto_agent_config()
{
// The diagnostics package gates its RealmJoin sections on this flag but never
// sees the remote response — the merger carries the one Analyzers knob over.
var agent = NewAgentConfig();
var remote = FullRemote();
remote.Analyzers = new AnalyzerConfiguration { EnableRealmJoinWatcher = true };
RemoteConfigMerger.Merge(agent, remote);
Assert.True(agent.EnableRealmJoinWatcher);
}
[Fact]
public void Merge_clears_realmjoin_watcher_when_remote_analyzers_missing()
{
var agent = NewAgentConfig();
agent.EnableRealmJoinWatcher = true; // stale cached true must not survive
var remote = FullRemote();
remote.Analyzers = null!;
RemoteConfigMerger.Merge(agent, remote);
Assert.False(agent.EnableRealmJoinWatcher);
}
[Fact]
public void Merge_replaces_diagnostics_log_paths_with_empty_list_when_remote_null()
{
@@ -114,6 +114,22 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.Equal(1, svc.LastFetchAttempts);
}
[Fact]
public async Task FetchConfig_DefaultsFallback_CarriesBuiltInCatalogs()
{
// A no-config/no-cache session must still get the compiled-in rule catalogs —
// an empty ImeLogPatterns list kills the entire IME tracking pipeline (the
// pre-catalog behaviour this pins against regressing to).
var svc = NewService(BackendBehaviour.AlwaysTransientFailure);
var result = await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(0, result.ConfigVersion);
Assert.NotEmpty(result.ImeLogPatterns);
Assert.All(result.ImeLogPatterns, p => Assert.True(p.Enabled));
Assert.All(result.GatherRules, r => Assert.True(r.Enabled));
}
[Fact]
public async Task FetchConfig_EmptyResponse_TreatedAsTransient()
{
@@ -128,6 +144,94 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.Equal(4, svc.LastFetchAttempts);
}
// ── Network-link gate (tenant aebdce78 audit 2026-08-24) ────────────
[Fact]
public async Task FetchConfig_InitialFetch_WaitsForNetworkLink()
{
// The initial startup fetch must gate on the link-level signal so the retry
// budget isn't burned into a NIC that is provably still down (agent relaunched
// at boot, Wi-Fi still associating).
var svc = NewService(BackendBehaviour.SuccessImmediately);
await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(1, svc.NetworkLinkWaits);
}
[Fact]
public async Task FetchConfig_RotatePath_SkipsNetworkLinkGate()
{
// rotate_config runs mid-session on a proven link and must stay latency-bounded.
var svc = NewService(BackendBehaviour.SuccessImmediately);
await svc.FetchConfigAsync(retryOnTransientErrors: false);
Assert.Equal(0, svc.NetworkLinkWaits);
}
// ── Post-registration recovery ──────────────────────────────────────
[Fact]
public async Task TryRecover_AfterFailedFetch_AppliesLiveConfig_KeepsFailureRecord()
{
// The core self-heal contract: registration proved the backend reachable, the
// recovery swaps in the live config — but the LastFetch* observability keeps
// describing the failed startup fetch so the wire event can report both.
var svc = NewService(BackendBehaviour.FailAllThenSucceed);
await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
var recovered = await svc.TryRecoverConfigAsync();
Assert.NotNull(recovered);
Assert.Equal(42, recovered.ConfigVersion);
Assert.Equal(42, svc.CurrentConfig.ConfigVersion);
Assert.True(svc.RecoveredAfterFetchFailure);
// Startup-failure record intact:
Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
Assert.Equal(1 + RemoteConfigService.InitialFetchRetryBackoff.Length, svc.LastFetchAttempts);
Assert.Equal(nameof(HttpRequestException), svc.LastFetchFailureType);
}
[Fact]
public async Task TryRecover_AfterSuccessfulFetch_IsNoOp()
{
var svc = NewService(BackendBehaviour.SuccessImmediately);
await svc.FetchConfigAsync(retryOnTransientErrors: true);
var recovered = await svc.TryRecoverConfigAsync();
Assert.Null(recovered);
Assert.False(svc.RecoveredAfterFetchFailure);
}
[Fact]
public async Task TryRecover_BeforeAnyFetch_IsNoOp()
{
var svc = NewService(BackendBehaviour.SuccessImmediately);
var recovered = await svc.TryRecoverConfigAsync();
Assert.Null(recovered);
Assert.False(svc.RecoveredAfterFetchFailure);
}
[Fact]
public async Task TryRecover_WhenBackendStillFailing_KeepsFallback_NeverThrows()
{
var svc = NewService(BackendBehaviour.AlwaysTransientFailure);
var fallback = await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(0, fallback.ConfigVersion);
var recovered = await svc.TryRecoverConfigAsync();
Assert.Null(recovered);
Assert.False(svc.RecoveredAfterFetchFailure);
Assert.Equal(0, svc.CurrentConfig.ConfigVersion); // fallback config untouched
Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
}
// ── Outcome state defaults ──────────────────────────────────────────
[Fact]
@@ -145,6 +249,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
SuccessImmediately,
AlwaysTransientFailure,
FailTwiceThenSucceed,
/// <summary>All 4 initial attempts fail; the NEXT call (the recovery fetch) succeeds.</summary>
FailAllThenSucceed,
AlwaysAuthFailure401,
AlwaysReturnNull,
}
@@ -177,6 +283,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
_behaviour = behaviour;
}
/// <summary>Number of times the network-link gate seam was awaited.</summary>
public int NetworkLinkWaits { get; private set; }
protected override Task<AgentConfigResponse> CallBackendAsync()
{
_calls++;
@@ -188,6 +297,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
if (_calls < 3)
throw new HttpRequestException("simulated cold-start timeout");
return Task.FromResult(new AgentConfigResponse { ConfigVersion = 42 });
case BackendBehaviour.FailAllThenSucceed:
if (_calls <= 1 + RemoteConfigService.InitialFetchRetryBackoff.Length)
throw new HttpRequestException("simulated dead network link");
return Task.FromResult(new AgentConfigResponse { ConfigVersion = 42 });
case BackendBehaviour.AlwaysTransientFailure:
throw new HttpRequestException("simulated network failure");
case BackendBehaviour.AlwaysAuthFailure401:
@@ -202,6 +315,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// Zero-delay so the suite finishes in ms instead of the live 10/30/60 s schedule.
protected override Task DelayBetweenAttemptsAsync(TimeSpan delay) => Task.CompletedTask;
// Record + skip the real link poll so the suite is deterministic on offline CI.
protected override Task WaitForNetworkLinkAsync()
{
NetworkLinkWaits++;
return Task.CompletedTask;
}
// In-memory + per-instance cache so a prior test's successful fetch can't
// bleed into a later test that expects UsedDefaults.
private AgentConfigResponse? _cached;
@@ -0,0 +1,109 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
{
/// <summary>
/// Pins the per-FILE isolation of the offset measurement across a full read cycle.
///
/// <para>
/// Background (2026-08-20, session e9753578): the field showed "+02:00 measured for
/// IntuneManagementExtension.log" although that file held no +2-era line — the anchor had
/// crossed file boundaries. The committed code was proven innocent (this very scenario is
/// green here); the leak lived in a binary built from an uncommitted tree. This test keeps
/// the property pinned so a real regression of the file scoping can never ship silently
/// again — it replays the exact field pass structure: a NEW file appearing mid-run with a
/// different writer era than the file already being tailed.
/// </para>
/// </summary>
public sealed class CmTraceCalibratorFileIsolationTests
{
private static readonly DateTime T0 = new DateTime(2026, 8, 20, 12, 13, 0, DateTimeKind.Utc);
private static List<ImeLogPattern> Patterns() => new List<ImeLogPattern>
{
new ImeLogPattern
{
PatternId = "T-MARK", Category = "always", Enabled = true,
Pattern = @"marker (?<n>\d+)",
Action = "noop",
Parameters = new Dictionary<string, string>(),
},
};
private static string Line(string message, DateTime utcInstant, TimeSpan writerOffset)
{
var local = utcInstant + writerOffset;
return $"<![LOG[{message}]LOG]!><time=\"{local:HH:mm:ss.fffffff}\" date=\"{local:M-d-yyyy}\" " +
"component=\"X\" context=\"\" type=\"1\" thread=\"1\" file=\"\">";
}
[Fact]
public async Task NewFileWithOtherEra_MustNotRecalibrateTheOldFile()
{
var pdt = TimeSpan.FromHours(-7); // IME service era
var cest = TimeSpan.FromHours(2); // fresh child-process era
using var tmp = new TempDirectory();
var now = T0;
var tracker = new ImeLogTracker(
logFolder: tmp.Path,
patterns: Patterns(),
logger: new AgentLogger(tmp.Path, AgentLogLevel.Info));
try
{
tracker.UtcNowProvider = () => now;
var imeLog = Path.Combine(tmp.Path, "IntuneManagementExtension.log");
var execLog = Path.Combine(tmp.Path, "AgentExecutor.log");
// Pass 1: IME.log first sight (no calibration).
File.AppendAllText(imeLog, Line("marker 1", now, pdt) + Environment.NewLine);
await tracker.CheckLogFilesAsync(CancellationToken.None);
// Pass 2: IME.log grew -> measures -7.
now = T0.AddSeconds(10);
File.AppendAllText(imeLog, Line("marker 2", now, pdt) + Environment.NewLine);
await tracker.CheckLogFilesAsync(CancellationToken.None);
TimeSpan imeMeasured;
Assert.True(tracker.OffsetCalibrator.TryGetOffset("IntuneManagementExtension.log", out imeMeasured));
Assert.Equal(pdt, imeMeasured);
// Pass 3: AgentExecutor.log appears for the FIRST time with fresh +2-era lines
// while IME.log grows too — the exact field pass structure. The +2 anchor must
// never land under IME.log's key.
now = T0.AddSeconds(20);
File.AppendAllText(execLog, Line("marker 3", now, cest) + Environment.NewLine);
File.AppendAllText(imeLog, Line("marker 4", now, pdt) + Environment.NewLine);
await tracker.CheckLogFilesAsync(CancellationToken.None);
Assert.True(tracker.OffsetCalibrator.TryGetOffset("IntuneManagementExtension.log", out imeMeasured));
Assert.Equal(pdt, imeMeasured);
// Pass 4: both grow — each file measures its own writer's era.
now = T0.AddSeconds(30);
File.AppendAllText(execLog, Line("marker 5", now, cest) + Environment.NewLine);
File.AppendAllText(imeLog, Line("marker 6", now, pdt) + Environment.NewLine);
await tracker.CheckLogFilesAsync(CancellationToken.None);
TimeSpan execMeasured;
Assert.True(tracker.OffsetCalibrator.TryGetOffset("AgentExecutor.log", out execMeasured));
Assert.Equal(cest, execMeasured);
Assert.True(tracker.OffsetCalibrator.TryGetOffset("IntuneManagementExtension.log", out imeMeasured));
Assert.Equal(pdt, imeMeasured);
}
finally
{
tracker.Dispose();
}
}
}
}
@@ -0,0 +1,441 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Logging;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
{
/// <summary>
/// Per-line self-anchoring — the era-aware successor to the reverted per-file offset
/// application (04b1a7c6).
///
/// <para>
/// The property under test: a line read in a provably FRESH pass resolves to the instant it
/// was written, whatever timezone its writer believed — and neighbouring lines from writers
/// with DIFFERENT beliefs each resolve to their own era, because no cross-line state is
/// involved. That interleaving (fixture: AgentExecutor.log with PDT and CEST children in one
/// file, flipping both ways) is exactly what broke every cross-line anchor design.
/// </para>
///
/// <para>
/// The freshness boundary is equally load-bearing: backlog (first sight, restart catch-up,
/// stalled polls) must NEVER anchor, because an old line's age can round onto the 15-minute
/// offset grid. Those paths keep the uniform reader-zone fallback.
/// </para>
/// </summary>
public sealed class ImeLogTrackerLineAnchoringTests
{
private static readonly DateTime T0 = new DateTime(2026, 8, 20, 12, 13, 0, DateTimeKind.Utc);
private static List<ImeLogPattern> Patterns() => new List<ImeLogPattern>
{
new ImeLogPattern
{
PatternId = "T-ALL", Category = "always", Enabled = true,
Pattern = @"marker (?<n>\d+)",
Action = "noop",
Parameters = new Dictionary<string, string>(),
},
};
/// <summary>A CMTrace line as a writer holding <paramref name="writerOffset"/> renders instant <paramref name="utcInstant"/>.</summary>
private static string Line(string message, DateTime utcInstant, TimeSpan writerOffset)
{
var local = utcInstant + writerOffset;
return $"<![LOG[{message}]LOG]!><time=\"{local:HH:mm:ss.fffffff}\" date=\"{local:M-d-yyyy}\" " +
"component=\"X\" context=\"\" type=\"1\" thread=\"1\" file=\"\">";
}
private sealed class Harness : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
public ImeLogTracker Tracker { get; }
public DateTime Now { get; set; } = T0;
public Harness(string? stateDirectory = null)
{
Tracker = new ImeLogTracker(
logFolder: _tmp.Path,
patterns: Patterns(),
logger: new AgentLogger(_tmp.Path, AgentLogLevel.Info),
stateDirectory: stateDirectory);
Tracker.UtcNowProvider = () => Now;
}
public string PathOf(string fileName) => Path.Combine(_tmp.Path, fileName);
public void Append(string fileName, string line)
=> File.AppendAllText(PathOf(fileName), line + Environment.NewLine);
public Task Pass() => Tracker.CheckLogFilesAsync(CancellationToken.None);
public void Dispose()
{
Tracker.Dispose();
_tmp.Dispose();
}
}
[Theory]
// The writer's belief. A fresh line must resolve to the instant it was written,
// WHATEVER that belief is — including agreement with this process (the majority case,
// which a fix to the agent's own zone cache would have broken).
[InlineData(2)] // CEST
[InlineData(1)] // BST
[InlineData(0)] // UTC
[InlineData(-7)] // PDT — the OOBE default behind the -9 h and -17 h field cases
[InlineData(10)] // E. Australia
public async Task FreshLine_ResolvesToItsTrueInstant_ForAnyWriterBelief(int writerOffsetHours)
{
var writerOffset = TimeSpan.FromHours(writerOffsetHours);
using var h = new Harness();
// First sight: backlog, must not anchor.
h.Append("IntuneManagementExtension.log", Line("marker 1", h.Now, writerOffset));
await h.Pass();
h.Now = T0.AddSeconds(10);
var writtenAt = h.Now.AddMilliseconds(-80); // written just before the poll sees it
h.Append("IntuneManagementExtension.log", Line("marker 2", writtenAt, writerOffset));
await h.Pass();
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
Assert.Equal(writtenAt, h.Tracker.LastMatchedLogTimestamp);
Assert.Equal(writerOffsetHours * 60, h.Tracker.LastMatchedSourceOffsetMinutes);
}
[Fact]
public async Task InterleavedEras_InOneGrowingPass_EachLineResolvesToItsOwnEra()
{
// The e9753578 file shape: two writer processes with different zone beliefs
// interleaved in ONE file. Any per-file offset gets one of them wrong by 9 h;
// per-line anchoring must get both right within the same chunk.
var pdt = TimeSpan.FromHours(-7);
var cest = TimeSpan.FromHours(2);
using var h = new Harness();
h.Append("AgentExecutor.log", Line("marker 1", h.Now, pdt));
await h.Pass();
h.Now = T0.AddSeconds(10);
var wrotePdt = h.Now.AddMilliseconds(-90);
var wroteCest = h.Now.AddMilliseconds(-50);
h.Append("AgentExecutor.log", Line("marker 2", wrotePdt, pdt));
h.Append("AgentExecutor.log", Line("marker 3", wroteCest, cest));
var resolved = new List<(DateTime? Ts, CmTraceOffsetOrigin Origin, int? Offset)>();
Action<string> capture = _ => resolved.Add((
h.Tracker.LastMatchedLogTimestamp,
h.Tracker.LastMatchedSourceOffsetOrigin,
h.Tracker.LastMatchedSourceOffsetMinutes));
h.Tracker.OnPatternMatched += capture;
try
{
await h.Pass();
}
finally
{
h.Tracker.OnPatternMatched -= capture;
}
Assert.Equal(2, resolved.Count);
Assert.Equal(wrotePdt, resolved[0].Ts);
Assert.Equal(-420, resolved[0].Offset);
Assert.Equal(wroteCest, resolved[1].Ts);
Assert.Equal(120, resolved[1].Offset);
Assert.All(resolved, r => Assert.Equal(CmTraceOffsetOrigin.LineAnchored, r.Origin));
}
[Fact]
public async Task FirstSightBacklog_FallsBackToReaderZone()
{
using var h = new Harness();
h.Append("IntuneManagementExtension.log", Line("marker 1", h.Now.AddHours(-3), TimeSpan.FromHours(2)));
await h.Pass();
Assert.Equal(CmTraceOffsetOrigin.None, h.Tracker.LastMatchedSourceOffsetOrigin);
}
[Fact]
public async Task FileFirstSeenEmpty_ItsFirstContentIsFresh()
{
// AgentExecutor.log in the field: the file APPEARS mid-enrollment. Seeing it empty
// (or not existing content-wise) counts as an observation, so its very first lines
// are provably written between passes and may anchor. This closes the warm-up gap
// for exactly the file whose eras flip the most.
using var h = new Harness();
File.WriteAllText(h.PathOf("AgentExecutor.log"), string.Empty);
await h.Pass();
h.Now = T0.AddSeconds(5);
var writtenAt = h.Now.AddMilliseconds(-60);
h.Append("AgentExecutor.log", Line("marker 1", writtenAt, TimeSpan.FromHours(2)));
await h.Pass();
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
Assert.Equal(writtenAt, h.Tracker.LastMatchedLogTimestamp);
}
[Fact]
public async Task PollGapBeyondFreshWindow_FallsBack()
{
// A stalled agent (GC, CPU starvation) re-polls after a long gap: the new bytes span
// the whole gap and their age is unbounded, so the pass must not anchor — even
// though most of its lines may in fact be recent. Conservative by design.
using var h = new Harness();
h.Append("IntuneManagementExtension.log", Line("marker 1", h.Now, TimeSpan.FromHours(2)));
await h.Pass();
h.Now = T0.AddSeconds(10);
h.Append("IntuneManagementExtension.log", Line("marker 2", h.Now, TimeSpan.FromHours(2)));
await h.Pass();
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
// Gap far beyond FreshLineMaxAge.
h.Now = h.Now + ImeLogTracker.FreshLineMaxAge + TimeSpan.FromSeconds(10);
h.Append("IntuneManagementExtension.log", Line("marker 3", h.Now, TimeSpan.FromHours(2)));
await h.Pass();
Assert.Equal(CmTraceOffsetOrigin.None, h.Tracker.LastMatchedSourceOffsetOrigin);
// And the very next tight poll is fresh again.
h.Now = h.Now.AddSeconds(1);
h.Append("IntuneManagementExtension.log", Line("marker 4", h.Now, TimeSpan.FromHours(2)));
await h.Pass();
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
}
[Fact]
public async Task RestartWithRestoredBookmark_FirstPassIsBacklog_SecondPassAnchors()
{
// The persisted position bookmark survives a restart; the freshness guarantee must
// not — the first pass after a restart reads the downtime backlog, whose lines can
// be arbitrarily old.
using var stateDir = new TempDirectory();
using var h = new Harness(stateDirectory: stateDir.Path);
h.Append("IntuneManagementExtension.log", Line("marker 1", h.Now, TimeSpan.FromHours(2)));
await h.Pass();
h.Tracker.SaveStateForTest();
// "Restart": a second tracker over the same folder restores the bookmark.
using var h2 = new HarnessOver(h, stateDir.Path);
h2.Tracker.LoadStateForTest();
// Backlog written during the downtime — 30 min old, exactly on the offset grid,
// the nastiest possible backlog shape. Must NOT anchor.
h2.Now = T0.AddMinutes(30);
h2.Append(Line("marker 2", T0.AddSeconds(20), TimeSpan.FromHours(2)));
await h2.Pass();
Assert.Equal(CmTraceOffsetOrigin.None, h2.Tracker.LastMatchedSourceOffsetOrigin);
// The next tight poll is fresh again.
h2.Now = h2.Now.AddSeconds(1);
var writtenAt = h2.Now.AddMilliseconds(-70);
h2.Append(Line("marker 3", writtenAt, TimeSpan.FromHours(2)));
await h2.Pass();
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h2.Tracker.LastMatchedSourceOffsetOrigin);
Assert.Equal(writtenAt, h2.Tracker.LastMatchedLogTimestamp);
}
/// <summary>Second tracker over the FIRST harness's log folder — the "restarted agent".</summary>
private sealed class HarnessOver : IDisposable
{
private readonly string _logFolder;
public ImeLogTracker Tracker { get; }
public DateTime Now { get; set; }
public HarnessOver(Harness original, string stateDirectory)
{
_logFolder = Path.GetDirectoryName(original.PathOf("x"));
Tracker = new ImeLogTracker(
logFolder: _logFolder,
patterns: Patterns(),
logger: new AgentLogger(_logFolder, AgentLogLevel.Info),
stateDirectory: stateDirectory);
Tracker.UtcNowProvider = () => Now;
}
public void Append(string line)
=> File.AppendAllText(Path.Combine(_logFolder, "IntuneManagementExtension.log"), line + Environment.NewLine);
public Task Pass() => Tracker.CheckLogFilesAsync(CancellationToken.None);
public void Dispose() => Tracker.Dispose();
}
[Fact]
public async Task FreshlyWrittenReplayLine_OnExactGridAge_ResolvesToRoughlyNow()
{
// Characterization of the one accepted edge: a line freshly WRITTEN but carrying a
// REPLAYED timestamp whose age is exactly a grid multiple (here 30 min) measures
// that age as an "offset" and resolves to ~now instead of its replayed past. The
// error is bounded by construction (an anchored line always lands at now ± 2 min),
// and sub-24h replays are treated as current by the historic-replay guard anyway.
using var h = new Harness();
h.Append("IntuneManagementExtension.log", Line("marker 1", h.Now, TimeSpan.Zero));
await h.Pass();
h.Now = T0.AddSeconds(10);
var replayedInstant = h.Now.AddMinutes(-30);
h.Append("IntuneManagementExtension.log", Line("marker 2", replayedInstant, TimeSpan.Zero));
await h.Pass();
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
Assert.NotNull(h.Tracker.LastMatchedLogTimestamp);
var error = (h.Tracker.LastMatchedLogTimestamp!.Value - h.Now).Duration();
Assert.True(error <= CmTraceOffsetCalibrator.MaxGridResidual,
$"anchored line must resolve to now ± residual, was off by {error}");
}
// =====================================================================
// The committed field fixture — session e9753578, ImeLogs/AgentExecutor.log.
// One file, two writer eras (PDT hour-05 / CEST hour-14), interleaved, flipping
// BOTH ways. Replayed here as the live tail the field agent saw: entry by entry,
// with the clock following the true write time.
// =====================================================================
private static string FixturePath()
{
var dir = AppContext.BaseDirectory;
for (int i = 0; i < 12; i++)
{
if (File.Exists(Path.Combine(dir, "AutopilotMonitor.sln")))
return Path.Combine(dir, "tests", "fixtures", "cmtrace-logs",
"agentexecutor-two-writer-eras-v1.cmtrace");
var parent = Directory.GetParent(dir)?.FullName;
if (parent == null || parent == dir) break;
dir = parent;
}
throw new DirectoryNotFoundException(
"Could not locate repo root (AutopilotMonitor.sln) walking up from " + AppContext.BaseDirectory);
}
/// <summary>Group the fixture's physical lines into complete CMTrace entries (multiline-safe).</summary>
private static List<string> FixtureEntries()
{
var entries = new List<string>();
string? pending = null;
foreach (var rawWithBom in File.ReadAllLines(FixturePath()))
{
// The fixture keeps the original file's BOM verbatim — it sat at the START of
// the real AgentExecutor.log, which is mid-file in this excerpt.
var raw = rawWithBom.TrimStart('');
if (pending == null)
{
if (raw.Length == 0 || raw[0] == '#') continue;
if (!raw.StartsWith("<![LOG[", StringComparison.Ordinal)) continue;
if (raw.Contains("]LOG]!>")) { entries.Add(raw); continue; }
pending = raw;
}
else
{
pending += "\n" + raw;
if (raw.Contains("]LOG]!>")) { entries.Add(pending!); pending = null; }
}
}
return entries;
}
[Fact]
public async Task TwoWriterEraFixture_EveryLineOfBothErasResolvesToItsTrueInstant()
{
// The committed fixture is a curated, verbatim excerpt of the transition window
// (commit policy: anonymized, versioned) — 20 entries covering both eras and the flip.
var entries = FixtureEntries();
Assert.True(entries.Count >= 15, $"fixture unexpectedly small: {entries.Count} entries");
using var h = new Harness();
// Match every fixture entry.
h.Tracker.CompilePatterns(new List<ImeLogPattern>
{
new ImeLogPattern
{
PatternId = "T-ANY", Category = "always", Enabled = true,
Pattern = ".+", Action = "noop",
Parameters = new Dictionary<string, string>(),
},
});
// The file exists (empty) before its first content — first sight is the observation
// baseline, everything after is a fresh tail, exactly like the field.
File.WriteAllText(h.PathOf("AgentExecutor.log"), string.Empty);
h.Now = new DateTime(2026, 8, 20, 12, 13, 30, DateTimeKind.Utc);
await h.Pass();
var anchored = 0;
var resolvedList = new List<(DateTime Resolved, DateTime Expected)>();
Action<string> capture = _ =>
{
if (h.Tracker.LastMatchedSourceOffsetOrigin == CmTraceOffsetOrigin.LineAnchored)
anchored++;
if (h.Tracker.LastMatchedLogTimestamp.HasValue && _expectedForCurrentEntry.HasValue)
resolvedList.Add((h.Tracker.LastMatchedLogTimestamp.Value, _expectedForCurrentEntry.Value));
};
h.Tracker.OnPatternMatched += capture;
try
{
foreach (var entry in entries)
{
CmTraceLogEntry? parsed;
Assert.True(CmTraceLogParser.TryParseLine(entry, out parsed), "fixture entry did not parse");
// Era by local hour: the fixture holds exactly two eras — PDT (UTC-7) lines
// render at local hour 05, CEST (UTC+2) lines at hour 14 (fixture header).
var eraOffset = parsed.LocalTimestamp.Hour == 14
? TimeSpan.FromHours(2)
: TimeSpan.FromHours(-7);
var trueUtc = DateTime.SpecifyKind(parsed.LocalTimestamp - eraOffset, DateTimeKind.Utc);
// The clock follows the write; never move backwards (interleaved eras are
// written in true-UTC order, but guard against sub-ms jitter).
var readAt = trueUtc.AddMilliseconds(120);
if (readAt > h.Now) h.Now = readAt;
// In production the 100 ms poll keeps running through write silences — every
// empty look refreshes the freshness window. The fixture's transition window
// spans multi-minute write gaps, so replay that cadence with one empty poll
// just before each append; without it the NEXT pass would (correctly!)
// classify the gap as a stall and refuse to anchor.
h.Now = h.Now.AddMilliseconds(-100);
await h.Pass();
h.Now = readAt;
_expectedForCurrentEntry = trueUtc;
File.AppendAllText(h.PathOf("AgentExecutor.log"), entry + Environment.NewLine);
await h.Pass();
_expectedForCurrentEntry = null;
}
}
finally
{
h.Tracker.OnPatternMatched -= capture;
}
// Every matched entry resolved to the instant its writer rendered — across BOTH
// eras, including the flips. This is the exact property whose absence produced the
// -9 h script durations.
Assert.Equal(entries.Count, resolvedList.Count);
Assert.All(resolvedList, r => Assert.Equal(r.Expected, r.Resolved));
Assert.True(anchored == resolvedList.Count,
$"expected every entry line-anchored, got {anchored}/{resolvedList.Count}");
// Both eras genuinely exercised: PDT entries render at local hour 05, CEST at 14.
Assert.Contains(resolvedList, r => (r.Expected + TimeSpan.FromHours(-7)).Hour == 5);
Assert.Contains(resolvedList, r => (r.Expected + TimeSpan.FromHours(2)).Hour == 14);
}
private DateTime? _expectedForCurrentEntry;
}
}
@@ -80,36 +80,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
}
[Theory]
// The writer's belief. Whatever it is, a line written at T must come back as T.
[InlineData(2)] // CEST — the bfff9b1c writer, while the agent still believed BST
[InlineData(1)] // BST — writer and a UK-defaulted agent agreeing
[InlineData(-7)] // PDT — the OOBE default behind the -17 h sessions
// The writer's belief. The MEASUREMENT must recover it exactly, whatever it is.
[InlineData(2)] // CEST
[InlineData(1)] // BST
[InlineData(-7)] // PDT — the OOBE default behind the -17 h and -9 h field cases
[InlineData(10)] // E. Australia
[InlineData(0)] // UTC
public async Task ResolvesLineToTheInstantItWasWritten_ForAnyWriterBelief(int writerOffsetHours)
public async Task MeasuresTheWritersOffsetExactly_ForAnyBelief(int writerOffsetHours)
{
// NOTE: the per-FILE measurement asserted here is OBSERVATIONAL — its application was
// reverted on 2026-08-20 (04b1a7c6) because one offset per file is wrong when a file
// holds interleaved writer eras. Resolution now anchors per LINE instead (see
// ImeLogTrackerLineAnchoringTests, which restores the "a line written at T resolves
// to T" property era-safely). The per-file measurement stays for the Info log and
// measuredWriterOffsetMinutes provenance.
var writerOffset = TimeSpan.FromHours(writerOffsetHours);
using var h = new Harness();
// Pass 1 — first sight of the file. Its content may be arbitrarily old, so nothing is
// calibrated from it.
h.Append(Line("marker 1", h.Now, writerOffset));
await h.Pass();
// Pass 2 — the file has grown against a size we already observed, so this line was
// written within a poll interval and is a valid anchor.
h.Now = T0.AddSeconds(10);
h.Append(Line("marker 2", h.Now, writerOffset));
await h.Pass();
// Pass 3 — resolved with the measured offset.
h.Now = T0.AddSeconds(20);
var writtenAt = h.Now;
h.Append(Line("marker 3", writtenAt, writerOffset));
await h.Pass();
Assert.True(h.Tracker.LastMatchedLogTimestamp.HasValue);
Assert.Equal(writtenAt, h.Tracker.LastMatchedLogTimestamp.Value);
TimeSpan measured;
Assert.True(h.Tracker.OffsetCalibrator.TryGetOffset(LogFileName, out measured));
Assert.Equal(writerOffset, measured);
}
[Fact]
@@ -175,8 +172,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
TimeSpan after;
Assert.True(h.Tracker.OffsetCalibrator.TryGetOffset(LogFileName, out after));
Assert.Equal(TimeSpan.FromHours(2), after);
Assert.True(h.Tracker.LastMatchedLogTimestamp.HasValue);
Assert.Equal(writtenAt, h.Tracker.LastMatchedLogTimestamp.Value);
// The measurement follows the restart. Applying it is what the 2026-08-20 revert
// removed — that is exactly the case a file with two writer eras breaks.
Assert.Equal(120, h.Tracker.LastMatchedMeasuredWriterOffsetMinutes);
}
[Fact]
@@ -204,12 +202,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
}
[Fact]
public async Task RecordsProvenance_ForACalibratedLine()
public async Task RecordsMeasuredOffsetSeparatelyFromTheAppliedOne()
{
// P8: what the writer wrote, what we subtracted, and how we knew — the evidence that
// makes a stored timestamp recomputable without the diagnostics archive.
// The regression that forced the revert shipped an event tagged "calibrated" while
// carrying an offset that did not hold for that line. Measured and applied are now
// separate fields so a future reader cannot mistake one for the other.
using var h = new Harness();
var writerOffset = TimeSpan.FromHours(-7); // PST/PDT, the OOBE default
var writerOffset = TimeSpan.FromHours(-7); // PDT, the OOBE default
h.Append(Line("marker 1", h.Now, writerOffset));
await h.Pass();
@@ -219,18 +218,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
await h.Pass();
h.Now = T0.AddSeconds(20);
var writtenAt = h.Now;
h.Append(Line("marker 3", writtenAt, writerOffset));
h.Append(Line("marker 3", h.Now, writerOffset));
await h.Pass();
Assert.Equal(CmTraceOffsetOrigin.Calibrated, h.Tracker.LastMatchedSourceOffsetOrigin);
Assert.Equal(-420, h.Tracker.LastMatchedSourceOffsetMinutes);
Assert.True(h.Tracker.LastMatchedSourceLocalTimestamp.HasValue);
// Measured: the writer's actual belief, per file.
Assert.Equal(-420, h.Tracker.LastMatchedMeasuredWriterOffsetMinutes);
// The raw local value is kept verbatim, with no zone attached — attaching one would
// re-introduce exactly the assumption this change removes.
Assert.Equal(writtenAt + writerOffset, h.Tracker.LastMatchedSourceLocalTimestamp.Value);
Assert.Equal(DateTimeKind.Unspecified, h.Tracker.LastMatchedSourceLocalTimestamp.Value.Kind);
// Applied: the line's OWN anchor (fresh pass), never the per-file "calibrated" value —
// the two must stay distinguishable even when they agree numerically.
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
Assert.NotEqual(CmTraceOffsetOrigin.Calibrated, h.Tracker.LastMatchedSourceOffsetOrigin);
}
[Fact]
@@ -244,8 +241,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
await h.Pass();
Assert.Equal(CmTraceOffsetOrigin.None, h.Tracker.LastMatchedSourceOffsetOrigin);
Assert.Null(h.Tracker.LastMatchedSourceOffsetMinutes);
Assert.True(h.Tracker.LastMatchedSourceLocalTimestamp.HasValue);
// The APPLIED offset is reported even on the fallback path — an event that states no
// offset at all cannot be recomputed later, which defeats the purpose of recording
// provenance. Asserted against the running machine's zone so this stays portable.
var readerZone = (int)TimeZoneInfo.Local
.GetUtcOffset(h.Tracker.LastMatchedSourceLocalTimestamp!.Value).TotalMinutes;
Assert.Equal(readerZone, h.Tracker.LastMatchedSourceOffsetMinutes);
}
[Fact]
@@ -0,0 +1,199 @@
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
{
/// <summary>
/// Sessions 81daa77f / 75d6ae8e (2026-08-25) — zero user-targeted apps starved every
/// user-phase completion probe. The IME zero-app branch returns before it writes the
/// "Completed user session" line, so a device whose user has no targeted Win32 apps never
/// produced the evidence <c>IsImeUserSessionGenuine</c> requires, and the AdvisoryCompletion
/// window failed the enrollment 30 minutes later — with the user at the desktop, Hello
/// provisioned and RealmJoin still installing.
/// <para>
/// IME-USER-SESSION-ZERO-APPS reads the line IME does write in that branch and routes it
/// through the same <see cref="ImeLogTracker.OnUserSessionCompleted"/> callback.
/// </para>
/// </summary>
public sealed class ImeLogTrackerZeroUserAppsTests
{
// Mirrors rules/ime-log-patterns/IME-USER-SESSION-ZERO-APPS.json. The JSON itself is
// pinned against a real IME line by BuiltInRulesTests in the backend suite.
private const string ZeroAppsPattern =
@"\[Win32App\] Get 0 apps for user session (?<sessionId>\d+), user id = (?<userId>[a-f0-9\-]+)";
private const string EspPhasePattern =
@"\[Win32App\] (?:In|The) EspPhase: (?<espPhase>\w+)";
private const string ZeroAppsLine =
"[Win32App] Get 0 apps for user session 1, user id = 4a1ba5db-44ed-4e69-8477-47ed88e76020";
private static ImeLogTracker BuildTracker(TempDirectory tmp) =>
new ImeLogTracker(
logFolder: tmp.Path,
patterns: new List<ImeLogPattern>
{
new ImeLogPattern
{
PatternId = "IME-USER-SESSION-ZERO-APPS",
Category = "always",
Pattern = ZeroAppsPattern,
Action = "userSessionZeroApps",
},
new ImeLogPattern
{
PatternId = "IME-ESP-PHASE",
Category = "always",
Pattern = EspPhasePattern,
Action = "espPhaseDetected",
},
},
logger: new AgentLogger(tmp.Path, AgentLogLevel.Info));
[Fact]
public void ZeroApps_line_during_user_phase_fires_user_session_completed()
{
using var tmp = new TempDirectory();
var tracker = BuildTracker(tmp);
var fired = 0;
tracker.OnUserSessionCompleted = () => fired++;
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: AccountSetup");
tracker.ProcessLogMessageForTest(ZeroAppsLine);
Assert.Equal(1, fired);
Assert.True(tracker.UserSessionZeroAppsObserved);
}
[Fact]
public void ZeroApps_line_before_user_phase_does_not_fire()
{
// Firing here would burn the adapter fire-once flag on a pre-sign-in timestamp that
// IsImeUserSessionGenuine (>= AccountSetupEnteredUtc) can never accept.
using var tmp = new TempDirectory();
var tracker = BuildTracker(tmp);
var fired = 0;
tracker.OnUserSessionCompleted = () => fired++;
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: DeviceSetup");
tracker.ProcessLogMessageForTest(ZeroAppsLine);
Assert.Equal(0, fired);
Assert.True(tracker.UserSessionZeroAppsObserved);
}
[Fact]
public void ZeroApps_observed_early_is_replayed_on_the_user_phase_transition()
{
// IME writes its check-in lines independently of when the agent first parses the ESP
// phase marker, so the evidence can legitimately arrive first. Dropping it would
// leave a zero-app device with no completion evidence at all.
using var tmp = new TempDirectory();
var tracker = BuildTracker(tmp);
var fired = 0;
tracker.OnUserSessionCompleted = () => fired++;
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: DeviceSetup");
tracker.ProcessLogMessageForTest(ZeroAppsLine);
Assert.Equal(0, fired);
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: AccountSetup");
Assert.Equal(1, fired);
}
[Fact]
public void Repeated_user_phase_matches_do_not_replay_again()
{
// IME re-emits the phase string as it re-evaluates app sets. Only a real transition
// replays; the adapter fire-once flag is the outer guard, but the tracker must not
// spam the callback either.
using var tmp = new TempDirectory();
var tracker = BuildTracker(tmp);
var fired = 0;
tracker.OnUserSessionCompleted = () => fired++;
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: DeviceSetup");
tracker.ProcessLogMessageForTest(ZeroAppsLine);
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: AccountSetup");
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: AccountSetup");
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: AccountSetup");
Assert.Equal(1, fired);
}
[Fact]
public void Nonzero_app_count_lines_do_not_match_the_pattern()
{
using var tmp = new TempDirectory();
var tracker = BuildTracker(tmp);
var fired = 0;
tracker.OnUserSessionCompleted = () => fired++;
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: AccountSetup");
tracker.ProcessLogMessageForTest(
"[Win32App] Get 3 apps for user session 1, user id = 4a1ba5db-44ed-4e69-8477-47ed88e76020");
tracker.ProcessLogMessageForTest(
"[Win32App] Get 10 apps for user session 1, user id = 4a1ba5db-44ed-4e69-8477-47ed88e76020");
Assert.Equal(0, fired);
Assert.False(tracker.UserSessionZeroAppsObserved);
}
[Fact]
public void AreUserEspAppsSettled_true_after_zero_apps_observation()
{
using var tmp = new TempDirectory();
var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: AccountSetup");
tracker.ProcessLogMessageForTest(ZeroAppsLine);
Assert.True(tracker.AreUserEspAppsSettled());
}
[Fact]
public void AreUserEspAppsSettled_stays_false_on_an_empty_list_without_the_observation()
{
// Mutation proof for the branch above: an empty live list on its own still means
// "phase just cleared / apps not surfaced yet", never vacuously settled.
using var tmp = new TempDirectory();
var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: AccountSetup");
Assert.False(tracker.AreUserEspAppsSettled());
}
[Fact]
public void AreUserEspAppsSettled_false_outside_the_user_phase_even_after_the_observation()
{
using var tmp = new TempDirectory();
var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: DeviceSetup");
tracker.ProcessLogMessageForTest(ZeroAppsLine);
Assert.True(tracker.UserSessionZeroAppsObserved);
Assert.False(tracker.AreUserEspAppsSettled());
}
[Fact]
public void GetPendingRequiredUserEspInstallApps_is_empty_for_a_zero_app_user_session()
{
// The adapter deferral probe must not park the signal when there is nothing to wait
// for — otherwise the zero-app path would starve exactly like the old one.
using var tmp = new TempDirectory();
var tracker = BuildTracker(tmp);
tracker.ProcessLogMessageForTest("[Win32App] In EspPhase: AccountSetup");
tracker.ProcessLogMessageForTest(ZeroAppsLine);
Assert.Empty(tracker.GetPendingRequiredUserEspInstallApps());
}
}
}
@@ -1,9 +1,11 @@
#nullable enable
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Text;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Shared.Models;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
@@ -46,7 +48,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
Directory.CreateDirectory(SpoolFolder);
}
public DiagnosticsPackageService Build(System.Action<AgentConfiguration>? mutateConfig = null)
public DiagnosticsPackageService Build(
System.Action<AgentConfiguration>? mutateConfig = null,
IReadOnlyDictionary<string, string>? sectionFolderOverrides = null,
System.Func<bool>? devicePreparationProbe = null)
{
// BackendApiClient is required by the public ctor but BuildArchiveBytes
// never touches it — construct with a throwaway HttpClient. Tests only
@@ -71,7 +76,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
imeLogFolderOverride: ImeTmp.Path,
agentStateFolderOverride: StateFolder,
agentSpoolFolderOverride: SpoolFolder,
agentDataFolderOverride: DataFolder);
agentDataFolderOverride: DataFolder,
sectionFolderOverrides: sectionFolderOverrides,
devicePreparationProbe: devicePreparationProbe);
}
public void Dispose()
@@ -287,6 +294,185 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
Assert.DoesNotContain(entries, e => e.StartsWith("AgentSpool/"));
}
// ===================================================== built-in catalog gates ====
// The RealmJoin sections ride behind the tenant's RealmJoin Watcher toggle, the
// Device Preparation bootstrapper event log behind the deterministic WDP marker.
// Every conditional section is skipped WITH a manifest line, never silently.
/// <summary>RealmJoin system-side folders redirected to temp (user-side goes via UserProfileResolver).</summary>
private sealed class RealmJoinRig : System.IDisposable
{
public TempDirectory WindowsLogs { get; } = new TempDirectory();
public TempDirectory Choco { get; } = new TempDirectory();
public TempDirectory UserProfile { get; } = new TempDirectory();
public string PackagesFolder => Path.Combine(WindowsLogs.Path, "RealmJoin");
public RealmJoinRig()
{
File.WriteAllText(Path.Combine(WindowsLogs.Path, "realmjoin.log"), "rj");
File.WriteAllText(Path.Combine(WindowsLogs.Path, "realmjoin2.log"), "rj2");
var pkg = Path.Combine(PackagesFolder, "Packages", "generic-7zip");
Directory.CreateDirectory(pkg);
File.WriteAllText(Path.Combine(pkg, "install.log"), "pkg");
var choco = Path.Combine(Choco.Path, "generic-7zip");
Directory.CreateDirectory(choco);
File.WriteAllText(Path.Combine(choco, "2026-06-30_install.log"), "choco");
var userRj = Path.Combine(UserProfile.Path, "AppData", "Local", "RealmJoin");
Directory.CreateDirectory(Path.Combine(userRj, "Logs", "generic-office"));
File.WriteAllText(Path.Combine(userRj, "tray.log"), "tray");
File.WriteAllText(Path.Combine(userRj, "config.pjson"), "not a log");
File.WriteAllText(Path.Combine(userRj, "Logs", "RjImeHost.log"), "host");
File.WriteAllText(Path.Combine(userRj, "Logs", "generic-office", "usersettings.log"), "pkg-user");
}
public IReadOnlyDictionary<string, string> Overrides => new Dictionary<string, string>
{
["RealmJoinWindows"] = WindowsLogs.Path,
["RealmJoinPackages"] = PackagesFolder,
["RealmJoinChoco"] = Choco.Path,
};
public void Dispose()
{
WindowsLogs.Dispose();
Choco.Dispose();
UserProfile.Dispose();
}
}
[Fact]
public void BuildArchiveBytes_skips_realmjoin_sections_when_watcher_disabled()
{
using var rig = new Rig();
using var rj = new RealmJoinRig();
UserProfileResolver.SetForTesting(rj.UserProfile.Path);
try
{
var bytes = rig.Build(sectionFolderOverrides: rj.Overrides).BuildArchiveBytes(enrollmentSucceeded: true);
var entries = ZipEntryNames(bytes);
Assert.DoesNotContain(entries, e => e.StartsWith("RealmJoinLogs/"));
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
Assert.Contains("SCENARIO: devicePreparation=False realmJoinWatcher=False", manifest);
Assert.Contains("BUILT-IN SKIPPED (RealmJoin Watcher disabled): RealmJoinWindows", manifest);
Assert.Contains("BUILT-IN SKIPPED (RealmJoin Watcher disabled): RealmJoinUserLogs", manifest);
}
finally
{
UserProfileResolver.Reset();
}
}
[Fact]
public void BuildArchiveBytes_includes_realmjoin_sections_when_watcher_enabled()
{
using var rig = new Rig();
using var rj = new RealmJoinRig();
UserProfileResolver.SetForTesting(rj.UserProfile.Path);
try
{
var bytes = rig.Build(cfg => cfg.EnableRealmJoinWatcher = true, rj.Overrides)
.BuildArchiveBytes(enrollmentSucceeded: false);
var entries = ZipEntryNames(bytes);
// ZIP layout mirrors the disk layout: flat client logs + package tree under
// Windows/, tray logs + per-user Logs tree under User/.
Assert.Contains("RealmJoinLogs/Windows/realmjoin.log", entries);
Assert.Contains("RealmJoinLogs/Windows/realmjoin2.log", entries);
Assert.Contains("RealmJoinLogs/Windows/RealmJoin/Packages/generic-7zip/install.log", entries);
Assert.Contains("RealmJoinLogs/Choco/generic-7zip/2026-06-30_install.log", entries);
Assert.Contains("RealmJoinLogs/User/tray.log", entries);
Assert.Contains("RealmJoinLogs/User/Logs/RjImeHost.log", entries);
Assert.Contains("RealmJoinLogs/User/Logs/generic-office/usersettings.log", entries);
// Only tray*.log is collected from the user RealmJoin root — config stays out.
Assert.DoesNotContain(entries, e => e.EndsWith("config.pjson"));
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
Assert.Contains("BUILT-IN: RealmJoinUserLogs -> folder=", manifest);
Assert.DoesNotContain("BUILT-IN SKIPPED (RealmJoin Watcher disabled)", manifest);
}
finally
{
UserProfileResolver.Reset();
}
}
[Fact]
public void BuildArchiveBytes_skips_realmjoin_user_sections_without_user_session()
{
using var rig = new Rig();
using var rj = new RealmJoinRig();
// Token present, no interactive user detected → user-profile sections skip with a
// manifest line; the system-side sections are unaffected.
UserProfileResolver.SetForTesting(null!);
try
{
var bytes = rig.Build(cfg => cfg.EnableRealmJoinWatcher = true, rj.Overrides)
.BuildArchiveBytes(enrollmentSucceeded: true);
var entries = ZipEntryNames(bytes);
Assert.Contains("RealmJoinLogs/Windows/realmjoin.log", entries);
Assert.DoesNotContain(entries, e => e.StartsWith("RealmJoinLogs/User"));
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
Assert.Contains("BUILT-IN SKIPPED (no user session for token): RealmJoinUserTray", manifest);
Assert.Contains("BUILT-IN SKIPPED (no user session for token): RealmJoinUserLogs", manifest);
}
finally
{
UserProfileResolver.Reset();
}
}
[Fact]
public void BuildArchiveBytes_collects_bootstrapper_event_log_only_on_device_preparation()
{
using var rig = new Rig();
using var winevt = new TempDirectory();
// Synthetic evtx: no registered channel claims the file, so the export falls
// back to a raw copy — the point here is the scenario gate, not wevtutil.
File.WriteAllBytes(Path.Combine(winevt.Path, "BootstrapperAgentServiceLogProvider.evtx"), new byte[] { 1, 2, 3 });
var overrides = new Dictionary<string, string> { ["ImeBootstrapperEventLog"] = winevt.Path };
var classic = rig.Build(sectionFolderOverrides: overrides, devicePreparationProbe: () => false)
.BuildArchiveBytes(enrollmentSucceeded: true);
Assert.DoesNotContain("ImeLogs/BootstrapperAgentServiceLogProvider.evtx", ZipEntryNames(classic));
Assert.Contains("BUILT-IN SKIPPED (not a Device Preparation enrollment): ImeBootstrapperEventLog",
ReadZipEntryText(classic, "package-manifest.txt"));
var wdp = rig.Build(sectionFolderOverrides: overrides, devicePreparationProbe: () => true)
.BuildArchiveBytes(enrollmentSucceeded: true);
Assert.Contains("ImeLogs/BootstrapperAgentServiceLogProvider.evtx", ZipEntryNames(wdp));
Assert.Contains("SCENARIO: devicePreparation=True", ReadZipEntryText(wdp, "package-manifest.txt"));
}
[Fact]
public void BuildArchiveBytes_treats_failing_device_preparation_probe_as_classic()
{
using var rig = new Rig();
var bytes = rig.Build(devicePreparationProbe: () => throw new System.InvalidOperationException("registry boom"))
.BuildArchiveBytes(enrollmentSucceeded: true);
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
Assert.Contains("SCENARIO PROBE FAILED (treated as Classic): registry boom", manifest);
Assert.Contains("BUILT-IN SKIPPED (not a Device Preparation enrollment): ImeBootstrapperEventLog", manifest);
Assert.Contains("sessioninfo.txt", ZipEntryNames(bytes)); // archive still built
}
[Theory]
[InlineData(DiagnosticsSectionCondition.Always, false, false, true)]
[InlineData(DiagnosticsSectionCondition.RealmJoinWatcher, false, true, false)]
[InlineData(DiagnosticsSectionCondition.RealmJoinWatcher, true, false, true)]
[InlineData(DiagnosticsSectionCondition.DevicePreparation, true, false, false)]
[InlineData(DiagnosticsSectionCondition.DevicePreparation, false, true, true)]
[InlineData((DiagnosticsSectionCondition)999, true, true, false)]
public void IsSectionActive_gates_on_condition(DiagnosticsSectionCondition condition, bool watcher, bool wdp, bool expected)
{
var section = new DiagnosticsBuiltInSection("X", "X", @"C:\X", new[] { "*.log" }, false, "x", condition);
var cfg = Cfg();
cfg.EnableRealmJoinWatcher = watcher;
Assert.Equal(expected, DiagnosticsPackageService.IsSectionActive(section, cfg, wdp));
}
private static string[] ZipEntryNames(byte[] zipBytes)
{
using var ms = new MemoryStream(zipBytes);
@@ -0,0 +1,258 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Runtime;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// Pure state machine behind the power watcher: AC↔battery edges, the 50/30/15 threshold
/// ladder (latch once per lifetime, lowest-crossed-only on multi-level jumps, baseline
/// coverage for sessions starting low), null-percent tolerance and the storm cap.
/// </summary>
public sealed class PowerStateTransitionTrackerTests
{
private static PowerStateResult Ac(int? percent = 80, bool charging = true) => new PowerStateResult
{
OnAcPower = true,
HasBattery = true,
BatteryPercent = percent,
IsCharging = charging,
BatteryLifeMinutes = null,
};
private static PowerStateResult Battery(int? percent, int? lifeMinutes = 90) => new PowerStateResult
{
OnAcPower = false,
HasBattery = true,
BatteryPercent = percent,
IsCharging = false,
BatteryLifeMinutes = lifeMinutes,
};
[Fact]
public void Baseline_on_ac_is_silent()
{
var tracker = new PowerStateTransitionTracker();
Assert.Empty(tracker.Baseline(Ac()));
}
[Fact]
public void Baseline_without_battery_or_with_probe_error_is_silent()
{
var tracker = new PowerStateTransitionTracker();
Assert.Empty(tracker.Baseline(new PowerStateResult { OnAcPower = true, HasBattery = false }));
Assert.Empty(tracker.Baseline(new PowerStateResult { ProbeError = "boom" }));
}
[Fact]
public void Baseline_on_battery_above_ladder_is_silent()
{
var tracker = new PowerStateTransitionTracker();
Assert.Empty(tracker.Baseline(Battery(80)));
}
[Fact]
public void Baseline_at_12_percent_emits_exactly_the_15_threshold()
{
var tracker = new PowerStateTransitionTracker();
var emissions = tracker.Baseline(Battery(12));
var e = Assert.Single(emissions);
Assert.Equal(PowerStateTransitionTracker.TransitionThresholdCrossed, e.Transition);
Assert.Equal(15, e.ThresholdPercent);
Assert.Equal(12, e.BatteryPercent);
Assert.Equal(EventSeverity.Error, e.Severity);
Assert.True(e.ImmediateUpload);
}
[Fact]
public void Baseline_at_25_percent_emits_30_and_latches_50()
{
var tracker = new PowerStateTransitionTracker();
var baseline = tracker.Baseline(Battery(25));
var e = Assert.Single(baseline);
Assert.Equal(30, e.ThresholdPercent);
Assert.Equal(EventSeverity.Warning, e.Severity);
Assert.True(e.ImmediateUpload);
// 50 and 30 are latched by the baseline; draining to 40% stays silent, 14% emits 15.
Assert.Empty(tracker.Evaluate(Battery(40)));
var next = Assert.Single(tracker.Evaluate(Battery(14)));
Assert.Equal(15, next.ThresholdPercent);
}
[Fact]
public void Ac_to_battery_emits_warning_with_immediate_upload()
{
var tracker = new PowerStateTransitionTracker();
tracker.Baseline(Ac(63));
var e = Assert.Single(tracker.Evaluate(Battery(63)));
Assert.Equal(PowerStateTransitionTracker.TransitionAcToBattery, e.Transition);
Assert.Null(e.ThresholdPercent);
Assert.Equal(EventSeverity.Warning, e.Severity);
Assert.True(e.ImmediateUpload);
Assert.False(e.OnAcPower);
Assert.Equal(63, e.BatteryPercent);
}
[Fact]
public void Battery_to_ac_emits_info_without_immediate_upload()
{
var tracker = new PowerStateTransitionTracker();
tracker.Baseline(Battery(70));
var e = Assert.Single(tracker.Evaluate(Ac(70)));
Assert.Equal(PowerStateTransitionTracker.TransitionBatteryToAc, e.Transition);
Assert.Equal(EventSeverity.Info, e.Severity);
Assert.False(e.ImmediateUpload);
}
[Fact]
public void No_diff_stays_silent()
{
var tracker = new PowerStateTransitionTracker();
tracker.Baseline(Ac());
Assert.Empty(tracker.Evaluate(Ac()));
Assert.Empty(tracker.Evaluate(Ac(79)));
}
[Fact]
public void Unplug_below_threshold_emits_transition_and_threshold_together()
{
var tracker = new PowerStateTransitionTracker();
tracker.Baseline(Ac(10));
var emissions = tracker.Evaluate(Battery(10));
Assert.Equal(2, emissions.Count);
Assert.Equal(PowerStateTransitionTracker.TransitionAcToBattery, emissions[0].Transition);
Assert.Equal(PowerStateTransitionTracker.TransitionThresholdCrossed, emissions[1].Transition);
Assert.Equal(15, emissions[1].ThresholdPercent);
}
[Fact]
public void Multi_level_drop_emits_lowest_crossed_level_only()
{
var tracker = new PowerStateTransitionTracker();
tracker.Baseline(Battery(60));
var e = Assert.Single(tracker.Evaluate(Battery(10)));
Assert.Equal(15, e.ThresholdPercent);
// All levels are latched — recovering and re-draining emits nothing more.
tracker.Evaluate(Ac(90));
Assert.Single(tracker.Evaluate(Battery(45)), x => x.Transition == PowerStateTransitionTracker.TransitionAcToBattery);
Assert.Empty(tracker.Evaluate(Battery(13)));
}
[Fact]
public void Each_level_latches_once_per_lifetime()
{
var tracker = new PowerStateTransitionTracker();
tracker.Baseline(Battery(70));
Assert.Single(tracker.Evaluate(Battery(48))); // 50
Assert.Empty(tracker.Evaluate(Battery(47))); // still inside 50, no re-emit
Assert.Single(tracker.Evaluate(Battery(29))); // 30
Assert.Single(tracker.Evaluate(Battery(15))); // 15 (boundary: <= level)
Assert.Empty(tracker.Evaluate(Battery(5))); // ladder exhausted
}
[Fact]
public void Threshold_severity_ladder_is_info_warning_error()
{
var tracker = new PowerStateTransitionTracker();
tracker.Baseline(Battery(70));
var fifty = Assert.Single(tracker.Evaluate(Battery(50)));
Assert.Equal(EventSeverity.Info, fifty.Severity);
Assert.False(fifty.ImmediateUpload);
var thirty = Assert.Single(tracker.Evaluate(Battery(30)));
Assert.Equal(EventSeverity.Warning, thirty.Severity);
Assert.True(thirty.ImmediateUpload);
var fifteen = Assert.Single(tracker.Evaluate(Battery(15)));
Assert.Equal(EventSeverity.Error, fifteen.Severity);
Assert.True(fifteen.ImmediateUpload);
}
[Fact]
public void Null_battery_percent_skips_thresholds_but_ac_diff_still_works()
{
var tracker = new PowerStateTransitionTracker();
tracker.Baseline(Ac(null));
var e = Assert.Single(tracker.Evaluate(Battery(null, lifeMinutes: null)));
Assert.Equal(PowerStateTransitionTracker.TransitionAcToBattery, e.Transition);
Assert.Null(e.BatteryPercent);
Assert.Contains("battery level unknown", e.Message);
Assert.Empty(tracker.Evaluate(Battery(null, lifeMinutes: null)));
}
[Fact]
public void Probe_error_mid_run_preserves_state()
{
var tracker = new PowerStateTransitionTracker();
tracker.Baseline(Ac());
Assert.Empty(tracker.Evaluate(new PowerStateResult { ProbeError = "transient" }));
// State was not corrupted by the error — the real transition still diffs cleanly.
Assert.Single(tracker.Evaluate(Battery(60)));
}
[Fact]
public void Evaluate_without_baseline_latches_like_a_baseline()
{
var tracker = new PowerStateTransitionTracker();
var e = Assert.Single(tracker.Evaluate(Battery(12)));
// No phantom ac_to_battery from the implicit default state — only the threshold.
Assert.Equal(PowerStateTransitionTracker.TransitionThresholdCrossed, e.Transition);
}
[Fact]
public void Emission_cap_suppresses_and_flips_flag_once()
{
var tracker = new PowerStateTransitionTracker(maxEmissions: 4);
tracker.Baseline(Ac());
// Flapping power source: each edge is one emission.
for (var i = 0; i < 2; i++)
{
Assert.Single(tracker.Evaluate(Battery(80)));
Assert.Single(tracker.Evaluate(Ac(80)));
}
Assert.False(tracker.EmissionCapReached);
Assert.Empty(tracker.Evaluate(Battery(80)));
Assert.True(tracker.EmissionCapReached);
Assert.Empty(tracker.Evaluate(Ac(80)));
}
[Fact]
public void Message_carries_percent_and_transition_context()
{
var tracker = new PowerStateTransitionTracker();
tracker.Baseline(Ac(62));
var toBattery = Assert.Single(tracker.Evaluate(Battery(62)));
Assert.Contains("AC to battery", toBattery.Message);
Assert.Contains("battery 62%", toBattery.Message);
var backToAc = Assert.Single(tracker.Evaluate(Ac(63)));
Assert.Contains("back to AC", backToAc.Message);
Assert.Contains("charging", backToAc.Message);
}
}
}
@@ -0,0 +1,429 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// SystemTimelineTracker — clock steps (Kernel-General 1) and completed sleep episodes
/// (Power-Troubleshooter 1 / Kernel-Power 507) from the System channel. Drives the primitive
/// <see cref="SystemTimelineTracker.ProcessEvent"/> test seam (no real EventRecord, which is
/// abstract + Windows-only), mirroring the WindowsUpdateTracker test pattern.
/// </summary>
public sealed class SystemTimelineTrackerTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 8, 21, 10, 0, 0, DateTimeKind.Utc);
private readonly TempDirectory _tmp = new TempDirectory();
private readonly FakeSignalIngressSink _sink;
private readonly SystemTimelineTracker _tracker;
private long _nextRecordId = 1;
public SystemTimelineTrackerTests()
{
_sink = new FakeSignalIngressSink();
var post = new InformationalEventPost(_sink, new VirtualClock(At));
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_tracker = new SystemTimelineTracker(
sessionId: "sess-stl",
tenantId: "tenant-stl",
post: post,
logger: logger,
backfillEnabled: false,
stateDirectory: null); // in-memory dedup for most tests
}
public void Dispose() => _tmp.Dispose();
// -------------------------------------------------------------------
// Emission helpers
// -------------------------------------------------------------------
private void EmitClockChange(
DateTime? oldTime, DateTime? newTime, string? rawDeltaMs = null,
int reason = 1, long? recordId = null, bool isBackfill = false, DateTime? timeCreatedUtc = null)
{
var data = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
if (oldTime.HasValue) data["OldTime"] = oldTime.Value.ToString("o");
if (newTime.HasValue) data["NewTime"] = newTime.Value.ToString("o");
if (rawDeltaMs != null) data["TimeDeltaInMs"] = rawDeltaMs;
data["Reason"] = reason.ToString();
data["ProcessName"] = @"\Device\HarddiskVolume3\Windows\System32\svchost.exe";
data["ProcessID"] = "6892";
_tracker.ProcessEvent(
providerName: SystemTimelineTracker.ProviderKernelGeneral,
eventId: SystemTimelineTracker.EventId_ClockChange,
recordId: recordId ?? _nextRecordId++,
timeCreatedUtc: timeCreatedUtc ?? At,
eventData: data,
isBackfill: isBackfill);
}
private void EmitClassicResume(
DateTime sleepTime, DateTime wakeTime, string? effectiveState = "1",
string? wakeSourceText = null, long? recordId = null, bool isBackfill = false, DateTime? timeCreatedUtc = null)
{
var data = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
{
["SleepTime"] = sleepTime.ToString("o"),
["WakeTime"] = wakeTime.ToString("o"),
["WakeSourceType"] = "0",
};
if (effectiveState != null) data["EffectiveState"] = effectiveState;
if (wakeSourceText != null) data["WakeSourceText"] = wakeSourceText;
_tracker.ProcessEvent(
providerName: SystemTimelineTracker.ProviderPowerTroubleshooter,
eventId: SystemTimelineTracker.EventId_ClassicResume,
recordId: recordId ?? _nextRecordId++,
timeCreatedUtc: timeCreatedUtc ?? wakeTime,
eventData: data,
isBackfill: isBackfill);
}
private void EmitModernStandbyExit(
long sleepDurationUs, long durationUs, string powerStateAc = "true",
long? recordId = null, bool isBackfill = false, DateTime? timeCreatedUtc = null)
{
var data = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
{
["SleepDurationInUs"] = sleepDurationUs.ToString(),
["DurationInUs"] = durationUs.ToString(),
["Reason"] = "32",
["PowerStateAc"] = powerStateAc,
["BatteryRemainingCapacityOnExit"] = "34180",
};
_tracker.ProcessEvent(
providerName: SystemTimelineTracker.ProviderKernelPower,
eventId: SystemTimelineTracker.EventId_ModernStandbyExit,
recordId: recordId ?? _nextRecordId++,
timeCreatedUtc: timeCreatedUtc ?? At,
eventData: data,
isBackfill: isBackfill);
}
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> ByType(string eventType) =>
_sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == eventType).ToList();
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
// -------------------------------------------------------------------
// system_clock_changed
// -------------------------------------------------------------------
[Fact]
public void ClockStep_AboveThreshold_EmitsInfo_WithSignedDeltaAndNormalizedTimes()
{
var oldTime = At;
var newTime = At.AddSeconds(10);
EmitClockChange(oldTime, newTime, reason: 1);
var s = Assert.Single(ByType(Constants.EventTypes.SystemClockChanged));
Assert.Equal("Info", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("false", s.Payload![SignalPayloadKeys.ImmediateUpload]);
var data = Data(s);
Assert.Equal(10_000L, data["timeDeltaMs"]);
Assert.Equal("application_set", data["reasonText"]);
Assert.Equal(1, data["reason"]);
Assert.Equal(oldTime.ToString("o"), data["oldTime"]);
Assert.Equal(newTime.ToString("o"), data["newTime"]);
Assert.Equal(6892, data["processId"]);
Assert.Contains("forward", (string)s.Payload![SignalPayloadKeys.Message]);
Assert.Contains("svchost.exe", (string)s.Payload![SignalPayloadKeys.Message]);
}
[Theory]
[InlineData(1)]
[InlineData(1999)]
public void ClockMicroStep_BelowThreshold_Suppressed(int deltaMs)
{
EmitClockChange(At, At.AddMilliseconds(deltaMs));
Assert.Empty(ByType(Constants.EventTypes.SystemClockChanged));
}
[Fact]
public void ClockStep_Backward_EmitsNegativeDelta()
{
EmitClockChange(oldTime: At, newTime: At.AddSeconds(-3));
var s = Assert.Single(ByType(Constants.EventTypes.SystemClockChanged));
Assert.Equal(-3000L, Data(s)["timeDeltaMs"]);
Assert.Contains("backward", (string)s.Payload![SignalPayloadKeys.Message]);
}
[Fact]
public void ClockStep_AtLeastFiveMinutes_WarningAndImmediateUpload()
{
EmitClockChange(At, At.AddMinutes(5), reason: 2);
var s = Assert.Single(ByType(Constants.EventTypes.SystemClockChanged));
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
Assert.Equal("hardware_clock_sync", Data(s)["reasonText"]);
}
[Fact]
public void ClockStep_NoUsableTimesOrDelta_Skipped()
{
EmitClockChange(oldTime: null, newTime: null, rawDeltaMs: "not-a-number");
Assert.Empty(ByType(Constants.EventTypes.SystemClockChanged));
}
[Fact]
public void ClockStep_FallsBackToTimeDeltaInMs_WhenTimesUnparseable()
{
// Only the raw delta parses — the direction information of NewTime/OldTime is gone,
// but the magnitude still gates and emits.
EmitClockChange(oldTime: null, newTime: null, rawDeltaMs: "7200000");
var s = Assert.Single(ByType(Constants.EventTypes.SystemClockChanged));
Assert.Equal(7_200_000L, Data(s)["timeDeltaMs"]);
}
// -------------------------------------------------------------------
// system_sleep_episode — classic (Power-Troubleshooter 1)
// -------------------------------------------------------------------
[Fact]
public void ClassicResume_EffectiveState5_EmitsHibernateEpisode()
{
var sleepAt = At.AddHours(-2);
EmitClassicResume(sleepAt, At, effectiveState: "5", wakeSourceText: "Power Button");
var s = Assert.Single(ByType(Constants.EventTypes.SystemSleepEpisode));
Assert.Equal("Info", s.Payload![SignalPayloadKeys.Severity]);
var data = Data(s);
Assert.Equal("hibernate", data["kind"]);
Assert.Equal(7200L, data["durationSeconds"]);
Assert.Equal(sleepAt.ToString("o"), data["enteredAt"]);
Assert.Equal(At.ToString("o"), data["exitedAt"]);
Assert.Equal("Power Button", data["wakeSourceText"]);
Assert.Contains("hibernate", (string)s.Payload![SignalPayloadKeys.Message]);
}
[Fact]
public void ClassicResume_NonHibernateState_EmitsSleepKind()
{
EmitClassicResume(At.AddMinutes(-30), At, effectiveState: "4");
Assert.Equal("sleep", Data(Assert.Single(ByType(Constants.EventTypes.SystemSleepEpisode)))["kind"]);
}
[Fact]
public void ClassicResume_ShortEpisode_StillEmits()
{
// No duration floor for classic sleep: the event only exists for genuinely completed
// S3/S4 transitions, so a 30 s real sleep is still signal.
EmitClassicResume(At.AddSeconds(-30), At);
var s = Assert.Single(ByType(Constants.EventTypes.SystemSleepEpisode));
Assert.Equal(30L, Data(s)["durationSeconds"]);
}
// -------------------------------------------------------------------
// system_sleep_episode — Modern Standby (Kernel-Power 507)
// -------------------------------------------------------------------
[Fact]
public void ModernStandbyExit_EmitsEpisode_WithDerivedEnterTime()
{
// Values from a live 507 on this dev machine: 38 min scenario, ~38 min actual sleep.
EmitModernStandbyExit(sleepDurationUs: 2_324_095_807, durationUs: 2_330_201_652);
var s = Assert.Single(ByType(Constants.EventTypes.SystemSleepEpisode));
var data = Data(s);
Assert.Equal("modern_standby", data["kind"]);
Assert.Equal(2330L, data["durationSeconds"]);
Assert.Equal(2324L, data["sleepDurationSeconds"]);
Assert.Equal(true, data["onAcPower"]);
Assert.Equal(34180L, data["batteryRemainingCapacityOnExit"]);
Assert.Equal(At.ToString("o"), data["exitedAt"]);
Assert.Equal(At.AddMilliseconds(-2_330_201.652).ToString("o"), data["enteredAt"]);
Assert.Contains("Modern Standby", (string)s.Payload![SignalPayloadKeys.Message]);
}
[Fact]
public void ModernStandbyExit_SleepBelowFloor_Suppressed()
{
EmitModernStandbyExit(sleepDurationUs: 30_000_000, durationUs: 90_000_000); // 30 s sleep
Assert.Empty(ByType(Constants.EventTypes.SystemSleepEpisode));
}
[Fact]
public void ModernStandbyExit_ScreenOffWithoutSleep_Suppressed()
{
// SleepEntered=false scenarios carry SleepDurationInUs 0 — the single duration gate
// suppresses them without a separate branch.
EmitModernStandbyExit(sleepDurationUs: 0, durationUs: 6_106_696);
Assert.Empty(ByType(Constants.EventTypes.SystemSleepEpisode));
}
[Fact]
public void ModernStandbyExit_ScenarioShorterThanSleep_ClampsToSleepDuration()
{
EmitModernStandbyExit(sleepDurationUs: 120_000_000, durationUs: 0);
var data = Data(Assert.Single(ByType(Constants.EventTypes.SystemSleepEpisode)));
Assert.Equal(120L, data["durationSeconds"]);
Assert.Equal(120L, data["sleepDurationSeconds"]);
}
// -------------------------------------------------------------------
// Backfill semantics
// -------------------------------------------------------------------
[Fact]
public void BackfilledEvent_UsesEventTimeForTimelineTimestamp_AndFlagsData()
{
// A pre-agent standby episode (record created before the agent started) must land on
// the timeline where the device actually woke, not at agent start.
var wakeTime = At.AddHours(-3);
EmitClassicResume(wakeTime.AddHours(-1), wakeTime, isBackfill: true, timeCreatedUtc: wakeTime);
var s = Assert.Single(ByType(Constants.EventTypes.SystemSleepEpisode));
Assert.Equal(wakeTime, s.OccurredAtUtc);
Assert.Equal(true, Data(s)["backfilled"]);
}
// -------------------------------------------------------------------
// Dedup / watermark
// -------------------------------------------------------------------
[Fact]
public void SameRecordId_ProcessedTwice_EmitsOnce()
{
EmitClockChange(At, At.AddMinutes(1), recordId: 42);
EmitClockChange(At, At.AddMinutes(1), recordId: 42);
Assert.Single(ByType(Constants.EventTypes.SystemClockChanged));
}
[Fact]
public void BackfillEventWithLowerRecordId_AfterLiveEvent_IsStillEmitted()
{
// The live watcher is armed BEFORE backfill runs, so a live record with a higher
// RecordId can be processed first. An older, never-emitted backfill record must NOT be
// suppressed — pre-agent episodes are the whole point (high-water-mark trap).
EmitClockChange(At, At.AddMinutes(1), recordId: 100); // live, high
EmitClockChange(At, At.AddMinutes(2), recordId: 50, isBackfill: true); // backfill, older
Assert.Equal(2, ByType(Constants.EventTypes.SystemClockChanged).Count);
}
[Fact]
public void Watermark_PersistsAcrossTrackerInstances()
{
var post = new InformationalEventPost(_sink, new VirtualClock(At));
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
var clockData = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase)
{
["OldTime"] = At.ToString("o"),
["NewTime"] = At.AddMinutes(10).ToString("o"),
["Reason"] = "1",
};
var first = new SystemTimelineTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
first.ProcessEvent(SystemTimelineTracker.ProviderKernelGeneral, SystemTimelineTracker.EventId_ClockChange,
recordId: 500, timeCreatedUtc: At, eventData: clockData, isBackfill: false);
// A fresh tracker (agent restart) must load the watermark and skip the already-emitted
// record when the backfill re-reads it.
var second = new SystemTimelineTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
second.LoadWatermark();
second.ProcessEvent(SystemTimelineTracker.ProviderKernelGeneral, SystemTimelineTracker.EventId_ClockChange,
recordId: 500, timeCreatedUtc: At, eventData: clockData, isBackfill: true);
Assert.Single(ByType(Constants.EventTypes.SystemClockChanged));
}
[Fact]
public void UnknownProviderOrEventId_Ignored()
{
_tracker.ProcessEvent("Microsoft-Windows-Kernel-Power", eventId: 566, recordId: _nextRecordId++,
timeCreatedUtc: At, eventData: new Dictionary<string, string>(), isBackfill: false);
_tracker.ProcessEvent("Some-Other-Provider", eventId: 1, recordId: _nextRecordId++,
timeCreatedUtc: At, eventData: new Dictionary<string, string>(), isBackfill: false);
Assert.Empty(_sink.Posted);
}
// -------------------------------------------------------------------
// XPath + parsing
// -------------------------------------------------------------------
[Fact]
public void BuildXPath_ContainsAllThreeProviderIdPairs_AndNo506()
{
var xpath = SystemTimelineTracker.BuildXPath();
Assert.Contains("Provider[@Name='Microsoft-Windows-Kernel-General'] and EventID=1", xpath);
Assert.Contains("Provider[@Name='Microsoft-Windows-Power-Troubleshooter'] and EventID=1", xpath);
Assert.Contains("Provider[@Name='Microsoft-Windows-Kernel-Power'] and EventID=507", xpath);
Assert.DoesNotContain("506", xpath);
}
[Fact]
public void BuildBackfillXPath_AppendsTimediffClause()
{
var xpath = SystemTimelineTracker.BuildBackfillXPath(86_400_000);
Assert.Contains("TimeCreated[timediff(@SystemTime) <= 86400000]", xpath);
Assert.Contains("EventID=507", xpath);
}
[Fact]
public void ParseEventData_RealKernelGeneralPayload_YieldsAuthoritativeFields()
{
// Verbatim (values shortened) from a live Kernel-General 1 record on a 26220 dev machine.
const string xml =
"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>" +
"<System><Provider Name='Microsoft-Windows-Kernel-General'/><EventID>1</EventID></System>" +
"<EventData>" +
"<Data Name='NewTime'>2026-08-21T09:40:02.1987028Z</Data>" +
"<Data Name='OldTime'>2026-08-21T09:40:02.1973028Z</Data>" +
"<Data Name='TimeDeltaInMs'>1</Data>" +
"<Data Name='Reason'>1</Data>" +
"<Data Name='ProcessName'>\\Device\\HarddiskVolume3\\Windows\\System32\\svchost.exe</Data>" +
"<Data Name='ProcessID'>6892</Data>" +
"</EventData></Event>";
var data = SystemTimelineTracker.ParseEventData(xml);
Assert.Equal("1", data["TimeDeltaInMs"]);
Assert.Equal("2026-08-21T09:40:02.1987028Z", data["newtime"]); // case-insensitive keys
Assert.NotNull(SystemTimelineTracker.TryParseUtc(data["OldTime"]));
}
[Fact]
public void FormatDuration_CompactForms()
{
Assert.Equal("2h 05m", SystemTimelineTracker.FormatDuration(TimeSpan.FromMinutes(125)));
Assert.Equal("56m 12s", SystemTimelineTracker.FormatDuration(TimeSpan.FromSeconds(56 * 60 + 12)));
Assert.Equal("45s", SystemTimelineTracker.FormatDuration(TimeSpan.FromSeconds(45)));
}
[Fact]
public void ProcessLeafName_StripsDevicePath()
{
Assert.Equal("svchost.exe",
SystemTimelineTracker.ProcessLeafName(@"\Device\HarddiskVolume3\Windows\System32\svchost.exe"));
Assert.Null(SystemTimelineTracker.ProcessLeafName(null));
Assert.Null(SystemTimelineTracker.ProcessLeafName(@"\Device\HarddiskVolume3\"));
}
}
}
@@ -0,0 +1,126 @@
using System;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
{
public class DoGroupIdServiceTests
{
// -------------------------------------------------------------------
// ComputeGroupId — RealmJoin byte-layout compatibility
// -------------------------------------------------------------------
[Fact]
public void ComputeGroupId_MatchesHandComputedVector()
{
// Layout: {101,48,6} + {0,0,0} + {ip[1],0,ip[2],ip[3]} + mac — through new Guid(byte[])
// the first three groups print little-endian, the last eight bytes verbatim.
var guid = DoGroupIdService.ComputeGroupId(
new byte[] { 192, 168, 178, 1 },
new byte[] { 0x00, 0x11, 0x22, 0x33, 0x44, 0x55 });
Assert.Equal("00063065-0000-00a8-b201-001122334455", guid.ToString());
}
[Theory]
[InlineData(new byte[] { 10, 0, 0, 1 }, new byte[] { 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF })]
[InlineData(new byte[] { 192, 168, 1, 254 }, new byte[] { 0x00, 0x50, 0x56, 0x01, 0x02, 0x03 })]
[InlineData(new byte[] { 172, 16, 254, 3 }, new byte[] { 0xF0, 0x9F, 0xC2, 0x00, 0x00, 0x01 })]
public void ComputeGroupId_MatchesRealmJoinReferenceImplementation(byte[] ip, byte[] mac)
{
Assert.Equal(RealmJoinReferenceGuid(ip, mac), DoGroupIdService.ComputeGroupId(ip, mac));
}
/// <summary>Verbatim port of RealmJoin's ConvertToGuid — the compatibility contract.</summary>
private static Guid RealmJoinReferenceGuid(byte[] ipBytes, byte[] address)
{
byte[] prefix = { 101, 48, 6 };
var stuff = new byte[4];
stuff[0] = ipBytes[1];
stuff[2] = ipBytes[2];
stuff[3] = ipBytes[3];
return new Guid(prefix
.Concat(new byte[16 - prefix.Length - stuff.Length - address.Length])
.Concat(stuff)
.Concat(address)
.ToArray());
}
[Fact]
public void ComputeGroupId_RejectsNonIpv4AndBadMacLengths()
{
var ip4 = new byte[] { 10, 0, 0, 1 };
var mac = new byte[] { 1, 2, 3, 4, 5, 6 };
Assert.Throws<ArgumentException>(() => DoGroupIdService.ComputeGroupId(new byte[16], mac));
Assert.Throws<ArgumentException>(() => DoGroupIdService.ComputeGroupId(null!, mac));
Assert.Throws<ArgumentException>(() => DoGroupIdService.ComputeGroupId(ip4, new byte[8]));
Assert.Throws<ArgumentException>(() => DoGroupIdService.ComputeGroupId(ip4, null!));
}
[Fact]
public void ComputeGroupId_RegistryFormat_IsBracelessLowercase()
{
// RealmJoin writes SetValue(name, Guid) which stringifies via Guid.ToString() ("d"
// format) — our written value must be the identical string form.
var text = DoGroupIdService.ComputeGroupId(
new byte[] { 10, 1, 2, 3 }, new byte[] { 0xAB, 0xCD, 0xEF, 0x01, 0x02, 0x03 }).ToString();
Assert.DoesNotContain("{", text);
Assert.Equal(text.ToLowerInvariant(), text);
Assert.Equal(36, text.Length);
}
// -------------------------------------------------------------------
// DecideAction — ownership matrix
// -------------------------------------------------------------------
private const string Computed = "00063065-0000-00a8-b201-001122334455";
private const string Other = "11111111-2222-3333-4444-555555555555";
[Theory]
[InlineData(null)]
[InlineData("")]
public void DecideAction_NoExistingValue_Writes(string? existing)
{
Assert.Equal(DoGroupIdAction.Write, DoGroupIdService.DecideAction(existing, Computed, null));
}
[Fact]
public void DecideAction_ExistingEqualsComputed_IsNoOp()
{
Assert.Equal(DoGroupIdAction.NoOp, DoGroupIdService.DecideAction(Computed, Computed, null));
}
[Fact]
public void DecideAction_ExistingEqualsComputed_ToleratesBracesAndCase()
{
// A value written by another tool in "{...}"/uppercase form is still semantically ours.
var braced = "{" + Computed.ToUpperInvariant() + "}";
Assert.Equal(DoGroupIdAction.NoOp, DoGroupIdService.DecideAction(braced, Computed, null));
}
[Fact]
public void DecideAction_ExistingIsOurStaleValue_WritesUpdate()
{
// Gateway changed: the registry still holds the GUID we wrote for the old network.
Assert.Equal(DoGroupIdAction.Write, DoGroupIdService.DecideAction(Other, Computed, Other));
}
[Fact]
public void DecideAction_ForeignValue_Skips()
{
Assert.Equal(DoGroupIdAction.SkipForeign, DoGroupIdService.DecideAction(Other, Computed, null));
Assert.Equal(DoGroupIdAction.SkipForeign, DoGroupIdService.DecideAction(Other, Computed, Computed));
}
[Fact]
public void DecideAction_NonGuidGarbageExisting_SkipsAsForeign()
{
// Never overwrite a value we can't even parse — conservative by design.
Assert.Equal(DoGroupIdAction.SkipForeign, DoGroupIdService.DecideAction("not-a-guid", Computed, null));
}
}
}
@@ -1,5 +1,6 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.DecisionCore.Signals;
@@ -11,8 +12,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
internal sealed class FakeSignalIngressSink : ISignalIngressSink
{
private readonly List<PostedSignal> _posted = new List<PostedSignal>();
private readonly object _gate = new object();
public IReadOnlyList<PostedSignal> Posted => _posted;
/// <summary>
/// Snapshot, not the live list: producers post from timer / worker threads (e.g. the
/// parked-dwell tripwire) while tests poll this in a SpinWait loop. Enumerating the live
/// list across an Add throws "Collection was modified" — a rare CI-only failure
/// (run 32644425162) with no assertion text.
/// </summary>
public IReadOnlyList<PostedSignal> Posted
{
get { lock (_gate) return _posted.ToArray(); }
}
/// <summary>When non-null, <see cref="Post"/> throws instead of capturing.</summary>
public Exception? ThrowOnPost { get; set; }
@@ -27,7 +38,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
object? typedPayload = null)
{
if (ThrowOnPost != null) throw ThrowOnPost;
_posted.Add(new PostedSignal(kind, occurredAtUtc, sourceOrigin, evidence, payload, kindSchemaVersion, typedPayload));
lock (_gate)
_posted.Add(new PostedSignal(kind, occurredAtUtc, sourceOrigin, evidence, payload, kindSchemaVersion, typedPayload));
}
internal sealed class PostedSignal
@@ -0,0 +1,162 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Runtime;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
{
/// <summary>
/// PowerStateWatcherHost — no-battery/probe-error devices never arm, a low-battery baseline
/// emits its threshold event onto the single rail, and debounced ticks translate tracker
/// emissions into <c>power_state_change</c> events. WMI arming itself stays untested
/// (ConsoleBypassWatcher precedent); <c>StartCore(armWmi: false)</c> exercises everything else.
/// </summary>
public sealed class PowerStateWatcherHostTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 8, 21, 9, 0, 0, DateTimeKind.Utc);
private readonly TempDirectory _tmp = new TempDirectory();
private readonly FakeSignalIngressSink _sink = new FakeSignalIngressSink();
private readonly AgentLogger _logger;
private PowerStateResult _probeResult = new PowerStateResult { OnAcPower = true, HasBattery = true, BatteryPercent = 80 };
public PowerStateWatcherHostTests()
{
_logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
}
public void Dispose() => _tmp.Dispose();
private PowerStateWatcherHost Build() => new PowerStateWatcherHost(
sessionId: "sess-power",
tenantId: "tenant-power",
ingress: _sink,
clock: new VirtualClock(At),
logger: _logger,
probe: () => _probeResult);
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> PowerEvents() =>
_sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == Constants.EventTypes.PowerStateChange).ToList();
[Fact]
public void No_battery_emits_nothing()
{
_probeResult = new PowerStateResult { OnAcPower = true, HasBattery = false };
Build().StartCore(armWmi: false);
Assert.Empty(_sink.Posted);
}
[Fact]
public void Probe_error_emits_nothing()
{
_probeResult = new PowerStateResult { ProbeError = "GetSystemPowerStatus returned false" };
Build().StartCore(armWmi: false);
Assert.Empty(_sink.Posted);
}
[Fact]
public void Baseline_below_threshold_emits_threshold_event_with_immediate_upload()
{
_probeResult = new PowerStateResult { OnAcPower = false, HasBattery = true, BatteryPercent = 12, BatteryLifeMinutes = 34 };
Build().StartCore(armWmi: false);
var s = Assert.Single(PowerEvents());
Assert.Equal("Error", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
Assert.Contains("15%", s.Payload![SignalPayloadKeys.Message]);
var data = (IReadOnlyDictionary<string, object>)s.TypedPayload!;
Assert.Equal("threshold_crossed", data["transition"]);
Assert.Equal(15, data["thresholdPercent"]);
Assert.Equal(12, data["batteryPercent"]);
Assert.Equal(false, data["onAcPower"]);
Assert.Equal(34, data["batteryLifeMinutes"]);
}
[Fact]
public void Tick_after_unplug_emits_ac_to_battery_warning()
{
var host = Build();
host.StartCore(armWmi: false);
Assert.Empty(PowerEvents());
_probeResult = new PowerStateResult { OnAcPower = false, HasBattery = true, BatteryPercent = 79 };
host.TickForTest();
var s = Assert.Single(PowerEvents());
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
var data = (IReadOnlyDictionary<string, object>)s.TypedPayload!;
Assert.Equal("ac_to_battery", data["transition"]);
Assert.Equal(79, data["batteryPercent"]);
Assert.False(data.ContainsKey("thresholdPercent"));
// Replug: Info recovery event without immediate upload.
_probeResult = new PowerStateResult { OnAcPower = true, HasBattery = true, BatteryPercent = 79, IsCharging = true };
host.TickForTest();
Assert.Equal(2, PowerEvents().Count);
var back = PowerEvents()[1];
Assert.Equal("Info", back.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("false", back.Payload![SignalPayloadKeys.ImmediateUpload]);
Assert.Equal("battery_to_ac", ((IReadOnlyDictionary<string, object>)back.TypedPayload!)["transition"]);
}
[Fact]
public void Tick_with_unchanged_state_emits_nothing()
{
var host = Build();
host.StartCore(armWmi: false);
host.TickForTest();
host.TickForTest();
Assert.Empty(PowerEvents());
}
[Fact]
public void Stopped_host_ignores_ticks()
{
var host = Build();
host.StartCore(armWmi: false);
host.Stop();
_probeResult = new PowerStateResult { OnAcPower = false, HasBattery = true, BatteryPercent = 50 };
host.TickForTest();
Assert.Empty(PowerEvents());
}
[Fact]
public void Unknown_battery_percent_serializes_as_unknown()
{
var host = Build();
host.StartCore(armWmi: false);
_probeResult = new PowerStateResult { OnAcPower = false, HasBattery = true, BatteryPercent = null, BatteryLifeMinutes = null };
host.TickForTest();
var data = (IReadOnlyDictionary<string, object>)Assert.Single(PowerEvents()).TypedPayload!;
Assert.Equal("unknown", data["batteryPercent"]);
Assert.Equal("unknown", data["batteryLifeMinutes"]);
}
}
}
@@ -35,6 +35,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
// OS-eventlog forwarding — observability, not enrollment progress (session 8bc1180f:
// an EventID-100 burst must not keep the periodic collectors alive).
[InlineData("modern_deployment_log")]
// Environment observation — battery drain on an idle device must not reset idle clocks.
[InlineData("power_state_change")]
// Agent health / control / transport — not device/enrollment progress (P2).
[InlineData("collector_degraded")]
[InlineData("telemetry_upload_poisoned")]
@@ -0,0 +1,57 @@
#nullable enable
using System;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.DecisionCore.Engine;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
{
/// <summary>
/// SystemTimelineWatcherHost — thin ICollectorHost wrapper over SystemTimelineTracker.
/// Real event-log arming stays untested by convention (ConsoleBypassWatcher precedent);
/// these tests pin the lifecycle contract only.
/// </summary>
public sealed class SystemTimelineWatcherHostTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
public void Dispose() => _tmp.Dispose();
private SystemTimelineWatcherHost CreateHost() => new SystemTimelineWatcherHost(
sessionId: "sess-stl",
tenantId: "tenant-stl",
logger: new AgentLogger(_tmp.Path, AgentLogLevel.Info),
ingress: new FakeSignalIngressSink(),
clock: new VirtualClock(new DateTime(2026, 8, 21, 10, 0, 0, DateTimeKind.Utc)),
backfillLookbackMinutes: 0,
stateDirectory: _tmp.Path);
[Fact]
public void Stop_BeforeStart_DoesNotThrow()
{
using (var host = CreateHost())
{
host.Stop();
}
}
[Fact]
public void Dispose_Twice_DoesNotThrow()
{
var host = CreateHost();
host.Dispose();
host.Dispose();
}
[Fact]
public void Name_IsStable()
{
using (var host = CreateHost())
{
Assert.Equal("SystemTimelineTracker", host.Name);
}
}
}
}
@@ -182,6 +182,73 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
Assert.Equal("FromCache", data["outcome"]);
}
// ── Post-registration recovery (Phase 6.5, tenant aebdce78 audit 2026-08-24) ─
[Fact]
public void EmitFetchFailedIfAny_AfterRecovery_MarksRecoveredAndDemotesToInfo()
{
using var rig = new EmitterRig();
// Startup fetch failed (network link not up yet), then the post-registration
// recovery fetched ConfigVersion 39. The event must keep the original failure
// details AND mark the heal.
rig.RemoteConfig.SetFakeOutcome(
RemoteConfigFetchOutcome.UsedDefaults,
configVersion: 39, // CurrentConfig now holds the recovered config
attempts: 4,
failureType: "HttpRequestException",
failureMessage: "simulated dead network link",
recovered: true);
LifecycleEmitters.EmitRemoteConfigFetchFailedIfAny(
rig.Post, rig.AgentConfig, rig.RemoteConfig, rig.Logger);
var data = AssertSingleEmit(rig.Sink, expectedEventType: "remote_config_fetch_failed");
Assert.Equal("UsedDefaults", data["outcome"]); // startup failure record intact
Assert.Equal(4, data["attempts"]);
Assert.Equal(true, data["recovered"]);
Assert.Equal(39, data["recoveredConfigVersion"]);
Assert.Equal(nameof(EventSeverity.Info), rig.Sink.Posted[0].Payload!["severity"]);
}
[Fact]
public void EmitFetchFailedIfAny_WithoutRecovery_StaysWarning()
{
using var rig = new EmitterRig();
rig.RemoteConfig.SetFakeOutcome(
RemoteConfigFetchOutcome.UsedDefaults,
configVersion: 0,
attempts: 4,
failureType: "HttpRequestException",
failureMessage: "still down");
LifecycleEmitters.EmitRemoteConfigFetchFailedIfAny(
rig.Post, rig.AgentConfig, rig.RemoteConfig, rig.Logger);
var data = AssertSingleEmit(rig.Sink, expectedEventType: "remote_config_fetch_failed");
Assert.False(data.ContainsKey("recovered"));
Assert.False(data.ContainsKey("recoveredConfigVersion"));
Assert.Equal(nameof(EventSeverity.Warning), rig.Sink.Posted[0].Payload!["severity"]);
}
[Fact]
public void EmitAgentStarted_AfterRecovery_FlagsLiveConfig()
{
using var rig = new EmitterRig();
rig.RemoteConfig.SetFakeOutcome(
RemoteConfigFetchOutcome.UsedDefaults,
configVersion: 39,
recovered: true);
LifecycleEmitters.EmitAgentStarted(
rig.Post, rig.AgentConfig, rig.PreviousExit, "2.0.806", rig.RemoteConfig, rig.Logger);
var data = AssertSingleEmit(rig.Sink, expectedEventType: "agent_started");
Assert.Equal(39, data["configVersion"]);
Assert.Equal(true, data["remoteConfigFetched"]); // running on a live-fetched config
Assert.Equal("UsedDefaults", data["remoteConfigOutcome"]); // startup outcome stays honest
Assert.Equal(true, data["remoteConfigRecovered"]);
}
// ── Helpers ─────────────────────────────────────────────────────────
private static Dictionary<string, object> AssertSingleEmit(
@@ -258,8 +325,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
int attempts = 1,
string? failureType = null,
string? failureMessage = null,
int? authStatusCode = null)
int? authStatusCode = null,
bool recovered = false)
{
typeof(RemoteConfigService).GetProperty(nameof(RecoveredAfterFetchFailure))?.SetValue(this, recovered);
// CurrentConfig is set via the protected SetConfig path — easiest:
// a real FetchConfigAsync replacement that just stashes a canned config.
typeof(RemoteConfigService).GetMethod("SetConfig",
@@ -339,6 +339,114 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Runtime
Assert.False(StartupEnvironmentProbes.IsNtpWithinTolerance(null!));
}
// ================================================================= DO GroupId auto-set
// Builder/fingerprint tests only — TrySetGroupId itself does live registry + ARP I/O and
// must never run in a unit test (it would write HKLM on the build machine).
[Fact]
public void BuildDoGroupIdEvent_written_success_is_info_with_full_data()
{
var result = new DoGroupIdSetResult
{
Success = true,
Written = true,
GroupId = "00063065-0000-00a8-b201-001122334455",
PreviousValue = null,
GatewayIp = "192.168.178.1",
GatewayMac = "00-11-22-33-44-55",
};
var evt = StartupEnvironmentProbes.BuildDoGroupIdEvent(Config(), result);
Assert.Equal("do_group_id_auto_set", evt.EventType);
Assert.Equal(EventSeverity.Info, evt.Severity);
Assert.Equal("StartupEnvironmentProbes", evt.Source);
Assert.Equal("S1", evt.SessionId);
Assert.Equal("T1", evt.TenantId);
Assert.Contains("00063065-0000-00a8-b201-001122334455", evt.Message);
Assert.Equal("192.168.178.1", evt.Data["gatewayIp"]);
Assert.Equal("00-11-22-33-44-55", evt.Data["gatewayMac"]);
Assert.Equal(true, evt.Data["written"]);
Assert.Equal(true, evt.Data["success"]);
Assert.Equal(false, evt.Data["skipped"]);
}
[Fact]
public void BuildDoGroupIdEvent_noop_success_says_already_set()
{
var result = new DoGroupIdSetResult
{
Success = true,
Written = false,
GroupId = "00063065-0000-00a8-b201-001122334455",
};
var evt = StartupEnvironmentProbes.BuildDoGroupIdEvent(Config(), result);
Assert.Equal(EventSeverity.Info, evt.Severity);
Assert.Contains("already set", evt.Message);
Assert.Equal(false, evt.Data["written"]);
}
[Fact]
public void BuildDoGroupIdEvent_skip_is_info_with_reason()
{
var result = new DoGroupIdSetResult
{
Skipped = true,
SkipReason = "foreign_value_present",
PreviousValue = "11111111-2222-3333-4444-555555555555",
};
var evt = StartupEnvironmentProbes.BuildDoGroupIdEvent(Config(), result);
Assert.Equal(EventSeverity.Info, evt.Severity);
Assert.Contains("foreign_value_present", evt.Message);
Assert.Equal("foreign_value_present", evt.Data["skipReason"]);
Assert.Equal("11111111-2222-3333-4444-555555555555", evt.Data["previousValue"]);
}
[Fact]
public void BuildDoGroupIdEvent_error_is_warning()
{
var result = new DoGroupIdSetResult { Error = "readback mismatch" };
var evt = StartupEnvironmentProbes.BuildDoGroupIdEvent(Config(), result);
Assert.Equal(EventSeverity.Warning, evt.Severity);
Assert.Contains("readback mismatch", evt.Message);
Assert.Equal("readback mismatch", evt.Data["error"]);
}
[Fact]
public void DoGroupIdFingerprint_roundtrips_ownership_marker_for_success()
{
var fp = StartupEnvironmentProbes.ComputeDoGroupIdFingerprint(new DoGroupIdSetResult
{
Success = true,
GroupId = "00063065-0000-00a8-b201-001122334455",
});
Assert.Equal("written:00063065-0000-00a8-b201-001122334455", fp);
Assert.Equal("00063065-0000-00a8-b201-001122334455", StartupEnvironmentProbes.ExtractLastWrittenGroupId(fp));
}
[Fact]
public void DoGroupIdFingerprint_skip_carries_no_ownership_and_errors_never_dedupe()
{
var skipFp = StartupEnvironmentProbes.ComputeDoGroupIdFingerprint(new DoGroupIdSetResult
{
Skipped = true,
SkipReason = "mdm_policy_present",
});
Assert.Equal("skipped:mdm_policy_present", skipFp);
Assert.Null(StartupEnvironmentProbes.ExtractLastWrittenGroupId(skipFp));
// Errors: no fingerprint → the probe re-runs AND re-emits each failing start.
Assert.Null(StartupEnvironmentProbes.ComputeDoGroupIdFingerprint(new DoGroupIdSetResult { Error = "boom" }));
Assert.Null(StartupEnvironmentProbes.ExtractLastWrittenGroupId(null));
}
[Fact]
public async Task RunAsync_skips_geo_timezone_and_ntp_once_all_succeeded_only_power_state_emits()
{
@@ -0,0 +1,146 @@
using System;
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
{
/// <summary>
/// Pins <see cref="EntraDeviceCertHelper.ParseGuidFromExtension"/>, the pure byte-level part of the
/// cert probe. Both encodings Microsoft actually ships are covered: the ASN.1 OCTET STRING wrapper
/// used by the MS-Organization-Access TenantId/DeviceId extensions (and Intune's
/// <c>1.2.840.113556.5.6</c> / <c>5.14</c>), and the BARE 16-byte payload used by Intune's DeviceId
/// extension <c>1.2.840.113556.5.4</c> — the latter was silently returning <c>null</c> before.
/// The store-iteration shell around it needs a real machine cert store and is not unit-tested.
/// </summary>
public sealed class EntraDeviceCertHelperParseTests
{
// Synthetic — never a real tenant/device id. ToByteArray() gives the MS little-endian layout
// that both cert encodings carry, so every expectation below is derived, not hand-transcribed.
private static readonly Guid Sample = new Guid("11223344-5566-7788-99aa-bbccddeeff00");
private static byte[] Wrap(byte[] payload, params byte[] header)
{
var buf = new byte[header.Length + payload.Length];
Buffer.BlockCopy(header, 0, buf, 0, header.Length);
Buffer.BlockCopy(payload, 0, buf, header.Length, payload.Length);
return buf;
}
[Fact]
public void OctetString_ShortForm_Parses()
{
var raw = Wrap(Sample.ToByteArray(), 0x04, 0x10);
Assert.Equal(Sample, EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void OctetString_LongForm_Parses()
{
var raw = Wrap(Sample.ToByteArray(), 0x04, 0x81, 0x10);
Assert.Equal(Sample, EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
/// <summary>
/// The regression this class exists for: Intune's <c>1.2.840.113556.5.4</c> extension value is
/// exactly 16 bytes with no <c>04 10</c> prefix, so the strict OCTET-STRING parser bailed on the
/// tag check and the DeviceId was unreadable.
/// </summary>
[Fact]
public void BareSixteenBytes_Parses()
{
var raw = Sample.ToByteArray();
Assert.Equal(Sample, EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
/// <summary>
/// A bare GUID may legitimately start with 0x04 (the OCTET STRING tag). That must not be
/// mistaken for a wrapper: a wrapped 16-byte GUID is always >= 18 bytes, so the two forms are
/// disjoint by length and the bare path has to win here.
/// </summary>
[Fact]
public void BareSixteenBytes_StartingWithOctetStringTag_StillParses()
{
var guid = new Guid("aabbcc04-0110-4020-8030-405060708090");
var raw = guid.ToByteArray();
Assert.Equal(0x04, raw[0]); // guard: the byte layout really does start with the ASN.1 tag
Assert.Equal(16, raw.Length);
Assert.Equal(guid, EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void OctetString_IsLittleEndianMsLayout()
{
// 04 10 + the canonical MS binary layout of {11223344-...} => first group byte-reversed.
var raw = new byte[]
{
0x04, 0x10,
0x44, 0x33, 0x22, 0x11,
0x66, 0x55,
0x88, 0x77,
0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF, 0x00
};
Assert.Equal(Sample, EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void Null_ReturnsNull()
{
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(null));
}
[Fact]
public void Empty_ReturnsNull()
{
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(new byte[0]));
}
[Theory]
[InlineData(15)]
[InlineData(17)]
public void BarePayload_OfWrongLength_ReturnsNull(int length)
{
var raw = new byte[length];
for (var i = 0; i < length; i++) raw[i] = (byte)(0x40 + i); // no 0x04 tag, no valid wrapper
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void NonOctetStringTag_ReturnsNull()
{
// e.g. Intune's 1.2.840.113556.5.15 => ASN.1 INTEGER 2.
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(new byte[] { 0x02, 0x01, 0x02 }));
}
[Fact]
public void OctetString_WithNonGuidContentLength_ReturnsNull()
{
var raw = Wrap(new byte[8], 0x04, 0x08);
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void OctetString_TruncatedContent_ReturnsNull()
{
// Declares 16 content bytes but only carries 10.
var raw = Wrap(new byte[10], 0x04, 0x10);
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void OctetString_LongFormWithOversizedLengthHeader_ReturnsNull()
{
// 0x85 => 5 length bytes, above the 4 the parser accepts.
var raw = Wrap(Sample.ToByteArray(), 0x04, 0x85, 0x00, 0x00, 0x00, 0x00, 0x10);
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
}
}
@@ -97,7 +97,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
}
[Fact]
public async Task Fails_after_five_exceptions_without_auth_error()
public async Task Fails_after_six_exceptions_without_auth_error()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
@@ -111,8 +111,50 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
backoffDelay: _ => Task.CompletedTask);
Assert.Equal(SessionRegistrationOutcome.Failed, result.Outcome);
Assert.Equal(5, apiClient.CallCount);
Assert.Contains("transient attempt 5", result.ErrorMessage);
Assert.Equal(6, apiClient.CallCount);
Assert.Contains("transient attempt 6", result.ErrorMessage);
}
[Fact]
public async Task Waits_for_network_link_before_first_attempt()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
var config = NewConfig();
var order = new System.Collections.Generic.List<string>();
var apiClient = new FakeApiClient(_ =>
{
order.Add("register");
return new RegisterSessionResponse { Success = true, SessionId = config.SessionId };
});
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
apiClient, config, agentVersion: "0.0.0", logger: logger,
backoffDelay: _ => Task.CompletedTask,
networkLinkWait: _ => { order.Add("link-wait"); return Task.CompletedTask; });
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
Assert.Equal(new[] { "link-wait", "register" }, order);
}
[Fact]
public async Task Network_link_wait_failure_does_not_block_registration()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
var config = NewConfig();
var apiClient = new FakeApiClient(_ =>
new RegisterSessionResponse { Success = true, SessionId = config.SessionId });
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
apiClient, config, agentVersion: "0.0.0", logger: logger,
backoffDelay: _ => Task.CompletedTask,
networkLinkWait: _ => throw new InvalidOperationException("probe broken"));
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
Assert.Equal(1, apiClient.CallCount);
}
[Fact]
@@ -245,11 +245,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
}
[Fact]
public void Terminal_to_active_retry_rearms_CompletedAt_and_final_terminal_covers_total_occupancy()
public void Terminal_to_active_retry_rearms_BothEndpoints_and_final_terminal_covers_last_attempt()
{
// Audit Q1: the IME retry path (Error -> Downloading -> Installed is permitted by
// AppPackageState.UpdateState). Pre-fix, CompletedAt froze at the first Error, so
// the final Installed payload carried the first failure's completedAt/duration.
// Audit Q1 + 2026-08 attempt-duration change: the IME retry path
// (Error -> Downloading -> Installed is permitted by AppPackageState.UpdateState).
// A terminal->active transition re-arms BOTH endpoints, so the final Installed
// payload describes the retry ATTEMPT (matching the backend's AppInstallSummaries
// semantics) — not the first-start -> last-terminal span across attempts.
using var f = new ImeLogTrackerAdapterFixture(T0);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-R", 0);
@@ -261,23 +263,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
adapter.TriggerAppStateFromTest(app, AppInstallationState.Error, AppInstallationState.Downloading);
// Re-armed while active again: the retry's download event carries no completedAt,
// and the FIRST start stamp is kept.
// and the start stamp is the RETRY's re-entry time (fresh attempt window).
Assert.Null(adapter.AppTimings["app-R"].CompletedAtUtc);
var retryDownload = f.InfoEvent(SharedEventTypes.AppDownloadStarted);
Assert.False(retryDownload.Payload!.ContainsKey("completedAt"));
Assert.False(retryDownload.Payload.ContainsKey("durationSeconds"));
Assert.Equal(T0.ToString("o"), retryDownload.Payload["startedAt"]);
Assert.Equal(T0.AddSeconds(15).ToString("o"), retryDownload.Payload["startedAt"]);
f.Clock.Advance(TimeSpan.FromSeconds(20));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Downloading, AppInstallationState.Installed);
var timing = adapter.AppTimings["app-R"];
Assert.Equal(T0, timing.StartedAtUtc);
Assert.Equal(T0.AddSeconds(15), timing.StartedAtUtc);
Assert.Equal(T0.AddSeconds(35), timing.CompletedAtUtc);
Assert.Equal(35.0, timing.DurationSeconds);
Assert.Equal(20.0, timing.DurationSeconds);
var completed = Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Equal("35.00", completed.Payload!["durationSeconds"]);
Assert.Equal("20.00", completed.Payload!["durationSeconds"]);
Assert.Equal(T0.AddSeconds(35).ToString("o"), completed.Payload["completedAt"]);
}
@@ -163,5 +163,45 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
Assert.Single(f.DecisionSignals(DecisionSignalKind.ImeUserSessionCompleted));
Assert.Single(f.InfoEvents(SharedEventTypes.ImeUserSessionCompleted));
}
// ==================================================== zero-app evidence label ====
[Fact]
public void Zero_app_user_session_labels_the_evidence_on_signal_and_timeline()
{
// Sessions 81daa77f / 75d6ae8e: with no user-targeted apps IME never writes its
// "Completed user session" line at all, so the verdict rests on a different
// observation. Whoever reads the timeline later must be able to tell which.
using var f = new ImeLogTrackerAdapterFixture();
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.SeedCurrentPhaseForTesting("AccountSetup");
f.Tracker.SeedUserSessionZeroAppsForTesting();
adapter.TriggerUserSessionCompletedFromTest();
Assert.Single(f.DecisionSignals(DecisionSignalKind.ImeUserSessionCompleted));
var evt = Assert.Single(f.InfoEvents(SharedEventTypes.ImeUserSessionCompleted));
Assert.Equal(
ImeLogTrackerAdapter.EvidenceZeroUserApps,
evt.Payload!["evidence"]);
}
[Fact]
public void Ordinary_user_session_completion_keeps_the_default_evidence_label()
{
// Mutation proof for the branch above.
using var f = new ImeLogTrackerAdapterFixture();
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
f.Tracker.SeedCurrentPhaseForTesting("AccountSetup");
AddApp(f.Tracker, "app-done", AppIntent.Install, AppInstallationState.Installed);
adapter.TriggerUserSessionCompletedFromTest();
var evt = Assert.Single(f.InfoEvents(SharedEventTypes.ImeUserSessionCompleted));
Assert.Equal(
ImeLogTrackerAdapter.EvidenceUserSessionCompleted,
evt.Payload!["evidence"]);
}
}
}
@@ -107,6 +107,31 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Transport
Assert.False(req.Headers.ContainsKey("X-Bootstrap-Token"));
}
[Fact]
public async Task UploadBatchAsync_stamps_send_time_header_per_attempt()
{
// P14: X-Send-Time-Utc carries the device clock at the SEND moment, round-trip
// ISO-8601, one value per request. A retry is a new send moment — the header is
// stamped inside UploadBatchAsync, so each attempt gets a fresh value.
using var rig = new Rig();
rig.Handler.QueueStatus(HttpStatusCode.ServiceUnavailable);
rig.Handler.QueueStatus(HttpStatusCode.OK);
var before = DateTime.UtcNow;
await rig.Sut.UploadBatchAsync(new[] { NewEventItem(1) }, CancellationToken.None);
await rig.Sut.UploadBatchAsync(new[] { NewEventItem(1) }, CancellationToken.None);
var after = DateTime.UtcNow;
Assert.Equal(2, rig.Handler.Captured.Count);
foreach (var req in rig.Handler.Captured)
{
Assert.True(req.TryGetHeader("X-Send-Time-Utc", out var raw));
var parsed = DateTime.Parse(raw, null, System.Globalization.DateTimeStyles.RoundtripKind);
Assert.Equal(DateTimeKind.Utc, parsed.Kind);
Assert.InRange(parsed, before.AddSeconds(-1), after.AddSeconds(1));
}
}
// ============================================================ Compression (bandwidth)
[Fact]
@@ -52,6 +52,11 @@
carries the V2 assembly namespace so GatherRuleGuards locates it under
"AutopilotMonitor.Agent.V2.Core.Resources.guardrails.json". -->
<EmbeddedResource Include="..\..\..\rules\guardrails.json" LogicalName="AutopilotMonitor.Agent.V2.Core.Resources.guardrails.json" />
<!-- Built-in fallback rule catalogs (BuiltInConfigCatalog): the SAME CI-combined
rules/dist artifacts the backend embeds and serves. Snapshotted per agent build so
the no-config/no-cache fallback still runs full IME tracking instead of nothing. -->
<EmbeddedResource Include="..\..\..\rules\dist\ime-log-patterns.json" LogicalName="AutopilotMonitor.Agent.V2.Core.Resources.ime-log-patterns.json" />
<EmbeddedResource Include="..\..\..\rules\dist\gather-rules.json" LogicalName="AutopilotMonitor.Agent.V2.Core.Resources.gather-rules.json" />
</ItemGroup>
<ItemGroup>
@@ -109,6 +109,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
/// </summary>
public bool EnableTimezoneAutoSet { get; set; } = false;
/// <summary>
/// Whether to set the Delivery Optimization group ID (DOGroupId policy value) from a
/// deterministic network fingerprint (default gateway IP + MAC, RealmJoin-compatible).
/// Existing DOGroupId/DOGroupIdSource policies are never overwritten.
/// Default: false
/// </summary>
public bool EnableDoGroupIdAutoSet { get; set; } = false;
/// <summary>
/// Continue-Anyway observation mode (operator-set tenant opt-in, delivered via
/// remote config): stamped into the esp_config_detected decision-signal payload so
@@ -254,6 +262,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
/// </summary>
public bool UnrestrictedMode { get; set; } = false;
/// <summary>
/// Mirror of <c>AnalyzerConfiguration.EnableRealmJoinWatcher</c> (remote, per tenant).
/// Gates the RealmJoin built-in sections of the diagnostics package — the package
/// service only sees this configuration, not the remote response. The RealmJoin host
/// itself keeps reading the remote Analyzers block. Default: false.
/// </summary>
public bool EnableRealmJoinWatcher { get; set; } = false;
/// <summary>
/// Sanitized command-line arguments for the agent_started event.
/// Secrets (e.g. bootstrap token) are redacted before assignment.
@@ -0,0 +1,74 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Models;
using Newtonsoft.Json;
namespace AutopilotMonitor.Agent.V2.Core.Configuration
{
/// <summary>
/// Compiled-in fallback copies of the global rule catalogs, embedded from the same
/// CI-generated <c>rules/dist</c> artifacts the backend seeds and serves — single
/// source by construction: each agent release snapshots the catalog state of its own
/// build, so the offline fallback can never drift further than the running agent
/// version itself (no manually maintained default to forget).
/// <para>
/// Consumed only by <see cref="RemoteConfigService"/>'s built-in default config,
/// i.e. when a session has NO live config and NO cache. Before this existed, that
/// path shipped an empty <c>ImeLogPatterns</c> list, which left the entire IME
/// tracking pipeline (app installs, platform/remediation scripts, health-script
/// compliance, IME ESP-phase signals) dead for the whole session — the core of
/// enrollment monitoring. A live fetch (startup, post-registration recovery or
/// rotate_config) always wins over this catalog.
/// </para>
/// Fail-soft: any load/parse error returns an empty list (the pre-catalog behaviour).
/// </summary>
internal static class BuiltInConfigCatalog
{
/// <summary>Enabled built-in IME log patterns from the embedded catalog.</summary>
public static List<ImeLogPattern> GetEnabledImeLogPatterns(AgentLogger logger)
=> LoadEnabled<ImeLogPattern>("ime-log-patterns.json", p => p.Enabled, logger);
/// <summary>Enabled built-in gather rules from the embedded catalog.</summary>
public static List<GatherRule> GetEnabledGatherRules(AgentLogger logger)
=> LoadEnabled<GatherRule>("gather-rules.json", r => r.Enabled, logger);
private static List<T> LoadEnabled<T>(string name, Func<T, bool> isEnabled, AgentLogger logger)
{
try
{
var assembly = typeof(BuiltInConfigCatalog).Assembly;
using (var stream = assembly.GetManifestResourceStream(
"AutopilotMonitor.Agent.V2.Core.Resources." + name))
{
if (stream == null)
{
logger?.Warning($"Built-in config catalog resource '{name}' not found — fallback runs without it.");
return new List<T>();
}
using (var reader = new StreamReader(stream))
{
var wrapper = JsonConvert.DeserializeObject<RulesFile<T>>(reader.ReadToEnd());
var all = wrapper?.Rules ?? new List<T>();
return all.Where(isEnabled).ToList();
}
}
}
catch (Exception ex)
{
logger?.Warning($"Built-in config catalog '{name}' failed to load ({ex.Message}) — fallback runs without it.");
return new List<T>();
}
}
/// <summary>Shape of the CI-combined catalog files: <c>{ "rules": [...] }</c>.</summary>
private sealed class RulesFile<T>
{
[JsonProperty("rules")]
public List<T> Rules { get; set; }
}
}
}
@@ -94,12 +94,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
// a tenant admin can clear a previously-applied custom NTP server.
agentConfig.NtpServer = remoteConfig.NtpServer;
agentConfig.EnableTimezoneAutoSet = remoteConfig.EnableTimezoneAutoSet;
agentConfig.EnableDoGroupIdAutoSet = remoteConfig.EnableDoGroupIdAutoSet;
// ---------------- Diagnostics — remote wins (including DiagnosticsUploadMode="Off"
// which must clear any CLI-initial default).
agentConfig.DiagnosticsUploadEnabled = remoteConfig.DiagnosticsUploadEnabled;
agentConfig.DiagnosticsUploadMode = remoteConfig.DiagnosticsUploadMode;
agentConfig.DiagnosticsLogPaths = remoteConfig.DiagnosticsLogPaths ?? new List<DiagnosticsLogPath>();
// The diagnostics package gates its RealmJoin sections on the tenant's watcher
// toggle but never sees the remote response — mirror the one Analyzers knob here.
// The remaining Analyzers flags flow untouched to DefaultComponentFactory.
agentConfig.EnableRealmJoinWatcher = remoteConfig.Analyzers?.EnableRealmJoinWatcher ?? false;
agentConfig.SendTraceEvents = remoteConfig.SendTraceEvents;
@@ -42,6 +42,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
TimeSpan.FromSeconds(60),
};
/// <summary>
/// Upper bound for the pre-fetch network-link wait on the initial startup fetch.
/// Deliberately longer than the registration gate's 15s: the fetch runs FIRST after a
/// boot-time relaunch, when Wi-Fi may still be associating (802.1X, OOBE), and a fetch
/// burned into a dead link strands the whole session on built-in defaults — tenant aebdce78
/// audit 2026-08-24: ~50% of sessions lost their tenant config exactly this way.
/// </summary>
internal static readonly TimeSpan InitialFetchNetworkLinkWaitMax = TimeSpan.FromSeconds(60);
/// <summary>
/// Gets the current configuration (thread-safe)
/// </summary>
@@ -76,6 +85,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
/// <summary>HTTP status for auth failures (401/403), null otherwise.</summary>
public int? LastFetchAuthStatusCode { get; private set; }
/// <summary>
/// True once <see cref="TryRecoverConfigAsync"/> replaced a failed startup fetch with a
/// live config. The <c>LastFetch*</c> properties intentionally keep describing the failed
/// startup fetch so the wire event can report both the original failure and the recovery.
/// </summary>
public bool RecoveredAfterFetchFailure { get; private set; }
public RemoteConfigService(BackendApiClient apiClient, string tenantId, AgentLogger logger, EmergencyReporter emergencyReporter = null, DistressReporter distressReporter = null, AuthFailureTracker authFailureTracker = null)
{
_apiClient = apiClient ?? throw new ArgumentNullException(nameof(apiClient));
@@ -116,6 +132,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
LastFetchFailureMessage = null;
LastFetchAuthStatusCode = null;
// Initial startup fetch only: give a still-associating NIC a chance to come up
// before the first attempt. Without this the retry budget (10/30/60s) is burned
// into a provably dead link — the agent starts at boot, Wi-Fi associates minutes
// later, and the session runs on defaults although the backend was never the
// problem. The rotate_config path (retryOnTransientErrors:false) runs mid-session
// on a proven link and must stay latency-bounded, so it skips the gate.
if (retryOnTransientErrors)
{
try { await WaitForNetworkLinkAsync().ConfigureAwait(false); }
catch (Exception ex) { _logger.Debug($"Pre-fetch network wait failed ({ex.Message}) — fetching anyway."); }
}
var maxAttempts = retryOnTransientErrors ? InitialFetchRetryBackoff.Length + 1 : 1;
Exception lastException = null;
@@ -203,14 +231,63 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
return cached;
}
// Fall back to defaults
_logger.Info("No cached config available, using defaults (all optional collectors disabled)");
// Fall back to built-in defaults (collector/analyzer defaults from the shared
// model initializers + the compiled-in rule catalogs).
var defaults = CreateDefaultConfig();
_logger.Info($"No cached config available, using built-in defaults (ConfigVersion=0, " +
$"{defaults.ImeLogPatterns?.Count ?? 0} built-in IME patterns, " +
$"{defaults.GatherRules?.Count ?? 0} built-in gather rules)");
SetConfig(defaults);
LastFetchOutcome = RemoteConfigFetchOutcome.UsedDefaults;
return defaults;
}
/// <summary>
/// Single-attempt recovery fetch for a session whose startup fetch failed. Called right
/// after session registration succeeded — registration proves the backend is reachable,
/// so the config that Phase 4 could not get is one cheap call away instead of the whole
/// session being stranded on cache/defaults (only a manual <c>rotate_config</c> action
/// could rescue it before this existed).
/// <para>
/// Unlike <see cref="FetchConfigAsync"/> this does NOT reset the <c>LastFetch*</c>
/// observability state — those keep describing the failed startup fetch so
/// <c>remote_config_fetch_failed</c> can report the original failure AND the recovery.
/// Returns the recovered config, or null when there is nothing to recover (startup fetch
/// succeeded / never ran) or the recovery attempt itself failed. Never throws.
/// </para>
/// </summary>
public async Task<AgentConfigResponse> TryRecoverConfigAsync()
{
if (LastFetchOutcome != RemoteConfigFetchOutcome.FromCache &&
LastFetchOutcome != RemoteConfigFetchOutcome.UsedDefaults)
return null;
try
{
_logger.Info("Retrying remote config fetch after successful session registration...");
var config = await CallBackendAsync().ConfigureAwait(false);
if (config == null)
{
_logger.Warning("Config recovery fetch returned empty response — keeping fallback config.");
return null;
}
_logger.Info($"Remote config recovered: ConfigVersion={config.ConfigVersion}, " +
$"GatherRules={config.GatherRules?.Count ?? 0}");
LogCollectorSettings(config.Collectors);
SetConfig(config);
CacheConfig(config);
_authFailureTracker?.RecordSuccess();
RecoveredAfterFetchFailure = true;
return config;
}
catch (Exception ex)
{
_logger.Warning($"Config recovery fetch failed ({ex.Message}) — keeping fallback config.");
return null;
}
}
private void SetConfig(AgentConfigResponse config)
{
lock (_configLock)
@@ -319,6 +396,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
private AgentConfigResponse CreateDefaultConfig()
{
// Everything not set here falls to the shared AgentConfigResponse property
// initializers — the same values the backend uses as its `??` fallbacks, so the
// scalar knobs cannot drift. The rule catalogs are the one thing initializers
// cannot supply: without them the whole IME tracking pipeline is dead, so they
// come from the compiled-in snapshot of the served catalogs (BuiltInConfigCatalog).
return new AgentConfigResponse
{
ConfigVersion = 0,
@@ -330,8 +412,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
EnableGatherRuleDebugLog = false,
EnableEspContinueAnywayObservation = false,
Collectors = CollectorConfiguration.CreateDefault(),
GatherRules = new System.Collections.Generic.List<GatherRule>(),
ImeLogPatterns = new System.Collections.Generic.List<ImeLogPattern>()
GatherRules = BuiltInConfigCatalog.GetEnabledGatherRules(_logger),
ImeLogPatterns = BuiltInConfigCatalog.GetEnabledImeLogPatterns(_logger)
};
}
@@ -358,6 +440,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
protected virtual Task DelayBetweenAttemptsAsync(TimeSpan delay)
=> Task.Delay(delay);
/// <summary>
/// Awaits the pre-fetch network-link gate (initial startup fetch only). Production
/// polls the link-level signal via <see cref="NetworkLinkWait"/> bounded by
/// <see cref="InitialFetchNetworkLinkWaitMax"/>; tests subclass to record the call
/// and return immediately.
/// </summary>
protected virtual Task WaitForNetworkLinkAsync()
=> NetworkLinkWait.WaitAsync(_logger, InitialFetchNetworkLinkWaitMax, "Remote config fetch");
public void Dispose()
{
// Nothing to dispose
@@ -10,10 +10,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// the <c>app_tracking_summary</c> terminal event.
/// <para>
/// <b>StartedAtUtc</b> is recorded on the first lifecycle transition into Downloading /
/// Installing / InProgress and never overwritten. <b>CompletedAtUtc</b> is recorded on a
/// transition to a terminal state (Installed / Skipped / Postponed / Error) and re-armed
/// to <c>null</c> when the app goes terminal → active again (IME retry / re-evaluation,
/// audit Q1) — so the final terminal event carries first-start → last-terminal.
/// Installing / InProgress. <b>CompletedAtUtc</b> is recorded on a transition to a
/// terminal state (Installed / Skipped / Postponed / Error). When the app goes
/// terminal → active again (IME retry / re-evaluation pass, audit Q1) BOTH endpoints
/// re-arm — CompletedAtUtc to <c>null</c>, StartedAtUtc to the re-entry time — so the
/// final terminal event carries the status-defining ATTEMPT's window (2026-08
/// attempt-duration change, matching the backend's AppInstallSummaries semantics),
/// never the first-start → last-terminal span across install passes.
/// </para>
/// </summary>
public sealed class AppInstallTiming
@@ -12,8 +12,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// <summary>The writer declared its offset in the line ("+480"). Authoritative.</summary>
Bias = 1,
/// <summary>Offset measured from a freshly appended line. See <see cref="CmTraceOffsetCalibrator"/>.</summary>
/// <summary>
/// A per-FILE measured offset was applied. Retired by the 2026-08-20 revert (04b1a7c6):
/// one offset per file is wrong when a file holds lines from two writer eras. Kept so
/// stored events from agent 2.0.1410 remain interpretable.
/// </summary>
Calibrated = 2,
/// <summary>
/// The line anchored ITSELF: it was read in a pass whose lines are provably fresh
/// (written since the previous poll), so its own distance to the agent clock, rounded
/// to the 15-minute offset grid, IS the writer's offset — no cross-line state involved.
/// Immune to interleaved writer eras by construction. See
/// <see cref="CmTraceOffsetCalibrator.TryMeasureOffset"/>.
/// </summary>
LineAnchored = 3,
}
/// <summary>
@@ -98,24 +111,51 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// <param name="localTimestamp">The line's local time exactly as written (Kind is ignored).</param>
/// <param name="agentUtcNow">The agent's UTC clock at parse time.</param>
/// <returns><c>true</c> when the anchor was accepted and the offset updated.</returns>
public bool TryCalibrate(string sourceKey, DateTime localTimestamp, DateTime agentUtcNow)
/// <summary>
/// The pure measurement: the distance between a FRESH line's local time and the agent's
/// UTC clock, rounded to the 15-minute offset grid, is the writer's UTC offset.
/// <para>
/// Freshness is the caller's obligation and is what makes this correct — for a line
/// written within the last poll interval, <c>local - now</c> differs from the writer's
/// true offset only by seconds, which the grid rounding absorbs. Returns <c>false</c>
/// when the residual exceeds <see cref="MaxGridResidual"/> (the line is not fresh, or
/// the writer's clock is broken) or the rounded value is not a real timezone offset.
/// </para>
/// <para>
/// Known, accepted edge: a freshly WRITTEN line carrying an old REPLAYED timestamp whose
/// age happens to sit within the residual of an exact grid multiple measures that age as
/// an "offset" — the line then resolves to roughly the read time instead of its replayed
/// past. The error is bounded by construction (an anchored line always resolves to
/// now ± <see cref="MaxGridResidual"/>), the case needs an age of exactly N×15 min ± 2 min,
/// and sub-24h replays are treated as current by the historic-replay guard anyway.
/// </para>
/// </summary>
public static bool TryMeasureOffset(DateTime localTimestamp, DateTime agentUtcNow, out TimeSpan offset)
{
if (string.IsNullOrEmpty(sourceKey)) return false;
var candidate = DateTime.SpecifyKind(localTimestamp, DateTimeKind.Unspecified)
- DateTime.SpecifyKind(agentUtcNow, DateTimeKind.Unspecified);
var gridUnits = Math.Round(candidate.TotalMinutes / OffsetGrid.TotalMinutes,
MidpointRounding.AwayFromZero);
var offset = TimeSpan.FromMinutes(gridUnits * OffsetGrid.TotalMinutes);
offset = TimeSpan.FromMinutes(gridUnits * OffsetGrid.TotalMinutes);
// Off-grid by more than the tolerance: the anchor was not fresh, or the writer's own
// clock is broken. Either way it must not become the offset.
// Off-grid by more than the tolerance: the line was not fresh, or the writer's own
// clock is broken. Either way it must not become an offset.
if ((candidate - offset).Duration() > MaxGridResidual) return false;
// Not a value any real timezone can produce.
if (offset < MinOffset || offset > MaxOffset) return false;
return true;
}
public bool TryCalibrate(string sourceKey, DateTime localTimestamp, DateTime agentUtcNow)
{
if (string.IsNullOrEmpty(sourceKey)) return false;
TimeSpan offset;
if (!TryMeasureOffset(localTimestamp, agentUtcNow, out offset)) return false;
lock (_lock)
{
if (_bySource.TryGetValue(sourceKey, out var existing))
@@ -1042,6 +1042,56 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// </summary>
internal string LastEspPhaseDetected => _lastEspPhaseDetected;
/// <summary>
/// True while the user-scope ESP phase is the tracker's current phase. Every user-phase
/// probe and the zero-apps evidence gate on this — outside AccountSetup the IME lines
/// belong to the device phase or the pre-sign-in (defaultuser0) frame.
/// </summary>
private bool IsUserEspPhaseActive =>
string.Equals(_lastEspPhaseDetected, "AccountSetup", StringComparison.OrdinalIgnoreCase);
/// <summary>
/// True once IME reported zero targeted Win32 apps for a user session while the user ESP
/// phase was active. Read by the adapter to label the evidence behind
/// <c>ime_user_session_completed</c>.
/// </summary>
internal bool UserSessionZeroAppsObserved => _userSessionZeroAppsObserved;
/// <summary>
/// IME-USER-SESSION-ZERO-APPS: IME enumerated the user session and found no targeted
/// Win32 apps. Its zero-app branch then <c>continue</c>s past the "Completed user
/// session" log statement, so this line is the only completion evidence such a device
/// ever produces — and IME itself treats the same condition as finished
/// (<c>ProviderResult.ProvisioningComplete</c> in the user check-in path, and an ESP
/// lock-in with an empty application list in the poller path).
/// <para>
/// Deliberately routed through <see cref="OnUserSessionCompleted"/> rather than a
/// separate callback: downstream this IS a completed user session, and the adapter's
/// pending-apps deferral plus its fire-once flag then apply unchanged.
/// </para>
/// <para>
/// Phase guard: firing before AccountSetup would burn the adapter's fire-once flag on a
/// pre-sign-in timestamp that <c>IsImeUserSessionGenuine</c> can never accept. The
/// observation is remembered instead and replayed by
/// <see cref="HandleEspPhaseDetected"/> on the transition into the user phase.
/// </para>
/// </summary>
private void HandleUserSessionZeroApps()
{
_userSessionZeroAppsObserved = true;
if (!IsUserEspPhaseActive)
{
_logger.Debug(
"ImeLogTracker: zero-app user session observed before the user ESP phase " +
$"(current phase: {_lastEspPhaseDetected ?? "(none)"}) - deferring until AccountSetup");
return;
}
_logger.Info("ImeLogTracker: user session enumerated with zero targeted apps - treating as user session completed");
OnUserSessionCompleted?.Invoke();
}
private void HandleEspPhaseDetected(string espPhaseString)
{
// Validate phase and get its ordinal for forward-only enforcement
@@ -1120,6 +1170,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
ActivatePatterns(logPhaseIsCurrentPhase);
OnEspPhaseChanged?.Invoke(espPhaseString);
// Replay a zero-app user session observed before this phase was known. IME logs its
// check-in lines independently of when the agent first parses the ESP phase marker,
// so the evidence can legitimately arrive first; without this the only completion
// evidence a zero-app device produces would be dropped on the floor.
if (isFirstOrChangedPhase && IsUserEspPhaseActive && _userSessionZeroAppsObserved)
{
_logger.Info("ImeLogTracker: replaying zero-app user session evidence observed before the user ESP phase");
OnUserSessionCompleted?.Invoke();
}
}
private void HandleEspTrackStatus(Match match)
@@ -77,12 +77,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
var fileInfo = new FileInfo(filePath);
if (!fileInfo.Exists) continue;
// Captured BEFORE GetSafePosition, which creates the entry as a side effect.
// Growth measured against a previously observed size is what makes the last
// line of this pass a valid "written now" calibration anchor.
// Captured BEFORE MarkChecked below stamps this pass. Growth measured
// against a previously observed state is what makes this pass's lines valid
// "written now" anchors — for the per-file measurement AND per-line anchoring.
var hadPreviousObservation = _positionTracker.HasSeen(filePath);
var lastCheckedUtc = _positionTracker.GetLastCheckedUtc(filePath);
var passNowUtc = UtcNowProvider();
var startPos = _positionTracker.GetSafePosition(filePath, fileInfo.Length);
// Every look counts, including "no new data" and an empty first sight: the
// NEXT pass's freshness window is measured from here. Restored bookmarks
// deliberately carry no LastCheckedUtc — the first pass after a restart reads
// downtime backlog and must never count as fresh.
_positionTracker.MarkChecked(filePath, passNowUtc);
if (startPos >= fileInfo.Length)
{
// M2: guard the interpolated string so it isn't built every 100 ms tick
@@ -95,6 +104,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
_logger.Trace($"ImeLogTracker: reading {Path.GetFileName(filePath)} from pos {startPos} (size={fileInfo.Length}, delta={fileInfo.Length - startPos})");
_currentSourceFileName = Path.GetFileName(filePath);
_currentPassLinesAreFresh = hadPreviousObservation
&& lastCheckedUtc > DateTime.MinValue
&& (passNowUtc - lastCheckedUtc) <= FreshLineMaxAge;
// Newest bias-less line of this pass — the calibration anchor. Bias-carrying
// lines are skipped: they already state the writer's offset, so they need no
@@ -210,7 +222,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
// growing pass, during which lines fall back to the reader zone and are
// flagged as such.
if (hadPreviousObservation && calibrationAnchor != null)
CalibrateFrom(_currentSourceFileName, calibrationAnchor);
CalibrateFrom(_currentSourceFileName, Path.GetFileName(filePath), calibrationAnchor);
}
}
catch (FileNotFoundException) { }
@@ -254,6 +266,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
LastMatchedSourceLocalTimestamp = null;
LastMatchedSourceOffsetMinutes = null;
LastMatchedSourceOffsetOrigin = CmTraceOffsetOrigin.None;
LastMatchedMeasuredWriterOffsetMinutes = null;
}
else
{
@@ -263,6 +276,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
LastMatchedSourceLocalTimestamp = entry.HasTimestamp ? entry.LocalTimestamp : (DateTime?)null;
LastMatchedSourceOffsetMinutes = offsetMinutes;
LastMatchedSourceOffsetOrigin = origin;
// Observational: what the calibrator measured for this file, distinct from what
// was applied above.
TimeSpan measuredWriterOffset;
LastMatchedMeasuredWriterOffsetMinutes =
_currentSourceFileName != null
&& OffsetCalibrator.TryGetOffset(_currentSourceFileName, out measuredWriterOffset)
? (int)measuredWriterOffset.TotalMinutes
: (int?)null;
}
// Generic pattern-match hook (M4.4.4). Invoked before action-specific callbacks so
@@ -365,6 +387,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
OnUserSessionCompleted?.Invoke();
break;
case "usersessionzeroapps":
// IME found no targeted user apps — its zero-app branch skips the
// "Completed user session" line, so this is the equivalent evidence.
HandleUserSessionZeroApps();
break;
case "updatestateinstalled":
if (!string.IsNullOrEmpty(id))
UpdateStateWithCallback(id, AppInstallationState.Installed);
@@ -640,52 +668,110 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
if (!entry.HasTimestamp) return UtcNowProvider();
// Measured offset of the process that wrote this file.
TimeSpan measured;
DateTime calibrated;
if (_currentSourceFileName != null
&& OffsetCalibrator.TryGetOffset(_currentSourceFileName, out measured)
&& OffsetCalibrator.TryResolveUtc(_currentSourceFileName, entry.LocalTimestamp, out calibrated))
// ERA-AWARE RESOLUTION (2026-08-20, second attempt): a FRESH line anchors ITSELF.
//
// The first attempt (e9dba11b, reverted in 04b1a7c6) applied one measured offset per
// FILE — wrong, because a single log file holds lines from multiple writer eras:
// AgentExecutor.log is written by short-lived child processes whose zone belief
// flips per process, interleaved in one file (fixture
// tests/fixtures/cmtrace-logs/agentexecutor-two-writer-eras-v1.cmtrace). Any
// cross-line anchor — per file, or nearest-in-write-order — inherits that trap.
//
// Per-line self-anchoring does not: at a 100 ms poll every line of a growing pass
// was written essentially "now", so its own distance to the agent clock IS its
// writer's offset, era by era. The line's timestamp still contributes the sub-poll
// precision and ordering (two lines 6 ms apart stay 6 ms apart — a plain
// occurredAt=now could not do that), while the offset grid absorbs poll latency.
// Freshness is load-bearing (see FreshLineMaxAge) — backlog passes, restart
// catch-up and replay logs must never anchor, because an old line's AGE can round
// onto the grid.
if (_currentPassLinesAreFresh)
{
origin = CmTraceOffsetOrigin.Calibrated;
offsetMinutes = (int)measured.TotalMinutes;
return calibrated;
TimeSpan lineOffset;
if (CmTraceOffsetCalibrator.TryMeasureOffset(entry.LocalTimestamp, UtcNowProvider(), out lineOffset))
{
origin = CmTraceOffsetOrigin.LineAnchored;
offsetMinutes = (int)lineOffset.TotalMinutes;
return DateTime.SpecifyKind(entry.LocalTimestamp - lineOffset, DateTimeKind.Utc);
}
}
// Not calibrated yet (first growing pass, or a file that never grows while we watch).
// Falls back to this process's zone, which is right only if the writer happens to
// share our belief. Origin stays None so the emitted event says so.
// Fallback for everything not provably fresh: assume the reader's zone, UNIFORMLY.
// Wrong in absolute terms whenever the writer held a different belief, but
// self-consistent — both ends of a duration are wrong by the same amount, so derived
// durations stay right. That asymmetry is the lesson of the 04b1a7c6 revert:
// partially corrected is strictly worse than uniformly wrong. Origin stays None so
// the emitted event says so.
var readerZoneOffset = TimeZoneInfo.Local.GetUtcOffset(entry.LocalTimestamp);
offsetMinutes = (int)readerZoneOffset.TotalMinutes;
return CmTraceLogParser.ResolveUtcAssumingReaderZone(entry.LocalTimestamp);
}
// Field-forensics (tripwire, 2026-08-20): sessions e9753578 and 067797d8 both logged a
// measurement whose LABEL and ANCHOR could not have come from the same file iteration —
// 067797d8's anchor value (19:51:26.948) does not even exist in ANY log file on the
// device, and IntuneManagementExtension.log never produced its expected first
// measurement at all. Mechanism unexplained; resolution is immune (per-line anchoring),
// but the observational layer must convict the culprit on the next occurrence. Hence:
// the stream's file name travels alongside the label (a mismatch is THE smoking gun),
// and the first rejected anchor per file becomes visible instead of silent.
private readonly HashSet<string> _calibrationRejectLogged = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
/// <summary>
/// Feed the pass's newest bias-less line to the calibrator and report a measured offset
/// that disagrees with this process's own zone — that disagreement is precisely the
/// condition that used to corrupt every IME-derived timestamp silently.
/// </summary>
private void CalibrateFrom(string sourceFileName, CmTraceLogEntry anchor)
/// <param name="sourceFileName">The calibration key (<c>_currentSourceFileName</c>).</param>
/// <param name="streamFileName">The file the enclosing iteration actually read. Must equal
/// <paramref name="sourceFileName"/> — logged loudly when it does not (see tripwire note).</param>
private void CalibrateFrom(string sourceFileName, string streamFileName, CmTraceLogEntry anchor)
{
if (!string.Equals(sourceFileName, streamFileName, StringComparison.OrdinalIgnoreCase))
{
_logger?.Warning(
$"ImeLogTracker: CALIBRATION LABEL MISMATCH — key='{sourceFileName}' but the iteration read '{streamFileName}' " +
$"(anchor local={anchor.LocalTimestamp:yyyy-MM-ddTHH:mm:ss.fffffff}). Skipping this anchor; see tripwire note.");
return;
}
TimeSpan previous;
var hadOffset = OffsetCalibrator.TryGetOffset(sourceFileName, out previous);
if (!OffsetCalibrator.TryCalibrate(sourceFileName, anchor.LocalTimestamp, UtcNowProvider()))
{
// Once per file: why a file that visibly grows never produces a measurement.
// 067797d8 ran 24 s of growing IME.log passes with no measurement at all — this
// silence is exactly what made the phenomenon undiagnosable from the log.
if (_calibrationRejectLogged.Add(sourceFileName))
{
_logger?.Debug(
$"ImeLogTracker: {sourceFileName} anchor rejected by calibrator " +
$"(anchor local={anchor.LocalTimestamp:yyyy-MM-ddTHH:mm:ss.fffffff}, now={UtcNowProvider():HH:mm:ss.fff}Z) — first rejection for this file, further ones stay silent.");
}
return;
}
TimeSpan measured;
if (!OffsetCalibrator.TryGetOffset(sourceFileName, out measured)) return;
if (hadOffset && measured == previous) return;
// The anchor's local timestamp is part of the record ON PURPOSE (tripwire, 2026-08-20):
// session e9753578 logged "+02:00 measured for IntuneManagementExtension.log" although
// that file provably held no +2-era line — an anchor/label pairing the committed code
// could not be shown to produce. Should it ever happen again, the logged anchor value
// itself will prove (or refute) the crossing without needing the device's log files.
var readerZoneOffset = TimeZoneInfo.Local.GetUtcOffset(UtcNowProvider());
if (measured == readerZoneOffset)
{
_logger?.Debug(
$"ImeLogTracker: {sourceFileName} writer offset measured {measured} (matches this process).");
$"ImeLogTracker: {sourceFileName} writer offset measured {measured} (matches this process, anchor local={anchor.LocalTimestamp:yyyy-MM-ddTHH:mm:ss.fff}).");
return;
}
_logger?.Info(
$"ImeLogTracker: {sourceFileName} writer offset measured {measured}, this process believes {readerZoneOffset} " +
$"— timestamps from this file are corrected by {(measured - readerZoneOffset).Negate()}.");
$"(anchor local={anchor.LocalTimestamp:yyyy-MM-ddTHH:mm:ss.fff}). Measurement is observational — resolution anchors per line.");
}
}
@@ -103,6 +103,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// </summary>
private string _currentSourceFileName;
/// <summary>
/// Whether every line of the file iteration currently in flight is provably FRESH —
/// written after the previous look at this file, which happened at most
/// <see cref="FreshLineMaxAge"/> ago in THIS process. Only then may a line anchor its
/// own UTC offset (<see cref="CmTraceOffsetCalibrator.TryMeasureOffset"/>): the grid
/// rounding is safe when the line's age is bounded by seconds, and becomes a trap on
/// backlog whose age can hit an exact grid multiple. Same single-task justification as
/// <see cref="_currentSourceFileName"/>.
/// </summary>
private bool _currentPassLinesAreFresh;
/// <summary>
/// Maximum gap between two looks at a file for its new bytes to count as fresh. Far
/// above the 100 ms poll (rides out GC pauses and ESP CPU starvation), far below the
/// 13-minute danger zone where a line's AGE could round onto the 15-minute offset grid.
/// </summary>
internal static readonly TimeSpan FreshLineMaxAge = TimeSpan.FromSeconds(30);
internal static DateTime NormalizeUtc(DateTime value) =>
value.Kind == DateTimeKind.Utc ? value : DateTime.SpecifyKind(value, DateTimeKind.Utc);
@@ -111,6 +129,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
private CancellationTokenSource _cts;
private bool _allAppsCompletedFired;
// Sticky: IME enumerated a user session and reported ZERO targeted Win32 apps
// (IME-USER-SESSION-ZERO-APPS). IME's zero-app branch returns before it writes the
// "Completed user session" line, so on a device with no user-targeted apps that line
// never appears and every user-phase completion probe would starve forever. This flag
// IS the user-phase completion evidence for that shape. Set only while the user ESP
// phase is active, or replayed on the transition into it.
private bool _userSessionZeroAppsObserved;
// State persistence: saves tracker state to disk so agent restart continues
// from the exact log position without re-parsing or re-building ignore lists.
private readonly ImeTrackerStatePersistence _statePersistence;
@@ -228,6 +254,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// <summary>How that offset was arrived at — declared, measured, or neither.</summary>
public CmTraceOffsetOrigin LastMatchedSourceOffsetOrigin { get; internal set; }
/// <summary>
/// The offset the calibrator MEASURED for the source file, which is not necessarily the
/// one applied.
/// <para>
/// Since the 2026-08-20 revert the measurement is observational only: it is recorded so the
/// field tells us what an era-aware calibrator would have to work with, without acting on
/// it. Reported separately from <see cref="LastMatchedSourceOffsetMinutes"/> precisely so
/// the two can never be confused — that event said "calibrated" while carrying a wrong
/// value, and an honest record has to distinguish measured from applied.
/// </para>
/// </summary>
public int? LastMatchedMeasuredWriterOffsetMinutes { get; internal set; }
// Callbacks to EnrollmentTracker
public Action<string> OnEspPhaseChanged { get; set; }
public Action<string> OnImeAgentVersion { get; set; }
@@ -310,6 +349,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
_lastEspPhaseDetected = phaseName;
}
/// <summary>
/// Test-only seam: marks the zero-targeted-apps user session as observed without driving
/// the IME-USER-SESSION-ZERO-APPS pattern through a log line. The pattern-to-flag path
/// itself is covered end-to-end by <c>ImeLogTrackerZeroUserAppsTests</c>.
/// </summary>
internal void SeedUserSessionZeroAppsForTesting()
{
_userSessionZeroAppsObserved = true;
}
/// <summary>
/// Returns a deduped union of phase-snapshotted apps plus the live
/// <see cref="PackageStates"/> (current phase). Live entries win on Id collision.
@@ -364,11 +413,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
{
try
{
if (!string.Equals(_lastEspPhaseDetected, "AccountSetup", StringComparison.OrdinalIgnoreCase))
if (!IsUserEspPhaseActive)
return false;
var apps = new List<AppPackageState>(_packageStates);
if (apps.Count == 0) return false;
// Zero user-targeted apps is the MOST settled state there is, but the
// conservative "at least one required app" rule below reads it as "not
// settled" and starves the AccountSetup synthesis. Only an explicit IME
// observation lifts that — never a merely empty list, which also means
// "phase just cleared" or "apps not surfaced yet".
if (apps.Count == 0)
return _userSessionZeroAppsObserved;
var sawRequired = false;
foreach (var pkg in apps)
@@ -715,6 +771,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
_currentPhaseOrder = state.CurrentPhaseOrder;
_lastEspPhaseDetected = state.LastEspPhaseDetected;
_allAppsCompletedFired = state.AllAppsCompletedFired;
_userSessionZeroAppsObserved = state.UserSessionZeroAppsObserved;
_logPhaseIsCurrentPhase = state.LogPhaseIsCurrentPhase;
// Restore seen app IDs
@@ -830,6 +887,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
CurrentPhaseOrder = _currentPhaseOrder,
LastEspPhaseDetected = _lastEspPhaseDetected,
AllAppsCompletedFired = _allAppsCompletedFired,
UserSessionZeroAppsObserved = _userSessionZeroAppsObserved,
LogPhaseIsCurrentPhase = _logPhaseIsCurrentPhase,
SeenAppIds = _seenAppIds.ToList(),
IgnoreList = _packageStates.IgnoreList.ToList(),
@@ -109,6 +109,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
public int CurrentPhaseOrder { get; set; }
public string LastEspPhaseDetected { get; set; }
public bool AllAppsCompletedFired { get; set; }
/// <summary>
/// IME reported zero targeted Win32 apps for a user session. Absent in state files
/// written before this field existed — deserializing to false there is correct: the
/// tracker simply re-observes the line on the next IME check-in.
/// </summary>
public bool UserSessionZeroAppsObserved { get; set; }
public bool LogPhaseIsCurrentPhase { get; set; }
public List<string> SeenAppIds { get; set; }
public List<string> IgnoreList { get; set; }
@@ -65,6 +65,42 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
state.LastReadTime = DateTime.UtcNow;
}
/// <summary>
/// Records that the file was looked at in a poll pass — whether or not it had new data.
/// Creates the entry on first sight (position 0), so a file first seen EMPTY counts as
/// observed and its first content is provably fresh.
/// <para>
/// This timestamp is the freshness reference for per-line offset anchoring: bytes found
/// in the NEXT pass were written after this instant, so
/// <c>now - LastCheckedUtc</c> bounds the age of every line in that pass's chunk.
/// The caller supplies the clock (the tracker's <c>UtcNowProvider</c>) so replay/test
/// clocks stay consistent with the freshness comparison.
/// </para>
/// </summary>
public void MarkChecked(string filePath, DateTime nowUtc)
{
FilePositionState state;
if (!_positions.TryGetValue(filePath, out state))
{
state = new FilePositionState();
_positions[filePath] = state;
}
state.LastCheckedUtc = nowUtc;
}
/// <summary>
/// When the file was last looked at, or <see cref="DateTime.MinValue"/> when it never
/// was in THIS process — including entries restored from persisted state, whose
/// bookmark survives an agent restart but whose freshness deliberately does not
/// (the restart gap can hold arbitrarily old backlog).
/// </summary>
public DateTime GetLastCheckedUtc(string filePath)
{
FilePositionState state;
return _positions.TryGetValue(filePath, out state) ? state.LastCheckedUtc : DateTime.MinValue;
}
/// <summary>
/// Gets the stored position for a file, or 0 if not tracked.
/// Does not perform rollover detection.
@@ -89,6 +125,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// <summary>
/// Restores a previously persisted position for a file.
/// Used on agent restart to continue reading from the last known position.
/// <para>
/// <see cref="FilePositionState.LastCheckedUtc"/> is deliberately NOT restored: the
/// bookmark survives the restart, the freshness guarantee does not — the first pass
/// after a restart reads the whole downtime backlog and must never anchor per-line
/// offsets on it.
/// </para>
/// </summary>
public void RestorePosition(string filePath, long position, long lastKnownSize)
{
@@ -96,7 +138,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
{
Position = position,
LastKnownSize = lastKnownSize,
LastReadTime = DateTime.UtcNow
LastReadTime = DateTime.UtcNow,
LastCheckedUtc = DateTime.MinValue
};
}
}
@@ -106,5 +149,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
public long Position { get; set; }
public long LastKnownSize { get; set; }
public DateTime LastReadTime { get; set; }
/// <summary>
/// When this file was last looked at by a poll pass IN THIS PROCESS. In-memory only —
/// never persisted, so a restart resets it (see <see cref="LogFilePositionTracker.RestorePosition"/>).
/// </summary>
public DateTime LastCheckedUtc { get; set; }
}
}
@@ -0,0 +1,197 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Runtime;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Pure state machine behind <c>PowerStateWatcherHost</c>: diffs successive
/// <see cref="PowerStateResult"/> snapshots into <c>power_state_change</c> emissions.
/// No WMI, no timers, no logging — fully deterministic and unit-testable.
/// <para>
/// Semantics:
/// <list type="bullet">
/// <item><b>AC↔battery transitions</b> — one emission per accepted edge
/// (<c>ac_to_battery</c> Warning/immediate, <c>battery_to_ac</c> Info).</item>
/// <item><b>Threshold ladder</b> (default 50/30/15 %) — evaluated only while ON battery;
/// each level latches once per tracker lifetime (no re-emit when the device charges
/// back up and drains again — avoids plug-cycle alarm spam). A multi-level jump
/// emits only the LOWEST newly-crossed level. Entering battery power already below
/// a level counts as crossing it (covers "unplugged at 12 %" and, via
/// <see cref="Baseline"/>, "enrollment started at 12 %").</item>
/// <item><b>Storm cap</b> — a lifetime emission cap (default 20) as backstop against a
/// flapping dock / dying battery controller; <see cref="EmissionCapReached"/> flips
/// once so the host can log a single warning.</item>
/// </list>
/// A snapshot with <see cref="PowerStateResult.ProbeError"/> is ignored (previous state
/// preserved); a null <see cref="PowerStateResult.BatteryPercent"/> skips threshold
/// evaluation but AC diffing still works.
/// </para>
/// </summary>
internal sealed class PowerStateTransitionTracker
{
public const string TransitionAcToBattery = "ac_to_battery";
public const string TransitionBatteryToAc = "battery_to_ac";
public const string TransitionThresholdCrossed = "threshold_crossed";
internal static readonly int[] DefaultThresholdLadder = { 50, 30, 15 };
internal const int DefaultMaxEmissions = 20;
private readonly int[] _ladder; // strictly descending
private readonly int _maxEmissions;
private readonly HashSet<int> _latchedLevels = new HashSet<int>();
private bool _hasState;
private bool _lastOnAc;
private int _emitted;
public PowerStateTransitionTracker(IReadOnlyList<int>? thresholdLadder = null, int maxEmissions = DefaultMaxEmissions)
{
_ladder = (thresholdLadder ?? DefaultThresholdLadder).OrderByDescending(l => l).ToArray();
_maxEmissions = maxEmissions;
}
/// <summary>True once the lifetime emission cap suppressed at least one emission.</summary>
public bool EmissionCapReached { get; private set; }
/// <summary>
/// Latches the arm-time state silently — except when the device is already on battery
/// at/below a ladder level, which yields one threshold emission for the lowest crossed
/// level (an enrollment STARTING at 12 % must still produce the event the analyze rule
/// matches on). No AC-transition emission is ever produced from a baseline.
/// </summary>
public IReadOnlyList<PowerStateEmission> Baseline(PowerStateResult probe)
{
if (probe == null || probe.ProbeError != null || !probe.HasBattery)
return Array.Empty<PowerStateEmission>();
_hasState = true;
_lastOnAc = probe.OnAcPower;
var emissions = new List<PowerStateEmission>();
EvaluateThresholds(probe, emissions);
return emissions;
}
/// <summary>
/// Diffs a fresh (host-debounced) snapshot against the last accepted state and returns
/// 0..n emissions — an AC transition and a threshold crossing can co-occur (unplugged
/// at 10 % ⇒ <c>ac_to_battery</c> + <c>threshold_crossed:15</c>).
/// </summary>
public IReadOnlyList<PowerStateEmission> Evaluate(PowerStateResult probe)
{
if (probe == null || probe.ProbeError != null)
return Array.Empty<PowerStateEmission>();
if (!_hasState)
return Baseline(probe);
var emissions = new List<PowerStateEmission>();
if (_lastOnAc && !probe.OnAcPower)
{
Emit(emissions, new PowerStateEmission(
transition: TransitionAcToBattery,
thresholdPercent: null,
probe: probe,
severity: EventSeverity.Warning,
immediateUpload: true,
message: $"Power source switched from AC to battery ({FormatPercent(probe)} remaining) during enrollment"));
}
else if (!_lastOnAc && probe.OnAcPower)
{
Emit(emissions, new PowerStateEmission(
transition: TransitionBatteryToAc,
thresholdPercent: null,
probe: probe,
severity: EventSeverity.Info,
immediateUpload: false,
message: $"Power source switched back to AC ({FormatPercent(probe)}{(probe.IsCharging ? ", charging" : string.Empty)})"));
}
_lastOnAc = probe.OnAcPower;
EvaluateThresholds(probe, emissions);
return emissions;
}
/// <summary>
/// While on battery with a known percentage: emit the LOWEST ladder level at/above the
/// current charge that is not yet latched, then latch every level at/above the charge
/// (a 60 %→10 % jump emits threshold 15 only, latching 50/30/15).
/// </summary>
private void EvaluateThresholds(PowerStateResult probe, List<PowerStateEmission> emissions)
{
if (probe.OnAcPower || !probe.BatteryPercent.HasValue) return;
var percent = probe.BatteryPercent.Value;
int? lowestCrossed = null;
foreach (var level in _ladder) // descending — the last match is the lowest crossed level
{
if (percent <= level && !_latchedLevels.Contains(level))
lowestCrossed = level;
}
if (!lowestCrossed.HasValue) return;
foreach (var level in _ladder)
{
if (percent <= level) _latchedLevels.Add(level);
}
var threshold = lowestCrossed.Value;
var severity = threshold <= 15 ? EventSeverity.Error
: threshold <= 30 ? EventSeverity.Warning
: EventSeverity.Info;
Emit(emissions, new PowerStateEmission(
transition: TransitionThresholdCrossed,
thresholdPercent: threshold,
probe: probe,
severity: severity,
immediateUpload: severity >= EventSeverity.Warning,
message: $"Battery dropped below {threshold}% ({FormatPercent(probe)} remaining) while enrolling on battery power"));
}
private void Emit(List<PowerStateEmission> emissions, PowerStateEmission emission)
{
if (_emitted >= _maxEmissions)
{
EmissionCapReached = true;
return;
}
_emitted++;
emissions.Add(emission);
}
private static string FormatPercent(PowerStateResult probe)
=> probe.BatteryPercent.HasValue ? $"battery {probe.BatteryPercent.Value}%" : "battery level unknown";
}
/// <summary>One decided <c>power_state_change</c> emission — the host maps this 1:1 onto an event.</summary>
internal sealed class PowerStateEmission
{
public PowerStateEmission(string transition, int? thresholdPercent, PowerStateResult probe,
EventSeverity severity, bool immediateUpload, string message)
{
Transition = transition;
ThresholdPercent = thresholdPercent;
OnAcPower = probe.OnAcPower;
BatteryPercent = probe.BatteryPercent;
IsCharging = probe.IsCharging;
BatteryLifeMinutes = probe.BatteryLifeMinutes;
Severity = severity;
ImmediateUpload = immediateUpload;
Message = message;
}
public string Transition { get; }
public int? ThresholdPercent { get; }
public bool OnAcPower { get; }
public int? BatteryPercent { get; }
public bool IsCharging { get; }
public int? BatteryLifeMinutes { get; }
public EventSeverity Severity { get; }
public bool ImmediateUpload { get; }
public string Message { get; }
}
}
@@ -0,0 +1,681 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.Eventing.Reader;
using System.Globalization;
using System.IO;
using System.Linq;
using System.Xml.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using Newtonsoft.Json;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Watches the <c>System</c> event log for two enrollment blind spots and forwards them as
/// timeline ground truth:
/// <list type="bullet">
/// <item><b>System clock steps</b> (Kernel-General EventID 1) → <c>system_clock_changed</c>.
/// The payload carries <c>oldTime</c>/<c>newTime</c>/<c>timeDeltaMs</c> plus the
/// setting process — the "clock set from X to Y" fact that today has to be inferred
/// from timestamp discontinuities. w32time also logs frequent 1 ms micro-slews on the
/// same EventID, so emissions are gated on <see cref="MinClockDeltaMs"/>.</item>
/// <item><b>Completed sleep episodes</b> → <c>system_sleep_episode</c>. Classic S3/S4 comes
/// from Power-Troubleshooter EventID 1 (one event per episode at wake, carrying
/// SleepTime + WakeTime). Modern Standby comes from Kernel-Power EventID 507 (exit),
/// whose payload carries the scenario duration and the actual sleep residency; the
/// enter time is derived as exit − duration. Kernel-Power 506 (enter) is deliberately
/// NOT consumed: 507 already carries everything, and pairing 506/507 would need an
/// orphan-tolerant state machine across agent restarts for zero information gain —
/// an episode without a 507 never completed and would not be emitted under any design.</item>
/// </list>
/// <para>
/// <b>Payload times are authoritative.</b> The decisive clock correction (w32time fixing a
/// wrong BIOS clock) happens exactly when the device clock is untrustworthy, and the backend
/// clamps implausible event timestamps to server time. All semantic instants therefore travel
/// as explicit ISO-8601 payload fields, never only as the event's own <c>Timestamp</c>.
/// </para>
/// <para>
/// <b>Backfill + cross-restart dedup</b> mirror <see cref="WindowsUpdateTracker"/>: the System
/// .evtx persists across the OOBE reboots, so a startup scan recovers pre-agent clock steps
/// and standby episodes; a persisted RecordId watermark plus an intra-run seen-set (a HashSet,
/// NOT a high-water mark — see the field comment) keep restarts from re-emitting.
/// </para>
/// </summary>
internal sealed class SystemTimelineTracker : IDisposable
{
internal const string Channel = "System";
internal const string ProviderKernelGeneral = "Microsoft-Windows-Kernel-General"; // EventID 1: system time changed
internal const string ProviderPowerTroubleshooter = "Microsoft-Windows-Power-Troubleshooter"; // EventID 1: resumed from classic sleep/hibernate
internal const string ProviderKernelPower = "Microsoft-Windows-Kernel-Power"; // EventID 507: Modern Standby exit
internal const int EventId_ClockChange = 1;
internal const int EventId_ClassicResume = 1;
internal const int EventId_ModernStandbyExit = 507;
internal const string WatermarkStateFileName = "system-timeline-watermark.json";
// w32time logs Kernel-General 1 for routine 1 ms slew corrections (verified live: several
// per hour). Only genuine steps are timeline-relevant; anything below this is claimed in
// the dedup set but never emitted.
internal const long MinClockDeltaMs = 2000;
// A step of at least 5 minutes can invalidate token/cert validation windows and visibly
// reorders the timeline (matches the web's TimeJumpBadge threshold) — surface it fast.
internal const long ClockDeltaWarningMs = 5 * 60 * 1000;
// Modern Standby logs a 506/507 pair for every screen-off scenario, including seconds-long
// ones where the device never actually slept (SleepDurationInUs 0). Only episodes with
// real sleep residency are emitted. Classic Power-Troubleshooter episodes have no floor:
// they only exist for genuinely completed S3/S4 transitions.
internal const long MinSleepDurationSeconds = 60;
private readonly AgentLogger _logger;
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly bool _backfillEnabled;
private readonly int _backfillLookbackMinutes;
private readonly string _stateDirectory;
private EventLogWatcher _watcher;
// Dedup. Guarded because EventRecordWritten callbacks run concurrently with the backfill scan.
//
// Cross-restart boundary loaded from disk: the highest RecordId emitted in a PRIOR run.
// Immutable during this run — records at/below it were already emitted before this process
// started and must not be re-emitted by the backfill scan.
private long _restartWatermark = -1;
// RecordIds claimed in THIS run — the intra-run dedup set. Deliberately a HashSet, NOT a
// high-water mark: the live watcher is armed before the backfill runs, so a live record with
// a HIGHER RecordId can be processed before/during backfill. A high-water mark would then
// make the dedup skip every older, never-emitted backfill record — silently dropping exactly
// the pre-agent clock corrections and standby episodes this tracker exists for. The targeted
// records are low-volume, so the set stays tiny.
private readonly HashSet<long> _seenThisRun = new HashSet<long>();
// Highest RecordId claimed so far (max of restart watermark + this run). Persisted so the
// next run's restart watermark is correct.
private long _maxEmittedRecordId = -1;
private readonly object _watermarkLock = new object();
public SystemTimelineTracker(
string sessionId,
string tenantId,
InformationalEventPost post,
AgentLogger logger,
bool backfillEnabled = true,
int backfillLookbackMinutes = 1440,
string stateDirectory = null)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_backfillEnabled = backfillEnabled;
_backfillLookbackMinutes = backfillLookbackMinutes;
_stateDirectory = stateDirectory != null ? Environment.ExpandEnvironmentVariables(stateDirectory) : null;
}
public void Start()
{
LoadWatermark();
StartWatcher();
if (_backfillEnabled && _backfillLookbackMinutes > 0)
BackfillRecentEvents();
else
_logger.Info("SystemTimeline backfill disabled by config");
}
public void Stop()
{
if (_watcher == null) return;
try
{
_watcher.Enabled = false;
_watcher.Dispose();
_logger.Info("SystemTimeline watcher stopped");
}
catch (Exception ex)
{
_logger.Error("Error stopping SystemTimeline watcher", ex);
}
finally
{
_watcher = null;
}
}
public void Dispose() => Stop();
// -----------------------------------------------------------------------
// Watcher lifecycle
// -----------------------------------------------------------------------
private void StartWatcher()
{
try
{
var query = new EventLogQuery(Channel, PathType.LogName, BuildXPath());
_watcher = new EventLogWatcher(query);
_watcher.EventRecordWritten += OnEventRecordWritten;
_watcher.Enabled = true;
_logger.Info($"SystemTimeline watcher started: {Channel} (clock changes + sleep episodes)");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"SystemTimeline event log not found: {Channel} (normal on non-Windows test environments)");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"SystemTimeline watcher access denied for {Channel}: {ex.Message}");
}
catch (Exception ex)
{
_logger.Error($"Failed to start SystemTimeline watcher for {Channel}", ex);
CollectorDegradationReporter.Report(_post, _sessionId, _tenantId,
collectorName: "SystemTimelineTracker", reason: $"watcher_arm_failed:{Channel}", ex: ex);
}
}
/// <summary>
/// Provider-qualified XPath for the three targeted signals. Provider filtering is mandatory
/// on the System channel — EventID 1 alone matches many unrelated providers.
/// Exposed as internal for tests.
/// </summary>
internal static string BuildXPath() =>
"*[System[" + ProviderIdClauses() + "]]";
/// <summary>Backfill variant of <see cref="BuildXPath"/> with the lookback window applied.</summary>
internal static string BuildBackfillXPath(long lookbackMs) =>
"*[System[(" + ProviderIdClauses() + $") and TimeCreated[timediff(@SystemTime) <= {lookbackMs}]]]";
private static string ProviderIdClauses() =>
$"(Provider[@Name='{ProviderKernelGeneral}'] and EventID={EventId_ClockChange})" +
$" or (Provider[@Name='{ProviderPowerTroubleshooter}'] and EventID={EventId_ClassicResume})" +
$" or (Provider[@Name='{ProviderKernelPower}'] and EventID={EventId_ModernStandbyExit})";
// -----------------------------------------------------------------------
// Event processing
// -----------------------------------------------------------------------
private void OnEventRecordWritten(object sender, EventRecordWrittenEventArgs e)
{
if (e.EventRecord == null) return;
try
{
ProcessRecord(e.EventRecord, isBackfill: false);
}
catch (Exception ex)
{
_logger.Error("Error processing SystemTimeline event", ex);
}
}
private void ProcessRecord(EventRecord record, bool isBackfill)
{
var recordId = record.RecordId ?? -1;
// Cheap dedup gate BEFORE the XML rendering work.
if (IsAlreadyProcessed(recordId))
return;
string xml = null;
try { xml = record.ToXml(); }
catch { /* fall through with empty EventData */ }
ProcessEvent(
providerName: record.ProviderName,
eventId: record.Id,
recordId: recordId,
timeCreatedUtc: record.TimeCreated?.ToUniversalTime(),
eventData: ParseEventData(xml),
isBackfill: isBackfill);
}
/// <summary>
/// Core processing extracted to primitive inputs so tests can drive it without synthesizing
/// an abstract, Windows-only <see cref="EventRecord"/> (WindowsUpdateTracker test-seam
/// pattern; a plain dictionary instead of named args because three providers with ~10
/// payload fields each make a fixed signature impractical).
/// </summary>
internal void ProcessEvent(
string providerName,
int eventId,
long recordId,
DateTime? timeCreatedUtc,
Dictionary<string, string> eventData,
bool isBackfill)
{
if (!MarkProcessed(recordId))
return; // already emitted (cross-restart or duplicate delivery)
eventData = eventData ?? new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
if (string.Equals(providerName, ProviderKernelGeneral, StringComparison.OrdinalIgnoreCase) && eventId == EventId_ClockChange)
HandleClockChange(recordId, timeCreatedUtc, eventData, isBackfill);
else if (string.Equals(providerName, ProviderPowerTroubleshooter, StringComparison.OrdinalIgnoreCase) && eventId == EventId_ClassicResume)
HandleClassicResume(recordId, timeCreatedUtc, eventData, isBackfill);
else if (string.Equals(providerName, ProviderKernelPower, StringComparison.OrdinalIgnoreCase) && eventId == EventId_ModernStandbyExit)
HandleModernStandbyExit(recordId, timeCreatedUtc, eventData, isBackfill);
// Anything else slipped past the XPath filter — ignore.
}
// -----------------------------------------------------------------------
// system_clock_changed (Kernel-General EventID 1)
// -----------------------------------------------------------------------
private void HandleClockChange(long recordId, DateTime? timeCreatedUtc, Dictionary<string, string> eventData, bool isBackfill)
{
eventData.TryGetValue("NewTime", out var newTimeRaw);
eventData.TryGetValue("OldTime", out var oldTimeRaw);
var newTime = TryParseUtc(newTimeRaw);
var oldTime = TryParseUtc(oldTimeRaw);
// Signed delta preferably from the authoritative instants (NewTime − OldTime carries
// the direction unambiguously); TimeDeltaInMs is the fallback when either fails to
// parse. Records with neither are unusable and skipped (already claimed by dedup).
long deltaMs;
if (newTime.HasValue && oldTime.HasValue)
{
deltaMs = (long)Math.Round((newTime.Value - oldTime.Value).TotalMilliseconds);
}
else if (eventData.TryGetValue("TimeDeltaInMs", out var deltaRaw) &&
long.TryParse(deltaRaw, NumberStyles.Integer, CultureInfo.InvariantCulture, out var parsedDelta))
{
deltaMs = parsedDelta;
}
else
{
_logger.Debug($"SystemTimeline: clock-change record {recordId} has no usable times/delta — skipped");
return;
}
if (Math.Abs(deltaMs) < MinClockDeltaMs)
return; // w32time micro-slew — claimed (MarkProcessed) but never emitted
eventData.TryGetValue("Reason", out var reasonRaw);
int.TryParse(reasonRaw, NumberStyles.Integer, CultureInfo.InvariantCulture, out var reason);
var reasonText = reason == 1 ? "application_set"
: reason == 2 ? "hardware_clock_sync"
: "unknown";
eventData.TryGetValue("ProcessName", out var processName);
var data = new Dictionary<string, object>
{
{ "timeDeltaMs", deltaMs },
{ "reason", reason },
{ "reasonText", reasonText },
{ "backfilled", isBackfill },
};
// Authoritative instants: re-normalized ISO-8601 when parseable, raw pass-through
// otherwise — the payload must survive even a malformed provider rendering.
data["newTime"] = newTime.HasValue ? newTime.Value.ToString("o") : (object)(newTimeRaw ?? string.Empty);
data["oldTime"] = oldTime.HasValue ? oldTime.Value.ToString("o") : (object)(oldTimeRaw ?? string.Empty);
if (recordId >= 0) data["recordId"] = recordId;
if (timeCreatedUtc.HasValue) data["timeCreated"] = timeCreatedUtc.Value.ToString("o");
if (!string.IsNullOrEmpty(processName)) data["processName"] = processName;
if (eventData.TryGetValue("ProcessID", out var pidRaw) &&
int.TryParse(pidRaw, NumberStyles.Integer, CultureInfo.InvariantCulture, out var pid))
{
data["processId"] = pid;
}
var direction = deltaMs >= 0 ? "forward" : "backward";
var processLeaf = ProcessLeafName(processName);
var message = $"System clock stepped {direction} {FormatDuration(TimeSpan.FromMilliseconds(Math.Abs(deltaMs)))}" +
$" (reason: {reasonText}" + (processLeaf != null ? $", process: {processLeaf})" : ")");
var isLargeStep = Math.Abs(deltaMs) >= ClockDeltaWarningMs;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
// The record's own creation time, not UtcNow: backfilled pre-agent clock steps must
// land on the timeline where they happened (WindowsUpdateTracker convention).
Timestamp = timeCreatedUtc ?? DateTime.UtcNow,
EventType = Constants.EventTypes.SystemClockChanged,
Severity = isLargeStep ? EventSeverity.Warning : EventSeverity.Info,
Source = "SystemTimelineWatcher",
Phase = EnrollmentPhase.Unknown,
Message = message,
Data = data,
ImmediateUpload = isLargeStep,
});
}
// -----------------------------------------------------------------------
// system_sleep_episode — classic sleep/hibernate (Power-Troubleshooter EventID 1)
// -----------------------------------------------------------------------
private void HandleClassicResume(long recordId, DateTime? timeCreatedUtc, Dictionary<string, string> eventData, bool isBackfill)
{
eventData.TryGetValue("SleepTime", out var sleepRaw);
eventData.TryGetValue("WakeTime", out var wakeRaw);
var enteredAt = TryParseUtc(sleepRaw);
var exitedAt = TryParseUtc(wakeRaw);
// EffectiveState 5 = hibernate (S4); TargetState is the fallback when the effective
// state is missing (older event versions).
eventData.TryGetValue("EffectiveState", out var effectiveState);
eventData.TryGetValue("TargetState", out var targetState);
var stateValue = !string.IsNullOrEmpty(effectiveState) ? effectiveState : targetState;
var kind = stateValue == "5" ? "hibernate" : "sleep";
long? durationSeconds = null;
if (enteredAt.HasValue && exitedAt.HasValue && exitedAt.Value > enteredAt.Value)
durationSeconds = (long)Math.Round((exitedAt.Value - enteredAt.Value).TotalSeconds);
var data = new Dictionary<string, object>
{
{ "kind", kind },
{ "backfilled", isBackfill },
};
data["enteredAt"] = enteredAt.HasValue ? enteredAt.Value.ToString("o") : (object)(sleepRaw ?? string.Empty);
data["exitedAt"] = exitedAt.HasValue ? exitedAt.Value.ToString("o") : (object)(wakeRaw ?? string.Empty);
if (durationSeconds.HasValue) data["durationSeconds"] = durationSeconds.Value;
if (recordId >= 0) data["recordId"] = recordId;
if (timeCreatedUtc.HasValue) data["timeCreated"] = timeCreatedUtc.Value.ToString("o");
if (eventData.TryGetValue("WakeSourceType", out var wakeTypeRaw) &&
int.TryParse(wakeTypeRaw, NumberStyles.Integer, CultureInfo.InvariantCulture, out var wakeType))
{
data["wakeSourceType"] = wakeType;
}
eventData.TryGetValue("WakeSourceText", out var wakeText);
if (!string.IsNullOrEmpty(wakeText)) data["wakeSourceText"] = wakeText;
var durationLabel = durationSeconds.HasValue
? " after " + FormatDuration(TimeSpan.FromSeconds(durationSeconds.Value))
: string.Empty;
var wakeLabel = !string.IsNullOrEmpty(wakeText) ? $" (wake source: {wakeText})" : string.Empty;
EmitSleepEpisode(timeCreatedUtc, data,
$"Device resumed from {kind}{durationLabel}{wakeLabel}");
}
// -----------------------------------------------------------------------
// system_sleep_episode — Modern Standby (Kernel-Power EventID 507)
// -----------------------------------------------------------------------
private void HandleModernStandbyExit(long recordId, DateTime? timeCreatedUtc, Dictionary<string, string> eventData, bool isBackfill)
{
eventData.TryGetValue("SleepDurationInUs", out var sleepUsRaw);
long.TryParse(sleepUsRaw, NumberStyles.Integer, CultureInfo.InvariantCulture, out var sleepUs);
var sleepDurationSeconds = sleepUs / 1_000_000;
// Every screen-off scenario logs a 507, including seconds-long ones where the device
// never slept (SleepDurationInUs 0, incl. SleepEntered=false) — one gate covers both.
if (sleepDurationSeconds < MinSleepDurationSeconds)
return; // claimed but not an episode worth the timeline
eventData.TryGetValue("DurationInUs", out var durationUsRaw);
long.TryParse(durationUsRaw, NumberStyles.Integer, CultureInfo.InvariantCulture, out var durationUs);
if (durationUs < sleepUs) durationUs = sleepUs; // scenario duration can never undercut its sleep residency
var durationSeconds = durationUs / 1_000_000;
var exitedAt = timeCreatedUtc; // 507 is written at scenario exit
DateTime? enteredAt = exitedAt.HasValue
? exitedAt.Value.AddMilliseconds(-(durationUs / 1000.0))
: (DateTime?)null;
var data = new Dictionary<string, object>
{
{ "kind", "modern_standby" },
{ "durationSeconds", durationSeconds },
{ "sleepDurationSeconds", sleepDurationSeconds },
{ "backfilled", isBackfill },
};
if (enteredAt.HasValue) data["enteredAt"] = enteredAt.Value.ToString("o");
if (exitedAt.HasValue) data["exitedAt"] = exitedAt.Value.ToString("o");
if (recordId >= 0) data["recordId"] = recordId;
if (timeCreatedUtc.HasValue) data["timeCreated"] = timeCreatedUtc.Value.ToString("o");
if (eventData.TryGetValue("Reason", out var reasonRaw) &&
int.TryParse(reasonRaw, NumberStyles.Integer, CultureInfo.InvariantCulture, out var reason))
{
data["reason"] = reason;
}
bool? onAcPower = null;
if (eventData.TryGetValue("PowerStateAc", out var acRaw) && bool.TryParse(acRaw, out var ac))
{
onAcPower = ac;
data["onAcPower"] = ac;
}
if (eventData.TryGetValue("BatteryRemainingCapacityOnExit", out var battRaw) &&
long.TryParse(battRaw, NumberStyles.Integer, CultureInfo.InvariantCulture, out var batt))
{
data["batteryRemainingCapacityOnExit"] = batt;
}
var powerLabel = onAcPower.HasValue ? (onAcPower.Value ? ", on AC" : ", on battery") : string.Empty;
EmitSleepEpisode(timeCreatedUtc, data,
$"Device exited Modern Standby after {FormatDuration(TimeSpan.FromSeconds(durationSeconds))}" +
$" (slept {FormatDuration(TimeSpan.FromSeconds(sleepDurationSeconds))}{powerLabel})");
}
private void EmitSleepEpisode(DateTime? timeCreatedUtc, Dictionary<string, object> data, string message)
{
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
// Wake/exit moment (the record's creation time) — backfilled pre-agent episodes
// must land on the timeline where the device actually woke.
Timestamp = timeCreatedUtc ?? DateTime.UtcNow,
EventType = Constants.EventTypes.SystemSleepEpisode,
Severity = EventSeverity.Info,
Source = "SystemTimelineWatcher",
Phase = EnrollmentPhase.Unknown,
Message = message,
Data = data,
ImmediateUpload = false,
});
}
// -----------------------------------------------------------------------
// Parsing helpers
// -----------------------------------------------------------------------
/// <summary>
/// Extracts <c>&lt;Data Name="..."&gt;value&lt;/Data&gt;</c> pairs from an event's rendered
/// XML, keyed case-insensitively. Namespace-agnostic (matches on local name). Returns an
/// empty map on null/malformed XML — never throws.
/// </summary>
internal static Dictionary<string, string> ParseEventData(string xml)
{
var result = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
if (string.IsNullOrEmpty(xml)) return result;
try
{
var doc = XDocument.Parse(xml);
foreach (var el in doc.Descendants().Where(e => e.Name.LocalName == "Data"))
{
var nameAttr = el.Attribute("Name");
if (nameAttr == null || string.IsNullOrEmpty(nameAttr.Value)) continue;
if (!result.ContainsKey(nameAttr.Value))
result[nameAttr.Value] = el.Value;
}
}
catch
{
// Malformed / unexpected XML — best effort, return what we have.
}
return result;
}
internal static DateTime? TryParseUtc(string value)
{
if (string.IsNullOrWhiteSpace(value)) return null;
return DateTime.TryParse(value, CultureInfo.InvariantCulture,
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var parsed)
? parsed
: (DateTime?)null;
}
/// <summary>Compact human duration for messages: "2h 05m", "56m 12s", "45s".</summary>
internal static string FormatDuration(TimeSpan span)
{
if (span.TotalHours >= 1)
return $"{(long)span.TotalHours}h {span.Minutes:D2}m";
if (span.TotalMinutes >= 1)
return $"{span.Minutes}m {span.Seconds:D2}s";
return $"{(long)Math.Ceiling(span.TotalSeconds)}s";
}
/// <summary>Leaf file name of a kernel device path like <c>\Device\HarddiskVolume3\Windows\System32\svchost.exe</c>.</summary>
internal static string ProcessLeafName(string devicePath)
{
if (string.IsNullOrEmpty(devicePath)) return null;
var idx = devicePath.LastIndexOf('\\');
var leaf = idx >= 0 ? devicePath.Substring(idx + 1) : devicePath;
return string.IsNullOrEmpty(leaf) ? null : leaf;
}
// -----------------------------------------------------------------------
// Backfill
// -----------------------------------------------------------------------
private void BackfillRecentEvents()
{
try
{
var lookbackMs = (long)_backfillLookbackMinutes * 60 * 1000;
_logger.Info($"SystemTimeline backfill: scanning {Channel} " +
$"(lookback={_backfillLookbackMinutes}min, restartWatermark={_restartWatermark})");
var query = new EventLogQuery(Channel, PathType.LogName, BuildBackfillXPath(lookbackMs));
int processed = 0;
using (var reader = new EventLogReader(query))
{
EventRecord record;
while ((record = reader.ReadEvent()) != null)
{
using (record)
{
ProcessRecord(record, isBackfill: true);
processed++;
}
}
}
if (processed == 0)
_logger.Debug($"SystemTimeline backfill: no targeted events in last {_backfillLookbackMinutes} minutes");
else
_logger.Info($"SystemTimeline backfill: scanned {processed} record(s)");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"SystemTimeline event log not found during backfill: {Channel} (normal on non-Windows test environments)");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"SystemTimeline backfill access denied: {ex.Message}");
}
catch (Exception ex)
{
_logger.Warning($"SystemTimeline backfill failed: {ex.Message}");
}
}
// -----------------------------------------------------------------------
// Watermark dedup (cross-restart)
// -----------------------------------------------------------------------
private bool IsAlreadyProcessed(long recordId)
{
if (recordId < 0) return false; // no RecordId → cannot dedup, let it through
lock (_watermarkLock)
{
return recordId <= _restartWatermark || _seenThisRun.Contains(recordId);
}
}
/// <summary>
/// Atomically claims <paramref name="recordId"/>. Returns true if this is the first time we
/// see it (caller should evaluate/emit), false if it was already processed — either in a
/// PRIOR run (at/below the restart watermark) or already this run (in the seen-set). Only
/// advances + persists the max on a genuinely-new, higher RecordId. Records without a
/// RecordId (-1) are always processed, never tracked or persisted. Note: threshold-suppressed
/// records are claimed too, so a restart's backfill does not re-evaluate them.
/// </summary>
private bool MarkProcessed(long recordId)
{
if (recordId < 0) return true;
long toPersist = -1;
lock (_watermarkLock)
{
if (recordId <= _restartWatermark || !_seenThisRun.Add(recordId))
return false;
if (recordId > _maxEmittedRecordId)
{
_maxEmittedRecordId = recordId;
toPersist = _maxEmittedRecordId;
}
}
if (toPersist >= 0) PersistWatermark(toPersist);
return true;
}
internal void LoadWatermark()
{
if (string.IsNullOrEmpty(_stateDirectory)) return;
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
if (!File.Exists(filePath)) return;
try
{
var json = File.ReadAllText(filePath);
var state = JsonConvert.DeserializeObject<WatermarkState>(json);
if (state != null)
{
lock (_watermarkLock)
{
_restartWatermark = state.LastRecordId;
_maxEmittedRecordId = state.LastRecordId;
}
_logger.Info($"SystemTimeline watermark loaded: lastRecordId={state.LastRecordId} (persisted {state.PersistedUtc:O})");
}
}
catch (Exception ex)
{
_logger.Warning($"Failed to load SystemTimeline watermark: {ex.Message}");
}
}
private void PersistWatermark(long recordId)
{
if (string.IsNullOrEmpty(_stateDirectory)) return;
try
{
Directory.CreateDirectory(_stateDirectory);
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
var state = new WatermarkState { LastRecordId = recordId, PersistedUtc = DateTime.UtcNow };
var json = JsonConvert.SerializeObject(state, Formatting.Indented);
var tempPath = filePath + ".tmp";
File.WriteAllText(tempPath, json);
File.Copy(tempPath, filePath, overwrite: true);
try { File.Delete(tempPath); } catch { }
}
catch (Exception ex)
{
_logger.Warning($"Failed to persist SystemTimeline watermark: {ex.Message}");
}
}
internal sealed class WatermarkState
{
public long LastRecordId { get; set; }
public DateTime PersistedUtc { get; set; }
}
}
}
@@ -0,0 +1,64 @@
#nullable enable
using System;
using System.Net;
using System.Net.Sockets;
using System.Runtime.InteropServices;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
{
/// <summary>
/// ARP resolution via <c>iphlpapi!SendARP</c> — resolves the MAC address of an on-link IPv4
/// host (in practice: the default gateway). IPv4 only by API design; the OS answers from its
/// ARP cache or sends a real ARP request, so the target must be on the local subnet.
/// </summary>
internal static class ArpNativeMethods
{
[DllImport("iphlpapi.dll", ExactSpelling = true)]
private static extern int SendARP(uint destIP, uint srcIP, byte[] macAddress, ref uint macAddressLength);
/// <summary>
/// Fail-soft MAC resolution. Returns false (with a diagnostic in <paramref name="error"/>)
/// for non-IPv4 addresses, SendARP failures, or unexpected reply lengths — never throws.
/// </summary>
internal static bool TryResolveMac(IPAddress? ipv4Address, out byte[] mac, out string? error)
{
mac = new byte[6];
error = null;
try
{
if (ipv4Address == null || ipv4Address.AddressFamily != AddressFamily.InterNetwork)
{
error = "not an IPv4 address";
return false;
}
// SendARP takes the IPv4 address as a uint in network byte order (first octet in
// the lowest byte).
var addressBytes = ipv4Address.GetAddressBytes();
uint dest = ((uint)addressBytes[3] << 24)
| ((uint)addressBytes[2] << 16)
| ((uint)addressBytes[1] << 8)
| addressBytes[0];
uint len = (uint)mac.Length;
var err = SendARP(dest, 0, mac, ref len);
if (err != 0)
{
error = $"SendARP returned {err}";
return false;
}
if (len != 6)
{
error = $"SendARP returned unexpected MAC length {len}";
return false;
}
return true;
}
catch (Exception ex)
{
error = ex.Message;
return false;
}
}
}
}
@@ -0,0 +1,68 @@
#nullable enable
using System.Net;
using System.Net.NetworkInformation;
using System.Net.Sockets;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring
{
/// <summary>
/// Shared "which NIC is the device actually using" heuristic: Up, not Loopback/Tunnel,
/// carries a non-0.0.0.0 default gateway. Fail-soft — any enumeration error yields null.
/// </summary>
internal static class NetworkInterfaceLocator
{
/// <summary>
/// Finds the active network interface: Up, not Loopback/Tunnel, has a non-0.0.0.0 gateway.
/// </summary>
internal static NetworkInterface? FindActiveNetworkInterface()
{
try
{
var interfaces = NetworkInterface.GetAllNetworkInterfaces();
foreach (var nic in interfaces)
{
if (nic.OperationalStatus != OperationalStatus.Up)
continue;
if (nic.NetworkInterfaceType == NetworkInterfaceType.Loopback ||
nic.NetworkInterfaceType == NetworkInterfaceType.Tunnel)
continue;
var ipProps = nic.GetIPProperties();
foreach (var gw in ipProps.GatewayAddresses)
{
if (gw.Address.ToString() != "0.0.0.0")
return nic;
}
}
}
catch
{
// Caller handles null
}
return null;
}
/// <summary>First IPv4 default gateway of the interface (0.0.0.0 excluded), or null.</summary>
internal static IPAddress? GetIpv4Gateway(NetworkInterface nic)
{
try
{
foreach (var gw in nic.GetIPProperties().GatewayAddresses)
{
var address = gw?.Address;
if (address != null
&& address.AddressFamily == AddressFamily.InterNetwork
&& address.ToString() != "0.0.0.0")
{
return address;
}
}
}
catch
{
// Caller handles null
}
return null;
}
}
}
@@ -10,7 +10,9 @@ using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
using AutopilotMonitor.Agent.V2.Core.Security;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
@@ -124,62 +126,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
}
}
private readonly string _imeLogFolder;
private readonly string _agentLogFolder;
private readonly string _agentStateFolder;
private readonly string _agentSpoolFolder;
private readonly string _agentDataFolder;
// Built-in sections come from the Shared catalog (DiagnosticsBuiltInSections.All) — the
// same list the backend serves to the portal, so what administrators see is exactly
// what this service collects. Test seam: section id → folder; production leaves the
// map empty and every section resolves from the catalog's (unexpanded) source folder.
private readonly IReadOnlyDictionary<string, string> _sectionFolderOverrides;
private static readonly string DefaultImeLogFolder =
Environment.ExpandEnvironmentVariables(@"%ProgramData%\Microsoft\IntuneManagementExtension\Logs");
private static readonly string DefaultAgentLogFolder =
Environment.ExpandEnvironmentVariables(Constants.LogDirectory);
private static readonly string DefaultAgentStateFolder =
Environment.ExpandEnvironmentVariables(Constants.StateDirectory);
private static readonly string DefaultAgentSpoolFolder =
Environment.ExpandEnvironmentVariables(Constants.SpoolDirectory);
private static readonly string DefaultAgentDataFolder =
Environment.ExpandEnvironmentVariables(@"%ProgramData%\AutopilotMonitor");
/// <summary>
/// File extensions collected from built-in log folders (Agent + IME).
/// Covers standard logs, structured data, traces, event logs, and diagnostics archives.
/// </summary>
private static readonly string[] LogFilePatterns = new[]
{
"*.log", "*.txt", "*.json", "*.jsonl",
"*.etl", "*.evtx", "*.xml", "*.csv", "*.cab"
};
/// <summary>
/// Patterns collected from the agent state folder. Adds completion markers on top of
/// the standard log patterns so files like <c>enrollment-complete.marker</c> and
/// <c>whiteglove-backfill-state.json</c> reach the diagnostics archive.
/// </summary>
private static readonly string[] StateFilePatterns = new[]
{
"*.log", "*.txt", "*.json", "*.jsonl",
"*.etl", "*.evtx", "*.xml", "*.csv", "*.cab",
"*.complete", "*.marker"
};
/// <summary>
/// Marker patterns from the top-level data directory. Kept tight on purpose: only
/// completion/exit markers, no session.id / bootstrap.json (those are config, not
/// forensic state).
/// </summary>
private static readonly string[] RootMarkerPatterns = new[] { "*.complete", "*.marker" };
/// <summary>
/// Telemetry spool: only the JSON-shaped files. The spool folder may also contain
/// quarantine subfolders, so subfolders are excluded to keep the archive focused
/// on what is actually pending upload.
/// </summary>
private static readonly string[] SpoolFilePatterns = new[] { "*.jsonl", "*.json" };
// Scenario oracle for DiagnosticsSectionCondition.DevicePreparation. Production reads
// the deterministic WDP registry marker (the same gate every other WDP switch uses);
// tests inject a constant so they never touch the registry.
private readonly Func<bool> _devicePreparationProbe;
public DiagnosticsPackageService(AgentConfiguration configuration, AgentLogger logger, BackendApiClient apiClient)
: this(configuration, logger, apiClient, null, null, null, null, null)
@@ -188,7 +144,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
// Test seam: allows xUnit fixtures to redirect log/state/spool/data folders to a
// temp dir without touching real %ProgramData% paths. Production callers always go
// through the public ctor.
// through the public ctor. The five named overrides map onto the catalog's section
// ids; sectionFolderOverrides covers every other section (RealmJoin*, ImeBootstrapper…).
internal DiagnosticsPackageService(
AgentConfiguration configuration,
AgentLogger logger,
@@ -197,18 +154,38 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
string imeLogFolderOverride,
string agentStateFolderOverride,
string agentSpoolFolderOverride,
string agentDataFolderOverride)
string agentDataFolderOverride,
IReadOnlyDictionary<string, string> sectionFolderOverrides = null,
Func<bool> devicePreparationProbe = null)
{
_configuration = configuration;
_logger = logger;
_apiClient = apiClient;
_httpClient = new HttpClient { Timeout = TimeSpan.FromMinutes(5) };
_agentLogFolder = agentLogFolderOverride ?? DefaultAgentLogFolder;
_imeLogFolder = imeLogFolderOverride ?? DefaultImeLogFolder;
_agentStateFolder = agentStateFolderOverride ?? DefaultAgentStateFolder;
_agentSpoolFolder = agentSpoolFolderOverride ?? DefaultAgentSpoolFolder;
_agentDataFolder = agentDataFolderOverride ?? DefaultAgentDataFolder;
var overrides = new Dictionary<string, string>(StringComparer.Ordinal);
void Map(string sectionId, string folder)
{
if (folder != null) overrides[sectionId] = folder;
}
Map("AgentLogs", agentLogFolderOverride);
Map("ImeLogs", imeLogFolderOverride);
Map("AgentState", agentStateFolderOverride);
Map("AgentSpool", agentSpoolFolderOverride);
Map("AgentMarkers", agentDataFolderOverride);
if (sectionFolderOverrides != null)
{
foreach (var kv in sectionFolderOverrides)
overrides[kv.Key] = kv.Value;
}
_sectionFolderOverrides = overrides;
// Tests that pass folder overrides but no probe must stay registry-free: a seam
// caller gets "not WDP" by default, the production ctor gets the real marker.
_devicePreparationProbe = devicePreparationProbe
?? (overrides.Count == 0
? (Func<bool>)EnrollmentRegistryDetector.IsDeterministicDevicePreparation
: () => false);
}
/// <summary>
@@ -370,33 +347,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
// 1. sessioninfo.txt — informational metadata, not subject to caps.
AddSessionInfo(archive, enrollmentSucceeded);
// 2. Agent logs
foreach (var pattern in LogFilePatterns)
AddLogFiles(archive, _agentLogFolder, "AgentLogs", pattern, tracker);
// 2. Built-in sections — ONE catalog shared with the backend (the portal
// shows exactly this list): agent logs, IME logs (+ the Device Preparation
// bootstrapper event log on WDP), agent state (recursive: `.quarantine`
// and `.part1-<utc>/` buckets ride along), telemetry spool, top-level
// markers, and the RealmJoin logs behind the tenant's watcher toggle.
// The scenario probe is a registry read — evaluate it once per build.
var isDevicePreparation = ProbeDevicePreparation();
ManifestLine($"SCENARIO: devicePreparation={isDevicePreparation} realmJoinWatcher={_configuration.EnableRealmJoinWatcher}");
foreach (var section in DiagnosticsBuiltInSections.All)
AddBuiltInSection(archive, section, tracker, isDevicePreparation);
// 3. IME logs (all relevant files in the IME log folder)
foreach (var pattern in LogFilePatterns)
AddLogFiles(archive, _imeLogFolder, "ImeLogs", pattern, tracker);
// 4. Agent state: snapshot, journal, signal-log, ime-tracker-state,
// enrollment-complete.marker, final-status.json, whiteglove-backfill-state.json.
// Recursive to include `.quarantine` subfolders AND `.part1-<utc>/` buckets
// that StateArchiver creates on WG Part-2 boot — both ride along under
// AgentState/ via the same recursive walk.
foreach (var pattern in StateFilePatterns)
AddLogFiles(archive, _agentStateFolder, "AgentState", pattern, tracker, includeSubfolders: true);
// 5. Telemetry spool — pending uploads + upload cursor (forensic: what didn't ship?)
foreach (var pattern in SpoolFilePatterns)
AddLogFiles(archive, _agentSpoolFolder, "AgentSpool", pattern, tracker, includeSubfolders: false);
// 6. Top-level markers (whiteglove.complete, clean-exit marker) — Part-1/Part-2
// bridge + ghost-restart guard. Top-directory only to avoid re-zipping
// State/Spool/Logs/Agent which already have their own sections.
foreach (var pattern in RootMarkerPatterns)
AddLogFiles(archive, _agentDataFolder, "AgentMarkers", pattern, tracker, includeSubfolders: false);
// 7. Configured additional log paths (global + tenant, validated by guards)
// 3. Configured additional log paths (global + tenant, validated by guards)
foreach (var entry in _configuration.DiagnosticsLogPaths ?? new System.Collections.Generic.List<Shared.Models.DiagnosticsLogPath>())
{
// Resolve %LOGGED_ON_USER_PROFILE% token and get profile path for guard exception
@@ -447,6 +409,81 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
}
}
private bool ProbeDevicePreparation()
{
try
{
return _devicePreparationProbe();
}
catch (Exception ex)
{
// Fail toward Classic like every other WDP gate: the evtx export is skipped,
// never the whole package.
_logger.Warning($"Device Preparation probe failed — treating as Classic: {ex.Message}");
ManifestLine($"SCENARIO PROBE FAILED (treated as Classic): {ex.Message}");
return false;
}
}
/// <summary>
/// Pure gate: is this built-in section collected under the given configuration and
/// enrollment scenario? Unknown conditions (a newer Shared catalog than this agent)
/// fail closed.
/// </summary>
internal static bool IsSectionActive(DiagnosticsBuiltInSection section, AgentConfiguration configuration, bool isDevicePreparation)
{
switch (section.Condition)
{
case DiagnosticsSectionCondition.Always:
return true;
case DiagnosticsSectionCondition.RealmJoinWatcher:
return configuration.EnableRealmJoinWatcher;
case DiagnosticsSectionCondition.DevicePreparation:
return isDevicePreparation;
default:
return false;
}
}
private static string DescribeInactiveCondition(DiagnosticsSectionCondition condition)
{
switch (condition)
{
case DiagnosticsSectionCondition.RealmJoinWatcher:
return "RealmJoin Watcher disabled";
case DiagnosticsSectionCondition.DevicePreparation:
return "not a Device Preparation enrollment";
default:
return $"unknown condition {condition}";
}
}
private void AddBuiltInSection(ZipArchive archive, DiagnosticsBuiltInSection section, BudgetTracker tracker, bool isDevicePreparation)
{
if (!IsSectionActive(section, _configuration, isDevicePreparation))
{
ManifestLine($"BUILT-IN SKIPPED ({DescribeInactiveCondition(section.Condition)}): {section.Id}");
return;
}
// Override (test seam) or catalog source folder with %ProgramData% /
// %LOGGED_ON_USER_PROFILE% expanded; null means the token is present but no
// interactive user has been detected yet — same skip as configured paths.
var folder = _sectionFolderOverrides.TryGetValue(section.Id, out var overrideFolder)
? overrideFolder
: UserProfileResolver.ExpandCustomTokens(section.SourceFolder);
if (folder == null)
{
_logger.Warning($"Built-in diagnostics section skipped (no user session for token): {section.Id}");
ManifestLine($"BUILT-IN SKIPPED (no user session for token): {section.Id} '{section.SourceFolder}'");
return;
}
ManifestLine($"BUILT-IN: {section.Id} -> folder='{folder}' zip='{section.ZipFolder}' recursive={section.IncludeSubfolders}");
foreach (var pattern in section.Patterns)
AddLogFiles(archive, folder, section.ZipFolder, pattern, tracker, section.IncludeSubfolders);
}
private void AddSessionInfo(ZipArchive archive, bool? enrollmentSucceeded)
{
var sb = new StringBuilder();
@@ -20,7 +20,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
/// </summary>
public static class UserProfileResolver
{
public const string Token = "%LOGGED_ON_USER_PROFILE%";
// Spelled once, in Shared: the built-in diagnostics catalog uses the same token.
public const string Token = Shared.Models.DiagnosticsBuiltInSections.UserProfileToken;
private static readonly object Lock = new object();
private static bool _resolved;
@@ -922,36 +922,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
}
}
/// <summary>
/// Finds the active network interface: Up, not Loopback/Tunnel, has a non-0.0.0.0 gateway.
/// </summary>
private static NetworkInterface FindActiveNetworkInterface()
{
try
{
var interfaces = NetworkInterface.GetAllNetworkInterfaces();
foreach (var nic in interfaces)
{
if (nic.OperationalStatus != OperationalStatus.Up)
continue;
if (nic.NetworkInterfaceType == NetworkInterfaceType.Loopback ||
nic.NetworkInterfaceType == NetworkInterfaceType.Tunnel)
continue;
var ipProps = nic.GetIPProperties();
foreach (var gw in ipProps.GatewayAddresses)
{
if (gw.Address.ToString() != "0.0.0.0")
return nic;
}
}
}
catch
{
// Caller handles null
}
return null;
}
private static NetworkInterface FindActiveNetworkInterface() =>
Monitoring.NetworkInterfaceLocator.FindActiveNetworkInterface();
/// <summary>
/// Collects WiFi signal info via the native WLAN API (<see cref="WifiInfoReader"/>,
@@ -420,36 +420,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
// Helpers
// -------------------------------------------------------------------
/// <summary>
/// Finds the active network interface: Up, not Loopback/Tunnel, has a non-0.0.0.0 gateway.
/// </summary>
private static NetworkInterface FindActiveNetworkInterface()
{
try
{
var interfaces = NetworkInterface.GetAllNetworkInterfaces();
foreach (var nic in interfaces)
{
if (nic.OperationalStatus != OperationalStatus.Up)
continue;
if (nic.NetworkInterfaceType == NetworkInterfaceType.Loopback ||
nic.NetworkInterfaceType == NetworkInterfaceType.Tunnel)
continue;
var ipProps = nic.GetIPProperties();
foreach (var gw in ipProps.GatewayAddresses)
{
if (gw.Address.ToString() != "0.0.0.0")
return nic;
}
}
}
catch
{
// Caller handles null
}
return null;
}
private static NetworkInterface FindActiveNetworkInterface() =>
NetworkInterfaceLocator.FindActiveNetworkInterface();
private static string BuildChangeMessage(NetworkStateSnapshot before, NetworkStateSnapshot after)
{
@@ -0,0 +1,217 @@
#nullable enable
using System;
using System.Net;
using System.Net.Sockets;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic;
using Microsoft.Win32;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
{
public class DoGroupIdSetResult
{
/// <summary>The DOGroupId value is the computed one after this run (written or already there).</summary>
public bool Success { get; set; }
/// <summary>True when this run actually wrote the registry value (false for the idempotent no-op).</summary>
public bool Written { get; set; }
/// <summary>Deliberate non-action (existing foreign policy/value, no usable network) — not an error.</summary>
public bool Skipped { get; set; }
public string? SkipReason { get; set; }
public string? GroupId { get; set; }
public string? PreviousValue { get; set; }
public string? GatewayIp { get; set; }
public string? GatewayMac { get; set; }
public string? Error { get; set; }
}
internal enum DoGroupIdAction
{
Write,
NoOp,
SkipForeign,
}
/// <summary>
/// Sets the Delivery Optimization group ID policy value (<c>DOGroupId</c> under the GPO
/// DeliveryOptimization key) from a network fingerprint: a deterministic GUID derived from the
/// default gateway's IPv4 address and MAC address, so all devices behind the same gateway land
/// in the same DO peering group. The byte layout is RealmJoin-compatible — devices managed by
/// RealmJoin on the same network compute the identical GUID and peer with ours.
/// <para>
/// Deliberately conservative where RealmJoin is not: no random-GUID fallback (a per-device
/// random ID would isolate the device in a one-member group — worse than the OS default
/// grouping), and existing foreign configuration always wins: a DOGroupIdSource policy, an
/// MDM-managed DOGroupId, or a GPO value we did not write ourselves all result in a skip.
/// Never throws.
/// </para>
/// </summary>
public static class DoGroupIdService
{
internal const string GroupIdValueName = "DOGroupId";
internal const string GroupIdSourceValueName = "DOGroupIdSource";
// RealmJoin's fingerprint-GUID prefix — part of the compatible byte layout, do not change.
private static readonly byte[] GuidPrefix = { 101, 48, 6 };
/// <summary>
/// Computes and applies the fingerprint GroupId. <paramref name="lastWrittenGroupId"/> is
/// the GUID a previous agent run wrote (null when none is recorded) — it lets us update our
/// own value after a gateway change while never touching a foreign one.
/// </summary>
public static DoGroupIdSetResult TrySetGroupId(string? lastWrittenGroupId, AgentLogger logger)
{
var result = new DoGroupIdSetResult();
try
{
// ---- Network fingerprint -------------------------------------------------
var nic = NetworkInterfaceLocator.FindActiveNetworkInterface();
var gateway = nic == null ? null : NetworkInterfaceLocator.GetIpv4Gateway(nic);
if (gateway == null)
{
result.Skipped = true;
result.SkipReason = "no_ipv4_gateway";
logger.Info("DO GroupId auto-set: no active interface with an IPv4 default gateway — skipping.");
return result;
}
result.GatewayIp = gateway.ToString();
if (!ArpNativeMethods.TryResolveMac(gateway, out var mac, out var arpError))
{
result.Skipped = true;
result.SkipReason = "arp_failed";
result.Error = arpError;
logger.Warning($"DO GroupId auto-set: could not resolve gateway MAC ({arpError}) — skipping.");
return result;
}
result.GatewayMac = BitConverter.ToString(mac);
var computed = ComputeGroupId(gateway.GetAddressBytes(), mac).ToString();
result.GroupId = computed;
// ---- Foreign-configuration checks ----------------------------------------
// DOGroupIdSource (either store) selects a dynamic group source — a fingerprint
// GroupId would fight it. An MDM-managed DOGroupId means Intune owns the setting.
if (ReadPolicyValue(DoThrottlePolicyReader.GpoKeyPath, GroupIdSourceValueName) != null
|| ReadPolicyValue(DoThrottlePolicyReader.MdmKeyPath, GroupIdSourceValueName) != null)
{
result.Skipped = true;
result.SkipReason = "group_id_source_policy";
logger.Info("DO GroupId auto-set: DOGroupIdSource policy is configured — skipping.");
return result;
}
if (ReadPolicyValue(DoThrottlePolicyReader.MdmKeyPath, GroupIdValueName) != null)
{
result.Skipped = true;
result.SkipReason = "mdm_policy_present";
logger.Info("DO GroupId auto-set: DOGroupId is managed via MDM policy — skipping.");
return result;
}
// ---- Ownership decision ---------------------------------------------------
var existing = ReadPolicyValue(DoThrottlePolicyReader.GpoKeyPath, GroupIdValueName) as string;
result.PreviousValue = existing;
switch (DecideAction(existing, computed, lastWrittenGroupId))
{
case DoGroupIdAction.NoOp:
result.Success = true;
logger.Info($"DO GroupId auto-set: already set to {computed} — nothing to do.");
return result;
case DoGroupIdAction.SkipForeign:
result.Skipped = true;
result.SkipReason = "foreign_value_present";
logger.Info($"DO GroupId auto-set: existing DOGroupId '{existing}' was not written by us — skipping.");
return result;
}
// ---- Write + readback -----------------------------------------------------
using (var hklm = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, RegistryView.Registry64))
using (var key = hklm.CreateSubKey(DoThrottlePolicyReader.GpoKeyPath))
{
key.SetValue(GroupIdValueName, computed, RegistryValueKind.String);
}
var readback = ReadPolicyValue(DoThrottlePolicyReader.GpoKeyPath, GroupIdValueName) as string;
if (!string.Equals(readback, computed, StringComparison.OrdinalIgnoreCase))
{
result.Error = $"readback mismatch: wrote '{computed}', read '{readback ?? "<null>"}'";
logger.Warning($"DO GroupId auto-set failed: {result.Error}");
return result;
}
result.Success = true;
result.Written = true;
logger.Info($"DO GroupId auto-set: DOGroupId set to {computed} (gateway {result.GatewayIp}, previous: {existing ?? "<none>"})");
return result;
}
catch (Exception ex)
{
result.Error = ex.Message;
logger.Warning($"DO GroupId auto-set failed: {ex.Message}");
return result;
}
}
/// <summary>
/// RealmJoin-compatible fingerprint GUID: 16 bytes laid out as
/// prefix {101,48,6} + {0,0,0} + {ip[1],0,ip[2],ip[3]} + mac[6], passed to
/// <c>new Guid(byte[])</c>. The IP byte shuffle mirrors RealmJoin's "swap for
/// new Guid()" step — the layout must stay bit-identical for cross-product peering.
/// </summary>
internal static Guid ComputeGroupId(byte[] ipv4Bytes, byte[] macBytes)
{
if (ipv4Bytes == null || ipv4Bytes.Length != 4)
throw new ArgumentException("IPv4 address bytes required", nameof(ipv4Bytes));
if (macBytes == null || macBytes.Length != 6)
throw new ArgumentException("6-byte MAC required", nameof(macBytes));
var bytes = new byte[16];
Array.Copy(GuidPrefix, 0, bytes, 0, GuidPrefix.Length);
// bytes[3..5] stay zero (RealmJoin's zero padding)
bytes[6] = ipv4Bytes[1];
// bytes[7] stays zero
bytes[8] = ipv4Bytes[2];
bytes[9] = ipv4Bytes[3];
Array.Copy(macBytes, 0, bytes, 10, macBytes.Length);
return new Guid(bytes);
}
/// <summary>
/// Pure ownership decision. GUID comparison tolerates case and braces so a value written
/// by another tool in "{...}" form still matches semantically.
/// </summary>
internal static DoGroupIdAction DecideAction(string? existing, string computed, string? lastWrittenByUs)
{
if (string.IsNullOrEmpty(existing))
return DoGroupIdAction.Write;
if (GuidEquals(existing, computed))
return DoGroupIdAction.NoOp;
if (GuidEquals(existing, lastWrittenByUs))
return DoGroupIdAction.Write; // our own stale value — gateway changed
return DoGroupIdAction.SkipForeign;
}
private static bool GuidEquals(string? a, string? b)
{
if (string.IsNullOrEmpty(a) || string.IsNullOrEmpty(b)) return false;
return Guid.TryParse(a, out var ga) && Guid.TryParse(b, out var gb) && ga == gb;
}
private static object? ReadPolicyValue(string keyPath, string valueName)
{
// Registry64 explicitly — SOFTWARE\Policies is WOW64-redirected, and the policy
// stores live in the 64-bit view regardless of the agent's bitness.
using (var hklm = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, RegistryView.Registry64))
using (var key = hklm.OpenSubKey(keyPath))
{
return key?.GetValue(valueName);
}
}
}
}
@@ -0,0 +1,38 @@
using System;
using System.Net.NetworkInformation;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
{
/// <summary>
/// Boot-time NIC grace shared by every first-contact backend call (remote-config fetch,
/// session registration, emergency-break reporter pattern): polls the cheap link-level
/// signal once per second for at most <paramref name="maxWait"/>. It cannot prove backend
/// reachability — the callers' retry loops handle that — it only keeps retry budgets from
/// being burned into a link that is provably still down (BootTrigger relaunch after a
/// mid-enrollment reboot, Wi-Fi still associating; tenant aebdce78 audits 2026-08-23/24).
/// Free on the normal path — a live link returns immediately. Probe errors end the wait,
/// never the caller's operation.
/// </summary>
internal static class NetworkLinkWait
{
internal static async Task WaitAsync(AgentLogger logger, TimeSpan maxWait, string context)
{
if (NetworkInterface.GetIsNetworkAvailable()) return;
logger?.Info($"{context}: no network link yet — waiting up to {maxWait.TotalSeconds:F0}s.");
var deadline = DateTime.UtcNow + maxWait;
while (DateTime.UtcNow < deadline)
{
await Task.Delay(1000).ConfigureAwait(false);
if (NetworkInterface.GetIsNetworkAvailable())
{
logger?.Info($"{context}: network link is up.");
return;
}
}
logger?.Warning($"{context}: still no network link after {maxWait.TotalSeconds:F0}s — attempting anyway.");
}
}
}
@@ -557,6 +557,22 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
stateDirectory: _stateDirectory));
}
// System timeline watcher — System channel: Kernel-General 1 (system clock stepped,
// ground truth for time jumps incl. old/new time + setting process) and completed
// sleep/standby episodes (Power-Troubleshooter 1, Kernel-Power 507). Backfill recovers
// the pre-agent OOBE clock correction and standby episodes from the persistent .evtx.
if (collectors.EnableSystemTimelineWatcher)
{
hosts.Add(new SystemTimelineWatcherHost(
sessionId: sessionId,
tenantId: tenantId,
logger: logger,
ingress: ingress,
clock: clock,
backfillLookbackMinutes: collectors.SystemEventBackfillLookbackMinutes,
stateDirectory: _stateDirectory));
}
// ----- Peripheral hosts (event-only; driven by remote-config toggles) --------------
// V1 parity (PeriodicCollectorManager) — combine Performance + AgentSelfMetrics under
@@ -595,6 +611,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
logger: logger,
apiBaseUrl: _agentConfig.ApiBaseUrl));
// Live AC/battery watcher — WMI Win32_PowerManagementEvent push (no polling), emits
// power_state_change on AC↔battery transitions and 50/30/15 % downward threshold
// crossings. Deliberately NOT under PeriodicCollectorLifecycleHost: it is push-based
// (zero idle cost) and must stay alive precisely when the device sits idle on battery.
// Never arms on devices without a battery. Kill-switchable via remote config.
if (collectors.EnablePowerStateWatcher)
{
hosts.Add(new PowerStateWatcherHost(
sessionId: sessionId,
tenantId: tenantId,
ingress: ingress,
clock: clock,
logger: logger));
}
// Microsoft 365 Apps (Office C2R) install detector — event-driven (Rev 2): woken by a WMI
// Win32_ProcessStartTrace push on OfficeC2RClient.exe (+ startup probe), progress via
// RegNotifyChangeKeyValue, stop via Process.Exited. No idle polling. Constructed first so
@@ -0,0 +1,332 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Management;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Runtime;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
/// <summary>
/// Live AC/battery watcher: subscribes to the WMI push event
/// <c>Win32_PowerManagementEvent WHERE EventCode = 10</c> (PBT_APMPOWERSTATUSCHANGE — fires on
/// AC↔DC switches and battery-percentage steps), falling back to the intrinsic
/// <c>__InstanceModificationEvent WITHIN 30 … Win32_Battery</c> where the legacy push provider
/// no longer loads (Windows 11 24H2/25H2, builds 26200+: activation fails WBEM_E_NOT_FOUND
/// although the provider is still registered — session 161b838c). Either way the arriving
/// event's payload is ignored; each arrival re-probes via <see cref="PowerStateProbe"/> and
/// diffs the snapshots through the pure <see cref="PowerStateTransitionTracker"/> into
/// <c>power_state_change</c> events. No polling in this process (the fallback's WITHIN
/// sampling runs inside WinMgmt). Complements the one-shot startup
/// <c>power_state_check</c> (<c>StartupEnvironmentProbes</c>), which stays the baseline event.
/// <para>
/// Devices without a battery (or with a failing probe) never arm the WMI subscription — zero
/// overhead on desktops/VMs. WMI arming runs on a background thread (WinMgmt can hang in OOBE;
/// same rationale as <c>ConsoleBypassWatcher</c>), and an arm failure is surfaced once as
/// <c>collector_degraded</c>. A 5 s trailing-edge debounce collapses docking flaps to a
/// no-diff; the tracker's lifetime emission cap is the storm backstop.
/// </para>
/// </summary>
internal sealed class PowerStateWatcherHost : ICollectorHost
{
public string Name => "PowerStateWatcher";
private static readonly TimeSpan DebounceDelay = TimeSpan.FromSeconds(5);
private readonly string _sessionId;
private readonly string _tenantId;
private readonly AgentLogger _logger;
private readonly InformationalEventPost _post;
private readonly Func<PowerStateResult> _probe;
private readonly PowerStateTransitionTracker _tracker = new PowerStateTransitionTracker();
private readonly object _lock = new object();
private readonly object _tickLock = new object();
private ManagementEventWatcher? _watcher;
private Timer? _debounceTimer;
private bool _started;
private bool _disposed;
private bool _capWarned;
/// <param name="probe">Power-state snapshot source; test seam, defaults to
/// <see cref="PowerStateProbe.Probe"/>. The ctor never calls it (the factory constructs
/// hosts eagerly; all probing happens in <see cref="Start"/>'s background thread).</param>
public PowerStateWatcherHost(
string sessionId,
string tenantId,
ISignalIngressSink ingress,
IClock clock,
AgentLogger logger,
Func<PowerStateResult>? probe = null)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
_sessionId = sessionId;
_tenantId = tenantId;
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_post = new InformationalEventPost(ingress, clock, logger);
_probe = probe ?? PowerStateProbe.Probe;
}
public void Start()
{
lock (_lock)
{
if (_started || _disposed) return;
_started = true;
}
// Baseline probe + WMI arming are synchronous WinMgmt-adjacent operations and WinMgmt
// can hang during OOBE — never run them inline on the orchestrator's sequential
// host-start loop (ConsoleBypassWatcher precedent). Disposal racing the background
// start is handled by the _disposed re-checks inside StartCore.
System.Threading.Tasks.Task.Run(() => StartCore(armWmi: true));
}
/// <summary>Internal for tests: the synchronous start body; tests pass
/// <paramref name="armWmi"/> = false so no real WMI subscription is created
/// (precedent: ConsoleBypassWatcher leaves WMI arming untested).</summary>
internal void StartCore(bool armWmi)
{
PowerStateResult baseline;
try
{
baseline = _probe();
}
catch (Exception ex)
{
_logger.Warning($"[{Name}] baseline probe threw ({ex.Message}) — power-state watcher not armed");
return;
}
if (baseline.ProbeError != null)
{
_logger.Info($"[{Name}] power probe failed ({baseline.ProbeError}) — watcher not armed");
return;
}
if (!baseline.HasBattery)
{
_logger.Info($"[{Name}] no battery present (desktop/VM) — watcher not armed");
return;
}
// Latch the baseline BEFORE arming so the first WMI push diffs against a real state.
// A session already on battery below a ladder level emits its threshold event here.
EmitAll(_tracker.Baseline(baseline));
if (!armWmi) return;
// Arm chain: the extrinsic push event first (zero WinMgmt cost, still alive on older
// builds), then the intrinsic Win32_Battery modification event. On Windows 11 24H2/25H2
// (observed on builds 26200/26220, session 161b838c) the MS_Power_Management_Event_Provider
// is still REGISTERED but its activation fails with WBEM_E_NOT_FOUND — the legacy provider
// no longer loads, so the push query throws on exactly the devices that enroll today.
// The fallback's WITHIN polling runs inside WinMgmt (a Win32_Battery snapshot every 30 s),
// NOT in this process; the handler path is identical (payload ignored, re-probe + diff),
// worst-case detection latency ≈ 35 s incl. debounce.
if (TryArmWatcher(
"SELECT * FROM Win32_PowerManagementEvent WHERE EventCode = 10",
"power_management_event (push)", baseline, out var pushError))
{
return;
}
if (TryArmWatcher(
"SELECT * FROM __InstanceModificationEvent WITHIN 30 WHERE TargetInstance ISA 'Win32_Battery'",
"battery_instance_poll (WinMgmt WITHIN 30s)", baseline, out var fallbackError))
{
return;
}
_logger.Warning($"[{Name}] could not arm any WMI power watcher — " +
$"push: {Describe(pushError)}; fallback: {Describe(fallbackError)} — " +
"live power-state transitions will not be observed this run");
CollectorDegradationReporter.Report(_post, _sessionId, _tenantId, Name, "watcher_arm_failed",
fallbackError ?? pushError);
}
private static string Describe(Exception? ex)
=> ex == null ? "n/a" : $"{ex.GetType().Name}: {ex.Message}";
/// <summary>
/// One arm attempt for the given WQL event query, with the full dispose-race dance
/// (ConsoleBypassWatcher pattern). Returns true when the chain should stop: the watcher
/// armed, or disposal won the race (then arming anything would leak a subscription).
/// Returns false with the captured exception so the caller can try the next query.
/// </summary>
private bool TryArmWatcher(string wql, string modeLabel, PowerStateResult baseline, out Exception? error)
{
error = null;
try
{
var watcher = new ManagementEventWatcher(new WqlEventQuery(wql));
watcher.EventArrived += OnPowerStatusChange;
lock (_lock)
{
if (_disposed)
{
// Dispose won the race against the background start — don't arm.
try { watcher.EventArrived -= OnPowerStatusChange; } catch { }
try { watcher.Dispose(); } catch { }
return true;
}
_watcher = watcher;
}
watcher.Start();
// Re-check after arming: Dispose() may have run between the lock release above and
// Start() — it then stopped a not-yet-armed watcher and this thread armed it right
// back. If disposal won that race, tear the watcher down here.
bool disposedDuringArm;
lock (_lock)
{
disposedDuringArm = _disposed;
if (disposedDuringArm) _watcher = null;
}
if (disposedDuringArm)
{
try { watcher.EventArrived -= OnPowerStatusChange; } catch { }
try { watcher.Stop(); } catch { }
try { watcher.Dispose(); } catch { }
return true;
}
_logger.Info($"[{Name}] armed via {modeLabel} — " +
$"baseline: {(baseline.OnAcPower ? "AC" : "battery")}, {baseline.BatteryPercent?.ToString() ?? "?"}%");
return true;
}
catch (Exception ex)
{
error = ex;
_logger.Info($"[{Name}] arm attempt '{modeLabel}' failed ({ex.GetType().Name}: {ex.Message})");
lock (_lock) { _watcher = null; }
return false;
}
}
private void OnPowerStatusChange(object sender, EventArrivedEventArgs e)
{
// Push and fallback events are treated identically: the payload is ignored, the event
// only tells us "power status may have changed". Debounce, then re-probe — never call
// watcher.Stop() from inside this callback.
try { ScheduleDebouncedTick(); }
catch (Exception ex) { _logger.Debug($"[{Name}] power event handler error: {ex.Message}"); }
}
private void ScheduleDebouncedTick()
{
lock (_lock) { if (_disposed) return; }
// One trailing-edge tick per change burst (ImeRegistryAppStateHost pattern): a
// dock-flap AC→battery→AC inside the window collapses to a no-diff in the tracker.
var timer = _debounceTimer;
if (timer == null)
{
var created = new Timer(_ => DebouncedTick(), null, DebounceDelay, Timeout.InfiniteTimeSpan);
if (Interlocked.CompareExchange(ref _debounceTimer, created, null) != null)
created.Dispose();
else
return;
timer = _debounceTimer;
}
try { timer?.Change(DebounceDelay, Timeout.InfiniteTimeSpan); }
catch (ObjectDisposedException) { /* stopping */ }
}
/// <summary>Internal for tests: runs one debounced probe/diff/emit cycle synchronously,
/// exactly what the trailing-edge timer callback executes after a WMI push.</summary>
internal void TickForTest() => DebouncedTick();
private void DebouncedTick()
{
lock (_lock) { if (_disposed) return; }
try
{
IReadOnlyList<PowerStateEmission> emissions;
lock (_tickLock) // tracker is not thread-safe; ticks must not interleave
{
emissions = _tracker.Evaluate(_probe());
}
EmitAll(emissions);
if (_tracker.EmissionCapReached && !_capWarned)
{
_capWarned = true;
_logger.Warning($"[{Name}] emission cap reached — further power_state_change events " +
"suppressed this run (flapping power source or dying battery controller)");
}
}
catch (Exception ex)
{
_logger.Debug($"[{Name}] debounced power tick failed: {ex.Message}");
}
}
private void EmitAll(IReadOnlyList<PowerStateEmission> emissions)
{
foreach (var emission in emissions)
{
try
{
var data = new Dictionary<string, object>
{
{ "transition", emission.Transition },
{ "onAcPower", emission.OnAcPower },
{ "batteryPercent", emission.BatteryPercent.HasValue ? (object)emission.BatteryPercent.Value : "unknown" },
{ "isCharging", emission.IsCharging },
{ "batteryLifeMinutes", emission.BatteryLifeMinutes.HasValue ? (object)emission.BatteryLifeMinutes.Value : "unknown" },
};
if (emission.ThresholdPercent.HasValue)
data["thresholdPercent"] = emission.ThresholdPercent.Value;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.PowerStateChange,
Severity = emission.Severity,
Source = Name,
Phase = EnrollmentPhase.Unknown,
Message = emission.Message,
ImmediateUpload = emission.ImmediateUpload,
Data = data,
});
}
catch (Exception ex)
{
_logger.Warning($"[{Name}] failed to emit power_state_change: {ex.Message}");
}
}
}
public void Stop()
{
ManagementEventWatcher? watcher;
lock (_lock)
{
if (_disposed) return;
_disposed = true;
watcher = _watcher;
_watcher = null;
}
if (watcher != null)
{
try { watcher.EventArrived -= OnPowerStatusChange; } catch { }
try { watcher.Stop(); } catch { }
try { watcher.Dispose(); } catch { }
}
_debounceTimer?.Dispose();
_debounceTimer = null;
}
public void Dispose() => Stop();
}
}
@@ -0,0 +1,68 @@
#nullable enable
using System;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
/// <summary>
/// Standalone host for the <see cref="SystemTimelineTracker"/> — watches the <c>System</c>
/// event log for system clock steps (Kernel-General 1) and completed sleep/standby episodes
/// (Power-Troubleshooter 1 / Kernel-Power 507) and forwards them via the shared single-rail
/// ingress. Mirrors <see cref="WindowsUpdateWatcherHost"/>: builds its own
/// <see cref="InformationalEventPost"/> and delegates Start/Stop to the tracker.
/// </summary>
internal sealed class SystemTimelineWatcherHost : ICollectorHost
{
public string Name => "SystemTimelineTracker";
private readonly SystemTimelineTracker _tracker;
private readonly AgentLogger _logger;
private int _disposed;
public SystemTimelineWatcherHost(
string sessionId,
string tenantId,
AgentLogger logger,
ISignalIngressSink ingress,
IClock clock,
int backfillLookbackMinutes,
string stateDirectory)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
var post = new InformationalEventPost(ingress, clock, logger);
_tracker = new SystemTimelineTracker(
sessionId: sessionId,
tenantId: tenantId,
post: post,
logger: logger,
backfillEnabled: backfillLookbackMinutes > 0,
backfillLookbackMinutes: backfillLookbackMinutes,
stateDirectory: stateDirectory);
}
public void Start()
{
_tracker.Start();
_logger.Info("SystemTimelineWatcherHost: started.");
}
public void Stop()
{
try { _tracker.Stop(); }
catch (Exception ex) { _logger.Warning($"SystemTimelineWatcherHost: stop failed: {ex.Message}"); }
}
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
Stop();
}
}
}
@@ -56,6 +56,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// real activity (rare, and worth keeping diagnostics running for).
SharedConstants.EventTypes.ModernDeploymentLog,
// Environment observation, not device/enrollment progress: an idle device draining
// its battery must not keep the periodic collectors alive or push out stall
// detection — the exact scenario the power watcher exists to observe.
SharedConstants.EventTypes.PowerStateChange,
// Same reasoning for the system timeline watcher: a clock step, a wake from standby,
// or a backfilled episode at agent start is environment observation and must not
// reset the idle/stall clocks.
SharedConstants.EventTypes.SystemClockChanged,
SharedConstants.EventTypes.SystemSleepEpisode,
// Agent health / control / transport — not device/enrollment progress.
SharedConstants.EventTypes.CollectorDegraded,
SharedConstants.EventTypes.TelemetryUploadPoisoned,
@@ -124,6 +124,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Persistence
}
}
/// <summary>
/// Read-only peek of the key's current fingerprint (possibly restored from a previous
/// agent run), or null when none is recorded. Like <see cref="HasFingerprint"/> this makes
/// NO claim — use it when the fingerprint carries state the caller needs to read back
/// (e.g. the DO GroupId ownership marker).
/// </summary>
public string? GetFingerprint(string key)
{
if (string.IsNullOrEmpty(key)) return null;
lock (_lock)
{
return _entries.TryGetValue(key, out var entry) ? entry.Fingerprint : null;
}
}
/// <summary>Retry-until-success: true when <see cref="MarkSucceeded"/> latched this key in any run.</summary>
public bool AlreadySucceeded(string key)
{
@@ -25,6 +25,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Runtime
/// Always emitted — offset &gt;60s is Warning, smaller is Info, failure is Warning.</item>
/// <item><b>power_state_check</b> — AC vs. battery snapshot via Win32 <c>GetSystemPowerStatus</c>.
/// On battery with &lt;80% charge is Warning, otherwise Info, probe failure is Warning.</item>
/// <item><b>do_group_id_auto_set</b> — only when <c>EnableDoGroupIdAutoSet=true</c>. Sets the
/// Delivery Optimization <c>DOGroupId</c> policy value from a network fingerprint via
/// <see cref="DoGroupIdService"/>; skips (Info) rather than fight foreign policy values.</item>
/// </list>
/// <para>
/// Each probe runs best-effort: failures are logged + optionally emitted as a Warning event,
@@ -78,6 +81,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Runtime
await RunNtpCheck(configuration, logger, post, gate).ConfigureAwait(false);
}
if (configuration.EnableDoGroupIdAutoSet)
RunDoGroupIdAutoSet(configuration, logger, post, gate);
RunPowerStateCheck(configuration, logger, post);
}
@@ -331,6 +337,99 @@ namespace AutopilotMonitor.Agent.V2.Core.Runtime
};
}
/// <summary>
/// One-shot DO GroupId apply with change-based restart dedup: the gate fingerprint doubles
/// as the ownership marker ("written:&lt;guid&gt;" records the GUID we last wrote, so a later
/// run can distinguish our own stale value from a foreign one after a gateway change).
/// Unchanged outcomes (same GUID, same skip reason) re-run silently without a duplicate
/// event; errors never claim a fingerprint so each failing start keeps its retry trail.
/// </summary>
private static void RunDoGroupIdAutoSet(
AgentConfiguration configuration,
AgentLogger logger,
Orchestration.InformationalEventPost post,
Persistence.StartupEventGate? gate)
{
try
{
var lastWritten = ExtractLastWrittenGroupId(gate?.GetFingerprint(Constants.EventTypes.DoGroupIdAutoSet));
var result = DoGroupIdService.TrySetGroupId(lastWritten, logger);
var fingerprint = ComputeDoGroupIdFingerprint(result);
if (fingerprint != null && gate != null
&& !gate.ShouldEmit(Constants.EventTypes.DoGroupIdAutoSet, fingerprint))
{
logger.Debug("StartupEnvironmentProbes: DO GroupId outcome unchanged — skipping duplicate event (restart dedup).");
return;
}
SafeEmit(post, logger, BuildDoGroupIdEvent(configuration, result));
if (fingerprint != null)
gate?.MarkEmitted(Constants.EventTypes.DoGroupIdAutoSet);
}
catch (Exception ex)
{
logger.Warning($"StartupEnvironmentProbes: DO GroupId auto-set threw: {ex.Message}");
}
}
internal const string DoGroupIdWrittenFingerprintPrefix = "written:";
/// <summary>
/// Success (written or already correct) fingerprints as "written:&lt;guid&gt;", a skip as
/// "skipped:&lt;reason&gt;"; errors return null so they are never deduped.
/// </summary>
internal static string? ComputeDoGroupIdFingerprint(DoGroupIdSetResult result)
{
if (result.Success) return DoGroupIdWrittenFingerprintPrefix + result.GroupId;
if (result.Skipped) return "skipped:" + result.SkipReason;
return null;
}
/// <summary>Ownership marker readback: the GUID from a "written:" fingerprint, else null.</summary>
internal static string? ExtractLastWrittenGroupId(string? fingerprint) =>
fingerprint != null && fingerprint.StartsWith(DoGroupIdWrittenFingerprintPrefix, StringComparison.Ordinal)
? fingerprint.Substring(DoGroupIdWrittenFingerprintPrefix.Length)
: null;
internal static EnrollmentEvent BuildDoGroupIdEvent(AgentConfiguration configuration, DoGroupIdSetResult result)
{
string message;
if (result.Success && result.Written)
message = $"Delivery Optimization GroupId set to {result.GroupId} (gateway {result.GatewayIp})";
else if (result.Success)
message = $"Delivery Optimization GroupId already set to {result.GroupId}";
else if (result.Skipped)
message = $"Delivery Optimization GroupId not set: {result.SkipReason}";
else
message = $"Delivery Optimization GroupId auto-set failed: {result.Error}";
return new EnrollmentEvent
{
SessionId = configuration.SessionId,
TenantId = configuration.TenantId,
EventType = Constants.EventTypes.DoGroupIdAutoSet,
Severity = result.Success || result.Skipped ? EventSeverity.Info : EventSeverity.Warning,
Source = "StartupEnvironmentProbes",
Phase = EnrollmentPhase.Unknown,
Timestamp = DateTime.UtcNow,
Message = message,
Data = new Dictionary<string, object>
{
{ "groupId", result.GroupId ?? "" },
{ "previousValue", result.PreviousValue ?? "" },
{ "gatewayIp", result.GatewayIp ?? "" },
{ "gatewayMac", result.GatewayMac ?? "" },
{ "written", result.Written },
{ "skipped", result.Skipped },
{ "skipReason", result.SkipReason ?? "" },
{ "success", result.Success },
{ "error", result.Error ?? "" },
},
};
}
private static void RunPowerStateCheck(
AgentConfiguration configuration,
AgentLogger logger,
@@ -2,6 +2,7 @@ using System;
using System.Linq;
using System.Security.Cryptography.X509Certificates;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Security;
namespace AutopilotMonitor.Agent.V2.Core.Security
{
@@ -23,6 +24,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <item><description><c>1.2.840.113556.1.5.284.2</c> → DeviceId (same encoding)</description></item>
/// </list>
/// <para>
/// The sibling <c>Microsoft Intune MDM Device CA</c> client cert (the one the agent authenticates
/// with, see <see cref="CertificateHelper"/>) carries a DIFFERENT OID arc that identifies the same
/// device from Intune's side. Observed on a live Intune-enrolled Windows 11 device:
/// </para>
/// <list type="table">
/// <listheader><term>OID</term><description>Meaning / encoding</description></listheader>
/// <item><term><c>1.2.840.113556.5.4</c></term><description>Intune DeviceId — BARE 16-byte GUID, no ASN.1 wrapper. Equals the cert's Subject CN.</description></item>
/// <item><term><c>1.2.840.113556.5.6</c></term><description>Intune AccountId (the Intune tenant account, NOT the Entra TenantId) — OCTET STRING GUID.</description></item>
/// <item><term><c>1.2.840.113556.5.14</c></term><description>Entra TenantId — OCTET STRING GUID. This is the one to compare against an Entra tenant id.</description></item>
/// <item><term><c>1.2.840.113556.5.11</c></term><description>Further GUID, OCTET STRING; semantics not documented by Microsoft and not verified here.</description></item>
/// <item><term><c>1.2.840.113556.5.15/.16/.17/.18/.19/.23</c></term><description>ASN.1 INTEGER enums (platform / enrollment flavour); semantics undocumented.</description></item>
/// </list>
/// <para>
/// The bare-GUID encoding of <c>1.2.840.113556.5.4</c> is why
/// <see cref="ParseGuidFromExtension"/> accepts an unwrapped 16-byte payload as well —
/// a strict OCTET-STRING parser silently returns <c>null</c> for that extension.
/// </para>
/// <para>
/// All probes are best-effort: never throw, return <c>null</c> on miss, log a Warning
/// with the specific failure reason (no matching cert / extension missing / ASN.1
/// parse failed) so the agent log explains why the fallback didn't help.
@@ -31,8 +50,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
public static class EntraDeviceCertHelper
{
public const string MsOrganizationAccessIssuer = "MS-Organization-Access";
public const string TenantIdOid = "1.2.840.113556.1.5.284.5";
public const string DeviceIdOid = "1.2.840.113556.1.5.284.2";
public const string TenantIdOid = MsDeviceCertificateOids.EntraJoinCertTenantIdOid;
public const string DeviceIdOid = MsDeviceCertificateOids.EntraJoinCertDeviceIdOid;
/// <summary>
/// Reads the Entra TenantId from the MS-Organization-Access device cert.
@@ -88,7 +107,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
continue;
}
var guid = ParseGuidFromAsn1OctetString(ext.RawData);
var guid = ParseGuidFromExtension(ext.RawData);
if (guid != null)
{
logger?.Info(
@@ -98,7 +117,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
}
logger?.Warning(
$"EntraDeviceCertHelper: failed to parse {label} ASN.1 OCTET STRING in cert thumbprint={cert.Thumbprint} " +
$"EntraDeviceCertHelper: failed to parse {label} GUID extension in cert thumbprint={cert.Thumbprint} " +
$"(raw {ext.RawData.Length} bytes); trying next candidate.");
}
@@ -116,46 +135,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <summary>
/// Parses a 16-byte MS-format binary GUID (little-endian for the first three groups,
/// per <see cref="Guid(byte[])"/>) wrapped in an ASN.1 OCTET STRING. Handles both
/// short-form (<c>04 10 ...</c>) and long-form (<c>04 81 10 ...</c>) length encoding.
/// per <see cref="Guid(byte[])"/>) out of a certificate extension's raw value.
/// Two encodings are accepted, because Microsoft uses both:
/// <list type="number">
/// <item><description>Wrapped in an ASN.1 OCTET STRING — short-form (<c>04 10 ...</c>) or
/// long-form (<c>04 81 10 ...</c>) length. Used by the MS-Organization-Access TenantId/DeviceId
/// extensions and by the Intune <c>…5.6</c> / <c>…5.14</c> extensions.</description></item>
/// <item><description>A bare, unwrapped 16-byte payload. Used by the Intune DeviceId extension
/// <c>1.2.840.113556.5.4</c> — verified against a live Intune MDM device cert, whose raw value is
/// exactly 16 bytes with no <c>04 10</c> prefix.</description></item>
/// </list>
/// A valid OCTET-STRING wrapper around 16 bytes is at least 18 bytes long, so the two forms
/// never collide for well-formed input. A 16-byte value is always read as the bare form —
/// a truncated wrapper of exactly that length is indistinguishable from it.
/// Returns <c>null</c> on any structural mismatch — never throws.
/// <para>
/// The decoding itself lives in <see cref="MsDeviceCertificateOids.TryParseGuid"/> so the agent
/// and the backend's client-cert tenant binding share one implementation; this stays as the
/// agent-side nullable-returning façade.
/// </para>
/// </summary>
public static Guid? ParseGuidFromAsn1OctetString(byte[] raw)
{
if (raw == null || raw.Length < 2) return null;
// ASN.1 OCTET STRING tag.
if (raw[0] != 0x04) return null;
int contentStart;
int contentLen;
if (raw[1] < 0x80)
{
// short-form length
contentLen = raw[1];
contentStart = 2;
}
else
{
// long-form length: lower 7 bits = number of following length bytes.
int numLenBytes = raw[1] & 0x7F;
if (numLenBytes < 1 || numLenBytes > 4 || raw.Length < 2 + numLenBytes) return null;
contentLen = 0;
for (int i = 0; i < numLenBytes; i++)
contentLen = (contentLen << 8) | raw[2 + i];
contentStart = 2 + numLenBytes;
}
if (contentLen != 16 || contentStart + 16 > raw.Length) return null;
var guidBytes = new byte[16];
Buffer.BlockCopy(raw, contentStart, guidBytes, 0, 16);
try { return new Guid(guidBytes); }
catch { return null; }
}
public static Guid? ParseGuidFromExtension(byte[] raw) =>
MsDeviceCertificateOids.TryParseGuid(raw, out var value) ? value : (Guid?)null;
}
}
@@ -19,7 +19,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <b>Retry contract (V1 <c>MonitoringService.RegisterSessionAsync</c>):</b>
/// </para>
/// <list type="bullet">
/// <item>5 attempts total; between attempts <c>2^attempt</c> seconds (2s, 4s, 8s, 16s).</item>
/// <item>Boot-time NIC grace before the first attempt: when no network link is up yet
/// (BootTrigger relaunch after a mid-enrollment reboot, Wi-Fi still associating) wait
/// up to <see cref="NetworkLinkWaitMax"/> for the link-level signal. Best-effort and
/// free on the normal path — a live link returns immediately.</item>
/// <item>6 attempts total; between attempts <c>2^attempt</c> seconds (2s, 4s, 8s, 16s, 32s —
/// ~62 s of backoff). The original V1 budget of 5 attempts / ~30 s was too short for the
/// reboot-relaunch on Wi-Fi kiosks: the relaunched agent gave up and exited 7 before
/// the network came back, leaving the session silent forever (tenant aebdce78,
/// 2026-08-23 audit). A relaunch that misses here is lost — nothing retries later.</item>
/// <item>On <c>response.Success == true</c> → stop immediately.</item>
/// <item>On <see cref="BackendAuthException"/> (401/403) → report to the
/// <see cref="AuthFailureTracker"/> (which fires the first-failure distress + may trip
@@ -37,16 +45,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// </summary>
public static class SessionRegistrationHelper
{
private const int MaxAttempts = 5;
private const int MaxAttempts = 6;
/// <summary>Upper bound for the pre-registration link wait (see class remarks).</summary>
internal static readonly TimeSpan NetworkLinkWaitMax = TimeSpan.FromSeconds(15);
/// <summary>
/// Executes the 5-retry register-session handshake against the backend. Returns the
/// Executes the retrying register-session handshake against the backend. Returns the
/// response + classified outcome. Never throws.
/// </summary>
/// <param name="backoffDelay">
/// Optional delay-provider used by tests to avoid real-time waits. Production callers
/// pass <c>null</c> → V1 parity <c>Task.Delay(2^attempt * 1000)</c> between attempts.
/// </param>
/// <param name="networkLinkWait">
/// Optional pre-registration link wait used by tests. Production callers pass
/// <c>null</c> → <see cref="WaitForNetworkLinkAsync"/> (bounded by <see cref="NetworkLinkWaitMax"/>).
/// </param>
public static async Task<SessionRegistrationResult> RegisterWithRetryAsync(
BackendApiClient apiClient,
AgentConfiguration agentConfig,
@@ -56,14 +71,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
EmergencyReporter emergencyReporter = null,
Func<int, Task> backoffDelay = null,
Func<Exception, Task> onTerminalTransportFailure = null,
(string Manufacturer, string Model, string SerialNumber)? deviceHardware = null)
(string Manufacturer, string Model, string SerialNumber)? deviceHardware = null,
Func<AgentLogger, Task> networkLinkWait = null)
{
if (apiClient == null) throw new ArgumentNullException(nameof(apiClient));
if (agentConfig == null) throw new ArgumentNullException(nameof(agentConfig));
if (logger == null) throw new ArgumentNullException(nameof(logger));
backoffDelay ??= DefaultBackoffDelay;
networkLinkWait ??= WaitForNetworkLinkAsync;
var registration = BuildRegistration(agentConfig, agentVersion, deviceHardware);
try { await networkLinkWait(logger).ConfigureAwait(false); }
catch (Exception ex) { logger.Debug($"Pre-registration network wait failed ({ex.Message}) — registering anyway."); }
string lastError = null;
Exception lastException = null;
@@ -113,7 +133,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
if (attempt < MaxAttempts)
{
var delaySeconds = (int)Math.Pow(2, attempt); // 2, 4, 8, 16
var delaySeconds = (int)Math.Pow(2, attempt); // 2, 4, 8, 16, 32
logger.Info($"Retrying session registration in {delaySeconds}s");
await backoffDelay(attempt).ConfigureAwait(false);
}
@@ -143,6 +163,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
private static Task DefaultBackoffDelay(int attempt)
=> Task.Delay(((int)Math.Pow(2, attempt)) * 1000);
/// <summary>
/// Boot-time NIC grace before the first registration attempt — shared polling loop in
/// <see cref="NetworkLinkWait"/>, bounded by <see cref="NetworkLinkWaitMax"/>. Probe
/// errors end the wait, never the registration.
/// </summary>
internal static Task WaitForNetworkLinkAsync(AgentLogger logger)
=> NetworkLinkWait.WaitAsync(logger, NetworkLinkWaitMax, "Session registration");
private static SessionRegistration BuildRegistration(
AgentConfiguration agentConfig,
string agentVersion,
@@ -194,7 +222,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <summary>401/403 from the backend — cert/token rejected. No retry was attempted.</summary>
AuthFailed = 1,
/// <summary>All 5 attempts failed with non-auth errors (network, 5xx, malformed response).</summary>
/// <summary>All attempts failed with non-auth errors (network, 5xx, malformed response).</summary>
Failed = 2,
}
@@ -56,6 +56,13 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
{
private const string SourceLabel = "ImeLogTracker";
/// <summary>Timeline label: IME wrote its "Completed user session" line and no required user-scope app was pending.</summary>
internal const string EvidenceUserSessionCompleted = "user_session_completed";
/// <summary>Timeline label: IME reported zero targeted apps for the user session, so it never wrote a
/// "Completed user session" line at all. The user phase had nothing to enforce.</summary>
internal const string EvidenceZeroUserApps = "zero_user_apps_enumerated";
private readonly ImeLogTracker _tracker;
private readonly ISignalIngressSink _ingress;
private readonly IClock _clock;
@@ -407,6 +414,13 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
var offsetMinutes = _tracker.LastMatchedSourceOffsetMinutes;
if (offsetMinutes.HasValue)
data["sourceOffsetMinutes"] = offsetMinutes.Value.ToString(culture);
// Observational only — what the calibrator measured, which since the 2026-08-20
// revert is deliberately NOT what was applied. Kept separate so the two can never
// be read as the same thing.
var measured = _tracker.LastMatchedMeasuredWriterOffsetMinutes;
if (measured.HasValue)
data["measuredWriterOffsetMinutes"] = measured.Value.ToString(culture);
}
if (!derivedFromClock) return;
@@ -420,9 +434,14 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
switch (origin)
{
case CmTraceOffsetOrigin.Bias: return "bias";
// Per-FILE offset application, retired by the 2026-08-20 revert (04b1a7c6).
// Kept so stored 2.0.1410 events remain interpretable; no current path emits it.
case CmTraceOffsetOrigin.Calibrated: return "calibrated";
// The writer's offset was never measured, so this process's own zone was assumed —
// correct only if the writer happens to share it.
// The line anchored itself: read provably fresh, its own distance to the agent
// clock (grid-rounded) was applied as the writer's offset. Era-safe per line.
case CmTraceOffsetOrigin.LineAnchored: return "line-anchored";
// The line was not provably fresh (backlog, restart catch-up, replay), so this
// process's own zone was assumed — correct only if the writer happens to share it.
default: return "reader-zone-fallback";
}
}
@@ -579,9 +598,24 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
var now = ResolveOccurredAt(out var derivedFromClock, out var rawSourceTs);
// Which observation carried the completion. The zero-app variant matters to whoever
// reads the timeline later: the verdict then rests on IME reporting that this user
// had no targeted apps at all, not on watching enforcement finish.
var zeroApps = _tracker.UserSessionZeroAppsObserved;
var evidenceLabel = zeroApps ? EvidenceZeroUserApps : EvidenceUserSessionCompleted;
var summary = zeroApps
? "IME enumerated the user session with zero targeted apps"
: "IME user session completed (no required user-scope apps pending)";
var message = zeroApps
? "IME enumerated the user session with zero targeted apps - no user-phase enforcement to wait for"
: "IME user session completed";
var derivationInputs = new Dictionary<string, string>(StringComparer.Ordinal)
{
["detectionSource"] = "IME log pattern (UserSessionCompleted)",
["detectionSource"] = zeroApps
? "IME log pattern (UserSessionZeroApps)"
: "IME log pattern (UserSessionCompleted)",
["evidence"] = evidenceLabel,
};
TagDerivedTimestamp(derivationInputs, derivedFromClock, rawSourceTs);
@@ -592,12 +626,13 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
evidence: new Evidence(
kind: EvidenceKind.Derived,
identifier: "ime-log-tracker-v1",
summary: "IME user session completed (no required user-scope apps pending)",
summary: summary,
derivationInputs: derivationInputs));
var data = new Dictionary<string, string>(StringComparer.Ordinal)
{
["detectedAt"] = now.ToString("o"),
["evidence"] = evidenceLabel,
};
var patternId = _tracker.LastMatchedPatternId;
if (!string.IsNullOrEmpty(patternId))
@@ -607,12 +642,12 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
_post.Emit(
eventType: SharedEventTypes.ImeUserSessionCompleted,
source: SourceLabel,
message: "IME user session completed",
message: message,
data: data,
occurredAtUtc: now);
// PR3-D4: terminal-ish lifecycle marker — INFO so it's visible at the default level.
_logger?.Info($"ImeAdapter: user session completed -> posting ImeUserSessionCompleted (patternId={patternId ?? "(none)"})");
_logger?.Info($"ImeAdapter: user session completed ({evidenceLabel}) -> posting ImeUserSessionCompleted (patternId={patternId ?? "(none)"})");
}
/// <summary>
@@ -792,12 +827,15 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
/// <see cref="AppInstallTiming"/> so the emitter can inline the values on the
/// current event's payload without a second dictionary lookup.
/// <para>
/// Audit Q1: a terminal→active transition (the IME retry path — <see
/// cref="AppPackageState.UpdateState"/> permits e.g. Error→Downloading and
/// Skipped→Downloading) re-arms <c>Completed</c> to <c>null</c>, so the NEXT terminal
/// transition stamps a fresh CompletedAt. StartedAt keeps its first value — the final
/// terminal payload then covers first-start → last-terminal (total occupancy across
/// attempts) instead of freezing at the first failure. Observed pre-fix: a user-scope
/// Audit Q1 + 2026-08 attempt-duration change: a terminal→active transition (the IME
/// retry / re-evaluation path — <see cref="AppPackageState.UpdateState"/> permits e.g.
/// Error→Downloading and Skipped→Downloading) re-arms BOTH endpoints: <c>Completed</c>
/// to <c>null</c> (the NEXT terminal transition stamps fresh) and <c>Started</c> to
/// the re-entry time. The final terminal payload therefore describes the
/// status-defining ATTEMPT — matching the backend's AppInstallSummaries semantics —
/// instead of first-start → last-terminal across passes: the IME processes the app
/// list repeatedly (device-ESP evaluation pass, user-context pass), and the idle gap
/// between passes is not install time. Observed pre-fix (old bug shape): a user-scope
/// app Skipped early in DeviceSetup, retried after sign-in, emitted its final
/// Installed event with the stale early CompletedAt (durationSeconds=-2342).
/// </para>
@@ -808,11 +846,13 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
{
_appTimings.TryGetValue(appId, out var current);
// Terminal→active (retry): the earlier terminal stamp no longer describes this
// app's outcome — re-arm so the next terminal transition stamps fresh.
// Terminal→active (retry / new pass): the earlier stamps no longer describe
// this app's outcome — re-arm both endpoints so the next terminal transition
// reports the fresh attempt's window.
if (current.Completed != null && IsLifecycleActive(newState))
{
current.Completed = null;
current.Started = now;
}
// First lifecycle-active transition → record StartedAt.
@@ -119,6 +119,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Transport.Telemetry
AddSecurityHeaders(request);
request.Headers.Add("X-Correlation-ID", correlationId);
// P14: device-clock send time, one value per request. Lets the backend separate
// spool delay (SentAt − OccurredUtc, both device frame) from device-vs-server
// clock offset (ReceivedAt − SentAt) — indistinguishable without this header.
// Stamped per attempt on purpose: a retry after backoff is a NEW send moment.
request.Headers.Add("X-Send-Time-Utc", DateTime.UtcNow.ToString("O"));
HttpResponseMessage response;
try
{
@@ -274,6 +274,15 @@ namespace AutopilotMonitor.Agent.V2
{
return registration.ExitCode;
}
// Phase 6.5 (config self-heal): when the Phase 4 fetch ran before the network link
// was up, registration has just proven the backend reachable — retry the fetch once
// so the session runs on the live tenant config instead of cache/defaults.
// See AgentRuntimeConfig.TryRecoverAfterRegistration for what is deliberately
// NOT re-run here (kill-signal check, endpoint migration).
Runtime.AgentRuntimeConfig.TryRecoverAfterRegistration(
bootstrap.AgentConfig, auth, runtimeConfig, GetAgentVersion(), consoleMode, logger);
// Phase 7+8 (orchestrator + shutdown lifecycle + onIngressReady wiring +
// initial signal posts + analyzer/probes startup + shutdown.Wait + finally)
// is encapsulated in AgentRuntimeHost. See Runtime/AgentRuntimeHost.cs.
@@ -69,6 +69,63 @@ namespace AutopilotMonitor.Agent.V2.Runtime
}
}
var configMergeResult = ApplyFetchedConfig(agentConfig, remoteConfig, auth, agentVersion, consoleMode, logger);
return new RuntimeConfigBundle(remoteConfigService, remoteConfig, configMergeResult, pendingMigrationTarget: null);
}
/// <summary>
/// Phase 6.5 — config self-heal. When the Phase 4 fetch ran before the network link was
/// up (agent relaunched at boot, Wi-Fi still associating — tenant aebdce78 audit 2026-08-24:
/// ~50% of sessions ran on ConfigVersion=0 exactly this way), a successful session
/// registration has just proven the backend reachable. Retry the fetch once and apply
/// the live tenant config instead of stranding the whole session on cache/defaults.
/// <para>
/// Deliberately NOT re-run here: the kill-signal check (post-registration kills arrive
/// via the telemetry-channel control signals on every upload response) and endpoint
/// migration (registration already succeeded against the current base URL; a migration
/// signal is honoured at the next agent start). Never throws — recovery is strictly
/// best-effort on top of an already-degraded-but-working fallback.
/// </para>
/// </summary>
public static void TryRecoverAfterRegistration(
AgentConfiguration agentConfig,
BackendAuthBundle auth,
RuntimeConfigBundle bundle,
string agentVersion,
bool consoleMode,
AgentLogger logger)
{
try
{
var service = bundle?.RemoteConfigService;
if (service == null) return;
var recovered = service.TryRecoverConfigAsync().GetAwaiter().GetResult();
if (recovered == null) return;
var mergeResult = ApplyFetchedConfig(agentConfig, recovered, auth, agentVersion, consoleMode, logger);
bundle.ApplyRecovered(recovered, mergeResult);
}
catch (Exception ex)
{
logger.Warning($"Post-registration config recovery failed: {ex.Message}");
}
}
/// <summary>
/// Applies a live-fetched config to the runtime: merge, tracker ceilings, log level,
/// integrity SHA + check, bootstrap-config cleanup. Shared by the Phase 4 startup path
/// and the post-registration recovery path so both apply identical semantics.
/// </summary>
private static RemoteConfigMergeResult ApplyFetchedConfig(
AgentConfiguration agentConfig,
AgentConfigResponse remoteConfig,
BackendAuthBundle auth,
string agentVersion,
bool consoleMode,
AgentLogger logger)
{
// Project remote tenant-controlled knobs onto the runtime AgentConfiguration so that
// downstream consumers (CleanupService, SummaryDialogLauncher, StartupEnvironmentProbes,
// DiagnosticsPackageService, EnrollmentTerminationHandler, the logger, the watchdog)
@@ -91,7 +148,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
TryCleanupBootstrapConfig(agentConfig, agentVersion, logger);
return new RuntimeConfigBundle(remoteConfigService, remoteConfig, configMergeResult, pendingMigrationTarget: null);
return configMergeResult;
}
/// <summary>
@@ -175,10 +232,21 @@ namespace AutopilotMonitor.Agent.V2.Runtime
internal sealed class RuntimeConfigBundle
{
public RemoteConfigService RemoteConfigService { get; }
public AgentConfigResponse RemoteConfig { get; }
public RemoteConfigMergeResult MergeResult { get; }
public AgentConfigResponse RemoteConfig { get; private set; }
public RemoteConfigMergeResult MergeResult { get; private set; }
public string PendingMigrationTarget { get; }
/// <summary>
/// Swaps in the config recovered by <see cref="AgentRuntimeConfig.TryRecoverAfterRegistration"/>
/// so Phase 7+8 (orchestrator, collectors, gather rules, IME patterns, the
/// unrestricted-mode audit) run on the live tenant config instead of the fallback.
/// </summary>
public void ApplyRecovered(AgentConfigResponse recoveredConfig, RemoteConfigMergeResult mergeResult)
{
RemoteConfig = recoveredConfig;
MergeResult = mergeResult;
}
public RuntimeConfigBundle(
RemoteConfigService remoteConfigService,
AgentConfigResponse remoteConfig,
@@ -11,7 +11,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{
/// <summary>
/// Phase 6 of <see cref="Program"/>'s <c>RunAgent</c>: blocking
/// <c>POST /api/agent/register-session</c> with 5-retry exponential backoff before the
/// <c>POST /api/agent/register-session</c> with link-wait + 6-retry exponential backoff before the
/// orchestrator starts. V1 parity (MonitoringService.RegisterSessionAsync) — without
/// this call the backend's Sessions table never gets a row for this session, so
/// <c>IncrementSessionEventCountAsync</c> / <c>UpdateSessionStatusAsync</c> silently
@@ -52,6 +52,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
// prior to this, ConfigVersion=0 was only deducible by inferring from
// the absence of tenant-controlled knobs in the DataJson.
var outcome = remoteConfigService?.LastFetchOutcome ?? RemoteConfigFetchOutcome.NotAttempted;
var recovered = remoteConfigService?.RecoveredAfterFetchFailure == true;
var configVersion = remoteConfigService?.CurrentConfig?.ConfigVersion ?? 0;
var data = new Dictionary<string, object>
@@ -66,8 +67,12 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ "diagnosticsUploadMode", agentConfig.DiagnosticsUploadMode ?? "Off" },
{ "previousExitType", previousExit?.ExitType ?? "unknown" },
{ "unrestrictedMode", agentConfig.UnrestrictedMode },
// remoteConfigFetched answers "is the agent running on a live-fetched
// config" — true also when the post-registration recovery replaced a
// failed startup fetch (remoteConfigOutcome then still names the startup
// failure; remoteConfigRecovered marks the heal).
{ "configVersion", configVersion },
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded },
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded || recovered },
{ "remoteConfigOutcome", outcome.ToString() },
// WinRT OOBE state sampled at this start. Together with previousExitType
// this makes late starts provable: "completed" on a first_run means OOBE
@@ -75,6 +80,9 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ "oobeStateAtAgentStart", OobeStateReader.Read() },
};
if (recovered)
data["remoteConfigRecovered"] = true;
if (!string.IsNullOrEmpty(previousExit?.CrashExceptionType))
data["previousCrashException"] = previousExit.CrashExceptionType;
@@ -135,12 +143,25 @@ namespace AutopilotMonitor.Agent.V2.Runtime
? $"Remote config fetch failed after {remoteConfigService.LastFetchAttempts} attempt(s); using cached config."
: $"Remote config fetch failed after {remoteConfigService.LastFetchAttempts} attempt(s); running with built-in defaults (ConfigVersion=0).";
// Post-registration recovery (Phase 6.5) replaced the fallback with a live
// fetch before this emitter ran: keep the failure details (the network-late
// condition is worth surfacing) but mark the heal and demote to Info — the
// session is NOT degraded.
var recovered = remoteConfigService.RecoveredAfterFetchFailure;
if (recovered)
{
var recoveredVersion = remoteConfigService.CurrentConfig?.ConfigVersion ?? 0;
data["recovered"] = true;
data["recoveredConfigVersion"] = recoveredVersion;
msg += $" Recovered after session registration (ConfigVersion={recoveredVersion}).";
}
post.Emit(new EnrollmentEvent
{
SessionId = agentConfig.SessionId,
TenantId = agentConfig.TenantId,
EventType = SharedConstants.EventTypes.RemoteConfigFetchFailed,
Severity = EventSeverity.Warning,
Severity = recovered ? EventSeverity.Info : EventSeverity.Warning,
Source = "Agent",
Phase = EnrollmentPhase.Unknown,
Message = msg,
@@ -0,0 +1,242 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using Xunit;
namespace AutopilotMonitor.DecisionCore.Tests
{
/// <summary>
/// Sessions 81daa77f / 75d6ae8e (2026-08-25) — the 30-min <c>advisory_completion</c> window
/// overran a completion gate that was deliberately holding the session. The RealmJoin gate
/// blocks completion until phase 110 and carries its own bounded resolution (60 min from
/// detection, re-armed on deployment activity, hard-capped at 4 h), so on every device whose
/// first deployment outlives 30 minutes the enrollment was failed while the engine own
/// <c>completion_waiting</c> still read "waiting on: realmjoin_resolution". 75d6ae8e
/// installed 25 RealmJoin packages successfully inside the window, the last one 2 min before
/// the failure; the customer tenant hit this on 10 of 30 enrollments in one week.
/// <para>
/// The guard re-arms while the gate holds. Convergence comes from the gate itself: Resolved,
/// FirstDeploymentIncomplete and Timeout all set <c>RealmJoinFacts.Outcome</c>, which opens
/// <c>RealmJoinGateOpen</c> and stops the guard applying.
/// </para>
/// </summary>
public sealed class AdvisoryCompletionGateHoldingTests
{
private static readonly DateTime T0 = new DateTime(2026, 8, 25, 6, 10, 0, DateTimeKind.Utc);
private static DecisionSignal MakeSignal(
long ordinal,
DecisionSignalKind kind,
DateTime occurredAtUtc,
IReadOnlyDictionary<string, string>? payload = null)
{
return new DecisionSignal(
sessionSignalOrdinal: ordinal,
sessionTraceOrdinal: ordinal,
kind: kind,
kindSchemaVersion: 1,
occurredAtUtc: occurredAtUtc,
sourceOrigin: "test",
evidence: new Evidence(EvidenceKind.Synthetic, $"{kind}-{ordinal}", "test"),
payload: payload);
}
private static DecisionSignal DeadlineFired(long ordinal, DateTime occurredAtUtc, string deadlineName) =>
MakeSignal(ordinal, DecisionSignalKind.DeadlineFired, occurredAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = deadlineName });
private static ActiveDeadline? FindDeadline(DecisionState state, string name) =>
state.Deadlines.FirstOrDefault(d => d.Name == name);
private static DecisionEffect SingleTimelineEffect(DecisionStep step, string eventType) =>
step.Effects.Single(e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry
&& e.Parameters != null
&& e.Parameters.TryGetValue("eventType", out var et)
&& et == eventType);
/// <summary>
/// Replays the shape both field sessions shared: Classic user-driven, full ESP, Hello
/// policy enabled, AccountSetup entered, then a guard-blocked post-AccountSetup ESP exit
/// (Shell-Core 62407, "Exiting page normally", errorCode 0) that arms the 30-min window.
/// The ESP registry froze at "1 of 5" and the device had zero user-targeted Intune apps,
/// so no arm of ShouldTransitionToAwaitingHello could ever open.
/// </summary>
private static DecisionState SetupEspExitDeadEnd(DecisionEngine engine)
{
var state = DecisionState.CreateInitial("sess-81daa77f", "tenant-81daa77f", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0)).NewState;
state = engine.Reduce(state, MakeSignal(
5, DecisionSignalKind.EspConfigDetected, T0.AddMinutes(1),
new Dictionary<string, string>
{
[SignalPayloadKeys.SkipUserEsp] = "false",
[SignalPayloadKeys.SkipDeviceEsp] = "false",
[SignalPayloadKeys.EspAllowContinueAnyway] = "false",
})).NewState;
state = engine.Reduce(state, MakeSignal(
10, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(2),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "DeviceSetup" })).NewState;
state = engine.Reduce(state, MakeSignal(
20, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(2.5),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "AccountSetup" })).NewState;
// Guard-blocked post-AccountSetup exit — arms advisory_completion at exit + 30 min.
state = engine.Reduce(state, MakeSignal(30, DecisionSignalKind.EspExiting, T0.AddMinutes(3))).NewState;
Assert.NotNull(FindDeadline(state, DeadlineNames.AdvisoryCompletion));
Assert.Null(state.Outcome);
return state;
}
/// <summary>Desktop (DAD-validated real user), RealmJoin detected, Hello provisioned.</summary>
private static DecisionState AddDesktopRealmJoinAndHello(DecisionEngine engine, DecisionState state)
{
state = engine.Reduce(state, MakeSignal(40, DecisionSignalKind.DesktopArrived, T0.AddMinutes(3.5))).NewState;
state = engine.Reduce(state, MakeSignal(
50, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(6),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "0" })).NewState;
state = engine.Reduce(state, MakeSignal(
60, DecisionSignalKind.HelloResolved, T0.AddMinutes(14),
new Dictionary<string, string> { [SignalPayloadKeys.HelloOutcome] = "Success" })).NewState;
return state;
}
// ==================================================== the gate-holding guard ====
[Fact]
public void DeadlineFired_WhileRealmJoinGateHolds_RearmsInsteadOfFailing()
{
var engine = new DecisionEngine();
var state = SetupEspExitDeadEnd(engine);
state = AddDesktopRealmJoinAndHello(engine, state);
// Preconditions: the gate is closed and owns an armed deadline of its own.
Assert.False(DecisionEngine.RealmJoinGateOpen(state));
Assert.NotNull(FindDeadline(state, DeadlineNames.RealmJoinTimeout));
var fireAt = T0.AddMinutes(33);
var step = engine.Reduce(state, DeadlineFired(70, fireAt, DeadlineNames.AdvisoryCompletion));
Assert.True(step.Transition.Taken);
Assert.Null(step.NewState.Outcome);
Assert.NotEqual(SessionStage.Failed, step.NewState.Stage);
var rearmed = FindDeadline(step.NewState, DeadlineNames.AdvisoryCompletion);
Assert.NotNull(rearmed);
Assert.Equal(fireAt.AddMinutes(30), rearmed!.DueAtUtc);
var schedule = step.Effects.Single(e => e.Kind == DecisionEffectKind.ScheduleDeadline);
Assert.Equal(DeadlineNames.AdvisoryCompletion, schedule.Deadline!.Name);
var waiting = SingleTimelineEffect(step, "completion_waiting");
Assert.Contains("CompletionGateHolding", waiting.Parameters!["trigger"]);
Assert.Contains("realmjoin_resolution", waiting.Parameters!["missingPrerequisites"]);
Assert.Equal(rearmed.DueAtUtc.ToString("o"), waiting.Parameters!["resolutionDeadlineDueAtUtc"]);
}
[Fact]
public void Session81daa77f_RealmJoinResolvesAfterTheWindow_CompletesInsteadOfFailing()
{
// End-to-end proof against the real field shape: before this guard the session was
// stamped Failed at exit+30 min with esp_exit_without_completion_evidence, 34 min
// before the RealmJoin gate would have released it.
var engine = new DecisionEngine();
var state = SetupEspExitDeadEnd(engine);
state = AddDesktopRealmJoinAndHello(engine, state);
var fired = engine.Reduce(state, DeadlineFired(70, T0.AddMinutes(33), DeadlineNames.AdvisoryCompletion));
Assert.Null(fired.NewState.Outcome);
// RealmJoin reaches phase 110 — the gate opens and the session completes.
var resolved = engine.Reduce(fired.NewState, MakeSignal(
80, DecisionSignalKind.RealmJoinResolved, T0.AddMinutes(50),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "110" }));
Assert.Equal(SessionStage.Finalizing, resolved.NewState.Stage);
Assert.Equal("Resolved", resolved.NewState.RealmJoinFacts.Outcome!.Value);
var grace = FindDeadline(resolved.NewState, DeadlineNames.FinalizingGrace);
Assert.NotNull(grace);
var completed = engine.Reduce(resolved.NewState,
DeadlineFired(90, grace!.DueAtUtc, DeadlineNames.FinalizingGrace));
Assert.Equal(SessionStage.Completed, completed.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentComplete, completed.NewState.Outcome);
}
// ============================================================== convergence ====
[Fact]
public void DeadlineFired_AfterTheGateOpened_ResolvesNormallyAgain()
{
// Convergence proof: the guard is not an escape hatch. Once RealmJoin timed out the
// gate is open, the guard stops applying, and a session with no completion evidence
// fails exactly as before.
var engine = new DecisionEngine();
var state = SetupEspExitDeadEnd(engine);
state = engine.Reduce(state, MakeSignal(
50, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(6),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "0" })).NewState;
var rjDeadline = FindDeadline(state, DeadlineNames.RealmJoinTimeout);
Assert.NotNull(rjDeadline);
state = engine.Reduce(state,
DeadlineFired(60, rjDeadline!.DueAtUtc, DeadlineNames.RealmJoinTimeout)).NewState;
Assert.True(DecisionEngine.RealmJoinGateOpen(state));
Assert.Null(FindDeadline(state, DeadlineNames.RealmJoinTimeout));
// No desktop, no Hello, no IME evidence — the genuine dead-end shape.
var step = engine.Reduce(state,
DeadlineFired(70, T0.AddMinutes(80), DeadlineNames.AdvisoryCompletion));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentFailed, step.NewState.Outcome);
var failed = SingleTimelineEffect(step, "enrollment_failed");
Assert.Equal("esp_exit_without_completion_evidence", failed.Parameters!["reason"]);
}
// =========================================================== mutation proofs ====
[Fact]
public void DeadlineFired_WithoutRealmJoin_StillFails()
{
// The guard must key on an actually-held gate, never on the absence of evidence.
var engine = new DecisionEngine();
var state = SetupEspExitDeadEnd(engine);
var step = engine.Reduce(state,
DeadlineFired(70, T0.AddMinutes(33), DeadlineNames.AdvisoryCompletion));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentFailed, step.NewState.Outcome);
var failed = SingleTimelineEffect(step, "enrollment_failed");
Assert.Equal("esp_exit_without_completion_evidence", failed.Parameters!["reason"]);
}
[Fact]
public void DeadlineFired_GateClosedButItsDeadlineNoLongerArmed_StillFails()
{
// A gate that can no longer resolve itself must not park the session forever. The
// hand-built state drops only the RealmJoin deadline, leaving the gate closed.
var engine = new DecisionEngine();
var state = SetupEspExitDeadEnd(engine);
state = AddDesktopRealmJoinAndHello(engine, state);
var builder = state.ToBuilder();
builder.CancelDeadline(DeadlineNames.RealmJoinTimeout);
state = builder.Build();
Assert.False(DecisionEngine.RealmJoinGateOpen(state));
Assert.Null(FindDeadline(state, DeadlineNames.RealmJoinTimeout));
var step = engine.Reduce(state,
DeadlineFired(70, T0.AddMinutes(33), DeadlineNames.AdvisoryCompletion));
Assert.Equal(SessionStage.Failed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentFailed, step.NewState.Outcome);
}
}
}
@@ -372,6 +372,9 @@ namespace AutopilotMonitor.DecisionCore.Tests
// Before RealmJoinPhaseChanged became a typed signal, LastDeploymentPhase was only
// written at detection time — every realmjoin_timeout claimed "last phase: 0" no
// matter how far RJ actually got, hiding this whole failure class from ops.
// Since the activity-based extension, the PhaseChanged at T0+6 counts as activity,
// so the first fire (59 min later) extends once; the timeout lands on the re-armed
// fire a full inactivity window after the last activity.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
@@ -384,7 +387,13 @@ namespace AutopilotMonitor.DecisionCore.Tests
})).NewState;
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.DesktopArrived, T0.AddMinutes(7))).NewState;
var step = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, T0.AddMinutes(65),
// First fire at the original due (T0+65): activity 59 min ago → extends to T0+66.
state = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, T0.AddMinutes(65),
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout })).NewState;
var rearmed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Equal(T0.AddMinutes(66), rearmed.DueAtUtc);
var step = engine.Reduce(state, MakeSignal(9, DecisionSignalKind.DeadlineFired, rearmed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
var timeout = Assert.Single(step.Effects, e =>
@@ -395,6 +404,248 @@ namespace AutopilotMonitor.DecisionCore.Tests
Assert.Equal("101", timeout.Parameters["lastSeenPhase"]);
}
// ============================================================== Activity-based extension (report 55e6afd61c9d)
[Fact]
public void Timeout_fire_with_recent_first_deployment_activity_extends_instead_of_timing_out()
{
// Report 55e6afd61c9d (Douglas): timer armed at detection (phase 0, RJ agent MSI
// install during DeviceSetup), first deployment only started 16 min later, Office
// completed 3 s before the deadline — the hard cut truncated an actively working
// deployment. The fire must now re-arm to lastActivity + 60 min instead.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "0" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(21),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "0",
})).NewState;
state = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.RealmJoinPackageCompleted, T0.AddMinutes(62),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.PackageId] = "generic-microsoft-office-2016-proplus",
[DecisionEngine.RealmJoinPayloadKeys.Scope] = RealmJoinPackageFact.ScopeMachine,
[DecisionEngine.RealmJoinPayloadKeys.Success] = "true",
[DecisionEngine.RealmJoinPayloadKeys.LastExitCode] = "0",
})).NewState;
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
var step = engine.Reduce(state, MakeSignal(9, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
// No completion, no outcome — the gate stays closed and the window is re-armed.
Assert.NotEqual(SessionStage.Finalizing, step.NewState.Stage);
Assert.Null(step.NewState.RealmJoinFacts.Outcome);
Assert.Equal($"DeadlineFired:{DeadlineNames.RealmJoinTimeout}:Extended", step.Transition.Trigger);
var rearmed = Assert.Single(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Equal(T0.AddMinutes(62 + 60), rearmed.DueAtUtc); // lastActivity + inactivity window
Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.ScheduleDeadline &&
e.Deadline != null &&
e.Deadline.Name == DeadlineNames.RealmJoinTimeout &&
e.Deadline.DueAtUtc == rearmed.DueAtUtc);
var extended = Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout_extended");
Assert.Equal("Info", extended.Parameters!["severity"]);
Assert.Equal("101", extended.Parameters["deploymentPhase"]);
Assert.DoesNotContain(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
}
[Fact]
public void Resolved_after_extension_completes_via_classic_path()
{
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(40),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "100",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "101",
})).NewState;
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
state = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout })).NewState;
Assert.Null(state.RealmJoinFacts.Outcome); // extended, not timed out
var step = engine.Reduce(state, MakeSignal(9, DecisionSignalKind.RealmJoinResolved, T0.AddMinutes(80),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "110" }));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("Resolved", step.NewState.RealmJoinFacts.Outcome!.Value);
Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
}
[Fact]
public void Rearmed_fire_after_quiet_window_times_out_with_inactivity_reason()
{
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "0" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(30),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "0",
})).NewState;
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
state = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout })).NewState;
var rearmed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Equal(T0.AddMinutes(90), rearmed.DueAtUtc); // lastActivity (T0+30) + 60 min
// No further activity — the re-armed fire must now time out for real.
var step = engine.Reduce(state, MakeSignal(9, DecisionSignalKind.DeadlineFired, rearmed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("Timeout", step.NewState.RealmJoinFacts.Outcome!.Value);
var timeout = Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
Assert.Equal("inactivity", timeout.Parameters!["reason"]);
Assert.Contains("no deployment activity", timeout.Parameters["message"]);
}
[Fact]
public void Extension_is_capped_at_absolute_ceiling_and_cap_fire_times_out_despite_activity()
{
// Late activity pushes the sliding window beyond detection + 4 h — the re-arm must
// clamp to the cap, and the cap fire must time out even though activity is recent.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
// Package completion whose sliding window (T0+230+60) would exceed the cap (T0+245).
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPackageCompleted, T0.AddMinutes(230),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.PackageId] = "generic-big-suite",
[DecisionEngine.RealmJoinPayloadKeys.Scope] = RealmJoinPackageFact.ScopeMachine,
[DecisionEngine.RealmJoinPayloadKeys.Success] = "true",
[DecisionEngine.RealmJoinPayloadKeys.LastExitCode] = "0",
})).NewState;
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
state = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout })).NewState;
var rearmed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Equal(T0.AddMinutes(5).AddHours(4), rearmed.DueAtUtc); // clamped to detection + cap
var step = engine.Reduce(state, MakeSignal(9, DecisionSignalKind.DeadlineFired, rearmed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("Timeout", step.NewState.RealmJoinFacts.Outcome!.Value);
var timeout = Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
Assert.Equal("absolute_cap", timeout.Parameters!["reason"]);
}
[Fact]
public void Stale_fire_of_old_incarnation_after_rearm_is_dead_end()
{
// Race: the OLD deadline incarnation's fire was already queued when the extension
// re-armed. The armed deadline is due LATER than the stale fire — dead-end, no
// second extension evaluation, no timeout.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "0" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(30),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "0",
})).NewState;
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout })).NewState;
Assert.Null(state.RealmJoinFacts.Outcome); // extended
var step = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
Assert.False(step.Transition.Taken);
Assert.Null(step.NewState.RealmJoinFacts.Outcome);
Assert.Empty(step.Effects);
var stillArmed = Assert.Single(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Equal(T0.AddMinutes(90), stillArmed.DueAtUtc);
}
[Fact]
public void Detection_phase_alone_is_not_activity_idle_case_times_out_at_60_min_unchanged()
{
// The phase captured AT detection must not count as activity — RJ detected in a
// first-deployment phase that then never moves still times out after the original
// 60 min (the extension exists for demonstrable progress, not for standing still).
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
Assert.Null(state.RealmJoinFacts.LastActivityUtc);
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("Timeout", step.NewState.RealmJoinFacts.Outcome!.Value);
var timeout = Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
Assert.Equal("hard_timeout", timeout.Parameters!["reason"]);
}
[Fact]
public void Regular_deployment_phase_with_activity_does_not_extend()
{
// Session 6f1959c0 shape: RJ stood at 210 when the agent booted and deploys
// regular (non-first) packages afterwards. Phase 200/210 is outside the
// first-deployment window — activity there must NOT hold the session hostage.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "210" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(40),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "210",
})).NewState;
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
var step = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("Timeout", step.NewState.RealmJoinFacts.Outcome!.Value);
}
[Fact]
public void SelfDeploying_deferred_terminal_released_by_first_deployment_incomplete()
{
@@ -198,4 +198,164 @@ public class AppInstallTerminalStateAndReconcileTests
// Column absent → Merge-mode preserves a prior batch's terminal state.
Assert.False(empty.ContainsKey("TerminalState"));
}
// ── attempt-duration semantics (2026-08): the field case that motivated it ──
// Session 44862cd3, "Set TimeZone": device-ESP evaluation pass ends Skipped, the real
// install happens ~66 min later in the user-context pass and takes 23 s. The old span
// semantics reported 4002 s.
[Fact]
public void TwoPasses_SingleBatch_DurationIsLastAttempt_NotSpan()
{
var s = Aggregate(
AppEvent("app_install_started", T0),
AppEvent("app_install_completed", T0.AddMinutes(6), "Skipped"),
AppEvent("app_install_started", T0.AddMinutes(66)),
AppEvent("app_install_completed", T0.AddMinutes(66).AddSeconds(23), "Installed"))["App"].Summary;
Assert.Equal(23, s.DurationSeconds); // last attempt, not 3983 s
Assert.Equal(T0, s.StartedAt); // first-seen anchor untouched
Assert.Equal(T0.AddMinutes(66), s.LastAttemptStartedAt);
Assert.Equal(2, s.InstallPassCount);
Assert.Equal("Installed", s.TerminalState);
Assert.Equal(T0.AddMinutes(66).AddSeconds(23), s.CompletedAt);
}
[Fact]
public void WeakerTerminal_CompletedSkipped_NeverOverridesInstalled_InBatch()
{
// Re-evaluation after a real install reports Skipped (already present) — the row
// keeps describing the install attempt, incl. CompletedAt and duration.
var s = Aggregate(
AppEvent("app_install_started", T0),
AppEvent("app_install_completed", T0.AddSeconds(40), "Installed"),
AppEvent("app_install_started", T0.AddMinutes(30)),
AppEvent("app_install_completed", T0.AddMinutes(30).AddSeconds(2), "Skipped"))["App"].Summary;
Assert.Equal("Installed", s.TerminalState);
Assert.Equal(T0.AddSeconds(40), s.CompletedAt);
Assert.Equal(40, s.DurationSeconds);
}
[Fact]
public void StartInOneBatch_TerminalInNext_ReconcileAnchorsOnStoredAttempt()
{
// Pass 2 start landed in batch N (stored anchor); the terminal-only batch N+1 must
// pair against it — not against the first-seen StartedAt an hour earlier.
var existing = new TableEntity("tenant", "session_App")
{
["Status"] = "InProgress",
["StartedAt"] = new DateTimeOffset(T0),
["LastAttemptStartedAt"] = new DateTimeOffset(T0.AddMinutes(66)),
["InstallPassCount"] = 2,
};
var summary = new AppInstallSummary
{
AppName = "App", SessionId = "session", TenantId = "tenant",
Status = "Succeeded", TerminalState = "Installed",
StartedAt = T0.AddMinutes(66).AddSeconds(23), // batch init stamp (terminal event)
CompletedAt = T0.AddMinutes(66).AddSeconds(23),
};
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal(T0, summary.StartedAt); // earliest wins (window key)
Assert.Equal(T0.AddMinutes(66), summary.LastAttemptStartedAt);
Assert.Equal(23, summary.DurationSeconds); // attempt-anchored, not 3983 s
Assert.Equal(2, summary.InstallPassCount);
}
[Fact]
public void Reconcile_ReplayedOlderBatch_NeverRegressesAnchorOrDoubleCountsPasses()
{
var existing = new TableEntity("tenant", "session_App")
{
["StartedAt"] = new DateTimeOffset(T0),
["LastAttemptStartedAt"] = new DateTimeOffset(T0.AddMinutes(66)),
["InstallPassCount"] = 2,
};
// Replay of the pass-1 batch: its newest start is OLDER than the stored anchor.
var summary = new AppInstallSummary
{
AppName = "App", Status = "InProgress",
StartedAt = T0, LastAttemptStartedAt = T0, InstallPassCount = 1,
};
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal(T0.AddMinutes(66), summary.LastAttemptStartedAt); // max-fold: no regression
Assert.Equal(2, summary.InstallPassCount); // not 3 — replay not re-counted
}
[Fact]
public void Reconcile_NewerAttemptBatch_AddsPassCount()
{
var existing = new TableEntity("tenant", "session_App")
{
["StartedAt"] = new DateTimeOffset(T0),
["LastAttemptStartedAt"] = new DateTimeOffset(T0),
["InstallPassCount"] = 1,
};
var summary = new AppInstallSummary
{
AppName = "App", Status = "InProgress",
StartedAt = T0.AddMinutes(66), LastAttemptStartedAt = T0.AddMinutes(66), InstallPassCount = 1,
};
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal(T0.AddMinutes(66), summary.LastAttemptStartedAt);
Assert.Equal(2, summary.InstallPassCount);
}
[Fact]
public void Reconcile_WeakerTerminalBatch_NeverOverridesStoredInstalled()
{
// Cross-batch mirror of the in-batch guard: a batch that only saw the Skipped
// re-evaluation must not re-describe an Installed row.
var existing = new TableEntity("tenant", "session_App")
{
["Status"] = "Succeeded",
["TerminalState"] = "Installed",
["StartedAt"] = new DateTimeOffset(T0),
["CompletedAt"] = new DateTimeOffset(T0.AddSeconds(40)),
["DurationSeconds"] = 40,
};
var summary = new AppInstallSummary
{
AppName = "App", Status = "Succeeded", TerminalState = "Skipped",
StartedAt = T0.AddMinutes(30),
LastAttemptStartedAt = T0.AddMinutes(30), InstallPassCount = 1,
CompletedAt = T0.AddMinutes(30).AddSeconds(2), DurationSeconds = 2,
};
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal("Installed", summary.TerminalState);
// The guard drops the skip-pass endpoints; the Q4 path then re-adopts the STORED
// CompletedAt (the surviving install attempt's) — same value as on disk, and the
// attempt anchor being newer keeps DurationSeconds at 0, so the omitted column
// lets Merge preserve the stored 40 s.
Assert.Equal(T0.AddSeconds(40), summary.CompletedAt);
Assert.Equal(0, summary.DurationSeconds);
}
[Fact]
public void EntityBuilder_AttemptColumns_AreSentinelGated()
{
var with = new AppInstallSummary
{
AppName = "App",
LastAttemptStartedAt = T0,
InstallPassCount = 2,
};
var entity = TableStorageService.BuildAppInstallSummaryEntity(with, "rk");
Assert.Equal(new DateTimeOffset(T0), entity.GetDateTimeOffset("LastAttemptStartedAt"));
Assert.Equal(2, entity.GetInt32("InstallPassCount"));
var without = new AppInstallSummary { AppName = "App" };
var empty = TableStorageService.BuildAppInstallSummaryEntity(without, "rk");
Assert.False(empty.ContainsKey("LastAttemptStartedAt"));
Assert.False(empty.ContainsKey("InstallPassCount"));
}
}
@@ -36,7 +36,8 @@ public class AppVersionRegressionRadarTests
string appName = App,
string status = "Succeeded",
string terminalState = "Installed",
bool collision = false)
bool collision = false,
bool attemptAnchored = true)
=> new()
{
TenantId = TenantA,
@@ -47,6 +48,9 @@ public class AppVersionRegressionRadarTests
TerminalState = terminalState,
DurationSeconds = durationSeconds,
StartedAt = startedAt ?? Now.AddDays(-1),
// Post-cutover rows carry the attempt anchor; the radar's cutover gate
// excludes rows without it (attemptAnchored: false = legacy span row).
LastAttemptStartedAt = attemptAnchored ? (startedAt ?? Now.AddDays(-1)) : null,
AppIdCollision = collision,
};
@@ -164,6 +168,26 @@ public class AppVersionRegressionRadarTests
Assert.Equal(900, finding.CurrentMedianSeconds);
}
[Fact]
public void Evaluate_CutoverGate_LegacySpanRows_ExcludedOnBothSides()
{
// 2026-08 attempt-duration cutover: rows without LastAttemptStartedAt carry the old
// full-span semantics and are excluded from BOTH comparison sides.
// Current side: 10 legacy span rows would fire (lift 3.0), but only 5 are
// attempt-anchored → below MinMeasuredInstalls → silent.
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
rows.AddRange(Enumerable.Range(0, 10).Select(_ => Install("2.0", 900, Now.AddDays(-5), attemptAnchored: false)));
rows.AddRange(Enumerable.Range(0, 5).Select(_ => Install("2.0", 900, Now.AddDays(-5))));
Assert.Empty(AppVersionRegressionRadar.Evaluate(rows));
// Previous side entirely legacy → gated away → no comparison basis → silent,
// even though the raw medians (300 → 900) would scream regression.
var rows2 = Enumerable.Range(0, 10).Select(_ => Install("1.0", 300, Now.AddDays(-20), attemptAnchored: false)).ToList();
rows2.AddRange(Batch("2.0", 10, 900, Now.AddDays(-5)));
Assert.Empty(AppVersionRegressionRadar.Evaluate(rows2));
}
[Fact]
public void Evaluate_DeterministicOrdering_LiftDesc_ThenAppNameOrdinal()
{

Some files were not shown because too many files have changed in this diff Show More