Compare commits

...

5 Commits

Author SHA1 Message Date
Oliver Kieselbach 21b8e054df docs: correct portal-prefetch root cause (h2, server-side queue) + measured outcome
The HTTP/1.1 / six-connection-cap claim was wrong — it came from the
Schannel curl build that cannot speak HTTP/2. ALPN (openssl s_client)
and .NET HttpClient confirm the portal negotiates h2; the queue is
server-side in the SWA SSR runtime. Doc, index, log and the
GlobalSidebar comment now describe the real mechanism.

Adds the 2026-07-28 outcome section: Standard plan changed nothing
(stratified p50s match, Mann-Whitney p=0.81); prefetch={false} removed
the freezes (worst totalDuration 225s -> 4.9s) and also removed an
accidental keep-alive (~150ms added to cold navigations).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 16:52:19 +02:00
Oliver Kieselbach 249c20e15a feat: mark backfilled pre-agent events as history in timeline + agent summary event
Customer feedback: backfilled modern_deployment_error events (pre-agent
Autopilot event-log errors replayed at agent start) rendered as plain
Error rows with one identical emission timestamp — indistinguishable
from a live failure.

Web (EventTimeline.tsx, effective for already-stored events):
- "⟲ Backfilled" badge (slate, explanatory tooltip) on data.backfilled
- row time now shows the original event-log timeCreated with a
  "(reported …)" suffix; metadata block gains a "Recorded" line;
  raw mode gets a compact ⟲ marker
- severity deliberately NOT downgraded — unresolved pre-agent errors
  must stay visible in the Error filter

Agent (ModernDeploymentTracker):
- backfill scan buffers the capped record set, emits an Info summary
  event first (modern_deployment_log, backfillSummary: true +
  replayedCount/droppedOverCap/lookbackMinutes) announcing the block,
  then replays oldest-first so timeline sequence follows event-log
  chronology (cap semantics unchanged: newest 25 win)

No new EventType, no ConfigVersion bump. Tests: summary emission x2,
tracker subset 12/12; tsc + vitest 651/651 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 16:10:25 +02:00
Oliver Kieselbach 4ebd60fb1e feat: EnableGatherRuleDebugLog — per-tenant gather-rule debug trace (ConfigVersion 36)
Customers cannot tell why a gather rule produces nothing in the timeline
(e.g. a logparser rule with emit_mode on_change that never emits). New
per-tenant boolean EnableGatherRuleDebugLog (EnableImeMatchLog-style
plumbing end to end): when enabled, the agent writes
%ProgramData%\AutopilotMonitor\Logs\gather_rules_debug.log with one
pipe-delimited line per evaluation decision across all previously silent
"produced nothing" exits — disabled rules, interval without
intervalSeconds, first-tick-after-one-full-interval, phase-scope skips
incl. currentPhase=Unknown, per-(rule, phase) dedup, empty collector
results (emitOnlyIfExists), on_change hash suppression (hash prefix +
streak), guard blocks, logparser per-file outcomes (position skips,
format=text hint when every line fails CMTrace parsing, maxLines and
wildcard caps), and full exception stacks the agent log drops.

- Backend: ConfigVersion 35 -> 36; TenantConfiguration/AgentConfigResponse
  field + TableConfigRepository Store+Map
- Agent: GatherRuleDebugLog writer (10 MB cap, one .old generation,
  failures swallowed, optional console echo), GatherRuleContext.DebugLog
  (null-safe no-op), trace points in executor + collectors; never writes
  under _scopeLock (reasons captured inside, written after release);
  merger bool->path projection with --gather-debug-log CLI override
  tolerance; deliberately NOT in the offline-cache strip lists;
  --run-gather-rules always traces; zero-rules delivery leaves a one-line
  file instead of nothing
- Side fix: GatherRuleExecutorHost now passes the logger into
  InformationalEventPost so gather-event emissions hit the DEBUG log
- Web: tenant settings toggle (Settings -> Agent) + admin tenant-config
  report rows
- Tests: merger on/off/override, writer format+rotation, executor debug
  lines, logparser debug lines (agent 2131, backend 3561, tsc green)
- Docs: docs/agent/gather-rule-debug-log.md + index/log entries

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 15:59:30 +02:00
Oliver Kieselbach 22093d0824 feat(agent): diagnostics upload at WhiteGlove Part-1 seal (preprov, V1 parity)
Upload the diagnostics package as a pre-provisioning intermediate when
Part 1 seals (suffix "preprov"). No new config field: the existing mode
semantics carry the desired behaviour — Always uploads at the seal,
OnFailure skips (the seal terminates as Succeeded; a Part-1 failure goes
through the regular Failed terminal path, which already uploads), Off
stays off. The upload runs before the whiteglove_part1_complete emit so
diagnostics_* events keep a lower sequence than the Part-1/Part-2
boundary marker. Best-effort: a fast reseal-reboot just loses the
intermediate package. Backend needs nothing — ingest already stamps
diagnostics_uploaded independent of terminal state (proven live by the
on-demand collection path).

Also make the on-demand path honest: CreateAndUploadAsync now takes
bool? enrollmentSucceeded — null means "no outcome yet" (mid-session),
so the OnFailure gate only skips a KNOWN success and sessioninfo.txt
records "In Progress" instead of the former enrollmentSucceeded:false
workaround that stamped in-flight packages "Failed".

10 new tests (WG seal mode matrix, preprov suffix, event ordering vs the
Part-1 marker, unchanged Part-1 exit contract; nullable-outcome gate
matrix via HTTP-free sentinel, sessioninfo labels). Agent suite
2112/2112 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 15:54:14 +02:00
Oliver Kieselbach dff1532be5 feat: project ConnectionType (WiFi/Ethernet) onto SessionsIndex
The agent already reports connectionType in every network_interface_info
event, but it was only stored as a JSON blob in DeviceSnapshot — answering
"how many machines enroll over Ethernet vs WiFi?" required a slow full-table
scan via deviceProperties. Project it as a typed column instead (template:
the AvgApiLatencyMs chain), making it a cheap indexed one-call query.

- Ingest: TryExtractConnectionType — last network_interface_info emission of
  the batch wins (a device switching media mid-enrollment is relabeled);
  only exact "WiFi"/"Ethernet" accepted, fire-and-forget fail-soft writer
- Storage: UpdateSessionConnectionTypeAsync (Merge on Sessions + index
  mirror, tombstone-safe), BuildSessionIndexEntity superset entry,
  MapToSessionSummary, re-registration preserve in StoreSessionAsync
- Search: connectionType= filter with OData eq push-down on the index,
  client-side backstop on the non-OData paths, fields=connectionType
  projection, pagination fingerprint entry, raw endpoint parity
- MCP: connectionType z.enum param on search_sessions/query_raw_sessions,
  fields doc updated (server redeploy required)
- Web: optional sortable/filterable "Connection" column in the session list
- No backfill: legacy sessions lack the column and are excluded by the
  filter (intended, same semantics as RebootCount); the deviceProperties
  scan path still covers historical data

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 14:25:17 +02:00
58 changed files with 1812 additions and 83 deletions
+13
View File
@@ -119,6 +119,19 @@ normal IME deployment flow. Spam containment, in order:
payload; the event's own timestamp is the emission time, so the session's `StartedAt`
(which the backend aligns to the earliest event) is not dragged backwards.
Presentation (2026-07-28, customer feedback — an unannounced wall of pre-agent Error
events read like a live failure):
- The scan buffers the (capped) record set, first emits an **Info summary event**
(`modern_deployment_log`, payload `backfillSummary: true` + `replayedCount` /
`droppedOverCap` / `lookbackMinutes`) announcing the replay block, then replays the
records **oldest-first** so the timeline sequence follows event-log chronology.
- The web timeline (`EventTimeline.tsx`) badges `backfilled: true` events („⟲ Backfilled",
slate, explanatory tooltip) and renders the original `timeCreated` as the row time with a
"(reported …)" suffix; the expanded metadata block gains a `Recorded` line. Severity is
deliberately **not** downgraded — unresolved pre-agent errors must still surface in the
Error filter.
# Schema — Error codes (backend known-issue map)
Codes documented in the known-issues / troubleshooting-FAQ pages that appear in ESP/enrollment
+108
View File
@@ -0,0 +1,108 @@
---
type: Concept
title: Gather Rule Debug Log — the per-tenant evaluation trace for rules that produce nothing
description: EnableGatherRuleDebugLog (ConfigVersion 36) makes the agent write a local trace file explaining every gather-rule evaluation decision — scope skips, missing intervalSeconds, on_change suppression, empty collector results, logparser position/parse-mode/no-match outcomes — so customers can self-diagnose rules that never show up in the timeline.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Gather/GatherRuleDebugLog.cs
tags:
- agent
- gather-rules
- diagnostics
- tenant-configuration
timestamp: 2026-07-28T00:00:00+02:00
---
# Gather Rule Debug Log
## Schema
### The problem
A gather rule that produces no timeline events is indistinguishable from a rule that
never ran. The pipeline has many *silent-nothing* exits — before this feature, none of
them were visible to the customer:
| # | Decision point | Where |
|---|---|---|
| 1 | Rule delivered but `Enabled=false` | `GatherRuleExecutor.UpdateRules` |
| 2 | `trigger=interval` without `intervalSeconds` — never runs | `UpdateRules` timer filter |
| 3 | Interval timer's first tick comes only after one **full** interval | `UpdateRules` timer creation |
| 4 | Out of phase scope (`activePhases` / `activeFromPhase` not latched); incl. `currentPhase=Unknown` before the first phase signal | `IsRuleInScopeLocked` |
| 5 | phase_change / phase_exit dedup — once per (rule, phase) | `OnPhaseChanged` |
| 6 | Collector returned an **empty** result (e.g. `emitOnlyIfExists=true` and target absent) | `ExecuteRule` |
| 7 | `emit_mode=on_change` — result hash unchanged, emit suppressed | `ShouldEmitOnChange` |
| 8 | Guard blocked the target (emits `security_warning`, but the rule yields no data) | `GatherRuleContext.EmitSecurityWarning` |
| 9 | logparser: no files matched, position tracker says nothing new, regex no-match, plain-text file parsed in default `cmtrace` mode, `maxLines` cap, wildcard capped at 20 files | `LogParserCollector` |
| 10 | Rule execution threw (agent log had only the message, no stack) | `ExecuteRule` catch |
### The setting
`TenantConfiguration.EnableGatherRuleDebugLog` (`bool?`, default null→false), served to the
agent as `AgentConfigResponse.EnableGatherRuleDebugLog` — **ConfigVersion 36**. Portal:
Settings → Agent → Settings, toggle "Gather Rule Debug Log", stored via the usual
tenant-config roundtrip (`TableConfigRepository` Store+Map). Also shown in the admin
tenant-config report.
`RemoteConfigMerger` projects the bool onto a runtime path (same pattern as
`EnableImeMatchLog`): `true` → `Constants.GatherRuleDebugLogPath` =
`%ProgramData%\AutopilotMonitor\Logs\gather_rules_debug.log`; `false` → null, unless a
`--gather-debug-log <path>` CLI override supplied a non-default path. The field is not
security-sensitive and deliberately **survives the offline config cache**.
### The trace file
`GatherRuleDebugLog` is an append-only writer held by `GatherRuleExecutor` and exposed
to collectors via `GatherRuleContext.DebugLog(ruleId, stage, message)` (null-safe no-op
when disabled — zero cost on the normal path). Line format:
```
2026-07-28T09:15:02.123Z | RULE-ID | stage | message
```
Stages: `config` (registration summary per rule, header, zero-rules note), `trigger`
(timer scheduled, phase/event fired, dedup), `scope` (skips with reason incl. current
phase), `exec` (execution start), `collector` (null/empty results with likely cause),
`guard` (allowlist block), `emit` (event emitted, hash change), `suppress` (on_change
suppression with hash prefix + streak), `logparser` (per-file outcome: lines read,
position range, match count, parse failures, regex timeouts, mode; plus the
`format=text` hint when every line fails CMTrace parsing), `error` (full exception
**with stack trace** — the agent log keeps only the message).
Volume control: 10 MB cap with a single `.old` rotation generation (~20 MB worst case).
Every write failure is swallowed — tracing must never break gathering. The trace never
leaves the device on its own — but because it lives in the agent log folder
(`Constants.LogDirectory`), the diagnostics package (`DiagnosticsPackageService`,
`*.log` pattern, `AgentLogs/` section) picks it up automatically, so a customer-initiated
diagnostics upload hands the trace to support without manual copying.
### Semantics
* **No hot-reload** — the flag is consumed once when `DefaultComponentFactory` builds
the `GatherRuleExecutorHost` at session start (per-enrollment lifecycle, same as
`EnableImeMatchLog`). Enable it in the tenant, then the *next* enrollment writes the
trace.
* **Zero-rules case is traced too** — flag on + backend delivered no gather rules →
a one-shot standalone `config` line, so the file is never just absent.
* **`--run-gather-rules` always traces** (operator diagnostic mode; CLI override or
default path) and mirrors every line to the console under `--console`.
## Examples
A logparser rule with `emit_mode=on_change` that "delivers nothing" becomes:
```
... | LP-01 | config | registered: trigger=interval, collector=logparser, target=C:\...\app.log, unscoped, emitMode=on_change, interval=300s
... | LP-01 | trigger | interval timer scheduled every 300s (first run after one full interval — no immediate execution)
... | LP-01 | exec | executing: collector=logparser, target=C:\...\app.log
... | LP-01 | logparser | app.log: read 1000 lines from position 0->81234, matched 0, parseFailures=1000, regexTimeouts=0, mode=cmtrace
... | LP-01 | logparser | every line failed CMTrace parsing — if this is a plain-text log, set parameter format=text
```
— the answer ("plain-text log parsed in CMTrace mode") is in the file.
## Citations
* Executor + trace points: `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Gather/GatherRuleExecutor.cs`
* Writer: `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Gather/GatherRuleDebugLog.cs`
* Config projection: `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Configuration/RemoteConfigMerger.cs`
* Backend serve + ConfigVersion: `../../src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs`
* Phase scoping / emit mode background: `../rules/gather-rule-phase-scoping.md`
+2 -1
View File
@@ -20,6 +20,7 @@ https://docs.autopilotmonitor.com (separate repository).
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [Gather Rule Debug Log](agent/gather-rule-debug-log.md) - EnableGatherRuleDebugLog (ConfigVersion 36): the per-tenant local trace file that explains every gather-rule evaluation decision — scope skips, on_change suppression, empty collector results, logparser position/parse-mode outcomes — for rules that produce no timeline events.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated DM-Enterprise 2800 observations (neutral, flush-time watermark) + the ANALYZE-ESP-005 advisory gated on an actually-observed reboot.
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait is keyed to the Intune-registered "Sidecar" provider by NAME: co-management's "ConfigMgr" parks the user ESP at "Apps (Identifying)" for days even with TrackingPoliciesCreated=1 (issue #106); the always-on two-arm esp_policy_provider_stalled tripwire (provider_incomplete | sidecar_provider_missing) and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
@@ -43,7 +44,7 @@ https://docs.autopilotmonitor.com (separate repository).
# Web Portal
* [Portal Navigation Prefetch & the SWA Runtime Bottleneck](web/portal-navigation-prefetch.md) - Why every navigation Link sets `prefetch={false}`: the App Router's viewport prefetch fans out one RSC request per visible link against an HTTP/1.1 Static Web Apps runtime, and its latency tail starves the six connections that also serve the document and the route chunks. Includes where browser telemetry actually lives.
* [Portal Navigation Prefetch & the SWA Runtime Bottleneck](web/portal-navigation-prefetch.md) - Why every navigation Link sets `prefetch={false}`: the App Router's viewport prefetch fans out one RSC request per visible link and the Static Web Apps SSR runtime queues them server-side into a multi-minute tail. Records the follow-up measurement (the Standard plan changed nothing; the prefetch storm was an accidental keep-alive), where browser telemetry actually lives, and why this curl build cannot be trusted for HTTP versions.
# Security & Identity
+9 -1
View File
@@ -1,8 +1,16 @@
# Log
## 2026-07-28
* **Update**: `agent/autopilot-ztd-diagnostics.md` — presentation layer for the startup error backfill (customer feedback: an unannounced wall of pre-agent Error events with a single identical timestamp read like a live failure). Agent: the backfill scan now buffers the capped record set, emits an Info summary event first (`modern_deployment_log`, `backfillSummary: true` + `replayedCount`/`droppedOverCap`/`lookbackMinutes`), then replays oldest-first so the timeline sequence follows event-log chronology. Web: `EventTimeline.tsx` badges `backfilled: true` events („⟲ Backfilled", tooltip) and renders the original `timeCreated` as the row time with a "(reported …)" suffix plus a `Recorded` metadata line — works retroactively for already-stored events. Severity is deliberately not downgraded.
* **Creation**: Added `agent/gather-rule-debug-log.md` — the new per-tenant setting `EnableGatherRuleDebugLog` (ConfigVersion 35 → 36, portal toggle under Settings → Agent, `EnableImeMatchLog`-style bool→path projection in `RemoteConfigMerger`). When enabled, the agent writes `%ProgramData%\AutopilotMonitor\Logs\gather_rules_debug.log`: one pipe-delimited line per evaluation decision across ten previously silent "produced nothing" exits — disabled rules, `interval` without `intervalSeconds` (never runs), the first-tick-after-one-full-interval window, phase-scope skips incl. `currentPhase=Unknown`, per-(rule, phase) dedup, empty collector results (`emitOnlyIfExists`), `on_change` hash suppression (hash prefix + streak), guard blocks, logparser per-file outcomes (position skips, parse failures with a `format=text` hint when every line fails CMTrace parsing, `maxLines`/wildcard caps), and full exception stacks that the agent log drops. 10 MB cap + one `.old` generation; no hot-reload (per-enrollment lifecycle); flag survives the offline config cache; `--run-gather-rules` always traces and echoes under `--console`; zero-rules delivery leaves a one-line file instead of nothing. Side fix: `GatherRuleExecutorHost` now passes the logger into `InformationalEventPost`, so gather-event emissions appear in the agent DEBUG log.
* **Correction + Update**: `web/portal-navigation-prefetch.md`. The HTTP/1.1 claim in the 2026-07-27 entry below is **wrong** and has been struck from the document: `portal.autopilotmonitor.com` negotiates **h2**, verified by `openssl s_client -alpn h2,http/1.1` and a .NET `HttpClient` returning `HTTP/2.0`. The false reading came from the Windows/Schannel curl build in this environment, which has no HTTP/2 support and therefore always reports `1.1` — do not use it to determine HTTP versions. There is no six-connection cap in play; the queue is server-side in the SWA SSR runtime, which does not change the fix or the evidence, only the mechanism. Added an `Outcome` section with 20 h of post-change telemetry: the Free → Standard plan change (16:02:44 UTC) produced **no measurable latency gain** once stratified by the gap to the previous request (p50 Free → Standard: `<10s` 85 → 95 ms, `5-30min` 429 → 441 ms; tail rates 1.64 %/0.374 % over 2 s/10 s vs 6 and 1 events at n = 258 against 4.2 and 1.0 expected; `browserTimings` Mann-Whitney z = −0.24, p = 0.81). The freezes were removed by `prefetch={false}` (16:34:13 UTC): worst `browserTimings.totalDuration` 225,277 → 4,856 ms. The prefetch storm was also an **accidental keep-alive** — requests under 10 s apart fell from 85 % to 46 %, and on the identical never-cacheable `/sessions/{id}` route the median rose 87 → 211 ms and p90 465 → 1,480 ms, with one 11,441 ms abandoned request (`resultCode 0`) after 517 minutes idle. Synthetic bursts warm the runtime and must not be used to benchmark this (round 1 p50 288 ms vs rounds 4–25 p50 113 ms).
## 2026-07-27
* **Creation**: Added `web/portal-navigation-prefetch.md` (and a `Web Portal` section to the index) — root cause of the minute-long portal freezes. `GlobalSidebar` and `Navbar` render the whole navigation on every page, so the App Router's default viewport prefetch issued one `?_rsc=` request per visible link against `portal.autopilotmonitor.com` (~10,400 in 14 days). That origin serves the document, the route chunks and the RSC payloads over **HTTP/1.1**, so the six-connection-per-origin cap applies; when the Static Web Apps managed runtime queued the prefetches (p50 91 ms, p95 896 ms, but max 234 s / 21 samples over 30 s), the in-flight prefetches held every socket and the navigation the user wanted could not get one. Measured on the 2026-07-27 17:25 CEST incident: `/admin/presence?_rsc=` 151 s, three siblings finishing within 340 ms of each other at ~37.5 s, all `200`, while the backend API answered in 1.6–2.2 s and `browserTimings` showed `networkDuration` 16 ms against `processingDuration` 225 s. Every navigation Link now sets `prefetch={false}` — these routes are client components behind `ProtectedRoute` that fetch their own data, so the prefetched RSC payload carried no benefit. The doc also records that browser telemetry lives in a **separate** App Insights resource (`AUTOPILOT_MONITOR_WEB_APPINSIGHTS_ID`) that the MCP `query_backend_logs` tool cannot reach, which is why the freezes left no trace in the backend logs.
* **Creation**: Added `web/portal-navigation-prefetch.md` (and a `Web Portal` section to the index) — root cause of the minute-long portal freezes. `GlobalSidebar` and `Navbar` render the whole navigation on every page, so the App Router's default viewport prefetch issued one `?_rsc=` request per visible link against `portal.autopilotmonitor.com` (~10,400 in 14 days). *(The original entry attributed the freeze to HTTP/1.1 and a six-connection-per-origin cap — see the 2026-07-28 correction; the queue is server-side.)* When the Static Web Apps managed runtime queued the prefetches (p50 91 ms, p95 896 ms, but max 234 s / 21 samples over 30 s), the navigation the user wanted was stuck behind that queue. Measured on the 2026-07-27 17:25 CEST incident: `/admin/presence?_rsc=` 151 s, three siblings finishing within 340 ms of each other at ~37.5 s, all `200`, while the backend API answered in 1.6–2.2 s and `browserTimings` showed `networkDuration` 16 ms against `processingDuration` 225 s. Every navigation Link now sets `prefetch={false}` — these routes are client components behind `ProtectedRoute` that fetch their own data, so the prefetched RSC payload carried no benefit. The doc also records that browser telemetry lives in a **separate** App Insights resource (`AUTOPILOT_MONITOR_WEB_APPINSIGHTS_ID`) that the MCP `query_backend_logs` tool cannot reach, which is why the freezes left no trace in the backend logs.
* **Creation**: Added `backend/rule-regression-radar.md` — F3 regression radar (analyze rules only; gather rules wait for their per-batch stats dedup fix): per (tenant, rule) daily hit rate from `RuleStats`, firing only when the trailing 7-day window has ≥5 hit sessions and ≥20 evaluated sessions, the rate is ≥2× the prior 28-day baseline AND the Wilson 95 % intervals are disjoint (deterministic `MetricsMath` primitive with hand-computed test vectors). Suppressed for deleted rules, rules younger than baseline+window (`CreatedAt` grace), rules edited inside the window (`UpdatedAt`), empty baselines, and the `RuleRegressionRadarDisabled` app-setting kill switch. Alert episodes live as `ruleregression|{ruleId}` tracker rows (one bell + one `RuleFrequencyRegression` ops event per episode; numbers refreshed while active; re-armed under 1.5× baseline, on fire-stop, or by the 30-day tracker retention). On-fire dimension correlation over osBuild/model/agentVersion/imeVersion with the "correlated, not necessarily causal" wording contract. Surfaces: `regressions[]` in the rule-stats response (MCP inherits), the analyze-rules page's 30-day sparkline and regression badge.
+43 -11
View File
@@ -1,7 +1,7 @@
---
type: Concept
title: Portal Navigation Prefetch & the SWA Runtime Bottleneck
description: Why every navigation Link in the portal sets prefetch={false} — the App Router's viewport prefetch fires one RSC request per visible link against an HTTP/1.1 Static Web Apps runtime, and its latency tail starves the connection pool that serves the document and the route chunks.
description: Why every navigation Link in the portal sets prefetch={false} — the App Router's viewport prefetch fires one RSC request per visible link, and the Static Web Apps SSR runtime queues them server-side into a multi-minute latency tail. Also records what the follow-up measurement showed about the Standard SKU and the keep-alive side effect.
resource: src/Web/autopilot-monitor-web/components/GlobalSidebar.tsx
tags:
- web
@@ -25,21 +25,26 @@ it enters the viewport.
# Mechanism
Three properties combine into a portal-wide freeze:
Two properties combine into a portal-wide freeze:
1. **Fan-out.** The navigation is fully rendered on each page load, so a single navigation issues one
RSC request per visible link — measured at ~10,400 requests over 14 days.
2. **Shared origin over HTTP/1.1.** `portal.autopilotmonitor.com` serves the document, the JS route
chunks *and* the RSC payloads, and it negotiates HTTP/1.1 — not HTTP/2. Browsers cap concurrent
connections per origin at six.
3. **A heavy latency tail on the SSR runtime.** RSC responses are prerendered and edge-cached
2. **A heavy latency tail on the SSR runtime.** RSC responses are prerendered and edge-cached
(`x-nextjs-cache: HIT`, `s-maxage=31536000`), so the median is fast. On a cache miss or cold start
the Static Web Apps managed runtime queues them.
the Static Web Apps managed runtime queues them **server-side**, and the queue drains all at once.
When the tail hits, the in-flight prefetches hold all six connections. The document request and the
route chunks of the page the user actually wants then have no socket available, and the tab shows a
loading spinner or a blank page until the queue drains. Nothing fails — the requests eventually
return `200`, which is why the failure leaves no error trace anywhere.
The navigation the user wants is then stuck behind that queue and the tab shows a loading spinner or a
blank page until it drains. Nothing fails — the requests eventually return `200`, which is why the
failure leaves no error trace anywhere.
> **Correction (2026-07-28).** The original version of this document named a third property: that
> `portal.autopilotmonitor.com` negotiates HTTP/1.1 and the prefetches exhaust the six-connection
> browser cap. **That is wrong.** It came from `curl -w '%{http_version}'` on a Windows build of curl
> whose Schannel backend has no HTTP/2 support, so it can only ever report `1.1`. ALPN says otherwise:
> `openssl s_client -alpn h2,http/1.1` negotiates **h2** on the portal host, confirmed by a .NET
> `HttpClient` request returning `HTTP/2.0`. There is no six-connection cap and connection starvation
> was never the mechanism — the queue is server-side. Verify HTTP versions with `openssl s_client
> -alpn` or `HttpClient`, never with this curl build.
# Evidence
@@ -62,6 +67,33 @@ document immediately and the browser then waited nearly four minutes for the `lo
Fourteen-day distribution of RSC prefetches: n = 10,428, p50 = 91 ms, p95 = 896 ms, max = 234,230 ms,
38 over 10 s, 21 over 30 s.
# Outcome, measured 2026-07-28
Two changes landed within half an hour of each other on 2026-07-27: the SWA plan went Free → Standard
at 16:02:44 UTC, and `prefetch={false}` went live at 16:34:13 UTC. Twenty hours of production
telemetry separate their effects.
**The Standard plan produced no measurable latency gain.** Latency is governed almost entirely by the
gap to the previous request — the SSR runtime cools down — so raw percentiles only measure the traffic
mix. Stratified by that gap, the two eras are indistinguishable (p50, Free → Standard): `<10s`
85 → 95 ms · `10-60s` 100 → 117 ms · `1-5min` 295 → 310 ms · `5-30min` 429 → 441 ms. Tail rates match
too: Free ran 1.64 % over 2 s and 0.374 % over 10 s (n = 10,163); Standard produced 6 and 1 events at
n = 258 against 4.2 and 1.0 expected. `browserTimings.totalDuration` by Mann-Whitney: n = 436 vs 48,
z = −0.24, **p = 0.81**.
**The prefetch fix is what removed the freezes** — worst `browserTimings.totalDuration` fell from
225,277 ms to 4,856 ms, and the 30-second events stopped.
**It also removed an accidental keep-alive.** The prefetch storm kept the SSR runtime warm: 85 % of
requests used to arrive less than 10 s apart, now 46 % do. On an identical never-cacheable route
(`/sessions/{id}`) the median rose from 87 ms to 211 ms and p90 from 465 ms to 1,480 ms. After 517
minutes idle, one request took 11,441 ms and was abandoned (`resultCode 0`). The trade — minute-long
freezes gone, ~150 ms added to a cold navigation — is worth it, but it is a trade.
Do not benchmark this with a synthetic burst. Firing requests in quick succession warms the runtime
and measures the best case: in a 25-round `curl --parallel-max 6` run, round 1 came in at p50 288 ms
and rounds 4–25 at p50 113 ms.
# Diagnosing this class of problem
The backend Application Insights resource is the wrong place to look — a freeze appears there as an
@@ -285,6 +285,54 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.Equal(@"C:\Temp\custom-ime.log", agent.ImeMatchLogPath);
}
[Fact]
public void Merge_maps_enable_gather_rule_debug_log_true_to_default_path()
{
var agent = NewAgentConfig();
agent.GatherRuleDebugLogPath = null;
var remote = FullRemote();
remote.EnableGatherRuleDebugLog = true;
RemoteConfigMerger.Merge(agent, remote);
Assert.False(string.IsNullOrEmpty(agent.GatherRuleDebugLogPath));
Assert.Equal(
Environment.ExpandEnvironmentVariables(Constants.GatherRuleDebugLogPath),
agent.GatherRuleDebugLogPath);
}
[Fact]
public void Merge_maps_enable_gather_rule_debug_log_false_clears_path()
{
var agent = NewAgentConfig();
agent.GatherRuleDebugLogPath = Environment.ExpandEnvironmentVariables(Constants.GatherRuleDebugLogPath);
var remote = FullRemote();
remote.EnableGatherRuleDebugLog = false;
RemoteConfigMerger.Merge(agent, remote);
Assert.Null(agent.GatherRuleDebugLogPath);
}
[Fact]
public void Merge_preserves_custom_gather_rule_debug_log_path_when_remote_disables()
{
// A dev-time --gather-debug-log <custom> override seeds a non-default path on the
// AgentConfiguration. When the tenant remote flips EnableGatherRuleDebugLog off we
// keep the custom path rather than clobbering a deliberate dev override.
var agent = NewAgentConfig();
agent.GatherRuleDebugLogPath = @"C:\Temp\custom-gather.log";
var remote = FullRemote();
remote.EnableGatherRuleDebugLog = false;
RemoteConfigMerger.Merge(agent, remote);
Assert.Equal(@"C:\Temp\custom-gather.log", agent.GatherRuleDebugLogPath);
}
// ============================================================= Null / partial tolerance
[Fact]
@@ -0,0 +1,104 @@
using System;
using System.IO;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
{
/// <summary>
/// Writer unit tests for <see cref="GatherRuleDebugLog"/>: line format, lazy directory
/// creation, size-cap rotation, and the null-writer no-op path on the context.
/// </summary>
public sealed class GatherRuleDebugLogTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
private readonly AgentLogger _logger;
public GatherRuleDebugLogTests()
{
_logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
}
public void Dispose() => _tmp.Dispose();
private string LogPath => Path.Combine(_tmp.Path, "sub", "gather_rules_debug.log");
[Fact]
public void Write_produces_pipe_delimited_line_and_creates_directory_lazily()
{
var log = new GatherRuleDebugLog(LogPath, _logger);
Assert.False(File.Exists(LogPath)); // lazy — nothing until the first write
log.Write("RULE-1", GatherRuleDebugLog.StageScope, "skipped: out of scope");
var lines = File.ReadAllLines(LogPath);
Assert.Equal(2, lines.Length); // header + entry
Assert.Contains("| - | config | -- gather rule debug log started --", lines[0]);
var parts = lines[1].Split(new[] { " | " }, StringSplitOptions.None);
Assert.Equal(4, parts.Length);
Assert.Equal("RULE-1", parts[1]);
Assert.Equal("scope", parts[2]);
Assert.Equal("skipped: out of scope", parts[3]);
}
[Fact]
public void Write_flattens_newlines_except_for_error_stage()
{
var log = new GatherRuleDebugLog(LogPath, _logger);
log.Write("RULE-1", GatherRuleDebugLog.StageCollector, "line1\r\nline2");
log.Write("RULE-1", GatherRuleDebugLog.StageError, "boom\r\n at Frame()");
var content = File.ReadAllText(LogPath);
Assert.Contains("line1 line2", content);
Assert.Contains(" at Frame()", content); // error stage keeps the stack multi-line
}
[Fact]
public void Write_null_ruleId_renders_dash()
{
var log = new GatherRuleDebugLog(LogPath, _logger);
log.Write(null, GatherRuleDebugLog.StageConfig, "no rules delivered");
Assert.Contains(" | - | config | no rules delivered", File.ReadAllText(LogPath));
}
[Fact]
public void Write_rotates_once_when_file_exceeds_cap()
{
var log = new GatherRuleDebugLog(LogPath, _logger, maxBytes: 512);
for (int i = 0; i < 20; i++)
log.Write("RULE-1", GatherRuleDebugLog.StageExec, new string('x', 100));
var oldPath = Path.Combine(Path.GetDirectoryName(LogPath), "gather_rules_debug.old.log");
Assert.True(File.Exists(oldPath), "expected one .old rotation generation");
Assert.True(File.Exists(LogPath));
Assert.Contains("-- rotated", File.ReadAllText(LogPath) + File.ReadAllText(oldPath));
// Never more than one .old generation
Assert.Single(Directory.GetFiles(Path.GetDirectoryName(LogPath), "*.old.log"));
}
[Fact]
public void WriteStandalone_writes_single_config_line()
{
GatherRuleDebugLog.WriteStandalone(LogPath, "no gather rules delivered by backend — nothing to execute", _logger);
var lines = File.ReadAllLines(LogPath);
Assert.Contains(lines, l => l.Contains("no gather rules delivered by backend"));
}
[Fact]
public void Context_DebugLog_is_noop_without_writer()
{
var context = new GatherRuleContext(
_logger, "sess", "tenant", _ => { }, null, new LogFilePositionTracker());
context.DebugLog("RULE-1", GatherRuleDebugLog.StageScope, "should not throw, should not write");
Assert.False(File.Exists(LogPath));
}
}
}
@@ -0,0 +1,156 @@
using System;
using System.Collections.Generic;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
{
/// <summary>
/// Debug-trace coverage of <see cref="GatherRuleExecutor"/> (EnableGatherRuleDebugLog):
/// every "silently produced nothing" outcome must leave an explanatory line in the trace
/// file — registration summary, missing interval, scope skips, on_change suppression,
/// empty collector results.
/// </summary>
[Collection("SerialThreading")] // startup rules execute on the shared ThreadPool
public sealed class GatherRuleExecutorDebugLogTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
private readonly GatherRuleExecutor _executor;
private readonly string _debugLogPath;
private const string AbsentPath =
"HKLM\\SOFTWARE\\AutopilotMonitorTests\\DefinitelyAbsent_7e3a1f4b-0000-0000-0000-000000000000";
public GatherRuleExecutorDebugLogTests()
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_debugLogPath = Path.Combine(_tmp.Path, "gather_rules_debug.log");
_executor = new GatherRuleExecutor(
"sess", "tenant", _ => { }, logger,
debugLogPath: _debugLogPath)
{
UnrestrictedMode = true
};
}
public void Dispose()
{
_executor.Dispose();
_tmp.Dispose();
}
private string TraceContent() => File.Exists(_debugLogPath) ? File.ReadAllText(_debugLogPath) : string.Empty;
private static GatherRule Rule(string id, Action<GatherRule>? mutate = null)
{
var rule = new GatherRule
{
RuleId = id,
Title = id,
CollectorType = "registry",
Target = AbsentPath,
Trigger = "startup",
OutputEventType = "gather_test",
Enabled = true,
};
mutate?.Invoke(rule);
return rule;
}
[Fact]
public void UpdateRules_writes_registration_summary_per_rule()
{
_executor.UpdateRules(new List<GatherRule>
{
Rule("GATHER-REG-001", r => { r.Trigger = "interval"; r.IntervalSeconds = 3600; r.EmitMode = "on_change"; }),
Rule("GATHER-REG-002", r => r.Enabled = false),
});
_executor.WaitForStartupRules(30);
var trace = TraceContent();
Assert.Contains("GATHER-REG-001 | config | registered: trigger=interval, collector=registry", trace);
Assert.Contains("emitMode=on_change", trace);
Assert.Contains("interval=3600s", trace);
Assert.Contains("GATHER-REG-002 | config | rule disabled — will never run", trace);
Assert.Contains("interval timer scheduled every 3600s (first run after one full interval", trace);
}
[Fact]
public void UpdateRules_flags_interval_rule_without_intervalSeconds()
{
_executor.UpdateRules(new List<GatherRule>
{
Rule("GATHER-NOINT-001", r => { r.Trigger = "interval"; r.IntervalSeconds = null; }),
});
Assert.Contains("GATHER-NOINT-001 | error | trigger=interval but intervalSeconds is missing — rule will NEVER run",
TraceContent());
}
[Fact]
public void ScopedRule_skip_traces_unknown_phase_reason()
{
var rule = Rule("GATHER-SCOPE-001", r => { r.Trigger = "interval"; r.IntervalSeconds = 3600; r.ActivePhases = new List<string> { "DeviceSetup" }; });
_executor.UpdateRules(new List<GatherRule> { rule });
// Synchronous internal check mirrors the timer-tick gate; the reason must name Unknown.
string reason;
Assert.False(_executor.IsRuleInScope(rule, out reason));
Assert.Contains("currentPhase=Unknown", reason);
}
[Fact]
public void ScopedStartupRule_deferral_is_traced()
{
_executor.UpdateRules(new List<GatherRule>
{
Rule("GATHER-DEFER-001", r => r.ActivePhases = new List<string> { "DeviceSetup" }),
});
Assert.Contains("GATHER-DEFER-001 | scope | startup rule deferred — currentPhase=Unknown", TraceContent());
}
[Fact]
public void OnChange_suppression_and_reemission_are_traced()
{
var rule = Rule("GATHER-OC-001", r => { r.Trigger = "interval"; r.IntervalSeconds = 3600; r.EmitMode = "on_change"; });
_executor.UpdateRules(new List<GatherRule> { rule });
string hashPrefix;
int streak;
var first = new Dictionary<string, object> { ["exists"] = false };
Assert.True(_executor.ShouldEmitOnChange(rule, first, out hashPrefix, out streak));
Assert.Equal(0, streak);
Assert.False(string.IsNullOrEmpty(hashPrefix));
Assert.False(_executor.ShouldEmitOnChange(rule, new Dictionary<string, object> { ["exists"] = false }, out hashPrefix, out streak));
Assert.Equal(1, streak);
Assert.False(_executor.ShouldEmitOnChange(rule, new Dictionary<string, object> { ["exists"] = false }, out hashPrefix, out streak));
Assert.Equal(2, streak);
var changed = new Dictionary<string, object> { ["exists"] = true };
Assert.True(_executor.ShouldEmitOnChange(rule, changed, out hashPrefix, out streak));
Assert.Equal(2, streak); // the streak that just ended
Assert.Equal(2, changed["suppressedPolls"]);
}
[Fact]
public void EmptyCollectorResult_is_traced_via_startup_execution()
{
_executor.UpdateRules(new List<GatherRule>
{
Rule("GATHER-EMPTY-001", r => r.Parameters = new Dictionary<string, string> { ["emitOnlyIfExists"] = "true" }),
});
Assert.True(_executor.WaitForStartupRules(30));
var trace = TraceContent();
Assert.Contains("GATHER-EMPTY-001 | exec | executing: collector=registry", trace);
Assert.Contains("GATHER-EMPTY-001 | collector | registry key/value not found", trace);
Assert.Contains("GATHER-EMPTY-001 | collector | collector returned EMPTY result — nothing emitted", trace);
}
}
}
@@ -0,0 +1,121 @@
using System;
using System.Collections.Generic;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
{
/// <summary>
/// Debug-trace coverage of <see cref="LogParserCollector"/>: the classic
/// "rule delivers nothing" cases must be explained in the trace — zero matches,
/// position-tracker skips, and plain-text logs parsed in the default cmtrace mode.
/// </summary>
public sealed class LogParserCollectorDebugLogTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
private readonly List<EnrollmentEvent> _events = new List<EnrollmentEvent>();
private readonly GatherRuleContext _context;
private readonly string _debugLogPath;
// %TEMP% lives under C:\Users (hard-blocked) — parse targets sit beside the
// test assembly instead, admitted via UnrestrictedMode.
private readonly string _outsideUsers = Path.Combine(
AppContext.BaseDirectory, "gather-debug-tests-" + Guid.NewGuid().ToString("N"));
public LogParserCollectorDebugLogTests()
{
Directory.CreateDirectory(_outsideUsers);
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_debugLogPath = Path.Combine(_tmp.Path, "gather_rules_debug.log");
_context = new GatherRuleContext(
logger, "sess", "tenant",
evt => _events.Add(evt),
null,
new LogFilePositionTracker(),
new GatherRuleDebugLog(_debugLogPath, logger))
{
UnrestrictedMode = true
};
}
public void Dispose()
{
_tmp.Dispose();
try { Directory.Delete(_outsideUsers, recursive: true); } catch { /* best effort */ }
}
private string TraceContent() => File.Exists(_debugLogPath) ? File.ReadAllText(_debugLogPath) : string.Empty;
private static GatherRule Rule(string target, string pattern, string? format = null) => new GatherRule
{
RuleId = "GATHER-LP-DEBUG",
Title = "logparser debug test",
CollectorType = "logparser",
Target = target,
Parameters = format == null
? new Dictionary<string, string> { ["pattern"] = pattern }
: new Dictionary<string, string> { ["pattern"] = pattern, ["format"] = format },
Trigger = "startup",
OutputEventType = "logparser_debug_test",
};
[Fact]
public void ZeroMatches_writes_per_file_outcome_line()
{
var path = Path.Combine(_outsideUsers, "app.log");
File.WriteAllText(path, "nothing interesting\nstill nothing\n");
new LogParserCollector().Execute(Rule(path, "WILL_NEVER_MATCH_[0-9]+", format: "text"), _context);
Assert.Empty(_events);
var trace = TraceContent();
Assert.Contains("app.log: read 2 lines", trace);
Assert.Contains("matched 0", trace);
Assert.Contains("mode=text", trace);
}
[Fact]
public void SecondRun_without_new_content_traces_position_skip()
{
var path = Path.Combine(_outsideUsers, "twice.log");
File.WriteAllText(path, "one line\n");
var rule = Rule(path, "one", format: "text");
var collector = new LogParserCollector();
collector.Execute(rule, _context); // reads the file, advances the position
collector.Execute(rule, _context); // nothing new
Assert.Contains("twice.log: no new content (position", TraceContent());
}
[Fact]
public void PlainTextFile_in_default_cmtrace_mode_gets_format_hint()
{
var path = Path.Combine(_outsideUsers, "plain.log");
File.WriteAllText(path, "plain line one\nplain line two\n");
new LogParserCollector().Execute(Rule(path, "plain"), _context); // default = cmtrace mode
Assert.Empty(_events);
var trace = TraceContent();
Assert.Contains("parseFailures=2", trace);
Assert.Contains("every line failed CMTrace parsing — if this is a plain-text log, set parameter format=text", trace);
}
[Fact]
public void MissingFile_traces_no_files_matched()
{
var path = Path.Combine(_outsideUsers, "does-not-exist.log");
new LogParserCollector().Execute(Rule(path, "x", format: "text"), _context);
Assert.Contains("no files matched target", TraceContent());
}
}
}
@@ -226,6 +226,95 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
return archive.Entries.Select(e => e.FullName).ToArray();
}
private static string ZipEntryText(byte[] zipBytes, string entryName)
{
using var ms = new MemoryStream(zipBytes);
using var archive = new ZipArchive(ms, ZipArchiveMode.Read);
var entry = archive.GetEntry(entryName);
Assert.NotNull(entry);
using var reader = new StreamReader(entry!.Open(), Encoding.UTF8);
return reader.ReadToEnd();
}
// ── Nullable outcome (on-demand mid-session) — gate + sessioninfo semantics ──
/// <summary>
/// Sentinel subclass: throws before any HTTP so gate-pass-through is observable as a
/// non-null error result (a skip returns null BEFORE the archive is ever built).
/// </summary>
private sealed class ThrowOnBuildService : DiagnosticsPackageService
{
public ThrowOnBuildService(AgentConfiguration cfg, AgentLogger logger, BackendApiClient api)
: base(cfg, logger, api) { }
internal override byte[] BuildArchiveBytes(bool? enrollmentSucceeded) =>
throw new System.InvalidOperationException("gate-passed-sentinel");
}
private static DiagnosticsPackageService BuildGateProbe(AgentLogger logger, string mode)
{
var cfg = Cfg();
cfg.DiagnosticsUploadEnabled = true;
cfg.DiagnosticsUploadMode = mode;
var apiClient = new BackendApiClient(
httpClient: new System.Net.Http.HttpClient(),
baseUrl: "http://localhost",
manufacturer: string.Empty,
model: string.Empty,
serialNumber: string.Empty,
useBootstrapTokenAuth: false,
bootstrapToken: null,
agentVersion: "0.0.0",
logger: logger);
return new ThrowOnBuildService(cfg, logger, apiClient);
}
[Fact]
public async System.Threading.Tasks.Task CreateAndUploadAsync_null_outcome_passes_OnFailure_gate()
{
// On-demand mid-session (no outcome yet): the OnFailure gate must NOT skip —
// there is no known success to skip on. Reaching the archive build (sentinel
// throw → error result) proves the gate was passed; a skip would return null.
using var rig = new Rig();
var result = await BuildGateProbe(rig.Logger, "OnFailure")
.CreateAndUploadAsync(enrollmentSucceeded: null, fileNameSuffix: "server-requested");
Assert.NotNull(result);
Assert.Equal("gate-passed-sentinel", result!.ErrorCode);
}
[Fact]
public async System.Threading.Tasks.Task CreateAndUploadAsync_known_success_still_skipped_by_OnFailure_gate()
{
using var rig = new Rig();
var result = await BuildGateProbe(rig.Logger, "OnFailure")
.CreateAndUploadAsync(enrollmentSucceeded: true);
Assert.Null(result);
}
[Fact]
public async System.Threading.Tasks.Task CreateAndUploadAsync_null_outcome_still_blocked_by_mode_Off()
{
using var rig = new Rig();
var result = await BuildGateProbe(rig.Logger, "Off")
.CreateAndUploadAsync(enrollmentSucceeded: null);
Assert.Null(result);
}
[Theory]
[InlineData(true, "Succeeded")]
[InlineData(false, "Failed")]
[InlineData(null, "In Progress")]
public void BuildArchiveBytes_sessioninfo_reflects_nullable_outcome(bool? outcome, string expected)
{
using var rig = new Rig();
var bytes = rig.Build().BuildArchiveBytes(outcome);
var sessionInfo = ZipEntryText(bytes, "sessioninfo.txt");
Assert.Contains($"Enrollment Result: {expected}", sessionInfo);
}
// ── BuildBlobUploadUrl — destination-aware URL construction ─────────────────
[Theory]
@@ -112,6 +112,36 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
Assert.Equal(false, data["backfilled"]);
}
[Fact]
public void ErrorBackfillSummary_EmitsInfoLogEvent_AnnouncingTheReplayBlock()
{
_tracker.EmitErrorBackfillSummary(replayedCount: 6, droppedOverCap: 0);
var posted = _sink.Posted.Single(p => p.Kind == DecisionSignalKind.InformationalEvent);
Assert.Equal("modern_deployment_log", posted.Payload![SignalPayloadKeys.EventType]);
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(posted.TypedPayload);
Assert.Equal(true, data["backfillSummary"]);
Assert.Equal(6, data["replayedCount"]);
Assert.Equal(0, data["droppedOverCap"]);
Assert.Equal(240, data["lookbackMinutes"]);
var message = (string)posted.Payload[SignalPayloadKeys.Message];
Assert.Contains("Replaying 6 error event(s)", message);
Assert.Contains("before the agent started", message);
Assert.DoesNotContain("cap", message);
}
[Fact]
public void ErrorBackfillSummary_MentionsDroppedRecords_WhenCapWasHit()
{
_tracker.EmitErrorBackfillSummary(replayedCount: 25, droppedOverCap: 7);
var posted = _sink.Posted.Single(p => p.Kind == DecisionSignalKind.InformationalEvent);
var message = (string)posted.Payload![SignalPayloadKeys.Message];
Assert.Contains("7 older record(s) beyond the 25-record cap", message);
}
[Fact]
public void BackfilledHarmlessId_StillRunsThroughRollupSuppression()
{
@@ -511,6 +511,88 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Termination
$"network_bandwidth_estimate (@{bandwidthIdx}) must precede whiteglove_part1_complete (@{part1CompleteIdx}) so the straggler stays in Part 1.");
}
// ===================================================== WG Part-1 preprov diagnostics
// V1 parity (restored 2026-07-28): the seal exit uploads the diagnostics package as a
// pre-provisioning intermediate under the SAME mode gating as the terminal path — the
// seal counts as a success, so Always uploads, OnFailure skips (a Part-1 failure
// terminates via the regular Failed path, which uploads), Off does nothing.
[Theory]
[InlineData("Always", true, 1)]
[InlineData("OnFailure", true, 0)]
[InlineData("Off", true, 0)]
[InlineData("Always", false, 0)]
public void Handle_whiteglove_part1_diagnostics_upload_respects_mode(
string mode, bool enabled, int expected)
{
using var rig = new Rig();
rig.State = new DecisionStateBuilder(DecisionState.CreateInitial("S1", "T1")) { Stage = SessionStage.WhiteGloveSealed }.Build();
var cfg = rig.BuildConfig(diagEnabled: enabled, diagMode: mode);
rig.Build(cfg).Handle(sender: null!,
Args(EnrollmentTerminationReason.DecisionTerminalStage, EnrollmentTerminationOutcome.Succeeded, SessionStage.WhiteGloveSealed));
Assert.Equal(expected, rig.DiagnosticsUploads);
}
[Fact]
public void Handle_whiteglove_part1_diagnostics_upload_uses_preprov_suffix_and_succeeded_flag()
{
using var rig = new Rig();
rig.State = new DecisionStateBuilder(DecisionState.CreateInitial("S1", "T1")) { Stage = SessionStage.WhiteGloveSealed }.Build();
var cfg = rig.BuildConfig(diagEnabled: true, diagMode: "Always");
rig.Build(cfg).Handle(sender: null!,
Args(EnrollmentTerminationReason.DecisionTerminalStage, EnrollmentTerminationOutcome.Succeeded, SessionStage.WhiteGloveSealed));
Assert.Equal(1, rig.DiagnosticsUploads);
Assert.Equal("preprov", rig.LastDiagnosticsSuffix);
Assert.Equal(true, rig.LastDiagnosticsSucceededFlag);
}
[Fact]
public void Handle_whiteglove_part1_diagnostics_events_precede_part1_complete_marker()
{
// whiteglove_part1_complete is the authoritative Part-1/Part-2 boundary for the
// web split (computeWhiteGloveSplitSequence) — diagnostics_collecting/uploaded
// must sequence BEFORE it or they get mis-filed into the resumed Part-2 block.
using var rig = new Rig();
rig.State = new DecisionStateBuilder(DecisionState.CreateInitial("S1", "T1")) { Stage = SessionStage.WhiteGloveSealed }.Build();
rig.DiagnosticsResult = new DiagnosticsUploadResult { BlobName = "diag-preprov.zip" };
var cfg = rig.BuildConfig(diagEnabled: true, diagMode: "Always");
rig.Build(cfg).Handle(sender: null!,
Args(EnrollmentTerminationReason.DecisionTerminalStage, EnrollmentTerminationOutcome.Succeeded, SessionStage.WhiteGloveSealed));
var emitted = rig.EmittedEventTypes.ToList();
var collectingIdx = emitted.IndexOf("diagnostics_collecting");
var uploadedIdx = emitted.IndexOf("diagnostics_uploaded");
var part1CompleteIdx = emitted.IndexOf(Constants.EventTypes.WhiteGlovePart1Complete);
Assert.True(collectingIdx >= 0, "diagnostics_collecting must have been emitted.");
Assert.True(uploadedIdx >= 0, "diagnostics_uploaded must have been emitted.");
Assert.True(part1CompleteIdx >= 0, "whiteglove_part1_complete must have been emitted.");
Assert.True(uploadedIdx < part1CompleteIdx,
$"diagnostics_uploaded (@{uploadedIdx}) must precede whiteglove_part1_complete (@{part1CompleteIdx}) so the upload stays in Part 1.");
}
[Fact]
public void Handle_whiteglove_part1_still_skips_cleanup_and_marker_when_diagnostics_enabled()
{
// The preprov upload must not change the Part-1 exit contract: no self-destruct,
// no enrollment-complete.marker, whiteglove.complete still written.
using var rig = new Rig();
rig.State = new DecisionStateBuilder(DecisionState.CreateInitial("S1", "T1")) { Stage = SessionStage.WhiteGloveSealed }.Build();
var cfg = rig.BuildConfig(diagEnabled: true, diagMode: "Always");
rig.Build(cfg).Handle(sender: null!,
Args(EnrollmentTerminationReason.DecisionTerminalStage, EnrollmentTerminationOutcome.Succeeded, SessionStage.WhiteGloveSealed));
Assert.Equal(0, rig.CleanupService.Invocations);
Assert.False(File.Exists(Path.Combine(rig.StateDir, "enrollment-complete.marker")));
Assert.True(rig.SessionPersistence.IsWhiteGloveResume());
}
[Fact]
public void Handle_standalone_reboot_fires_when_reboot_enabled_and_no_self_destruct()
{
@@ -121,6 +121,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
/// </summary>
public string ImeMatchLogPath { get; set; }
/// <summary>
/// Optional path to write a gather-rule evaluation trace (for debugging).
/// If empty, no trace is written. Set from remote config (EnableGatherRuleDebugLog)
/// or the --gather-debug-log CLI override.
/// </summary>
public string GatherRuleDebugLogPath { get; set; }
/// <summary>
/// Path to directory with real IME log files for log replay.
/// When set, the agent replays these logs with time compression instead of reading live logs.
@@ -135,6 +135,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
}
}
// ---------------- GatherRuleDebugLog: same bool→path projection as ImeMatchLog
// above. `true` expands Constants.GatherRuleDebugLogPath, `false`
// nulls the runtime path unless a --gather-debug-log CLI override
// supplied a non-default path.
if (remoteConfig.EnableGatherRuleDebugLog)
{
agentConfig.GatherRuleDebugLogPath = Environment.ExpandEnvironmentVariables(Constants.GatherRuleDebugLogPath);
}
else
{
var defaultGatherDebugPath = Environment.ExpandEnvironmentVariables(Constants.GatherRuleDebugLogPath);
if (string.IsNullOrEmpty(agentConfig.GatherRuleDebugLogPath)
|| string.Equals(agentConfig.GatherRuleDebugLogPath, defaultGatherDebugPath, StringComparison.OrdinalIgnoreCase))
{
agentConfig.GatherRuleDebugLogPath = null;
}
}
// ---------------- Upload interval / batch — direct, V1 parity (no > 0 gate so a
// misconfigured tenant can in theory break uploads, but this
// surfaces the config bug instead of silently masking it).
@@ -327,6 +327,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
KeepLogFile = false,
EnableGeoLocation = true,
EnableImeMatchLog = false,
EnableGatherRuleDebugLog = false,
Collectors = CollectorConfiguration.CreateDefault(),
GatherRules = new System.Collections.Generic.List<GatherRule>(),
ImeLogPatterns = new System.Collections.Generic.List<ImeLogPattern>()
@@ -631,10 +631,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// the regular <see cref="ProcessRecord"/> pipeline with isBackfill=true, so they arrive
/// as modern_deployment_error events carrying backfilled=true plus the original
/// timeCreated; the event's own timestamp is emission time, so session StartedAt is not
/// dragged backwards. Newest-first, capped, watermark-deduped across restarts.
/// dragged backwards. The scan reads newest-first (the cap keeps the newest records),
/// but replay is buffered so an Info summary event announces the block BEFORE the errors
/// hit the timeline (customer feedback 2026-07-28: an unannounced wall of pre-agent
/// Error events reads like a live failure) and the records emit oldest-first, giving the
/// timeline sequence the original event-log chronology. Watermark-deduped across restarts.
/// </summary>
private void BackfillAutopilotErrors()
{
var buffered = new List<EventRecord>();
try
{
long watermark;
@@ -649,7 +654,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
ReverseDirection = true
};
int processed = 0, skippedWatermark = 0, droppedOverCap = 0;
int skippedWatermark = 0, droppedOverCap = 0;
long maxSeen = watermark;
using (var reader = new EventLogReader(query))
@@ -657,37 +662,48 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
EventRecord record;
while ((record = reader.ReadEvent()) != null)
{
using (record)
var recordId = record.RecordId ?? 0;
if (recordId > maxSeen) maxSeen = recordId;
if (recordId != 0 && recordId <= watermark)
{
var recordId = record.RecordId ?? 0;
if (recordId > maxSeen) maxSeen = recordId;
if (recordId != 0 && recordId <= watermark)
{
skippedWatermark++;
continue;
}
if (processed >= ErrorBackfillMaxRecords)
{
droppedOverCap++;
continue;
}
processed++;
ProcessRecord(record, "Autopilot", AutopilotChannel, isBackfill: true);
skippedWatermark++;
record.Dispose();
continue;
}
if (buffered.Count >= ErrorBackfillMaxRecords)
{
droppedOverCap++;
record.Dispose();
continue;
}
buffered.Add(record);
}
}
if (maxSeen > watermark)
AdvanceAutopilotErrorWatermark(maxSeen);
if (processed == 0 && droppedOverCap == 0)
if (buffered.Count == 0 && droppedOverCap == 0)
{
_logger.Debug($"ModernDeployment error backfill: nothing new (skippedByWatermark={skippedWatermark})");
}
else
_logger.Info($"ModernDeployment error backfill: forwarded {processed} record(s) " +
{
EmitErrorBackfillSummary(buffered.Count, droppedOverCap);
// Scan buffered newest-first — replay in reverse so the emitted sequence
// follows the original event-log chronology.
for (var i = buffered.Count - 1; i >= 0; i--)
{
ProcessRecord(buffered[i], "Autopilot", AutopilotChannel, isBackfill: true);
}
_logger.Info($"ModernDeployment error backfill: forwarded {buffered.Count} record(s) " +
$"(skippedByWatermark={skippedWatermark}, droppedOverCap={droppedOverCap})");
}
}
catch (EventLogNotFoundException)
{
@@ -701,6 +717,49 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
_logger.Warning($"ModernDeployment error backfill failed: {ex.Message}");
}
finally
{
foreach (var record in buffered)
{
try { record.Dispose(); } catch { }
}
}
}
/// <summary>
/// Announces the error-backfill block on the session timeline BEFORE the replayed
/// errors, so pre-agent Error events are readable as history rather than a live
/// failure. The web timeline additionally badges each replayed event via its
/// backfilled=true payload marker; this summary carries backfillSummary=true instead —
/// it is a live emission, not a replayed record.
/// </summary>
internal void EmitErrorBackfillSummary(int replayedCount, int droppedOverCap)
{
var message = $"[Autopilot] Replaying {replayedCount} error event(s) recorded in the Autopilot event log " +
$"before the agent started (lookback {ErrorBackfillLookbackMinutes / 60}h). " +
"These predate monitoring — if enrollment progressed afterwards, they were already resolved by a retry.";
if (droppedOverCap > 0)
{
message += $" {droppedOverCap} older record(s) beyond the {ErrorBackfillMaxRecords}-record cap were not replayed.";
}
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
EventType = Constants.EventTypes.ModernDeploymentLog,
Severity = EventSeverity.Info,
Source = "ModernDeploymentWatcher",
Phase = EnrollmentPhase.Unknown,
Message = message,
Data = new Dictionary<string, object>
{
{ "backfillSummary", true },
{ "replayedCount", replayedCount },
{ "droppedOverCap", droppedOverCap },
{ "lookbackMinutes", ErrorBackfillLookbackMinutes }
}
});
}
/// <summary>
@@ -218,15 +218,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
/// This method is non-fatal: all exceptions are caught and logged.
/// </summary>
/// <param name="enrollmentSucceeded">
/// True for a successful enrollment (affects OnFailure mode check and sessioninfo.txt content).
/// Pass true for WhiteGlove pre-provisioning (it succeeded up to this point).
/// True for a successful enrollment, false for a failed one (drives the OnFailure mode
/// check and sessioninfo.txt content). Pass true for WhiteGlove pre-provisioning (it
/// succeeded up to this point). Pass <b>null</b> when the session has no outcome yet
/// (on-demand server-requested collection mid-enrollment): the OnFailure gate does not
/// apply — there is no success to skip on — and sessioninfo.txt records "In Progress".
/// </param>
/// <param name="fileNameSuffix">
/// Optional suffix inserted before the .zip extension.
/// Example: "preprov" → AgentDiagnostics-{sessionId}-{timestamp}-preprov.zip
/// Null (default) → AgentDiagnostics-{sessionId}-{timestamp}.zip
/// </param>
public virtual async Task<DiagnosticsUploadResult> CreateAndUploadAsync(bool enrollmentSucceeded, string fileNameSuffix = null)
public virtual async Task<DiagnosticsUploadResult> CreateAndUploadAsync(bool? enrollmentSucceeded, string fileNameSuffix = null)
{
try
{
@@ -244,13 +247,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
return null;
}
if (string.Equals(mode, "OnFailure", StringComparison.OrdinalIgnoreCase) && enrollmentSucceeded)
// Null outcome (on-demand mid-session) passes through: the OnFailure gate only
// skips a KNOWN success — an in-flight session has nothing to skip on.
if (string.Equals(mode, "OnFailure", StringComparison.OrdinalIgnoreCase) && enrollmentSucceeded == true)
{
_logger.Info("Diagnostics upload skipped: enrollment succeeded and mode=OnFailure");
return null;
}
_logger.Info($"Creating diagnostics package (mode={mode}, enrollmentSucceeded={enrollmentSucceeded}{(fileNameSuffix != null ? $", suffix={fileNameSuffix}" : "")})...");
_logger.Info($"Creating diagnostics package (mode={mode}, enrollmentSucceeded={(enrollmentSucceeded.HasValue ? enrollmentSucceeded.ToString() : "n/a")}{(fileNameSuffix != null ? $", suffix={fileNameSuffix}" : "")})...");
// Build ZIP in memory
var timestamp = DateTime.UtcNow.ToString("yyyyMMddHHmmss");
@@ -343,7 +348,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
// Builds the diagnostics ZIP body in-memory. Extracted from CreateAndUploadAsync so
// tests can assert archive contents without going through the upload path.
internal virtual byte[] BuildArchiveBytes(bool enrollmentSucceeded)
internal virtual byte[] BuildArchiveBytes(bool? enrollmentSucceeded)
{
var tracker = new BudgetTracker(Budget);
using (var ms = new MemoryStream())
@@ -414,14 +419,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
}
}
private void AddSessionInfo(ZipArchive archive, bool enrollmentSucceeded)
private void AddSessionInfo(ZipArchive archive, bool? enrollmentSucceeded)
{
var sb = new StringBuilder();
sb.AppendLine($"Session ID: {_configuration.SessionId}");
sb.AppendLine($"Tenant ID: {_configuration.TenantId}");
sb.AppendLine($"Device Name: {Environment.MachineName}");
sb.AppendLine($"Timestamp: {DateTime.UtcNow:O}");
sb.AppendLine($"Enrollment Result: {(enrollmentSucceeded ? "Succeeded" : "Failed")}");
sb.AppendLine($"Enrollment Result: {(enrollmentSucceeded == null ? "In Progress" : enrollmentSucceeded.Value ? "Succeeded" : "Failed")}");
// Hardware info via existing DeviceInfoProvider (WMI)
sb.AppendLine($"Manufacturer: {DeviceInfoProvider.GetManufacturer()}");
@@ -25,14 +25,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
{
var filePath = rule.Target;
if (string.IsNullOrEmpty(filePath))
{
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageLogParser, "empty target — nothing to parse");
return null;
}
// Expand custom tokens (%LOGGED_ON_USER_PROFILE%) and standard environment variables
var userProfilePath = UserProfileResolver.ContainsUserProfileToken(filePath)
? UserProfileResolver.GetLoggedOnUserProfilePath() : null;
filePath = UserProfileResolver.ExpandCustomTokens(filePath);
if (filePath == null)
{
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageLogParser,
"target contains %LOGGED_ON_USER_PROFILE% but no user is logged on — skipped this run");
return null; // Token present but no user logged on — skip silently
}
// Apply IME log path override if set. The override comes from the local
// --ime-log-path flag only (never from remote config), so the operator
@@ -66,6 +73,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
string.IsNullOrEmpty(patternStr))
{
context.Logger.Warning($"LogParser rule {rule.RuleId} has no 'pattern' parameter");
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageError,
"missing 'pattern' parameter — rule can never match");
return null;
}
@@ -93,6 +102,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
catch (Exception ex)
{
context.Logger.Warning($"LogParser rule {rule.RuleId} has invalid regex: {ex.Message}");
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageError,
$"invalid regex pattern — rule can never match: {ex.Message}");
return null;
}
@@ -101,6 +112,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
if (resolvedPaths.Count == 0)
{
context.Logger.Debug($"LogParser rule {rule.RuleId}: no files found for: {filePath}");
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageLogParser,
$"no files matched target {filePath}" + (HasWildcard(Path.GetFileName(filePath)) ? " (wildcard)" : "") + " — nothing to parse");
return null;
}
@@ -144,7 +157,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
try
{
// Return matched files sorted by last write time (newest first), capped at 20
return Directory.GetFiles(directory, fileNamePart)
var allMatches = Directory.GetFiles(directory, fileNamePart);
if (allMatches.Length > 20)
context.DebugLog(ruleId, GatherRuleDebugLog.StageLogParser,
$"wildcard matched {allMatches.Length} files — capped to the 20 newest");
return allMatches
.OrderByDescending(f => new FileInfo(f).LastWriteTimeUtc)
.Take(20)
.ToList();
@@ -152,6 +169,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
catch (Exception ex)
{
context.Logger.Warning($"LogParser rule {ruleId}: wildcard expansion failed for {filePath}: {ex.Message}");
context.DebugLog(ruleId, GatherRuleDebugLog.StageError, $"wildcard expansion failed for {filePath}: {ex.Message}");
return new List<string>();
}
}
@@ -168,10 +186,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
// Nothing new to read
if (startPosition >= fileInfo.Length)
{
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageLogParser,
$"{Path.GetFileName(filePath)}: no new content (position {startPosition} >= length {fileInfo.Length}) — trackPosition=true, nothing re-read");
return;
}
int matchCount = 0;
int linesRead = 0;
int parseFailCount = 0;
int timeoutCount = 0;
long endPosition = startPosition;
using (var stream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
@@ -195,6 +219,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
}
catch (RegexMatchTimeoutException)
{
timeoutCount++;
continue;
}
if (!match.Success)
@@ -239,7 +264,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
// CMTrace mode: parse line as CMTrace format, match regex against message
CmTraceLogEntry entry;
if (!CmTraceLogParser.TryParseLine(line, out entry))
{
parseFailCount++;
continue;
}
Match match;
try
@@ -248,6 +276,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
}
catch (RegexMatchTimeoutException)
{
timeoutCount++;
continue;
}
if (!match.Success)
@@ -301,10 +330,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
if (matchCount > 0)
context.Logger.Debug($"LogParser rule {rule.RuleId}: {matchCount} matches from {linesRead} lines in {Path.GetFileName(filePath)}");
// Per-file outcome — always written so a "matched 0" run is visible too.
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageLogParser,
$"{Path.GetFileName(filePath)}: read {linesRead} lines from position {startPosition}->{endPosition}, " +
$"matched {matchCount}, parseFailures={parseFailCount}, regexTimeouts={timeoutCount}, mode={(isTextMode ? "text" : "cmtrace")}");
if (!isTextMode && linesRead > 0 && parseFailCount == linesRead)
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageLogParser,
"every line failed CMTrace parsing — if this is a plain-text log, set parameter format=text");
if (linesRead == maxLines)
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageLogParser,
$"stopped at maxLines={maxLines} — remaining content is deferred to the next run");
}
catch (Exception ex)
{
context.Logger.Warning($"LogParser rule {rule.RuleId} failed reading {filePath}: {ex.Message}");
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageError, $"failed reading {filePath}: {ex}");
}
}
@@ -113,7 +113,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
// Suppress the "not found" event when the rule only wants a positive signal.
if (emitOnlyIfExists)
{
context.DebugLog(rule.RuleId, GatherRuleDebugLog.StageCollector,
$"registry key/value not found: {path} — emitOnlyIfExists=true, empty result");
return new Dictionary<string, object>(); // empty → executor emits nothing
}
data["exists"] = false;
data["path"] = path;
@@ -22,13 +22,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
public LogFilePositionTracker FilePositionTracker { get; }
public bool UnrestrictedMode { get; set; }
/// <summary>
/// Optional gather-rule debug trace writer (EnableGatherRuleDebugLog). Null when
/// tracing is disabled — use <see cref="DebugLog"/> for null-safe writes.
/// </summary>
public GatherRuleDebugLog DebugWriter { get; }
public GatherRuleContext(
AgentLogger logger,
string sessionId,
string tenantId,
Action<EnrollmentEvent> onEventCollected,
string imeLogPathOverride,
LogFilePositionTracker filePositionTracker)
LogFilePositionTracker filePositionTracker,
GatherRuleDebugLog debugLog = null)
{
Logger = logger ?? throw new ArgumentNullException(nameof(logger));
SessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
@@ -36,8 +43,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
OnEventCollected = onEventCollected ?? throw new ArgumentNullException(nameof(onEventCollected));
ImeLogPathOverride = imeLogPathOverride;
FilePositionTracker = filePositionTracker ?? throw new ArgumentNullException(nameof(filePositionTracker));
DebugWriter = debugLog;
}
/// <summary>
/// Null-safe write to the gather-rule debug trace. No-op when tracing is disabled.
/// </summary>
public void DebugLog(string ruleId, string stage, string message) => DebugWriter?.Write(ruleId, stage, message);
/// <summary>
/// Emits a security_warning event and returns an empty result dictionary.
/// Called when a gather rule targets a path/query not on the allowlist.
@@ -45,6 +58,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
public Dictionary<string, object> EmitSecurityWarning(GatherRule rule, string collectorType, string target)
{
Logger.Warning($"SECURITY: {collectorType} path blocked by guard: {target} (Rule: {rule.RuleId})");
DebugLog(rule.RuleId, GatherRuleDebugLog.StageGuard,
$"{collectorType} target blocked by allowlist guard: {target} — security_warning emitted, rule produced no data");
var data = new Dictionary<string, object>
{
@@ -0,0 +1,115 @@
using System;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Logging;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
{
/// <summary>
/// Append-only trace file for gather-rule evaluation, enabled per tenant via
/// EnableGatherRuleDebugLog (or --gather-debug-log). Explains every evaluation
/// decision — including the outcomes that produce no event — so customers can
/// diagnose rules that never show up in the timeline. Every write failure is
/// swallowed: tracing must never break gathering.
/// </summary>
public sealed class GatherRuleDebugLog
{
public const string StageConfig = "config";
public const string StageTrigger = "trigger";
public const string StageScope = "scope";
public const string StageExec = "exec";
public const string StageCollector = "collector";
public const string StageGuard = "guard";
public const string StageEmit = "emit";
public const string StageSuppress = "suppress";
public const string StageLogParser = "logparser";
public const string StageError = "error";
private const long DefaultMaxBytes = 10 * 1024 * 1024;
private readonly string _path;
private readonly long _maxBytes;
private readonly Action<string> _echo;
private readonly object _lock = new object();
private bool _headerWritten;
public GatherRuleDebugLog(string path, AgentLogger logger, Action<string> echo = null, long maxBytes = DefaultMaxBytes)
{
if (string.IsNullOrEmpty(path)) throw new ArgumentNullException(nameof(path));
_path = path;
_maxBytes = maxBytes;
_echo = echo;
logger?.Info($"GatherRuleDebugLog: enabled -> {path}");
}
/// <summary>
/// Writes one trace line: <c>{utc} | {ruleId} | {stage} | {message}</c>.
/// Newlines in <paramref name="message"/> are flattened except for the error
/// stage, where exception stack traces stay multi-line.
/// </summary>
public void Write(string ruleId, string stage, string message)
{
try
{
if (stage != StageError && message != null && message.IndexOf('\n') >= 0)
message = message.Replace("\r", "").Replace('\n', ' ');
var line = $"{DateTime.UtcNow:yyyy-MM-ddTHH:mm:ss.fffZ} | {ruleId ?? "-"} | {stage} | {message}";
lock (_lock)
{
EnsureFileReadyLocked();
File.AppendAllText(_path, line + Environment.NewLine);
}
_echo?.Invoke(line);
}
catch { }
}
/// <summary>
/// One-shot helper for callers that have no executor (e.g. the flag is on but the
/// backend delivered zero gather rules) — the customer's trace file should explain
/// that too instead of silently not existing.
/// </summary>
public static void WriteStandalone(string path, string message, AgentLogger logger)
{
if (string.IsNullOrEmpty(path)) return;
try
{
new GatherRuleDebugLog(path, logger).Write(null, StageConfig, message);
}
catch { }
}
// Lazy on first write: create the directory and emit a header line. Before every
// append, rotate when the file exceeds maxBytes — keeping exactly one .old
// generation (worst case ~2x maxBytes on disk).
private void EnsureFileReadyLocked()
{
if (!_headerWritten)
{
var dir = Path.GetDirectoryName(_path);
if (!string.IsNullOrEmpty(dir))
Directory.CreateDirectory(dir);
}
var info = new FileInfo(_path);
if (info.Exists && info.Length > _maxBytes)
{
var oldPath = Path.ChangeExtension(_path, null) + ".old.log";
if (File.Exists(oldPath))
File.Delete(oldPath);
File.Move(_path, oldPath);
File.AppendAllText(_path,
$"{DateTime.UtcNow:yyyy-MM-ddTHH:mm:ss.fffZ} | - | {StageConfig} | -- rotated (previous file exceeded {_maxBytes} bytes) --" + Environment.NewLine);
}
if (!_headerWritten)
{
File.AppendAllText(_path,
$"{DateTime.UtcNow:yyyy-MM-ddTHH:mm:ss.fffZ} | - | {StageConfig} | -- gather rule debug log started --" + Environment.NewLine);
_headerWritten = true;
}
}
}
}
@@ -22,6 +22,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
public class GatherRuleExecutor : IDisposable
{
private readonly AgentLogger _logger;
private readonly GatherRuleDebugLog _debug; // null unless EnableGatherRuleDebugLog / --gather-debug-log
private readonly GatherRuleContext _context;
private readonly Dictionary<string, IGatherRuleCollector> _collectors;
@@ -61,7 +62,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
}
public GatherRuleExecutor(string sessionId, string tenantId, Action<EnrollmentEvent> onEventCollected,
AgentLogger logger, string imeLogPathOverride = null)
AgentLogger logger, string imeLogPathOverride = null, string debugLogPath = null,
Action<string> debugEcho = null)
{
if (sessionId == null) throw new ArgumentNullException(nameof(sessionId));
if (tenantId == null) throw new ArgumentNullException(nameof(tenantId));
@@ -69,9 +71,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
if (logger == null) throw new ArgumentNullException(nameof(logger));
_logger = logger;
_debug = string.IsNullOrEmpty(debugLogPath) ? null : new GatherRuleDebugLog(debugLogPath, logger, debugEcho);
var filePositionTracker = new LogFilePositionTracker();
_context = new GatherRuleContext(logger, sessionId, tenantId, onEventCollected, imeLogPathOverride, filePositionTracker);
_context = new GatherRuleContext(logger, sessionId, tenantId, onEventCollected, imeLogPathOverride, filePositionTracker, _debug);
// Register all collector strategies
var collectorList = new IGatherRuleCollector[]
@@ -96,6 +99,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
_collectors["command"] = _collectors["command_allowlisted"];
}
/// <summary>
/// Null-safe write to the gather-rule debug trace. Never call while holding
/// <see cref="_scopeLock"/> — capture the message under the lock, write after release.
/// </summary>
private void DebugLog(string ruleId, string stage, string message) => _debug?.Write(ruleId, stage, message);
/// <summary>
/// Updates the active rules and starts/stops execution accordingly
/// </summary>
@@ -111,11 +120,43 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
_activeRules = rules.Where(r => r.Enabled).ToList();
// Registration summary for the debug trace: one line per delivered rule so the
// customer can see exactly what the agent received and how it will be triggered.
if (_debug != null)
{
foreach (var rule in rules)
{
if (!rule.Enabled)
{
DebugLog(rule.RuleId, GatherRuleDebugLog.StageConfig, "rule disabled — will never run");
continue;
}
var scope = rule.ActivePhases != null && rule.ActivePhases.Count > 0
? $"activePhases=[{string.Join(",", rule.ActivePhases)}]"
: !string.IsNullOrEmpty(rule.ActiveFromPhase)
? $"activeFromPhase={rule.ActiveFromPhase}"
: "unscoped";
DebugLog(rule.RuleId, GatherRuleDebugLog.StageConfig,
$"registered: trigger={rule.Trigger}" +
(string.IsNullOrEmpty(rule.TriggerPhase) ? "" : $" triggerPhase={rule.TriggerPhase}") +
(string.IsNullOrEmpty(rule.TriggerEventType) ? "" : $" triggerEventType={rule.TriggerEventType}") +
$", collector={rule.CollectorType}, target={rule.Target}, {scope}" +
$", emitMode={(string.IsNullOrEmpty(rule.EmitMode) ? "always" : rule.EmitMode)}" +
(rule.IntervalSeconds.HasValue ? $", interval={rule.IntervalSeconds}s" : ""));
if (rule.Trigger == "interval" && !rule.IntervalSeconds.HasValue)
DebugLog(rule.RuleId, GatherRuleDebugLog.StageError,
"trigger=interval but intervalSeconds is missing — rule will NEVER run");
}
}
// Execute startup rules — track completion via CountdownEvent so callers can wait.
// Scoped startup rules that are not yet in scope are deferred: they run once from
// OnPhaseChanged when their scope activates (same _startupRulesExecuted dedup — a
// rule runs either here or there, never both). Unscoped rules behave as before.
List<GatherRule> pendingStartup;
var deferredTraces = _debug != null ? new List<KeyValuePair<string, string>>() : null;
lock (_scopeLock)
{
// A config refresh may deliver new from-phase rules after their phase was
@@ -124,7 +165,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
pendingStartup = _activeRules
.Where(r => r.Trigger == "startup" && !_startupRulesExecuted.Contains(r.RuleId))
.Where(r => IsRuleInScopeLocked(r))
.Where(r =>
{
string reason;
var inScope = IsRuleInScopeLocked(r, out reason);
if (!inScope && deferredTraces != null)
deferredTraces.Add(new KeyValuePair<string, string>(r.RuleId,
$"startup rule deferred — {reason}; runs once when its scope activates"));
return inScope;
})
.ToList();
foreach (var rule in pendingStartup)
@@ -133,6 +182,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
}
}
if (deferredTraces != null)
{
foreach (var trace in deferredTraces)
DebugLog(trace.Key, GatherRuleDebugLog.StageScope, trace.Value);
}
if (pendingStartup.Count > 0)
{
_startupRulesLatch?.Dispose();
@@ -158,8 +213,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
var timer = new Timer(
_ =>
{
if (!IsRuleInScope(intervalRule))
string skipReason;
if (!IsRuleInScope(intervalRule, out skipReason))
{
DebugLog(intervalRule.RuleId, GatherRuleDebugLog.StageScope, $"interval tick skipped: {skipReason}");
return;
}
ExecuteRule(intervalRule);
},
null,
@@ -168,6 +227,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
);
_intervalTimers[intervalRule.RuleId] = timer;
_logger.Info($" Interval rule {intervalRule.RuleId} scheduled every {intervalRule.IntervalSeconds}s");
DebugLog(intervalRule.RuleId, GatherRuleDebugLog.StageTrigger,
$"interval timer scheduled every {intervalRule.IntervalSeconds}s (first run after one full interval — no immediate execution)");
}
_logger.Info($"GatherRuleExecutor: {_activeRules.Count(r => r.Trigger == "startup")} startup, " +
@@ -187,6 +248,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
// _currentPhase moves below. A transition INTO Failed still fires the exit rules of the
// phase that failed — that snapshot is the whole point at a failure boundary.
List<GatherRule> exitRules = null;
var exitTraces = _debug != null ? new List<KeyValuePair<string, string>>() : null;
lock (_scopeLock)
{
var previousPhase = _currentPhase;
@@ -204,12 +266,22 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
// Scope gate before the dedup add — an out-of-scope pass must not consume
// the once-per-(rule, phase) slot. IsRuleInScopeLocked still sees the old phase.
if (!IsRuleInScopeLocked(rule)) continue;
string scopeReason;
if (!IsRuleInScopeLocked(rule, out scopeReason))
{
if (exitTraces != null)
exitTraces.Add(new KeyValuePair<string, string>(rule.RuleId,
$"phase_exit of {previousPhaseName} skipped: {scopeReason}"));
continue;
}
// "exit:" prefix keeps this key space disjoint from the phase_change keys.
if (!_phaseRulesExecuted.Add($"exit:{rule.RuleId}|{previousPhaseName}"))
{
_logger.Debug($"Phase-exit rule {rule.RuleId} already executed for exit of {previousPhaseName}, skipping");
if (exitTraces != null)
exitTraces.Add(new KeyValuePair<string, string>(rule.RuleId,
$"phase_exit already executed for exit of {previousPhaseName} — once per (rule, phase)"));
continue;
}
@@ -219,6 +291,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
}
}
if (exitTraces != null)
{
foreach (var trace in exitTraces)
DebugLog(trace.Key, GatherRuleDebugLog.StageScope, trace.Value);
}
if (exitRules != null)
{
foreach (var rule in exitRules)
@@ -252,6 +330,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
{
var deferredRule = rule;
_logger.Info($"Phase {phaseName} activated deferred startup rule {deferredRule.RuleId}");
DebugLog(deferredRule.RuleId, GatherRuleDebugLog.StageTrigger,
$"phase {phaseName} activated deferred startup rule");
ThreadPool.QueueUserWorkItem(_ => ExecuteRule(deferredRule));
}
@@ -262,18 +342,26 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
{
// Scope gate BEFORE the dedup add — an out-of-scope pass must not consume
// the once-per-(rule, phase) execution slot.
if (!IsRuleInScope(rule))
string scopeReason;
if (!IsRuleInScope(rule, out scopeReason))
{
DebugLog(rule.RuleId, GatherRuleDebugLog.StageScope,
$"phase_change to {phaseName} skipped: {scopeReason}");
continue;
}
// Deduplicate: only fire once per (ruleId, phase) combination
var deduplicationKey = $"{rule.RuleId}|{phaseName}";
if (!_phaseRulesExecuted.Add(deduplicationKey))
{
_logger.Debug($"Phase rule {rule.RuleId} already executed for phase {phaseName}, skipping");
DebugLog(rule.RuleId, GatherRuleDebugLog.StageTrigger,
$"phase_change already executed for phase {phaseName} — once per (rule, phase)");
continue;
}
_logger.Info($"Phase change triggered rule {rule.RuleId} (phase: {phaseName})");
DebugLog(rule.RuleId, GatherRuleDebugLog.StageTrigger, $"phase_change to {phaseName} fired");
ThreadPool.QueueUserWorkItem(_ => ExecuteRule(rule));
}
}
@@ -288,10 +376,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
{
if (string.Equals(rule.TriggerEventType, eventType, StringComparison.OrdinalIgnoreCase))
{
if (!IsRuleInScope(rule))
string scopeReason;
if (!IsRuleInScope(rule, out scopeReason))
{
DebugLog(rule.RuleId, GatherRuleDebugLog.StageScope,
$"on_event {eventType} skipped: {scopeReason}");
continue;
}
_logger.Info($"Event triggered rule {rule.RuleId} (event: {eventType})");
DebugLog(rule.RuleId, GatherRuleDebugLog.StageTrigger, $"on_event {eventType} fired");
ThreadPool.QueueUserWorkItem(_ => ExecuteRule(rule));
}
}
@@ -302,6 +396,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
try
{
_logger.Info($"Executing gather rule: {rule.RuleId} ({rule.Title})");
DebugLog(rule.RuleId, GatherRuleDebugLog.StageExec, $"executing: collector={rule.CollectorType}, target={rule.Target}");
var collectorType = rule.CollectorType?.ToLower();
@@ -309,6 +404,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
if (collectorType == null || !_collectors.TryGetValue(collectorType, out collector))
{
_logger.Warning($"Unknown collector type: {rule.CollectorType} for rule {rule.RuleId}");
DebugLog(rule.RuleId, GatherRuleDebugLog.StageError, $"unknown collector type '{rule.CollectorType}' — nothing emitted");
return;
}
@@ -316,7 +412,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
// LogParser (and potentially others) emit events directly and return null
if (result == null)
{
DebugLog(rule.RuleId, GatherRuleDebugLog.StageCollector,
collectorType == "logparser"
? "logparser emits per-match events directly — see logparser lines above for the per-file outcome"
: "collector returned null — nothing emitted");
return;
}
if (result.Count == 0)
{
DebugLog(rule.RuleId, GatherRuleDebugLog.StageCollector,
"collector returned EMPTY result — nothing emitted (typical causes: target not found with emitOnlyIfExists, or guard block — see guard line if present)");
}
if (result.Count > 0)
{
@@ -325,8 +433,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
// empty result never reaches this point, so an emitOnlyIfExists miss neither
// emits nor updates the hash — composing to "one event on appearance, then
// only on change".
if (IsOnChangeMode(rule) && !ShouldEmitOnChange(rule, result))
return;
if (IsOnChangeMode(rule))
{
string hashPrefix;
int suppressedStreak;
if (!ShouldEmitOnChange(rule, result, out hashPrefix, out suppressedStreak))
{
DebugLog(rule.RuleId, GatherRuleDebugLog.StageSuppress,
$"result unchanged (hash {hashPrefix}) — emit suppressed (on_change), suppressed streak={suppressedStreak}");
return;
}
DebugLog(rule.RuleId, GatherRuleDebugLog.StageEmit,
$"result changed (hash {hashPrefix})" + (suppressedStreak > 0 ? $" after {suppressedStreak} suppressed polls" : ""));
}
result["ruleId"] = rule.RuleId;
result["ruleTitle"] = rule.Title;
@@ -357,11 +476,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
Message = $"Gather: {rule.Title}",
Data = result
});
DebugLog(rule.RuleId, GatherRuleDebugLog.StageEmit,
$"emitted {eventType} (severity={severity}, {result.Count} fields)");
}
}
catch (Exception ex)
{
_logger.Warning($"Gather rule {rule.RuleId} failed: {ex.Message}");
// Full exception incl. stack trace only in the debug trace — the agent log stays terse.
DebugLog(rule.RuleId, GatherRuleDebugLog.StageError, $"rule execution failed: {ex}");
}
}
@@ -375,15 +498,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
// Internal so the scope matrix is testable synchronously (no timer/ThreadPool races).
internal bool IsRuleInScope(GatherRule rule)
{
string reason;
return IsRuleInScope(rule, out reason);
}
/// <summary>
/// Scope gate with a human-readable skip reason for the debug trace. The reason is
/// captured under the lock but must be WRITTEN outside it (never trace under _scopeLock).
/// </summary>
internal bool IsRuleInScope(GatherRule rule, out string reason)
{
lock (_scopeLock)
{
return IsRuleInScopeLocked(rule);
return IsRuleInScopeLocked(rule, out reason);
}
}
private bool IsRuleInScopeLocked(GatherRule rule)
{
string reason;
return IsRuleInScopeLocked(rule, out reason);
}
private bool IsRuleInScopeLocked(GatherRule rule, out string reason)
{
reason = null;
if (!HasPhaseScope(rule))
return true; // unscoped — legacy behavior
@@ -400,7 +541,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
// No phase signal yet (or scope tokens that never match, e.g. "Unknown"):
// scoped rules stay inactive.
if (_currentPhase == EnrollmentPhase.Unknown)
{
reason = "currentPhase=Unknown — scoped rules stay inactive until the first phase signal";
return false;
}
var phaseName = _currentPhase.ToString();
foreach (var phase in rule.ActivePhases)
@@ -408,10 +552,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
if (string.Equals(phase, phaseName, StringComparison.OrdinalIgnoreCase))
return true;
}
reason = $"out of scope (currentPhase={phaseName}, activePhases=[{string.Join(",", rule.ActivePhases)}])";
return false;
}
return _fromPhaseLatched.Contains(rule.RuleId);
if (_fromPhaseLatched.Contains(rule.RuleId))
return true;
reason = $"activeFromPhase={rule.ActiveFromPhase} not latched yet (currentPhase={_currentPhase})";
return false;
}
/// <summary>
@@ -457,8 +606,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
/// Internal so the state machine is testable synchronously (no timer/ThreadPool races).
/// </summary>
internal bool ShouldEmitOnChange(GatherRule rule, Dictionary<string, object> result)
{
string hashPrefix;
int suppressedStreak;
return ShouldEmitOnChange(rule, result, out hashPrefix, out suppressedStreak);
}
/// <summary>
/// Overload exposing the hash prefix and suppression streak for the debug trace:
/// on suppress, <paramref name="suppressedStreak"/> is the updated streak count;
/// on emit, it is the streak that just ended (0 when none).
/// </summary>
internal bool ShouldEmitOnChange(GatherRule rule, Dictionary<string, object> result,
out string hashPrefix, out int suppressedStreak)
{
var hash = ComputeCanonicalHash(result);
hashPrefix = hash.Length > 8 ? hash.Substring(0, 8) : hash;
suppressedStreak = 0;
bool emit;
lock (_scopeLock)
{
@@ -469,6 +633,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
_suppressed[rule.RuleId] = _suppressed.TryGetValue(rule.RuleId, out streak)
? (streak.Count + 1, streak.SinceUtc)
: (1, DateTime.UtcNow);
suppressedStreak = _suppressed[rule.RuleId].Count;
emit = false;
}
else
@@ -479,6 +644,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
{
result["suppressedPolls"] = streak.Count;
result["suppressedSinceUtc"] = streak.SinceUtc.ToString("o", CultureInfo.InvariantCulture);
suppressedStreak = streak.Count;
}
_suppressed.Remove(rule.RuleId);
emit = true;
@@ -483,7 +483,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
logger: logger,
rules: _remoteConfig.GatherRules,
imeLogPathOverride: _agentConfig.ImeLogPathOverride,
unrestrictedMode: _agentConfig.UnrestrictedMode));
unrestrictedMode: _agentConfig.UnrestrictedMode,
gatherDebugLogPath: _agentConfig.GatherRuleDebugLogPath));
}
else if (!string.IsNullOrEmpty(_agentConfig.GatherRuleDebugLogPath))
{
// Debug trace is on but the backend delivered zero gather rules — the trace
// file must say so instead of silently not existing.
Monitoring.Telemetry.Gather.GatherRuleDebugLog.WriteStandalone(
_agentConfig.GatherRuleDebugLogPath,
"no gather rules delivered by backend — nothing to execute", logger);
}
return new CollectorSurfaces(hosts, imeLogHost, espAndHelloHost, aadJoinHost);
@@ -39,7 +39,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
AgentLogger logger,
List<GatherRule> rules,
string? imeLogPathOverride,
bool unrestrictedMode = false)
bool unrestrictedMode = false,
string? gatherDebugLogPath = null)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
@@ -55,9 +56,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// session mode we wrap post.Emit so every session-mode gather event still
// flows through the InformationalEvent ingress pipe before hitting the
// telemetry spool — Rail-A semantics for ordering / replay determinism.
var post = new InformationalEventPost(ingress, clock);
var post = new InformationalEventPost(ingress, clock, logger);
_executor = new Monitoring.Telemetry.Gather.GatherRuleExecutor(
sessionId, tenantId, evt => post.Emit(evt), logger, imeLogPathOverride);
sessionId, tenantId, evt => post.Emit(evt), logger, imeLogPathOverride,
debugLogPath: gatherDebugLogPath);
_observableIngress = ingress as SignalIngress;
}
@@ -39,8 +39,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
/// <see cref="SessionStage.WhiteGloveSealed"/> (Part-1 exit, session resumes Part 2).</item>
/// <item>WhiteGlove Part-1 path: stop peripheral collectors first (so their stop-time
/// stragglers — e.g. <c>network_bandwidth_estimate</c> — sequence BEFORE the marker),
/// then emit <c>whiteglove_part1_complete</c>, drain spool, and write
/// <c>whiteglove.complete</c> marker via <see cref="SessionIdPersistence.SaveWhiteGloveComplete"/>
/// upload the diagnostics package as a pre-provisioning intermediate (suffix
/// <c>preprov</c>, same mode gating as the terminal path — the seal counts as a
/// success, so <c>OnFailure</c> skips), then emit <c>whiteglove_part1_complete</c>,
/// drain spool, and write <c>whiteglove.complete</c> marker via
/// <see cref="SessionIdPersistence.SaveWhiteGloveComplete"/>
/// so Part-2 resume is detected on the next boot.</item>
/// <item>Signal the caller-owned shutdown <see cref="ManualResetEventSlim"/>.</item>
/// </list>
@@ -241,6 +244,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
// the full Stop() later is a no-op for the hosts.
StopPeripheralCollectorsBestEffort();
// V1 parity (restored 2026-07-28) — upload the diagnostics package as a
// pre-provisioning intermediate. Between reseal and user OOBE the device
// may sit boxed for weeks, and if Part 2 never uploads (never runs, or
// OnFailure + success) the Part-1 forensics would be lost to self-destruct.
// No dedicated flag: the existing mode semantics apply — Always uploads,
// OnFailure skips (the seal is a success; a Part-1 failure terminates via
// the regular Failed path below, which uploads), Off does nothing. Must run
// BEFORE the whiteglove_part1_complete emit so diagnostics_* events keep a
// lower sequence than the Part-1/Part-2 boundary marker. Best-effort: a
// reseal-reboot racing the upload just loses the intermediate package.
RunUploadDiagnosticsWithEvents(args, suffixOverride: "preprov");
EmitEventSafe(new EnrollmentEvent
{
SessionId = _configuration.SessionId,
@@ -777,7 +792,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
StringComparison.Ordinal);
}
private void RunUploadDiagnosticsWithEvents(EnrollmentTerminatedEventArgs args)
/// <param name="args">Termination payload — outcome drives the OnFailure gate.</param>
/// <param name="suffixOverride">
/// Blob-name suffix override. Null (terminal path) derives "success"/"failure" from
/// the outcome; the WhiteGlove Part-1 seal passes "preprov" so the intermediate
/// package is distinguishable from the later Part-2 terminal upload.
/// </param>
private void RunUploadDiagnosticsWithEvents(EnrollmentTerminatedEventArgs args, string suffixOverride = null)
{
var mode = _configuration.DiagnosticsUploadMode ?? "Off";
if (!_configuration.DiagnosticsUploadEnabled || string.Equals(mode, "Off", StringComparison.OrdinalIgnoreCase))
@@ -813,7 +834,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
DiagnosticsUploadResult result = null;
try
{
var suffix = enrollmentSucceeded ? "success" : "failure";
var suffix = suffixOverride ?? (enrollmentSucceeded ? "success" : "failure");
result = _uploadDiagnosticsAsync(enrollmentSucceeded, suffix).GetAwaiter().GetResult();
}
catch (Exception ex)
@@ -221,8 +221,18 @@ namespace AutopilotMonitor.Agent.V2
},
});
// Step 4 — run the executor and wait.
using (var executor = new GatherRuleExecutor(sessionId, tenantId, emitEvent, logger, config.ImeLogPathOverride))
// Step 4 — run the executor and wait. Diagnostic mode always writes the
// gather-rule debug trace (CLI override or default path) and mirrors it
// to the console in --console mode.
var gatherDebugLogPath = !string.IsNullOrEmpty(config.GatherRuleDebugLogPath)
? config.GatherRuleDebugLogPath
: Environment.ExpandEnvironmentVariables(Constants.GatherRuleDebugLogPath);
logger.Info($"--run-gather-rules: debug trace -> {gatherDebugLogPath}");
if (consoleMode) Console.WriteLine($"Debug trace: {gatherDebugLogPath}");
using (var executor = new GatherRuleExecutor(sessionId, tenantId, emitEvent, logger, config.ImeLogPathOverride,
debugLogPath: gatherDebugLogPath,
debugEcho: consoleMode ? (Action<string>)(line => Console.WriteLine($" {line}")) : null))
{
// Diagnostic mode has no enrollment phase context — scoped rules
// (activePhases / activeFromPhase) execute unconditionally here.
@@ -303,6 +303,7 @@ namespace AutopilotMonitor.Agent.V2
var apiBaseUrl = GetArgValue(args, "--api-url", "--backend-api") ?? Constants.ApiBaseUrl;
var imeLogPathOverride = GetArgValue(args, "--ime-log-path");
var imeMatchLogPath = GetArgValue(args, "--ime-match-log");
var gatherDebugLogPath = GetArgValue(args, "--gather-debug-log");
// Dev / test — IME log replay (V1 compat mode). Feeds ImeLogTracker.SimulationMode
// + SpeedFactor so recorded raw IME logs are replayed at an accelerated rate; signal
@@ -373,6 +374,7 @@ namespace AutopilotMonitor.Agent.V2
EnableGeoLocation = !disableGeoLocation,
ImeLogPathOverride = imeLogPathOverride,
ImeMatchLogPath = imeMatchLogPath,
GatherRuleDebugLogPath = gatherDebugLogPath,
KeepLogFile = keepLogFile,
SelfDestructOnComplete = !noCleanup,
CommandLineArgs = FormatArgsForLog(args),
@@ -485,6 +487,7 @@ namespace AutopilotMonitor.Agent.V2
Console.Out.WriteLine(" --api-url <URL> Override backend API base URL (alias: --backend-api)");
Console.Out.WriteLine(" --ime-log-path <PATH> Override IME logs directory");
Console.Out.WriteLine(" --ime-match-log <PATH> Write matched IME log lines to file (debug)");
Console.Out.WriteLine(" --gather-debug-log <PATH> Write gather-rule evaluation trace to file (debug)");
Console.Out.WriteLine();
Console.Out.WriteLine("Dev / Test:");
Console.Out.WriteLine(" --replay-log-dir <PATH> Replay IME logs from this directory (simulation mode)");
@@ -46,8 +46,12 @@ namespace AutopilotMonitor.Agent.V2.Runtime
try { var _ = await remoteConfigService.FetchConfigAsync(); return true; }
catch (Exception ex) { logger.Warning($"ServerAction rotate_config failed: {ex.Message}"); return false; }
},
// On-demand request mid-enrollment — the session has no outcome yet. Null keeps
// sessioninfo.txt honest ("In Progress") and passes the OnFailure gate by
// semantics (there is no success to skip on) instead of the former
// enrollmentSucceeded:false workaround that stamped in-flight packages "Failed".
uploadDiagnosticsAsync: async (suffix) =>
await diagnosticsService.CreateAndUploadAsync(enrollmentSucceeded: false, fileNameSuffix: suffix),
await diagnosticsService.CreateAndUploadAsync(enrollmentSucceeded: null, fileNameSuffix: suffix),
onTerminateRequested: action => OnTerminateRequested(
action, agentConfig, orchestrator, terminationHandler, shutdown, shutdownComplete, logger),
post: post);
@@ -0,0 +1,48 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Cover for the ConnectionType predicate used on every search path that does NOT push the
/// filter to OData — the deviceProperties batch-get (<c>ApplyBasicFilters</c>) and the legacy
/// unpaged scan. Sessions that predate the projection (null ConnectionType) must never match
/// a set filter; an unset filter matches everything.
/// </summary>
public class ConnectionTypeFilterTests
{
private static SessionSummary Session(string? connectionType) =>
new() { SessionId = "s", TenantId = "t", ConnectionType = connectionType };
[Theory]
[InlineData(null)]
[InlineData("WiFi")]
[InlineData("Ethernet")]
public void NoFilter_AlwaysMatches(string? connectionType)
{
var filter = new SessionSearchFilter();
Assert.True(TableStorageService.MatchesConnectionType(Session(connectionType), filter));
}
[Theory]
[InlineData("Ethernet", true)]
[InlineData("ethernet", true)] // case-insensitive, matching the other string filters
[InlineData("WiFi", false)]
[InlineData(null, false)] // legacy session without the projected column
public void EthernetFilter_MatchesOnlyEthernet(string? connectionType, bool expected)
{
var filter = new SessionSearchFilter { ConnectionType = "Ethernet" };
Assert.Equal(expected, TableStorageService.MatchesConnectionType(Session(connectionType), filter));
}
[Theory]
[InlineData("WiFi", true)]
[InlineData("Ethernet", false)]
[InlineData(null, false)]
public void WiFiFilter_MatchesOnlyWiFi(string? connectionType, bool expected)
{
var filter = new SessionSearchFilter { ConnectionType = "WiFi" };
Assert.Equal(expected, TableStorageService.MatchesConnectionType(Session(connectionType), filter));
}
}
@@ -0,0 +1,124 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the per-session connection-type projection (2026-07-28): the agent stamps
/// <c>connectionType</c> ("WiFi"/"Ethernet") into every <c>network_interface_info</c>
/// emission (initial collect + re-collect on change); ingest takes the LAST matching
/// event of the batch and persists it via <c>UpdateSessionConnectionTypeAsync</c>
/// (last-write-wins → a device that switches media mid-enrollment is relabeled).
/// The no-NIC payload ({"status":"no_active_interface"}) carries no connectionType
/// and must never match.
/// </summary>
public class IngestConnectionTypeProjectionTests
{
private static EnrollmentEvent NicInfo(string? connectionType)
{
var evt = new EnrollmentEvent
{
EventType = "network_interface_info",
Data = new Dictionary<string, object>
{
["adapterName"] = "Ethernet 2",
["linkSpeedMbps"] = 1000L,
},
};
if (connectionType != null) evt.Data["connectionType"] = connectionType;
return evt;
}
[Fact]
public void EthernetEmission_IsExtracted()
{
var events = new List<EnrollmentEvent> { NicInfo("Ethernet") };
Assert.True(EventIngestProcessor.TryExtractConnectionType(events, out var type));
Assert.Equal("Ethernet", type);
}
[Fact]
public void LastEmissionOfBatch_Wins()
{
// Media switch mid-batch (dock removed → WiFi) — the most recent state wins.
var events = new List<EnrollmentEvent>
{
NicInfo("Ethernet"),
new EnrollmentEvent { EventType = "download_progress" },
NicInfo("WiFi"),
};
Assert.True(EventIngestProcessor.TryExtractConnectionType(events, out var type));
Assert.Equal("WiFi", type);
}
[Fact]
public void NoActiveInterfacePayload_DoesNotMatch()
{
// The no-NIC emission carries only {"status":"no_active_interface"}.
var events = new List<EnrollmentEvent>
{
new EnrollmentEvent
{
EventType = "network_interface_info",
Data = new Dictionary<string, object> { ["status"] = "no_active_interface" },
},
};
Assert.False(EventIngestProcessor.TryExtractConnectionType(events, out _));
}
[Fact]
public void NoActiveInterfaceAfterValidEmission_DoesNotShadowIt()
{
// The predicate selects the last event that CARRIES connectionType.
var events = new List<EnrollmentEvent>
{
NicInfo("WiFi"),
new EnrollmentEvent
{
EventType = "network_interface_info",
Data = new Dictionary<string, object> { ["status"] = "no_active_interface" },
},
};
Assert.True(EventIngestProcessor.TryExtractConnectionType(events, out var type));
Assert.Equal("WiFi", type);
}
[Theory]
[InlineData("Cellular")]
[InlineData("wifi")] // agent emits exact casing — anything else is unexpected
[InlineData("")]
public void UnexpectedValues_AreDroppedDefensively(string raw)
{
var events = new List<EnrollmentEvent> { NicInfo(raw) };
Assert.False(EventIngestProcessor.TryExtractConnectionType(events, out _));
}
[Fact]
public void BatchWithoutNicEvents_DoesNotMatch()
{
var events = new List<EnrollmentEvent>
{
new EnrollmentEvent { EventType = "esp_phase_changed" },
new EnrollmentEvent { EventType = "agent_metrics_snapshot" },
};
Assert.False(EventIngestProcessor.TryExtractConnectionType(events, out _));
}
[Fact]
public void NullData_DoesNotMatch()
{
var events = new List<EnrollmentEvent>
{
new EnrollmentEvent { EventType = "network_interface_info", Data = null! },
};
Assert.False(EventIngestProcessor.TryExtractConnectionType(events, out _));
}
}
@@ -45,6 +45,12 @@ public class SessionIndexProjectionTests
["IsSelfDeployingProfile"] = true,
// Device-validation path recorded at registration — shown on the session detail page.
["ValidatedBy"] = "AutopilotV1",
// Agent→backend HTTP latency projection (UpdateSessionNetworkLatencyAsync merges these).
["AvgApiLatencyMs"] = 513.5,
["ApiRequestCount"] = 100,
// Connection-type projection (UpdateSessionConnectionTypeAsync merges this) —
// search-filterable (connectionType= pushes an OData eq on the index).
["ConnectionType"] = "Ethernet",
};
var idx = TableStorageService.BuildSessionIndexEntity(session, IndexRowKey, StartedAt);
@@ -67,6 +73,9 @@ public class SessionIndexProjectionTests
Assert.True(idx.GetBoolean("ExcessiveEventsAutoActioned"));
Assert.True(idx.GetBoolean("IsSelfDeployingProfile"));
Assert.Equal("AutopilotV1", idx.GetString("ValidatedBy"));
Assert.Equal(513.5, idx.GetDouble("AvgApiLatencyMs"));
Assert.Equal(100, idx.GetInt32("ApiRequestCount"));
Assert.Equal("Ethernet", idx.GetString("ConnectionType"));
}
[Fact]
@@ -86,6 +95,8 @@ public class SessionIndexProjectionTests
Assert.False(idx.ContainsKey("FailureSource"));
Assert.False(idx.ContainsKey("StalledAt"));
Assert.False(idx.ContainsKey("ValidatedBy"));
Assert.False(idx.ContainsKey("AvgApiLatencyMs"));
Assert.False(idx.ContainsKey("ConnectionType"));
// Always-present counts/flags mirror the Sessions defaults.
Assert.Equal(0, idx.GetInt32("PlatformScriptCount"));
Assert.Equal(0, idx.GetInt32("RemediationScriptCount"));
@@ -40,6 +40,7 @@ public class StoreSessionReregistrationPreserveTests
["PlatformScriptCount"] = 3,
["RemediationScriptCount"] = 5,
["RebootCount"] = 7,
["ConnectionType"] = "Ethernet",
};
existing.ETag = new ETag("0xEXISTING");
@@ -71,6 +72,9 @@ public class StoreSessionReregistrationPreserveTests
Assert.Equal(7, harness.Written.GetInt32("RebootCount"));
// EventCount preservation is the established sibling behaviour — assert it as a sanity anchor.
Assert.Equal(42, harness.Written.GetInt32("EventCount"));
// Merge-mode field (UpdateSessionConnectionTypeAsync) — must survive the Replace so the
// value isn't lost in the window until the restarted agent re-emits network_interface_info.
Assert.Equal("Ethernet", harness.Written.GetString("ConnectionType"));
}
[Fact]
@@ -413,6 +413,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
{ "EnableTimezoneAutoSet", config.EnableTimezoneAutoSet },
{ "NtpServer", config.NtpServer },
{ "EnableImeMatchLog", config.EnableImeMatchLog },
{ "EnableGatherRuleDebugLog", config.EnableGatherRuleDebugLog },
{ "LogLevel", config.LogLevel },
{ "MaxBatchSize", config.MaxBatchSize },
{ "DiagnosticsBlobSasUrl", config.DiagnosticsBlobSasUrl },
@@ -514,6 +515,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
EnableTimezoneAutoSet = entity.GetBoolean("EnableTimezoneAutoSet"),
NtpServer = string.IsNullOrWhiteSpace(entity.GetString("NtpServer")) ? "time.windows.com" : entity.GetString("NtpServer"),
EnableImeMatchLog = entity.GetBoolean("EnableImeMatchLog"),
EnableGatherRuleDebugLog = entity.GetBoolean("EnableGatherRuleDebugLog"),
LogLevel = entity.GetString("LogLevel"),
MaxBatchSize = entity.GetInt32("MaxBatchSize"),
DiagnosticsBlobSasUrl = entity.GetString("DiagnosticsBlobSasUrl"),
@@ -152,6 +152,10 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
double avgLatencyMs, int requestCount)
=> _storage.UpdateSessionNetworkLatencyAsync(tenantId, sessionId, avgLatencyMs, requestCount);
public Task UpdateSessionConnectionTypeAsync(string tenantId, string sessionId,
string connectionType)
=> _storage.UpdateSessionConnectionTypeAsync(tenantId, sessionId, connectionType);
public Task<List<SessionSummary>> GetSessionsWithEventCountAboveAsync(string tenantId, int threshold)
=> _storage.GetSessionsWithEventCountAboveAsync(tenantId, threshold);
@@ -270,12 +270,13 @@ namespace AutopilotMonitor.Functions.Functions.Config
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new AgentConfigResponse
{
ConfigVersion = 35, // MDM reboot-policy watcher (mdm_policy_reboot_required / esp_reboot_coalescing)
ConfigVersion = 36, // EnableGatherRuleDebugLog (local gather-rule evaluation trace)
UploadIntervalSeconds = 10,
SelfDestructOnComplete = tenantConfig.SelfDestructOnComplete ?? true,
KeepLogFile = tenantConfig.KeepLogFile ?? false,
EnableGeoLocation = tenantConfig.EnableGeoLocation ?? true,
EnableImeMatchLog = tenantConfig.EnableImeMatchLog ?? false,
EnableGatherRuleDebugLog = tenantConfig.EnableGatherRuleDebugLog ?? false,
MaxAuthFailures = tenantConfig.MaxAuthFailures ?? 5,
AuthFailureTimeoutMinutes = tenantConfig.AuthFailureTimeoutMinutes ?? 0,
LogLevel = tenantConfig.LogLevel ?? "Info",
@@ -130,6 +130,8 @@ namespace AutopilotMonitor.Functions.Functions.Raw
filter.IsSelfDeployingProfile = sdp;
if (int.TryParse(query["rebootCountMin"], out var rcMin)) filter.RebootCountMin = rcMin;
if (int.TryParse(query["rebootCountMax"], out var rcMax)) filter.RebootCountMax = rcMax;
var connectionType = query["connectionType"];
if (!string.IsNullOrEmpty(connectionType)) filter.ConnectionType = connectionType;
var callerTenantId = TenantHelper.GetTenantId(req);
@@ -140,6 +140,7 @@ public class SearchSessionsFunction
ImeAgentVersion = query["imeAgentVersion"],
AgentVersionPrefix = query["agentVersionPrefix"],
ImeAgentVersionPrefix = query["imeAgentVersionPrefix"],
ConnectionType = query["connectionType"],
// Limit field is a no-op in the paged path — pagination drives count.
};
@@ -212,6 +213,7 @@ public class SearchSessionsFunction
if (fields.Contains("rebootCount")) dict["rebootCount"] = s.RebootCount;
if (fields.Contains("avgApiLatencyMs")) dict["avgApiLatencyMs"] = s.AvgApiLatencyMs;
if (fields.Contains("apiRequestCount")) dict["apiRequestCount"] = s.ApiRequestCount;
if (fields.Contains("connectionType")) dict["connectionType"] = s.ConnectionType;
if (dict.Count == 0)
{
@@ -53,6 +53,7 @@ namespace AutopilotMonitor.Functions.Pagination
new KeyValuePair<string, string?>("startedBefore", filter.StartedBefore?.ToUniversalTime().ToString("o", CultureInfo.InvariantCulture)),
new KeyValuePair<string, string?>("rebootCountMin", filter.RebootCountMin?.ToString(CultureInfo.InvariantCulture)),
new KeyValuePair<string, string?>("rebootCountMax", filter.RebootCountMax?.ToString(CultureInfo.InvariantCulture)),
new KeyValuePair<string, string?>("connectionType", filter.ConnectionType),
};
if (filter.DeviceProperties != null)
{
@@ -151,6 +151,14 @@ namespace AutopilotMonitor.Functions.Services
"Fire-and-forget UpdateSessionNetworkLatencyAsync failed"), TaskContinuationOptions.OnlyOnFaulted);
}
if (TryExtractConnectionType(request.Events, out var connectionType))
{
_ = _sessionRepo.UpdateSessionConnectionTypeAsync(
request.TenantId, request.SessionId, connectionType)
.ContinueWith(t => _logger.LogWarning(t.Exception?.InnerException,
"Fire-and-forget UpdateSessionConnectionTypeAsync failed"), TaskContinuationOptions.OnlyOnFaulted);
}
var classification = ClassifyEvents(storedEvents);
foreach (var summary in classification.AppInstallUpdates.Values)
@@ -432,6 +440,28 @@ namespace AutopilotMonitor.Functions.Services
return true;
}
/// <summary>
/// Connection-type projection: the agent stamps connectionType ("WiFi"/"Ethernet")
/// into every network_interface_info emission (initial collect + re-collect), so the
/// LAST event of the batch carries the current media and a plain overwrite is
/// idempotent against replays. The no-NIC payload ({"status":"no_active_interface"})
/// carries no connectionType and never matches. Values outside WiFi/Ethernet are
/// dropped defensively. Exposed as internal for unit testing.
/// </summary>
internal static bool TryExtractConnectionType(
List<EnrollmentEvent> events, out string connectionType)
{
connectionType = string.Empty;
var nicEvent = events.LastOrDefault(e =>
e.EventType == Shared.Constants.EventTypes.NetworkInterfaceInfo &&
e.Data?.ContainsKey("connectionType") == true);
var raw = nicEvent?.Data?["connectionType"]?.ToString();
if (raw != "WiFi" && raw != "Ethernet")
return false;
connectionType = raw;
return true;
}
/// <summary>
/// Numeric reader for agent event Data values, which arrive as boxed Newtonsoft
/// primitives (integer → long, decimal → double) — same coercion ladder as
@@ -654,6 +654,11 @@ namespace AutopilotMonitor.Functions.Services
parts.Add($"RebootCount ge {filter.RebootCountMin.Value}");
if (filter.RebootCountMax.HasValue)
parts.Add($"RebootCount le {filter.RebootCountMax.Value}");
// ConnectionType exact match pushed to OData. Legacy index rows that predate
// the projected column lack the property and are excluded by the eq (they
// carry no connection data, which is the intended result).
if (!string.IsNullOrEmpty(filter.ConnectionType))
parts.Add($"ConnectionType eq '{ODataSanitizer.EscapeValue(filter.ConnectionType)}'");
return parts.Count == 0 ? null : string.Join(" and ", parts);
}
@@ -670,6 +675,19 @@ namespace AutopilotMonitor.Functions.Services
return true;
}
/// <summary>
/// Client-side ConnectionType predicate — same backstop role as
/// <see cref="MatchesRebootCountBounds"/>: applied on the search paths that do NOT
/// push the filter to OData (device-snapshot batch-get, legacy unpaged scan) and
/// redundantly on the OData scan paths. Sessions without the projected column
/// (null) never match a set filter.
/// </summary>
internal static bool MatchesConnectionType(SessionSummary session, SessionSearchFilter filter)
{
if (string.IsNullOrEmpty(filter.ConnectionType)) return true;
return string.Equals(session.ConnectionType, filter.ConnectionType, StringComparison.OrdinalIgnoreCase);
}
private static bool MatchesScanClientFilters(SessionSummary session, SessionSearchFilter filter)
{
if (!string.IsNullOrEmpty(filter.SerialNumber) &&
@@ -683,9 +701,10 @@ namespace AutopilotMonitor.Functions.Services
return false;
if (filter.StartedAfter.HasValue && session.StartedAt < filter.StartedAfter.Value) return false;
if (filter.StartedBefore.HasValue && session.StartedAt > filter.StartedBefore.Value) return false;
// RebootCount is also pushed to OData in BuildSearchScanFilter; this is a defensive
// backstop so the bound holds even if that push-down is ever weakened.
// RebootCount / ConnectionType are also pushed to OData in BuildSearchScanFilter;
// these are defensive backstops so the bound holds even if that push-down is ever weakened.
if (!MatchesRebootCountBounds(session, filter)) return false;
if (!MatchesConnectionType(session, filter)) return false;
// AgentVersion / ImeAgentVersion (exact + prefix) are pushed to OData
// in BuildSearchScanFilter — the server has already filtered them out
// before we see the page. No client-side check needed.
@@ -951,9 +970,11 @@ namespace AutopilotMonitor.Functions.Services
&& (s.ImeAgentVersion == null
|| !s.ImeAgentVersion.StartsWith(filter.ImeAgentVersionPrefix!, StringComparison.OrdinalIgnoreCase)))
return false;
// Device-snapshot path has no RebootCount OData push-down — enforce it here so a
// deviceProperties + rebootCountMin query can't return sub-threshold sessions.
// Device-snapshot path has no RebootCount/ConnectionType OData push-down —
// enforce them here so a deviceProperties + rebootCountMin/connectionType
// query can't return non-matching sessions.
if (!MatchesRebootCountBounds(s, filter)) return false;
if (!MatchesConnectionType(s, filter)) return false;
return true;
}).ToList();
}
@@ -225,6 +225,12 @@ namespace AutopilotMonitor.Functions.Services
if (apiRequestCount.HasValue)
indexEntity["ApiRequestCount"] = apiRequestCount.Value;
// Connection-type projection (UpdateSessionConnectionTypeAsync merges this) —
// must be in this superset or a StartedAt-shift full upsert drops it.
var connectionType = sessionEntity.GetString("ConnectionType");
if (!string.IsNullOrEmpty(connectionType))
indexEntity["ConnectionType"] = connectionType;
return indexEntity;
}
@@ -398,6 +404,7 @@ namespace AutopilotMonitor.Functions.Services
string geoRegion = string.Empty;
string geoCity = string.Empty;
string geoLoc = string.Empty;
string? connectionType = null;
string? existingIndexRowKey = null;
// PR3: cascade-delete state-machine columns. Re-registration MUST preserve
// these — losing them would silently clear the lock and let the agent's writes
@@ -444,6 +451,7 @@ namespace AutopilotMonitor.Functions.Services
geoRegion = existingEntity.GetString("GeoRegion") ?? string.Empty;
geoCity = existingEntity.GetString("GeoCity") ?? string.Empty;
geoLoc = existingEntity.GetString("GeoLoc") ?? string.Empty;
connectionType = existingEntity.GetString("ConnectionType");
existingIndexRowKey = existingEntity.GetString("IndexRowKey");
existingDeletionState = existingEntity.GetString("DeletionState");
existingPendingDeletionManifestId = existingEntity.GetString("PendingDeletionManifestId");
@@ -553,6 +561,12 @@ namespace AutopilotMonitor.Functions.Services
if (!string.IsNullOrEmpty(geoLoc))
entity["GeoLoc"] = geoLoc;
// Merge-mode field (UpdateSessionConnectionTypeAsync) — preserve through the
// Replace; the agent re-emits network_interface_info after restart, but the
// value must not be lost in the window until it does.
if (!string.IsNullOrEmpty(connectionType))
entity["ConnectionType"] = connectionType;
// Preserve IndexRowKey through the Replace so that:
// 1. Concurrent UpdateSessionStatusAsync calls can find it (prevents index stale-status)
// 2. UpsertSessionIndexAsync can detect and delete old index entries when startedAt shifts
@@ -2714,6 +2728,52 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// Stores the active network connection type ("WiFi"/"Ethernet") from the latest
/// network_interface_info event (Merge-mode, single field, last-write-wins).
/// Same UpdateEntity-not-Upsert rationale as <see cref="UpdateSessionNetworkLatencyAsync"/>:
/// a tombstoned Sessions row must stay tombstoned (404 → no-op).
/// Non-fatal: failures are logged as warnings and do not block ingest.
/// </summary>
public async Task UpdateSessionConnectionTypeAsync(string tenantId, string sessionId,
string connectionType)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.Sessions);
var update = new TableEntity(tenantId, sessionId)
{
["ConnectionType"] = connectionType
};
await tableClient.UpdateEntityAsync(update, ETag.All, TableUpdateMode.Merge);
// Mirror to SessionsIndex — serves the session list and the cheap
// connectionType= search filter (OData eq push-down on the index).
var idxRef = await tableClient.GetEntityAsync<TableEntity>(
tenantId, sessionId, select: new[] { "IndexRowKey" });
var indexRowKey = idxRef.Value.GetString("IndexRowKey");
if (!string.IsNullOrEmpty(indexRowKey))
{
await MergeSessionIndexAsync(tenantId, indexRowKey,
new TableEntity(tenantId, indexRowKey)
{
["ConnectionType"] = connectionType
});
}
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
_logger.LogDebug(
"UpdateSessionConnectionType no-op: Sessions row {Tenant}/{Session} is absent (tombstoned or never registered)",
tenantId, sessionId);
}
catch (Exception ex)
{
// Non-fatal — don't block ingest
_logger.LogWarning(ex, "Failed to update connection type for session {SessionId}", sessionId);
}
}
// ===== SESSION/EVENT MAPPING HELPERS =====
/// <summary>
@@ -2809,6 +2869,7 @@ namespace AutopilotMonitor.Functions.Services
DurationSeconds = ComputeEffectiveDuration(entity, status, startedAt, completedAt),
AvgApiLatencyMs = SafeGetDouble(entity, "AvgApiLatencyMs"),
ApiRequestCount = SafeGetInt32(entity, "ApiRequestCount"),
ConnectionType = entity.GetString("ConnectionType"),
EnrollmentType = entity.GetString("EnrollmentType") ?? "v1",
DiagnosticsBlobName = entity.GetString("DiagnosticsBlobName"),
// Legacy rows that predate this field surface as null → download path treats
@@ -894,6 +894,8 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
isPreProvisioned: z.boolean().optional().describe('Filter pre-provisioned (white-glove) sessions'),
isHybridJoin: z.boolean().optional().describe('Filter hybrid AAD-join sessions'),
isSelfDeployingProfile: z.boolean().optional().describe('Filter self-deploying/kiosk profile sessions (CloudAssignedOobeConfig 0x20|0x40)'),
connectionType: z.enum(['WiFi', 'Ethernet']).optional()
.describe('Active network connection type ("WiFi" or "Ethernet", exact match on the indexed column; last emission wins). Sessions predating the projection lack the column and are excluded.'),
fields: z.string().optional().describe('Comma-separated pass-through projection over the literal stored column names (case-insensitive, PascalCase, e.g. "Status,OsEdition,ImeAgentVersion,GeoCity"); narrows the row but never drops a real column. PartitionKey + RowKey are always kept. Omit for the full raw row.'),
pageSize: z.coerce.number().int().min(1).max(1000).optional().default(200)
.describe('Page size (1-1000, default 200). Returns this many sessions per call; follow nextLink to fetch more.'),
@@ -906,14 +908,14 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
try {
const { tenantId: rawTenantId, status, startedAfter, startedBefore, serialNumber, agentVersion, agentVersionPrefix,
imeAgentVersion, imeAgentVersionPrefix, manufacturer, model, enrollmentType, deviceName, osBuild,
geoCountry, isPreProvisioned, isHybridJoin, isSelfDeployingProfile, fields, pageSize, continuation } = args;
geoCountry, isPreProvisioned, isHybridJoin, isSelfDeployingProfile, connectionType, fields, pageSize, continuation } = args;
const tenantId = enforceDelegatedTenantForPage(rawTenantId, continuation);
const basePath = pickGlobalOrTenantPath('/api/global/raw/sessions', '/api/raw/sessions', tenantId);
const path = followNextLink(
basePath,
{ tenantId, status, startedAfter, startedBefore, serialNumber, agentVersion, agentVersionPrefix,
imeAgentVersion, imeAgentVersionPrefix, manufacturer, model, enrollmentType, deviceName, osBuild,
geoCountry, isPreProvisioned, isHybridJoin, isSelfDeployingProfile, fields, pageSize },
geoCountry, isPreProvisioned, isHybridJoin, isSelfDeployingProfile, connectionType, fields, pageSize },
continuation,
);
const data = await apiFetch(path);
@@ -139,6 +139,10 @@ export function registerSessionTools(server: McpServer, ga: boolean, delegated:
'e.g. rebootCountMin=5. Only populated for v2 enrollments; sessions that predate the field are excluded.'),
rebootCountMax: z.coerce.number().int().min(0).optional()
.describe('Maximum number of reboots observed during enrollment (<=).'),
connectionType: z.enum(['WiFi', 'Ethernet']).optional()
.describe('Active network connection type during enrollment ("WiFi" or "Ethernet"), indexed for cheap exact-match ' +
'filtering (e.g. "how many machines enrolled over WiFi?"). Last emission wins — a device that switches media ' +
'mid-enrollment reports the most recent state. Sessions that predate the projection lack the column and are excluded.'),
fields: z.string().optional()
.describe('Comma-separated lean projection (e.g. "sessionId,status,agentVersion,startedAt"). ' +
'Use for counting / aggregation to avoid the response cap. Available: sessionId, tenantId, status, ' +
@@ -146,7 +150,8 @@ export function registerSessionTools(server: McpServer, ga: boolean, delegated:
'durationSeconds, currentPhase, failureReason, eventCount, enrollmentType, isPreProvisioned, ' +
'isUserDriven, isHybridJoin, isSelfDeployingProfile, agentVersion, imeAgentVersion, geoCountry, rebootCount, ' +
'avgApiLatencyMs, apiRequestCount (agent→backend HTTP round-trip; weight latency by apiRequestCount when aggregating, ' +
'e.g. fields=geoCountry,avgApiLatencyMs,apiRequestCount for a per-country latency sweep; null on sessions from agents predating the field).'),
'e.g. fields=geoCountry,avgApiLatencyMs,apiRequestCount for a per-country latency sweep; null on sessions from agents predating the field), ' +
'connectionType ("WiFi"/"Ethernet"; null on sessions predating the projection).'),
deviceProperties: z.record(z.string(), z.string()).optional().describe(
'Dynamic device property filters. Keys use "eventType.propertyName" dot notation. ' +
'See the device_properties catalog (call get_resource(name="device_properties")) for all available keys and types. ' +
@@ -62,6 +62,13 @@ namespace AutopilotMonitor.Shared
/// </summary>
public const string ImeMatchLogPath = @"%ProgramData%\AutopilotMonitor\Logs\ime_pattern_matches.log";
/// <summary>
/// Default path for the gather rule debug trace file (debugging/diagnostics).
/// Written when the tenant enables EnableGatherRuleDebugLog; explains every
/// gather-rule evaluation decision, including outcomes that produce no event.
/// </summary>
public const string GatherRuleDebugLogPath = @"%ProgramData%\AutopilotMonitor\Logs\gather_rules_debug.log";
/// <summary>
/// Scheduled Task name used to run the agent as SYSTEM
/// </summary>
@@ -115,6 +115,13 @@ namespace AutopilotMonitor.Shared.DataAccess
/// </summary>
Task UpdateSessionNetworkLatencyAsync(string tenantId, string sessionId,
double avgLatencyMs, int requestCount);
/// <summary>
/// Persists the active network connection type ("WiFi" or "Ethernet") from the
/// latest <c>network_interface_info</c> event. Last-write-wins; fail-soft like
/// the other supplementary-field updaters.
/// </summary>
Task UpdateSessionConnectionTypeAsync(string tenantId, string sessionId,
string connectionType);
// --- Server→Agent Actions ---
/// <summary>
@@ -40,6 +40,12 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public bool EnableImeMatchLog { get; set; } = false;
/// <summary>
/// Whether to write a local gather-rule evaluation trace file.
/// When true, the default path is used: Constants.GatherRuleDebugLogPath.
/// </summary>
public bool EnableGatherRuleDebugLog { get; set; } = false;
public CollectorConfiguration Collectors { get; set; } = default!;
/// <summary>
@@ -334,6 +334,15 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public bool? EnableImeMatchLog { get; set; }
/// <summary>
/// Whether the agent writes a local gather-rule evaluation trace file so customers
/// can diagnose rules that produce no timeline events (scope skips, on_change
/// suppression, empty collector results, logparser details). When true, the agent
/// writes to Constants.GatherRuleDebugLogPath. The trace never leaves the device.
/// null = use agent default (false).
/// </summary>
public bool? EnableGatherRuleDebugLog { get; set; }
/// <summary>
/// Log verbosity level override for this tenant's agents.
/// null = use agent default ("Info"). Values: "Info", "Debug", "Verbose", "Trace".
@@ -275,6 +275,14 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public int? ApiRequestCount { get; set; }
/// <summary>
/// Active network connection type during enrollment: "WiFi" or "Ethernet".
/// Projected at ingest from the latest <c>network_interface_info</c> event
/// (last emission wins — a device that switches media mid-enrollment reports
/// the most recent state). Null for sessions that predate the projection.
/// </summary>
public string? ConnectionType { get; set; }
/// <summary>
/// Enrollment type: "v1" (Autopilot Classic/ESP) or "v2" (Windows Device Preparation).
/// Defaults to "v1" for sessions that predate this field.
@@ -37,6 +37,11 @@ namespace AutopilotMonitor.Shared.Models
public int? RebootCountMin { get; set; }
public int? RebootCountMax { get; set; }
// Active network connection type ("WiFi" or "Ethernet"). Exact match on the
// stored ConnectionType column. Sessions that predate the projection lack the
// property and are excluded (acceptable — they have no connection data).
public string? ConnectionType { get; set; }
public int Limit { get; set; } = 50;
// Dynamic device property filters (key = "eventType.propertyName", value = filter expression)
@@ -58,6 +58,7 @@ interface TenantConfig {
enableTimezoneAutoSet?: boolean | null;
ntpServer?: string | null;
enableImeMatchLog?: boolean | null;
enableGatherRuleDebugLog?: boolean | null;
logLevel?: string | null;
maxBatchSize?: number | null;
showEnrollmentSummary?: boolean | null;
@@ -130,6 +131,7 @@ const DEFAULTS: Record<string, unknown> = {
enableTimezoneAutoSet: null,
ntpServer: 'time.windows.com',
enableImeMatchLog: null,
enableGatherRuleDebugLog: null,
logLevel: null,
maxBatchSize: null,
showEnrollmentSummary: null,
@@ -168,6 +170,7 @@ const RUNTIME_DEFAULTS: Record<string, unknown> = {
enableTimezoneAutoSet: false,
ntpServer: 'time.windows.com',
enableImeMatchLog: false,
enableGatherRuleDebugLog: false,
maxAuthFailures: 5,
authFailureTimeoutMinutes: 0,
logLevel: 'Info',
@@ -296,6 +299,7 @@ function computeRuntime(c: TenantConfig): Record<string, unknown> {
enableTimezoneAutoSet: c.enableTimezoneAutoSet ?? false,
ntpServer: c.ntpServer ?? 'time.windows.com',
enableImeMatchLog: c.enableImeMatchLog ?? false,
enableGatherRuleDebugLog: c.enableGatherRuleDebugLog ?? false,
maxAuthFailures: c.maxAuthFailures ?? 5,
authFailureTimeoutMinutes: c.authFailureTimeoutMinutes ?? 0,
logLevel: c.logLevel ?? 'Info',
@@ -575,6 +579,7 @@ export function SectionTenantConfigReport() {
<ConfigRow label="Timezone Auto-Set" value={config.enableTimezoneAutoSet} configKey="enableTimezoneAutoSet" defaults={DEFAULTS} />
<ConfigRow label="NTP Server" value={config.ntpServer} configKey="ntpServer" defaults={DEFAULTS} />
<ConfigRow label="IME Match Log" value={config.enableImeMatchLog} configKey="enableImeMatchLog" defaults={DEFAULTS} />
<ConfigRow label="Gather Rule Debug Log" value={config.enableGatherRuleDebugLog} configKey="enableGatherRuleDebugLog" defaults={DEFAULTS} />
<ConfigRow label="Log Level" value={config.logLevel} configKey="logLevel" defaults={DEFAULTS} />
<ConfigRow label="Max Batch Size" value={config.maxBatchSize} configKey="maxBatchSize" defaults={DEFAULTS} />
<ConfigRow label="Show Script Output" value={config.showScriptOutput} configKey="showScriptOutput" defaults={DEFAULTS} />
@@ -647,6 +652,7 @@ export function SectionTenantConfigReport() {
<ConfigRow label="Timezone Auto-Set" value={runtime.enableTimezoneAutoSet} configKey="enableTimezoneAutoSet" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="NTP Server" value={runtime.ntpServer} configKey="ntpServer" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="IME Match Log" value={runtime.enableImeMatchLog} configKey="enableImeMatchLog" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="Gather Rule Debug Log" value={runtime.enableGatherRuleDebugLog} configKey="enableGatherRuleDebugLog" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="Log Level" value={runtime.logLevel} configKey="logLevel" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="Send Trace Events" value={runtime.sendTraceEvents} configKey="sendTraceEvents" defaults={RUNTIME_DEFAULTS} />
<ConfigRow label="Unrestricted Mode" value={runtime.unrestrictedMode} configKey="unrestrictedMode" defaults={RUNTIME_DEFAULTS} />
@@ -31,6 +31,7 @@ const ALL_COLUMNS: ColumnDef[] = [
{ key: "started", label: "Started", defaultVisible: true, sortKey: "startedAt" },
{ key: "country", label: "Country", defaultVisible: false, sortKey: "geoCountry", filterKey: "geoCountry" },
{ key: "apiLatency", label: "API Latency", defaultVisible: false, sortKey: "avgApiLatencyMs" },
{ key: "connection", label: "Connection", defaultVisible: false, sortKey: "connectionType", filterKey: "connectionType" },
{ key: "agentVersion", label: "Agent Version", defaultVisible: false, sortKey: "agentVersion", filterKey: "agentVersion" },
{ key: "osName", label: "OS Name", defaultVisible: false, sortKey: "osName", filterKey: "osName" },
{ key: "osBuild", label: "OS Build", defaultVisible: false, sortKey: "osBuild", filterKey: "osBuild" },
@@ -641,7 +642,7 @@ export function SessionTable({
currentSort={sortColumn}
direction={sortDirection}
onSort={onSort}
className={["eventCount", "duration", "started", "country", "apiLatency", "agentVersion", "osName", "osBuild", "osDisplayVersion", "osEdition", "osLanguage"].includes(col.key) ? "px-3" : undefined}
className={["eventCount", "duration", "started", "country", "apiLatency", "connection", "agentVersion", "osName", "osBuild", "osDisplayVersion", "osEdition", "osLanguage"].includes(col.key) ? "px-3" : undefined}
filterKey={col.filterKey}
filterValues={col.filterKey ? uniqueValuesByField[col.filterKey] : undefined}
activeFilter={col.filterKey ? columnFilters[col.filterKey] : undefined}
@@ -922,6 +923,19 @@ function SessionCell({
</td>
);
case "connection":
return (
<td className="px-3 py-4 whitespace-nowrap text-sm text-gray-500">
{session.connectionType ? (
<span title="Active network connection during enrollment (last reported state)">
{session.connectionType}
</span>
) : (
<span className="text-gray-300">—</span>
)}
</td>
);
case "agentVersion": {
const fullVersion = session.agentVersion || "";
const shortVersion = fullVersion.split("+")[0];
@@ -355,6 +355,24 @@ function PhaseSection({
);
}
// Backfilled events (ModernDeploymentTracker error backfill): pre-agent event-log records
// replayed at agent start carry backfilled=true plus the original event-log timeCreated —
// the event's own timestamp is agent emission time, so without this the whole block renders
// with one identical (wrong) time and full Error severity, reading like a live failure.
function getBackfillInfo(event: EnrollmentEvent): { isBackfilled: boolean; recordedAt: Date | null } {
const raw = event.data?.backfilled ?? event.data?.Backfilled;
const isBackfilled = raw === true || raw === "true" || raw === "True";
if (!isBackfilled) return { isBackfilled: false, recordedAt: null };
const t = event.data?.timeCreated ?? event.data?.TimeCreated;
const recordedAt = typeof t === "string" && t !== "" && !isNaN(Date.parse(t)) ? new Date(t) : null;
return { isBackfilled, recordedAt };
}
const BACKFILL_TOOLTIP =
"Recorded by Windows before the agent started — replayed from the event log for visibility. " +
"If enrollment progressed afterwards, this error was already resolved by a retry. " +
"The timestamp shown is the original event-log time.";
function EventRow({ event, showScriptOutput }: { event: EnrollmentEvent; showScriptOutput?: boolean }) {
const [showDetails, setShowDetails] = useState(false);
const [showRaw, setShowRaw] = useState(false);
@@ -429,15 +447,28 @@ function EventRow({ event, showScriptOutput }: { event: EnrollmentEvent; showScr
const hasData = isTruncated || (detailData && Object.keys(detailData).length > 0);
const hasDetails = true; // Every event has at least the metadata block
const { isBackfilled, recordedAt } = getBackfillInfo(event);
return (
<div id={`event-${event.eventId}`} className="bg-gray-50 rounded-lg p-3 hover:bg-gray-100 transition-colors">
<div className="flex items-start justify-between">
<div className="flex-1 min-w-0">
<div className="flex items-center gap-3">
<span className="text-xs text-gray-500 font-mono">
{new Date(event.timestamp).toLocaleTimeString()}
{(recordedAt ?? new Date(event.timestamp)).toLocaleTimeString()}
{recordedAt && (
<span className="text-gray-400 font-sans"> (reported {new Date(event.timestamp).toLocaleTimeString()})</span>
)}
</span>
<SeverityBadge severity={event.severity} />
{isBackfilled && (
<span
className="px-2 py-0.5 rounded text-xs font-medium bg-slate-200 text-slate-600"
title={BACKFILL_TOOLTIP}
>
⟲ Backfilled
</span>
)}
<span className="text-sm font-medium text-gray-900">{event.eventType}</span>
</div>
<p className="mt-1 text-sm text-gray-600" title={event.message || undefined}>{shortenBuildHashInMessage(event.message)}</p>
@@ -608,6 +639,15 @@ function EventRow({ event, showScriptOutput }: { event: EnrollmentEvent; showScr
<span className="w-16 flex-shrink-0 text-gray-400">EventId</span>
<span className="font-mono">{event.eventId}</span>
</div>
{recordedAt && (
<div className="flex mt-0.5">
<span className="w-16 flex-shrink-0 text-gray-400">Recorded</span>
<span className="font-mono">
{recordedAt.toISOString().replace('T', ' ').replace('Z', '')}
<span className="text-gray-400 ml-1" title={BACKFILL_TOOLTIP}>(event log, pre-agent)</span>
</span>
</div>
)}
<div className="flex mt-0.5">
<span className="w-16 flex-shrink-0 text-gray-400">Created</span>
<span className="font-mono">{event.timestamp}</span>
@@ -751,6 +791,7 @@ function RawEventRow({ event }: { event: EnrollmentEvent }) {
const [expanded, setExpanded] = useState(false);
const detailData = useMemo(() => normalizeEventDataForDisplay(event.data), [event.data]);
const hasDetails = detailData && Object.keys(detailData).length > 0;
const { isBackfilled, recordedAt } = getBackfillInfo(event);
const sevColor: Record<string, string> = {
Trace: "text-purple-500",
@@ -765,8 +806,11 @@ function RawEventRow({ event }: { event: EnrollmentEvent }) {
<div id={`event-${event.eventId}`} className="py-1.5 text-xs font-mono">
<div className="flex items-start gap-2">
<span className="text-gray-400 w-8 text-right flex-shrink-0">{event.sequence}</span>
<span className="text-gray-500 flex-shrink-0">{new Date(event.timestamp).toLocaleTimeString()}</span>
<span className="text-gray-500 flex-shrink-0">{(recordedAt ?? new Date(event.timestamp)).toLocaleTimeString()}</span>
<span className={`flex-shrink-0 w-14 ${sevColor[event.severity] || "text-gray-500"}`}>{event.severity}</span>
{isBackfilled && (
<span className="text-slate-500 flex-shrink-0" title={BACKFILL_TOOLTIP}>⟲</span>
)}
<span className="text-gray-900 font-medium flex-shrink-0">{event.eventType}</span>
<span className="text-gray-500 truncate flex-1 min-w-0" title={event.message || undefined}>{shortenBuildHashInMessage(event.message)}</span>
{hasDetails && (
@@ -167,6 +167,8 @@ interface TenantConfigContextValue {
setEnableTimezoneAutoSet: (v: boolean) => void;
enableImeMatchLog: boolean;
setEnableImeMatchLog: (v: boolean) => void;
enableGatherRuleDebugLog: boolean;
setEnableGatherRuleDebugLog: (v: boolean) => void;
logLevel: string;
setLogLevel: (v: string) => void;
showScriptOutput: boolean;
@@ -384,6 +386,7 @@ export function TenantConfigProvider({ children }: { children: React.ReactNode }
const [enableGeoLocation, setEnableGeoLocation] = useState(true);
const [enableTimezoneAutoSet, setEnableTimezoneAutoSet] = useState(false);
const [enableImeMatchLog, setEnableImeMatchLog] = useState(false);
const [enableGatherRuleDebugLog, setEnableGatherRuleDebugLog] = useState(false);
const [logLevel, setLogLevel] = useState("Info");
const [showScriptOutput, setShowScriptOutput] = useState(true);
const [showEnrollmentSummary, setShowEnrollmentSummary] = useState(false);
@@ -503,6 +506,7 @@ export function TenantConfigProvider({ children }: { children: React.ReactNode }
setEnableGeoLocation(data.enableGeoLocation ?? true);
setEnableTimezoneAutoSet(data.enableTimezoneAutoSet ?? false);
setEnableImeMatchLog(data.enableImeMatchLog ?? false);
setEnableGatherRuleDebugLog(data.enableGatherRuleDebugLog ?? false);
setLogLevel(data.logLevel ?? "Info");
setShowScriptOutput(data.showScriptOutput ?? true);
setShowEnrollmentSummary(data.showEnrollmentSummary ?? false);
@@ -700,6 +704,7 @@ export function TenantConfigProvider({ children }: { children: React.ReactNode }
enableGeoLocation,
enableTimezoneAutoSet,
enableImeMatchLog,
enableGatherRuleDebugLog,
logLevel,
showScriptOutput,
showEnrollmentSummary,
@@ -777,7 +782,7 @@ export function TenantConfigProvider({ children }: { children: React.ReactNode }
manufacturerWhitelist, modelWhitelist, validateAutopilotDevice, validateCorporateIdentifier, validateDeviceAssociation,
dataRetentionDays, sessionTimeoutHours, enablePerformanceCollector, performanceCollectorInterval,
helloWaitTimeoutSeconds, selfDestructOnComplete, keepLogFile, rebootOnComplete, rebootDelaySeconds,
contactEmail, enableGeoLocation, enableTimezoneAutoSet, enableImeMatchLog, logLevel, showScriptOutput, showEnrollmentSummary,
contactEmail, enableGeoLocation, enableTimezoneAutoSet, enableImeMatchLog, enableGatherRuleDebugLog, logLevel, showScriptOutput, showEnrollmentSummary,
enrollmentSummaryTimeoutSeconds, enrollmentSummaryBrandingImageUrl, enrollmentSummaryLaunchRetrySeconds,
notificationChannels,
slaTargetSuccessRate, slaTargetMaxDurationMinutes, slaTargetAppInstallSuccessRate,
@@ -1113,6 +1118,7 @@ export function TenantConfigProvider({ children }: { children: React.ReactNode }
setEnableGeoLocation(config.enableGeoLocation ?? true);
setEnableTimezoneAutoSet(config.enableTimezoneAutoSet ?? false);
setEnableImeMatchLog(config.enableImeMatchLog ?? false);
setEnableGatherRuleDebugLog(config.enableGatherRuleDebugLog ?? false);
setLogLevel(config.logLevel ?? "Info");
setShowScriptOutput(config.showScriptOutput ?? true);
setShowEnrollmentSummary(config.showEnrollmentSummary ?? false);
@@ -1542,6 +1548,7 @@ export function TenantConfigProvider({ children }: { children: React.ReactNode }
enableGeoLocation, setEnableGeoLocation,
enableTimezoneAutoSet, setEnableTimezoneAutoSet,
enableImeMatchLog, setEnableImeMatchLog,
enableGatherRuleDebugLog, setEnableGatherRuleDebugLog,
logLevel, setLogLevel,
showScriptOutput, setShowScriptOutput,
showEnrollmentSummary, setShowEnrollmentSummary,
@@ -17,6 +17,7 @@ export function SectionAgentSettings() {
enableGeoLocation, setEnableGeoLocation,
enableTimezoneAutoSet, setEnableTimezoneAutoSet,
enableImeMatchLog, setEnableImeMatchLog,
enableGatherRuleDebugLog, setEnableGatherRuleDebugLog,
logLevel, setLogLevel,
showScriptOutput, setShowScriptOutput,
showEnrollmentSummary, setShowEnrollmentSummary,
@@ -51,6 +52,8 @@ export function SectionAgentSettings() {
setEnableTimezoneAutoSet={setEnableTimezoneAutoSet}
enableImeMatchLog={enableImeMatchLog}
setEnableImeMatchLog={setEnableImeMatchLog}
enableGatherRuleDebugLog={enableGatherRuleDebugLog}
setEnableGatherRuleDebugLog={setEnableGatherRuleDebugLog}
logLevel={logLevel}
setLogLevel={setLogLevel}
showScriptOutput={showScriptOutput}
@@ -24,6 +24,8 @@ interface AgentSettingsSectionProps {
setEnableTimezoneAutoSet: (value: boolean) => void;
enableImeMatchLog: boolean;
setEnableImeMatchLog: (value: boolean) => void;
enableGatherRuleDebugLog: boolean;
setEnableGatherRuleDebugLog: (value: boolean) => void;
logLevel: string;
setLogLevel: (value: string) => void;
showScriptOutput: boolean;
@@ -64,6 +66,8 @@ export default function AgentSettingsSection({
setEnableTimezoneAutoSet,
enableImeMatchLog,
setEnableImeMatchLog,
enableGatherRuleDebugLog,
setEnableGatherRuleDebugLog,
logLevel,
setLogLevel,
showScriptOutput,
@@ -199,6 +203,21 @@ export default function AgentSettingsSection({
</button>
</div>
{/* Gather Rule Debug Log */}
<div className="flex items-center justify-between p-4 rounded-lg border border-gray-200 hover:border-violet-200 transition-colors">
<div>
<p className="font-medium text-gray-900">Gather Rule Debug Log</p>
<p className="text-sm text-gray-500">
Trace gather rule evaluation to a local file to diagnose rules that produce no timeline events. The trace never leaves the device.
{enableGatherRuleDebugLog && <span className="block text-xs text-gray-400 mt-0.5 font-mono">%ProgramData%\AutopilotMonitor\Logs\gather_rules_debug.log</span>}
</p>
</div>
<button onClick={() => setEnableGatherRuleDebugLog(!enableGatherRuleDebugLog)}
className={`relative inline-flex h-6 w-11 flex-shrink-0 items-center rounded-full transition-colors ${enableGatherRuleDebugLog ? 'bg-violet-500' : 'bg-gray-300'}`}>
<span className={`inline-block h-4 w-4 transform rounded-full bg-white transition-transform ${enableGatherRuleDebugLog ? 'translate-x-6' : 'translate-x-1'}`} />
</button>
</div>
{/* Show Script Output */}
<div className="flex items-center justify-between p-4 rounded-lg border border-gray-200 hover:border-violet-200 transition-colors">
<div className="flex-1">
@@ -60,6 +60,7 @@ export interface TenantConfiguration {
enableGeoLocation?: boolean;
enableTimezoneAutoSet?: boolean;
enableImeMatchLog?: boolean;
enableGatherRuleDebugLog?: boolean;
logLevel?: string;
// Teams notifications (legacy)
teamsWebhookUrl?: string;
@@ -315,10 +315,9 @@ export function GlobalSidebar({ children }: { children: ReactNode }) {
// Every nav Link below sets prefetch={false} on purpose. The sidebar renders the whole
// navigation on every page, so the App Router's default viewport prefetch fires one RSC
// request per visible link against the SWA-hosted Next.js runtime on each page load
// (measured: ~10k requests / 14 days). That host speaks HTTP/1.1, so those prefetches
// occupy the browser's 6 connections to the SAME origin that serves the document and the
// route chunks — when the runtime queues them (p95 ~0.9s but a tail reaching 150s+), the
// portal freezes until the queue drains. Every route here is a client component behind
// (measured: ~10k requests / 14 days). The SSR runtime queues that burst server-side and
// drains it all at once (p95 ~0.9s but a tail reaching 150s+), so the navigation the user
// actually wants is stuck behind it and the portal freezes. Every route here is a client component behind
// ProtectedRoute that fetches its own data after mount, so the prefetched RSC payload
// buys nothing. See docs/web/portal-navigation-prefetch.md.
// --- Render a global nav link ---
@@ -47,6 +47,8 @@ export interface Session {
avgApiLatencyMs?: number;
/** Number of HTTP requests behind avgApiLatencyMs — the weight when averaging across sessions. */
apiRequestCount?: number;
/** Active network connection type during enrollment ("WiFi" or "Ethernet", last emission wins). Absent for sessions predating the projection. */
connectionType?: string;
/**
* Compact JSON snapshot of "last known session state" written by the maintenance
* 5h-timeout sweep when a session graduates to terminal Failed (Hybrid User-Driven