Compare commits

...

10 Commits

Author SHA1 Message Date
Oliver Kieselbach a654c7309e feat(agent): attribute mid-ESP coalesced reboots to MDM policy URIs (second sign-in detection)
Device-assigned MDM policies whose CSP URIs match the OS reboot-required
catalog force a coalesced reboot at the end of ESP DeviceSetup - the user
sees an unexpected restart and a second sign-in before AccountSetup.
Invisible in the Intune console; admins simply lived with it. Now attributed
and actionable (PMPC research).

- MdmRebootPolicyTracker (+host): watches DeviceManagement-Enterprise-
  Diagnostics-Provider/Admin EventID 2800, emits one mdm_policy_reboot_required
  per reboot-forcing URI (Warning, ImmediateUpload; tolerant EventData-first
  URI extraction with regex fallback; fail-soft OMADM RebootRequired probe).
  Cross-restart RecordId watermark (mdm-reboot-watermark.json) - mandatory,
  the coalesced reboot restarts the agent and the backfill re-reads the log.
- ShellCoreTracker: RebootCoalescing branch on the already-watched 62407
  stream -> esp_reboot_coalescing corroboration event; deliberate deviation:
  emits on backfill too (live emit races the reboot kill).
- ANALYZE-ESP-005 advisory (warning): names the first URI via {{rebootUri}},
  remediation = reassign profiles device -> user groups; +20 confidence on
  coalescing corroboration, +10 on >=2 URIs.
- Config: Collectors.MdmRebootPolicyWatcherEnabled (default ON, internal
  steering only - agent-native collector, NOT an IAgentAnalyzer, no admin UI),
  MdmRebootPolicyBackfillLookbackMinutes=60; ConfigVersion 35.
- Event types deliberately distinct from system_reboot_detected so the
  backend RebootCount (exact-string match, 2 sites) stays uninflated.
- Docs: docs/agent/mdm-reboot-coalescing.md (+index/log). Exact 2800/62407
  message texts are unverified until E2E VM validation (tolerant matching).

Tests: Agent.V2.Core 1868/1868 (21 new), Functions 3099/3099, MCP 400 passed
(event-types drift), solution build 0 errors.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-20 18:05:11 +02:00
Oliver Kieselbach 6d1ff8d7f0 fix: code-review follow-ups (watcher dispose race, revoke-enforcement tests, scan guard)
- ConsoleBypassWatcher: re-check _disposed after watcher.Start() — a Dispose()
  racing the background arm could leave an armed WMI subscription no code path
  could ever stop (leaked until process exit).
- SignalRContext: collapse rapid accessRevoked pushes into one restart via an
  in-flight guard — a second start() on a non-Disconnected connection throws
  and burns the retry budget.
- config/all: negative-cache managed ids a rescue scan proved missing (5 min
  TTL) — one stale grant (offboarded tenant) no longer makes every delegated
  config/all call pay a full table scan.
- DelegatedAdmin/TenantGroup functions: extract testable revoke-enforcement
  cores (RevokeCoreAsync, SetEnabledCoreAsync, UnassignCoreAsync,
  DeleteGroupCoreAsync, ValidateGrantRequest) and cover the security flows:
  404-on-missing-row writes no audit row and fires no disconnect; real
  revoke/disable/unassign/group-delete always cuts live streams; grant
  validation is fail-closed (GUID tenantId, recognized roles only).
- dashboard/page.tsx: drop unused next/link import left by the banner removal.

Tests: backend 3099 passed, web 557 passed, agent ConsoleBypassWatcher suite
passed, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 13:13:26 +02:00
Oliver Kieselbach 5c8b273f37 chore: bump skills submodule (system-Timestamp warning in raw-query skills)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 12:38:04 +02:00
Oliver Kieselbach da958b5398 fix(backend): business timestamps survive storage migrations (OccurredUtc + RowKey decode)
The 2026-07-18 storage migration reset every row's system Timestamp,
exposing that AuditLogs, OpsEvents and Events displayed, filtered and
retention-swept on it (custom "Timestamp" writes were silent no-ops —
reserved system property). All migrated rows read as 2026-07-18.

- New Helpers/BusinessTimestamp: RowKey decoders (audit !revtick_guid,
  ops bare revtick, event yyyyMMddHHmmssfff_seq), RowKey-range OData
  clause builders (tick-exact, with the mandatory RowKey lt '"' guard
  so no lower bound can ever match legacy bare-GUID audit rows), and
  dual DateTimeOffset/DateTime column reads.
- Writes in all three tables now persist OccurredUtc (custom column,
  copied verbatim by future migrations); no-op ["Timestamp"] removed.
- Read mappers resolve OccurredUtc -> RowKey decode -> system Timestamp;
  GetEarliestSessionEventTimestampAsync uses the same resolution so
  StartedAt/Duration reconciliation can no longer absorb the migration
  date on migrated sessions.
- Date-window filters and retention moved off the system Timestamp onto
  RowKey ranges; ops retention loses its full-table scan and the
  post-migration mass-delete cliff; audit retention runs a temporary
  second pass for legacy GUID rows (TODO: remove after 2027-03-01).
- New GlobalAdmin backfill endpoint POST /api/maintenance/
  backfill-occurred-utc (dry-run by default, batched, idempotent MERGE
  of only OccurredUtc, table=audit|ops) — Events rows are deliberately
  never mutated; their read path decodes the RowKey instead.
- Tests: decoders, clause boundary proofs incl. GUID-safety pins,
  mapper precedence, backfill row decisions (Functions.Tests 3078).
- Docs: docs/backend/business-timestamps.md (system Timestamp is never
  authoritative; migration checklist rule).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 11:29:53 +02:00
Oliver Kieselbach e17eb607ca docs: gather-rule phase scoping + emit-on-change concept doc (OKF)
New rules/ section in the knowledge bundle: semantics table, from-phase latch
behavior (never via Failed, sticky through it), on_change x emitOnlyIfExists
composition, restart re-emit, no-phase-signal caveat, old-agent degradation,
and the prerequisite toggle-wipe fix. Index + log updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 09:51:14 +02:00
Oliver Kieselbach 71000a5dd2 feat(web): gather-rule scope + emit-mode UI, phase dropdowns, edit tags wipe fix
Step 4: portal round-trip for the new fields.

- types.ts: GatherRule + activePhases/activeFromPhase/emitMode; NewRuleForm +
  scopeMode ("always" | "during" | "from") as UI-only view state; canonical
  GATHER_PHASES list (EnrollmentPhase names Start..Complete with display
  labels) + EMIT_MODES; new rules default to emitMode "on_change".
- withDerivedScopeMode: JSON-mode merges rule-shaped JSON (no scopeMode key)
  over form state — the helper derives scopeMode from the data in all four
  JSON paths (create apply/save, edit apply, card edit save) so pasted scope
  fields survive into the payload instead of being silently dropped.
- GatherRuleFormFields: "Active During" mode select + phase multi-select /
  from-phase dropdown (mode switch clears the other mode's fields), "Emit
  Mode" select with helper text; triggerPhase free-text replaced by the same
  canonical dropdown (placeholder used to suggest invalid tokens like
  "DeviceESP"; legacy stored values remain selectable).
- page.tsx: create/save payloads carry the three fields; startEditing maps
  them back (existing rules load as "always", the on_change default is for
  NEW rules only); handleSaveEdit now also sends tags — same wipe class as
  the toggle bug, the full-replace PUT dropped them on every edit.
- GatherRuleCard: trigger section shows the scope + emit mode, e.g.
  "Interval every 60 seconds - from Account Setup - emit on change".

npx tsc --noEmit silent; vitest 557/557.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 09:51:04 +02:00
Oliver Kieselbach a88db34af5 feat(agent): gather-rule phase scoping + on_change emit dedup in GatherRuleExecutor
Step 3: the executor honors the new GatherRule fields.

Phase scope (ActivePhases / ActiveFromPhase), applied to ALL trigger types:
- interval: timers keep running, an out-of-scope tick is a free early-return
  (no start/stop lifecycle races on phase transitions).
- startup: scoped rules not yet in scope at UpdateRules are deferred and run
  exactly once from OnPhaseChanged when their scope activates (same
  _startupRulesExecuted dedup; not part of the WaitForStartupRules latch).
- phase_change: evaluated against the NEW phase; scope gate runs BEFORE the
  (rule, phase) dedup add so an out-of-scope pass does not consume the slot.
- on_event: gated by current phase.
- From-phase latch: phase != Unknown/Failed && ordinal >= threshold; sticky for
  the session incl. through Failed; re-evaluated on config refresh so a rule
  delivered after its threshold passed latches immediately.
- Before the first phase signal, scoped rules are inactive.
- IgnorePhaseScope bypass for --run-gather-rules (no phase context; logged).

EmitMode on_change:
- Canonical hash (keys ordinal-sorted at every level, invariant formatting,
  SHA-256) of the raw collector result BEFORE ruleId/ruleTitle injection.
- First non-empty result always emits; unchanged results increment a per-rule
  suppression counter; the next emitted event carries suppressedPolls +
  suppressedSinceUtc. Empty results (emitOnlyIfExists miss) touch nothing, so
  the combo yields zero noise, one event on appearance, then only on change.
- State survives UpdateRules refreshes (agent restart => at most one re-emit).

Tests: GatherRuleExecutorPhaseScopeTests (scope matrix synchronous via internal
IsRuleInScope; trigger wiring async with positive waits),
GatherRuleExecutorEmitModeTests (state machine via internal ShouldEmitOnChange,
canonical-hash properties, ExecuteRule wiring). V2.Core.Tests 1847/1847.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 09:50:50 +02:00
Oliver Kieselbach 21dd35f787 feat(backend,shared): gather-rule phase scoping + emit-mode fields, custom-rule toggle wipe fix
Step 0-2 of the gather-rule anti-spam plan:
- GatherRuleService.UpdateRuleAsync: toggle-style partial PUTs ({enabled,
  isBuiltIn, isCommunity}) previously full-replaced the stored custom rule and
  wiped Title/Target/Trigger/Parameters. Partials now merge Enabled into the
  existing row; full edit payloads preserve the original CreatedAt.
- GatherRule: new opt-in fields ActivePhases (run only during listed phases),
  ActiveFromPhase (sticky latch from a phase onwards), EmitMode (always |
  on_change). Absent fields = legacy behavior; schema updated accordingly.
- TableStorageService.Rules: Store+Map for the three columns (absent column /
  stored-empty -> null); MapToGatherRule internal for roundtrip tests.
- ContentEquivalent covers the new fields, null-tolerant for ActivePhases so
  pre-existing rows never diff against seeds (no reseed churn).
- GatherRulesFunction.ValidateScopeAndEmitMode: canonical EnrollmentPhase name
  list Start..Complete (Unknown/Failed and numeric tokens rejected), mutual
  exclusion of the two scope fields, emitMode whitelist; 400 on violation.
- ConfigVersion 33 -> 34 (33 was taken by MigrateToApiBaseUrl).

Tests: GatherRuleUpdatePartialMergeTests (merge/replace/unknown-rule/built-in
path), GatherRuleScopeFieldsTests (Store<->Map roundtrip incl. legacy rows,
ContentEquivalent matrix, validation accept/reject). Functions.Tests 3029/3029.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-20 09:50:35 +02:00
Oliver Kieselbach 9933c039ed refactor(shared,backend): MigrateToApiBaseUrl nullable, drop null-forgiving operator
null = no migration is the documented contract and the agent already
assigns null when stripping the field from the on-disk cache; annotate
the property as string? so the backend assignment needs no ! suppression.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 13:33:57 +02:00
Oliver Kieselbach ef67dec263 fix(backend): system-health URLs post EU-migration + GA endpoint visibility
- Agent Binaries check + ops watcher now probe the canonical download alias
  (download.autopilotmonitor.com) AND the legacy blob keepalive; alias failure
  is unhealthy, legacy failure a warning
- McpServerBaseUrl fallback moved from the stale kindwave WEU ACA FQDN to the
  stable custom domain mcp.autopilotmonitor.com (CNAME to the new GWC instance)
- health/detailed and health/mcp include the probed endpoint URLs in check
  details for Global Admins only (storage table endpoint, WEBSITE_HOSTNAME,
  alias/legacy download URLs, MCP server URL)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-19 11:42:40 +02:00
59 changed files with 5066 additions and 214 deletions
+1
View File
@@ -7,5 +7,6 @@ How the on-device agent works:
* [Logs, Persistence & Crash Recovery](logs-and-persistence.md) - every persisted file and why it exists; signal log vs spool; snapshot/journal; the replay-based recovery flow and quarantine.
* [Hello Wizard Un-Skip](hello-wizard-unskip.md) - why a single "WHfB disabled" policy read never decides completion: the HelloWizardStarted signal (prevention + cure) and the HelloTracker confirmation second read.
* [Autopilot ZTD Diagnostics](autopilot-ztd-diagnostics.md) - Windows' own diagnostic surfaces for the profile-download flow: ModernDeployment event IDs (807/815/164/...), the Diagnostics\Autopilot registry key, deployment-service endpoints, and the known-issue error-code map.
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: 2800 watcher with cross-restart watermark, Shell-Core RebootCoalescing corroboration, ANALYZE-ESP-005 advisory.
Reading order for newcomers: overview → decision engine → persistence.
+109
View File
@@ -0,0 +1,109 @@
---
type: Concept
title: MDM Reboot Coalescing — attributing the mid-ESP reboot and second sign-in to policy URIs
description: How the agent detects device-assigned MDM policies that force a coalesced reboot during ESP DeviceSetup (DM-Enterprise EventID 2800 + Shell-Core 62407 RebootCoalescing), and the ANALYZE-ESP-005 advisory that makes the invisible "second sign-in" fixable.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals
tags:
- agent
- esp
- reboot
- policy-attribution
- second-sign-in
timestamp: 2026-07-20T00:00:00+02:00
---
# MDM Reboot Coalescing
Device-assigned MDM policies whose CSP URIs match the OS reboot-required catalog
(`HKLM\SOFTWARE\Microsoft\Provisioning\SyncML\RebootRequiredURIs`) force a **coalesced
reboot at the end of ESP DeviceSetup** — the user sees an unexpected restart and a
**second sign-in screen** before AccountSetup. This is expected OS behavior, invisible in
the Intune console; most admins simply live with it, not knowing that reassigning the
profiles to user groups removes the reboot entirely. Source research: Patch My PC,
["Avoid the second sign-in screen after Autopilot"](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/).
The session already counts reboots (`system_reboot_detected` → `session.rebootCount`);
this feature adds the **cause attribution**.
# Signal chain
1. **EventID 2800**, `Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin`
— logged by the SyncML engine once per policy URI that matched the reboot-required
catalog, during device-policy application in DeviceSetup.
2. **EventID 62407**, `Microsoft-Windows-Shell-Core/Operational`, description containing
`RebootCoalescing` — ESP/CloudExperienceHost initiating the coalesced reboot at the end
of DeviceSetup.
3. The reboot kills the agent; the post-reboot restart re-observes both channels via
backfill (the .evtx persists across the reboot).
# Detectors
## `MdmRebootPolicyTracker` (new collector, `MdmRebootPolicyWatcherHost`)
Structural clone of `WindowsUpdateTracker`: live `EventLogWatcher` on the DM-Enterprise
Admin channel (EventID 2800) + startup backfill (default 60 min lookback) + **cross-restart
RecordId watermark** (`mdm-reboot-watermark.json`). The watermark is mandatory here: the
coalesced reboot restarts the agent, and without it every reboot-forcing URI would be
re-emitted after the very reboot it caused.
Per 2800 record it emits one `mdm_policy_reboot_required` event (Warning,
`ImmediateUpload=true` — the announced reboot will kill the agent, flush now) with
`Data.rebootUri` (the CSP URI), `recordId`, `backfilled`, `description` (truncated 1000),
and a fail-soft `omadmRebootRequiredFlag` corroboration probe of
`HKLM\SOFTWARE\Microsoft\Provisioning\OMADM\SyncML\RebootRequired` (null = unknown, never
blocks the emit). Per-URI emission (not aggregated) is deliberate: an aggregate would need
a flush point that races the reboot kill.
**Tolerant matching**: the exact 2800 message text is unverified — emission is gated on
the EventID only. URI extraction prefers structured EventData values starting with `./`
and falls back to a permissive regex over the description (trailing punctuation trimmed).
Events with an unparseable URI are still emitted (timeline-visible, rule stays silent, and
the captured description tells us how to fix the extraction). Exact texts are validated
post-deploy on the E2E VM and recorded here.
Config (internal steering only, no admin UI — this is an agent-native collector, NOT an
`IAgentAnalyzer`): `Collectors.MdmRebootPolicyWatcherEnabled` (default true),
`Collectors.MdmRebootPolicyBackfillLookbackMinutes` (default 60). ConfigVersion 35.
## `ShellCoreTracker` — `RebootCoalescing` branch
The already-watched 62407 stream gets a fourth branch (checked AFTER
WhiteGlove/failure/exiting so their semantics stay untouched): description containing
`RebootCoalescing` → one `esp_reboot_coalescing` event (Warning, ImmediateUpload), pure
informational — no `FinalizingSetup` transition, no C# event raise, no DecisionCore
involvement.
**Deliberate deviation**: unlike the other 62407 branches, the backfill path DOES emit
this event. The live emit races the very reboot that kills the process; the post-restart
5-min backfill re-emission is the only reliable delivery. A live+backfill cross-process
duplicate is possible and accepted (fire-once bool resets per process; both copies carry
the identical historical timestamp; the backfill copy is marked `backfilled:true`; the
analyze rule uses exists-semantics). The shared 5-min lookback is NOT enlarged — the
rule's required condition is the watermarked 2800 event; coalescing is optional
corroboration only.
# Advisory — ANALYZE-ESP-005
"MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)", severity `warning`,
`markSessionAsFailedDefault: false`. Required: `mdm_policy_reboot_required.rebootUri`
exists (first URI interpolates as `{{rebootUri}}`). Optional corroboration:
`esp_reboot_coalescing` observed (+20 confidence) and ≥2 reboot URIs (+10) on base 70 /
threshold 70. Remediation: map each URI to its Intune profile and reassign device → user
groups (or exclude the Autopilot device group), or accept + communicate the restart.
Because `{{token}}` interpolation is scalar-only, the explanation points at the timeline
for the complete URI list — each URI is its own event.
# Non-goals / invariants
* The new event types are **deliberately distinct from `system_reboot_detected`** — the
backend RebootCount matches that string exactly in two places
(`EventIngestProcessor.Classification` + the terminal recount); attribution events must
never inflate the reboot count.
* No session-level field / SessionInfoCard change in this iteration (follow-up candidate).
* The `RebootRequiredURIs` catalog itself is not read (format unverified); only the 2800
events (which carry the actually-triggered URIs) are consumed.
# Citations
* [Patch My PC — Avoid the second sign-in screen after Autopilot](https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/)
* [Microsoft Learn — Enrollment Status Page tracking](https://learn.microsoft.com/en-us/autopilot/enrollment-status)
+95
View File
@@ -0,0 +1,95 @@
---
type: Concept
title: Business Timestamps vs. the Azure Tables System Timestamp
description: Why the system Timestamp is never authoritative, how the 2026-07-18 migration exposed that, and the OccurredUtc + RowKey-decode compensation (mappers, RowKey-range date filters, retention, backfill).
resource: /src/Backend/AutopilotMonitor.Functions/Helpers/BusinessTimestamp.cs
tags:
- backend
- table-storage
- audit-log
- ops-events
- events
- migration
timestamp: 2026-07-20T00:00:00+02:00
---
# Problem
Azure Table Storage's `Timestamp` is a **system property**: the server ignores any supplied
value on write and stamps the row's *write* time; every row rewrite (storage-account
migration, MERGE) resets it. Three tables nevertheless treated it as the business event
time — writes like `["Timestamp"] = evt.Timestamp` were silent no-ops, unnoticed because
write time ≈ event time in normal operation.
The 2026-07-18 EU storage migration rewrote every row, so all migrated rows in
**AuditLogs**, **OpsEvents**, and **Events** read as "2026-07-18 ~13:58 UTC": displays were
wrong, date-window filters matched the wrong rows, and retention froze (nothing older than
the migration moment) with a mass-delete cliff one retention period later.
# Rule
**The system Timestamp is never authoritative.** Every table that needs an event time
stores it in the custom column `OccurredUtc` (`BusinessTimestamp.OccurredUtcColumn`).
Custom columns are copied verbatim by `scripts/Migration/Copy-TableStorage.ps1`, so future
migrations preserve chronology. Migration checklist: any table whose reads/filters depend
on a timestamp must carry it as a custom column — audit this before every storage move.
# Recovery sources per table
The original time survives migrations inside the RowKey:
| Table | RowKey scheme | Decodable |
|---|---|---|
| AuditLogs | `!{MaxTicks-ticks:D19}_{guid:N}` (since 2026-05-06) | Yes; legacy bare-GUID rows (pre-2026-05) are lost |
| OpsEvents | `{MaxTicks-ticks:D19}` | Yes, all rows |
| Events | `{ts:yyyyMMddHHmmssfff}_{seq:D10}` (sanitized agent time) | Yes, all rows (ms precision) |
# Mechanism
`BusinessTimestamp` (Functions/Helpers) is the single source of truth: RowKey decoders,
the OData clause builders, and `GetUtcDateTime` (datetime columns materialize as
`DateTimeOffset` *or* `DateTime` depending on payload shape — always read via this helper).
1. **Read mappers** resolve `OccurredUtc` → RowKey decode → system Timestamp (last
resort; only audit legacy GUID rows end there). Resolvers:
`TableStorageService.ResolveAuditTimestamp`, `TableOpsEventRepository.ResolveTimestamp`,
`TableStorageService.ResolveEventTimestamp` (also used by
`GetEarliestSessionEventTimestampAsync`, which feeds StartedAt/Duration reconciliation —
a system-Timestamp read there would rewrite migrated sessions' derived fields).
2. **Date-window filters and retention use RowKey ranges**, not property comparisons:
OData excludes rows missing a compared property (an `OccurredUtc` filter would drop
every not-yet-backfilled row), and RowKey ranges are index-backed. Boundary semantics
are tick-exact equivalents of the old `Timestamp ge/le` clauses.
**Trap:** audit legacy GUID RowKeys (hex first char) sort *after* all `!`-rows, so
every lower bound `RowKey ge '!…'` must be paired with `RowKey lt '"'`
(`AuditTimeEncodedUpperBound`) — without the guard, a retention sweep would delete all
legacy rows. Pinned by `BusinessTimestampFilterClauseTests`.
3. **Writes** in all three tables now set `OccurredUtc`; the no-op `["Timestamp"]`
entries were removed.
4. **Backfill** (`POST /api/maintenance/backfill-occurred-utc`, GlobalAdminOnly, dry-run
by default, `table=audit|ops`, batched via `maxRows`/`continuation`): MERGE that adds
only `OccurredUtc` decoded from the RowKey — deterministic and idempotent. **Events
rows are deliberately never backfilled** (read-path decode is sufficient; no bulk
mutation of enrollment evidence). Note: a MERGE resets the row's system Timestamp —
irrelevant once nothing depends on it.
5. **Audit retention** runs two passes until ~2027-01: RowKey-range for `!`-rows plus a
`RowKey ge '0' and Timestamp lt …` pass that ages legacy GUID rows out via their frozen
migration-date system Timestamp (remove after 2027-03-01).
# Consequences
* Audit legacy GUID rows have "unknown date" semantics: excluded from date-filtered
views, shown with the (wrong but honest) 2026-07-18 fallback in unfiltered views.
* Raw endpoints (`/api/raw/*`, `query_table`) intentionally keep showing the raw system
Timestamp — they are storage-state inspectors, not business views.
* Event timestamps for pre-cutover rows resolve to the RowKey's sanitized agent time
(ms precision); `ReceivedAt` (server ingest time, rows since 2026-03-19) and
`OriginalTimestamp` (pre-clamp agent time, clamped rows only) remain separate fields.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/BusinessTimestamp.cs` — decoders + clause builders
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Maintenance.cs` — audit write/mapper/filters/retention
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableOpsEventRepository.cs` — ops write/mapper/filters/retention
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Sessions.cs` — event writes, `ResolveEventTimestamp`, earliest-event read
* `src/Backend/AutopilotMonitor.Functions/Services/OccurredUtcBackfillService.cs` + `Functions/Admin/BackfillOccurredUtcFunction.cs` — backfill
+9
View File
@@ -18,6 +18,15 @@ https://docs.autopilotmonitor.com (separate repository).
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [MDM Reboot Coalescing](agent/mdm-reboot-coalescing.md) - Attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: DM-Enterprise 2800 watcher with cross-restart watermark, Shell-Core RebootCoalescing corroboration, ANALYZE-ESP-005 advisory.
# Rules
* [Gather Rule Phase Scoping & Emit-on-Change](rules/gather-rule-phase-scoping.md) - Restricting gather rules to enrollment phases (activePhases / activeFromPhase sticky latch) and emitMode "on_change" result-dedup — the anti-spam pair for interval rules.
# Backend
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
# Security & Identity
+7
View File
@@ -1,5 +1,12 @@
# Log
## 2026-07-20
* **Creation**: Added `agent/mdm-reboot-coalescing.md` — the agent now attributes the "unexpected mid-ESP reboot + second sign-in" pattern (PMPC research) to the device-assigned policy URIs that forced it: new `MdmRebootPolicyTracker` (DM-Enterprise EventID 2800, per-URI `mdm_policy_reboot_required` events, cross-restart RecordId watermark), a `RebootCoalescing` branch in `ShellCoreTracker` (`esp_reboot_coalescing`, backfill-emitting by design), and the `ANALYZE-ESP-005` advisory (reassign device → user groups). ConfigVersion 35; event types deliberately distinct from `system_reboot_detected` so RebootCount stays uninflated.
* **Creation**: Added `backend/business-timestamps.md` (new `backend/` section) — the 2026-07-18 migration reset every row's system Timestamp, exposing that AuditLogs/OpsEvents/Events had (unknowingly) displayed and filtered on it all along; documents the `OccurredUtc` column, the RowKey decoders and RowKey-range date/retention clauses in `BusinessTimestamp`, the GUID-guard trap, the audit/ops backfill endpoint, and the "system Timestamp is never authoritative" migration rule.
* **Creation**: Added `rules/gather-rule-phase-scoping.md` (new `rules/` section) — gather rules gain `activePhases` (run only during listed phases), `activeFromPhase` (sticky latch from a phase onwards, never via Failed) and `emitMode: "on_change"` (canonical-hash result dedup with `suppressedPolls` observability), ConfigVersion 34. Includes the prerequisite backend fix: the custom-rule toggle PUT no longer wipes the stored rule definition (partial-merge in `GatherRuleService.UpdateRuleAsync`).
## 2026-07-18
* **Creation**: Added `agent/endpoint-migration.md` — config-channel agent re-home (`MigrateToApiBaseUrl` on `GET agent/config`, ConfigVersion 33): backend or per-tenant API-URL moves without DNS indirection (impossible due to mTLS/Flex-TLS). Live-fetch-only with cache strip, `AgentEndpointMigrationRules` allowlist validated on both sides, one hop, kill-switch wins. Built as the lesson from the 2026-07-18 WEU→GWC cutover, which hard-cut all in-flight agents.
+97
View File
@@ -0,0 +1,97 @@
---
type: Concept
title: Gather Rule Phase Scoping & Emit-on-Change
description: How gather rules are restricted to enrollment phases (activePhases / activeFromPhase) and how emitMode "on_change" suppresses repeated identical results — the anti-spam pair for interval rules.
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Gather/GatherRuleExecutor.cs
tags:
- agent
- backend
- gather-rules
- config
timestamp: 2026-07-20T00:00:00+02:00
---
# Problem
Interval gather rules (e.g. a registry poll every 60 s) ran during **all** enrollment phases
and emitted an event on **every** tick — including `exists=false` events while the target key
did not exist yet. A rule needed only from Account Setup onwards produced hundreds of
identical noise events per session and wasted collector runs.
# Schema
Three opt-in fields on `GatherRule` (Shared model, `rules/schema/gather-rule.schema.json`,
delivered to the agent via `GET agent/config`; ConfigVersion 34). Absent fields = legacy
behavior — full back-compat.
| Field | Type | Semantics |
|---|---|---|
| `activePhases` | `string[]` \| null | Rule runs only while the current phase is in the list. Null/empty = unrestricted. |
| `activeFromPhase` | `string` \| null | Sticky latch: activates when the phase first reaches the threshold, then stays active for the rest of the session — including through `Failed`. Null = unrestricted. |
| `emitMode` | `"always"` \| `"on_change"` \| null | `on_change`: poll on the trigger cadence, emit only when the collected result changes. Null/`always` = emit every collection (legacy). |
Canonical phase tokens are the `EnrollmentPhase` enum **names** `Start`(0) …`Complete`(7).
Backend validation (`GatherRulesFunction.ValidateScopeAndEmitMode`) rejects `Unknown`/`Failed`,
numeric tokens, unknown emit modes, and both scope fields set at once (400). The agent
defensively prefers `activePhases` if both arrive anyway.
# Scope semantics (agent, `GatherRuleExecutor`)
* Scope applies to **all trigger types**:
* `interval` — timers keep running; an out-of-scope tick is a free early-return (no
start/stop lifecycle races on phase transitions).
* `startup` — a scoped startup rule not yet in scope at `UpdateRules` is **deferred**: it
runs exactly once from `OnPhaseChanged` when its scope activates (same
`_startupRulesExecuted` dedup; it is not part of the `WaitForStartupRules` latch).
* `phase_change` — evaluated against the **new** phase; the scope gate runs before the
per-(rule, phase) dedup so an out-of-scope pass does not consume the execution slot.
* `on_event` — gated by the current phase.
* **Before the first phase signal** of a session (`_currentPhase == Unknown`) scoped rules are
inactive. Sessions that never produce phase signals never run them (documented behavior).
* **From-phase latch**: activates when `phase != Unknown && phase != Failed && (int)phase >=
(int)fromPhase`. `Failed=99` would ordinal-satisfy every threshold, hence the explicit
exclusion — but once latched, a rule stays active through `Failed`. A config refresh that
delivers a from-phase rule after its threshold passed latches it immediately.
* `--run-gather-rules` (diagnostic CLI) sets `IgnorePhaseScope = true` — no phase context
exists there, scoped rules execute unconditionally (log line marks the bypass).
# Emit-on-change semantics
* The collector result dict is canonically hashed (keys ordinal-sorted at every nesting
level, invariant value formatting, SHA-256) **before** `ruleId`/`ruleTitle` injection.
* First in-scope non-empty result always emits — on an absent registry key that first
`exists=false` event *is* the visible "we're polling now" indicator.
* Unchanged results increment a per-rule suppression counter; the next emitted event carries
`suppressedPolls` + `suppressedSinceUtc` in its `Data` so the gap is observable.
* Composition with `emitOnlyIfExists=true` (collector-level): a miss returns an empty dict —
no emit AND no hash update. Result: zero noise while absent, one event on appearance, then
only on change.
* State is in-memory and survives `UpdateRules` config refreshes; an agent restart causes at
most one re-emit per rule (acceptable under the per-enrollment lifecycle).
* Volatile collectors (eventlog timestamps, logparser positions) degrade `on_change` to
`always` — every poll hashes differently. The option stays visible for them since some
outputs are stable; this is documented, not hidden.
# Portal & delivery
* Create form defaults new rules to `emitMode: "on_change"`; existing rules load as
`always`. The scope UI is a three-way mode (always / during phases / from phase) backed by
a canonical `GATHER_PHASES` list — the same list replaced the free-text `triggerPhase`
input, whose placeholder used to suggest invalid tokens.
* The custom-rule toggle PUT (`{enabled, isBuiltIn, isCommunity}`) previously full-replaced
the stored rule and wiped Title/Target/Trigger — `GatherRuleService.UpdateRuleAsync` now
merges toggle-style partials (empty Title/CollectorType/Target) into the existing row and
preserves `CreatedAt` on full edits. This fix is a prerequisite: without it every toggle
would silently clear the new fields.
* Old agents (Newtonsoft) ignore unknown config properties: scoped rules run everywhere,
`emitMode` is ignored — graceful degradation until the fleet rolls forward.
* `ContentEquivalent` treats null and empty `activePhases` as equal, so pre-existing DB rows
(absent columns → null) never diff against seeds — no reseed churn.
# Citations
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Gather/GatherRuleExecutor.cs` — scope gates, latch, canonical hash
* `src/Backend/AutopilotMonitor.Functions/Functions/Rules/GatherRulesFunction.cs` — `ValidateScopeAndEmitMode`
* `src/Backend/AutopilotMonitor.Functions/Services/GatherRuleService.cs` — partial-PUT merge, `ContentEquivalent`
* `src/Shared/AutopilotMonitor.Shared/Models/Rules/GatherRule.cs` — field contracts
* Tests: `GatherRuleExecutorPhaseScopeTests`, `GatherRuleExecutorEmitModeTests` (agent); `GatherRuleUpdatePartialMergeTests`, `GatherRuleScopeFieldsTests` (backend)
+74
View File
@@ -0,0 +1,74 @@
{
"$schema": "../schema/analyze-rule.schema.json",
"ruleId": "ANALYZE-ESP-005",
"title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)",
"description": "Detects the 'unexpected reboot + second sign-in screen' pattern: one or more device-assigned MDM policies matched the OS reboot-required catalog during ESP DeviceSetup, and Windows coalesced them into a restart at the end of DeviceSetup — forcing the user through a second sign-in before AccountSetup. Attributes the reboot to the specific CSP policy URIs so the profiles can be reassigned.",
"severity": "warning",
"category": "esp",
"version": "1.0.0",
"author": "Autopilot Monitor",
"enabled": true,
"trigger": "single",
"baseConfidence": 70,
"conditions": [
{
"signal": "reboot_forcing_policy_uri",
"source": "event_data",
"eventType": "mdm_policy_reboot_required",
"dataField": "rebootUri",
"operator": "exists",
"required": true
},
{
"signal": "reboot_uri_count",
"source": "event_count",
"eventType": "mdm_policy_reboot_required",
"operator": "count_gte",
"value": "1",
"required": false
},
{
"signal": "esp_coalesced_reboot_observed",
"source": "event_type",
"eventType": "esp_reboot_coalescing",
"operator": "exists",
"required": false
}
],
"confidenceFactors": [
{ "signal": "esp_coalesced_reboot_observed", "condition": "exists", "weight": 20 },
{ "signal": "mdm_policy_reboot_required", "condition": "count >= 2", "weight": 10 }
],
"confidenceThreshold": 70,
"explanation": "During ESP DeviceSetup, a device-assigned MDM policy matched the OS reboot-required catalog and Windows scheduled a **coalesced reboot** at the end of DeviceSetup — the user saw an unexpected restart and a **second sign-in screen** before AccountSetup continued.\n\nFirst reboot-forcing policy URI: `{{rebootUri}}`\n\nWhen several policies force a reboot they are coalesced into one restart; **each** reboot-forcing policy appears as its own `mdm_policy_reboot_required` event on the session timeline — check the timeline for the complete URI list.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
"remediation": [
{
"title": "Reassign the reboot-forcing policies to user groups",
"steps": [
"Open each `mdm_policy_reboot_required` timeline event and note the CSP URI (e.g. `./Device/Vendor/MSFT/...`)",
"Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area",
"Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup",
"Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment"
]
},
{
"title": "Or accept the reboot and set expectations",
"steps": [
"If the policy must be device-assigned (e.g. a security baseline), keep it and document the expected mid-ESP restart",
"Communicate to users that a restart and second sign-in during provisioning is expected for these deployments"
]
}
],
"relatedDocs": [
{
"title": "Patch My PC: Avoid the second sign-in screen after Autopilot",
"url": "https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/"
},
{
"title": "Microsoft Learn: Enrollment Status Page tracking",
"url": "https://learn.microsoft.com/en-us/autopilot/enrollment-status"
}
],
"markSessionAsFailedDefault": false,
"tags": ["esp", "reboot", "coalescing", "second-sign-in", "policy-attribution", "user-experience"]
}
+88
View File
@@ -1785,6 +1785,94 @@
"enrollment"
]
},
{
"ruleId": "ANALYZE-ESP-005",
"title": "MDM Policy Forced a Mid-ESP Reboot (Second Sign-In)",
"description": "Detects the 'unexpected reboot + second sign-in screen' pattern: one or more device-assigned MDM policies matched the OS reboot-required catalog during ESP DeviceSetup, and Windows coalesced them into a restart at the end of DeviceSetup — forcing the user through a second sign-in before AccountSetup. Attributes the reboot to the specific CSP policy URIs so the profiles can be reassigned.",
"severity": "warning",
"category": "esp",
"version": "1.0.0",
"author": "Autopilot Monitor",
"enabled": true,
"trigger": "single",
"baseConfidence": 70,
"conditions": [
{
"signal": "reboot_forcing_policy_uri",
"source": "event_data",
"eventType": "mdm_policy_reboot_required",
"dataField": "rebootUri",
"operator": "exists",
"required": true
},
{
"signal": "reboot_uri_count",
"source": "event_count",
"eventType": "mdm_policy_reboot_required",
"operator": "count_gte",
"value": "1",
"required": false
},
{
"signal": "esp_coalesced_reboot_observed",
"source": "event_type",
"eventType": "esp_reboot_coalescing",
"operator": "exists",
"required": false
}
],
"confidenceFactors": [
{
"signal": "esp_coalesced_reboot_observed",
"condition": "exists",
"weight": 20
},
{
"signal": "mdm_policy_reboot_required",
"condition": "count >= 2",
"weight": 10
}
],
"confidenceThreshold": 70,
"explanation": "During ESP DeviceSetup, a device-assigned MDM policy matched the OS reboot-required catalog and Windows scheduled a **coalesced reboot** at the end of DeviceSetup — the user saw an unexpected restart and a **second sign-in screen** before AccountSetup continued.\n\nFirst reboot-forcing policy URI: `{{rebootUri}}`\n\nWhen several policies force a reboot they are coalesced into one restart; **each** reboot-forcing policy appears as its own `mdm_policy_reboot_required` event on the session timeline — check the timeline for the complete URI list.\n\nThis is expected OS behavior, not a failure — but it degrades the enrollment experience, confuses users, and is invisible in the Intune console.",
"remediation": [
{
"title": "Reassign the reboot-forcing policies to user groups",
"steps": [
"Open each `mdm_policy_reboot_required` timeline event and note the CSP URI (e.g. `./Device/Vendor/MSFT/...`)",
"Map each URI to its Intune profile (Settings Catalog / configuration profile) — the URI path names the CSP area",
"Reassign those profiles from device groups to user groups so they apply during/after AccountSetup instead of DeviceSetup",
"Alternatively, exclude the Autopilot device group from the profile and deliver the setting post-enrollment"
]
},
{
"title": "Or accept the reboot and set expectations",
"steps": [
"If the policy must be device-assigned (e.g. a security baseline), keep it and document the expected mid-ESP restart",
"Communicate to users that a restart and second sign-in during provisioning is expected for these deployments"
]
}
],
"relatedDocs": [
{
"title": "Patch My PC: Avoid the second sign-in screen after Autopilot",
"url": "https://patchmypc.com/blog/autopilot-unexpected-reboot-autopilot-second-login-screen/"
},
{
"title": "Microsoft Learn: Enrollment Status Page tracking",
"url": "https://learn.microsoft.com/en-us/autopilot/enrollment-status"
}
],
"markSessionAsFailedDefault": false,
"tags": [
"esp",
"reboot",
"coalescing",
"second-sign-in",
"policy-attribution",
"user-experience"
]
},
{
"ruleId": "ANALYZE-ID-001",
"title": "Expected Machine Certificate Not Found",
+18
View File
@@ -96,6 +96,24 @@
"type": "string",
"description": "Event type to trigger on (only used when trigger = \"on_event\")."
},
"activePhases": {
"type": "array",
"description": "Run the rule only while the current enrollment phase is one of these phases (EnrollmentPhase enum names). Empty/absent = unrestricted. Mutually exclusive with activeFromPhase.",
"items": {
"type": "string",
"enum": ["Start", "DevicePreparation", "DeviceSetup", "AppsDevice", "AccountSetup", "AppsUser", "FinalizingSetup", "Complete"]
}
},
"activeFromPhase": {
"type": "string",
"description": "Activate the rule once the enrollment phase first reaches this phase, then keep it active for the rest of the session (sticky latch). Absent = unrestricted. Mutually exclusive with activePhases.",
"enum": ["Start", "DevicePreparation", "DeviceSetup", "AppsDevice", "AccountSetup", "AppsUser", "FinalizingSetup", "Complete"]
},
"emitMode": {
"type": "string",
"description": "Emit behavior: \"always\" (default — emit on every collection) or \"on_change\" (poll on the trigger cadence but emit only when the collected result changes; the first in-scope result always emits).",
"enum": ["always", "on_change"]
},
"outputEventType": {
"type": "string",
"description": "EventType for the emitted event (e.g., \"gather_proxy_settings\")."
@@ -0,0 +1,249 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
{
/// <summary>
/// EmitMode "on_change" on <see cref="GatherRuleExecutor"/>: poll on the trigger cadence but
/// emit only when the collected result changes. The dedup state machine is asserted
/// synchronously via the internal <c>ShouldEmitOnChange</c> / <c>ComputeCanonicalHash</c>
/// (no timer races); the ExecuteRule wiring (hash-before-injection, empty-result no-op,
/// emitOnlyIfExists composition) through the public startup path.
/// </summary>
[Collection("SerialThreading")] // startup rules execute on the shared ThreadPool
public sealed class GatherRuleExecutorEmitModeTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
private readonly List<EnrollmentEvent> _events = new List<EnrollmentEvent>();
private readonly object _eventsGate = new object();
private readonly GatherRuleExecutor _executor;
private const string AbsentPath =
"HKLM\\SOFTWARE\\AutopilotMonitorTests\\DefinitelyAbsent_7e3a1f4b-0000-0000-0000-000000000000";
public GatherRuleExecutorEmitModeTests()
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_executor = new GatherRuleExecutor(
"sess", "tenant",
evt => { lock (_eventsGate) _events.Add(evt); },
logger)
{
UnrestrictedMode = true
};
}
public void Dispose()
{
_executor.Dispose();
_tmp.Dispose();
}
private int EventCount { get { lock (_eventsGate) return _events.Count; } }
private EnrollmentEvent SingleEvent()
{
lock (_eventsGate) return Assert.Single(_events);
}
private bool WaitForEventCount(int count, int timeoutMs = 10000)
{
var sw = Stopwatch.StartNew();
while (sw.ElapsedMilliseconds < timeoutMs)
{
if (EventCount >= count) return true;
Thread.Sleep(25);
}
return EventCount >= count;
}
private static GatherRule OnChangeRule(string id = "GATHER-OC-001") => new GatherRule
{
RuleId = id,
Title = id,
CollectorType = "registry",
Target = AbsentPath,
Trigger = "interval",
IntervalSeconds = 3600,
OutputEventType = "gather_test",
Enabled = true,
EmitMode = "on_change",
};
private static Dictionary<string, object> Result(params (string Key, object Value)[] pairs)
{
var d = new Dictionary<string, object>();
foreach (var (key, value) in pairs) d[key] = value;
return d;
}
// ── on_change state machine (synchronous) ──────────────────────────
[Fact]
public void FirstResult_AlwaysEmits_ThenUnchangedIsSuppressed_ThenChangeReemits()
{
var rule = OnChangeRule();
Assert.True(_executor.ShouldEmitOnChange(rule, Result(("exists", false)))); // first
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false)))); // unchanged
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false)))); // unchanged
Assert.True(_executor.ShouldEmitOnChange(rule, Result(("exists", true), ("Version", "1")))); // changed
}
[Fact]
public void EmitAfterSuppressionStreak_CarriesSuppressedPollCount_AndResets()
{
var rule = OnChangeRule();
Assert.True(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
var changed = Result(("exists", true));
Assert.True(_executor.ShouldEmitOnChange(rule, changed));
Assert.Equal(3, changed["suppressedPolls"]);
Assert.True(changed.ContainsKey("suppressedSinceUtc"));
// Counter must reset: the next change emits without a suppression payload.
var changedAgain = Result(("exists", false));
Assert.True(_executor.ShouldEmitOnChange(rule, changedAgain));
Assert.False(changedAgain.ContainsKey("suppressedPolls"));
Assert.False(changedAgain.ContainsKey("suppressedSinceUtc"));
}
[Fact]
public void SuppressionState_IsIsolatedPerRule()
{
var ruleA = OnChangeRule("GATHER-OC-A");
var ruleB = OnChangeRule("GATHER-OC-B");
var payload = Result(("exists", false));
Assert.True(_executor.ShouldEmitOnChange(ruleA, new Dictionary<string, object>(payload)));
// Same content, different rule: B has no hash yet — must emit.
Assert.True(_executor.ShouldEmitOnChange(ruleB, new Dictionary<string, object>(payload)));
Assert.False(_executor.ShouldEmitOnChange(ruleA, new Dictionary<string, object>(payload)));
Assert.False(_executor.ShouldEmitOnChange(ruleB, new Dictionary<string, object>(payload)));
}
[Fact]
public void DedupState_SurvivesUpdateRulesRefresh()
{
var rule = OnChangeRule();
Assert.True(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
// Config refresh mid-session must not reset the last-emitted hash (that would
// re-emit an unchanged result on every refresh).
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.False(_executor.ShouldEmitOnChange(rule, Result(("exists", false))));
}
// ── Canonical hashing ──────────────────────────────────────────────
[Fact]
public void CanonicalHash_IsKeyOrderInsensitive()
{
var a = new Dictionary<string, object> { ["alpha"] = 1, ["beta"] = "x" };
var b = new Dictionary<string, object> { ["beta"] = "x", ["alpha"] = 1 };
Assert.Equal(
GatherRuleExecutor.ComputeCanonicalHash(a),
GatherRuleExecutor.ComputeCanonicalHash(b));
}
[Fact]
public void CanonicalHash_DetectsValueChanges_IncludingNested()
{
var baseline = new Dictionary<string, object>
{
["exists"] = true,
["entries"] = new List<object> { new Dictionary<string, object> { ["id"] = 1 } },
};
var changedScalar = new Dictionary<string, object>
{
["exists"] = false,
["entries"] = new List<object> { new Dictionary<string, object> { ["id"] = 1 } },
};
var changedNested = new Dictionary<string, object>
{
["exists"] = true,
["entries"] = new List<object> { new Dictionary<string, object> { ["id"] = 2 } },
};
var baseHash = GatherRuleExecutor.ComputeCanonicalHash(baseline);
Assert.NotEqual(baseHash, GatherRuleExecutor.ComputeCanonicalHash(changedScalar));
Assert.NotEqual(baseHash, GatherRuleExecutor.ComputeCanonicalHash(changedNested));
}
[Fact]
public void CanonicalHash_DistinguishesTypeShapes()
{
// "1" (string) vs 1 (int) and null vs "<null>"-like strings must not collide.
Assert.NotEqual(
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = "1" }),
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = 1 }));
Assert.NotEqual(
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = null }),
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = "<null>" }));
}
// ── ExecuteRule wiring (async via startup path) ────────────────────
[Fact]
public void AlwaysMode_EmitsEveryCollection()
{
// EmitMode absent (legacy) — repeated identical results all emit. Two executions of
// the same collector via two rules sharing the target prove no dedup is applied.
var rule1 = OnChangeRule("GATHER-AL-1"); rule1.EmitMode = null; rule1.Trigger = "startup";
var rule2 = OnChangeRule("GATHER-AL-2"); rule2.EmitMode = null; rule2.Trigger = "startup";
_executor.UpdateRules(new List<GatherRule> { rule1, rule2 });
Assert.True(_executor.WaitForStartupRules(30));
Assert.True(WaitForEventCount(2));
}
[Fact]
public void OnChange_FirstInScopePoll_EmitsExistsFalse_AsPollingIndicator()
{
// The first result on an absent key IS the visible "we're polling now" indicator.
var rule = OnChangeRule();
rule.Trigger = "startup";
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.True(_executor.WaitForStartupRules(30));
Assert.True(WaitForEventCount(1));
var evt = SingleEvent();
Assert.Equal(false, evt.Data["exists"]);
Assert.Equal("GATHER-OC-001", evt.Data["ruleId"]); // injection happened after hashing
}
[Fact]
public void OnChange_ComposedWithEmitOnlyIfExists_EmptyResultDoesNotTouchHashState()
{
// emitOnlyIfExists miss → empty result → no emit AND no hash update: the eventual
// first real result must still count as "first emit".
var rule = OnChangeRule();
rule.Trigger = "startup";
rule.Parameters = new Dictionary<string, string> { ["emitOnlyIfExists"] = "true" };
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.True(_executor.WaitForStartupRules(30));
Thread.Sleep(300);
Assert.Equal(0, EventCount); // zero noise while the key is absent
// When the key appears (simulated via the state machine), the result emits as first.
Assert.True(_executor.ShouldEmitOnChange(rule, Result(("exists", true), ("Version", "1"))));
}
}
}
@@ -0,0 +1,326 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
{
/// <summary>
/// Phase scoping on <see cref="GatherRuleExecutor"/> (ActivePhases / ActiveFromPhase):
/// the scope matrix is asserted synchronously via the internal <c>IsRuleInScope</c>
/// (no timer races); the trigger-path wiring (deferred startup, phase_change / on_event
/// gating) is asserted through the public API with positive event waits.
/// Rules use the absent-registry-path pattern from
/// <see cref="RegistryCollectorEmitOnlyIfExistsTests"/> so no real system state is touched.
/// </summary>
[Collection("SerialThreading")] // deferred startup rules execute on the shared ThreadPool
public sealed class GatherRuleExecutorPhaseScopeTests : IDisposable
{
private readonly TempDirectory _tmp = new TempDirectory();
private readonly List<EnrollmentEvent> _events = new List<EnrollmentEvent>();
private readonly object _eventsGate = new object();
private readonly GatherRuleExecutor _executor;
private const string AbsentPath =
"HKLM\\SOFTWARE\\AutopilotMonitorTests\\DefinitelyAbsent_5b1c8d2e-0000-0000-0000-000000000000";
public GatherRuleExecutorPhaseScopeTests()
{
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_executor = new GatherRuleExecutor(
"sess", "tenant",
evt => { lock (_eventsGate) _events.Add(evt); },
logger)
{
// Bypass the registry allowlist guard for the synthetic test path.
UnrestrictedMode = true
};
}
public void Dispose()
{
_executor.Dispose();
_tmp.Dispose();
}
private int EventCount { get { lock (_eventsGate) return _events.Count; } }
private bool WaitForEventCount(int count, int timeoutMs = 10000)
{
var sw = Stopwatch.StartNew();
while (sw.ElapsedMilliseconds < timeoutMs)
{
if (EventCount >= count) return true;
Thread.Sleep(25);
}
return EventCount >= count;
}
/// <summary>Interval rule with a 1h period — the timer never ticks within a test run,
/// so these rules exist purely for synchronous scope-matrix assertions.</summary>
private static GatherRule ScopeProbeRule(string id, List<string> activePhases = null, string fromPhase = null)
=> new GatherRule
{
RuleId = id,
Title = id,
CollectorType = "registry",
Target = AbsentPath,
Trigger = "interval",
IntervalSeconds = 3600,
OutputEventType = "gather_test",
Enabled = true,
ActivePhases = activePhases,
ActiveFromPhase = fromPhase,
};
private static GatherRule StartupRule(string id, List<string> activePhases = null, string fromPhase = null)
=> new GatherRule
{
RuleId = id,
Title = id,
CollectorType = "registry",
Target = AbsentPath,
Trigger = "startup",
OutputEventType = "gather_test",
Enabled = true,
ActivePhases = activePhases,
ActiveFromPhase = fromPhase,
};
// ── Scope matrix (synchronous) ─────────────────────────────────────
[Fact]
public void UnscopedRule_IsInScope_EvenBeforeAnyPhaseSignal()
{
var rule = ScopeProbeRule("GATHER-SCOPE-000");
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.True(_executor.IsRuleInScope(rule));
}
[Fact]
public void ActivePhases_InactiveBeforeFirstPhaseSignal()
{
var rule = ScopeProbeRule("GATHER-SCOPE-001", activePhases: new List<string> { "AccountSetup" });
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.False(_executor.IsRuleInScope(rule));
}
[Fact]
public void ActivePhases_ActiveDuringListedPhase_InactiveAfterLeaving()
{
var rule = ScopeProbeRule("GATHER-SCOPE-002", activePhases: new List<string> { "AccountSetup" });
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Assert.False(_executor.IsRuleInScope(rule));
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(_executor.IsRuleInScope(rule));
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser);
Assert.False(_executor.IsRuleInScope(rule));
}
[Fact]
public void ActivePhases_MatchIsCaseInsensitive()
{
var rule = ScopeProbeRule("GATHER-SCOPE-003", activePhases: new List<string> { "accountsetup" });
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(_executor.IsRuleInScope(rule));
}
[Fact]
public void FromPhase_LatchesAtThreshold_AndStaysSticky_IncludingThroughFailed()
{
var rule = ScopeProbeRule("GATHER-SCOPE-010", fromPhase: "AccountSetup");
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Assert.False(_executor.IsRuleInScope(rule)); // threshold not reached
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(_executor.IsRuleInScope(rule)); // latched
_executor.OnPhaseChanged(EnrollmentPhase.Failed);
Assert.True(_executor.IsRuleInScope(rule)); // sticky through Failed
}
[Fact]
public void FromPhase_NoSpuriousActivationViaFailed()
{
// Failed=99 would ordinal-satisfy every threshold — it must never latch.
var rule = ScopeProbeRule("GATHER-SCOPE-011", fromPhase: "AccountSetup");
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.Failed);
Assert.False(_executor.IsRuleInScope(rule));
}
[Fact]
public void FromPhase_LatchesWhenThresholdIsSkippedOver()
{
// The session may never report the exact threshold phase — a later phase latches too.
var rule = ScopeProbeRule("GATHER-SCOPE-012", fromPhase: "AccountSetup");
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.FinalizingSetup);
Assert.True(_executor.IsRuleInScope(rule));
}
[Fact]
public void FromPhase_RuleDeliveredByRefreshAfterPhaseReached_LatchesImmediately()
{
_executor.UpdateRules(new List<GatherRule>());
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
// Config refresh delivers a new from-phase rule whose threshold already passed.
var rule = ScopeProbeRule("GATHER-SCOPE-013", fromPhase: "DeviceSetup");
_executor.UpdateRules(new List<GatherRule> { rule });
Assert.True(_executor.IsRuleInScope(rule));
}
[Fact]
public void BothFieldsSet_ActivePhasesWins()
{
// Backend validation rejects this shape; the agent defensively prefers ActivePhases.
var rule = ScopeProbeRule("GATHER-SCOPE-020",
activePhases: new List<string> { "DeviceSetup" }, fromPhase: "Start");
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.False(_executor.IsRuleInScope(rule)); // FromPhase(Start) would say yes — list wins
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Assert.True(_executor.IsRuleInScope(rule));
}
[Fact]
public void IgnorePhaseScope_BypassesScope_WithoutPhaseContext()
{
_executor.IgnorePhaseScope = true;
var during = ScopeProbeRule("GATHER-SCOPE-030", activePhases: new List<string> { "AccountSetup" });
var from = ScopeProbeRule("GATHER-SCOPE-031", fromPhase: "Complete");
_executor.UpdateRules(new List<GatherRule> { during, from });
Assert.True(_executor.IsRuleInScope(during));
Assert.True(_executor.IsRuleInScope(from));
}
// ── Trigger-path wiring (async, positive waits) ────────────────────
[Fact]
public void UnscopedStartupRule_RunsImmediately()
{
_executor.UpdateRules(new List<GatherRule> { StartupRule("GATHER-START-000") });
Assert.True(_executor.WaitForStartupRules(30));
Assert.True(WaitForEventCount(1));
}
[Fact]
public void ScopedStartupRule_IsDeferred_ThenFiresExactlyOnce_WhenScopeActivates()
{
_executor.UpdateRules(new List<GatherRule>
{
StartupRule("GATHER-START-001", fromPhase: "AccountSetup")
});
// Not in scope at UpdateRules: nothing queued, latch trivially satisfied.
Assert.True(_executor.WaitForStartupRules(30));
Assert.Equal(0, EventCount);
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Assert.Equal(0, EventCount);
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(WaitForEventCount(1), "deferred startup rule should fire when its scope activates");
// Later phase changes must not re-fire it (dedup via _startupRulesExecuted).
_executor.OnPhaseChanged(EnrollmentPhase.AppsUser);
_executor.OnPhaseChanged(EnrollmentPhase.FinalizingSetup);
Thread.Sleep(300);
Assert.Equal(1, EventCount);
}
[Fact]
public void PhaseChangeRule_OutOfScopePhase_DoesNotFire_InScopePhaseFires()
{
// TriggerPhase empty = fires on every phase change; scope narrows it to AccountSetup.
var rule = new GatherRule
{
RuleId = "GATHER-PC-001",
Title = "pc",
CollectorType = "registry",
Target = AbsentPath,
Trigger = "phase_change",
TriggerPhase = "",
OutputEventType = "gather_test",
Enabled = true,
ActivePhases = new List<string> { "AccountSetup" },
};
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnPhaseChanged(EnrollmentPhase.DeviceSetup);
Thread.Sleep(300);
Assert.Equal(0, EventCount);
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Assert.True(WaitForEventCount(1));
// Same phase again: the (rule, phase) dedup slot is now consumed.
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
Thread.Sleep(300);
Assert.Equal(1, EventCount);
}
[Fact]
public void OnEventRule_GatedByScope()
{
var rule = new GatherRule
{
RuleId = "GATHER-OE-001",
Title = "oe",
CollectorType = "registry",
Target = AbsentPath,
Trigger = "on_event",
TriggerEventType = "app_install_failed",
OutputEventType = "gather_test",
Enabled = true,
ActivePhases = new List<string> { "AccountSetup" },
};
_executor.UpdateRules(new List<GatherRule> { rule });
_executor.OnEvent("app_install_failed"); // no phase signal yet → out of scope
Thread.Sleep(300);
Assert.Equal(0, EventCount);
_executor.OnPhaseChanged(EnrollmentPhase.AccountSetup);
_executor.OnEvent("app_install_failed");
Assert.True(WaitForEventCount(1));
}
[Fact]
public void IgnorePhaseScope_ScopedStartupRule_RunsImmediately()
{
// The --run-gather-rules diagnostic path: scoped rules must execute unconditionally.
_executor.IgnorePhaseScope = true;
_executor.UpdateRules(new List<GatherRule>
{
StartupRule("GATHER-START-002", activePhases: new List<string> { "AccountSetup" })
});
Assert.True(_executor.WaitForStartupRules(30));
Assert.True(WaitForEventCount(1));
}
}
}
@@ -0,0 +1,195 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// Shell-Core 62407 "RebootCoalescing" branch — ESP-initiated coalesced reboot (MDM policies
/// forced a mid-ESP restart → second sign-in). Pure informational corroboration for the
/// MdmRebootPolicyTracker's per-URI 2800 attribution: no FinalizingSetup transition, no C#
/// event raise, and — as a DELIBERATE deviation from the other 62407 backfill branches — the
/// backfill path DOES emit (the live emit races the very reboot that kills the process).
/// </summary>
public sealed class ShellCoreTrackerRebootCoalescingTests
{
private static readonly DateTime ClockNow = new DateTime(2026, 7, 20, 10, 5, 0, DateTimeKind.Utc);
private const string CoalescingDescription =
"CloudExperienceHost Web App Event 2. Name: 'CommercialOOBE_DeviceSetup_RebootCoalescing', Value: '...'.";
private static (ShellCoreTracker tracker, FakeSignalIngressSink sink) MakeTracker(TempDirectory tmp)
{
var sink = new FakeSignalIngressSink();
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
var tracker = new ShellCoreTracker(
sessionId: "S1",
tenantId: "T1",
post: new InformationalEventPost(sink, new VirtualClock(ClockNow)),
logger: logger,
helloTracker: null);
return (tracker, sink);
}
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> ByType(FakeSignalIngressSink sink, string eventType) =>
sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == eventType).ToList();
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
[Fact]
public void ProcessEvent_62407_RebootCoalescing_EmitsWarning_ImmediateUpload()
{
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
var sourceTime = ClockNow.AddSeconds(-5);
tracker.ProcessEvent(
eventId: ShellCoreTracker.EventId_ShellCore_WebAppEvent,
description: CoalescingDescription,
timestamp: sourceTime,
providerName: "Microsoft-Windows-Shell-Core",
isBackfill: false);
var s = Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
Assert.Equal(sourceTime, s.OccurredAtUtc);
Assert.Equal(CoalescingDescription, Data(s)["description"]);
}
}
[Fact]
public void ProcessEvent_62407_RebootCoalescing_FiresOncePerRun()
{
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
CoalescingDescription, ClockNow, "Microsoft-Windows-Shell-Core", isBackfill: false);
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
CoalescingDescription, ClockNow.AddSeconds(1), "Microsoft-Windows-Shell-Core", isBackfill: false);
Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
}
}
[Fact]
public void ProcessEvent_62407_RebootCoalescing_DoesNotTriggerEspSemantics()
{
// The coalescing record is corroboration only — it must not trip the ESP-exit /
// FinalizingSetup / failure machinery.
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
var raised = new List<string>();
tracker.FinalizingSetupPhaseTriggered += (_, reason) => raised.Add($"finalizing:{reason}");
tracker.EspExited += (_, _) => raised.Add("espExited");
tracker.EspFailureDetected += (_, type) => raised.Add($"failure:{type}");
tracker.WhiteGloveCompleted += (_, _) => raised.Add("whiteGlove");
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
CoalescingDescription, ClockNow, "Microsoft-Windows-Shell-Core", isBackfill: false);
Assert.Empty(raised);
Assert.False(tracker.IsEspExitedForTest);
Assert.Empty(ByType(sink, Constants.EventTypes.EspExiting));
Assert.Empty(ByType(sink, Constants.EventTypes.EspFailure));
}
}
[Fact]
public void ProcessEvent_62407_ExistingBranches_Unaffected()
{
// Ordering regression guard: WhiteGlove/failure/exiting still classify as before —
// the RebootCoalescing check comes AFTER them.
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
"Name: 'CommercialOOBE_ESPProgress_Page_Exiting'", ClockNow, "p", isBackfill: false);
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
"Name: 'CommercialOOBE_ESPProgress_WhiteGlove_Success'", ClockNow, "p", isBackfill: false);
Assert.Single(ByType(sink, Constants.EventTypes.EspExiting));
Assert.Single(ByType(sink, Constants.EventTypes.WhiteGloveComplete));
Assert.Empty(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
}
}
[Fact]
public void Backfill_RebootCoalescing_Emits_WithBackfilledTrue_AndHistoricalTimestamp()
{
// Deliberate deviation from the other 62407 backfill branches: after the coalesced
// reboot restarts the agent, the backfill re-emission is the only reliable delivery.
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
var historical = ClockNow.AddMinutes(-3);
tracker.HandleBackfillRecord(
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, historical);
var s = Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
Assert.Equal(historical, s.OccurredAtUtc);
Assert.Equal(true, Data(s)["backfilled"]);
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
}
}
[Fact]
public void Backfill_RebootCoalescing_AfterLiveFire_DoesNotDoubleEmit_SameProcess()
{
// Same-process dedup via the fire-once flag. (Cross-process live+backfill duplication
// is possible and accepted — exists-semantics rule, identical historical timestamp.)
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
tracker.ProcessEvent(ShellCoreTracker.EventId_ShellCore_WebAppEvent,
CoalescingDescription, ClockNow, "p", isBackfill: false);
tracker.HandleBackfillRecord(
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, ClockNow.AddMinutes(-1));
Assert.Single(ByType(sink, Constants.EventTypes.EspRebootCoalescing));
}
}
[Fact]
public void Backfill_RebootCoalescing_DoesNotFallThroughToExitOrFailureHandling()
{
using var tmp = new TempDirectory();
var (tracker, sink) = MakeTracker(tmp);
using (tracker)
{
var raised = new List<string>();
tracker.FinalizingSetupPhaseTriggered += (_, reason) => raised.Add($"finalizing:{reason}");
tracker.EspExited += (_, _) => raised.Add("espExited");
tracker.EspFailureDetected += (_, type) => raised.Add($"failure:{type}");
tracker.HandleBackfillRecord(
ShellCoreTracker.EventId_ShellCore_WebAppEvent, CoalescingDescription, ClockNow.AddMinutes(-2));
Assert.Empty(raised);
Assert.False(tracker.IsEspExitedForTest);
}
}
}
}
@@ -0,0 +1,235 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// MdmRebootPolicyTracker — attributes the "unexpected reboot + second sign-in" pattern to the
/// device-assigned policy URIs that forced the coalesced reboot (DM-Enterprise EventID 2800).
/// Drives the primitive <see cref="MdmRebootPolicyTracker.ProcessEvent"/> test seam (no real
/// EventRecord, which is abstract + Windows-only), mirroring the WindowsUpdateTracker pattern.
/// The registry probe is always overridden — tests never touch the live registry.
/// </summary>
public sealed class MdmRebootPolicyTrackerTests : IDisposable
{
private static readonly DateTime At = new DateTime(2026, 7, 20, 9, 30, 0, DateTimeKind.Utc);
private const string SampleUri = "./Device/Vendor/MSFT/Policy/Config/DeviceGuard/EnableVirtualizationBasedSecurity";
private readonly TempDirectory _tmp = new TempDirectory();
private readonly FakeSignalIngressSink _sink;
private readonly MdmRebootPolicyTracker _tracker;
private long _nextRecordId = 1;
public MdmRebootPolicyTrackerTests()
{
_sink = new FakeSignalIngressSink();
var post = new InformationalEventPost(_sink, new VirtualClock(At));
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
_tracker = new MdmRebootPolicyTracker(
sessionId: "sess-mdm",
tenantId: "tenant-mdm",
post: post,
logger: logger,
backfillEnabled: false,
stateDirectory: null); // in-memory dedup for most tests
_tracker.RebootRequiredFlagProbe = () => null; // registry untouched by default
}
public void Dispose() => _tmp.Dispose();
private void Emit(string? rebootUri = SampleUri, string? description = null, long? recordId = null, bool isBackfill = false, DateTime? timeCreatedUtc = null)
{
_tracker.ProcessEvent(
eventId: MdmRebootPolicyTracker.EventId_PolicyRebootRequired,
recordId: recordId ?? _nextRecordId++,
timeCreatedUtc: timeCreatedUtc ?? At,
rebootUri: rebootUri,
formattedDescription: description,
isBackfill: isBackfill);
}
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> Emitted() =>
_sink.Posted.Where(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == Constants.EventTypes.MdmPolicyRebootRequired).ToList();
private static IReadOnlyDictionary<string, object> Data(FakeSignalIngressSink.PostedSignal s) =>
(IReadOnlyDictionary<string, object>)s.TypedPayload!;
[Fact]
public void ProcessEvent_EmitsWarning_WithUri_AndImmediateUpload()
{
Emit();
var s = Assert.Single(Emitted());
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
var data = Data(s);
Assert.Equal(SampleUri, data["rebootUri"]);
Assert.Equal(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, data["windowsEventId"]);
Assert.Equal(MdmRebootPolicyTracker.Channel, data["eventLogChannel"]);
}
[Fact]
public void ProcessEvent_NoUri_StillEmits_WithoutRebootUriField()
{
// Tolerant contract: an unparsed URI must stay timeline-visible (with the captured
// description telling us how to fix the extraction) — the analyze rule keys on
// "rebootUri exists" and simply stays silent.
Emit(rebootUri: null, description: "Some unexpected 2800 wording");
var s = Assert.Single(Emitted());
var data = Data(s);
Assert.False(data.ContainsKey("rebootUri"));
Assert.Equal("Some unexpected 2800 wording", data["description"]);
}
[Fact]
public void BackfilledEvent_UsesEventTimeForTimelineTimestamp_AndMarksBackfilled()
{
var eventTime = At.AddMinutes(-25); // policy applied well before agent start
Emit(isBackfill: true, timeCreatedUtc: eventTime);
var s = Assert.Single(Emitted());
Assert.Equal(eventTime, s.OccurredAtUtc);
Assert.Equal(true, Data(s)["backfilled"]);
}
[Fact]
public void SameRecordId_ProcessedTwice_EmitsOnce()
{
Emit(recordId: 42);
Emit(recordId: 42);
Assert.Single(Emitted());
}
[Fact]
public void BackfillEventWithLowerRecordId_AfterLiveEvent_IsStillEmitted()
{
// Same contract as WindowsUpdateTracker: the live watcher is armed BEFORE the backfill
// scan, so a live record with a higher RecordId can arrive first — the older,
// never-emitted backfill record must not be suppressed by a high-water mark.
Emit(recordId: 100);
Emit(recordId: 50, isBackfill: true);
Assert.Equal(2, Emitted().Count);
}
[Fact]
public void NegativeRecordId_IsNeverDeduped()
{
Emit(recordId: -1);
Emit(recordId: -1);
Assert.Equal(2, Emitted().Count);
}
[Fact]
public void Watermark_PersistsAcrossTrackerInstances()
{
// The coalesced reboot restarts the agent and the post-reboot backfill re-reads the
// same 2800 records — the persisted watermark must make that re-read a no-op.
var post = new InformationalEventPost(_sink, new VirtualClock(At));
var logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
var first = new MdmRebootPolicyTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
first.RebootRequiredFlagProbe = () => null;
first.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
timeCreatedUtc: At, rebootUri: SampleUri, formattedDescription: null, isBackfill: false);
var second = new MdmRebootPolicyTracker("s", "t", post, logger, backfillEnabled: false, stateDirectory: _tmp.Path);
second.RebootRequiredFlagProbe = () => null;
second.LoadWatermark();
second.ProcessEvent(MdmRebootPolicyTracker.EventId_PolicyRebootRequired, recordId: 500,
timeCreatedUtc: At, rebootUri: SampleUri, formattedDescription: null, isBackfill: true);
Assert.Single(Emitted());
}
[Fact]
public void RebootRequiredFlagProbe_True_AddsField_Null_OmitsIt()
{
_tracker.RebootRequiredFlagProbe = () => true;
Emit(recordId: 1);
_tracker.RebootRequiredFlagProbe = () => null;
Emit(recordId: 2);
var emitted = Emitted();
Assert.Equal(2, emitted.Count);
Assert.Equal(true, Data(emitted[0])["omadmRebootRequiredFlag"]);
Assert.False(Data(emitted[1]).ContainsKey("omadmRebootRequiredFlag"));
}
[Fact]
public void RebootRequiredFlagProbe_Throwing_IsFailSoft()
{
_tracker.RebootRequiredFlagProbe = () => throw new InvalidOperationException("boom");
Emit();
var s = Assert.Single(Emitted());
Assert.False(Data(s).ContainsKey("omadmRebootRequiredFlag"));
}
[Fact]
public void BuildXPath_Targets2800()
{
Assert.Equal("*[System[(EventID=2800)]]", MdmRebootPolicyTracker.BuildXPath());
}
// ------------------------------------------------------------------ URI extraction
[Fact]
public void ExtractRebootUri_PrefersEventDataValue_OverDescription()
{
var eventData = new Dictionary<string, string> { ["Data0"] = " " + SampleUri + " " };
var uri = MdmRebootPolicyTracker.ExtractRebootUri(eventData, "unrelated ./Other/Uri text");
Assert.Equal(SampleUri, uri);
}
[Fact]
public void ExtractRebootUri_RegexFallback_FromDescription()
{
var uri = MdmRebootPolicyTracker.ExtractRebootUri(
new Dictionary<string, string> { ["Data0"] = "not a uri" },
$"The following URI has triggered a reboot: {SampleUri}.");
Assert.Equal(SampleUri, uri);
}
[Fact]
public void ExtractRebootUri_ReturnsNull_WhenNothingUriLike()
{
Assert.Null(MdmRebootPolicyTracker.ExtractRebootUri(
new Dictionary<string, string> { ["Data0"] = "plain" }, "no uri here"));
Assert.Null(MdmRebootPolicyTracker.ExtractRebootUri(null, null));
}
[Fact]
public void ParseEventData_CollectsNamedAndPositionalData()
{
const string xml =
"<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>" +
"<System><EventID>2800</EventID></System>" +
"<EventData><Data Name='Uri'>./Device/Vendor/MSFT/X</Data><Data>positional</Data></EventData>" +
"</Event>";
var parsed = MdmRebootPolicyTracker.ParseEventData(xml);
Assert.Equal("./Device/Vendor/MSFT/X", parsed["Uri"]);
Assert.Equal("positional", parsed["Data0"]);
}
}
}
@@ -0,0 +1,519 @@
using System;
using System.Collections.Generic;
using System.Diagnostics.Eventing.Reader;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using System.Xml.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using Newtonsoft.Json;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
{
/// <summary>
/// Watches <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> for
/// EventID 2800 — logged once per device-assigned MDM policy whose CSP URI matches the OS
/// reboot-required catalog — and forwards each as a <c>mdm_policy_reboot_required</c> event
/// carrying the reboot-forcing URI.
/// <para>
/// This attributes the "unexpected reboot + second sign-in screen" pattern (PMPC research):
/// device-assigned policies applied during ESP DeviceSetup can demand a reboot; Windows
/// coalesces them into one restart at the end of DeviceSetup, forcing the user through a second
/// sign-in before AccountSetup. Admins are blind to this in the Intune console — the attributed
/// URIs make it actionable (reassign the profiles to user groups and the reboot disappears).
/// </para>
/// <para>
/// <b>Backfill.</b> Device policies can apply before the agent starts, so a startup backfill
/// scans the lookback window. The .evtx persists across the coalesced reboot, so the
/// post-reboot agent restart re-reads the same records — which is why the
/// <b>cross-restart RecordId watermark</b> (WindowsUpdateTracker pattern) is mandatory here:
/// without it every reboot-forcing URI would be re-emitted after the very reboot it caused.
/// </para>
/// <para>
/// <b>Tolerant matching.</b> The exact 2800 message text is unverified (validated post-deploy on
/// the E2E VM): emission is gated on the EventID only, never on message wording. URI extraction
/// prefers structured EventData values that look like an OMA-DM URI (<c>./…</c>) and falls back
/// to a permissive regex over the formatted description. Events with an unparseable URI are
/// still emitted (with the captured description) so the timeline shows them and the captured
/// text tells us how to fix the extraction — the analyze rule keys on <c>rebootUri exists</c>
/// and simply stays silent for those.
/// </para>
/// </summary>
internal sealed class MdmRebootPolicyTracker : IDisposable
{
internal const string Channel = "Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin";
internal const int EventId_PolicyRebootRequired = 2800;
internal const string WatermarkStateFileName = "mdm-reboot-watermark.json";
/// <summary>Registry key whose <c>RebootRequired</c> value/subkey flags a pending OMA-DM coalesced reboot.</summary>
internal const string OmadmSyncMlKeyPath = @"SOFTWARE\Microsoft\Provisioning\OMADM\SyncML";
// Permissive OMA-DM URI shape: "./Device/Vendor/MSFT/...", "./User/...", "./Vendor/...".
// Kept deliberately broad — the exact 2800 text is unverified; post-deploy E2E validates
// and tightens this against real records if needed.
private static readonly Regex UriPattern = new Regex(
@"\./[A-Za-z0-9_\-./\[\]{}%~+]+", RegexOptions.Compiled);
private readonly AgentLogger _logger;
private readonly string _sessionId;
private readonly string _tenantId;
private readonly InformationalEventPost _post;
private readonly bool _backfillEnabled;
private readonly int _backfillLookbackMinutes;
private readonly string _stateDirectory;
private EventLogWatcher _watcher;
// Dedup — same contract as WindowsUpdateTracker (see its field docs): immutable
// cross-restart boundary from disk + intra-run HashSet (NOT a high-water mark, because the
// live watcher is armed before the backfill scan and can deliver higher RecordIds first).
private long _restartWatermark = -1;
private readonly HashSet<long> _seenThisRun = new HashSet<long>();
private long _maxEmittedRecordId = -1;
private readonly object _watermarkLock = new object();
/// <summary>
/// Fail-soft corroboration probe for the OMA-DM RebootRequired flag — null = unknown
/// (key absent or unreadable). Settable test seam; the default reads the live registry.
/// </summary>
internal Func<bool?> RebootRequiredFlagProbe { get; set; } = ReadOmadmRebootRequiredFlag;
public MdmRebootPolicyTracker(
string sessionId,
string tenantId,
InformationalEventPost post,
AgentLogger logger,
bool backfillEnabled = true,
int backfillLookbackMinutes = 60,
string stateDirectory = null)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_backfillEnabled = backfillEnabled;
_backfillLookbackMinutes = backfillLookbackMinutes;
_stateDirectory = stateDirectory != null ? Environment.ExpandEnvironmentVariables(stateDirectory) : null;
}
public void Start()
{
LoadWatermark();
StartWatcher();
if (_backfillEnabled && _backfillLookbackMinutes > 0)
BackfillRecentEvents();
else
_logger.Info("MdmRebootPolicy backfill disabled by config");
}
public void Stop()
{
if (_watcher == null) return;
try
{
_watcher.Enabled = false;
_watcher.Dispose();
_logger.Info("MdmRebootPolicy watcher stopped");
}
catch (Exception ex)
{
_logger.Error("Error stopping MdmRebootPolicy watcher", ex);
}
finally
{
_watcher = null;
}
}
public void Dispose() => Stop();
// -----------------------------------------------------------------------
// Watcher lifecycle
// -----------------------------------------------------------------------
private void StartWatcher()
{
try
{
var query = new EventLogQuery(Channel, PathType.LogName, BuildXPath());
_watcher = new EventLogWatcher(query);
_watcher.EventRecordWritten += OnEventRecordWritten;
_watcher.Enabled = true;
_logger.Info($"MdmRebootPolicy watcher started: {Channel} (EventID {EventId_PolicyRebootRequired})");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"MdmRebootPolicy event log not found: {Channel} (normal on non-MDM-enrolled test environments)");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"MdmRebootPolicy watcher access denied for {Channel}: {ex.Message}");
}
catch (Exception ex)
{
_logger.Error($"Failed to start MdmRebootPolicy watcher for {Channel}", ex);
// MON-D1: surface a dead watcher as one-shot telemetry, not just a local log line.
CollectorDegradationReporter.Report(_post, _sessionId, _tenantId,
collectorName: "MdmRebootPolicyTracker", reason: $"watcher_arm_failed:{Channel}", ex: ex);
}
}
/// <summary>Targeted-EventID XPath filter. Exposed as internal for tests.</summary>
internal static string BuildXPath() =>
$"*[System[(EventID={EventId_PolicyRebootRequired})]]";
// -----------------------------------------------------------------------
// Event processing
// -----------------------------------------------------------------------
private void OnEventRecordWritten(object sender, EventRecordWrittenEventArgs e)
{
if (e.EventRecord == null) return;
try
{
ProcessRecord(e.EventRecord, isBackfill: false);
}
catch (Exception ex)
{
_logger.Error("Error processing MdmRebootPolicy event", ex);
}
}
private void ProcessRecord(EventRecord record, bool isBackfill)
{
var recordId = record.RecordId ?? -1;
// Cheap dedup gate BEFORE the expensive XML/FormatDescription work.
if (IsAlreadyProcessed(recordId))
return;
string xml = null;
try { xml = record.ToXml(); }
catch { /* fall back to description-only parse below */ }
string description = null;
try { description = record.FormatDescription(); }
catch { /* some events lack formatting resources */ }
var rebootUri = ExtractRebootUri(ParseEventData(xml), description);
ProcessEvent(
eventId: record.Id,
recordId: recordId,
timeCreatedUtc: record.TimeCreated?.ToUniversalTime(),
rebootUri: rebootUri,
formattedDescription: description,
isBackfill: isBackfill);
}
/// <summary>
/// Core processing extracted to primitive inputs so tests can drive it without synthesizing
/// an abstract, Windows-only <see cref="EventRecord"/>. Mirrors the WindowsUpdateTracker
/// test-seam pattern.
/// </summary>
internal void ProcessEvent(
int eventId,
long recordId,
DateTime? timeCreatedUtc,
string rebootUri,
string formattedDescription,
bool isBackfill)
{
if (!MarkProcessed(recordId))
return; // already emitted (cross-restart or duplicate delivery)
var data = new Dictionary<string, object>
{
{ "windowsEventId", eventId },
{ "backfilled", isBackfill },
{ "eventLogChannel", Channel },
};
if (recordId >= 0) data["recordId"] = recordId;
if (timeCreatedUtc.HasValue) data["eventTime"] = timeCreatedUtc.Value.ToString("o");
if (!string.IsNullOrEmpty(rebootUri)) data["rebootUri"] = rebootUri;
if (!string.IsNullOrEmpty(formattedDescription))
{
data["description"] = formattedDescription.Length > 1000
? formattedDescription.Substring(0, 1000) + "…"
: formattedDescription;
}
var flag = SafeProbeRebootRequiredFlag();
if (flag.HasValue) data["omadmRebootRequiredFlag"] = flag.Value;
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
// The record's own time, not UtcNow: backfilled pre-agent (and post-reboot re-read)
// records must land on the timeline at when the policy actually demanded the reboot.
Timestamp = timeCreatedUtc ?? DateTime.UtcNow,
EventType = Constants.EventTypes.MdmPolicyRebootRequired,
Severity = EventSeverity.Warning,
Source = "MdmRebootPolicyWatcher",
Phase = EnrollmentPhase.Unknown,
Message = !string.IsNullOrEmpty(rebootUri)
? $"MDM policy requires a reboot during ESP: {rebootUri}"
: "MDM policy requires a reboot during ESP (URI not parsed from event — see description)",
Data = data,
// The reboot this event announces will kill the agent — flush now.
ImmediateUpload = true,
});
}
/// <summary>
/// Extracts the reboot-forcing OMA-DM URI. Preference order: (1) a structured EventData
/// value that starts with <c>./</c> (the CSP-URI shape), (2) a permissive regex over the
/// formatted description. Returns null when nothing URI-like is found — the caller still
/// emits, just without the <c>rebootUri</c> field.
/// </summary>
internal static string ExtractRebootUri(Dictionary<string, string> eventData, string description)
{
if (eventData != null)
{
foreach (var value in eventData.Values)
{
var trimmed = value?.Trim();
if (!string.IsNullOrEmpty(trimmed) && trimmed.StartsWith("./", StringComparison.Ordinal))
return trimmed;
}
}
if (!string.IsNullOrEmpty(description))
{
var match = UriPattern.Match(description);
if (match.Success)
{
// The permissive pattern can swallow sentence punctuation trailing the URI
// ("...Security."). OMA-DM URI segments never end in punctuation — trim it.
var candidate = match.Value.TrimEnd('.', ',', ';', ':', '\'', '"', ')');
if (candidate.Length > 2) return candidate;
}
}
return null;
}
/// <summary>
/// Extracts <c>&lt;Data Name="..."&gt;value&lt;/Data&gt;</c> pairs from an event's rendered
/// XML, keyed case-insensitively. Also collects UNNAMED Data elements under positional keys
/// (<c>Data0</c>, <c>Data1</c>, …) — DM-Enterprise events frequently carry positional-only
/// payloads. Namespace-agnostic; returns an empty map on null/malformed XML — never throws.
/// </summary>
internal static Dictionary<string, string> ParseEventData(string xml)
{
var result = new Dictionary<string, string>(StringComparer.OrdinalIgnoreCase);
if (string.IsNullOrEmpty(xml)) return result;
try
{
var doc = XDocument.Parse(xml);
var positionalIndex = 0;
foreach (var el in doc.Descendants().Where(e => e.Name.LocalName == "Data"))
{
var nameAttr = el.Attribute("Name");
var key = nameAttr != null && !string.IsNullOrEmpty(nameAttr.Value)
? nameAttr.Value
: "Data" + positionalIndex++;
if (!result.ContainsKey(key))
result[key] = el.Value;
}
}
catch
{
// Malformed / unexpected XML — best effort, return what we have.
}
return result;
}
// -----------------------------------------------------------------------
// OMA-DM RebootRequired flag probe (fail-soft corroboration)
// -----------------------------------------------------------------------
private bool? SafeProbeRebootRequiredFlag()
{
try
{
return RebootRequiredFlagProbe?.Invoke();
}
catch
{
return null; // corroboration only — never let the probe break the emit
}
}
/// <summary>
/// Reads the pending-reboot flag under <see cref="OmadmSyncMlKeyPath"/>. The exact shape is
/// unverified (PMPC research names the key, not the value type), so both a value and a
/// subkey named <c>RebootRequired</c> count as "pending". Fail-soft: null = unknown
/// (key absent / unreadable), matching the EspSkipConfigurationProbe convention.
/// </summary>
internal static bool? ReadOmadmRebootRequiredFlag()
{
try
{
using (var key = Microsoft.Win32.Registry.LocalMachine.OpenSubKey(OmadmSyncMlKeyPath))
{
if (key == null) return null;
if (key.GetValue("RebootRequired") != null) return true;
using (var sub = key.OpenSubKey("RebootRequired"))
{
return sub != null ? (bool?)true : false;
}
}
}
catch
{
return null;
}
}
// -----------------------------------------------------------------------
// Backfill
// -----------------------------------------------------------------------
private void BackfillRecentEvents()
{
try
{
var lookbackMs = (long)_backfillLookbackMinutes * 60 * 1000;
var xpath = $"*[System[(EventID={EventId_PolicyRebootRequired}) and TimeCreated[timediff(@SystemTime) <= {lookbackMs}]]]";
_logger.Info($"MdmRebootPolicy backfill: scanning {Channel} " +
$"(lookback={_backfillLookbackMinutes}min, restartWatermark={_restartWatermark})");
var query = new EventLogQuery(Channel, PathType.LogName, xpath);
int processed = 0;
using (var reader = new EventLogReader(query))
{
EventRecord record;
while ((record = reader.ReadEvent()) != null)
{
using (record)
{
ProcessRecord(record, isBackfill: true);
processed++;
}
}
}
if (processed == 0)
_logger.Debug($"MdmRebootPolicy backfill: no EventID {EventId_PolicyRebootRequired} records in last {_backfillLookbackMinutes} minutes");
else
_logger.Info($"MdmRebootPolicy backfill: scanned {processed} record(s)");
}
catch (EventLogNotFoundException)
{
_logger.Warning($"MdmRebootPolicy event log not found during backfill: {Channel} (normal on non-MDM-enrolled test environments)");
}
catch (UnauthorizedAccessException ex)
{
_logger.Warning($"MdmRebootPolicy backfill access denied: {ex.Message}");
}
catch (Exception ex)
{
_logger.Warning($"MdmRebootPolicy backfill failed: {ex.Message}");
}
}
// -----------------------------------------------------------------------
// Watermark dedup (cross-restart) — WindowsUpdateTracker pattern verbatim
// -----------------------------------------------------------------------
private bool IsAlreadyProcessed(long recordId)
{
if (recordId < 0) return false; // no RecordId → cannot dedup, let it through
lock (_watermarkLock)
{
return recordId <= _restartWatermark || _seenThisRun.Contains(recordId);
}
}
/// <summary>
/// Atomically claims <paramref name="recordId"/>. Returns true if this is the first time we
/// see it (caller should emit), false if it was already processed — either in a PRIOR run
/// (at/below the restart watermark) or already this run (in the seen-set). Records without
/// a RecordId (-1) are always emitted, never tracked or persisted.
/// </summary>
private bool MarkProcessed(long recordId)
{
if (recordId < 0) return true;
long toPersist = -1;
lock (_watermarkLock)
{
if (recordId <= _restartWatermark || !_seenThisRun.Add(recordId))
return false;
if (recordId > _maxEmittedRecordId)
{
_maxEmittedRecordId = recordId;
toPersist = _maxEmittedRecordId;
}
}
if (toPersist >= 0) PersistWatermark(toPersist);
return true;
}
internal void LoadWatermark()
{
if (string.IsNullOrEmpty(_stateDirectory)) return;
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
if (!File.Exists(filePath)) return;
try
{
var json = File.ReadAllText(filePath);
var state = JsonConvert.DeserializeObject<WatermarkState>(json);
if (state != null)
{
lock (_watermarkLock)
{
_restartWatermark = state.LastRecordId;
_maxEmittedRecordId = state.LastRecordId;
}
_logger.Info($"MdmRebootPolicy watermark loaded: lastRecordId={state.LastRecordId} (persisted {state.PersistedUtc:O})");
}
}
catch (Exception ex)
{
_logger.Warning($"Failed to load MdmRebootPolicy watermark: {ex.Message}");
}
}
private void PersistWatermark(long recordId)
{
if (string.IsNullOrEmpty(_stateDirectory)) return;
try
{
Directory.CreateDirectory(_stateDirectory);
var filePath = Path.Combine(_stateDirectory, WatermarkStateFileName);
var state = new WatermarkState { LastRecordId = recordId, PersistedUtc = DateTime.UtcNow };
var json = JsonConvert.SerializeObject(state, Formatting.Indented);
var tempPath = filePath + ".tmp";
File.WriteAllText(tempPath, json);
File.Copy(tempPath, filePath, overwrite: true);
try { File.Delete(tempPath); } catch { }
}
catch (Exception ex)
{
_logger.Warning($"Failed to persist MdmRebootPolicy watermark: {ex.Message}");
}
}
internal sealed class WatermarkState
{
public long LastRecordId { get; set; }
public DateTime PersistedUtc { get; set; }
}
}
}
@@ -16,6 +16,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// CommercialOOBE_ESPProgress_Page_Exiting — normal ESP exit
/// CommercialOOBE_ESPProgress_WhiteGlove_Success — WhiteGlove complete
/// CommercialOOBE_ESPProgress_Failure/_Timeout/_Abort/WhiteGlove_Failed — ESP failure
/// "RebootCoalescing" (DeviceSetup) — ESP-initiated coalesced reboot
/// (MDM policies forced a mid-ESP restart → second sign-in; corroborates the
/// MdmRebootPolicyTracker's per-URI 2800 attribution events)
///
/// Raises <see cref="FinalizingSetupPhaseTriggered"/>, <see cref="WhiteGloveCompleted"/>,
/// and <see cref="EspFailureDetected"/>. Cross-notifies the <see cref="HelloTracker"/>
@@ -47,6 +50,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
private bool _espExitDetected;
private bool _whiteGloveDetected;
private bool _helloWizardStartDetected;
private bool _rebootCoalescingDetected;
private readonly object _stateLock = new object();
/// <summary>
@@ -247,6 +251,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_logger.Info("ESP phase exit detected - detected via Shell-Core event 62407");
}
else if (description.IndexOf("RebootCoalescing", StringComparison.OrdinalIgnoreCase) >= 0)
{
// ESP-initiated coalesced reboot (end of DeviceSetup): MDM policies forced a
// restart, the user will see a second sign-in screen. Checked AFTER the
// WhiteGlove/failure/exiting patterns so their semantics stay untouched even
// if a description ever carried both tokens. Fire-once per process run — the
// reboot restarts the agent, and the fresh run re-observes it via backfill.
lock (_stateLock)
{
if (_rebootCoalescingDetected) return;
_rebootCoalescingDetected = true;
}
eventType = Constants.EventTypes.EspRebootCoalescing;
severity = EventSeverity.Warning;
message = "ESP initiated a coalesced reboot (RebootCoalescing) — device will restart and show a second sign-in";
// No FinalizingSetup transition, no C# event raise — pure informational
// corroboration for the MdmRebootPolicyTracker's per-URI attribution.
_logger.Warning("ESP coalesced reboot detected via Shell-Core event 62407 (RebootCoalescing)");
}
else
{
return;
@@ -421,6 +446,29 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
return;
}
if (description.IndexOf("RebootCoalescing", StringComparison.OrdinalIgnoreCase) >= 0)
{
// DELIBERATE deviation from the other 62407 backfill branches (which only raise C#
// events): this is a pure informational type with no reducer/subscriber — the
// emission IS the signal, and the live emit races the very reboot that kills the
// process. Re-emitting after the restart is the only reliable delivery path. A
// live+backfill cross-process duplicate is possible (no watermark here); accepted:
// the analyze rule uses exists-semantics, both copies carry the identical historical
// timestamp, and the backfill copy is marked backfilled:true.
bool shouldEmit;
lock (_stateLock)
{
shouldEmit = !_rebootCoalescingDetected;
_rebootCoalescingDetected = true;
}
if (shouldEmit)
{
_logger.Info($"Backfill: ESP coalesced reboot (RebootCoalescing) found in recent Shell-Core logs (originalAt={occurredAtUtc:o})");
EmitRebootCoalescing(description, occurredAtUtc, isBackfill: true);
}
return; // a RebootCoalescing record is not an ESP exit/failure — stop here
}
if (EspExitingPattern.IsMatch(description))
{
bool shouldNotify = false;
@@ -459,6 +507,39 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
}
}
/// <summary>
/// Emits the <c>esp_reboot_coalescing</c> event with the standard Shell-Core eventData
/// shape. Used by the backfill path (the live path flows through the shared emit tail in
/// <see cref="ProcessEvent"/>); <paramref name="isBackfill"/> adds the
/// <c>backfilled</c> marker that distinguishes an accepted post-restart re-emission.
/// </summary>
private void EmitRebootCoalescing(string description, DateTime occurredAtUtc, bool isBackfill)
{
var eventData = new Dictionary<string, object>
{
{ "windowsEventId", EventId_ShellCore_WebAppEvent },
{ "providerName", "" },
{ "description", description },
{ "eventLogChannel", ShellCoreEventLogChannel },
{ "eventTime", occurredAtUtc.ToString("o") },
{ "backfilled", isBackfill }
};
_post.Emit(new EnrollmentEvent
{
SessionId = _sessionId,
TenantId = _tenantId,
Timestamp = occurredAtUtc,
EventType = Constants.EventTypes.EspRebootCoalescing,
Severity = EventSeverity.Warning,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = "ESP initiated a coalesced reboot (RebootCoalescing) — device will restart and show a second sign-in",
Data = eventData,
ImmediateUpload = true
});
}
// =====================================================================
// Pattern helpers
// =====================================================================
@@ -1,6 +1,9 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using System.Security.Cryptography;
using System.Text;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors;
@@ -28,6 +31,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
private readonly HashSet<string> _phaseRulesExecuted = new HashSet<string>();
private CountdownEvent _startupRulesLatch; // non-null only while startup rules are pending
// Phase-scope + emit-mode state (ActivePhases / ActiveFromPhase / EmitMode on GatherRule).
// Guarded by _scopeLock; collector work always runs outside the lock. The state deliberately
// survives UpdateRules config refreshes — an agent restart causes at most one re-emit per
// on_change rule (acceptable under the per-enrollment lifecycle).
private readonly object _scopeLock = new object();
private EnrollmentPhase _currentPhase = EnrollmentPhase.Unknown;
private readonly HashSet<string> _fromPhaseLatched = new HashSet<string>();
private readonly Dictionary<string, string> _lastEmittedHash = new Dictionary<string, string>();
private readonly Dictionary<string, (int Count, DateTime SinceUtc)> _suppressed =
new Dictionary<string, (int Count, DateTime SinceUtc)>();
private readonly HashSet<string> _scopeBypassLogged = new HashSet<string>();
/// <summary>
/// When true, phase scoping (ActivePhases / ActiveFromPhase) is bypassed and scoped rules
/// execute unconditionally. Used by the standalone <c>--run-gather-rules</c> diagnostic
/// mode, which has no enrollment phase context. EmitMode is unaffected.
/// </summary>
public bool IgnorePhaseScope { get; set; }
/// <summary>
/// When true, guardrails are relaxed: all registry, WMI, and command targets are allowed.
/// File paths allow everything except C:\Users. Set from tenant configuration.
@@ -89,10 +111,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
_activeRules = rules.Where(r => r.Enabled).ToList();
// Execute startup rules — track completion via CountdownEvent so callers can wait
var pendingStartup = _activeRules
.Where(r => r.Trigger == "startup" && !_startupRulesExecuted.Contains(r.RuleId))
.ToList();
// Execute startup rules — track completion via CountdownEvent so callers can wait.
// Scoped startup rules that are not yet in scope are deferred: they run once from
// OnPhaseChanged when their scope activates (same _startupRulesExecuted dedup — a
// rule runs either here or there, never both). Unscoped rules behave as before.
List<GatherRule> pendingStartup;
lock (_scopeLock)
{
// A config refresh may deliver new from-phase rules after their phase was
// already reached — evaluate latches against the current phase first.
EvaluateFromPhaseLatchesLocked();
pendingStartup = _activeRules
.Where(r => r.Trigger == "startup" && !_startupRulesExecuted.Contains(r.RuleId))
.Where(r => IsRuleInScopeLocked(r))
.ToList();
foreach (var rule in pendingStartup)
{
_startupRulesExecuted.Add(rule.RuleId);
}
}
if (pendingStartup.Count > 0)
{
@@ -101,7 +140,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
foreach (var rule in pendingStartup)
{
_startupRulesExecuted.Add(rule.RuleId);
ThreadPool.QueueUserWorkItem(_ =>
{
try { ExecuteRule(rule); }
@@ -110,18 +148,26 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
}
}
// Set up interval timers
// Set up interval timers. Timers for scoped rules keep running while out of scope —
// the tick gate makes an out-of-scope tick a free no-op, avoiding start/stop
// lifecycle races on phase transitions.
foreach (var rule in _activeRules.Where(r => r.Trigger == "interval" && r.IntervalSeconds.HasValue))
{
var interval = TimeSpan.FromSeconds(rule.IntervalSeconds.Value);
var intervalRule = rule;
var interval = TimeSpan.FromSeconds(intervalRule.IntervalSeconds.Value);
var timer = new Timer(
_ => ExecuteRule(rule),
_ =>
{
if (!IsRuleInScope(intervalRule))
return;
ExecuteRule(intervalRule);
},
null,
interval, // Initial delay = one interval
interval
);
_intervalTimers[rule.RuleId] = timer;
_logger.Info($" Interval rule {rule.RuleId} scheduled every {rule.IntervalSeconds}s");
_intervalTimers[intervalRule.RuleId] = timer;
_logger.Info($" Interval rule {intervalRule.RuleId} scheduled every {intervalRule.IntervalSeconds}s");
}
_logger.Info($"GatherRuleExecutor: {_activeRules.Count(r => r.Trigger == "startup")} startup, " +
@@ -135,11 +181,43 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
{
var phaseName = newPhase.ToString();
// Update the phase + from-phase latches FIRST so deferred startup activation and
// the phase_change rules below all evaluate against the NEW phase.
List<GatherRule> deferredStartup;
lock (_scopeLock)
{
_currentPhase = newPhase;
EvaluateFromPhaseLatchesLocked();
deferredStartup = _activeRules
.Where(r => r.Trigger == "startup" && HasPhaseScope(r)
&& !_startupRulesExecuted.Contains(r.RuleId)
&& IsRuleInScopeLocked(r))
.ToList();
foreach (var rule in deferredStartup)
{
_startupRulesExecuted.Add(rule.RuleId);
}
}
foreach (var rule in deferredStartup)
{
var deferredRule = rule;
_logger.Info($"Phase {phaseName} activated deferred startup rule {deferredRule.RuleId}");
ThreadPool.QueueUserWorkItem(_ => ExecuteRule(deferredRule));
}
foreach (var rule in _activeRules.Where(r => r.Trigger == "phase_change"))
{
if (string.IsNullOrEmpty(rule.TriggerPhase) ||
string.Equals(rule.TriggerPhase, phaseName, StringComparison.OrdinalIgnoreCase))
{
// Scope gate BEFORE the dedup add — an out-of-scope pass must not consume
// the once-per-(rule, phase) execution slot.
if (!IsRuleInScope(rule))
continue;
// Deduplicate: only fire once per (ruleId, phase) combination
var deduplicationKey = $"{rule.RuleId}|{phaseName}";
if (!_phaseRulesExecuted.Add(deduplicationKey))
@@ -163,6 +241,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
{
if (string.Equals(rule.TriggerEventType, eventType, StringComparison.OrdinalIgnoreCase))
{
if (!IsRuleInScope(rule))
continue;
_logger.Info($"Event triggered rule {rule.RuleId} (event: {eventType})");
ThreadPool.QueueUserWorkItem(_ => ExecuteRule(rule));
}
@@ -192,6 +273,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
if (result.Count > 0)
{
// EmitMode "on_change": hash the raw collector result (before ruleId/title
// injection) and only emit when it differs from the last emitted one. An
// empty result never reaches this point, so an emitOnlyIfExists miss neither
// emits nor updates the hash — composing to "one event on appearance, then
// only on change".
if (IsOnChangeMode(rule) && !ShouldEmitOnChange(rule, result))
return;
result["ruleId"] = rule.RuleId;
result["ruleTitle"] = rule.Title;
@@ -229,6 +318,199 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
}
}
// ===== Phase scope (ActivePhases / ActiveFromPhase) =====
private static bool HasPhaseScope(GatherRule rule)
{
return (rule.ActivePhases != null && rule.ActivePhases.Count > 0)
|| !string.IsNullOrEmpty(rule.ActiveFromPhase);
}
// Internal so the scope matrix is testable synchronously (no timer/ThreadPool races).
internal bool IsRuleInScope(GatherRule rule)
{
lock (_scopeLock)
{
return IsRuleInScopeLocked(rule);
}
}
private bool IsRuleInScopeLocked(GatherRule rule)
{
if (!HasPhaseScope(rule))
return true; // unscoped — legacy behavior
if (IgnorePhaseScope)
{
if (_scopeBypassLogged.Add(rule.RuleId))
_logger.Info($"Gather rule {rule.RuleId}: phase scope ignored (diagnostic mode)");
return true;
}
// Both fields set is rejected by backend validation; defensively prefer ActivePhases.
if (rule.ActivePhases != null && rule.ActivePhases.Count > 0)
{
// No phase signal yet (or scope tokens that never match, e.g. "Unknown"):
// scoped rules stay inactive.
if (_currentPhase == EnrollmentPhase.Unknown)
return false;
var phaseName = _currentPhase.ToString();
foreach (var phase in rule.ActivePhases)
{
if (string.Equals(phase, phaseName, StringComparison.OrdinalIgnoreCase))
return true;
}
return false;
}
return _fromPhaseLatched.Contains(rule.RuleId);
}
/// <summary>
/// Latches every from-phase rule whose threshold the current phase has reached.
/// Never latches on Unknown/Failed (Failed=99 would ordinal-satisfy every threshold);
/// once latched a rule stays active for the session — including through Failed.
/// Caller must hold <see cref="_scopeLock"/>.
/// </summary>
private void EvaluateFromPhaseLatchesLocked()
{
if (_currentPhase == EnrollmentPhase.Unknown || _currentPhase == EnrollmentPhase.Failed)
return;
foreach (var rule in _activeRules)
{
if (string.IsNullOrEmpty(rule.ActiveFromPhase)) continue;
if (rule.ActivePhases != null && rule.ActivePhases.Count > 0) continue; // ActivePhases wins
if (_fromPhaseLatched.Contains(rule.RuleId)) continue;
if (Enum.TryParse<EnrollmentPhase>(rule.ActiveFromPhase, ignoreCase: true, out var fromPhase)
&& fromPhase != EnrollmentPhase.Unknown
&& fromPhase != EnrollmentPhase.Failed
&& (int)_currentPhase >= (int)fromPhase)
{
_fromPhaseLatched.Add(rule.RuleId);
_logger.Info($"Gather rule {rule.RuleId}: from-phase scope latched at {_currentPhase} (activeFromPhase={rule.ActiveFromPhase})");
}
}
}
// ===== Emit mode (on_change) =====
private static bool IsOnChangeMode(GatherRule rule)
{
return string.Equals(rule.EmitMode, "on_change", StringComparison.OrdinalIgnoreCase);
}
/// <summary>
/// on_change decision for a non-empty collector result: true = emit (first result for the
/// rule, or the result changed), false = suppress. Hashes the result BEFORE ruleId/title
/// injection. On emit after a suppression streak, injects <c>suppressedPolls</c> and
/// <c>suppressedSinceUtc</c> into the result so the gap is observable.
/// Internal so the state machine is testable synchronously (no timer/ThreadPool races).
/// </summary>
internal bool ShouldEmitOnChange(GatherRule rule, Dictionary<string, object> result)
{
var hash = ComputeCanonicalHash(result);
bool emit;
lock (_scopeLock)
{
string lastHash;
if (_lastEmittedHash.TryGetValue(rule.RuleId, out lastHash) && lastHash == hash)
{
(int Count, DateTime SinceUtc) streak;
_suppressed[rule.RuleId] = _suppressed.TryGetValue(rule.RuleId, out streak)
? (streak.Count + 1, streak.SinceUtc)
: (1, DateTime.UtcNow);
emit = false;
}
else
{
_lastEmittedHash[rule.RuleId] = hash;
(int Count, DateTime SinceUtc) streak;
if (_suppressed.TryGetValue(rule.RuleId, out streak) && streak.Count > 0)
{
result["suppressedPolls"] = streak.Count;
result["suppressedSinceUtc"] = streak.SinceUtc.ToString("o", CultureInfo.InvariantCulture);
}
_suppressed.Remove(rule.RuleId);
emit = true;
}
}
if (!emit)
_logger.Debug($"Gather rule {rule.RuleId}: result unchanged, emit suppressed (on_change)");
return emit;
}
/// <summary>
/// Canonical hash of a collector result: keys ordinal-sorted at every dictionary level,
/// values rendered with invariant formatting, then SHA-256. Internal for direct test
/// coverage of the canonicalization.
/// </summary>
internal static string ComputeCanonicalHash(Dictionary<string, object> result)
{
var sb = new StringBuilder();
AppendCanonicalValue(sb, result);
using (var sha = SHA256.Create())
{
var hashBytes = sha.ComputeHash(Encoding.UTF8.GetBytes(sb.ToString()));
return Convert.ToBase64String(hashBytes);
}
}
private static void AppendCanonicalValue(StringBuilder sb, object value)
{
if (value == null)
{
sb.Append("<null>");
return;
}
// string first — it is IEnumerable and must not be treated as a sequence
if (value is string str)
{
sb.Append('"').Append(str).Append('"');
return;
}
if (value is System.Collections.IDictionary dict)
{
sb.Append('{');
var keys = dict.Keys.Cast<object>()
.Select(k => k?.ToString() ?? string.Empty)
.OrderBy(k => k, StringComparer.Ordinal);
foreach (var key in keys)
{
sb.Append(key).Append('=');
AppendCanonicalValue(sb, dict[key]);
sb.Append(';');
}
sb.Append('}');
return;
}
if (value is System.Collections.IEnumerable sequence)
{
sb.Append('[');
foreach (var item in sequence)
{
AppendCanonicalValue(sb, item);
sb.Append(';');
}
sb.Append(']');
return;
}
if (value is IFormattable formattable)
{
sb.Append(formattable.ToString(null, CultureInfo.InvariantCulture));
return;
}
sb.Append(value);
}
private EventSeverity ParseSeverity(string severity)
{
return GatherRuleContext.ParseSeverity(severity);
@@ -245,7 +527,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather
/// <summary>
/// Blocks until all startup rules that were queued by the most recent <see cref="UpdateRules"/>
/// call have finished executing, or the timeout elapses.
/// call have finished executing, or the timeout elapses. Deferred scoped startup rules
/// (waiting for their phase scope) are not part of the latch.
/// Returns true if all rules completed within the timeout; false if timed out.
/// </summary>
public bool WaitForStartupRules(int timeoutSeconds = 120)
@@ -136,6 +136,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Security
}
watcher.Start();
// Re-check after arming: Dispose() may have run between the lock release above and
// Start() — it then stopped a not-yet-armed watcher and this thread armed it right
// back, leaving a WMI subscription no code path could ever reach again. If disposal
// won that race, tear the watcher down here.
bool disposedDuringArm;
lock (_lock)
{
disposedDuringArm = _disposed;
if (disposedDuringArm) _startWatcher = null;
}
if (disposedDuringArm)
{
try { watcher.EventArrived -= OnProcessStartTrace; } catch { }
try { watcher.Stop(); } catch { }
try { watcher.Dispose(); } catch { }
return;
}
_logger.Info($"[ConsoleBypassWatcher] watching Win32_ProcessStartTrace for " +
$"{string.Join(", ", WatchedProcessNames)} (interactive-session + bare command line)");
}
@@ -335,6 +335,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
osBuildChangedProvider: () => osBuildChangeHost.BuildChanged));
}
// MDM reboot-policy watcher — subscribes to DeviceManagement-Enterprise-Diagnostics-
// Provider/Admin EventID 2800 and attributes the "unexpected reboot + second sign-in"
// pattern to the device-assigned policy URIs that forced the coalesced reboot.
// Agent-native collector (NOT an IAgentAnalyzer); the config flag is an internal
// steering lever only.
if (collectors.MdmRebootPolicyWatcherEnabled)
{
hosts.Add(new MdmRebootPolicyWatcherHost(
sessionId: sessionId,
tenantId: tenantId,
logger: logger,
ingress: ingress,
clock: clock,
backfillLookbackMinutes: collectors.MdmRebootPolicyBackfillLookbackMinutes,
stateDirectory: _stateDirectory));
}
// ----- Peripheral hosts (event-only; driven by remote-config toggles) --------------
// V1 parity (PeriodicCollectorManager) — combine Performance + AgentSelfMetrics under
@@ -0,0 +1,68 @@
#nullable enable
using System;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
/// <summary>
/// Standalone host for the <see cref="MdmRebootPolicyTracker"/> — watches
/// <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> (EventID
/// 2800) for MDM policies that force a coalesced reboot during ESP DeviceSetup and forwards
/// them via the shared single-rail ingress. Mirrors <see cref="WindowsUpdateWatcherHost"/>:
/// builds its own <see cref="InformationalEventPost"/> and delegates Start/Stop to the tracker.
/// </summary>
internal sealed class MdmRebootPolicyWatcherHost : ICollectorHost
{
public string Name => "MdmRebootPolicyTracker";
private readonly MdmRebootPolicyTracker _tracker;
private readonly AgentLogger _logger;
private int _disposed;
public MdmRebootPolicyWatcherHost(
string sessionId,
string tenantId,
AgentLogger logger,
ISignalIngressSink ingress,
IClock clock,
int backfillLookbackMinutes,
string stateDirectory)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
var post = new InformationalEventPost(ingress, clock, logger);
_tracker = new MdmRebootPolicyTracker(
sessionId: sessionId,
tenantId: tenantId,
post: post,
logger: logger,
backfillEnabled: backfillLookbackMinutes > 0,
backfillLookbackMinutes: backfillLookbackMinutes,
stateDirectory: stateDirectory);
}
public void Start()
{
_tracker.Start();
_logger.Info("MdmRebootPolicyWatcherHost: started.");
}
public void Stop()
{
try { _tracker.Stop(); }
catch (Exception ex) { _logger.Warning($"MdmRebootPolicyWatcherHost: stop failed: {ex.Message}"); }
}
public void Dispose()
{
if (Interlocked.Exchange(ref _disposed, 1) == 1) return;
Stop();
}
}
}
@@ -224,6 +224,11 @@ namespace AutopilotMonitor.Agent.V2
// Step 4 — run the executor and wait.
using (var executor = new GatherRuleExecutor(sessionId, tenantId, emitEvent, logger, config.ImeLogPathOverride))
{
// Diagnostic mode has no enrollment phase context — scoped rules
// (activePhases / activeFromPhase) execute unconditionally here.
executor.IgnorePhaseScope = true;
logger.Info("--run-gather-rules: phase scoping bypassed (no phase context in diagnostic mode).");
executor.UpdateRules(rules);
if (consoleMode) Console.WriteLine($"Running {startupRules.Count} rule(s)...");
@@ -0,0 +1,83 @@
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// RowKey → business-time decoders. These are the read-path compensation for rows whose
/// system Timestamp was reset by the 2026-07-18 storage migration: the RowKey is the only
/// time carrier that survives a row rewrite, so the decode must round-trip exactly and
/// must reject every foreign format (falling back instead of mis-dating rows).
/// </summary>
public class BusinessTimestampDecodeTests
{
private static readonly DateTime Ts = new DateTime(2026, 6, 15, 10, 30, 45, 123, DateTimeKind.Utc).AddTicks(4567);
[Fact]
public void Audit_rowkey_roundtrips_tick_exact()
{
var rowKey = TableStorageService.BuildAuditLogRowKey(Ts, Guid.NewGuid());
Assert.True(BusinessTimestamp.TryDecodeAuditRowKey(rowKey, out var decoded));
Assert.Equal(Ts, decoded);
Assert.Equal(DateTimeKind.Utc, decoded.Kind);
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData("!")]
[InlineData("!123")] // too short
[InlineData("!123456789012345678x_guid")] // non-digit in revtick
[InlineData("d6f6dd90f41c4b16b290aac042e5b466")] // legacy bare GUID (:N)
[InlineData("0000000000000000000")] // ops-style bare revtick is not an audit key
public void Audit_decoder_rejects_foreign_formats(string? rowKey)
{
Assert.False(BusinessTimestamp.TryDecodeAuditRowKey(rowKey, out _));
}
[Fact]
public void Ops_rowkey_roundtrips_tick_exact()
{
var rowKey = $"{DateTime.MaxValue.Ticks - Ts.Ticks:D19}";
Assert.True(BusinessTimestamp.TryDecodeOpsRowKey(rowKey, out var decoded));
Assert.Equal(Ts, decoded);
Assert.Equal(DateTimeKind.Utc, decoded.Kind);
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData("123456789012345678")] // 18 digits
[InlineData("12345678901234567890")] // 20 digits
[InlineData("123456789012345678x")] // non-digit
[InlineData("!1234567890123456789")] // audit-prefixed
public void Ops_decoder_rejects_foreign_formats(string? rowKey)
{
Assert.False(BusinessTimestamp.TryDecodeOpsRowKey(rowKey, out _));
}
[Fact]
public void Event_rowkey_prefix_decodes_to_millisecond_truncated_agent_time()
{
var rowKey = $"{Ts:yyyyMMddHHmmssfff}_{42L:D10}";
Assert.True(BusinessTimestamp.TryDecodeEventRowKeyPrefix(rowKey, out var decoded));
// The RowKey format truncates sub-millisecond ticks at write time.
var expected = new DateTime(2026, 6, 15, 10, 30, 45, 123, DateTimeKind.Utc);
Assert.Equal(expected, decoded);
Assert.Equal(DateTimeKind.Utc, decoded.Kind);
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData("20260615103045123")] // no separator/sequence
[InlineData("2026061510304512_0000000042")] // 16-digit prefix
[InlineData("202606151030451234_0000000042")] // 18-digit prefix
[InlineData("2026061510304512x_0000000042")] // non-digit in prefix
[InlineData("20261315103045123_0000000042")] // month 13
[InlineData("d6f6dd90f41c4b16b290aac042e5b466")] // GUID
public void Event_decoder_rejects_foreign_formats(string? rowKey)
{
Assert.False(BusinessTimestamp.TryDecodeEventRowKeyPrefix(rowKey, out _));
}
}
@@ -0,0 +1,181 @@
using System.Text.RegularExpressions;
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// RowKey-range date-window / retention clauses. Date filtering moved off the system
/// Timestamp (reset by storage migrations) onto the reverse-tick RowKey; these tests pin
/// (a) tick-exact inclusive/exclusive boundary semantics identical to the old
/// `Timestamp ge/le` clauses, and (b) the GUID-safety invariant: legacy bare-GUID audit
/// rows sort AFTER all '!'-rows, so any lower RowKey bound without the `RowKey lt '"'`
/// guard would match — and a retention sweep would DELETE — every legacy row.
/// </summary>
public class BusinessTimestampFilterClauseTests
{
private static readonly DateTime T = new DateTime(2026, 6, 15, 12, 0, 0, DateTimeKind.Utc).AddTicks(789);
private static readonly string Tenant = "11111111-1111-1111-1111-111111111111";
private static string AuditRk(DateTime ts) => TableStorageService.BuildAuditLogRowKey(ts, Guid.NewGuid());
private static string OpsRk(DateTime ts) => $"{DateTime.MaxValue.Ticks - ts.Ticks:D19}";
/// <summary>Evaluates a clause of the form "RowKey op 'literal' [and RowKey op 'literal']…" against a RowKey.</summary>
private static bool Matches(string clause, string rowKey)
{
var parts = Regex.Matches(clause, @"RowKey (ge|gt|le|lt) '([^']*)'");
Assert.True(parts.Count > 0, $"no RowKey comparisons in clause: {clause}");
foreach (Match m in parts)
{
var cmp = string.CompareOrdinal(rowKey, m.Groups[2].Value);
var ok = m.Groups[1].Value switch
{
"ge" => cmp >= 0,
"gt" => cmp > 0,
"le" => cmp <= 0,
"lt" => cmp < 0,
_ => throw new InvalidOperationException(),
};
if (!ok) return false;
}
return true;
}
// ===== Audit boundaries =====
[Fact]
public void Audit_dateFrom_is_inclusive_to_the_tick()
{
var clause = BusinessTimestamp.AuditDateFromClause(T);
Assert.True(Matches(clause, AuditRk(T))); // ts == from → included
Assert.True(Matches(clause, AuditRk(T.AddTicks(1)))); // newer → included
Assert.False(Matches(clause, AuditRk(T.AddTicks(-1)))); // 1 tick older → excluded
}
[Fact]
public void Audit_dateTo_is_inclusive_to_the_tick()
{
var clause = BusinessTimestamp.AuditDateToClause(T);
Assert.True(Matches(clause, AuditRk(T))); // ts == to → included
Assert.True(Matches(clause, AuditRk(T.AddTicks(-1)))); // older → included
Assert.False(Matches(clause, AuditRk(T.AddTicks(1)))); // 1 tick newer → excluded
}
[Fact]
public void Audit_retention_is_strictly_older_than_cutoff()
{
var clause = BusinessTimestamp.AuditRetentionClause(T);
Assert.False(Matches(clause, AuditRk(T))); // ts == cutoff → kept
Assert.True(Matches(clause, AuditRk(T.AddTicks(-1)))); // 1 tick older → deleted
Assert.False(Matches(clause, AuditRk(T.AddTicks(1)))); // newer → kept
}
// ===== Audit GUID safety (the single most dangerous invariant of this change) =====
[Fact]
public void Legacy_guid_rowkeys_sort_after_the_time_encoded_guard()
{
for (var i = 0; i < 20; i++)
{
var guidRk = Guid.NewGuid().ToString("N");
Assert.True(string.CompareOrdinal(guidRk, BusinessTimestamp.AuditTimeEncodedUpperBound) > 0);
}
}
[Fact]
public void No_audit_date_or_retention_clause_matches_legacy_guid_rowkeys()
{
var clauses = new[]
{
BusinessTimestamp.AuditDateFromClause(T),
BusinessTimestamp.AuditDateToClause(T),
BusinessTimestamp.AuditDateToClause(DateTime.MaxValue.AddTicks(-1)), // widest possible window
BusinessTimestamp.AuditRetentionClause(T),
BusinessTimestamp.AuditRetentionClause(DateTime.MaxValue.AddTicks(-1)), // widest possible sweep
};
foreach (var clause in clauses)
{
for (var i = 0; i < 20; i++)
{
Assert.False(Matches(clause, Guid.NewGuid().ToString("N")),
$"clause matched a legacy GUID RowKey: {clause}");
}
}
}
[Fact]
public void Audit_lower_bounded_clauses_carry_the_time_encoded_guard()
{
Assert.Contains($"RowKey lt '{BusinessTimestamp.AuditTimeEncodedUpperBound}'",
BusinessTimestamp.AuditDateToClause(T));
Assert.Contains($"RowKey lt '{BusinessTimestamp.AuditTimeEncodedUpperBound}'",
BusinessTimestamp.AuditRetentionClause(T));
}
// ===== Ops boundaries =====
[Fact]
public void Ops_dateFrom_is_inclusive_to_the_tick()
{
var clause = BusinessTimestamp.OpsDateFromClause(T);
Assert.True(Matches(clause, OpsRk(T)));
Assert.True(Matches(clause, OpsRk(T.AddTicks(1))));
Assert.False(Matches(clause, OpsRk(T.AddTicks(-1))));
}
[Fact]
public void Ops_dateTo_is_inclusive_to_the_tick()
{
var clause = BusinessTimestamp.OpsDateToClause(T);
Assert.True(Matches(clause, OpsRk(T)));
Assert.True(Matches(clause, OpsRk(T.AddTicks(-1))));
Assert.False(Matches(clause, OpsRk(T.AddTicks(1))));
}
[Fact]
public void Ops_retention_is_strictly_older_than_cutoff()
{
var clause = BusinessTimestamp.OpsRetentionClause(T);
Assert.False(Matches(clause, OpsRk(T)));
Assert.True(Matches(clause, OpsRk(T.AddTicks(-1))));
Assert.False(Matches(clause, OpsRk(T.AddTicks(1))));
}
// ===== Composition into the filter builders =====
[Fact]
public void Audit_filter_builders_emit_rowkey_windows_not_timestamp_clauses()
{
var from = new DateTime(2026, 6, 1, 0, 0, 0, DateTimeKind.Utc);
var to = new DateTime(2026, 7, 1, 0, 0, 0, DateTimeKind.Utc);
var plain = TableStorageService.BuildAuditLogFilter(Tenant, from, to);
var fanOut = TableStorageService.BuildAuditLogFilterWithRowKeyBound(
Tenant, from, to, lastRowKey: "!0123_x", excludeDeletions: false);
foreach (var f in new[] { plain!, fanOut })
{
Assert.DoesNotContain("Timestamp ge", f);
Assert.DoesNotContain("Timestamp le", f);
Assert.Contains(BusinessTimestamp.AuditDateFromClause(from), f);
Assert.Contains(BusinessTimestamp.AuditDateToClause(to), f);
}
Assert.Contains("RowKey gt '!0123_x'", fanOut); // fan-out cursor composes with the window
}
[Fact]
public void Ops_filter_builder_emits_rowkey_window_not_timestamp_clauses()
{
var from = new DateTime(2026, 6, 1, 0, 0, 0, DateTimeKind.Utc);
var to = new DateTime(2026, 7, 1, 0, 0, 0, DateTimeKind.Utc);
var f = TableOpsEventRepository.BuildFilter("Security", from, to);
Assert.NotNull(f);
Assert.DoesNotContain("Timestamp ge", f);
Assert.DoesNotContain("Timestamp le", f);
Assert.Contains(BusinessTimestamp.OpsDateFromClause(from), f);
Assert.Contains(BusinessTimestamp.OpsDateToClause(to), f);
}
}
@@ -0,0 +1,118 @@
using Azure.Data.Tables;
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Read-mapper resolution order for the three affected tables:
/// OccurredUtc column → RowKey decode → system Timestamp. The last hop is the
/// migration-corrupted value (a row rewrite resets it), so it must only ever win when
/// the row carries no better source (audit legacy bare-GUID rows).
/// </summary>
public class BusinessTimestampPrecedenceTests
{
private static readonly DateTime Occurred = new(2026, 5, 10, 8, 0, 0, DateTimeKind.Utc);
private static readonly DateTime RowKeyTime = new(2026, 6, 20, 9, 30, 0, DateTimeKind.Utc);
private static readonly DateTimeOffset SystemTs = new(2026, 7, 18, 13, 58, 0, TimeSpan.Zero); // migration moment
private static TableEntity WithSystemTimestamp(TableEntity e)
{
e["Timestamp"] = SystemTs;
return e;
}
// ===== Audit =====
[Fact]
public void Audit_occurredutc_wins_over_rowkey_and_system()
{
var e = WithSystemTimestamp(new TableEntity("t", TableStorageService.BuildAuditLogRowKey(RowKeyTime, Guid.NewGuid()))
{
[BusinessTimestamp.OccurredUtcColumn] = new DateTimeOffset(Occurred),
});
Assert.Equal(Occurred, TableStorageService.ResolveAuditTimestamp(e));
}
[Fact]
public void Audit_rowkey_decode_wins_over_system_when_column_missing()
{
var e = WithSystemTimestamp(new TableEntity("t", TableStorageService.BuildAuditLogRowKey(RowKeyTime, Guid.NewGuid())));
Assert.Equal(RowKeyTime, TableStorageService.ResolveAuditTimestamp(e));
}
[Fact]
public void Audit_legacy_guid_row_falls_back_to_system_timestamp()
{
var e = WithSystemTimestamp(new TableEntity("t", Guid.NewGuid().ToString("N")));
Assert.Equal(SystemTs.UtcDateTime, TableStorageService.ResolveAuditTimestamp(e));
}
// ===== Ops =====
[Fact]
public void Ops_occurredutc_wins_over_rowkey_and_system()
{
var e = WithSystemTimestamp(new TableEntity("Security", $"{DateTime.MaxValue.Ticks - RowKeyTime.Ticks:D19}")
{
[BusinessTimestamp.OccurredUtcColumn] = new DateTimeOffset(Occurred),
});
Assert.Equal(Occurred, TableOpsEventRepository.ResolveTimestamp(e));
}
[Fact]
public void Ops_rowkey_decode_wins_over_system_when_column_missing()
{
var e = WithSystemTimestamp(new TableEntity("Security", $"{DateTime.MaxValue.Ticks - RowKeyTime.Ticks:D19}"));
Assert.Equal(RowKeyTime, TableOpsEventRepository.ResolveTimestamp(e));
}
[Fact]
public void Ops_undecodable_rowkey_falls_back_to_system_timestamp()
{
var e = WithSystemTimestamp(new TableEntity("Security", "not-a-revtick"));
Assert.Equal(SystemTs.UtcDateTime, TableOpsEventRepository.ResolveTimestamp(e));
}
// ===== Events =====
private static string EventRowKey(DateTime ts, long seq = 42) => $"{ts:yyyyMMddHHmmssfff}_{seq:D10}";
[Fact]
public void Event_occurredutc_wins_over_rowkey_and_system()
{
var e = WithSystemTimestamp(new TableEntity("tenant_session", EventRowKey(RowKeyTime))
{
[BusinessTimestamp.OccurredUtcColumn] = new DateTimeOffset(Occurred),
});
Assert.Equal(Occurred, TableStorageService.ResolveEventTimestamp(e));
}
[Fact]
public void Event_rowkey_prefix_wins_over_system_when_column_missing()
{
// The pre-cutover case: no OccurredUtc column, system Timestamp = migration moment.
var e = WithSystemTimestamp(new TableEntity("tenant_session", EventRowKey(RowKeyTime)));
Assert.Equal(RowKeyTime, TableStorageService.ResolveEventTimestamp(e));
}
[Fact]
public void Event_undecodable_rowkey_falls_back_to_system_timestamp()
{
var e = WithSystemTimestamp(new TableEntity("tenant_session", "weird-rowkey"));
Assert.Equal(SystemTs.UtcDateTime, TableStorageService.ResolveEventTimestamp(e));
}
[Fact]
public void Event_occurredutc_read_tolerates_datetime_materialization()
{
// SDK reads surface Edm.DateTime as DateTimeOffset or DateTime depending on payload
// shape — the resolver must accept both (dual-reader lesson).
var e = new TableEntity("tenant_session", EventRowKey(RowKeyTime))
{
[BusinessTimestamp.OccurredUtcColumn] = Occurred, // plain DateTime, not DateTimeOffset
};
Assert.Equal(Occurred, TableStorageService.ResolveEventTimestamp(e));
}
}
@@ -0,0 +1,283 @@
using AutopilotMonitor.Functions.Functions.Rules;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Azure;
using Azure.Data.Tables;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Covers the three gather-rule scope/emit fields (ActivePhases, ActiveFromPhase, EmitMode):
/// Store↔Map table-serialization roundtrip (table-serialization rule: every model field in BOTH
/// directions), legacy rows without the new columns, the <see cref="GatherRuleService.ContentEquivalent"/>
/// null-tolerance (no reseed churn), and the create/update validation matrix in
/// <see cref="GatherRulesFunction.ValidateScopeAndEmitMode"/>.
/// </summary>
public class GatherRuleScopeFieldsTests
{
// ── Store ↔ Map roundtrip ──────────────────────────────────────────────
private static (TableStorageService service, Func<TableEntity?> lastUpserted) BuildStorageHarness()
{
TableEntity? captured = null;
var tableClient = new Mock<TableClient>();
tableClient
.Setup(c => c.UpsertEntityAsync(
It.IsAny<TableEntity>(), It.IsAny<TableUpdateMode>(), It.IsAny<CancellationToken>()))
.Returns<TableEntity, TableUpdateMode, CancellationToken>((e, _, _) =>
{
captured = e;
return Task.FromResult(new Mock<Response>().Object);
});
var serviceClient = new Mock<TableServiceClient>();
serviceClient.Setup(c => c.GetTableClient(It.IsAny<string>())).Returns(tableClient.Object);
var service = new TableStorageService(serviceClient.Object, NullLogger<TableStorageService>.Instance);
return (service, () => captured);
}
[Fact]
public async Task Roundtrip_ScopeFields_SurviveStoreAndMap()
{
var (service, lastUpserted) = BuildStorageHarness();
var rule = new GatherRule
{
RuleId = "GATHER-SCOPE-001",
Title = "Scoped rule",
CollectorType = "registry",
Target = "HKLM\\SOFTWARE\\Contoso",
Trigger = "interval",
IntervalSeconds = 60,
OutputEventType = "gather_scoped",
ActivePhases = new List<string> { "AccountSetup", "AppsUser" },
EmitMode = "on_change",
};
Assert.True(await service.StoreGatherRuleAsync(rule, "global"));
var mapped = service.MapToGatherRule(lastUpserted()!);
Assert.NotNull(mapped.ActivePhases);
Assert.Equal(new[] { "AccountSetup", "AppsUser" }, mapped.ActivePhases);
Assert.Null(mapped.ActiveFromPhase);
Assert.Equal("on_change", mapped.EmitMode);
}
[Fact]
public async Task Roundtrip_FromPhase_SurvivesStoreAndMap()
{
var (service, lastUpserted) = BuildStorageHarness();
var rule = new GatherRule
{
RuleId = "GATHER-SCOPE-002",
Title = "From-phase rule",
CollectorType = "registry",
Target = "HKLM\\SOFTWARE\\Contoso",
Trigger = "interval",
OutputEventType = "gather_scoped",
ActiveFromPhase = "AccountSetup",
};
Assert.True(await service.StoreGatherRuleAsync(rule, "global"));
var mapped = service.MapToGatherRule(lastUpserted()!);
Assert.Null(mapped.ActivePhases);
Assert.Equal("AccountSetup", mapped.ActiveFromPhase);
Assert.Null(mapped.EmitMode);
}
[Fact]
public async Task Roundtrip_NullScopeFields_StayNull()
{
var (service, lastUpserted) = BuildStorageHarness();
var rule = new GatherRule
{
RuleId = "GATHER-SCOPE-003",
Title = "Unscoped rule",
CollectorType = "registry",
Target = "HKLM\\SOFTWARE\\Contoso",
Trigger = "startup",
OutputEventType = "gather_plain",
};
Assert.True(await service.StoreGatherRuleAsync(rule, "global"));
var mapped = service.MapToGatherRule(lastUpserted()!);
Assert.Null(mapped.ActivePhases);
Assert.Null(mapped.ActiveFromPhase);
Assert.Null(mapped.EmitMode);
}
[Fact]
public void Map_LegacyRow_WithoutScopeColumns_DefaultsToNull()
{
var (service, _) = BuildStorageHarness();
// A row written before the scope fields existed: none of the new columns present.
var legacy = new TableEntity("global", "GATHER-LEGACY-001")
{
["Title"] = "Legacy rule",
["CollectorType"] = "registry",
["Target"] = "HKLM\\SOFTWARE\\Contoso",
["Trigger"] = "startup",
["OutputEventType"] = "gather_legacy",
};
var mapped = service.MapToGatherRule(legacy);
Assert.Null(mapped.ActivePhases);
Assert.Null(mapped.ActiveFromPhase);
Assert.Null(mapped.EmitMode);
}
// ── ContentEquivalent ──────────────────────────────────────────────────
private static GatherRule BaseRule() => new()
{
RuleId = "GATHER-EQ-001",
Title = "Rule",
Description = "Desc",
Version = "1.0.0",
CollectorType = "registry",
Target = "HKLM\\SOFTWARE\\Contoso",
Trigger = "interval",
TriggerPhase = "",
OutputEventType = "gather_eq",
};
[Fact]
public void ContentEquivalent_NullAndEmptyActivePhases_AreEquivalent()
{
// Pre-existing DB rows map absent columns to null; seeds are also null — and an
// explicitly-empty list means the same "unrestricted". No reseed churn allowed.
var a = BaseRule();
var b = BaseRule();
b.ActivePhases = new List<string>();
Assert.True(GatherRuleService.ContentEquivalent(a, b));
Assert.True(GatherRuleService.ContentEquivalent(b, a));
}
[Fact]
public void ContentEquivalent_DetectsChangesInEachScopeField()
{
var baseline = BaseRule();
var phases = BaseRule();
phases.ActivePhases = new List<string> { "AccountSetup" };
Assert.False(GatherRuleService.ContentEquivalent(baseline, phases));
var from = BaseRule();
from.ActiveFromPhase = "AccountSetup";
Assert.False(GatherRuleService.ContentEquivalent(baseline, from));
var emit = BaseRule();
emit.EmitMode = "on_change";
Assert.False(GatherRuleService.ContentEquivalent(baseline, emit));
}
[Fact]
public void ContentEquivalent_SameScopeFields_AreEquivalent()
{
var a = BaseRule();
a.ActivePhases = new List<string> { "DeviceSetup", "AccountSetup" };
a.EmitMode = "on_change";
var b = BaseRule();
b.ActivePhases = new List<string> { "DeviceSetup", "AccountSetup" };
b.EmitMode = "on_change";
Assert.True(GatherRuleService.ContentEquivalent(a, b));
}
// ── Validation ─────────────────────────────────────────────────────────
private static GatherRule ValidationRule(
List<string>? activePhases = null, string? activeFromPhase = null, string? emitMode = null) => new()
{
RuleId = "GATHER-VAL-001",
Title = "Rule",
CollectorType = "registry",
Target = "HKLM\\SOFTWARE\\Contoso",
Trigger = "interval",
OutputEventType = "gather_val",
ActivePhases = activePhases,
ActiveFromPhase = activeFromPhase,
EmitMode = emitMode,
};
[Theory]
[InlineData(null)]
[InlineData("Start")]
[InlineData("DevicePreparation")]
[InlineData("AccountSetup")]
[InlineData("Complete")]
[InlineData("accountsetup")] // case-insensitive
public void Validate_AcceptsValidFromPhase(string? fromPhase)
{
Assert.Null(GatherRulesFunction.ValidateScopeAndEmitMode(ValidationRule(activeFromPhase: fromPhase)));
}
[Theory]
[InlineData("Unknown")]
[InlineData("Failed")]
[InlineData("4")] // numeric enum value — canonical names only
[InlineData("DeviceESP")] // the invalid token the old free-text UI suggested
public void Validate_RejectsInvalidFromPhase(string fromPhase)
{
Assert.NotNull(GatherRulesFunction.ValidateScopeAndEmitMode(ValidationRule(activeFromPhase: fromPhase)));
}
[Fact]
public void Validate_AcceptsValidActivePhases_AndRejectsInvalidEntries()
{
Assert.Null(GatherRulesFunction.ValidateScopeAndEmitMode(
ValidationRule(activePhases: new List<string> { "DeviceSetup", "AppsDevice" })));
Assert.NotNull(GatherRulesFunction.ValidateScopeAndEmitMode(
ValidationRule(activePhases: new List<string> { "DeviceSetup", "Failed" })));
Assert.NotNull(GatherRulesFunction.ValidateScopeAndEmitMode(
ValidationRule(activePhases: new List<string> { "" })));
}
[Fact]
public void Validate_RejectsBothScopeFieldsSet()
{
var error = GatherRulesFunction.ValidateScopeAndEmitMode(ValidationRule(
activePhases: new List<string> { "AccountSetup" }, activeFromPhase: "DeviceSetup"));
Assert.NotNull(error);
Assert.Contains("mutually exclusive", error);
}
[Theory]
[InlineData(null)]
[InlineData("always")]
[InlineData("on_change")]
[InlineData("ON_CHANGE")] // case-insensitive
public void Validate_AcceptsValidEmitModes(string? emitMode)
{
Assert.Null(GatherRulesFunction.ValidateScopeAndEmitMode(ValidationRule(emitMode: emitMode)));
}
[Fact]
public void Validate_RejectsUnknownEmitMode()
{
Assert.NotNull(GatherRulesFunction.ValidateScopeAndEmitMode(ValidationRule(emitMode: "sometimes")));
}
[Fact]
public void Validate_TogglePartialPayload_PassesThrough()
{
// The portal toggle sends only { enabled, isBuiltIn, isCommunity } — none of the
// scope fields — and must not be rejected.
var toggle = new GatherRule { RuleId = "GATHER-VAL-002", Enabled = false };
Assert.Null(GatherRulesFunction.ValidateScopeAndEmitMode(toggle));
}
[Fact]
public void Validate_EmptyActivePhasesList_IsUnrestricted_AndPasses()
{
Assert.Null(GatherRulesFunction.ValidateScopeAndEmitMode(ValidationRule(activePhases: new List<string>())));
}
}
@@ -0,0 +1,146 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the custom-rule partial-PUT fix on <see cref="GatherRuleService.UpdateRuleAsync"/>:
/// the portal toggle PUTs only { enabled, isBuiltIn, isCommunity }, and full-storing that
/// payload wiped Title/Target/Trigger/Parameters of the custom rule. Toggle-style partials
/// (empty Title/CollectorType/Target) must merge Enabled into the existing row; full payloads
/// (edit flow) must still replace the row but preserve the original CreatedAt.
/// </summary>
public class GatherRuleUpdatePartialMergeTests
{
private const string TenantId = "a1b2c3d4-e5f6-7890-abcd-ef1234567890";
private const string RuleId = "GATHER-CUST-001";
private static GatherRule ExistingCustomRule() => new()
{
RuleId = RuleId,
Title = "Collect BIOS Config",
Description = "Reads the RealmJoin BIOS key",
Category = "device",
CollectorType = "registry",
Target = "HKLM\\SOFTWARE\\RealmJoin\\Custom\\BIOS",
Parameters = new Dictionary<string, string> { ["valueName"] = "Version" },
Trigger = "interval",
IntervalSeconds = 60,
OutputEventType = "gather_bios_config",
Enabled = true,
IsBuiltIn = false,
IsCommunity = false,
CreatedAt = new DateTime(2026, 1, 15, 12, 0, 0, DateTimeKind.Utc),
ActiveFromPhase = "AccountSetup",
EmitMode = "on_change",
};
private static (GatherRuleService service, Mock<IRuleRepository> repo, List<GatherRule> stored) BuildService(
GatherRule? existingTenantRule)
{
var repo = new Mock<IRuleRepository>(MockBehavior.Loose);
var stored = new List<GatherRule>();
repo.Setup(r => r.GetGatherRulesAsync("global")).ReturnsAsync(new List<GatherRule>());
repo.Setup(r => r.GetGatherRulesAsync(TenantId)).ReturnsAsync(
existingTenantRule == null ? new List<GatherRule>() : new List<GatherRule> { existingTenantRule });
repo.Setup(r => r.StoreGatherRuleAsync(It.IsAny<GatherRule>(), TenantId))
.Callback<GatherRule, string>((rule, _) => stored.Add(rule))
.ReturnsAsync(true);
return (new GatherRuleService(repo.Object, NullLogger<GatherRuleService>.Instance), repo, stored);
}
[Fact]
public async Task TogglePartial_merges_enabled_and_preserves_definition_fields()
{
var existing = ExistingCustomRule();
var (service, _, stored) = BuildService(existing);
// Exactly what the portal toggle sends: { enabled, isBuiltIn, isCommunity }.
var togglePayload = new GatherRule { RuleId = RuleId, Enabled = false, IsBuiltIn = false, IsCommunity = false };
var ok = await service.UpdateRuleAsync(TenantId, togglePayload);
Assert.True(ok);
var written = Assert.Single(stored);
Assert.False(written.Enabled);
Assert.Equal("Collect BIOS Config", written.Title);
Assert.Equal("HKLM\\SOFTWARE\\RealmJoin\\Custom\\BIOS", written.Target);
Assert.Equal("registry", written.CollectorType);
Assert.Equal("interval", written.Trigger);
Assert.Equal(60, written.IntervalSeconds);
Assert.Equal("Version", written.Parameters["valueName"]);
Assert.Equal("AccountSetup", written.ActiveFromPhase);
Assert.Equal("on_change", written.EmitMode);
Assert.Equal(existing.CreatedAt, written.CreatedAt);
}
[Fact]
public async Task TogglePartial_for_unknown_custom_rule_fails_without_storing()
{
var (service, repo, stored) = BuildService(existingTenantRule: null);
var togglePayload = new GatherRule { RuleId = RuleId, Enabled = false, IsBuiltIn = false, IsCommunity = false };
var ok = await service.UpdateRuleAsync(TenantId, togglePayload);
Assert.False(ok);
Assert.Empty(stored);
repo.Verify(r => r.StoreGatherRuleAsync(It.IsAny<GatherRule>(), It.IsAny<string>()), Times.Never);
}
[Fact]
public async Task FullPayload_replaces_rule_but_preserves_original_CreatedAt()
{
var existing = ExistingCustomRule();
var (service, _, stored) = BuildService(existing);
var fullPayload = new GatherRule
{
RuleId = RuleId,
Title = "Collect BIOS Config v2",
CollectorType = "registry",
Target = "HKLM\\SOFTWARE\\RealmJoin\\Custom\\BIOS2",
Trigger = "interval",
IntervalSeconds = 120,
OutputEventType = "gather_bios_config",
Enabled = true,
// Edit flow sends createdAt from the client; a JSON-mode edit may omit it and
// default to UtcNow — either way the stored row must keep the original.
CreatedAt = DateTime.UtcNow,
};
var ok = await service.UpdateRuleAsync(TenantId, fullPayload);
Assert.True(ok);
var written = Assert.Single(stored);
Assert.Equal("Collect BIOS Config v2", written.Title);
Assert.Equal("HKLM\\SOFTWARE\\RealmJoin\\Custom\\BIOS2", written.Target);
Assert.Equal(120, written.IntervalSeconds);
Assert.False(written.IsBuiltIn);
Assert.False(written.IsCommunity);
Assert.Equal(existing.CreatedAt, written.CreatedAt);
}
[Fact]
public async Task BuiltInRule_toggle_still_goes_through_RuleState_path()
{
var repo = new Mock<IRuleRepository>(MockBehavior.Loose);
repo.Setup(r => r.GetGatherRulesAsync("global")).ReturnsAsync(new List<GatherRule>
{
new() { RuleId = RuleId, IsBuiltIn = true, Title = "Built-in", Enabled = true }
});
repo.Setup(r => r.StoreRuleStateAsync(TenantId, RuleId, It.IsAny<RuleState>())).ReturnsAsync(true);
var service = new GatherRuleService(repo.Object, NullLogger<GatherRuleService>.Instance);
var ok = await service.UpdateRuleAsync(TenantId, new GatherRule { RuleId = RuleId, Enabled = false });
Assert.True(ok);
repo.Verify(r => r.StoreRuleStateAsync(TenantId, RuleId, It.Is<RuleState>(s => !s.Enabled)), Times.Once);
repo.Verify(r => r.StoreGatherRuleAsync(It.IsAny<GatherRule>(), It.IsAny<string>()), Times.Never);
}
}
@@ -129,4 +129,30 @@ public class GetAllTenantConfigurationsFunctionTests
[Fact]
public void DelegatedBareArrayView_Empty_Empty()
=> Assert.Empty(GetAllTenantConfigurationsFunction.DelegatedBareArrayView(new TenantConfiguration[0]));
[Fact]
public void MissingManagedIds_ReturnsOnlyIdsWithoutConfig_CaseInsensitive()
{
var configs = new List<TenantConfiguration>
{
new() { TenantId = TenantB.ToUpperInvariant() }, // case variant still counts as present
};
var missing = GetAllTenantConfigurationsFunction.MissingManagedIds(new[] { TenantB, TenantC }, configs);
var only = Assert.Single(missing);
Assert.Equal(TenantC, only);
}
[Fact]
public void MissingManagedIds_AllPresent_Empty()
=> Assert.Empty(GetAllTenantConfigurationsFunction.MissingManagedIds(
new[] { TenantB },
new List<TenantConfiguration> { new() { TenantId = TenantB } }));
[Fact]
public void MissingConfigCacheKey_LowercasesId_SoCaseVariantsShareOneVerdict()
=> Assert.Equal(
GetAllTenantConfigurationsFunction.MissingConfigCacheKey(TenantB),
GetAllTenantConfigurationsFunction.MissingConfigCacheKey(TenantB.ToUpperInvariant()));
}
@@ -0,0 +1,107 @@
using System.Collections.Generic;
using System.Net;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Functions.Tests.GraphResolution;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging.Abstractions;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Service-level tests for the Agent Binaries HealthCheck card. The check probes the
/// canonical download alias (download.autopilotmonitor.com) AND the legacy blob keepalive
/// account: alias failure is unhealthy (new installs break), legacy failure is only a
/// warning (existing customers break until they migrate to the alias).
/// </summary>
public class HealthCheckServiceAgentBinariesTests
{
private const string AliasHost = "download.autopilotmonitor.com";
private const string LegacyHost = "autopilotmonitor.blob.core.windows.net";
[Fact]
public async Task Check_AllEndpointsAvailable_ReportsHealthyWithoutUrlDetails()
{
var handler = new StubHttpMessageHandler()
.When(AliasHost, HttpStatusCode.OK, "")
.When(LegacyHost, HttpStatusCode.OK, "");
var check = await BuildService(handler).CheckAgentBinariesAsync(includeEndpointUrls: false);
Assert.Equal("Agent Binaries", check.Name);
Assert.Equal("healthy", check.Status);
Assert.Contains("download alias", check.Message);
Assert.Null(check.Details);
// Both hosts must actually have been probed (ZIP + PS1 each).
Assert.Equal(2, handler.Requests.FindAll(u => u.Contains(AliasHost)).Count);
Assert.Equal(2, handler.Requests.FindAll(u => u.Contains(LegacyHost)).Count);
}
[Fact]
public async Task Check_WithEndpointUrls_SurfacesAliasAndLegacyUrlsInDetails()
{
var handler = new StubHttpMessageHandler()
.When(AliasHost, HttpStatusCode.OK, "")
.When(LegacyHost, HttpStatusCode.OK, "");
var check = await BuildService(handler).CheckAgentBinariesAsync(includeEndpointUrls: true);
Assert.NotNull(check.Details);
Assert.Equal("https://download.autopilotmonitor.com/agent/AutopilotMonitor-Agent.zip", check.Details!["Agent ZIP"]);
Assert.Equal("https://download.autopilotmonitor.com/agent/Install-AutopilotMonitor.ps1", check.Details["Bootstrap script"]);
Assert.Equal("https://autopilotmonitor.blob.core.windows.net/agent", check.Details["Legacy blob (keepalive)"]);
}
[Fact]
public async Task Check_LegacyBlobMissing_ReportsWarningNamingTheLegacyEndpoint()
{
var handler = new StubHttpMessageHandler()
.When(AliasHost, HttpStatusCode.OK, "")
.When(LegacyHost, HttpStatusCode.NotFound, "");
var check = await BuildService(handler).CheckAgentBinariesAsync(includeEndpointUrls: false);
Assert.Equal("warning", check.Status);
Assert.Contains("legacy", check.Message, System.StringComparison.OrdinalIgnoreCase);
Assert.Contains("404", check.Message);
}
[Fact]
public async Task Check_AliasMissing_ReportsUnhealthy()
{
var handler = new StubHttpMessageHandler()
.When(AliasHost, HttpStatusCode.NotFound, "")
.When(LegacyHost, HttpStatusCode.OK, "");
var check = await BuildService(handler).CheckAgentBinariesAsync(includeEndpointUrls: false);
Assert.Equal("unhealthy", check.Status);
Assert.Contains("download alias", check.Message);
}
[Fact]
public async Task Check_EndpointsUnreachable_ReportsUnhealthy()
{
// Un-scripted URLs make the stub throw — models a connection failure.
var check = await BuildService(new StubHttpMessageHandler()).CheckAgentBinariesAsync(includeEndpointUrls: false);
Assert.Equal("unhealthy", check.Status);
Assert.Contains("unreachable", check.Message);
}
private static HealthCheckService BuildService(StubHttpMessageHandler handler)
{
var config = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>())
.Build();
return new HealthCheckService(
NullLogger<HealthCheckService>.Instance,
adminConfigService: null!,
httpClientFactory: new StubHttpClientFactory(handler),
metricsReader: null!,
poisonQueueProbe: null!,
configuration: config);
}
}
@@ -53,6 +53,29 @@ public class HealthCheckServiceMcpServerTests
Assert.DoesNotContain("http", check.Message, System.StringComparison.OrdinalIgnoreCase);
}
[Fact]
public async Task Check_WithEndpointUrl_SurfacesServerUrlAlongsideVersion()
{
var svc = BuildService(HttpStatusCode.OK, "{\"status\":\"healthy\",\"version\":\"1.4.0\"}");
var check = await svc.CheckMcpServerAsync(includeEndpointUrl: true);
// The Version detail must merge into — not replace — the Server URL detail.
Assert.Equal("https://mcp.example.test", check.Details!["Server URL"]);
Assert.Equal("1.4.0", check.Details["Version"]);
}
[Fact]
public async Task Check_WithEndpointUrl_SurfacesServerUrlEvenWhenUnhealthy()
{
var svc = BuildService(HttpStatusCode.ServiceUnavailable, "");
var check = await svc.CheckMcpServerAsync(includeEndpointUrl: true);
Assert.Equal("unhealthy", check.Status);
Assert.Equal("https://mcp.example.test", check.Details!["Server URL"]);
}
[Fact]
public async Task Check_TimeoutOrConnectionFailure_ReportsWarningForColdStart()
{
@@ -0,0 +1,61 @@
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Per-row backfill decision (pure function): rows that already carry OccurredUtc are
/// skipped (idempotent re-runs), decodable RowKeys yield the deterministic decoded value,
/// anything else is reported undecodable — never guessed.
/// </summary>
public class OccurredUtcBackfillServiceTests
{
private static readonly DateTime Ts = new(2026, 6, 20, 9, 30, 0, DateTimeKind.Utc);
[Fact]
public void Audit_row_with_existing_column_is_skipped()
{
var e = new TableEntity("t", TableStorageService.BuildAuditLogRowKey(Ts, Guid.NewGuid()))
{
[BusinessTimestamp.OccurredUtcColumn] = new DateTimeOffset(Ts),
};
Assert.Equal(OccurredUtcBackfillService.RowDecision.AlreadySet,
OccurredUtcBackfillService.DecideRow(OccurredUtcBackfillService.TableAudit, e, out _));
}
[Fact]
public void Audit_time_encoded_row_yields_decoded_write()
{
var e = new TableEntity("t", TableStorageService.BuildAuditLogRowKey(Ts, Guid.NewGuid()));
var decision = OccurredUtcBackfillService.DecideRow(OccurredUtcBackfillService.TableAudit, e, out var occurred);
Assert.Equal(OccurredUtcBackfillService.RowDecision.Write, decision);
Assert.Equal(Ts, occurred);
}
[Fact]
public void Audit_legacy_guid_row_is_undecodable_not_guessed()
{
var e = new TableEntity("t", Guid.NewGuid().ToString("N"));
Assert.Equal(OccurredUtcBackfillService.RowDecision.Undecodable,
OccurredUtcBackfillService.DecideRow(OccurredUtcBackfillService.TableAudit, e, out _));
}
[Fact]
public void Ops_row_yields_decoded_write_and_existing_column_is_skipped()
{
var rowKey = $"{DateTime.MaxValue.Ticks - Ts.Ticks:D19}";
var bare = new TableEntity("Security", rowKey);
var decision = OccurredUtcBackfillService.DecideRow(OccurredUtcBackfillService.TableOps, bare, out var occurred);
Assert.Equal(OccurredUtcBackfillService.RowDecision.Write, decision);
Assert.Equal(Ts, occurred);
var backfilled = new TableEntity("Security", rowKey)
{
[BusinessTimestamp.OccurredUtcColumn] = new DateTimeOffset(Ts),
};
Assert.Equal(OccurredUtcBackfillService.RowDecision.AlreadySet,
OccurredUtcBackfillService.DecideRow(OccurredUtcBackfillService.TableOps, backfilled, out _));
}
}
@@ -0,0 +1,269 @@
using AutopilotMonitor.Functions.Functions.Admin;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Covers the SignalR revoke-enforcement cores of <see cref="DelegatedAdminManagementFunction"/> and
/// <see cref="TenantGroupManagementFunction"/> (extracted seams — the HTTP entry points are intentionally
/// not exercised, same rationale as <see cref="GetAllBlockedDevicesFunctionTests"/>). Locks in three
/// security invariants: (1) a miss (no assignment row) neither audits nor disconnects — a typo must not
/// write false customer-visible "access removed" rows while the real grant stays live; (2) a real
/// revoke/disable/unassign/group-delete always cuts the affected UPN's live streams (group authorization
/// is join-time only); (3) grant validation is fail-closed (GUID tenantId, recognized roles only).
/// </summary>
public class RevokeEnforcementTests
{
private const string Upn = "MSP-Admin@Partner.Example";
private const string UpnLower = "msp-admin@partner.example";
private const string TenantA = "11111111-1111-1111-1111-111111111111";
private const string TenantB = "22222222-2222-2222-2222-222222222222";
private const string GroupId = "tpl-aaaaaaaa";
private static DelegatedAdminService BuildService(Mock<IAdminRepository> repo) =>
new(
repo.Object,
new StubTenantEntitlementService(AutopilotMonitor.Functions.Security.TenantEdition.Enterprise),
new MemoryCache(new MemoryCacheOptions()),
NullLogger<DelegatedAdminService>.Instance);
private static Mock<IMaintenanceRepository> BuildAuditRepo()
{
var audit = new Mock<IMaintenanceRepository>();
audit.Setup(a => a.LogAuditEntryAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(),
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<Dictionary<string, string>?>()))
.ReturnsAsync(true);
return audit;
}
private static void VerifyNoAudit(Mock<IMaintenanceRepository> audit) =>
audit.Verify(a => a.LogAuditEntryAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(),
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<Dictionary<string, string>?>()), Times.Never);
private static (DelegatedAdminManagementFunction Fn, Mock<IAdminRepository> Repo,
Mock<IMaintenanceRepository> Audit, FakeSignalRNotificationService SignalR) BuildAdminFn()
{
var repo = new Mock<IAdminRepository>();
var audit = BuildAuditRepo();
var signalR = new FakeSignalRNotificationService();
var fn = new DelegatedAdminManagementFunction(
NullLogger<DelegatedAdminManagementFunction>.Instance, BuildService(repo), audit.Object, signalR);
return (fn, repo, audit, signalR);
}
private static (TenantGroupManagementFunction Fn, Mock<IAdminRepository> Repo,
Mock<IMaintenanceRepository> Audit, FakeSignalRNotificationService SignalR) BuildGroupFn()
{
var repo = new Mock<IAdminRepository>();
// Loose-mock defaults return a null Task — every repo read the flows touch needs a concrete setup.
repo.Setup(r => r.GetTenantGroupAsync(It.IsAny<string>())).ReturnsAsync((TenantGroup?)null);
repo.Setup(r => r.GetGroupAssignmentsForUpnAsync(It.IsAny<string>()))
.ReturnsAsync(new List<TenantGroupAssignment>());
repo.Setup(r => r.GetGroupAssigneesAsync(It.IsAny<string>()))
.ReturnsAsync(new List<TenantGroupAssignment>());
var audit = BuildAuditRepo();
var signalR = new FakeSignalRNotificationService();
var fn = new TenantGroupManagementFunction(
NullLogger<TenantGroupManagementFunction>.Instance, BuildService(repo), audit.Object, signalR);
return (fn, repo, audit, signalR);
}
// --- Delegated grant validation (fail-closed) ---
[Fact]
public void ValidateGrant_NullBody_Rejected()
=> Assert.Equal("upn and tenantId are required",
DelegatedAdminManagementFunction.ValidateGrantRequest(null, out _));
[Theory]
[InlineData("", TenantA)]
[InlineData(" ", TenantA)]
[InlineData(Upn, "")]
public void ValidateGrant_MissingFields_Rejected(string upn, string tenantId)
=> Assert.Equal("upn and tenantId are required",
DelegatedAdminManagementFunction.ValidateGrantRequest(
new GrantDelegatedAdminRequest { Upn = upn, TenantId = tenantId }, out _));
[Theory]
[InlineData("contoso.com")]
[InlineData("not-a-guid")]
[InlineData("11111111-1111-1111-1111")]
public void ValidateGrant_NonGuidTenantId_Rejected(string tenantId)
=> Assert.Equal("a valid tenantId (GUID) is required",
DelegatedAdminManagementFunction.ValidateGrantRequest(
new GrantDelegatedAdminRequest { Upn = Upn, TenantId = tenantId }, out _));
[Fact]
public void ValidateGrant_UnknownRole_Rejected()
{
var error = DelegatedAdminManagementFunction.ValidateGrantRequest(
new GrantDelegatedAdminRequest { Upn = Upn, TenantId = TenantA, Role = "GlobalAdmin" }, out _);
Assert.NotNull(error);
Assert.Contains(Constants.DelegatedRoles.DelegatedReader, error);
}
[Fact]
public void ValidateGrant_EmptyRole_DefaultsToLeastPrivilegedReader()
{
Assert.Null(DelegatedAdminManagementFunction.ValidateGrantRequest(
new GrantDelegatedAdminRequest { Upn = Upn, TenantId = TenantA }, out var role));
Assert.Equal(Constants.DelegatedRoles.DelegatedReader, role);
}
[Fact]
public void ValidateGrant_ExplicitDelegatedAdmin_Accepted()
{
Assert.Null(DelegatedAdminManagementFunction.ValidateGrantRequest(
new GrantDelegatedAdminRequest { Upn = Upn, TenantId = TenantA, Role = Constants.DelegatedRoles.DelegatedAdmin },
out var role));
Assert.Equal(Constants.DelegatedRoles.DelegatedAdmin, role);
}
// --- Revoke: 404-on-missing-row vs. audit + disconnect ---
[Fact]
public async Task RevokeCore_MissingRow_NoAuditNoDisconnect()
{
var (fn, repo, audit, signalR) = BuildAdminFn();
repo.Setup(r => r.RemoveDelegatedAdminAsync(UpnLower, TenantA)).ReturnsAsync(false);
var removed = await fn.RevokeCoreAsync(Upn, TenantA, "ga@vendor.example");
Assert.False(removed);
VerifyNoAudit(audit);
Assert.Empty(signalR.DisconnectedUsers);
}
[Fact]
public async Task RevokeCore_RemovedRow_AuditsAndDisconnectsLowercasedUpn()
{
var (fn, repo, audit, signalR) = BuildAdminFn();
repo.Setup(r => r.RemoveDelegatedAdminAsync(UpnLower, TenantA)).ReturnsAsync(true);
var removed = await fn.RevokeCoreAsync(Upn, TenantA, "ga@vendor.example");
Assert.True(removed);
// SignalR user ids are lowercased UPNs — a mixed-case route param must still hit the user's connections.
Assert.Equal(new[] { UpnLower }, signalR.DisconnectedUsers);
audit.Verify(a => a.LogAuditEntryAsync(
TenantA, "DELETE", "DelegatedAdmin", UpnLower, "ga@vendor.example", null), Times.Once);
}
// --- Disable is a revocation too; enable never disconnects ---
[Fact]
public async Task SetEnabledCore_MissingRow_NoAuditNoDisconnect()
{
var (fn, repo, audit, signalR) = BuildAdminFn();
repo.Setup(r => r.SetDelegatedAdminEnabledAsync(UpnLower, TenantA, false)).ReturnsAsync(false);
var ok = await fn.SetEnabledCoreAsync(Upn, TenantA, isEnabled: false, "ga@vendor.example");
Assert.False(ok);
VerifyNoAudit(audit);
Assert.Empty(signalR.DisconnectedUsers);
}
[Fact]
public async Task SetEnabledCore_Disable_Disconnects()
{
var (fn, repo, _, signalR) = BuildAdminFn();
repo.Setup(r => r.SetDelegatedAdminEnabledAsync(UpnLower, TenantA, false)).ReturnsAsync(true);
var ok = await fn.SetEnabledCoreAsync(Upn, TenantA, isEnabled: false, "ga@vendor.example");
Assert.True(ok);
Assert.Equal(new[] { UpnLower }, signalR.DisconnectedUsers);
}
[Fact]
public async Task SetEnabledCore_Enable_DoesNotDisconnect()
{
var (fn, repo, _, signalR) = BuildAdminFn();
repo.Setup(r => r.SetDelegatedAdminEnabledAsync(UpnLower, TenantA, true)).ReturnsAsync(true);
var ok = await fn.SetEnabledCoreAsync(Upn, TenantA, isEnabled: true, "ga@vendor.example");
Assert.True(ok);
Assert.Empty(signalR.DisconnectedUsers);
}
// --- Group unassign: 404-on-not-assigned vs. disconnect + per-tenant audit ---
[Fact]
public async Task UnassignCore_NotAssigned_NoAuditNoDisconnect()
{
var (fn, _, audit, signalR) = BuildGroupFn(); // default: no assignments for the UPN
var unassigned = await fn.UnassignCoreAsync(GroupId, Upn, "ga@vendor.example");
Assert.False(unassigned);
VerifyNoAudit(audit);
Assert.Empty(signalR.DisconnectedUsers);
}
[Fact]
public async Task UnassignCore_Assigned_DisconnectsAndAuditsPerGroupTenant()
{
var (fn, repo, audit, signalR) = BuildGroupFn();
repo.Setup(r => r.GetTenantGroupAsync(GroupId)).ReturnsAsync(new TenantGroup
{
GroupId = GroupId,
Name = "MSP Customers",
TenantIds = new List<string> { TenantA, TenantB },
AssigneeCount = 1,
});
repo.Setup(r => r.GetGroupAssignmentsForUpnAsync(UpnLower))
.ReturnsAsync(new List<TenantGroupAssignment> { new() { Upn = UpnLower, GroupId = GroupId } });
repo.Setup(r => r.UnassignGroupAsync(UpnLower, GroupId)).ReturnsAsync(true);
var unassigned = await fn.UnassignCoreAsync(GroupId, Upn, "ga@vendor.example");
Assert.True(unassigned);
Assert.Equal(new[] { UpnLower }, signalR.DisconnectedUsers);
// Lost access is audited under EACH of the group's tenants (customer-visible trail).
audit.Verify(a => a.LogAuditEntryAsync(
TenantA, "DELETE", "DelegatedGroupAccess", UpnLower, "ga@vendor.example",
It.IsAny<Dictionary<string, string>?>()), Times.Once);
audit.Verify(a => a.LogAuditEntryAsync(
TenantB, "DELETE", "DelegatedGroupAccess", UpnLower, "ga@vendor.example",
It.IsAny<Dictionary<string, string>?>()), Times.Once);
}
// --- Group delete: every (former) assignee's streams are cut ---
[Fact]
public async Task DeleteGroupCore_DisconnectsEveryAssignee_AndAuditsPerTenant()
{
var (fn, repo, audit, signalR) = BuildGroupFn();
var assignees = new List<TenantGroupAssignment>
{
new() { Upn = "one@partner.example", GroupId = GroupId },
new() { Upn = "two@partner.example", GroupId = GroupId },
};
repo.Setup(r => r.GetTenantGroupAsync(GroupId)).ReturnsAsync(new TenantGroup
{
GroupId = GroupId,
Name = "MSP Customers",
TenantIds = new List<string> { TenantA },
AssigneeCount = assignees.Count,
});
repo.Setup(r => r.GetGroupAssigneesAsync(GroupId)).ReturnsAsync(assignees);
repo.Setup(r => r.DeleteTenantGroupAsync(GroupId)).ReturnsAsync(true);
await fn.DeleteGroupCoreAsync(GroupId, "ga@vendor.example");
Assert.Equal(new[] { "one@partner.example", "two@partner.example" }, signalR.DisconnectedUsers);
audit.Verify(a => a.LogAuditEntryAsync(
TenantA, "DELETE", "DelegatedGroupAccess", "*", "ga@vendor.example",
It.IsAny<Dictionary<string, string>?>()), Times.Once);
}
}
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.Linq;
using System.Threading.Tasks;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Pagination;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
@@ -43,7 +44,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
["UserId"] = Truncate(entry.UserId, 128),
["Message"] = Truncate(entry.Message, 512),
["Details"] = Truncate(entry.Details, 4096),
["Timestamp"] = entry.Timestamp,
// "Timestamp" is a reserved system property (supplied values are ignored and
// any row rewrite resets it), so the event time needs its own column.
[BusinessTimestamp.OccurredUtcColumn] = entry.Timestamp,
};
await _table.UpsertEntityAsync(entity);
@@ -168,15 +171,16 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
if (!string.IsNullOrEmpty(lastRowKey))
clauses.Add($"RowKey gt '{lastRowKey!.Replace("'", "''")}'");
if (dateFrom.HasValue)
clauses.Add($"Timestamp ge datetime'{ToUtc(dateFrom.Value):o}'");
clauses.Add(BusinessTimestamp.OpsDateFromClause(ToUtc(dateFrom.Value)));
if (dateTo.HasValue)
clauses.Add($"Timestamp le datetime'{ToUtc(dateTo.Value):o}'");
clauses.Add(BusinessTimestamp.OpsDateToClause(ToUtc(dateTo.Value)));
return string.Join(" and ", clauses);
}
// Filters on the system-managed Timestamp; user-defined "Timestamp" property
// mirrors it on insert via SaveOpsEventAsync.
private static string? BuildFilter(string? category, DateTime? dateFrom, DateTime? dateTo)
// Date windows filter on the RowKey (reverse-tick, index-backed): the system
// Timestamp is reset by storage migrations, and an OccurredUtc property filter
// would exclude rows written before that column existed.
internal static string? BuildFilter(string? category, DateTime? dateFrom, DateTime? dateTo)
{
var clauses = new List<string>();
if (!string.IsNullOrEmpty(category))
@@ -185,11 +189,11 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
}
if (dateFrom.HasValue)
{
clauses.Add($"Timestamp ge datetime'{ToUtc(dateFrom.Value):o}'");
clauses.Add(BusinessTimestamp.OpsDateFromClause(ToUtc(dateFrom.Value)));
}
if (dateTo.HasValue)
{
clauses.Add($"Timestamp le datetime'{ToUtc(dateTo.Value):o}'");
clauses.Add(BusinessTimestamp.OpsDateToClause(ToUtc(dateTo.Value)));
}
return clauses.Count == 0 ? null : string.Join(" and ", clauses);
}
@@ -201,27 +205,30 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
{
var deleted = 0;
await foreach (var entity in _table.QueryAsync<TableEntity>())
// Server-side RowKey range instead of a full-table scan over the system
// Timestamp: every ops RowKey is a reverse-tick, so age is index-derivable,
// and the system Timestamp lies after storage migrations (it would freeze
// deletions for one retention period, then drop the whole pre-migration
// corpus in a single run).
var filter = BusinessTimestamp.OpsRetentionClause(cutoff);
await foreach (var entity in _table.QueryAsync<TableEntity>(
filter: filter, select: new[] { "PartitionKey", "RowKey" }))
{
var timestamp = entity.GetDateTimeOffset("Timestamp")?.UtcDateTime ?? DateTime.MinValue;
if (timestamp < cutoff)
try
{
try
{
await _table.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
deleted++;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to delete ops event {PK}/{RK}", entity.PartitionKey, entity.RowKey);
}
await _table.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
deleted++;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to delete ops event {PK}/{RK}", entity.PartitionKey, entity.RowKey);
}
}
return deleted;
}
private static OpsEventEntry MapToEntry(TableEntity entity)
internal static OpsEventEntry MapToEntry(TableEntity entity)
{
return new OpsEventEntry
{
@@ -233,10 +240,22 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
UserId = entity.GetString("UserId"),
Message = entity.GetString("Message") ?? string.Empty,
Details = entity.GetString("Details"),
Timestamp = entity.GetDateTimeOffset("Timestamp")?.UtcDateTime ?? DateTime.MinValue,
Timestamp = ResolveTimestamp(entity),
};
}
// Business time: OccurredUtc column → reverse-tick RowKey decode → system
// Timestamp (last resort; a row rewrite — e.g. storage migration — resets it).
internal static DateTime ResolveTimestamp(TableEntity entity)
{
var occurred = BusinessTimestamp.GetUtcDateTime(entity, BusinessTimestamp.OccurredUtcColumn);
if (occurred.HasValue)
return occurred.Value;
if (BusinessTimestamp.TryDecodeOpsRowKey(entity.RowKey, out var decoded))
return decoded;
return BusinessTimestamp.GetUtcDateTime(entity, "Timestamp") ?? DateTime.MinValue;
}
private static string? Truncate(string? value, int maxLength)
{
if (value == null) return null;
@@ -0,0 +1,103 @@
using System.Net;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Admin
{
/// <summary>
/// Global-Admin trigger for the one-time OccurredUtc backfill on AuditLogs / OpsEvents
/// (business timestamps lost to the system-Timestamp reset of the 2026-07-18 storage
/// migration; values are recovered from the reverse-tick RowKeys). DRY-RUN BY DEFAULT —
/// a real run requires the explicit <c>dryRun=false</c> query parameter, so the operator
/// always sees counts and samples before anything is written.
/// </summary>
public class BackfillOccurredUtcFunction
{
private readonly ILogger<BackfillOccurredUtcFunction> _logger;
private readonly OccurredUtcBackfillService _service;
private const int DefaultMaxRows = 250;
private const int MaxMaxRows = 1000;
public BackfillOccurredUtcFunction(
ILogger<BackfillOccurredUtcFunction> logger,
OccurredUtcBackfillService service)
{
_logger = logger;
_service = service;
}
/// <summary>
/// POST /api/maintenance/backfill-occurred-utc
/// Query parameters:
/// - table: "audit" | "ops" (required).
/// - dryRun: "false" to actually write; anything else (or absent) reports only.
/// - maxRows: rows examined per invocation (default 250, max 1000). Follow
/// nextContinuation to continue; null nextContinuation = table done.
/// - continuation: opaque token from the previous response (URL-encode it).
/// </summary>
[Function("BackfillOccurredUtc")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "maintenance/backfill-occurred-utc")] HttpRequestData req,
FunctionContext context)
{
// Authentication + GlobalAdminOnly authorization enforced by PolicyEnforcementMiddleware
var userEmail = TenantHelper.GetUserIdentifier(req);
var table = (req.Query["table"] ?? string.Empty).Trim().ToLowerInvariant();
if (table != OccurredUtcBackfillService.TableAudit && table != OccurredUtcBackfillService.TableOps)
{
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
await bad.WriteAsJsonAsync(new
{
error = $"table must be '{OccurredUtcBackfillService.TableAudit}' or '{OccurredUtcBackfillService.TableOps}'"
});
return bad;
}
var dryRun = !string.Equals(req.Query["dryRun"], "false", StringComparison.OrdinalIgnoreCase);
var continuation = req.Query["continuation"];
var maxRows = DefaultMaxRows;
if (!string.IsNullOrEmpty(req.Query["maxRows"]))
{
if (!int.TryParse(req.Query["maxRows"], out maxRows) || maxRows < 1)
{
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
await bad.WriteAsJsonAsync(new { error = $"maxRows must be a positive integer (max {MaxMaxRows})" });
return bad;
}
maxRows = Math.Min(maxRows, MaxMaxRows);
}
_logger.LogInformation(
"OccurredUtc backfill requested by {User}: table={Table} dryRun={DryRun} maxRows={MaxRows} hasContinuation={HasContinuation}",
userEmail, table, dryRun, maxRows, !string.IsNullOrEmpty(continuation));
try
{
var result = await _service.RunAsync(table, dryRun, maxRows, continuation);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new
{
success = true,
result,
triggeredBy = userEmail,
triggeredAt = DateTime.UtcNow,
});
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "OccurredUtc backfill run failed");
var error = req.CreateResponse(HttpStatusCode.InternalServerError);
await error.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
return error;
}
}
}
}
@@ -64,19 +64,9 @@ public class DelegatedAdminManagementFunction
var currentUpn = context.GetRequestContext().UserPrincipalName;
var body = await req.ReadFromJsonAsync<GrantDelegatedAdminRequest>();
if (body == null || string.IsNullOrWhiteSpace(body.Upn) || string.IsNullOrWhiteSpace(body.TenantId))
return await Bad(req, "upn and tenantId are required");
// Mirror the tenant-group path: a mistyped tenantId must not create a garbage scope entry
// that pollutes the allow-list and seeds a per-tenant audit trail keyed on the typo.
if (!Guid.TryParse(body.TenantId, out _))
return await Bad(req, "a valid tenantId (GUID) is required");
// Default to the least-privileged role; reject anything we don't recognize (fail-closed) so a typo
// can never silently widen access.
var role = string.IsNullOrWhiteSpace(body.Role) ? Constants.DelegatedRoles.DelegatedReader : body.Role;
if (role != Constants.DelegatedRoles.DelegatedReader && role != Constants.DelegatedRoles.DelegatedAdmin)
return await Bad(req, $"role must be '{Constants.DelegatedRoles.DelegatedReader}' or '{Constants.DelegatedRoles.DelegatedAdmin}'");
var validationError = ValidateGrantRequest(body, out var role);
if (validationError != null)
return await Bad(req, validationError);
// Edition note: TARGET tenants may be any edition — an MSP on Enterprise may manage Community
// customers. The Enterprise requirement applies to the delegated admin's HOME tenant and is
@@ -84,7 +74,7 @@ public class DelegatedAdminManagementFunction
// be checked here: at grant time only the UPN is known, and UPN-domain → tenant mapping is not
// reliable (multi-domain tenants). Grants to non-Enterprise-homed admins are simply inert.
var entry = await _delegatedAdminService.UpsertAsync(
body.Upn, body.TenantId, role,
body!.Upn, body.TenantId, role,
Constants.DelegatedStatus.Active, Constants.DelegatedSource.OperatorGranted, currentUpn ?? "");
await _maintenanceRepo.LogAuditEntryAsync(
@@ -112,10 +102,7 @@ public class DelegatedAdminManagementFunction
string upn, string tenantId, FunctionContext context)
{
var currentUpn = context.GetRequestContext().UserPrincipalName;
// The service reports whether a row was actually deleted — a mistyped upn/tenantId must not
// 200 and write a false customer-visible "access removed" audit row while the real grant
// stays live (mirrors the tenant-group revoke semantics).
var removed = await _delegatedAdminService.RemoveAsync(upn, tenantId);
var removed = await RevokeCoreAsync(upn, tenantId, currentUpn);
if (!removed)
{
var notFound = req.CreateResponse(HttpStatusCode.NotFound);
@@ -123,20 +110,34 @@ public class DelegatedAdminManagementFunction
return notFound;
}
await _maintenanceRepo.LogAuditEntryAsync(
tenantId.ToLowerInvariant(), "DELETE", "DelegatedAdmin", upn.ToLowerInvariant(), currentUpn ?? "");
// Enforcement: cut the revoked caller's already-joined live streams — group authorization is
// join-time only, so without this they keep receiving tenant telemetry until the connection drops.
await _signalRService.DisconnectUserAsync(upn.ToLowerInvariant());
_logger.LogInformation("Delegated revoke: {Upn} -> {TenantId} by {By}", upn, tenantId, currentUpn);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { message = "Delegated assignment revoked" });
return response;
}
/// <summary>
/// Testable core of the revoke flow (no HTTP plumbing — see GetAllBlockedDevicesFunctionTests for the
/// convention). Returns false when no assignment row existed (caller 404s): a mistyped upn/tenantId
/// must not 200 and write a false customer-visible "access removed" audit row while the real grant
/// stays live (mirrors the tenant-group revoke semantics). On a real removal writes the audit entry
/// and cuts the revoked UPN's live streams — group authorization is join-time only, so without the
/// kick they keep receiving tenant telemetry until the connection drops.
/// </summary>
internal async Task<bool> RevokeCoreAsync(string upn, string tenantId, string? currentUpn)
{
var removed = await _delegatedAdminService.RemoveAsync(upn, tenantId);
if (!removed)
return false;
await _maintenanceRepo.LogAuditEntryAsync(
tenantId.ToLowerInvariant(), "DELETE", "DelegatedAdmin", upn.ToLowerInvariant(), currentUpn ?? "");
await _signalRService.DisconnectUserAsync(upn.ToLowerInvariant());
_logger.LogInformation("Delegated revoke: {Upn} -> {TenantId} by {By}", upn, tenantId, currentUpn);
return true;
}
/// <summary>PATCH /api/global/delegated-admins/{upn}/{tenantId}/enable — re-enable a disabled assignment. GlobalAdminOnly.</summary>
[Function("EnableDelegatedAdmin")]
[Authorize]
@@ -157,7 +158,7 @@ public class DelegatedAdminManagementFunction
HttpRequestData req, string upn, string tenantId, bool isEnabled, FunctionContext context)
{
var currentUpn = context.GetRequestContext().UserPrincipalName;
var ok = await _delegatedAdminService.SetEnabledAsync(upn, tenantId, isEnabled);
var ok = await SetEnabledCoreAsync(upn, tenantId, isEnabled, currentUpn);
if (!ok)
{
var notFound = req.CreateResponse(HttpStatusCode.NotFound);
@@ -165,20 +166,56 @@ public class DelegatedAdminManagementFunction
return notFound;
}
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { message = isEnabled ? "Delegated assignment enabled" : "Delegated assignment disabled" });
return response;
}
/// <summary>
/// Testable core of enable/disable (no HTTP plumbing). Returns false when no assignment row
/// existed (caller 404s — no audit, no disconnect). Disable is a revocation too, so it cuts the
/// UPN's already-joined live streams (see <see cref="RevokeCoreAsync"/>); enable never disconnects.
/// </summary>
internal async Task<bool> SetEnabledCoreAsync(string upn, string tenantId, bool isEnabled, string? currentUpn)
{
var ok = await _delegatedAdminService.SetEnabledAsync(upn, tenantId, isEnabled);
if (!ok)
return false;
await _maintenanceRepo.LogAuditEntryAsync(
tenantId.ToLowerInvariant(), "UPDATE", "DelegatedAdmin", upn.ToLowerInvariant(), currentUpn ?? "",
new Dictionary<string, string> { { "IsEnabled", isEnabled.ToString() } });
// Disable is a revocation too — cut already-joined live streams (see RevokeDelegatedAdmin).
if (!isEnabled)
await _signalRService.DisconnectUserAsync(upn.ToLowerInvariant());
_logger.LogInformation("Delegated {Action}: {Upn} -> {TenantId} by {By}",
isEnabled ? "enable" : "disable", upn, tenantId, currentUpn);
return true;
}
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { message = isEnabled ? "Delegated assignment enabled" : "Delegated assignment disabled" });
return response;
/// <summary>
/// Pure grant-request validation (testable seam). Returns the 400 error message, or null when valid —
/// then <paramref name="role"/> carries the effective role. A mistyped tenantId must not create a
/// garbage scope entry that pollutes the allow-list and seeds a per-tenant audit trail keyed on the
/// typo; the role defaults to least-privileged and unknown roles are rejected (fail-closed) so a typo
/// can never silently widen access.
/// </summary>
internal static string? ValidateGrantRequest(GrantDelegatedAdminRequest? body, out string role)
{
role = Constants.DelegatedRoles.DelegatedReader;
if (body == null || string.IsNullOrWhiteSpace(body.Upn) || string.IsNullOrWhiteSpace(body.TenantId))
return "upn and tenantId are required";
if (!Guid.TryParse(body.TenantId, out _))
return "a valid tenantId (GUID) is required";
role = string.IsNullOrWhiteSpace(body.Role) ? Constants.DelegatedRoles.DelegatedReader : body.Role!;
if (role != Constants.DelegatedRoles.DelegatedReader && role != Constants.DelegatedRoles.DelegatedAdmin)
return $"role must be '{Constants.DelegatedRoles.DelegatedReader}' or '{Constants.DelegatedRoles.DelegatedAdmin}'";
return null;
}
private static async Task<HttpResponseData> Bad(HttpRequestData req, string error)
@@ -112,15 +112,26 @@ public class TenantGroupManagementFunction
string groupId, FunctionContext context)
{
var currentUpn = context.GetRequestContext().UserPrincipalName;
await DeleteGroupCoreAsync(groupId, currentUpn);
// Capture tenants + assignees BEFORE the cascade delete removes them, so we can audit the
// bulk access removal under each affected tenant and cut every assignee's live streams.
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { message = "Group deleted" });
return response;
}
/// <summary>
/// Testable core of the group cascade delete (no HTTP plumbing). Captures tenants + assignees
/// BEFORE the delete removes them, so the bulk access removal is audited under each affected
/// tenant and EVERY (former) assignee's live streams are cut — group authorization is join-time
/// only, so without the kick they keep receiving the managed tenants' telemetry until their
/// connection drops.
/// </summary>
internal async Task DeleteGroupCoreAsync(string groupId, string? currentUpn)
{
var group = await _delegatedAdminService.GetGroupAsync(groupId);
var assignees = await _delegatedAdminService.GetGroupAssigneesAsync(groupId);
await _delegatedAdminService.DeleteGroupAsync(groupId);
// Enforcement: group authorization is join-time only — without the kick, every (former)
// assignee keeps receiving the managed tenants' live telemetry until their connection drops.
foreach (var assignee in assignees)
await _signalRService.DisconnectUserAsync(assignee.Upn);
@@ -137,10 +148,6 @@ public class TenantGroupManagementFunction
}
_logger.LogInformation("Tenant group deleted: {GroupId} by {By}", groupId, currentUpn);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { message = "Group deleted" });
return response;
}
/// <summary>POST /api/global/tenant-groups/{groupId}/tenants — add a tenant. GlobalAdminOnly. Body: { "tenantId": "&lt;guid&gt;" }.</summary>
@@ -271,17 +278,31 @@ public class TenantGroupManagementFunction
string groupId, string upn, FunctionContext context)
{
var currentUpn = context.GetRequestContext().UserPrincipalName;
var unassigned = await UnassignCoreAsync(groupId, upn, currentUpn);
if (!unassigned)
return await NotFound(req);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { message = "Unassigned from group" });
return response;
}
/// <summary>
/// Testable core of the unassign flow (no HTTP plumbing). Returns false when the UPN was not
/// actually assigned (caller 404s) — so a mistyped UPN can't 200 and write false "access removed"
/// audit rows, and no disconnect fires. On a real unassign cuts the UPN's already-joined live
/// streams (join-time-only authz) and audits the lost access under each of the group's tenants.
/// </summary>
internal async Task<bool> UnassignCoreAsync(string groupId, string upn, string? currentUpn)
{
var normalizedUpn = upn.ToLowerInvariant();
// Read the group's tenants (unchanged by unassign) so we can audit the UPN's lost access per tenant.
var group = await _delegatedAdminService.GetGroupAsync(groupId);
// The service returns false when the UPN was not actually assigned — so a mistyped UPN can't 200 and
// write false "access removed" audit rows. Only audit after a real unassign.
var unassigned = await _delegatedAdminService.UnassignGroupAsync(normalizedUpn, groupId);
if (!unassigned)
return await NotFound(req);
return false;
// Enforcement: cut the unassigned caller's already-joined live streams (join-time-only authz).
await _signalRService.DisconnectUserAsync(normalizedUpn);
if (group != null)
@@ -296,10 +317,7 @@ public class TenantGroupManagementFunction
}
_logger.LogInformation("Group {GroupId} unassigned from {Upn} by {By}", groupId, normalizedUpn, currentUpn);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { message = "Unassigned from group" });
return response;
return true;
}
/// <summary>Logs one audit entry per tenant (the customer-visible "who can read my tenant" trail).</summary>
@@ -270,7 +270,7 @@ namespace AutopilotMonitor.Functions.Functions.Config
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new AgentConfigResponse
{
ConfigVersion = 33, // Agent endpoint migration on the config channel (MigrateToApiBaseUrl; live-fetch-only, allowlist-validated, one hop, kill wins)
ConfigVersion = 35, // MDM reboot-policy watcher (mdm_policy_reboot_required / esp_reboot_coalescing)
UploadIntervalSeconds = 10,
SelfDestructOnComplete = tenantConfig.SelfDestructOnComplete ?? true,
KeepLogFile = tenantConfig.KeepLogFile ?? false,
@@ -314,7 +314,7 @@ namespace AutopilotMonitor.Functions.Functions.Config
DeviceBlocked = killVerdict.IsBlocked,
DeviceKillSignal = killVerdict.IsKill,
UnblockAt = killVerdict.UnblockAt,
MigrateToApiBaseUrl = migrateTarget!,
MigrateToApiBaseUrl = migrateTarget,
});
return response;
@@ -10,6 +10,7 @@ using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using AutopilotMonitor.Shared.Pagination;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
@@ -17,15 +18,23 @@ namespace AutopilotMonitor.Functions.Functions.Config
{
public class GetAllTenantConfigurationsFunction
{
// TTL for the "this managed id has NO config row in any casing" verdict a rescue scan proved.
// Bounds how long a tenant onboarded moments after that scan stays absent from a delegated
// caller's config/all (matches the positive config cache TTL in TenantConfigurationService).
private static readonly TimeSpan MissingConfigNegativeCacheDuration = TimeSpan.FromMinutes(5);
private readonly ILogger<GetAllTenantConfigurationsFunction> _logger;
private readonly TenantConfigurationService _configService;
private readonly IMemoryCache _cache;
public GetAllTenantConfigurationsFunction(
ILogger<GetAllTenantConfigurationsFunction> logger,
TenantConfigurationService configService)
TenantConfigurationService configService,
IMemoryCache cache)
{
_logger = logger;
_configService = configService;
_cache = cache;
}
[Function("GetAllTenantConfigurations")]
@@ -73,9 +82,21 @@ namespace AutopilotMonitor.Functions.Functions.Config
// (agent paths auto-create rows with the caller-supplied id verbatim) — the same caveat
// the delegated sessions fan-out handles case-insensitively. A point-read is exact-case,
// so any id it missed gets ONE scan+filter rescue pass (the pre-point-read behavior).
// Only stale grants or case-variant rows pay for it; the common path stays read-bounded.
// An id a previous scan already proved missing in ANY casing is negative-cached and
// skipped for the TTL — otherwise a single stale grant (offboarded / never-onboarded
// tenant) would make EVERY config/all call pay a full table scan.
if (subset.Count < managedIds.Count)
subset = MergeRescuedConfigs(managedIds, subset, await _configService.GetAllConfigurationsAsync());
{
var unproven = MissingManagedIds(managedIds, subset)
.Where(id => !_cache.TryGetValue(MissingConfigCacheKey(id), out _))
.ToList();
if (unproven.Count > 0)
{
subset = MergeRescuedConfigs(managedIds, subset, await _configService.GetAllConfigurationsAsync());
foreach (var id in MissingManagedIds(unproven, subset))
_cache.Set(MissingConfigCacheKey(id), true, MissingConfigNegativeCacheDuration);
}
}
_logger.LogInformation("GetAllTenantConfigurations (delegated subset, {Count} tenants) by {User}",
subset.Count, userIdentifier);
@@ -177,6 +198,23 @@ namespace AutopilotMonitor.Functions.Functions.Config
IEnumerable<TenantConfiguration?> reads)
=> reads.Where(c => c != null).Select(c => c!).ToList();
/// <summary>
/// The managed ids not present (case-insensitively) in <paramref name="configs"/> — before the
/// rescue scan these are the point-read misses, after it the proven-missing ids to negative-cache.
/// Pure + testable.
/// </summary>
internal static List<string> MissingManagedIds(
IEnumerable<string> managedIds, IReadOnlyCollection<TenantConfiguration> configs)
{
var have = new HashSet<string>(configs.Select(c => c.TenantId), StringComparer.OrdinalIgnoreCase);
return managedIds.Where(id => !have.Contains(id)).ToList();
}
/// <summary>Negative-cache key for a managed id a rescue scan proved has no config row.
/// Lowercased so case variants of the same grant share one verdict.</summary>
internal static string MissingConfigCacheKey(string tenantId)
=> $"tenant-config-missing:{tenantId.ToLowerInvariant()}";
/// <summary>
/// Case-variant rescue for managed ids the exact-case point-reads missed: from the full config scan,
/// add every config whose TenantId matches a managed id case-insensitively and is not already in the
@@ -102,17 +102,19 @@ namespace AutopilotMonitor.Functions.Functions.Infrastructure
// Authentication + GlobalAdminOnly authorization enforced by PolicyEnforcementMiddleware
// Perform comprehensive health checks
var healthCheckResult = await _healthCheckService.PerformAllChecksAsync();
// SignalR Quota check exposes the subscription/resource-group path of the
// backing SignalR resource, and Poison Queues exposes internal async-worker
// queue topology — restrict both to Global Admins. Non-GA callers
// (Tenant Admins, Operators) get the rest of the report unchanged.
// SignalR Quota exposes the subscription/resource-group path of the backing
// SignalR resource, Poison Queues exposes internal async-worker queue topology,
// and endpoint URLs in the check details expose infrastructure hostnames —
// restrict all of that to Global Admins. Non-GA callers (Tenant Admins,
// Operators) get the rest of the report unchanged.
// The route is registered as AuthenticatedUser, so PolicyEnforcementMiddleware
// does NOT compute IsGlobalAdmin; resolve it directly via GlobalAdminService.
var requestCtx = context.GetRequestContext();
var isGlobalAdmin = await _globalAdminService.IsGlobalAdminAsync(requestCtx.UserPrincipalName);
// Perform comprehensive health checks (endpoint URLs only for Global Admins)
var healthCheckResult = await _healthCheckService.PerformAllChecksAsync(includeEndpointUrls: isGlobalAdmin);
var visibleChecks = isGlobalAdmin
? healthCheckResult.Checks
: healthCheckResult.Checks
@@ -146,10 +148,14 @@ namespace AutopilotMonitor.Functions.Functions.Infrastructure
/// </summary>
[Function("McpHealthCheck")]
public async Task<HttpResponseData> GetMcpHealthCheck(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "health/mcp")] HttpRequestData req)
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "health/mcp")] HttpRequestData req,
FunctionContext context)
{
// Authentication enforced by PolicyEnforcementMiddleware (AuthenticatedUser).
var check = await _healthCheckService.CheckMcpServerAsync();
// The MCP server URL is infrastructure topology — include it only for Global Admins.
var requestCtx = context.GetRequestContext();
var isGlobalAdmin = await _globalAdminService.IsGlobalAdminAsync(requestCtx.UserPrincipalName);
var check = await _healthCheckService.CheckMcpServerAsync(includeEndpointUrl: isGlobalAdmin);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new
@@ -64,6 +64,14 @@ namespace AutopilotMonitor.Functions.Functions.Rules
return badRequest;
}
var scopeError = ValidateScopeAndEmitMode(rule);
if (scopeError != null)
{
var badRequest = req.CreateResponse(HttpStatusCode.BadRequest);
await badRequest.WriteAsJsonAsync(new { success = false, message = scopeError });
return badRequest;
}
try
{
var success = await _ruleService.CreateRuleAsync(tenantId, rule);
@@ -108,12 +116,71 @@ namespace AutopilotMonitor.Functions.Functions.Rules
rule.RuleId = ruleId;
var scopeError = ValidateScopeAndEmitMode(rule);
if (scopeError != null)
{
var badRequest = req.CreateResponse(HttpStatusCode.BadRequest);
await badRequest.WriteAsJsonAsync(new { success = false, message = scopeError });
return badRequest;
}
var success = await _ruleService.UpdateRuleAsync(tenantId, rule);
var response = req.CreateResponse(success ? HttpStatusCode.OK : HttpStatusCode.InternalServerError);
await response.WriteAsJsonAsync(new { success, message = success ? "Rule updated" : "Failed to update rule" });
return response;
}
/// <summary>
/// Selectable phase-scope tokens: the <see cref="EnrollmentPhase"/> enum NAMES from
/// Start(0) through Complete(7). Explicit name list (not Enum.TryParse) so numeric
/// tokens ("4") and the non-selectable Unknown/Failed members are rejected — the
/// canonical vocabulary the agent, schema, and portal share.
/// </summary>
private static readonly HashSet<string> ValidScopePhases = new(StringComparer.OrdinalIgnoreCase)
{
nameof(EnrollmentPhase.Start),
nameof(EnrollmentPhase.DevicePreparation),
nameof(EnrollmentPhase.DeviceSetup),
nameof(EnrollmentPhase.AppsDevice),
nameof(EnrollmentPhase.AccountSetup),
nameof(EnrollmentPhase.AppsUser),
nameof(EnrollmentPhase.FinalizingSetup),
nameof(EnrollmentPhase.Complete),
};
/// <summary>
/// Validates the phase-scope + emit-mode fields on create/update.
/// Returns an error message for a 400 response, or null when valid.
/// Toggle-style partial payloads carry none of these fields and pass through.
/// </summary>
internal static string? ValidateScopeAndEmitMode(GatherRule rule)
{
var hasActivePhases = rule.ActivePhases != null && rule.ActivePhases.Count > 0;
var hasFromPhase = !string.IsNullOrEmpty(rule.ActiveFromPhase);
if (hasActivePhases && hasFromPhase)
return "activePhases and activeFromPhase are mutually exclusive — set only one.";
if (hasActivePhases)
{
foreach (var phase in rule.ActivePhases!)
{
if (string.IsNullOrEmpty(phase) || !ValidScopePhases.Contains(phase))
return $"Invalid phase '{phase}' in activePhases. Valid phases: {string.Join(", ", ValidScopePhases)}.";
}
}
if (hasFromPhase && !ValidScopePhases.Contains(rule.ActiveFromPhase!))
return $"Invalid activeFromPhase '{rule.ActiveFromPhase}'. Valid phases: {string.Join(", ", ValidScopePhases)}.";
if (!string.IsNullOrEmpty(rule.EmitMode)
&& !string.Equals(rule.EmitMode, "always", StringComparison.OrdinalIgnoreCase)
&& !string.Equals(rule.EmitMode, "on_change", StringComparison.OrdinalIgnoreCase))
return "emitMode must be \"always\" or \"on_change\".";
return null;
}
[Function("DeleteGatherRule")]
public async Task<HttpResponseData> DeleteRule(
[HttpTrigger(AuthorizationLevel.Anonymous, "delete", Route = "rules/gather/{ruleId}")] HttpRequestData req,
@@ -0,0 +1,154 @@
using System;
using System.Globalization;
using Azure.Data.Tables;
namespace AutopilotMonitor.Functions.Helpers
{
/// <summary>
/// Business-timestamp handling for tables whose rows historically relied on the
/// Azure Tables system <c>Timestamp</c> (= row write time). That system property is
/// server-managed: supplied values are ignored on write and every row rewrite
/// (storage migration, merge) resets it. The authoritative event time therefore
/// lives in (a) the explicit <see cref="OccurredUtcColumn"/> custom property on
/// rows written after its introduction, and (b) the time encoded in the RowKey
/// for older rows. Readers resolve OccurredUtc → RowKey decode → system Timestamp.
///
/// RowKey schemes decoded here:
/// AuditLogs: "!{MaxTicks - ticks:D19}_{guid:N}" (legacy pre-2026-05 rows are bare GUIDs — not decodable)
/// OpsEvents: "{MaxTicks - ticks:D19}"
/// Events: "{timestamp:yyyyMMddHHmmssfff}_{sequence:D10}"
/// </summary>
internal static class BusinessTimestamp
{
/// <summary>Custom column carrying the business event time. Survives migrations verbatim
/// (unlike the system Timestamp, which any row rewrite resets).</summary>
internal const string OccurredUtcColumn = "OccurredUtc";
internal const string AuditRowKeyPrefix = "!";
/// <summary>
/// '"' (0x22) is the character immediately after '!' (0x21). `RowKey lt '"'` confines a
/// range to time-encoded audit rows: legacy bare-GUID RowKeys start with a hex digit
/// (≥ '0', 0x30) and would otherwise match ANY lower bound of the form `RowKey ge '!…'`.
/// </summary>
internal const string AuditTimeEncodedUpperBound = "\"";
private const string EventRowKeyTimestampFormat = "yyyyMMddHHmmssfff";
/// <summary>
/// Reads a datetime column as UTC regardless of the SDK's materialized type —
/// service reads surface Edm.DateTime as either DateTimeOffset or DateTime
/// depending on payload shape (the pre-existing mappers already read both).
/// Unspecified kinds are treated as UTC (all stored values are UTC).
/// </summary>
internal static DateTime? GetUtcDateTime(TableEntity entity, string column)
{
if (!entity.TryGetValue(column, out var value) || value == null)
return null;
return value switch
{
DateTimeOffset dto => dto.UtcDateTime,
DateTime dt => dt.Kind == DateTimeKind.Local
? dt.ToUniversalTime()
: DateTime.SpecifyKind(dt, DateTimeKind.Utc),
_ => null,
};
}
// ===== RowKey decoders =====
/// <summary>Decodes an audit RowKey ("!{rev:D19}_{guid:N}") to the original UTC write time.
/// Returns false for legacy bare-GUID RowKeys and any other foreign format.</summary>
internal static bool TryDecodeAuditRowKey(string? rowKey, out DateTime utc)
{
utc = default;
if (rowKey == null || rowKey.Length < 21 || rowKey[0] != '!')
return false;
return TryDecodeReverseTicks(rowKey.AsSpan(1, 19), out utc);
}
/// <summary>Decodes an ops-event RowKey (bare "{rev:D19}") to the original UTC write time.</summary>
internal static bool TryDecodeOpsRowKey(string? rowKey, out DateTime utc)
{
utc = default;
if (rowKey == null || rowKey.Length != 19)
return false;
return TryDecodeReverseTicks(rowKey.AsSpan(), out utc);
}
/// <summary>Decodes an enrollment-event RowKey ("{yyyyMMddHHmmssfff}_{seq:D10}") to the
/// (sanitized) agent event time, millisecond precision, Kind=Utc.</summary>
internal static bool TryDecodeEventRowKeyPrefix(string? rowKey, out DateTime utc)
{
utc = default;
if (rowKey == null)
return false;
var separator = rowKey.IndexOf('_');
if (separator != 17)
return false;
if (!DateTime.TryParseExact(rowKey.Substring(0, 17), EventRowKeyTimestampFormat,
CultureInfo.InvariantCulture,
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out utc))
return false;
return true;
}
private static bool TryDecodeReverseTicks(ReadOnlySpan<char> revTickDigits, out DateTime utc)
{
utc = default;
foreach (var c in revTickDigits)
{
if (c < '0' || c > '9')
return false;
}
if (!long.TryParse(revTickDigits, NumberStyles.None, CultureInfo.InvariantCulture, out var rev))
return false;
var ticks = DateTime.MaxValue.Ticks - rev;
if (ticks < DateTime.MinValue.Ticks || ticks > DateTime.MaxValue.Ticks)
return false;
utc = new DateTime(ticks, DateTimeKind.Utc);
return true;
}
// ===== OData date-window clauses (RowKey ranges) =====
// Date windows filter on the RowKey rather than a property: rows missing a property are
// excluded from OData comparisons (so an OccurredUtc filter would drop every not-yet-
// backfilled row), and RowKey ranges are index-backed. String order equals numeric order
// because the reverse-tick is fixed-width D19. rev(t) = MaxTicks - t.Ticks, so larger
// timestamps have lexically smaller RowKeys (newest-first).
private static long Rev(DateTime utc) => DateTime.MaxValue.Ticks - utc.Ticks;
/// <summary>Audit rows with business time ≥ <paramref name="fromUtc"/> (inclusive):
/// rev ≤ rev(from) &lt; rev(from)+1. The exclusive upper bound also excludes all legacy
/// GUID rows (they sort after every '!' row).</summary>
internal static string AuditDateFromClause(DateTime fromUtc)
=> $"RowKey lt '{AuditRowKeyPrefix}{Rev(fromUtc) + 1:D19}'";
/// <summary>Audit rows with business time ≤ <paramref name="toUtc"/> (inclusive):
/// rev ≥ rev(to). The lower bound alone would match every legacy GUID row, hence the
/// mandatory time-encoded guard.</summary>
internal static string AuditDateToClause(DateTime toUtc)
=> $"RowKey ge '{AuditRowKeyPrefix}{Rev(toUtc):D19}' and RowKey lt '{AuditTimeEncodedUpperBound}'";
/// <summary>Audit rows strictly older than <paramref name="cutoffUtc"/> (retention):
/// rev ≥ rev(cutoff)+1 ⇔ ts &lt; cutoff. MUST keep the time-encoded guard — without it
/// this clause matches (and a retention sweep would delete) every legacy GUID row.</summary>
internal static string AuditRetentionClause(DateTime cutoffUtc)
=> $"RowKey ge '{AuditRowKeyPrefix}{Rev(cutoffUtc) + 1:D19}' and RowKey lt '{AuditTimeEncodedUpperBound}'";
/// <summary>Ops rows with business time ≥ <paramref name="fromUtc"/> (inclusive). Bare D19
/// RowKey (no suffix), so `le` is exact.</summary>
internal static string OpsDateFromClause(DateTime fromUtc)
=> $"RowKey le '{Rev(fromUtc):D19}'";
/// <summary>Ops rows with business time ≤ <paramref name="toUtc"/> (inclusive).</summary>
internal static string OpsDateToClause(DateTime toUtc)
=> $"RowKey ge '{Rev(toUtc):D19}'";
/// <summary>Ops rows strictly older than <paramref name="cutoffUtc"/> (retention):
/// `gt` not `ge` — a row at exactly the cutoff is kept, matching `Timestamp lt` semantics.</summary>
internal static string OpsRetentionClause(DateTime cutoffUtc)
=> $"RowKey gt '{Rev(cutoffUtc):D19}'";
}
}
@@ -252,6 +252,7 @@ builder.Services.AddHttpClient<GitHubRuleRepository>()
.AddPolicyHandler((sp, _) => sp.GetRequiredService<ResiliencePolicies>().ExternalDataApi);
builder.Services.AddSingleton<MaintenanceService>();
builder.Services.AddSingleton<LegacyReclassificationService>();
builder.Services.AddSingleton<OccurredUtcBackfillService>();
builder.Services.AddSingleton<
AutopilotMonitor.Functions.Services.Monitoring.IAzureMonitorMetricsReader,
AutopilotMonitor.Functions.Services.Monitoring.AzureMonitorMetricsReader>();
@@ -399,6 +399,9 @@ public static class EndpointAccessPolicyCatalog
// One-time retro-reconcile of misclassified historical sessions (audit 2026-07-16) —
// mutating, dry-run by default, GA only.
new("POST", "maintenance/reclassify-legacy", EndpointPolicy.GlobalAdminOnly),
// One-time OccurredUtc backfill on AuditLogs/OpsEvents (2026-07-18 migration reset
// the system Timestamps) — mutating, dry-run by default, GA only.
new("POST", "maintenance/backfill-occurred-utc", EndpointPolicy.GlobalAdminOnly),
// ── Critical-Table Backup (plan §PR1+§PR2) ─────────────────────────────
new("POST", "global/backups/trigger", EndpointPolicy.GlobalAdminOnly),
@@ -151,10 +151,38 @@ namespace AutopilotMonitor.Functions.Services
return await _ruleRepo.StoreRuleStateAsync(tenantId, rule.RuleId, new RuleState { Enabled = rule.Enabled });
}
// Custom rule. The portal toggle PUTs only { enabled, isBuiltIn, isCommunity } —
// full-storing that payload would wipe Title/Target/Trigger/Parameters. Merge
// toggle-style partials into the existing row; full payloads (edit flow) replace
// the row but keep the original CreatedAt.
var existing = (await _ruleRepo.GetGatherRulesAsync(tenantId))
.FirstOrDefault(r => r.RuleId == rule.RuleId && !r.IsBuiltIn && !r.IsCommunity);
var isTogglePartial = string.IsNullOrEmpty(rule.Title)
&& string.IsNullOrEmpty(rule.CollectorType)
&& string.IsNullOrEmpty(rule.Target);
if (isTogglePartial)
{
if (existing == null)
{
_logger.LogWarning(
"Toggle-style partial update for unknown custom gather rule {RuleId} (tenant {TenantId}); nothing to update",
rule.RuleId, tenantId);
return false;
}
existing.Enabled = rule.Enabled;
existing.UpdatedAt = DateTime.UtcNow;
return await _ruleRepo.StoreGatherRuleAsync(existing, tenantId);
}
// Ensure custom rules keep correct flags (the incoming payload may
// omit them, causing them to default to IsBuiltIn=true in the model).
rule.IsBuiltIn = false;
rule.IsCommunity = false;
if (existing != null)
rule.CreatedAt = existing.CreatedAt;
rule.UpdatedAt = DateTime.UtcNow;
return await _ruleRepo.StoreGatherRuleAsync(rule, tenantId);
}
@@ -413,7 +441,22 @@ namespace AutopilotMonitor.Functions.Services
&& a.Title == b.Title
&& a.CollectorType == b.CollectorType
&& a.Trigger == b.Trigger
&& a.TriggerPhase == b.TriggerPhase;
&& a.TriggerPhase == b.TriggerPhase
&& ActivePhasesEquivalent(a.ActivePhases, b.ActivePhases)
&& a.ActiveFromPhase == b.ActiveFromPhase
&& a.EmitMode == b.EmitMode;
}
/// <summary>
/// Null-tolerant ordered comparison for <see cref="GatherRule.ActivePhases"/>:
/// null and empty are both "unrestricted" and therefore equivalent, so pre-existing
/// DB rows (absent column → null) never diff against seeds (also null) — no reseed churn.
/// </summary>
private static bool ActivePhasesEquivalent(List<string>? a, List<string>? b)
{
var left = a ?? new List<string>();
var right = b ?? new List<string>();
return left.SequenceEqual(right, StringComparer.Ordinal);
}
}
}
@@ -34,9 +34,12 @@ public class HealthCheckService
}
/// <summary>
/// Performs all health checks and returns the results
/// Performs all health checks and returns the results.
/// <paramref name="includeEndpointUrls"/> adds the probed/backing endpoint URLs to the
/// per-check details — infrastructure topology, so callers must only enable it for
/// Global Admins.
/// </summary>
public async Task<HealthCheckResult> PerformAllChecksAsync()
public async Task<HealthCheckResult> PerformAllChecksAsync(bool includeEndpointUrls = false)
{
var result = new HealthCheckResult
{
@@ -50,9 +53,9 @@ public class HealthCheckService
// one slow check. It is exposed via its own endpoint (GET health/mcp) that the
// frontend fetches independently and folds into the card grid incrementally.
var checks = await Task.WhenAll(
CheckStorageBackendAsync(),
CheckProcessingBackendAsync(),
CheckAgentBinariesAsync(),
CheckStorageBackendAsync(includeEndpointUrls),
CheckProcessingBackendAsync(includeEndpointUrls),
CheckAgentBinariesAsync(includeEndpointUrls),
CheckSignalRQuotaAsync(),
CheckPoisonQueuesAsync()
);
@@ -73,7 +76,7 @@ public class HealthCheckService
/// <summary>
/// Checks Azure Table Storage connectivity by querying the admin configuration table
/// </summary>
private async Task<HealthCheck> CheckStorageBackendAsync()
private async Task<HealthCheck> CheckStorageBackendAsync(bool includeEndpointUrls)
{
var check = new HealthCheck
{
@@ -97,29 +100,62 @@ public class HealthCheckService
_logger.LogError(ex, "Storage backend health check failed");
}
if (includeEndpointUrls)
{
// Same resolution order as TableStorageService: account name (Managed Identity)
// is the production path, connection string is the local-dev fallback.
var accountName = _configuration["AzureStorageAccountName"];
check.Details = new Dictionary<string, object>
{
["Table endpoint"] = string.IsNullOrWhiteSpace(accountName)
? "connection string (local dev)"
: $"https://{accountName}.table.core.windows.net"
};
}
return check;
}
/// <summary>
/// Checks that the Azure Functions host is running
/// </summary>
private static Task<HealthCheck> CheckProcessingBackendAsync()
private static Task<HealthCheck> CheckProcessingBackendAsync(bool includeEndpointUrls)
{
var uptime = DateTime.UtcNow - System.Diagnostics.Process.GetCurrentProcess().StartTime.ToUniversalTime();
return Task.FromResult(new HealthCheck
var check = new HealthCheck
{
Name = "Processing Backend",
Description = "Application host process",
Status = "healthy",
Message = $"Host running (uptime: {(int)uptime.TotalMinutes}m {uptime.Seconds}s)"
});
};
if (includeEndpointUrls)
{
// WEBSITE_HOSTNAME is stamped by App Service and reflects the host actually
// serving this process — no hardcoded URL to go stale after a migration.
var hostname = Environment.GetEnvironmentVariable("WEBSITE_HOSTNAME");
if (!string.IsNullOrWhiteSpace(hostname))
{
check.Details = new Dictionary<string, object>
{
["API endpoint"] = $"https://{hostname}"
};
}
}
return Task.FromResult(check);
}
/// <summary>
/// Checks that the agent binaries (ZIP) and bootstrap script (PS1) are reachable in blob storage
/// Checks that the agent binaries (ZIP) and bootstrap script (PS1) are reachable on the
/// canonical download alias (Front Door → current blob origin) AND on the legacy blob
/// keepalive account that already-deployed customer bootstrap scripts still use.
/// Alias failure is unhealthy (new installs break); legacy failure is a warning
/// (existing customers break until they migrate to the alias).
/// </summary>
private async Task<HealthCheck> CheckAgentBinariesAsync()
internal async Task<HealthCheck> CheckAgentBinariesAsync(bool includeEndpointUrls)
{
var check = new HealthCheck
{
@@ -127,8 +163,20 @@ public class HealthCheckService
Description = "Agent download package availability"
};
var zipUrl = $"{Constants.AgentBlobBaseUrl}/{Constants.AgentZipFileName}";
var ps1Url = $"{Constants.AgentBlobBaseUrl}/Install-AutopilotMonitor.ps1";
var aliasZipUrl = $"{Constants.AgentDownloadBaseUrl}/{Constants.AgentZipFileName}";
var aliasPs1Url = $"{Constants.AgentDownloadBaseUrl}/{Constants.BootstrapScriptName}";
var legacyZipUrl = $"{Constants.AgentBlobBaseUrl}/{Constants.AgentZipFileName}";
var legacyPs1Url = $"{Constants.AgentBlobBaseUrl}/{Constants.BootstrapScriptName}";
if (includeEndpointUrls)
{
check.Details = new Dictionary<string, object>
{
["Agent ZIP"] = aliasZipUrl,
["Bootstrap script"] = aliasPs1Url,
["Legacy blob (keepalive)"] = Constants.AgentBlobBaseUrl
};
}
try
{
@@ -138,40 +186,48 @@ public class HealthCheckService
var sw = System.Diagnostics.Stopwatch.StartNew();
// HEAD requests to check availability without downloading
var zipRequest = new HttpRequestMessage(HttpMethod.Head, zipUrl);
var ps1Request = new HttpRequestMessage(HttpMethod.Head, ps1Url);
var results = await Task.WhenAll(
client.SendAsync(zipRequest),
client.SendAsync(ps1Request)
client.SendAsync(new HttpRequestMessage(HttpMethod.Head, aliasZipUrl)),
client.SendAsync(new HttpRequestMessage(HttpMethod.Head, aliasPs1Url)),
client.SendAsync(new HttpRequestMessage(HttpMethod.Head, legacyZipUrl)),
client.SendAsync(new HttpRequestMessage(HttpMethod.Head, legacyPs1Url))
);
sw.Stop();
var zipResponse = results[0];
var ps1Response = results[1];
var aliasZipOk = results[0].IsSuccessStatusCode;
var aliasPs1Ok = results[1].IsSuccessStatusCode;
var legacyZipOk = results[2].IsSuccessStatusCode;
var legacyPs1Ok = results[3].IsSuccessStatusCode;
var zipOk = zipResponse.IsSuccessStatusCode;
var ps1Ok = ps1Response.IsSuccessStatusCode;
var issues = new List<string>();
if (!aliasZipOk) issues.Add($"Agent ZIP (download alias): {(int)results[0].StatusCode}");
if (!aliasPs1Ok) issues.Add($"Bootstrap script (download alias): {(int)results[1].StatusCode}");
if (!legacyZipOk) issues.Add($"Agent ZIP (legacy blob): {(int)results[2].StatusCode}");
if (!legacyPs1Ok) issues.Add($"Bootstrap script (legacy blob): {(int)results[3].StatusCode}");
if (zipOk && ps1Ok)
if (issues.Count == 0)
{
check.Status = "healthy";
check.Message = $"Agent package and bootstrap script available ({sw.ElapsedMilliseconds}ms)";
check.Message = $"Agent package and bootstrap script available via download alias + legacy blob ({sw.ElapsedMilliseconds}ms)";
}
else if (aliasZipOk && aliasPs1Ok)
{
// Alias intact, legacy keepalive degraded — existing customer bootstrap
// scripts still download from the legacy account until they migrate.
check.Status = "warning";
check.Message = $"Download alias OK, but legacy keepalive degraded — {string.Join(", ", issues)}";
}
else
{
check.Status = "unhealthy";
var issues = new List<string>();
if (!zipOk) issues.Add($"Agent ZIP: {(int)zipResponse.StatusCode}");
if (!ps1Ok) issues.Add($"Bootstrap script: {(int)ps1Response.StatusCode}");
check.Message = $"Missing: {string.Join(", ", issues)}";
}
}
catch (Exception ex)
{
check.Status = "unhealthy";
check.Message = $"Blob storage unreachable: {ex.Message}";
check.Message = $"Agent download endpoints unreachable: {ex.Message}";
_logger.LogError(ex, "Agent binaries health check failed");
}
@@ -182,7 +238,8 @@ public class HealthCheckService
/// Probes the MCP server's public <c>/health</c> endpoint (which sits in front of the
/// <c>/mcp</c> access guard, so no auth is required) and surfaces reachability + the
/// deployed MCP build version. Visible to all authenticated users, so the server URL is
/// deliberately NOT included in the message or details.
/// only included in the details when <paramref name="includeEndpointUrl"/> is set —
/// callers must only enable it for Global Admins.
/// <para>
/// The Container App runs with minReplicas=0 and scales to zero on idle. Azure Container
/// Apps holds an inbound request while it activates a cold replica, so this probe doubles
@@ -193,7 +250,7 @@ public class HealthCheckService
/// the blocking all-checks batch, so the dashboard never waits on the wake.
/// </para>
/// </summary>
internal async Task<HealthCheck> CheckMcpServerAsync()
internal async Task<HealthCheck> CheckMcpServerAsync(bool includeEndpointUrl = false)
{
var check = new HealthCheck
{
@@ -203,6 +260,11 @@ public class HealthCheckService
var baseUrl = (_configuration["McpServerUrl"] ?? Constants.McpServerBaseUrl).TrimEnd('/');
var healthUrl = $"{baseUrl}/health";
if (includeEndpointUrl)
{
check.Details = new Dictionary<string, object> { ["Server URL"] = baseUrl };
}
var timeoutSeconds = ParsePositiveInt("McpServerHealthTimeoutSeconds", 30);
// A cold start that takes longer than this is treated as a successful wake rather
@@ -233,7 +295,8 @@ public class HealthCheckService
: $"MCP server reachable ({sw.ElapsedMilliseconds}ms)";
if (!string.IsNullOrWhiteSpace(version))
{
check.Details = new Dictionary<string, object> { ["Version"] = version };
check.Details ??= new Dictionary<string, object>();
check.Details["Version"] = version;
}
}
catch (Exception ex) when (ex is TaskCanceledException or OperationCanceledException or HttpRequestException)
@@ -554,52 +554,53 @@ namespace AutopilotMonitor.Functions.Services
}
/// <summary>
/// Verifies that agent binaries and bootstrap script are available on blob storage.
/// Verifies that agent binaries and bootstrap script are available on the canonical
/// download alias (Front Door → current blob origin) AND on the legacy blob keepalive
/// account that already-deployed customer bootstrap scripts still use.
/// Records an OpsEvent for each missing item so Global Admins see it in the dashboard.
/// </summary>
private async Task CheckAgentBlobStorageAsync()
{
_logger.LogInformation("Checking agent blob storage availability...");
_logger.LogInformation("Checking agent download endpoint availability...");
var zipUrl = $"{AutopilotMonitor.Shared.Constants.AgentBlobBaseUrl}/{AutopilotMonitor.Shared.Constants.AgentZipFileName}";
var ps1Url = $"{AutopilotMonitor.Shared.Constants.AgentBlobBaseUrl}/Install-AutopilotMonitor.ps1";
var probes = new (string Label, string Url)[]
{
($"Agent ZIP ({AutopilotMonitor.Shared.Constants.AgentZipFileName}, download alias)",
$"{AutopilotMonitor.Shared.Constants.AgentDownloadBaseUrl}/{AutopilotMonitor.Shared.Constants.AgentZipFileName}"),
($"Bootstrap script ({AutopilotMonitor.Shared.Constants.BootstrapScriptName}, download alias)",
$"{AutopilotMonitor.Shared.Constants.AgentDownloadBaseUrl}/{AutopilotMonitor.Shared.Constants.BootstrapScriptName}"),
($"Agent ZIP ({AutopilotMonitor.Shared.Constants.AgentZipFileName}, legacy blob keepalive)",
$"{AutopilotMonitor.Shared.Constants.AgentBlobBaseUrl}/{AutopilotMonitor.Shared.Constants.AgentZipFileName}"),
($"Bootstrap script ({AutopilotMonitor.Shared.Constants.BootstrapScriptName}, legacy blob keepalive)",
$"{AutopilotMonitor.Shared.Constants.AgentBlobBaseUrl}/{AutopilotMonitor.Shared.Constants.BootstrapScriptName}")
};
try
{
var client = _httpClientFactory.CreateClient();
client.Timeout = TimeSpan.FromSeconds(15);
var zipRequest = new HttpRequestMessage(HttpMethod.Head, zipUrl);
var ps1Request = new HttpRequestMessage(HttpMethod.Head, ps1Url);
var results = await Task.WhenAll(
client.SendAsync(zipRequest),
client.SendAsync(ps1Request)
probes.Select(p => client.SendAsync(new HttpRequestMessage(HttpMethod.Head, p.Url)))
);
var zipResponse = results[0];
var ps1Response = results[1];
if (!zipResponse.IsSuccessStatusCode)
var allOk = true;
for (var i = 0; i < probes.Length; i++)
{
_logger.LogError("Agent ZIP not available on blob storage: HTTP {StatusCode}", (int)zipResponse.StatusCode);
await _opsEventService.RecordBlobStorageMissingAsync("Agent ZIP (AutopilotMonitor-Agent.zip)", (int)zipResponse.StatusCode);
if (results[i].IsSuccessStatusCode) continue;
allOk = false;
_logger.LogError("{Item} not available: HTTP {StatusCode}", probes[i].Label, (int)results[i].StatusCode);
await _opsEventService.RecordBlobStorageMissingAsync(probes[i].Label, (int)results[i].StatusCode);
}
if (!ps1Response.IsSuccessStatusCode)
if (allOk)
{
_logger.LogError("Bootstrap script not available on blob storage: HTTP {StatusCode}", (int)ps1Response.StatusCode);
await _opsEventService.RecordBlobStorageMissingAsync("Bootstrap script (Install-AutopilotMonitor.ps1)", (int)ps1Response.StatusCode);
}
if (zipResponse.IsSuccessStatusCode && ps1Response.IsSuccessStatusCode)
{
_logger.LogInformation("Agent blob storage check passed: all binaries available");
_logger.LogInformation("Agent download endpoint check passed: all binaries available on alias + legacy blob");
}
}
catch (Exception ex)
{
_logger.LogError(ex, "Agent blob storage check failed — storage unreachable");
_logger.LogError(ex, "Agent download endpoint check failed — endpoints unreachable");
await _opsEventService.RecordBlobStorageUnreachableAsync(ex.Message);
}
}
@@ -0,0 +1,169 @@
using Azure;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Pagination;
using AutopilotMonitor.Shared;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// One-time backfill of the <c>OccurredUtc</c> business-timestamp column on
/// AuditLogs and OpsEvents rows written before the column existed. The value is
/// decoded deterministically from the reverse-tick RowKey, so re-runs are
/// idempotent and the write is a pure MERGE that adds exactly one property
/// (nothing existing is modified or deleted). Audit legacy bare-GUID rows carry
/// no recoverable time and are excluded server-side by the scan filter.
/// Readers do not depend on this backfill (they decode the RowKey on the fly);
/// its value is future-migration robustness and raw-query readability.
/// </summary>
public class OccurredUtcBackfillService
{
public const string TableAudit = "audit";
public const string TableOps = "ops";
private const int MaxSamplesInResponse = 100;
private const int TransactionChunkSize = 100;
private readonly TableStorageService _storage;
private readonly ILogger<OccurredUtcBackfillService> _logger;
public OccurredUtcBackfillService(TableStorageService storage, ILogger<OccurredUtcBackfillService> logger)
{
_storage = storage;
_logger = logger;
}
public class BackfillResult
{
public string Table { get; set; } = string.Empty;
public bool DryRun { get; set; }
public int RowsExamined { get; set; }
public int WouldWrite { get; set; }
public int Written { get; set; }
public int SkippedAlreadySet { get; set; }
public int SkippedUndecodable { get; set; }
public int Errors { get; set; }
public string? NextContinuation { get; set; }
public List<BackfillSample> Samples { get; set; } = new();
}
public class BackfillSample
{
public string PartitionKey { get; set; } = string.Empty;
public string RowKey { get; set; } = string.Empty;
public DateTime DecodedUtc { get; set; }
}
internal enum RowDecision { AlreadySet, Write, Undecodable }
/// <summary>Pure per-row decision, extracted for unit testing.</summary>
internal static RowDecision DecideRow(string table, TableEntity entity, out DateTime occurredUtc)
{
occurredUtc = default;
if (BusinessTimestamp.GetUtcDateTime(entity, BusinessTimestamp.OccurredUtcColumn).HasValue)
return RowDecision.AlreadySet;
var decoded = table == TableAudit
? BusinessTimestamp.TryDecodeAuditRowKey(entity.RowKey, out occurredUtc)
: BusinessTimestamp.TryDecodeOpsRowKey(entity.RowKey, out occurredUtc);
return decoded ? RowDecision.Write : RowDecision.Undecodable;
}
public async Task<BackfillResult> RunAsync(string table, bool dryRun, int maxRows, string? continuation)
{
var (tableName, scanFilter) = table switch
{
// Audit: server-side restriction to time-encoded '!'-rows — legacy bare-GUID
// rows (which sort after '"') have nothing to decode and are never touched.
TableAudit => (Constants.TableNames.AuditLogs,
$"RowKey ge '{BusinessTimestamp.AuditRowKeyPrefix}' and RowKey lt '{BusinessTimestamp.AuditTimeEncodedUpperBound}'"),
TableOps => (Constants.TableNames.OpsEvents, (string?)null),
_ => throw new ArgumentException($"Unknown table '{table}'", nameof(table)),
};
var tableClient = _storage.GetTableClient(tableName);
var result = new BackfillResult { Table = table, DryRun = dryRun };
var (entities, nextToken) = await AzureTablesPaginator.FetchPageAsync<TableEntity>(
client: tableClient,
filter: scanFilter,
pageSize: maxRows,
continuation: continuation,
select: new[] { "PartitionKey", "RowKey", BusinessTimestamp.OccurredUtcColumn });
result.NextContinuation = nextToken;
var pending = new List<(string PartitionKey, string RowKey, DateTime OccurredUtc)>();
foreach (var entity in entities)
{
result.RowsExamined++;
switch (DecideRow(table, entity, out var occurredUtc))
{
case RowDecision.AlreadySet:
result.SkippedAlreadySet++;
break;
case RowDecision.Undecodable:
result.SkippedUndecodable++;
_logger.LogWarning("OccurredUtc backfill: undecodable RowKey {PK}/{RK} in {Table}",
entity.PartitionKey, entity.RowKey, tableName);
break;
case RowDecision.Write:
result.WouldWrite++;
pending.Add((entity.PartitionKey, entity.RowKey, occurredUtc));
if (result.Samples.Count < MaxSamplesInResponse)
{
result.Samples.Add(new BackfillSample
{
PartitionKey = entity.PartitionKey,
RowKey = entity.RowKey,
DecodedUtc = occurredUtc,
});
}
break;
}
}
if (!dryRun && pending.Count > 0)
result.Written = await WritePendingAsync(tableClient, pending, result);
_logger.LogInformation(
"OccurredUtc backfill {Table}: dryRun={DryRun} examined={Examined} wouldWrite={WouldWrite} written={Written} alreadySet={AlreadySet} undecodable={Undecodable} errors={Errors} more={More}",
table, dryRun, result.RowsExamined, result.WouldWrite, result.Written,
result.SkippedAlreadySet, result.SkippedUndecodable, result.Errors, result.NextContinuation != null);
return result;
}
private async Task<int> WritePendingAsync(TableClient tableClient,
List<(string PartitionKey, string RowKey, DateTime OccurredUtc)> pending, BackfillResult result)
{
var written = 0;
foreach (var partitionGroup in pending.GroupBy(p => p.PartitionKey))
{
foreach (var chunk in partitionGroup.Chunk(TransactionChunkSize))
{
// Merge entity carries ONLY the new column; unconditional (ETag *) merge is
// safe because the value is a pure function of the immutable RowKey.
var actions = chunk.Select(p => new TableTransactionAction(
TableTransactionActionType.UpdateMerge,
new TableEntity(p.PartitionKey, p.RowKey)
{
[BusinessTimestamp.OccurredUtcColumn] = p.OccurredUtc,
},
ETag.All)).ToList();
try
{
await tableClient.SubmitTransactionAsync(actions);
written += actions.Count;
}
catch (Exception ex)
{
result.Errors += actions.Count;
_logger.LogError(ex, "OccurredUtc backfill: merge transaction failed for partition {PK} ({Count} rows)",
partitionGroup.Key, actions.Count);
}
}
}
return written;
}
}
}
@@ -1,5 +1,6 @@
using Azure;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Pagination;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Shared;
@@ -55,7 +56,10 @@ namespace AutopilotMonitor.Functions.Services
{ "EntityType", entityType },
{ "EntityId", entityId },
{ "PerformedBy", performedBy },
{ "Timestamp", timestamp },
// Business event time. "Timestamp" is a reserved system property (server
// ignores supplied values and resets it on every row rewrite), so the
// event time must live in a regular custom column to survive migrations.
{ BusinessTimestamp.OccurredUtcColumn, timestamp },
{ "Details", details != null ? JsonConvert.SerializeObject(details) : string.Empty }
};
@@ -71,32 +75,26 @@ namespace AutopilotMonitor.Functions.Services
}
/// <summary>
/// Retention cleanup: deletes audit log entries whose Timestamp is older than the cutoff.
/// The AuditLogs table is append-only (one row per admin action) and is otherwise only
/// wiped on tenant offboarding, so without this it grows unbounded. Filters server-side on
/// the per-row "Timestamp" property and selects only PK/RK to keep the scan cheap.
/// Retention cleanup: deletes audit log entries whose business time is older than the
/// cutoff. The AuditLogs table is append-only (one row per admin action) and is otherwise
/// only wiped on tenant offboarding, so without this it grows unbounded.
/// Two passes: (1) time-encoded '!'-rows via an index-backed RowKey range — the system
/// Timestamp cannot be used because storage migrations reset it (post-migration it would
/// delete nothing for a retention period, then the whole pre-migration corpus at once);
/// (2) legacy bare-GUID rows (true event time unrecoverable) via their frozen system
/// Timestamp — they all age out ~180d after the 2026-07-18 migration.
/// TODO: Remove pass 2 after 2027-03-01 (no bare-GUID rows can remain by then).
/// </summary>
public async Task<int> DeleteAuditLogsOlderThanAsync(DateTime cutoffUtc)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.AuditLogs);
var filter = $"Timestamp lt datetime'{cutoffUtc:yyyy-MM-ddTHH:mm:ss}Z'";
var query = tableClient.QueryAsync<TableEntity>(filter: filter, select: new[] { "PartitionKey", "RowKey" });
int deleted = 0;
await foreach (var entity in query)
{
try
{
await tableClient.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
deleted++;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to delete audit log {PK}/{RK}", entity.PartitionKey, entity.RowKey);
}
}
var deleted = 0;
deleted += await DeleteAuditLogsByFilterAsync(tableClient, BusinessTimestamp.AuditRetentionClause(cutoffUtc));
// 'RowKey ge '0'' cannot match any '!'-row (0x21 < 0x30), only legacy GUIDs.
deleted += await DeleteAuditLogsByFilterAsync(tableClient,
$"RowKey ge '0' and Timestamp lt datetime'{cutoffUtc:yyyy-MM-ddTHH:mm:ss}Z'");
if (deleted > 0)
_logger.LogInformation("Deleted {Count} audit log entries older than {Cutoff:yyyy-MM-dd}", deleted, cutoffUtc);
@@ -110,6 +108,25 @@ namespace AutopilotMonitor.Functions.Services
}
}
private async Task<int> DeleteAuditLogsByFilterAsync(TableClient tableClient, string filter)
{
var query = tableClient.QueryAsync<TableEntity>(filter: filter, select: new[] { "PartitionKey", "RowKey" });
int deleted = 0;
await foreach (var entity in query)
{
try
{
await tableClient.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
deleted++;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to delete audit log {PK}/{RK}", entity.PartitionKey, entity.RowKey);
}
}
return deleted;
}
/// <summary>
/// Gets audit log entries for a tenant within an optional UTC date window.
/// No row cap — returns the full filtered set, sorted newest-first.
@@ -384,9 +401,9 @@ namespace AutopilotMonitor.Functions.Services
if (!string.IsNullOrEmpty(lastRowKey))
clauses.Add($"RowKey gt '{lastRowKey!.Replace("'", "''")}'");
if (dateFrom.HasValue)
clauses.Add($"Timestamp ge datetime'{ToUtc(dateFrom.Value):o}'");
clauses.Add(BusinessTimestamp.AuditDateFromClause(ToUtc(dateFrom.Value)));
if (dateTo.HasValue)
clauses.Add($"Timestamp le datetime'{ToUtc(dateTo.Value):o}'");
clauses.Add(BusinessTimestamp.AuditDateToClause(ToUtc(dateTo.Value)));
return string.Join(" and ", clauses);
}
@@ -398,14 +415,30 @@ namespace AutopilotMonitor.Functions.Services
EntityType = entity.GetString("EntityType") ?? string.Empty,
EntityId = entity.GetString("EntityId") ?? string.Empty,
PerformedBy = entity.GetString("PerformedBy") ?? string.Empty,
Timestamp = entity.GetDateTimeOffset("Timestamp")?.UtcDateTime ?? DateTime.UtcNow,
Timestamp = ResolveAuditTimestamp(entity),
Details = entity.GetString("Details") ?? string.Empty,
};
// The system Timestamp is auto-managed by Azure on insert and reliably
// sortable; the user-defined "Timestamp" property is set by LogAuditEntry
// for parity with the model. We filter on the system Timestamp since it
// is always indexed.
/// <summary>
/// Business time of an audit row: OccurredUtc column → time decoded from the
/// reverse-tick RowKey → system Timestamp. The system Timestamp is only a last
/// resort (legacy bare-GUID rows): it is reset by any row rewrite, so for rows
/// migrated between storage accounts it holds the migration moment, not the
/// event time.
/// </summary>
internal static DateTime ResolveAuditTimestamp(TableEntity entity)
{
var occurred = BusinessTimestamp.GetUtcDateTime(entity, BusinessTimestamp.OccurredUtcColumn);
if (occurred.HasValue)
return occurred.Value;
if (BusinessTimestamp.TryDecodeAuditRowKey(entity.RowKey, out var decoded))
return decoded;
return BusinessTimestamp.GetUtcDateTime(entity, "Timestamp") ?? DateTime.UtcNow;
}
// Date windows filter on the RowKey (reverse-tick, index-backed) rather than any
// timestamp property: the system Timestamp is reset by migrations, and an
// OccurredUtc property filter would exclude rows that lack the column.
internal static string? BuildAuditLogFilter(string? tenantId, DateTime? dateFrom, DateTime? dateTo,
bool excludeDeletions = false, AuditLogQueryFilters? filters = null)
{
@@ -424,11 +457,11 @@ namespace AutopilotMonitor.Functions.Services
}
if (dateFrom.HasValue)
{
clauses.Add($"Timestamp ge datetime'{ToUtc(dateFrom.Value):o}'");
clauses.Add(BusinessTimestamp.AuditDateFromClause(ToUtc(dateFrom.Value)));
}
if (dateTo.HasValue)
{
clauses.Add($"Timestamp le datetime'{ToUtc(dateTo.Value):o}'");
clauses.Add(BusinessTimestamp.AuditDateToClause(ToUtc(dateTo.Value)));
}
return string.Join(" and ", clauses);
}
@@ -158,6 +158,9 @@ namespace AutopilotMonitor.Functions.Services
["IntervalSeconds"] = rule.IntervalSeconds,
["TriggerPhase"] = rule.TriggerPhase ?? string.Empty,
["TriggerEventType"] = rule.TriggerEventType ?? string.Empty,
["ActivePhasesJson"] = JsonConvert.SerializeObject(rule.ActivePhases ?? new List<string>()),
["ActiveFromPhase"] = rule.ActiveFromPhase ?? string.Empty,
["EmitMode"] = rule.EmitMode ?? string.Empty,
["OutputEventType"] = rule.OutputEventType ?? string.Empty,
["OutputSeverity"] = rule.OutputSeverity ?? "Info",
["TagsJson"] = JsonConvert.SerializeObject(rule.Tags ?? new string[0]),
@@ -222,7 +225,9 @@ namespace AutopilotMonitor.Functions.Services
}
}
private GatherRule MapToGatherRule(TableEntity entity)
// Internal (not private) so the Store↔Map roundtrip is testable per the
// table-serialization rule: every model field must appear in BOTH directions.
internal GatherRule MapToGatherRule(TableEntity entity)
{
return new GatherRule
{
@@ -244,6 +249,9 @@ namespace AutopilotMonitor.Functions.Services
IntervalSeconds = entity.GetInt32("IntervalSeconds"),
TriggerPhase = entity.GetString("TriggerPhase") ?? string.Empty,
TriggerEventType = entity.GetString("TriggerEventType") ?? string.Empty,
ActivePhases = MapActivePhases(entity.GetString("ActivePhasesJson")),
ActiveFromPhase = string.IsNullOrEmpty(entity.GetString("ActiveFromPhase")) ? null : entity.GetString("ActiveFromPhase"),
EmitMode = string.IsNullOrEmpty(entity.GetString("EmitMode")) ? null : entity.GetString("EmitMode"),
OutputEventType = entity.GetString("OutputEventType") ?? string.Empty,
OutputSeverity = entity.GetString("OutputSeverity") ?? "Info",
Tags = DeserializeJsonArray(entity.GetString("TagsJson")),
@@ -252,6 +260,17 @@ namespace AutopilotMonitor.Functions.Services
};
}
/// <summary>
/// Absent column (pre-existing rows) and the stored-empty "[]" both map to null —
/// null and empty are the same "unrestricted" state on <see cref="GatherRule.ActivePhases"/>,
/// and null keeps <see cref="GatherRuleService.ContentEquivalent"/> reseed-stable.
/// </summary>
private List<string>? MapActivePhases(string? json)
{
var phases = DeserializeJson<List<string>>(json);
return phases.Count == 0 ? null : phases;
}
// ===== RULE STATES METHODS =====
/// <summary>
@@ -681,7 +681,10 @@ namespace AutopilotMonitor.Functions.Services
["EventId"] = evt.EventId,
["SessionId"] = evt.SessionId,
["TenantId"] = evt.TenantId,
["Timestamp"] = evt.Timestamp,
// "Timestamp" is a reserved system property — supplied values are ignored
// and any row rewrite (storage migration) resets it. The sanitized agent
// event time therefore lives in its own column.
[BusinessTimestamp.OccurredUtcColumn] = EnsureUtc(evt.Timestamp),
["EventType"] = evt.EventType ?? string.Empty,
["Severity"] = (int)evt.Severity,
["Source"] = evt.Source ?? string.Empty,
@@ -777,7 +780,8 @@ namespace AutopilotMonitor.Functions.Services
["EventId"] = evt.EventId,
["SessionId"] = evt.SessionId,
["TenantId"] = evt.TenantId,
["Timestamp"] = evt.Timestamp,
// Reserved-name caveat: see StoreEventAsync.
[BusinessTimestamp.OccurredUtcColumn] = EnsureUtc(evt.Timestamp),
["EventType"] = evt.EventType ?? string.Empty,
["Severity"] = (int)evt.Severity,
["Source"] = evt.Source ?? string.Empty,
@@ -2673,10 +2677,7 @@ namespace AutopilotMonitor.Functions.Services
EventId = entity.GetString("EventId") ?? string.Empty,
SessionId = entity.GetString("SessionId") ?? string.Empty,
TenantId = tenantId ?? partitionKey,
Timestamp = DateTime.SpecifyKind(
entity.GetDateTimeOffset("Timestamp")?.UtcDateTime
?? entity.GetDateTime("Timestamp")
?? DateTime.UtcNow, DateTimeKind.Utc),
Timestamp = DateTime.SpecifyKind(ResolveEventTimestamp(entity), DateTimeKind.Utc),
EventType = entity.GetString("EventType") ?? string.Empty,
Severity = (EventSeverity)(entity.GetInt32("Severity") ?? 0),
Source = entity.GetString("Source") ?? string.Empty,
@@ -2827,7 +2828,10 @@ namespace AutopilotMonitor.Functions.Services
/// <summary>
/// Returns the earliest event timestamp persisted for a session, if any.
/// Events are written with RowKey "{Timestamp}_{Sequence}", so querying the partition
/// and taking the first row yields the earliest event.
/// and taking the first row yields the earliest event. Resolution must NOT rely on
/// the system Timestamp: it is reset by storage migrations, and this value feeds the
/// StartedAt/Duration reconciliation — a migration-reset read would rewrite those
/// session fields with the migration date.
/// </summary>
private async Task<DateTime?> GetEarliestSessionEventTimestampAsync(string tenantId, string sessionId)
{
@@ -2838,13 +2842,12 @@ namespace AutopilotMonitor.Functions.Services
var query = tableClient.QueryAsync<TableEntity>(
filter: $"PartitionKey eq '{partitionKey}'",
maxPerPage: 1,
select: new[] { "Timestamp", "RowKey" }
select: new[] { "RowKey", BusinessTimestamp.OccurredUtcColumn, "Timestamp" }
);
await foreach (var entity in query)
{
return entity.GetDateTimeOffset("Timestamp")?.UtcDateTime
?? entity.GetDateTime("Timestamp");
return ResolveEventTimestampOrNull(entity);
}
}
catch (Exception ex)
@@ -2855,6 +2858,26 @@ namespace AutopilotMonitor.Functions.Services
return null;
}
/// <summary>
/// Business time of an event row: OccurredUtc column → agent time decoded from the
/// "{yyyyMMddHHmmssfff}_{seq}" RowKey → system Timestamp (last resort — reset by any
/// row rewrite, e.g. a storage migration; for migrated rows it holds the migration
/// moment). Event rows are deliberately never backfilled, so the RowKey decode is the
/// primary path for all pre-OccurredUtc rows.
/// </summary>
internal static DateTime ResolveEventTimestamp(TableEntity entity)
=> ResolveEventTimestampOrNull(entity) ?? DateTime.UtcNow;
internal static DateTime? ResolveEventTimestampOrNull(TableEntity entity)
{
var occurred = BusinessTimestamp.GetUtcDateTime(entity, BusinessTimestamp.OccurredUtcColumn);
if (occurred.HasValue)
return occurred.Value;
if (BusinessTimestamp.TryDecodeEventRowKeyPrefix(entity.RowKey, out var decoded))
return decoded;
return BusinessTimestamp.GetUtcDateTime(entity, "Timestamp");
}
/// <summary>
/// Returns the authoritative (total, reboot) event-row counts for a session, or <c>null</c>
/// if they could not be determined (storage error). Events are written with UpsertReplace
@@ -24,6 +24,8 @@ export const EVENT_TYPES_CATALOG = {
'esp_failure_settle_started',
'esp_failure_settle_recovered',
'esp_exiting',
'esp_reboot_coalescing',
'mdm_policy_reboot_required',
'esp_config_detected',
'completion_check',
'completion_waiting',
@@ -73,19 +73,32 @@ namespace AutopilotMonitor.Shared
public const string ApiBaseUrl = "https://autopilotmonitor-api-eu.azurewebsites.net";
/// <summary>
/// Public base URL of the MCP server Container App (no trailing slash, no /mcp suffix).
/// Used by the backend health check to probe <c>{McpServerBaseUrl}/health</c>. The
/// Container App runs with minReplicas=0, so the first probe after idle may incur a
/// cold start. Overridable via the <c>McpServerUrl</c> app setting (e.g. for dev).
/// Public base URL of the MCP server (no trailing slash, no /mcp suffix). Stable custom
/// domain in front of the Container App — survives region/environment moves, unlike the
/// generated *.azurecontainerapps.io FQDN. Used by the backend health check to probe
/// <c>{McpServerBaseUrl}/health</c>. The Container App runs with minReplicas=0, so the
/// first probe after idle may incur a cold start. Overridable via the
/// <c>McpServerUrl</c> app setting (e.g. for dev).
/// </summary>
public const string McpServerBaseUrl = "https://autopilotmonitor-mcp.kindwave-58b4b547.westeurope.azurecontainerapps.io";
public const string McpServerBaseUrl = "https://mcp.autopilotmonitor.com";
// -----------------------------------------------------------------------
// Agent self-update
// -----------------------------------------------------------------------
/// <summary>
/// Base URL for agent binaries in Azure Blob Storage (public read)
/// Canonical public base URL for agent binaries: Front Door alias in front of the
/// current blob origin (autopilotmonitoreu, container "agent"). Storage-account moves
/// happen behind this name without touching customers. New bootstrap scripts and all
/// availability probes use this URL.
/// </summary>
public const string AgentDownloadBaseUrl = "https://download.autopilotmonitor.com/agent";
/// <summary>
/// LEGACY base URL for agent binaries: the pre-migration storage account, kept alive
/// (dual-push on every release) because bootstrap scripts already deployed in customer
/// Intune tenants still download from it. Remove only after the customer migration to
/// <see cref="AgentDownloadBaseUrl"/> is complete and the legacy account is torn down.
/// </summary>
public const string AgentBlobBaseUrl = "https://autopilotmonitor.blob.core.windows.net/agent";
@@ -392,6 +405,18 @@ namespace AutopilotMonitor.Shared
public const string OsBuildChanged = "os_build_changed"; // OS build (CurrentBuild.UBR) differs across an agent restart — an update was installed during enrollment
public const string WindowsUpdateChannelCensus = "windows_update_channel_census"; // One-shot EventID histogram of WU Client + UpdateOrchestrator channels when os_build_changed but zero targeted WU events captured
// MDM-policy-forced coalesced reboot during ESP (MdmRebootPolicyTracker + ShellCoreTracker;
// PMPC "unexpected reboot / second sign-in" research). Device-assigned policies whose CSP
// URIs match the OS reboot-required catalog force a coalesced reboot at the end of
// DeviceSetup — the user sees an unexpected restart and a second sign-in screen before
// AccountSetup. 2800 (DeviceManagement-Enterprise-Diagnostics-Provider/Admin) fires once
// per reboot-forcing URI; Shell-Core 62407 "RebootCoalescing" records ESP initiating the
// reboot. Deliberately DISTINCT from system_reboot_detected: the backend RebootCount
// matches that string exactly (EventIngestProcessor.Classification + terminal recount) —
// these attribution events must never inflate the reboot count.
public const string MdmPolicyRebootRequired = "mdm_policy_reboot_required"; // DM-Enterprise EventID 2800 — Data.rebootUri = CSP URI that demands the reboot (one event per URI)
public const string EspRebootCoalescing = "esp_reboot_coalescing"; // Shell-Core 62407 "RebootCoalescing" — ESP-initiated coalesced reboot imminent (second sign-in follows)
// TEMPORARY: shadow SM rollout verification — remove when CompletionStateMachine is promoted to primary
public const string ShadowDiscrepancy = "shadow_discrepancy";
@@ -242,7 +242,7 @@ namespace AutopilotMonitor.Shared.Models
/// config cannot chain another migration), and a kill signal always wins over it.
/// Old binaries ignore the field. null = no migration.
/// </summary>
public string MigrateToApiBaseUrl { get; set; }
public string? MigrateToApiBaseUrl { get; set; }
}
/// <summary>
@@ -482,6 +482,28 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public bool WindowsUpdateChannelCensusEnabled { get; set; } = true;
// -----------------------------------------------------------------------
// MDM reboot-policy watcher (DeviceManagement-Enterprise-Diagnostics-Provider/Admin)
// -----------------------------------------------------------------------
/// <summary>
/// Master switch for the MDM reboot-policy watcher. When enabled the agent subscribes to
/// <c>Microsoft-Windows-DeviceManagement-Enterprise-Diagnostics-Provider/Admin</c> EventID
/// 2800 and emits one <c>mdm_policy_reboot_required</c> event per device-assigned policy
/// URI that forces a coalesced reboot during ESP DeviceSetup (the "unexpected reboot +
/// second sign-in" pattern). Internal steering lever only — not surfaced in any admin UI;
/// disable by pushing config server-side. Default: true.
/// </summary>
public bool MdmRebootPolicyWatcherEnabled { get; set; } = true;
/// <summary>
/// Lookback window in minutes for the MDM reboot-policy backfill scan on startup. Device
/// policies can apply before the agent starts, and after the coalesced reboot the restarted
/// agent must re-scan past the reboot gap (the cross-restart watermark keeps re-reads from
/// re-emitting). 0 = backfill disabled. Default: 60 minutes.
/// </summary>
public int MdmRebootPolicyBackfillLookbackMinutes { get; set; } = 60;
/// <summary>
/// Creates default collector configuration
/// </summary>
@@ -128,6 +128,38 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public string TriggerEventType { get; set; } = default!;
/// <summary>
/// Restricts the rule to run only while the current enrollment phase is one of these
/// phases. Canonical tokens are the <see cref="EnrollmentPhase"/> enum names
/// ("Start", "DevicePreparation", "DeviceSetup", "AppsDevice", "AccountSetup",
/// "AppsUser", "FinalizingSetup", "Complete"); "Unknown" and "Failed" are rejected
/// by backend validation. Null or empty = unrestricted (runs in every phase —
/// legacy behavior). Mutually exclusive with <see cref="ActiveFromPhase"/>;
/// if both are set the agent defensively prefers this list.
/// Applies to ALL trigger types. Before the first phase signal of a session,
/// scoped rules are inactive.
/// </summary>
public List<string>? ActivePhases { get; set; }
/// <summary>
/// Activates the rule once the enrollment phase first reaches this phase
/// (ordinal comparison on <see cref="EnrollmentPhase"/>, ignoring Unknown/Failed),
/// then keeps it active for the rest of the session (sticky latch — including
/// through Failed). Canonical tokens as in <see cref="ActivePhases"/>.
/// Null = unrestricted. Mutually exclusive with <see cref="ActivePhases"/>.
/// </summary>
public string? ActiveFromPhase { get; set; }
/// <summary>
/// Emit behavior for collected results:
/// null / "always" = emit on every collection (legacy behavior);
/// "on_change" = poll on the trigger cadence but emit only when the collected
/// result differs from the last emitted one. The first in-scope result always
/// emits; the suppressed poll count is carried on the next emitted event
/// (suppressedPolls / suppressedSinceUtc in the event data).
/// </summary>
public string? EmitMode { get; set; }
// ===== OUTPUT =====
/// <summary>
@@ -1,7 +1,6 @@
"use client";
import { Suspense, useCallback, useEffect, useMemo, useRef, useState } from "react";
import Link from "next/link";
import { useRouter, useSearchParams } from "next/navigation";
import { ProtectedRoute } from "../../components/ProtectedRoute";
import { GlobalAdminBanner } from "@/components/GlobalAdminBanner";
@@ -1,7 +1,7 @@
"use client";
import { useMemo, useState } from "react";
import { GatherRule, NewRuleForm, CATEGORY_COLORS, COLLECTOR_TYPE_LABELS, EMPTY_FORM, formatTrigger } from "../types";
import { GatherRule, NewRuleForm, CATEGORY_COLORS, COLLECTOR_TYPE_LABELS, EMPTY_FORM, formatTrigger, formatGatherPhase, withDerivedScopeMode } from "../types";
import { GatherRuleFormFields } from "./GatherRuleFormFields";
import { FormJsonToggle, JsonModeToggleButtons, ReadOnlyJsonView } from "@/components/rules/FormJsonToggle";
import { validateGatherRuleTarget } from "@/utils/guardValidation";
@@ -301,6 +301,19 @@ export function GatherRuleCard({
on event: <code className="px-1 bg-gray-200 rounded text-xs">{rule.triggerEventType}</code>
</span>
)}
{rule.activePhases && rule.activePhases.length > 0 && (
<span className="text-gray-500">
· during {rule.activePhases.map(formatGatherPhase).join(", ")}
</span>
)}
{!rule.activePhases?.length && rule.activeFromPhase && (
<span className="text-gray-500">
· from {formatGatherPhase(rule.activeFromPhase)}
</span>
)}
{rule.emitMode === "on_change" && (
<span className="text-gray-500">· emit on change</span>
)}
</div>
</div>
</div>
@@ -430,7 +443,7 @@ export function GatherRuleCard({
if (jsonModeEdit) {
try {
const parsed = JSON.parse(jsonText) as NewRuleForm;
onSaveEdit(rule, { ...editForm, ...parsed });
onSaveEdit(rule, withDerivedScopeMode({ ...editForm, ...parsed }));
} catch (e) {
// jsonError is handled by parent
}
@@ -10,6 +10,8 @@ import {
SEVERITIES,
TARGET_PLACEHOLDERS,
TARGET_HINTS,
GATHER_PHASES,
EMIT_MODES,
formatTrigger,
} from "../types";
import { KNOWN_EVENT_TYPES, findEventType } from "../eventTypes";
@@ -426,14 +428,20 @@ export function GatherRuleFormFields({ form, setForm, showRuleId, unrestrictedMo
{form.trigger === "phase_change" && (
<div>
<label className="block text-sm font-medium text-gray-700 mb-1">Trigger Phase</label>
<input
type="text"
<select
value={form.triggerPhase}
onChange={(e) => setForm({ ...form, triggerPhase: e.target.value })}
placeholder="e.g., DeviceESP, AccountESP, DevicePreparation"
autoComplete="off"
className="w-full px-4 py-2 border border-gray-300 rounded-lg text-sm text-gray-900 placeholder-gray-500 focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:border-indigo-500 transition-colors"
/>
className="w-full px-4 py-2 border border-gray-300 rounded-lg text-sm text-gray-900 bg-white focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:border-indigo-500 transition-colors"
>
<option value="">Any phase change</option>
{form.triggerPhase && !GATHER_PHASES.some((p) => p.value === form.triggerPhase) && (
<option value={form.triggerPhase}>{form.triggerPhase} (legacy value)</option>
)}
{GATHER_PHASES.map((p) => (
<option key={p.value} value={p.value}>{p.label}</option>
))}
</select>
<p className="text-xs text-gray-400 mt-1">Run this rule when the enrollment reaches this phase. &quot;Any phase change&quot; fires once per phase transition.</p>
</div>
)}
@@ -472,6 +480,89 @@ export function GatherRuleFormFields({ form, setForm, showRuleId, unrestrictedMo
</div>
)}
{/* Row: Active During (phase scope) + Emit Mode */}
<div className="grid grid-cols-1 sm:grid-cols-2 gap-4">
<div>
<label className="block text-sm font-medium text-gray-700 mb-1">Active During</label>
<select
value={form.scopeMode}
onChange={(e) => {
const scopeMode = e.target.value as NewRuleForm["scopeMode"];
// Keep the form state consistent: only the fields of the selected mode survive.
setForm({
...form,
scopeMode,
activePhases: scopeMode === "during" ? form.activePhases : [],
activeFromPhase: scopeMode === "from" ? form.activeFromPhase : "",
});
}}
className="w-full px-4 py-2 border border-gray-300 rounded-lg text-sm text-gray-900 bg-white focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:border-indigo-500 transition-colors"
>
<option value="always">All phases (always)</option>
<option value="during">Only during specific phases</option>
<option value="from">From a phase onwards</option>
</select>
<p className="text-xs text-gray-400 mt-1">Restrict when this rule runs. Outside its scope the rule is idle — interval rules stop polling, triggers are ignored.</p>
</div>
<div>
<label className="block text-sm font-medium text-gray-700 mb-1">Emit Mode</label>
<select
value={form.emitMode}
onChange={(e) => setForm({ ...form, emitMode: e.target.value })}
className="w-full px-4 py-2 border border-gray-300 rounded-lg text-sm text-gray-900 bg-white focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:border-indigo-500 transition-colors"
>
{EMIT_MODES.map((m) => (
<option key={m.value} value={m.value}>{m.label}</option>
))}
</select>
<p className="text-xs text-gray-400 mt-1">On change: polls on the trigger cadence but only emits an event when the collected result changes. The first collection always emits.</p>
</div>
</div>
{form.scopeMode === "during" && (
<div>
<label className="block text-sm font-medium text-gray-700 mb-1">Active Phases</label>
<div className="grid grid-cols-2 sm:grid-cols-4 gap-2">
{GATHER_PHASES.map((p) => (
<label key={p.value} className="flex items-center space-x-2 text-sm text-gray-700 cursor-pointer">
<input
type="checkbox"
checked={form.activePhases.includes(p.value)}
onChange={(e) =>
setForm({
...form,
activePhases: e.target.checked
? [...form.activePhases, p.value]
: form.activePhases.filter((x) => x !== p.value),
})
}
className="rounded border-gray-300 text-indigo-600 focus:ring-indigo-500"
/>
<span>{p.label}</span>
</label>
))}
</div>
<p className="text-xs text-gray-400 mt-1">The rule runs only while the enrollment is in one of the selected phases. No selection = all phases.</p>
</div>
)}
{form.scopeMode === "from" && (
<div>
<label className="block text-sm font-medium text-gray-700 mb-1">Active From Phase</label>
<select
value={form.activeFromPhase}
onChange={(e) => setForm({ ...form, activeFromPhase: e.target.value })}
className="w-full px-4 py-2 border border-gray-300 rounded-lg text-sm text-gray-900 bg-white focus:outline-none focus:ring-2 focus:ring-indigo-500 focus:border-indigo-500 transition-colors"
>
<option value="">Select a phase…</option>
{GATHER_PHASES.map((p) => (
<option key={p.value} value={p.value}>{p.label}</option>
))}
</select>
<p className="text-xs text-gray-400 mt-1">Once the enrollment reaches this phase the rule activates and stays active for the rest of the session. No selection = all phases.</p>
</div>
)}
{/* Row 3: Output Event Type, Output Severity */}
<div className="grid grid-cols-1 sm:grid-cols-2 gap-4">
<div>
@@ -16,7 +16,7 @@ import { FormJsonToggle, JsonModeToggleButtons } from "@/components/rules/FormJs
import { useAuthenticatedFetch, useNotificationMessages, useGlobalAdminScope } from "@/hooks";
import { GlobalAdminBanner, globalAdminSubtitle } from "@/components/GlobalAdminBanner";
import { TenantScopeSelector } from "@/components/TenantScopeSelector";
import { GatherRule, NewRuleForm, EMPTY_FORM, CATEGORY_COLORS } from "./types";
import { GatherRule, NewRuleForm, EMPTY_FORM, CATEGORY_COLORS, withDerivedScopeMode } from "./types";
import { GatherRuleFormFields } from "./components/GatherRuleFormFields";
import { GatherRuleCard } from "./components/GatherRuleCard";
@@ -211,6 +211,9 @@ export default function GatherRulesPage() {
intervalSeconds: form.trigger === "interval" ? form.intervalSeconds : null,
triggerPhase: form.trigger === "phase_change" ? form.triggerPhase : null,
triggerEventType: form.trigger === "on_event" ? form.triggerEventType : null,
activePhases: form.scopeMode === "during" && form.activePhases.length > 0 ? form.activePhases : null,
activeFromPhase: form.scopeMode === "from" && form.activeFromPhase ? form.activeFromPhase : null,
emitMode: form.emitMode || null,
outputEventType: form.outputEventType,
outputSeverity: form.outputSeverity,
};
@@ -265,6 +268,12 @@ export default function GatherRulesPage() {
intervalSeconds: rule.intervalSeconds || 60,
triggerPhase: rule.triggerPhase || "",
triggerEventType: rule.triggerEventType || "",
scopeMode: rule.activePhases?.length ? "during" : rule.activeFromPhase ? "from" : "always",
activePhases: rule.activePhases || [],
activeFromPhase: rule.activeFromPhase || "",
// Existing rules without the field keep today's behavior ("always") — the on_change
// default applies to NEW rules only.
emitMode: rule.emitMode === "on_change" ? "on_change" : "always",
outputEventType: rule.outputEventType,
outputSeverity: rule.outputSeverity,
});
@@ -289,12 +298,17 @@ export default function GatherRulesPage() {
intervalSeconds: form.trigger === "interval" ? form.intervalSeconds : null,
triggerPhase: form.trigger === "phase_change" ? form.triggerPhase : null,
triggerEventType: form.trigger === "on_event" ? form.triggerEventType : null,
activePhases: form.scopeMode === "during" && form.activePhases.length > 0 ? form.activePhases : null,
activeFromPhase: form.scopeMode === "from" && form.activeFromPhase ? form.activeFromPhase : null,
emitMode: form.emitMode || null,
outputEventType: form.outputEventType,
outputSeverity: form.outputSeverity,
version: bumpVersion(rule.version),
enabled: rule.enabled,
author: rule.author,
createdAt: rule.createdAt,
// Same wipe class as the toggle bug: the full-replace PUT must carry tags or they vanish.
tags: rule.tags ?? [],
isBuiltIn: rule.isBuiltIn ?? false,
isCommunity: rule.isCommunity ?? false,
};
@@ -505,7 +519,9 @@ export default function GatherRulesPage() {
try {
const parsed = JSON.parse(jsonText) as NewRuleForm;
if (!parsed.ruleId && !parsed.title) throw new Error("JSON must include at least ruleId and title");
setNewRule({ ...EMPTY_FORM, ...parsed });
// Rule-shaped JSON carries activePhases/activeFromPhase but no scopeMode
// — derive it so the scope fields survive into the create payload.
setNewRule(withDerivedScopeMode({ ...EMPTY_FORM, ...parsed }));
setJsonModeCreate(false);
setJsonError(null);
} catch (e) {
@@ -538,7 +554,7 @@ export default function GatherRulesPage() {
try {
const parsed = JSON.parse(jsonText) as NewRuleForm;
setJsonError(null);
handleCreateRule({ ...EMPTY_FORM, ...parsed });
handleCreateRule(withDerivedScopeMode({ ...EMPTY_FORM, ...parsed }));
} catch (e) {
setJsonError(e instanceof Error ? e.message : "Invalid JSON");
}
@@ -619,7 +635,7 @@ export default function GatherRulesPage() {
try {
const parsed = JSON.parse(jsonText) as NewRuleForm;
if (!parsed.title) throw new Error("JSON must include title");
setEditForm({ ...editForm, ...parsed });
setEditForm(withDerivedScopeMode({ ...editForm, ...parsed }));
setJsonModeEdit(false);
setJsonError(null);
} catch (e) {
@@ -15,6 +15,9 @@ export interface GatherRule {
intervalSeconds: number | null;
triggerPhase: string | null;
triggerEventType: string | null;
activePhases?: string[] | null;
activeFromPhase?: string | null;
emitMode?: string | null;
outputEventType: string;
outputSeverity: string;
tags: string[];
@@ -48,6 +51,10 @@ export interface NewRuleForm {
intervalSeconds: number;
triggerPhase: string;
triggerEventType: string;
scopeMode: "always" | "during" | "from";
activePhases: string[];
activeFromPhase: string;
emitMode: string;
outputEventType: string;
outputSeverity: string;
}
@@ -57,6 +64,30 @@ export const COLLECTOR_TYPES = ["registry", "eventlog", "wmi", "file", "command_
export const TRIGGERS = ["startup", "phase_change", "interval", "on_event"] as const;
export const SEVERITIES = ["info", "warning", "error", "critical"] as const;
// Canonical phase-scope tokens: the backend EnrollmentPhase enum NAMES from Start(0) through
// Complete(7) — Unknown/Failed are not selectable. Local mirror (like eventTypes.ts); display
// labels are gather-rule-specific, deliberately NOT reusing phaseConstants.ts timeline names.
export const GATHER_PHASES: ReadonlyArray<{ value: string; label: string }> = [
{ value: "Start", label: "Start" },
{ value: "DevicePreparation", label: "Device Preparation" },
{ value: "DeviceSetup", label: "Device Setup" },
{ value: "AppsDevice", label: "Apps (Device)" },
{ value: "AccountSetup", label: "Account Setup" },
{ value: "AppsUser", label: "Apps (User)" },
{ value: "FinalizingSetup", label: "Finalizing Setup" },
{ value: "Complete", label: "Complete" },
];
export const EMIT_MODES: ReadonlyArray<{ value: string; label: string }> = [
{ value: "always", label: "Always (emit every collection)" },
{ value: "on_change", label: "On change (emit only when the result changes)" },
];
/** Display label for a phase-scope token; falls back to the raw token for unknown values. */
export function formatGatherPhase(token: string): string {
return GATHER_PHASES.find((p) => p.value === token)?.label ?? token;
}
export const CATEGORY_COLORS: Record<string, { bg: string; text: string }> = {
network: { bg: "bg-blue-100", text: "text-blue-700" },
identity: { bg: "bg-purple-100", text: "text-purple-700" },
@@ -126,6 +157,11 @@ export const EMPTY_FORM: NewRuleForm = {
intervalSeconds: 60,
triggerPhase: "",
triggerEventType: "",
scopeMode: "always",
activePhases: [],
activeFromPhase: "",
// New rules default to on_change (anti-spam); existing rules load as "always" in startEditing.
emitMode: "on_change",
outputEventType: "",
outputSeverity: "info",
};
@@ -137,3 +173,19 @@ export function formatTrigger(trigger: string) {
default: return trigger.charAt(0).toUpperCase() + trigger.slice(1);
}
}
/**
* Normalizes a form object after JSON-mode merges. Pasted JSON is usually rule-shaped
* (activePhases/activeFromPhase/emitMode, no scopeMode key), so the UI-only scopeMode must
* be derived from the data — otherwise the create/save payload would silently drop the
* scope fields. Also coerces null/unknown emitMode values to the "always" select option.
*/
export function withDerivedScopeMode(form: NewRuleForm): NewRuleForm {
const activePhases = (Array.isArray(form.activePhases) ? form.activePhases : [])
.filter((p): p is string => typeof p === "string" && p.length > 0);
const activeFromPhase = typeof form.activeFromPhase === "string" ? form.activeFromPhase : "";
const scopeMode: NewRuleForm["scopeMode"] =
activePhases.length > 0 ? "during" : activeFromPhase ? "from" : "always";
const emitMode = form.emitMode === "on_change" ? "on_change" : "always";
return { ...form, activePhases, activeFromPhase, scopeMode, emitMode };
}
@@ -31,6 +31,9 @@ export function SignalRProvider({ children }: { children: React.ReactNode }) {
const [joinedGroups, setJoinedGroups] = useState<string[]>([]);
const retryCountRef = useRef(0);
const retryTimeoutRef = useRef<ReturnType<typeof setTimeout> | null>(null);
// Collapses rapid successive accessRevoked pushes (one per revoked grant) into a single
// restart — a second start() on a non-Disconnected connection throws and burns the retry budget.
const revokeRestartInFlightRef = useRef(false);
// Set in onreconnecting, read in onreconnected so we can report downtime as a measurement.
const disconnectStartedAtRef = useRef<number | null>(null);
const maxRetries = 3;
@@ -157,12 +160,20 @@ export function SignalRProvider({ children }: { children: React.ReactNode }) {
// the next Connected transition — through the join endpoint, which re-runs authorization
// against the fresh scope. Revoked groups 403; still-authorized streams recover automatically.
newConnection.on('accessRevoked', async () => {
// Dropping an event that arrives mid-restart is safe: the rejoin after start() re-runs
// authorization against the CURRENT scope, which already reflects that later revoke.
if (revokeRestartInFlightRef.current) return;
revokeRestartInFlightRef.current = true;
console.warn('[SignalR] accessRevoked received — restarting connection to re-authorize group memberships');
trackEvent('signalr_access_revoked');
try {
await newConnection.stop();
} catch { /* proceed to restart regardless */ }
startConnection();
try {
await startConnection();
} finally {
revokeRestartInFlightRef.current = false;
}
});
// Start as soon as the user is authenticated. The previous 2s setTimeout