mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-25 05:00:07 +00:00
Compare commits
30 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| dbd7d2ac6f | |||
| 8f24876682 | |||
| 2d5ee887d7 | |||
| 81c14bb3ed | |||
| 3a0bd74b12 | |||
| 50e7590060 | |||
| 61e53a2e9b | |||
| 7e7b6f8086 | |||
| 2e20476559 | |||
| 990641b19e | |||
| 3ca2f3b499 | |||
| b9fa760404 | |||
| ab61839685 | |||
| 34fd40c3ad | |||
| a0eb1b9e35 | |||
| 9d677db2a8 | |||
| c728a0bd41 | |||
| 64336c0c89 | |||
| 59c944618b | |||
| 3c25172dfb | |||
| eda61eaade | |||
| 04b1a7c6e7 | |||
| 3b59dab2df | |||
| 389426f8f2 | |||
| e9dba11b96 | |||
| 828cbb5cd0 | |||
| 93a42ba5e4 | |||
| aaacc6ecb2 | |||
| bbc7a1d8b2 | |||
| 5f00124a86 |
+1
-1
Submodule .claude/commands updated: 3f98090747...2866e07882
@@ -0,0 +1,127 @@
|
||||
---
|
||||
type: concept
|
||||
title: CMTrace Time Resolution — Per-Line Self-Anchoring
|
||||
description: How the agent converts bias-less CMTrace local times to UTC without trusting any process's timezone belief — each provably fresh line anchors its own offset against the agent clock on the 15-minute grid; everything not provably fresh falls back uniformly and says so.
|
||||
resource: src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/Ime
|
||||
tags: [agent, ime, cmtrace, timestamps, timezone, calibration, provenance]
|
||||
timestamp: 2026-08-20
|
||||
---
|
||||
|
||||
# Problem
|
||||
|
||||
A CMTrace line carries local time and no offset — IME 1.104 writes `DateTime.Now.TimeOfDay`
|
||||
in both of its trace listeners. Converting such a line with the reader's own
|
||||
`TimeZoneInfo.Local` silently assumes the writing and reading processes agree on the zone.
|
||||
They do not: each caches its zone at process start and follows neither a later `tzutil` nor
|
||||
a Windows auto-timezone change. The error is `Offset_writer − Offset_reader` — zero only
|
||||
when both believe the same thing, right or wrong. Field measurement over 11,068 sessions
|
||||
found 26 sessions with a real error, from +1 h to −17 h.
|
||||
|
||||
Two designs failed before this one:
|
||||
|
||||
* **Fix the reader's belief** (`TimeZoneInfo.ClearCachedData`, live OS offset): makes the
|
||||
agent isolated-correct and thereby breaks the majority of sessions, where two equally
|
||||
stale beliefs cancel.
|
||||
* **One measured offset per file** (agent 2.0.1410, reverted in `04b1a7c6`): a single log
|
||||
file holds lines from multiple writer *eras*. `AgentExecutor.log` is written by
|
||||
short-lived child processes whose zone belief flips per process — interleaved in one
|
||||
file, and flipping *back* (fixture:
|
||||
`tests/fixtures/cmtrace-logs/agentexecutor-two-writer-eras-v1.cmtrace`). Any cross-line
|
||||
anchor — per file, or nearest-in-write-order — applies one era's offset to the other
|
||||
era's lines. In the field this shifted `script_started` events by −9 h against their
|
||||
correct completions.
|
||||
|
||||
# Schema
|
||||
|
||||
The resolution rule, in order:
|
||||
|
||||
1. **Writer-declared bias** (`+480` suffix): authoritative, used as-is. Never a calibration
|
||||
anchor — it needs no measurement and must not overwrite one.
|
||||
2. **Per-line self-anchoring** — only for lines read in a *provably fresh* pass:
|
||||
|
||||
```
|
||||
offset_line = round((local − agentUtcNow) / 15 min) · 15 min
|
||||
lineUtc = local − offset_line
|
||||
```
|
||||
|
||||
Every real UTC offset is a whole multiple of 15 minutes, so the rounding absorbs poll
|
||||
and flush latency. No cross-line state exists, so interleaved eras resolve line by
|
||||
line — the property the fixture test pins. The line's own timestamp still contributes
|
||||
sub-poll precision and ordering (two lines 6 ms apart stay 6 ms apart; a plain
|
||||
`occurredAt = now` could not do that). Guards: grid residual ≤ 2 min, offset within
|
||||
UTC−12 … UTC+14.
|
||||
3. **Uniform reader-zone fallback** for everything else: wrong in absolute terms whenever
|
||||
the writer held a different belief, but *self-consistent* — both ends of a duration are
|
||||
wrong by the same amount, so derived durations stay right. The revert's core lesson:
|
||||
partially corrected is strictly worse than uniformly wrong.
|
||||
|
||||
**Freshness** is what makes rule 2 sound, and it is strict:
|
||||
|
||||
* Every poll look at a file stamps `LastCheckedUtc` (`LogFilePositionTracker.MarkChecked`),
|
||||
including "no new data" and an empty first sight — so a file first seen empty
|
||||
(`AgentExecutor.log` appearing mid-enrollment) anchors from its very first content.
|
||||
* A pass's lines are fresh only if the file was previously observed *in this process* and
|
||||
the gap since the last look is ≤ `FreshLineMaxAge` (30 s — far above the 100 ms poll,
|
||||
far below the 13-minute zone where a line's *age* could round onto the offset grid).
|
||||
* A restart-restored position bookmark deliberately carries **no** freshness: the first
|
||||
pass after a restart reads downtime backlog and must never anchor.
|
||||
|
||||
**Provenance** (`sourceOffsetOrigin` in event DataJson): `bias` | `line-anchored` |
|
||||
`reader-zone-fallback` (plus retired `calibrated` from 2.0.1410 events). The applied
|
||||
offset is always recorded (`sourceOffsetMinutes`), the raw local time too
|
||||
(`sourceLocalTs`), so any event can be re-derived later. The per-*file* measurement still
|
||||
runs, purely observationally (`measuredWriterOffsetMinutes`, Info log when it disagrees
|
||||
with the process zone).
|
||||
|
||||
# Constraints
|
||||
|
||||
* An anchored line resolves to `now ± 2 min` *by construction*. The accepted edge: a
|
||||
freshly written line carrying a **replayed** old timestamp whose age is exactly
|
||||
N×15 min ± 2 min anchors to ≈now instead of its replayed past — bounded error, and
|
||||
sub-24 h replays are treated as current by the historic-replay guard anyway.
|
||||
* Snapshot readers (`StallProbeCollector`, `LogParserCollector`) cannot tail and therefore
|
||||
cannot prove freshness — they stay on the marked fallback. This is a documented limit,
|
||||
not an omission; sharing the per-file measurement with them would re-apply exactly what
|
||||
the revert removed.
|
||||
* Mixed pairs remain possible across a restart (start resolved from backlog fallback,
|
||||
completion line-anchored) and are recognizable via provenance.
|
||||
|
||||
# Backend regression tripwire
|
||||
|
||||
The backend watches for the failure mode this design eliminates. On every terminal ingest
|
||||
batch, the counter reconcile's existing event-partition scan additionally collects
|
||||
`Δ = ReceivedAt − OccurredUtc` per event, split by `Source == "ImeLogTracker"` versus the
|
||||
rest (`CmTraceSkewTripwire`, hooked in `EventIngestProcessor`). If
|
||||
`median(Δ_IME) − median(Δ_other)` is a clean 15-minute multiple (residual < 2 min — the
|
||||
same constants as the anchoring grid guard above) with ≥ 20 samples over ≥ 3 distinct
|
||||
upload batches per side, a `CmTraceTimeSkewRegression` ops event fires (category Agent,
|
||||
warning), carrying all numbers plus the session's `sourceOffsetOrigin` histogram.
|
||||
Bias-dominated sessions are suppressed (writer-declared offsets cannot be an anchoring
|
||||
regression); `measuredWriterOffsetMinutes` is never consulted (sticky after era
|
||||
flip-backs). Kill switch: app setting `CmTraceSkewTripwireDisabled`. Goal state: the event
|
||||
never fires — any hit is an anchoring case the per-line design misses, or a detector bug.
|
||||
|
||||
Device-clock problems (the customer-actionable cousin) are covered separately by the
|
||||
`clock_skew` analyze condition behind `ANALYZE-DEV-008`, which excludes IME-derived events
|
||||
precisely so an anchoring regression can never surface as a customer finding.
|
||||
|
||||
# Citations
|
||||
|
||||
* `CmTraceOffsetCalibrator.TryMeasureOffset` — the pure grid measurement, shared by
|
||||
anchoring and the observational per-file path.
|
||||
* `CmTraceSkewTripwire` + `CmTraceSkewTripwireTests` — the backend tripwire above.
|
||||
* `RuleEngine.ConditionEvaluators.EvaluateClockSkewCondition` + `RuleEngineClockSkewTests`
|
||||
— the customer-facing device-clock metrics (batch-median frames, spool-spread filter,
|
||||
plateau step detection, end-state persistence).
|
||||
* `ImeLogTracker.LogProcessing.cs` `ResolveEntryUtc` — the resolution order above.
|
||||
* `ImeLogTrackerLineAnchoringTests` — resolves-to-T across writer beliefs, interleaved
|
||||
eras in one chunk, freshness boundaries (first sight, empty first sight, poll gap,
|
||||
restart restore), the grid-replay edge, and the field fixture replayed at production
|
||||
poll cadence.
|
||||
* `CmTraceCalibratorFileIsolationTests` — pins per-file isolation of the observational
|
||||
measurement (session e9753578 logged a cross-file pairing the committed code could not
|
||||
be shown to produce; the mechanism remains open, the resolution path is immune by
|
||||
design, and `CalibrateFrom` logs the anchor timestamp as a tripwire).
|
||||
* `tasks/todo.md` (2026-08-20) — full evidence trail: field measurements, the revert,
|
||||
the two-writer-era fixture, and the disproven foreign-binary hypothesis (the SHA
|
||||
mismatch was the known release race against the 5-minute AdminConfiguration cache).
|
||||
@@ -143,11 +143,29 @@ advisory backstop. All routes converge on `CompleteThroughFinalizingOrDefer`:
|
||||
first-deployment window 100/101 for 200/210 without 110 — the RJ ESP was aborted,
|
||||
typically by an interactive logon reclassifying the run as secondary-user; emits
|
||||
the `realmjoin_first_deployment_incomplete` Warning, session 224b2087), and
|
||||
`Timeout` (hard 60-min deadline). The abort rule reads the *persisted*
|
||||
`Timeout` (60-min deadline from detection). The abort rule reads the *persisted*
|
||||
`RealmJoinFacts.LastDeploymentPhase` (updated by the typed `RealmJoinPhaseChanged`
|
||||
signal), so it survives agent restarts — the `RealmJoinDetected` replay path
|
||||
evaluates it too. A first observation of 210 does **not** release (RJ may have
|
||||
completed before agent boot and deploy afterwards).
|
||||
|
||||
**Activity-based timeout extension** (report 55e6afd61c9d — a large RJ package
|
||||
catalog outlived the 60-min window mid-install): when the timeout deadline fires
|
||||
while the first deployment is demonstrably active — `LastDeploymentPhase` 100/101
|
||||
*and* deployment activity (`RealmJoinPhaseChanged` or a package observation, tracked
|
||||
in `RealmJoinFacts.LastActivityUtc`; the phase captured at detection deliberately
|
||||
does not count) within the last 60 min — the handler re-arms the deadline to
|
||||
`lastActivity + 60 min` instead of timing out, emitting an Info
|
||||
`realmjoin_timeout_extended` event. Re-arms clamp to an absolute cap of
|
||||
**detection + 4 h**. Package activity is only observable at package *completion*
|
||||
(RJ writes the registry subkey when the install finishes), so a single install
|
||||
that is registry-silent for over 60 min still times out — the eventual
|
||||
`realmjoin_timeout` carries a `reason` payload (`hard_timeout` / `inactivity` /
|
||||
`absolute_cap`) telling the three shapes apart. A stale fire of an older deadline
|
||||
incarnation (armed due later than the fire time; `OccurredAtUtc = DueAtUtc` per
|
||||
scheduler contract) dead-ends as `realmjoin_timeout_stale_superseded_by_rearm`.
|
||||
The idle case (detected, no deployment activity ever) times out at 60 min
|
||||
unchanged, and phase 200/210 activity never extends.
|
||||
2. `Finalizing` stage + ~5 s `FinalizingGrace` deadline.
|
||||
3. Grace fires → `Completed`, `EnrollmentComplete` outcome, `enrollment_complete`
|
||||
timeline event with full audit payload.
|
||||
|
||||
@@ -8,5 +8,6 @@ How the on-device agent works:
|
||||
* [Hello Wizard Un-Skip](hello-wizard-unskip.md) - why a single "WHfB disabled" policy read never decides completion: the HelloWizardStarted signal (prevention + cure) and the HelloTracker confirmation second read.
|
||||
* [Autopilot ZTD Diagnostics](autopilot-ztd-diagnostics.md) - Windows' own diagnostic surfaces for the profile-download flow: ModernDeployment event IDs (807/815/164/...), the Diagnostics\Autopilot registry key, deployment-service endpoints, and the known-issue error-code map.
|
||||
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated 2800 observations (neutral, flush-time watermark) + ANALYZE-ESP-005 gated on an actually-observed reboot.
|
||||
* [Power-State Watcher](power-state-watcher.md) - live AC↔battery transitions and 50/30/15% downward threshold crossings via a WMI push subscription (no polling): power_state_change semantics (latch/debounce/cap, never armed without a battery) and the single-condition analyze rules ANALYZE-DEV-009/-010 on top.
|
||||
|
||||
Reading order for newcomers: overview → decision engine → persistence.
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
type: Concept
|
||||
title: Power-State Watcher — live AC/battery transitions and threshold events during enrollment
|
||||
description: How the agent observes AC↔battery switches and downward battery-threshold crossings mid-enrollment via a WMI push subscription (no polling), the power_state_change event semantics (transition field, 50/30/15 ladder, latch/debounce/cap), and the two analyze rules on top (ANALYZE-DEV-009/-010).
|
||||
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/PowerStateWatcherHost.cs
|
||||
tags:
|
||||
- agent
|
||||
- power
|
||||
- battery
|
||||
- collector
|
||||
- wmi
|
||||
- analyze-rules
|
||||
timestamp: 2026-08-21T00:00:00+02:00
|
||||
---
|
||||
|
||||
# Power-State Watcher
|
||||
|
||||
The startup probe (`power_state_check`, one-shot from `StartupEnvironmentProbes` via
|
||||
`PowerStateProbe`/`GetSystemPowerStatus`) only captures the power state at agent start. A
|
||||
device unplugged mid-enrollment — or draining its battery across a multi-hour ESP — was
|
||||
invisible. The watcher closes that gap **event-driven, with zero polling**.
|
||||
|
||||
# Mechanism
|
||||
|
||||
- `PowerStateWatcherHost` (`Orchestration/Hosts`) subscribes to the WMI push event
|
||||
`SELECT * FROM Win32_PowerManagementEvent WHERE EventCode = 10` (PBT_APMPOWERSTATUSCHANGE:
|
||||
fires on AC↔DC switches and battery-percentage steps, granularity set by the battery
|
||||
driver, typically 1–3%). The event carries no payload — each arrival re-probes via the
|
||||
existing `PowerStateProbe.Probe()` (cheap P/Invoke).
|
||||
- Snapshots are diffed by the pure `PowerStateTransitionTracker`
|
||||
(`Monitoring/Enrollment/SystemSignals`), which decides what to emit. The host only owns
|
||||
WMI, a 5 s trailing-edge debounce timer (dock-flap AC→battery→AC inside the window
|
||||
collapses to a no-diff) and the emit path (`InformationalEventPost`, single rail).
|
||||
- **No battery / probe error ⇒ WMI is never armed** — desktops and VMs pay nothing. WMI
|
||||
arming runs on a background thread (WinMgmt can hang in OOBE; `ConsoleBypassWatcher`
|
||||
precedent); an arm failure emits one `collector_degraded` (`watcher_arm_failed`).
|
||||
- Registered as a plain always-on peripheral host after `NetworkChangeHost`, deliberately
|
||||
NOT under `PeriodicCollectorLifecycleHost`: it is push-based (zero idle cost) and must be
|
||||
alive precisely when the device idles on battery. Kill-switch:
|
||||
`Collectors.EnablePowerStateWatcher` (default true, compile-time default like the other
|
||||
collector toggles; ConfigVersion 38).
|
||||
|
||||
# Event: power_state_change
|
||||
|
||||
`data.transition` discriminates three shapes (payload always carries `onAcPower`,
|
||||
`batteryPercent`, `isCharging`, `batteryLifeMinutes`; unknowns as `"unknown"`):
|
||||
|
||||
| transition | Severity | Immediate | Notes |
|
||||
|---|---|---|---|
|
||||
| `ac_to_battery` | Warning | yes | one per accepted edge |
|
||||
| `battery_to_ac` | Info | no | recovery |
|
||||
| `threshold_crossed` | Info (50) / Warning (30) / Error (15) | 30+15 only | extra field `thresholdPercent` |
|
||||
|
||||
Threshold semantics (tracker):
|
||||
|
||||
- Ladder 50/30/15, evaluated only while ON battery; **each level latches once per agent
|
||||
lifetime** (charge-up + re-drain does not re-emit — no plug-cycle alarm spam).
|
||||
- A multi-level jump emits only the LOWEST newly-crossed level (60%→10% ⇒ one `15` event).
|
||||
- Entering battery power already below a level counts as crossing it; the arm-time baseline
|
||||
does too — an enrollment STARTING at 12% battery emits its `thresholdPercent: 15` event
|
||||
immediately (the analyze rule would otherwise never see sessions that began low).
|
||||
- Lifetime emission cap 20 (storm backstop against a flapping dock / dying battery
|
||||
controller); the host logs one warning when it engages.
|
||||
|
||||
Backend/agent hygiene: `power_state_change` is classified as non-activity on both sides —
|
||||
`SignalActivityClassifier.NonActivityEventTypes` (agent: must not reset the
|
||||
PeriodicCollectorLifecycleHost/StallProbeHost idle clocks) and
|
||||
`EventIngestProcessor.IsPeriodicOrStallEvent` (backend: must not heal a Stalled session —
|
||||
battery drain is not enrollment progress).
|
||||
|
||||
# Analyze Rules
|
||||
|
||||
- **ANALYZE-DEV-009** (high): battery crossed 15% while on battery — matches the
|
||||
agent-stamped `thresholdPercent equals 15`.
|
||||
- **ANALYZE-DEV-010** (warning): AC→battery switch — matches `transition equals
|
||||
ac_to_battery`.
|
||||
- Both: `evaluateOn: ["enrollment_end", "whiteglove_sealed", "on_event:power_state_change"]`
|
||||
(live interim finding + terminal finalization), `markSessionAsFailedDefault: false`.
|
||||
- Both are deliberately **single-required-condition**: `event_data` conditions scan events
|
||||
independently (no same-instance join), so a two-condition rule (e.g. `transition ==
|
||||
threshold_crossed AND batteryPercent < 15`) could false-positive across two different
|
||||
power events. `thresholdPercent` exists only on threshold events and its value pins the
|
||||
ladder level, so one condition fully identifies the instance.
|
||||
- `{{token}}` caveat: `AddDataFieldsToEvidence` whitelists only generic identifiers
|
||||
(appId/errorCode/…), so `{{batteryPercent}}` would render literally. DEV-009 interpolates
|
||||
`{{thresholdPercent}}` (the matched condition's own dataField); the exact percentage lives
|
||||
on the linked event in the timeline.
|
||||
|
||||
# Citations
|
||||
|
||||
- `src/Agent/AutopilotMonitor.Agent.V2.Core/Orchestration/Hosts/PowerStateWatcherHost.cs`
|
||||
- `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Enrollment/SystemSignals/PowerStateTransitionTracker.cs`
|
||||
- `src/Agent/AutopilotMonitor.Agent.V2.Core/Runtime/PowerStateProbe.cs` (probe, unchanged)
|
||||
- `rules/analyze/ANALYZE-DEV-009.json`, `rules/analyze/ANALYZE-DEV-010.json`
|
||||
- [analyze-rule-triggers](../rules/analyze-rule-triggers.md) (interim-trigger design; the WG battery rule was its motivating case)
|
||||
@@ -0,0 +1,82 @@
|
||||
---
|
||||
type: concept
|
||||
title: Attempt-Scoped App Install Durations
|
||||
description: AppInstallSummary.DurationSeconds measures the status-defining (last) install attempt via a LastAttemptStartedAt max-fold anchor — replacing the first-observation-to-last-terminal span that billed multi-pass IME processing gaps to trivial apps; time attribution counts the union of active attempt segments.
|
||||
resource: src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.AppInstall.cs
|
||||
tags: [backend, agent, ingest, app-installs, metrics, time-attribution, durations]
|
||||
timestamp: 2026-08-21
|
||||
---
|
||||
|
||||
# Problem
|
||||
|
||||
The IME processes the ESP app list in multiple passes: a device-ESP evaluation pass can
|
||||
touch an app (often ending `Skipped`) long before the real install runs in a later pass.
|
||||
`AppInstallSummary` is keyed by `{SessionId}_{AppName}` and folded from ALL of the app's
|
||||
events, so `DurationSeconds = last terminal − first observation` spanned those passes.
|
||||
Field case (session `44862cd3`, "Set TimeZone"): evaluation pass 09:16, real install
|
||||
10:22:58 → 10:23:21 — reported 4002 s for a 23-second install. Fleet averages, the
|
||||
Slowest-Apps ranking, p95s, and the version-regression radar all inherited the inflation;
|
||||
the rule engine's `app_install_duration` (already attempt-scoped) contradicted the
|
||||
dashboard for the same install.
|
||||
|
||||
# Schema
|
||||
|
||||
Three durable anchors on the row — each with a distinct, non-interchangeable job:
|
||||
|
||||
| Column | Fold | Role |
|
||||
|---|---|---|
|
||||
| `StartedAt` | min (earliest wins) | First observation. Server-side window/bucket filter of every AppInstallSummaries scan and the radar's version-ordering key. Never re-anchored. |
|
||||
| `LastAttemptStartedAt` | **max** (in-batch and in `ReconcileAppInstallSummaryWithExisting`) | Anchor of `DurationSeconds`. Null = sentinel (row predates 2026-08, or no start observed) → duration falls back to the historical `StartedAt` span. Replay of an older batch can never regress it. |
|
||||
| `InstallPassCount` | additive, guarded by `LastAttemptStartedAt` monotonicity (a batch whose newest start is not newer than the stored anchor adds 0) | Explains "finished long after first seen" (web ×N chip); 2+ is normal. |
|
||||
|
||||
`DurationSeconds = CompletedAt − LastAttemptStartedAt`, computed at the terminal event and
|
||||
re-derived in one central reconcile site covering in-order and Q4 out-of-order batches.
|
||||
A `LastAttemptStartedAt` newer than `CompletedAt` means a fresh pass is in flight — the
|
||||
completed attempt's stored duration stands (the 0 value omits the column; Merge preserves).
|
||||
`DownloadDurationSeconds` pairs the latest `app_download_started` with the NEXT
|
||||
`app_install_started` (attempt-scoped, in-batch). The full span stays derivable as
|
||||
`CompletedAt − StartedAt` — deliberately not a column.
|
||||
|
||||
Weaker-terminal guard (both in-batch and cross-batch): `app_install_completed` with
|
||||
state `Skipped`/`Postponed` never overrides an `Installed`/`Error` terminal, nor its
|
||||
`CompletedAt`/`DurationSeconds` — mirroring the long-pinned `app_install_skipped` rule.
|
||||
|
||||
# Consumers
|
||||
|
||||
* **Fleet metrics / apps dashboard / PDF** (`MetricsMath`, `AppsAnalyticsHelper`): values
|
||||
drop to attempt scale; labels say "install time (final attempt)". The 6 h
|
||||
`MaxPlausibleInstallDurationSeconds` gate remains (an attempt longer than the agent's
|
||||
lifetime is still unobservable).
|
||||
* **AppVersionRegressionRadar**: cutover gate in `MeasuredAppGroups` — only rows with
|
||||
`LastAttemptStartedAt` enter the duration population, on BOTH comparison sides. A mixed
|
||||
population would mask real regressions and falsely re-arm active episodes for up to the
|
||||
35-day horizon. `MinMeasuredInstalls` keeps it silent until enough post-cutover rows
|
||||
exist. Thresholds (2.0× / 300 s) were tuned on span data — re-evaluate ~4 weeks after
|
||||
rollout.
|
||||
* **Time attribution** (`TimeAttributionCalculator.BuildBlockingAppIntervals`):
|
||||
sequence-ordered start→terminal pairing yields one segment per pass/attempt. Per-app
|
||||
seconds = SUM of clamped segments (failed attempts and evaluation passes count — they
|
||||
occupied the path); occupancy merges the segments, never the per-app hulls; hull
|
||||
endpoints remain first-start→last-terminal for display and what-if. Deliberately
|
||||
BROADER than the metrics duration: active time across all passes vs. final attempt.
|
||||
* **Agent payloads** (`AppInstallTiming` / `ImeLogTrackerAdapter.UpdateAppTiming`): a
|
||||
terminal→active transition re-arms BOTH endpoints, so event `durationSeconds` also
|
||||
describes the attempt (needs the agent release carrying this change).
|
||||
* **Rule engine** `app_install_duration`: was already attempt-scoped — now consistent.
|
||||
|
||||
# Rollout
|
||||
|
||||
No backfill (user decision, analog to the CMTrace legacy): pre-cutover rows keep span
|
||||
values, so time-series charts step down visibly at the changeover (documented in the
|
||||
public platform changelog). Sentinel-gated columns follow the standard chain:
|
||||
`BuildAppInstallSummaryEntity` + `MapToAppInstallSummary` + `AppsDashboardProjection`
|
||||
(the only projection whose consumers read the new columns).
|
||||
|
||||
# Citations
|
||||
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.AppInstall.cs` — fold + weaker-terminal guard + attempt pairing
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Metrics.cs` — reconcile max-fold, pass-count guard, central duration recompute
|
||||
* `src/Backend/AutopilotMonitor.Functions/Helpers/AppVersionRegressionRadar.cs` — cutover gate
|
||||
* `src/Backend/AutopilotMonitor.Functions/Helpers/TimeAttributionCalculator.cs` — per-pass segments + occupancy
|
||||
* `src/Agent/AutopilotMonitor.Agent.V2.Core/SignalAdapters/ImeLogTrackerAdapter.cs` — payload timing re-arm
|
||||
* Tests: `AppInstallTerminalStateAndReconcileTests`, `EventTimestampValidationTests`, `AppVersionRegressionRadarTests`, `TimeAttributionCalculatorTests`
|
||||
@@ -0,0 +1,84 @@
|
||||
---
|
||||
type: concept
|
||||
title: Send-Time Frame Separation (X-Send-Time-Utc)
|
||||
description: One device-clock send timestamp per upload batch splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset — turning clock diagnostics from median statistics into direct per-request measurement.
|
||||
resource: src/Backend/AutopilotMonitor.Functions/Functions/Ingest/IngestTelemetryFunction.cs
|
||||
tags: [backend, agent, ingest, timestamps, clock-skew, telemetry, provenance]
|
||||
timestamp: 2026-08-21
|
||||
---
|
||||
|
||||
# Problem
|
||||
|
||||
Events carry two time anchors: `OccurredUtc` (device clock at emission) and `ReceivedAt`
|
||||
(server clock at ingest). Their difference conflates two physically distinct phenomena:
|
||||
|
||||
* **Spool delay** — the event waited offline or in retry backoff before upload. Expected,
|
||||
harmless, minutes to hours.
|
||||
* **Device-vs-server clock offset** — the actual diagnostic signal.
|
||||
|
||||
A uniform spool backlog (events emitted close together, uploaded after an outage) is
|
||||
indistinguishable from a clock offset in that delta; a draining backlog fakes a clock jump
|
||||
ramp. Every consumer had to filter this statistically (spread caps, plateau guards,
|
||||
persistence tests) — and the first field clock-jump measurement failed exactly on this
|
||||
ambiguity.
|
||||
|
||||
# Schema
|
||||
|
||||
The agent stamps one header per upload attempt (`BackendTelemetryUploader`):
|
||||
|
||||
```
|
||||
X-Send-Time-Utc: <DateTime.UtcNow, ISO-8601 round-trip>
|
||||
```
|
||||
|
||||
Per **attempt**, not per batch content: a retry after backoff is a new send moment. The
|
||||
ingest function parses it (`ParseSendTimeHeader`: round-trip parse, UTC-normalized, values
|
||||
before year 2000 rejected as garbage — deliberately **no upper clamp**, a future-dated
|
||||
send time IS the clock-error measurement), and `StampServerFields` writes it as `SentAt`
|
||||
onto every event of the request — server-stamped, request-level, never trusted per-event
|
||||
from the wire. Stored as the nullable `SentAt` column on Events (absent for pre-header
|
||||
agents), mapped back everywhere events are read, exposed as `sentAt` in the `fields=`
|
||||
projection and verbatim on the raw endpoints.
|
||||
|
||||
The separation is then exact:
|
||||
|
||||
```
|
||||
SentAt − OccurredUtc = pure spool delay (both device frame; offset cancels)
|
||||
ReceivedAt − SentAt = network latency + clock offset (the only frame crossing)
|
||||
```
|
||||
|
||||
Network latency is seconds; anything beyond is device clock error, per request, no
|
||||
statistics required.
|
||||
|
||||
# Consumers
|
||||
|
||||
* **`clock_skew` analyze condition** (ANALYZE-DEV-008): batches whose events carry
|
||||
`SentAt` measure the batch offset directly as `SentAt − ReceivedAt` — spool-immune by
|
||||
construction, so the spread cap and the IME/clamped exclusions are unnecessary there
|
||||
(no event timestamps are read at all; IME-only sessions become measurable). Batches
|
||||
without `SentAt` keep the legacy per-event median + spread-cap path. Downstream logic
|
||||
(stability check, plateau step detection, end-state persistence) is shared; evidence
|
||||
carries `sentAtBatchCount`.
|
||||
* **`CmTraceSkewTripwire` deliberately unchanged**: `median(Δ_IME) − median(Δ_other)`
|
||||
cancels the device clock offset as common mode — send time adds nothing to that
|
||||
differential measurement.
|
||||
|
||||
# Constraints
|
||||
|
||||
* `SentAt` comes from the same device wall clock as the event timestamps — a clock jump
|
||||
between emission and send lands in the spool-delay term. That is the intended place for
|
||||
it to become visible, not a defect.
|
||||
* Signals and DecisionTransitions do not carry `SentAt` (no consumer); the header is
|
||||
request-level, so extending them later is a storage-only change.
|
||||
* Fleet coverage grows with agent rollout; all analyses must treat `SentAt == null`
|
||||
(legacy rows, old agents) as "legacy path", never as zero.
|
||||
|
||||
# Citations
|
||||
|
||||
* `BackendTelemetryUploader.UploadBatchAsync` — header stamping per attempt.
|
||||
* `IngestTelemetryFunction.ParseSendTimeHeader` + `IngestTelemetryFunctionTests` — parse
|
||||
contract incl. the no-upper-clamp decision.
|
||||
* `EventIngestProcessor.StampServerFields` + `IngestCriticalPathTests` — server-side
|
||||
stamping, forged per-event values overridden.
|
||||
* `RuleEngine.ConditionEvaluators.EvaluateClockSkewCondition` + `RuleEngineClockSkewTests`
|
||||
— the two measurement paths (the backlog-alias test pair pins why SentAt exists).
|
||||
* `tasks/todo.md` P14 (2026-08-21) — decision trail incl. why the tripwire stays as is.
|
||||
@@ -27,8 +27,10 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
* [ESP Completion Starvation — the sits-d Cloud-PC complex](agent/post-reboot-completion-recovery.md) - Why a SkipUser=true flow (no user ESP page → no Shell-Core 62407, no AccountSetup registry) left the completion gate satisfied but never evaluated — five Cloud PCs idled six hours at "waiting on: hello_resolution" — and the fix: the observed skip stands in for the final-exit evidence in the IME user-session completion attempt. Plus the two adjacent defects fixed alongside: the edge-only user-apps-settled synthesis (now level-triggered) and the never-wired Shell-Core restart replay (now Hello-wizard-only, with the reasoning why a replayed 62407 must never reach the reducer).
|
||||
* [Secure Boot CA-2023 Detection](agent/secureboot-ca2023-detection.md) - Why ANALYZE-SEC-001 v3 reads the UEFI db/KEK variables directly instead of trusting the SecureBoot\Servicing registry, and how the one-sided uefiCa2023FirmwareConfirmed marker plus a not_exists precondition give absence-tolerant suppression across old and new agents.
|
||||
* [Continue-Anyway Observation Mode](agent/continue-anyway-observation.md) - EnableEspContinueAnywayObservation (ConfigVersion 37, operator-set): Device-phase ESP terminal failures on Continue-Anyway profiles get a bounded 60-min observation instead of an immediate hard fail; the DAD-validated real-user desktop + Hello gate detect the true end, the session completes Succeeded + EspSoftFailure (amber "with issues" badge), expiry un-defangs to the original esp_terminal_failure.
|
||||
* [CMTrace Time Resolution](agent/cmtrace-time-resolution.md) - Converting bias-less CMTrace local times to UTC without trusting any process's timezone belief: per-line self-anchoring on the 15-minute offset grid for provably fresh lines, uniform marked reader-zone fallback for backlog, and the provenance vocabulary (bias / line-anchored / reader-zone-fallback) that keeps every event re-derivable.
|
||||
* [Registry App-State Second Pillar](agent/registry-app-state-pillar.md) - IME's authoritative per-app state read straight from the registry (Win32Apps EnforcementStateMessage, EspTrackingWin32Apps, StatusServiceReports) via a snapshot-and-diff watcher; registry/log reconciliation doubles as the built-in IME log-pattern drift alarm.
|
||||
* [WDP Scenario Gates & Completion Backstop](agent/wdp-scenario-gates.md) - How the agent gates Autopilot/ESP-only machinery off on Windows Device Preparation sessions (deterministic ExecutionContext marker, fail-toward-Classic), which noise it suppresses (ZTD probe, esp_config unknown, ModernDeployment 100/417/418/1005, starved sweep), and the WDP completion model: High-confidence seed, desktop-based arm D, and the 30-min device_prep_completion backstop.
|
||||
* [Power-State Watcher](agent/power-state-watcher.md) - Live AC↔battery transitions and 50/30/15% downward battery-threshold crossings during enrollment via a WMI Win32_PowerManagementEvent push subscription (zero polling, never armed on battery-less devices): the power_state_change event (transition field, per-level lifetime latch, 5s debounce, emission cap) and the analyze rules ANALYZE-DEV-009 (battery <15%, high) / ANALYZE-DEV-010 (AC→battery, warning) with on_event interim triggers.
|
||||
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait is keyed to the Intune-registered "Sidecar" provider by NAME: co-management's "ConfigMgr" parks the user ESP at "Apps (Identifying)" for days even with TrackingPoliciesCreated=1 (issue #106); the always-on two-arm esp_policy_provider_stalled tripwire (provider_incomplete | sidecar_provider_missing) and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
|
||||
|
||||
# Rules
|
||||
@@ -36,14 +38,17 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
* [Gather Rule Phase Scoping, One-Shot Triggers & Emit-on-Change](rules/gather-rule-phase-scoping.md) - Restricting gather rules to enrollment phases (activePhases / activeFromPhase sticky latch), one-shot collection at a phase's start/end (phase_change / phase_exit), and emitMode "on_change" result-dedup.
|
||||
* [Gather Rule Guardrails](rules/gather-rule-guardrails.md) - Why the allowlists in rules/guardrails.json are enforced on the agent and nowhere else, which hard blocks survive unrestricted mode, and the rule that every collector must call a guard and emit security_warning on refusal.
|
||||
* [Rule Authoring Surface](rules/rule-authoring-surface.md) - How customer AIs author rules against the real contract: the baked MCP resources (guide/schemas/guardrails via combine.js), the MCP-local validate_rule pre-flight, and the side-effect-free POST rules/analyze/dryrun whose trace twin shares the production evaluators (parity + side-effect-freedom pinned by tests).
|
||||
* [Rule ID Namespace & Collision Guards](rules/rule-id-namespace.md) - Why rule identity is the human-readable ruleId (no GUIDs) with tenant isolation via Table Storage partitioning, the reserved built-in namespace (RuleIdPolicy), which collisions are possible (custom-vs-custom across tenants: allowed and harmless), and the guards keeping the global RuleStats aggregate catalog-only and the global partition inside the reserved namespace (combine.js, reseed, cross-tenant RuleState GC).
|
||||
* [Analyze Rule Evaluation Triggers](rules/analyze-rule-triggers.md) - Opt-in `evaluateOn` interim triggers (`whiteglove_sealed`, `on_event:<type>`) so rules can fire before a session is terminal, with RuleResults update-semantics (interim → finalize/resolve, NotifiedAt notification dedupe) replacing the permanent dedupe freeze, KO/stat suppression on interim runs, the ingest-side cached trigger registry, and the authoring lints that keep interim rules monotonic.
|
||||
|
||||
# Backend
|
||||
|
||||
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
|
||||
* [Send-Time Frame Separation](backend/send-time-frame-separation.md) - One device-clock send timestamp per upload batch (X-Send-Time-Utc → SentAt on Events) splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset; the clock_skew condition measures SentAt batches directly (spool-immune), the CMTrace tripwire deliberately stays on its differential math.
|
||||
* [OOBE Bootstrap go-Code Flow](backend/oobe-bootstrap-go-flow.md) - The bootstrap short-code lifecycle (portal-created session → `irm go.autopilotmonitor.com/CODE | iex` at Shift+F10 → GetBootstrapScriptFunction), the always-HTTP-200 text/plain contract, the two-line injection defense (GUID/URL/expiry validator + single-quoted PS literals, golden-pinned against the legacy TS template), and the per-IP rate-limit story behind Front Door.
|
||||
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
|
||||
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
|
||||
* [Attempt-Scoped App Install Durations](backend/app-install-duration-attempt.md) - AppInstallSummary.DurationSeconds = status-defining (last) install attempt via the LastAttemptStartedAt max-fold anchor instead of the first-observation→last-terminal span that billed multi-pass IME gaps to trivial apps; InstallPassCount, the weaker-terminal guard, the radar's cutover gate, and time attribution's union-of-attempt-segments occupancy.
|
||||
* [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract.
|
||||
* [App Version Duration Regression Radar](backend/app-version-regression-radar.md) - How app versions whose median install duration regressed vs. the previous version are detected (first-seen-ordered versions, median lift ≥2× plus a 5-minute absolute floor over measured installs only), how alert episodes live in the appversionregression tracker keyspace with drain/median re-arm semantics, and the bell/ops-event/analytics surfaces.
|
||||
* [Tenant Activation & the Once-Only Welcome Mail](backend/tenant-activation-welcome-mail.md) - How a signup becomes an activated tenant (manual approve or the auto-approve queue), and the write-then-read + conditional-marker arbitration that sends the welcome mail exactly once even though activation and the user-typed notification address race each other.
|
||||
|
||||
+18
@@ -1,7 +1,25 @@
|
||||
# Log
|
||||
|
||||
## 2026-08-21
|
||||
|
||||
* **New**: `agent/power-state-watcher.md` — live AC/battery observation during enrollment: `PowerStateWatcherHost` subscribes to the WMI push `Win32_PowerManagementEvent EventCode 10` (no polling; re-probes `GetSystemPowerStatus` per push, 5 s trailing-edge debounce, never armed on battery-less devices, `Collectors.EnablePowerStateWatcher` kill-switch, ConfigVersion 38). The pure `PowerStateTransitionTracker` emits `power_state_change` with `transition` = ac_to_battery (Warning, immediate) / battery_to_ac (Info) / threshold_crossed at 50/30/15% (each level latched once per lifetime, lowest-crossed-only on jumps, baseline-below-level covered, emission cap 20). Classified non-activity on both sides (SignalActivityClassifier + IsPeriodicOrStallEvent). New analyze rules ANALYZE-DEV-009 (thresholdPercent=15, high) and ANALYZE-DEV-010 (transition=ac_to_battery, warning), both with `on_event:power_state_change` interim triggers and deliberately single-required-condition (event_data conditions have no same-instance join).
|
||||
|
||||
* **Update**: `rules/rule-id-namespace.md` — the export→paste round-trip is no longer lossy: the JSON editors now map rule-shaped exports into the form (`gatherRuleToForm` unflattens the nested gather `parameters` object, analyze `jsonToForm` maps `evaluateOn` onto the eval checkboxes; `tags` and the effective `markSessionAsFailed` carry into the created rule). Both editors still accept the serialized-form shape; edit-mode pastes merge, with an explicit `parameters`/`evaluateOn` winning over stale flat fields.
|
||||
|
||||
* **New**: `rules/rule-id-namespace.md` — rule identity model: human-readable `ruleId` strings (no GUIDs), tenant isolation via Table Storage partitioning (`global` = built-in/community, tenantId = custom), the reserved built-in namespace from `RuleIdPolicy`, and the four collision guards: catalog-only global `RuleStats` aggregate (same-ID custom rules of unrelated tenants no longer sum into one `global_{ruleId}` row), combine.js duplicate/namespace fail, GitHub-reseed rejection of non-reserved IDs into the global partition, and the namespace gate on the cross-tenant RuleState GC.
|
||||
|
||||
* **New**: `backend/app-install-duration-attempt.md` — attempt-scoped app install durations: `DurationSeconds` = status-defining (last) attempt anchored on the new max-folded `LastAttemptStartedAt` (StartedAt stays the earliest-wins window/version-ordering key; legacy rows without the anchor keep the span fallback); `InstallPassCount` with replay-guarded additive fold; weaker-terminal guard (completed-with-Skipped never overrides Installed/Error, in-batch and cross-batch); attempt-scoped `DownloadDurationSeconds` pairing; the version-regression radar's both-sides cutover gate; time attribution counts the union of per-pass active segments (occupancy merges segments, hulls stay for display/what-if); agent payload timing re-arms both endpoints on terminal→active. No backfill — pre-cutover rows keep span semantics.
|
||||
|
||||
* **Update**: `agent/decision-engine.md` — activity-based RealmJoin timeout extension (report 55e6afd61c9d: a large RJ package catalog outlived the 60-min window mid-install — the timer arms at RJ-agent-MSI detection, so the first deployment had effectively ~44 min). When the `RealmJoinTimeout` deadline fires with `LastDeploymentPhase` 100/101 and deployment activity (phase change or package observation, persisted in `RealmJoinFacts.LastActivityUtc`; the detection-time phase capture deliberately excluded) inside the last 60 min, the deadline re-arms to `lastActivity + 60 min` (Info event `realmjoin_timeout_extended`), clamped at detection + 4 h. `realmjoin_timeout` now carries `reason` = `hard_timeout` / `inactivity` / `absolute_cap`; a fire of a superseded deadline incarnation dead-ends. Idle detection and phase-200/210 sessions keep the original 60-min behavior.
|
||||
|
||||
* **New**: `backend/send-time-frame-separation.md` — P14: the agent stamps one device-clock send timestamp per upload attempt (`X-Send-Time-Utc`), the ingest function server-stamps it as `SentAt` on every event of the request. `SentAt − OccurredUtc` is pure spool delay, `ReceivedAt − SentAt` is network + device-vs-server clock offset — the two were indistinguishable before and had to be separated statistically. The `clock_skew` condition measures SentAt-carrying batches directly (spool-immune, no spread cap, IME-only sessions become measurable; evidence carries `sentAtBatchCount`), legacy batches keep the median path; the CMTrace tripwire deliberately stays on its differential math (common-mode clock offset cancels there).
|
||||
|
||||
## 2026-08-20
|
||||
|
||||
* **Update**: `agent/cmtrace-time-resolution.md` — new "Backend regression tripwire" section: the P6 split. Operator side: `CmTraceSkewTripwire` piggybacks on the terminal counter reconcile's partition scan (`Δ = ReceivedAt − OccurredUtc` per source, 15-min grid + 2-min residual, ≥20 samples / ≥3 distinct batches per side, bias-dominated sessions suppressed, `measuredWriterOffsetMinutes` never consulted) and emits the `CmTraceTimeSkewRegression` ops event — goal state: never fires; kill switch `CmTraceSkewTripwireDisabled`. Customer side: new `clock_skew` analyze condition source (`skewMetric: clock_jump | sustained_offset`, threshold in seconds, batch-median frames with spool-spread filter, plateau step detection, end-state persistence, IME-derived events excluded) behind the new built-in `ANALYZE-DEV-008` (300 s = the Kerberos/Entra 5-minute skew boundary) — the during-enrollment complement of ANALYZE-DEV-007's one-shot NTP check. P5 (agent-side observability event) is subsumed by the tripwire.
|
||||
|
||||
* **New**: `agent/cmtrace-time-resolution.md` — the era-aware answer to the CMTrace timezone skew: a bias-less line read in a provably fresh pass anchors its own UTC offset against the agent clock (15-minute grid, 2-minute residual), so interleaved writer eras in one file resolve line by line; everything not provably fresh keeps the uniform reader-zone fallback and says so via `sourceOffsetOrigin`. Freshness is a per-file `LastCheckedUtc` stamped on every poll look (empty first sights included) with a 30-second window, never restored across restarts. Replaces the reverted per-file offset application (04b1a7c6); the per-file measurement stays observational, now with the anchor timestamp logged as a tripwire for the still-unexplained e9753578 label/anchor crossing.
|
||||
|
||||
* **Update**: `agent/post-reboot-completion-recovery.md` — retitled to "ESP Completion Starvation — the sits-d Cloud-PC complex" and reframed on the evidence: the reboot was a red herring. The five stuck Cloud PCs carried `SkipUserStatusPage=True` (read 10+ times across three agent processes, `True` every time) and Windows honoured it — no user ESP page, hence no Shell-Core 62407 and no AccountSetup provisioning registry, verified over a six-hour uninterrupted post-reboot run with the watcher armed. Arm B of `ShouldTransitionToAwaitingHello` was open from the bootstrap onward, but every evaluation site hangs off a signal a skip-user flow cannot produce, and the `AdvisoryCompletion` backstop arms exclusively from an `EspExiting` signal — an open gate nobody knocks on. Fix: `HandleImeUserSessionCompletedV1` accepts the observed skip in place of the final-exit evidence (`skipUserExitEquivalent`; anchor + genuine IME completion + real-user desktop stay mandatory, WDP excluded). The knock deliberately lives on the IME edge, not desktop arrival — ordering-robust against Fix 10's AccountSetup bounce-back and semantically "user-phase work finished" rather than mid-install. `ClassicSkipUserEspUserSessionCompletionTests` replays the real 8110e262 ordering end to end. Docs and comments no longer claim an ESP-exit recovery that does not exist (stray duplicate summary in `ShellCoreTracker`, `ImeLogHost` restore-callback comment); the residual `SkipUser=false` exit-in-downtime gap is named honestly and surfaced by `ANALYZE-ESP-005` via `session_stalled` instead of being patched with an unorderable replay.
|
||||
|
||||
## 2026-08-19
|
||||
|
||||
@@ -98,7 +98,7 @@ Interim evaluation changes what a rule's conditions mean mid-flight. Three new l
|
||||
|
||||
# Application to the Motivating Cases
|
||||
|
||||
- WG battery rule (sits-d): `evaluateOn: ["enrollment_end", "whiteglove_sealed"]` — technician gets the channel alert at the bench; Part-2 terminal pass finalizes or resolves the finding.
|
||||
- WG battery rule (sits-d): shipped 2026-08-21 as ANALYZE-DEV-009 (battery below 15% while on battery) and ANALYZE-DEV-010 (AC→battery switch), both `evaluateOn: ["enrollment_end", "whiteglove_sealed", "on_event:power_state_change"]` on the live power watcher's `power_state_change` events (see `../agent/power-state-watcher.md`) — technician gets the channel alert at the bench; the terminal pass finalizes the finding. Both are deliberately single-required-condition rules: `event_data` conditions scan events independently (no same-instance join), so DEV-009 matches the agent-stamped `thresholdPercent` and DEV-010 matches `transition`.
|
||||
- Hybrid-affinity stuck (esa): ANALYZE-ID-004 gets `evaluateOn: ["enrollment_end", "on_event:hybrid_login_pending"]` — the second overdue-login event (threshold 65) produces a live finding on the still-InProgress session. ANALYZE-APP-015 stays `enrollment_end`-only until its conditions are hardened (lint 1).
|
||||
|
||||
- Stuck after a forced mid-ESP reboot (sits-d Cloud PCs, 2026-08-19): `on_event:session_stalled` is the established trigger for the never-terminal stuck session. The agent emits `session_stalled` exactly once per session (60-minute stall probe), so the trigger is rare by construction and needs no throttle. Five Cloud PCs sat at `InProgress` with **zero findings** for hours — a mid-ESP reboot had killed the agent in `Stage=EspAccountSetup` — before the max-lifetime watchdog eventually terminalised them as `Incomplete`. ANALYZE-ESP-005 carries the trigger and fires on that stream at confidence 95 (verified against session `8110e262`): both its conditions (`mdm_policy_reboot_required.firstRebootUri` exists, `system_reboot_detected` count ≥ 1) are **monotonic** and it has no terminal preconditions, so lint 1 does not apply. ANALYZE-APP-016 is interim-safe for the same reason (`app_install_starved` count ≥ 1 only ever grows).
|
||||
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
type: concept
|
||||
title: Rule ID Namespace & Collision Guards
|
||||
description: How gather/analyze rule identity is scoped (tenant partition vs global partition), why there are no GUIDs, which collisions are possible, and the guards that keep the reserved built-in namespace and the global RuleStats aggregate clean.
|
||||
resource: backend, rules
|
||||
tags: [rules, ruleid, namespace, collision, rulestats, tenant-isolation]
|
||||
timestamp: 2026-08-21
|
||||
---
|
||||
|
||||
# Schema
|
||||
|
||||
Rule identity is the human-readable `ruleId` string — there are no GUIDs, deliberately: the ID appears in logs, mails, notifications, RuleResults, the agent config payload, and docs, and tenant isolation is provided by Table Storage partitioning instead.
|
||||
|
||||
| Store | PartitionKey | RowKey | Content |
|
||||
|---|---|---|---|
|
||||
| `GatherRules` / `AnalyzeRules` | `"global"` | ruleId | Built-in + community rules (shared, all tenants) |
|
||||
| `GatherRules` / `AnalyzeRules` | tenantId | ruleId | Custom rules (visible only to that tenant) |
|
||||
| `RuleStates` | tenantId | ruleId | Per-tenant enable/notify overrides — built-in/community rules ONLY (custom rules fold state into their own row) |
|
||||
| `RuleStats` | date | `{tenantId}_{ruleId}` and `global_{ruleId}` | Daily fire/eval telemetry; the global row is catalog-only (see guards) |
|
||||
|
||||
Namespace split (`RuleIdPolicy.cs`, shared model):
|
||||
|
||||
- **Reserved built-in namespace**: `(ANALYZE|GATHER)-<CATEGORY>-<NUMBER>` with `CATEGORY != CUSTOM`, case-insensitive, including currently unused numbers (gaps = retired rules that may return).
|
||||
- **Tenant namespace**: everything else — `ANALYZE-CUSTOM-NNN` (portal suggestion), `-CUSTOM` suffixes, org prefixes (`CONTOSO-WIFI-001`), freeform.
|
||||
|
||||
# Collision model
|
||||
|
||||
- **Custom vs built-in**: impossible via API — `CreateRuleAsync`/`UpdateRuleAsync` reject reserved IDs (409) and check existence against `global` ∪ own tenant partition. Legacy debris is neutralized at merge time: global wins, tenant copy dropped with a warning (pinned by `RuleMergeCollisionGuardTests`, motivated by the 5ca2b350 duplicate-key incident).
|
||||
- **Custom vs custom across tenants**: allowed and harmless — separate partitions, separate evaluation, separate RuleResults/tenant stats. Two tenants both using `ANALYZE-CUSTOM-101` never interact.
|
||||
- **JSON copy between tenants** (export → paste into create editor): goes through the same create path, so an ID already taken in the receiving tenant surfaces as 409 — the user renames. There is no bulk import endpoint. The JSON editors accept both shapes — a rule export and the serialized form — via `gatherRuleToForm` / `jsonToForm` (web `types.ts` of each rules page): nested gather `parameters` are unflattened, analyze `evaluateOn` maps onto the eval checkboxes, and `tags` plus the effective `markSessionAsFailed` survive the import.
|
||||
|
||||
# Guards (all enforced, 2026-08-21)
|
||||
|
||||
1. **Global RuleStats aggregate is catalog-only.** The `global_{ruleId}` row would otherwise sum same-ID custom rules of unrelated tenants (title/severity last-writer-wins). Analyze gates on `rule.IsBuiltIn || rule.IsCommunity` (`AnalyzeOnEnrollmentEndHandler`), gather gates on `RuleIdPolicy.IsReservedBuiltInId` (`EventIngestProcessor.RuleStats`) because agent events carry no flag — valid because custom rules can never occupy the reserved pattern.
|
||||
2. **Catalog build fails on namespace violations.** `rules/scripts/combine.js` rejects duplicate IDs (case-insensitive) and any gather/analyze built-in outside the reserved pattern; `BuiltInRuleCatalogPolicyTests` pins the same invariants on the embedded resource actually deployed.
|
||||
3. **GitHub reseed rejects non-reserved IDs** before writing to the `global` partition (`ReseedFromGitHubFunction`) — a seeded `*-CUSTOM-*` rule would silently shadow every tenant's same-ID custom rule at merge time.
|
||||
4. **Cross-tenant RuleState GC is namespace-gated.** `DeleteRuleStatesForRuleIdAcrossTenantsAsync` deletes by bare RowKey across all partitions; it now refuses non-reserved IDs with `failed=-1` (callers then skip the global catalog delete, same contract as enumeration failure).
|
||||
|
||||
Invariant chain: schema pattern + combine.js + reseed guard ⇒ the `global` partition only ever contains reserved-pattern IDs ⇒ the pattern is a safe built-in/custom discriminator everywhere no `IsBuiltIn` flag is available.
|
||||
|
||||
# Citations
|
||||
|
||||
- `src/Shared/AutopilotMonitor.Shared/Models/Rules/RuleIdPolicy.cs` — reserved pattern + rationale
|
||||
- `src/Backend/AutopilotMonitor.Functions/Services/AnalyzeRuleService.cs`, `GatherRuleService.cs` — create/update guards, merge collision guard
|
||||
- `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Rules.cs` — key schemes, cross-tenant GC guard
|
||||
- `src/Backend/AutopilotMonitor.Functions.Tests/BuiltInRuleCatalogPolicyTests.cs`, `RuleIdReservationTests.cs`, `RuleMergeCollisionGuardTests.cs`
|
||||
- `rules/scripts/combine.js` — build-time namespace/duplicate guard
|
||||
@@ -0,0 +1,69 @@
|
||||
{
|
||||
"$schema": "../schema/analyze-rule.schema.json",
|
||||
"ruleId": "ANALYZE-DEV-008",
|
||||
"title": "Device Clock Wrong During Enrollment — Jump or Sustained Offset",
|
||||
"description": "The device's clock changed by five or more minutes mid-session (clock jump) or ran five or more minutes off server time for the whole session (sustained offset), measured against the server receive frame of every uploaded event. Complements ANALYZE-DEV-007, which is a one-shot NTP check at agent startup and misses devices where NTP was unreachable or the clock changed later. IME-log-derived events are excluded from the measurement.",
|
||||
"severity": "warning",
|
||||
"category": "device",
|
||||
"version": "1.0.0",
|
||||
"author": "Autopilot Monitor",
|
||||
"enabled": true,
|
||||
"evaluateOn": [
|
||||
"enrollment_end",
|
||||
"whiteglove_sealed"
|
||||
],
|
||||
"trigger": "single",
|
||||
"baseConfidence": 60,
|
||||
"confidenceThreshold": 50,
|
||||
"conditions": [
|
||||
{
|
||||
"signal": "clock_jump",
|
||||
"source": "clock_skew",
|
||||
"skewMetric": "clock_jump",
|
||||
"operator": "gte",
|
||||
"value": "300",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"signal": "sustained_offset",
|
||||
"source": "clock_skew",
|
||||
"skewMetric": "sustained_offset",
|
||||
"operator": "gte",
|
||||
"value": "300",
|
||||
"required": false
|
||||
}
|
||||
],
|
||||
"confidenceFactors": [],
|
||||
"explanation": "This device's {{skewSummary}} (five-minute threshold). A clock that is off by five or more minutes breaks time-sensitive enrollment mechanisms: Kerberos tolerates at most 5 minutes of skew (Hybrid Join), Entra ID token validation allows about 5 minutes on token lifetimes, and freshly issued SCEP/PKCS certificates can be rejected as \"not yet valid\" when the clock is more than ~10 minutes behind the CA.\n\nUnlike ANALYZE-DEV-007 (a one-shot NTP comparison at agent startup), this finding is measured across the whole session against the server's receive time — it also catches devices where the NTP check could not run (UDP 123 blocked) or where the clock changed after startup (time sync correcting a wrong RTC, Windows auto-timezone, manual change).",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Fix the device clock and time source",
|
||||
"steps": [
|
||||
"Check the BIOS/RTC clock and the CMOS battery if the wrong time recurs on the same hardware.",
|
||||
"At OOBE, run 'w32tm /resync /force' (Shift+F10 command prompt) to sync with time.windows.com.",
|
||||
"Ensure UDP port 123 to time.windows.com is open in the deployment network.",
|
||||
"Review 'Set time automatically' and 'Set time zone automatically' behavior on the deployment image — automatic timezone changes mid-enrollment show up as clock jumps."
|
||||
]
|
||||
}
|
||||
],
|
||||
"relatedDocs": [
|
||||
{
|
||||
"title": "Microsoft Learn: Windows Autopilot known issues — device provisioning can fail (clock skew)",
|
||||
"url": "https://learn.microsoft.com/en-us/autopilot/known-issues"
|
||||
},
|
||||
{
|
||||
"title": "Microsoft Learn: Windows Autopilot requirements — Networking (NTP)",
|
||||
"url": "https://learn.microsoft.com/en-us/autopilot/requirements?tabs=networking"
|
||||
}
|
||||
],
|
||||
"markSessionAsFailedDefault": false,
|
||||
"tags": [
|
||||
"device",
|
||||
"clock-skew",
|
||||
"clock-jump",
|
||||
"time-sync",
|
||||
"kerberos",
|
||||
"certificates",
|
||||
"known-issue"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
{
|
||||
"$schema": "../schema/analyze-rule.schema.json",
|
||||
"ruleId": "ANALYZE-DEV-009",
|
||||
"title": "Battery Critically Low During Enrollment",
|
||||
"description": "Detects that the device's battery dropped to 15% or less while enrolling on battery power (power_state_change threshold event from the agent's live power watcher). A device dying mid-enrollment leaves a half-provisioned state that typically needs a full reset; this rule surfaces the risk while there is still time to plug the device in. Uses a single required condition on the agent-stamped thresholdPercent field (the data field 'transition' on the event is unrelated to this rule's 'trigger' property).",
|
||||
"severity": "high",
|
||||
"category": "device",
|
||||
"version": "1.0.0",
|
||||
"enabled": true,
|
||||
"evaluateOn": [
|
||||
"enrollment_end",
|
||||
"whiteglove_sealed",
|
||||
"on_event:power_state_change"
|
||||
],
|
||||
"trigger": "single",
|
||||
"baseConfidence": 90,
|
||||
"confidenceThreshold": 60,
|
||||
"conditions": [
|
||||
{
|
||||
"signal": "battery_threshold_15",
|
||||
"source": "event_data",
|
||||
"eventType": "power_state_change",
|
||||
"dataField": "thresholdPercent",
|
||||
"operator": "equals",
|
||||
"value": "15",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"explanation": "The device's battery fell below **{{thresholdPercent}}%** while the enrollment was running on battery power. If the battery drains completely mid-enrollment, the device is left in a half-provisioned state — apps, policies, or the user profile may be incomplete, and in most cases the device has to be reset and re-enrolled. The agent detected the threshold crossing live via a Windows power-status notification; the exact percentage at that moment is recorded on the matched power_state_change event in the session timeline.",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Secure power for the running enrollment",
|
||||
"steps": [
|
||||
"Connect the device to AC power immediately — the enrollment continues normally once power is stable.",
|
||||
"If the device shut down before completing, check whether the session reached the desktop; if not, reset the device (Autopilot reset or reimage) and re-enroll it plugged in.",
|
||||
"For technician/White Glove flows, make AC power part of the bench checklist — pre-provisioning on battery is the most common source of this alert.",
|
||||
"If this fires across many devices of one model, review that model's battery health (the hardware_spec event carries design vs. full-charge capacity)."
|
||||
]
|
||||
}
|
||||
],
|
||||
"markSessionAsFailedDefault": false,
|
||||
"tags": [
|
||||
"device",
|
||||
"power",
|
||||
"battery",
|
||||
"environment"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,48 @@
|
||||
{
|
||||
"$schema": "../schema/analyze-rule.schema.json",
|
||||
"ruleId": "ANALYZE-DEV-010",
|
||||
"title": "Enrollment Switched From AC to Battery Power",
|
||||
"description": "Detects that the device lost AC power and switched to battery while the enrollment was running (power_state_change event with transition ac_to_battery from the agent's live power watcher). Uses a single required condition so the match is pinned to one event instance; the event payload carries the battery percentage at the moment of the switch. The agent debounces power flapping (5s) so a brief unplug/replug does not produce this event.",
|
||||
"severity": "warning",
|
||||
"category": "device",
|
||||
"version": "1.0.0",
|
||||
"enabled": true,
|
||||
"evaluateOn": [
|
||||
"enrollment_end",
|
||||
"whiteglove_sealed",
|
||||
"on_event:power_state_change"
|
||||
],
|
||||
"trigger": "single",
|
||||
"baseConfidence": 85,
|
||||
"confidenceThreshold": 60,
|
||||
"conditions": [
|
||||
{
|
||||
"signal": "switched_to_battery",
|
||||
"source": "event_data",
|
||||
"eventType": "power_state_change",
|
||||
"dataField": "transition",
|
||||
"operator": "equals",
|
||||
"value": "ac_to_battery",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"explanation": "The device switched from AC to battery power mid-enrollment. Enrollments are power-hungry — app downloads, Windows Update and disk encryption drain a battery quickly, and Windows may also throttle CPU and network on battery, slowing the enrollment down. The battery percentage at the moment of the switch is recorded on the matched power_state_change event in the session timeline; if the device stayed on battery, look there for subsequent threshold events (50/30/15%) — a later battery_to_ac event means power was restored.",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Restore AC power",
|
||||
"steps": [
|
||||
"Reconnect the device to AC power; the enrollment continues normally.",
|
||||
"Check the session timeline for a subsequent battery_to_ac event (power restored) or threshold_crossed events (battery draining) to judge urgency.",
|
||||
"If this alert is frequent in a deployment area, check docking stations/USB-C hubs for loose power delivery — a flapping dock is a common cause.",
|
||||
"Make 'plugged in during setup' part of the end-user/technician enrollment instructions."
|
||||
]
|
||||
}
|
||||
],
|
||||
"markSessionAsFailedDefault": false,
|
||||
"tags": [
|
||||
"device",
|
||||
"power",
|
||||
"battery",
|
||||
"environment"
|
||||
]
|
||||
}
|
||||
+162
@@ -1516,6 +1516,168 @@
|
||||
"known-issue"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ruleId": "ANALYZE-DEV-008",
|
||||
"title": "Device Clock Wrong During Enrollment — Jump or Sustained Offset",
|
||||
"description": "The device's clock changed by five or more minutes mid-session (clock jump) or ran five or more minutes off server time for the whole session (sustained offset), measured against the server receive frame of every uploaded event. Complements ANALYZE-DEV-007, which is a one-shot NTP check at agent startup and misses devices where NTP was unreachable or the clock changed later. IME-log-derived events are excluded from the measurement.",
|
||||
"severity": "warning",
|
||||
"category": "device",
|
||||
"version": "1.0.0",
|
||||
"author": "Autopilot Monitor",
|
||||
"enabled": true,
|
||||
"evaluateOn": [
|
||||
"enrollment_end",
|
||||
"whiteglove_sealed"
|
||||
],
|
||||
"trigger": "single",
|
||||
"baseConfidence": 60,
|
||||
"confidenceThreshold": 50,
|
||||
"conditions": [
|
||||
{
|
||||
"signal": "clock_jump",
|
||||
"source": "clock_skew",
|
||||
"skewMetric": "clock_jump",
|
||||
"operator": "gte",
|
||||
"value": "300",
|
||||
"required": false
|
||||
},
|
||||
{
|
||||
"signal": "sustained_offset",
|
||||
"source": "clock_skew",
|
||||
"skewMetric": "sustained_offset",
|
||||
"operator": "gte",
|
||||
"value": "300",
|
||||
"required": false
|
||||
}
|
||||
],
|
||||
"confidenceFactors": [],
|
||||
"explanation": "This device's {{skewSummary}} (five-minute threshold). A clock that is off by five or more minutes breaks time-sensitive enrollment mechanisms: Kerberos tolerates at most 5 minutes of skew (Hybrid Join), Entra ID token validation allows about 5 minutes on token lifetimes, and freshly issued SCEP/PKCS certificates can be rejected as \"not yet valid\" when the clock is more than ~10 minutes behind the CA.\n\nUnlike ANALYZE-DEV-007 (a one-shot NTP comparison at agent startup), this finding is measured across the whole session against the server's receive time — it also catches devices where the NTP check could not run (UDP 123 blocked) or where the clock changed after startup (time sync correcting a wrong RTC, Windows auto-timezone, manual change).",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Fix the device clock and time source",
|
||||
"steps": [
|
||||
"Check the BIOS/RTC clock and the CMOS battery if the wrong time recurs on the same hardware.",
|
||||
"At OOBE, run 'w32tm /resync /force' (Shift+F10 command prompt) to sync with time.windows.com.",
|
||||
"Ensure UDP port 123 to time.windows.com is open in the deployment network.",
|
||||
"Review 'Set time automatically' and 'Set time zone automatically' behavior on the deployment image — automatic timezone changes mid-enrollment show up as clock jumps."
|
||||
]
|
||||
}
|
||||
],
|
||||
"relatedDocs": [
|
||||
{
|
||||
"title": "Microsoft Learn: Windows Autopilot known issues — device provisioning can fail (clock skew)",
|
||||
"url": "https://learn.microsoft.com/en-us/autopilot/known-issues"
|
||||
},
|
||||
{
|
||||
"title": "Microsoft Learn: Windows Autopilot requirements — Networking (NTP)",
|
||||
"url": "https://learn.microsoft.com/en-us/autopilot/requirements?tabs=networking"
|
||||
}
|
||||
],
|
||||
"markSessionAsFailedDefault": false,
|
||||
"tags": [
|
||||
"device",
|
||||
"clock-skew",
|
||||
"clock-jump",
|
||||
"time-sync",
|
||||
"kerberos",
|
||||
"certificates",
|
||||
"known-issue"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ruleId": "ANALYZE-DEV-009",
|
||||
"title": "Battery Critically Low During Enrollment",
|
||||
"description": "Detects that the device's battery dropped to 15% or less while enrolling on battery power (power_state_change threshold event from the agent's live power watcher). A device dying mid-enrollment leaves a half-provisioned state that typically needs a full reset; this rule surfaces the risk while there is still time to plug the device in. Uses a single required condition on the agent-stamped thresholdPercent field (the data field 'transition' on the event is unrelated to this rule's 'trigger' property).",
|
||||
"severity": "high",
|
||||
"category": "device",
|
||||
"version": "1.0.0",
|
||||
"enabled": true,
|
||||
"evaluateOn": [
|
||||
"enrollment_end",
|
||||
"whiteglove_sealed",
|
||||
"on_event:power_state_change"
|
||||
],
|
||||
"trigger": "single",
|
||||
"baseConfidence": 90,
|
||||
"confidenceThreshold": 60,
|
||||
"conditions": [
|
||||
{
|
||||
"signal": "battery_threshold_15",
|
||||
"source": "event_data",
|
||||
"eventType": "power_state_change",
|
||||
"dataField": "thresholdPercent",
|
||||
"operator": "equals",
|
||||
"value": "15",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"explanation": "The device's battery fell below **{{thresholdPercent}}%** while the enrollment was running on battery power. If the battery drains completely mid-enrollment, the device is left in a half-provisioned state — apps, policies, or the user profile may be incomplete, and in most cases the device has to be reset and re-enrolled. The agent detected the threshold crossing live via a Windows power-status notification; the exact percentage at that moment is recorded on the matched power_state_change event in the session timeline.",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Secure power for the running enrollment",
|
||||
"steps": [
|
||||
"Connect the device to AC power immediately — the enrollment continues normally once power is stable.",
|
||||
"If the device shut down before completing, check whether the session reached the desktop; if not, reset the device (Autopilot reset or reimage) and re-enroll it plugged in.",
|
||||
"For technician/White Glove flows, make AC power part of the bench checklist — pre-provisioning on battery is the most common source of this alert.",
|
||||
"If this fires across many devices of one model, review that model's battery health (the hardware_spec event carries design vs. full-charge capacity)."
|
||||
]
|
||||
}
|
||||
],
|
||||
"markSessionAsFailedDefault": false,
|
||||
"tags": [
|
||||
"device",
|
||||
"power",
|
||||
"battery",
|
||||
"environment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ruleId": "ANALYZE-DEV-010",
|
||||
"title": "Enrollment Switched From AC to Battery Power",
|
||||
"description": "Detects that the device lost AC power and switched to battery while the enrollment was running (power_state_change event with transition ac_to_battery from the agent's live power watcher). Uses a single required condition so the match is pinned to one event instance; the event payload carries the battery percentage at the moment of the switch. The agent debounces power flapping (5s) so a brief unplug/replug does not produce this event.",
|
||||
"severity": "warning",
|
||||
"category": "device",
|
||||
"version": "1.0.0",
|
||||
"enabled": true,
|
||||
"evaluateOn": [
|
||||
"enrollment_end",
|
||||
"whiteglove_sealed",
|
||||
"on_event:power_state_change"
|
||||
],
|
||||
"trigger": "single",
|
||||
"baseConfidence": 85,
|
||||
"confidenceThreshold": 60,
|
||||
"conditions": [
|
||||
{
|
||||
"signal": "switched_to_battery",
|
||||
"source": "event_data",
|
||||
"eventType": "power_state_change",
|
||||
"dataField": "transition",
|
||||
"operator": "equals",
|
||||
"value": "ac_to_battery",
|
||||
"required": true
|
||||
}
|
||||
],
|
||||
"explanation": "The device switched from AC to battery power mid-enrollment. Enrollments are power-hungry — app downloads, Windows Update and disk encryption drain a battery quickly, and Windows may also throttle CPU and network on battery, slowing the enrollment down. The battery percentage at the moment of the switch is recorded on the matched power_state_change event in the session timeline; if the device stayed on battery, look there for subsequent threshold events (50/30/15%) — a later battery_to_ac event means power was restored.",
|
||||
"remediation": [
|
||||
{
|
||||
"title": "Restore AC power",
|
||||
"steps": [
|
||||
"Reconnect the device to AC power; the enrollment continues normally.",
|
||||
"Check the session timeline for a subsequent battery_to_ac event (power restored) or threshold_crossed events (battery draining) to judge urgency.",
|
||||
"If this alert is frequent in a deployment area, check docking stations/USB-C hubs for loose power delivery — a flapping dock is a common cause.",
|
||||
"Make 'plugged in during setup' part of the end-user/technician enrollment instructions."
|
||||
]
|
||||
}
|
||||
],
|
||||
"markSessionAsFailedDefault": false,
|
||||
"tags": [
|
||||
"device",
|
||||
"power",
|
||||
"battery",
|
||||
"environment"
|
||||
]
|
||||
},
|
||||
{
|
||||
"ruleId": "ANALYZE-ENRL-001",
|
||||
"title": "Enrollment Failed",
|
||||
|
||||
@@ -169,7 +169,12 @@
|
||||
"source": {
|
||||
"type": "string",
|
||||
"description": "Source of the signal.",
|
||||
"enum": ["event_type", "event_data", "event_data_array", "phase_duration", "event_count", "app_install_duration", "event_correlation"]
|
||||
"enum": ["event_type", "event_data", "event_data_array", "phase_duration", "event_count", "app_install_duration", "event_correlation", "clock_skew"]
|
||||
},
|
||||
"skewMetric": {
|
||||
"type": "string",
|
||||
"description": "For clock_skew only: which device-clock metric to evaluate. clock_jump = persistent step in the device's clock frame mid-session; sustained_offset = whole session ran on a clock off by at least the threshold. value is the threshold in seconds; operator is limited to gt/gte on the magnitude. IME-log-derived events are excluded from the measurement.",
|
||||
"enum": ["clock_jump", "sustained_offset"]
|
||||
},
|
||||
"eventType": {
|
||||
"type": "string",
|
||||
|
||||
@@ -13,18 +13,26 @@ const rulesRoot = path.resolve(__dirname, '..');
|
||||
|
||||
// ── Rule combination ────────────────────────────────────────────────────────
|
||||
|
||||
// Shipped gather/analyze rules must stay inside the reserved built-in namespace:
|
||||
// (ANALYZE|GATHER)-<CATEGORY>-<NNN> with CATEGORY !== CUSTOM. The CUSTOM category is
|
||||
// the sanctioned tenant namespace (RuleIdPolicy.cs) — a built-in shipped there would
|
||||
// silently shadow every tenant's same-ID custom rule at merge time (global wins).
|
||||
const RESERVED_BUILTIN_ID = /^(ANALYZE|GATHER)-(?!CUSTOM-)[A-Z]+-\d{3}$/;
|
||||
|
||||
const configs = [
|
||||
{
|
||||
dir: path.join(rulesRoot, 'gather'),
|
||||
output: path.join(rulesRoot, 'dist', 'gather-rules.json'),
|
||||
schema: '../schema/gather-rule.schema.json',
|
||||
idField: 'ruleId'
|
||||
idField: 'ruleId',
|
||||
reservedNamespace: RESERVED_BUILTIN_ID
|
||||
},
|
||||
{
|
||||
dir: path.join(rulesRoot, 'analyze'),
|
||||
output: path.join(rulesRoot, 'dist', 'analyze-rules.json'),
|
||||
schema: '../schema/analyze-rule.schema.json',
|
||||
idField: 'ruleId'
|
||||
idField: 'ruleId',
|
||||
reservedNamespace: RESERVED_BUILTIN_ID
|
||||
},
|
||||
{
|
||||
dir: path.join(rulesRoot, 'ime-log-patterns'),
|
||||
@@ -37,12 +45,27 @@ const configs = [
|
||||
for (const config of configs) {
|
||||
const files = fs.readdirSync(config.dir).filter(f => f.endsWith('.json')).sort();
|
||||
const rules = [];
|
||||
const seenIds = new Map(); // lowercased id -> filename (case variants collide for humans and RuleIdPolicy)
|
||||
|
||||
for (const file of files) {
|
||||
const content = fs.readFileSync(path.join(config.dir, file), 'utf8');
|
||||
const rule = JSON.parse(content);
|
||||
// Remove $schema from individual entries (it's on the wrapper)
|
||||
delete rule['$schema'];
|
||||
|
||||
const id = rule[config.idField] || '';
|
||||
const idKey = id.toLowerCase();
|
||||
if (seenIds.has(idKey)) {
|
||||
console.error(`ERROR: duplicate ${config.idField} "${id}" in ${file} (already defined in ${seenIds.get(idKey)})`);
|
||||
process.exit(1);
|
||||
}
|
||||
seenIds.set(idKey, file);
|
||||
|
||||
if (config.reservedNamespace && !config.reservedNamespace.test(id)) {
|
||||
console.error(`ERROR: ${file}: ${config.idField} "${id}" is outside the reserved built-in namespace ${config.reservedNamespace} — built-ins must never use the tenant CUSTOM namespace`);
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
rules.push(rule);
|
||||
}
|
||||
|
||||
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
|
||||
{
|
||||
/// <summary>
|
||||
/// Pins the per-FILE isolation of the offset measurement across a full read cycle.
|
||||
///
|
||||
/// <para>
|
||||
/// Background (2026-08-20, session e9753578): the field showed "+02:00 measured for
|
||||
/// IntuneManagementExtension.log" although that file held no +2-era line — the anchor had
|
||||
/// crossed file boundaries. The committed code was proven innocent (this very scenario is
|
||||
/// green here); the leak lived in a binary built from an uncommitted tree. This test keeps
|
||||
/// the property pinned so a real regression of the file scoping can never ship silently
|
||||
/// again — it replays the exact field pass structure: a NEW file appearing mid-run with a
|
||||
/// different writer era than the file already being tailed.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class CmTraceCalibratorFileIsolationTests
|
||||
{
|
||||
private static readonly DateTime T0 = new DateTime(2026, 8, 20, 12, 13, 0, DateTimeKind.Utc);
|
||||
|
||||
private static List<ImeLogPattern> Patterns() => new List<ImeLogPattern>
|
||||
{
|
||||
new ImeLogPattern
|
||||
{
|
||||
PatternId = "T-MARK", Category = "always", Enabled = true,
|
||||
Pattern = @"marker (?<n>\d+)",
|
||||
Action = "noop",
|
||||
Parameters = new Dictionary<string, string>(),
|
||||
},
|
||||
};
|
||||
|
||||
private static string Line(string message, DateTime utcInstant, TimeSpan writerOffset)
|
||||
{
|
||||
var local = utcInstant + writerOffset;
|
||||
return $"<![LOG[{message}]LOG]!><time=\"{local:HH:mm:ss.fffffff}\" date=\"{local:M-d-yyyy}\" " +
|
||||
"component=\"X\" context=\"\" type=\"1\" thread=\"1\" file=\"\">";
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task NewFileWithOtherEra_MustNotRecalibrateTheOldFile()
|
||||
{
|
||||
var pdt = TimeSpan.FromHours(-7); // IME service era
|
||||
var cest = TimeSpan.FromHours(2); // fresh child-process era
|
||||
|
||||
using var tmp = new TempDirectory();
|
||||
var now = T0;
|
||||
var tracker = new ImeLogTracker(
|
||||
logFolder: tmp.Path,
|
||||
patterns: Patterns(),
|
||||
logger: new AgentLogger(tmp.Path, AgentLogLevel.Info));
|
||||
try
|
||||
{
|
||||
tracker.UtcNowProvider = () => now;
|
||||
var imeLog = Path.Combine(tmp.Path, "IntuneManagementExtension.log");
|
||||
var execLog = Path.Combine(tmp.Path, "AgentExecutor.log");
|
||||
|
||||
// Pass 1: IME.log first sight (no calibration).
|
||||
File.AppendAllText(imeLog, Line("marker 1", now, pdt) + Environment.NewLine);
|
||||
await tracker.CheckLogFilesAsync(CancellationToken.None);
|
||||
|
||||
// Pass 2: IME.log grew -> measures -7.
|
||||
now = T0.AddSeconds(10);
|
||||
File.AppendAllText(imeLog, Line("marker 2", now, pdt) + Environment.NewLine);
|
||||
await tracker.CheckLogFilesAsync(CancellationToken.None);
|
||||
|
||||
TimeSpan imeMeasured;
|
||||
Assert.True(tracker.OffsetCalibrator.TryGetOffset("IntuneManagementExtension.log", out imeMeasured));
|
||||
Assert.Equal(pdt, imeMeasured);
|
||||
|
||||
// Pass 3: AgentExecutor.log appears for the FIRST time with fresh +2-era lines
|
||||
// while IME.log grows too — the exact field pass structure. The +2 anchor must
|
||||
// never land under IME.log's key.
|
||||
now = T0.AddSeconds(20);
|
||||
File.AppendAllText(execLog, Line("marker 3", now, cest) + Environment.NewLine);
|
||||
File.AppendAllText(imeLog, Line("marker 4", now, pdt) + Environment.NewLine);
|
||||
await tracker.CheckLogFilesAsync(CancellationToken.None);
|
||||
|
||||
Assert.True(tracker.OffsetCalibrator.TryGetOffset("IntuneManagementExtension.log", out imeMeasured));
|
||||
Assert.Equal(pdt, imeMeasured);
|
||||
|
||||
// Pass 4: both grow — each file measures its own writer's era.
|
||||
now = T0.AddSeconds(30);
|
||||
File.AppendAllText(execLog, Line("marker 5", now, cest) + Environment.NewLine);
|
||||
File.AppendAllText(imeLog, Line("marker 6", now, pdt) + Environment.NewLine);
|
||||
await tracker.CheckLogFilesAsync(CancellationToken.None);
|
||||
|
||||
TimeSpan execMeasured;
|
||||
Assert.True(tracker.OffsetCalibrator.TryGetOffset("AgentExecutor.log", out execMeasured));
|
||||
Assert.Equal(cest, execMeasured);
|
||||
Assert.True(tracker.OffsetCalibrator.TryGetOffset("IntuneManagementExtension.log", out imeMeasured));
|
||||
Assert.Equal(pdt, imeMeasured);
|
||||
}
|
||||
finally
|
||||
{
|
||||
tracker.Dispose();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
+21
-12
@@ -67,29 +67,34 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
|
||||
Assert.Equal("EnforcementState: Installing app X", entry.Message);
|
||||
Assert.Equal("AppEnforce", entry.Component);
|
||||
Assert.Equal(1, entry.Type);
|
||||
Assert.NotEqual(default, entry.Timestamp);
|
||||
Assert.True(entry.HasTimestamp);
|
||||
Assert.NotEqual(default, entry.LocalTimestamp);
|
||||
// Bias-less line: the parser refuses to guess the writer's zone, so no UTC value.
|
||||
Assert.Null(entry.BiasMinutes);
|
||||
Assert.Null(entry.TimestampUtc);
|
||||
Assert.Equal(DateTimeKind.Unspecified, entry.LocalTimestamp.Kind);
|
||||
}
|
||||
|
||||
// ── Structurally valid but unparseable timestamp → UtcNow fallback ────
|
||||
// ── Structurally valid but unparseable timestamp → flagged, not invented ──
|
||||
|
||||
[Fact]
|
||||
public void TryParseLine_falls_back_to_utcnow_for_unparseable_timestamp()
|
||||
public void TryParseLine_flags_unparseable_timestamp_instead_of_inventing_one()
|
||||
{
|
||||
// time/date satisfy the regex character classes ([\d:.]+ / [\d-]+) so the line MATCHES,
|
||||
// but "13-45-2026" / "25:99:99" fail every DateTime format → the parser falls back to
|
||||
// DateTime.UtcNow (CmTraceLogParser.cs:57) instead of throwing.
|
||||
var before = DateTime.UtcNow;
|
||||
// but "13-45-2026" / "25:99:99" fail every DateTime format. The parser used to stamp
|
||||
// DateTime.UtcNow here, which made a fabricated value indistinguishable from a real
|
||||
// one downstream. It now reports HasTimestamp=false and leaves the choice of fallback
|
||||
// to the caller, which is the only party that knows what a safe default is in context.
|
||||
var line = CmTraceLine("some message", "25:99:99.0000000", "13-45-2026");
|
||||
|
||||
var ok = CmTraceLogParser.TryParseLine(line, out var entry);
|
||||
var after = DateTime.UtcNow;
|
||||
|
||||
Assert.True(ok);
|
||||
Assert.NotNull(entry);
|
||||
Assert.Equal("some message", entry.Message);
|
||||
// Don't assert an exact instant — only that the fallback stamped a fresh "now"
|
||||
// (a successfully-parsed date would land in 2026, far outside this window).
|
||||
Assert.InRange(entry.Timestamp, before.AddSeconds(-5), after.AddSeconds(5));
|
||||
Assert.False(entry.HasTimestamp);
|
||||
Assert.Equal(default, entry.LocalTimestamp);
|
||||
Assert.Null(entry.TimestampUtc);
|
||||
}
|
||||
|
||||
// ── UTC-bias suffix on the time field → honored, not dropped ──────────
|
||||
@@ -108,10 +113,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
|
||||
|
||||
Assert.True(ok);
|
||||
Assert.NotNull(entry);
|
||||
// A writer-declared bias IS the fact we otherwise have to measure, so it is honored
|
||||
// and resolved to UTC right here — unlike the bias-less form.
|
||||
Assert.Equal(expectedOffsetMinutes, entry.BiasMinutes);
|
||||
Assert.NotNull(entry.TimestampUtc);
|
||||
Assert.Equal(
|
||||
new DateTime(2026, 7, 29, 11, 46, 19, DateTimeKind.Utc).AddTicks(226610).AddMinutes(expectedOffsetMinutes),
|
||||
entry.Timestamp);
|
||||
Assert.Equal(DateTimeKind.Utc, entry.Timestamp.Kind);
|
||||
entry.TimestampUtc!.Value);
|
||||
Assert.Equal(DateTimeKind.Utc, entry.TimestampUtc!.Value.Kind);
|
||||
}
|
||||
|
||||
// ── Oversized line → parses, no throw, no truncation of the message ───
|
||||
|
||||
+190
@@ -0,0 +1,190 @@
|
||||
using System;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
|
||||
{
|
||||
/// <summary>
|
||||
/// Pins the measurement contract of <see cref="CmTraceOffsetCalibrator"/>.
|
||||
///
|
||||
/// <para>
|
||||
/// The cases are taken from real sessions: +1 h (GMT to W. Europe, session bfff9b1c) and
|
||||
/// -17 h (a writer left on the Pacific OOBE default while the agent had moved to
|
||||
/// E. Australia). The backlog case is the one the residual guard cannot catch on its own and
|
||||
/// is the reason the freshness rule exists.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class CmTraceOffsetCalibratorTests
|
||||
{
|
||||
private const string Source = "IntuneManagementExtension.log";
|
||||
private static readonly DateTime UtcNow = new DateTime(2026, 08, 20, 07, 31, 46, DateTimeKind.Utc);
|
||||
|
||||
/// <summary>A line written "now" by a writer that believes it is at <paramref name="offset"/>.</summary>
|
||||
private static DateTime LineWrittenNow(TimeSpan offset, DateTime utcNow)
|
||||
=> DateTime.SpecifyKind(utcNow + offset, DateTimeKind.Unspecified);
|
||||
|
||||
[Theory]
|
||||
[InlineData(2)] // CEST — the bfff9b1c writer
|
||||
[InlineData(1)] // BST — what the agent wrongly assumed there
|
||||
[InlineData(-7)] // PDT — the OOBE default seen in the -17 h sessions
|
||||
[InlineData(10)] // E. Australia
|
||||
[InlineData(14)] // Line Islands, the upper bound
|
||||
[InlineData(-12)] // Dateline, the lower bound
|
||||
public void MeasuresWholeHourOffsets(int hours)
|
||||
{
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
var offset = TimeSpan.FromHours(hours);
|
||||
|
||||
Assert.True(sut.TryCalibrate(Source, LineWrittenNow(offset, UtcNow), UtcNow));
|
||||
|
||||
TimeSpan measured;
|
||||
Assert.True(sut.TryGetOffset(Source, out measured));
|
||||
Assert.Equal(offset, measured);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(330)] // India, UTC+5:30
|
||||
[InlineData(345)] // Nepal, UTC+5:45
|
||||
[InlineData(570)] // Adelaide, UTC+9:30 — the +30 min sessions in the field data
|
||||
public void MeasuresQuarterHourOffsets(int minutes)
|
||||
{
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
var offset = TimeSpan.FromMinutes(minutes);
|
||||
|
||||
Assert.True(sut.TryCalibrate(Source, LineWrittenNow(offset, UtcNow), UtcNow));
|
||||
|
||||
TimeSpan measured;
|
||||
Assert.True(sut.TryGetOffset(Source, out measured));
|
||||
Assert.Equal(offset, measured);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AbsorbsPollAndFlushLatency()
|
||||
{
|
||||
// The line was written a moment before we parsed it. That lag must vanish into the
|
||||
// grid rather than shift the measured offset.
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
var lineLocal = LineWrittenNow(TimeSpan.FromHours(2), UtcNow.AddMilliseconds(-450));
|
||||
|
||||
Assert.True(sut.TryCalibrate(Source, lineLocal, UtcNow));
|
||||
|
||||
TimeSpan measured;
|
||||
Assert.True(sut.TryGetOffset(Source, out measured));
|
||||
Assert.Equal(TimeSpan.FromHours(2), measured);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RejectsBacklogLine_EvenWhenItRoundsCleanly()
|
||||
{
|
||||
// THE case the residual guard cannot catch alone: a line exactly 30 minutes old is a
|
||||
// clean multiple of the grid, so it rounds with zero residual — and would calibrate
|
||||
// the offset half an hour wrong. Only the caller's freshness rule prevents this from
|
||||
// ever being offered; this test pins that a 30-minute-old line does not silently
|
||||
// produce the true offset.
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
var thirtyMinutesOld = LineWrittenNow(TimeSpan.FromHours(2), UtcNow.AddMinutes(-30));
|
||||
|
||||
Assert.True(sut.TryCalibrate(Source, thirtyMinutesOld, UtcNow));
|
||||
|
||||
TimeSpan measured;
|
||||
Assert.True(sut.TryGetOffset(Source, out measured));
|
||||
Assert.Equal(TimeSpan.FromMinutes(90), measured);
|
||||
Assert.NotEqual(TimeSpan.FromHours(2), measured);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RejectsOffGridCandidate()
|
||||
{
|
||||
// 7 minutes off the grid: not fresh, and not a timezone either.
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
var offGrid = LineWrittenNow(TimeSpan.FromHours(2), UtcNow.AddMinutes(-7));
|
||||
|
||||
Assert.False(sut.TryCalibrate(Source, offGrid, UtcNow));
|
||||
|
||||
TimeSpan ignored;
|
||||
Assert.False(sut.TryGetOffset(Source, out ignored));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RejectsOffsetOutsideAnyRealTimezone()
|
||||
{
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
var absurd = LineWrittenNow(TimeSpan.FromHours(20), UtcNow);
|
||||
|
||||
Assert.False(sut.TryCalibrate(Source, absurd, UtcNow));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SelfHeals_WhenTheWritingProcessPicksUpANewZone()
|
||||
{
|
||||
// The writer restarts mid-session and now believes CEST instead of the stale BST.
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
Assert.True(sut.TryCalibrate(Source, LineWrittenNow(TimeSpan.FromHours(1), UtcNow), UtcNow));
|
||||
|
||||
var later = UtcNow.AddMinutes(5);
|
||||
Assert.True(sut.TryCalibrate(Source, LineWrittenNow(TimeSpan.FromHours(2), later), later));
|
||||
|
||||
TimeSpan measured;
|
||||
Assert.True(sut.TryGetOffset(Source, out measured));
|
||||
Assert.Equal(TimeSpan.FromHours(2), measured);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void IgnoresAnchorOlderThanTheCurrentOne()
|
||||
{
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
Assert.True(sut.TryCalibrate(Source, LineWrittenNow(TimeSpan.FromHours(2), UtcNow), UtcNow));
|
||||
|
||||
// An out-of-order line from before the current anchor must not move the offset back.
|
||||
var earlier = UtcNow.AddMinutes(-10);
|
||||
Assert.False(sut.TryCalibrate(Source, LineWrittenNow(TimeSpan.FromHours(1), earlier), earlier));
|
||||
|
||||
TimeSpan measured;
|
||||
Assert.True(sut.TryGetOffset(Source, out measured));
|
||||
Assert.Equal(TimeSpan.FromHours(2), measured);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void KeepsSourcesIndependent()
|
||||
{
|
||||
// IntuneManagementExtension.log and AppWorkload.log are written by different
|
||||
// processes and may hold different beliefs.
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
Assert.True(sut.TryCalibrate("IntuneManagementExtension.log", LineWrittenNow(TimeSpan.FromHours(2), UtcNow), UtcNow));
|
||||
Assert.True(sut.TryCalibrate("AppWorkload.log", LineWrittenNow(TimeSpan.FromHours(-7), UtcNow), UtcNow));
|
||||
|
||||
TimeSpan ime, appWorkload;
|
||||
Assert.True(sut.TryGetOffset("IntuneManagementExtension.log", out ime));
|
||||
Assert.True(sut.TryGetOffset("AppWorkload.log", out appWorkload));
|
||||
Assert.Equal(TimeSpan.FromHours(2), ime);
|
||||
Assert.Equal(TimeSpan.FromHours(-7), appWorkload);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ResolvesTheRealFieldLine_FromSessionBfff9b1c()
|
||||
{
|
||||
// Raw line: <![LOG[Agent version is: 1.104.102.0]LOG]!><time="09:31:46.5192284" ...>
|
||||
// Server-side ReceivedAt puts the true UTC at 07:31:46.4. The old code produced
|
||||
// 08:31:46.5192284 — local minus the agent's stale +1 h belief.
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
var trueUtc = new DateTime(2026, 08, 20, 07, 31, 46, 400, DateTimeKind.Utc);
|
||||
var lineLocal = new DateTime(2026, 08, 20, 09, 31, 46, 519, DateTimeKind.Unspecified);
|
||||
|
||||
Assert.True(sut.TryCalibrate(Source, lineLocal, trueUtc));
|
||||
|
||||
DateTime resolved;
|
||||
Assert.True(sut.TryResolveUtc(Source, lineLocal, out resolved));
|
||||
Assert.Equal(DateTimeKind.Utc, resolved.Kind);
|
||||
Assert.Equal(new DateTime(2026, 08, 20, 07, 31, 46, 519, DateTimeKind.Utc), resolved);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReportsUncalibratedSource_SoTheCallerCanFlagIt()
|
||||
{
|
||||
var sut = new CmTraceOffsetCalibrator();
|
||||
|
||||
DateTime resolved;
|
||||
Assert.False(sut.TryResolveUtc(Source, UtcNow, out resolved));
|
||||
}
|
||||
}
|
||||
}
|
||||
+441
@@ -0,0 +1,441 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
|
||||
using AutopilotMonitor.Shared.Logging;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
|
||||
{
|
||||
/// <summary>
|
||||
/// Per-line self-anchoring — the era-aware successor to the reverted per-file offset
|
||||
/// application (04b1a7c6).
|
||||
///
|
||||
/// <para>
|
||||
/// The property under test: a line read in a provably FRESH pass resolves to the instant it
|
||||
/// was written, whatever timezone its writer believed — and neighbouring lines from writers
|
||||
/// with DIFFERENT beliefs each resolve to their own era, because no cross-line state is
|
||||
/// involved. That interleaving (fixture: AgentExecutor.log with PDT and CEST children in one
|
||||
/// file, flipping both ways) is exactly what broke every cross-line anchor design.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// The freshness boundary is equally load-bearing: backlog (first sight, restart catch-up,
|
||||
/// stalled polls) must NEVER anchor, because an old line's age can round onto the 15-minute
|
||||
/// offset grid. Those paths keep the uniform reader-zone fallback.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class ImeLogTrackerLineAnchoringTests
|
||||
{
|
||||
private static readonly DateTime T0 = new DateTime(2026, 8, 20, 12, 13, 0, DateTimeKind.Utc);
|
||||
|
||||
private static List<ImeLogPattern> Patterns() => new List<ImeLogPattern>
|
||||
{
|
||||
new ImeLogPattern
|
||||
{
|
||||
PatternId = "T-ALL", Category = "always", Enabled = true,
|
||||
Pattern = @"marker (?<n>\d+)",
|
||||
Action = "noop",
|
||||
Parameters = new Dictionary<string, string>(),
|
||||
},
|
||||
};
|
||||
|
||||
/// <summary>A CMTrace line as a writer holding <paramref name="writerOffset"/> renders instant <paramref name="utcInstant"/>.</summary>
|
||||
private static string Line(string message, DateTime utcInstant, TimeSpan writerOffset)
|
||||
{
|
||||
var local = utcInstant + writerOffset;
|
||||
return $"<![LOG[{message}]LOG]!><time=\"{local:HH:mm:ss.fffffff}\" date=\"{local:M-d-yyyy}\" " +
|
||||
"component=\"X\" context=\"\" type=\"1\" thread=\"1\" file=\"\">";
|
||||
}
|
||||
|
||||
private sealed class Harness : IDisposable
|
||||
{
|
||||
private readonly TempDirectory _tmp = new TempDirectory();
|
||||
public ImeLogTracker Tracker { get; }
|
||||
public DateTime Now { get; set; } = T0;
|
||||
|
||||
public Harness(string? stateDirectory = null)
|
||||
{
|
||||
Tracker = new ImeLogTracker(
|
||||
logFolder: _tmp.Path,
|
||||
patterns: Patterns(),
|
||||
logger: new AgentLogger(_tmp.Path, AgentLogLevel.Info),
|
||||
stateDirectory: stateDirectory);
|
||||
Tracker.UtcNowProvider = () => Now;
|
||||
}
|
||||
|
||||
public string PathOf(string fileName) => Path.Combine(_tmp.Path, fileName);
|
||||
|
||||
public void Append(string fileName, string line)
|
||||
=> File.AppendAllText(PathOf(fileName), line + Environment.NewLine);
|
||||
|
||||
public Task Pass() => Tracker.CheckLogFilesAsync(CancellationToken.None);
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
Tracker.Dispose();
|
||||
_tmp.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
[Theory]
|
||||
// The writer's belief. A fresh line must resolve to the instant it was written,
|
||||
// WHATEVER that belief is — including agreement with this process (the majority case,
|
||||
// which a fix to the agent's own zone cache would have broken).
|
||||
[InlineData(2)] // CEST
|
||||
[InlineData(1)] // BST
|
||||
[InlineData(0)] // UTC
|
||||
[InlineData(-7)] // PDT — the OOBE default behind the -9 h and -17 h field cases
|
||||
[InlineData(10)] // E. Australia
|
||||
public async Task FreshLine_ResolvesToItsTrueInstant_ForAnyWriterBelief(int writerOffsetHours)
|
||||
{
|
||||
var writerOffset = TimeSpan.FromHours(writerOffsetHours);
|
||||
using var h = new Harness();
|
||||
|
||||
// First sight: backlog, must not anchor.
|
||||
h.Append("IntuneManagementExtension.log", Line("marker 1", h.Now, writerOffset));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(10);
|
||||
var writtenAt = h.Now.AddMilliseconds(-80); // written just before the poll sees it
|
||||
h.Append("IntuneManagementExtension.log", Line("marker 2", writtenAt, writerOffset));
|
||||
await h.Pass();
|
||||
|
||||
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
Assert.Equal(writtenAt, h.Tracker.LastMatchedLogTimestamp);
|
||||
Assert.Equal(writerOffsetHours * 60, h.Tracker.LastMatchedSourceOffsetMinutes);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task InterleavedEras_InOneGrowingPass_EachLineResolvesToItsOwnEra()
|
||||
{
|
||||
// The e9753578 file shape: two writer processes with different zone beliefs
|
||||
// interleaved in ONE file. Any per-file offset gets one of them wrong by 9 h;
|
||||
// per-line anchoring must get both right within the same chunk.
|
||||
var pdt = TimeSpan.FromHours(-7);
|
||||
var cest = TimeSpan.FromHours(2);
|
||||
using var h = new Harness();
|
||||
|
||||
h.Append("AgentExecutor.log", Line("marker 1", h.Now, pdt));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(10);
|
||||
var wrotePdt = h.Now.AddMilliseconds(-90);
|
||||
var wroteCest = h.Now.AddMilliseconds(-50);
|
||||
h.Append("AgentExecutor.log", Line("marker 2", wrotePdt, pdt));
|
||||
h.Append("AgentExecutor.log", Line("marker 3", wroteCest, cest));
|
||||
|
||||
var resolved = new List<(DateTime? Ts, CmTraceOffsetOrigin Origin, int? Offset)>();
|
||||
Action<string> capture = _ => resolved.Add((
|
||||
h.Tracker.LastMatchedLogTimestamp,
|
||||
h.Tracker.LastMatchedSourceOffsetOrigin,
|
||||
h.Tracker.LastMatchedSourceOffsetMinutes));
|
||||
h.Tracker.OnPatternMatched += capture;
|
||||
try
|
||||
{
|
||||
await h.Pass();
|
||||
}
|
||||
finally
|
||||
{
|
||||
h.Tracker.OnPatternMatched -= capture;
|
||||
}
|
||||
|
||||
Assert.Equal(2, resolved.Count);
|
||||
Assert.Equal(wrotePdt, resolved[0].Ts);
|
||||
Assert.Equal(-420, resolved[0].Offset);
|
||||
Assert.Equal(wroteCest, resolved[1].Ts);
|
||||
Assert.Equal(120, resolved[1].Offset);
|
||||
Assert.All(resolved, r => Assert.Equal(CmTraceOffsetOrigin.LineAnchored, r.Origin));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FirstSightBacklog_FallsBackToReaderZone()
|
||||
{
|
||||
using var h = new Harness();
|
||||
|
||||
h.Append("IntuneManagementExtension.log", Line("marker 1", h.Now.AddHours(-3), TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
|
||||
Assert.Equal(CmTraceOffsetOrigin.None, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FileFirstSeenEmpty_ItsFirstContentIsFresh()
|
||||
{
|
||||
// AgentExecutor.log in the field: the file APPEARS mid-enrollment. Seeing it empty
|
||||
// (or not existing content-wise) counts as an observation, so its very first lines
|
||||
// are provably written between passes and may anchor. This closes the warm-up gap
|
||||
// for exactly the file whose eras flip the most.
|
||||
using var h = new Harness();
|
||||
|
||||
File.WriteAllText(h.PathOf("AgentExecutor.log"), string.Empty);
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(5);
|
||||
var writtenAt = h.Now.AddMilliseconds(-60);
|
||||
h.Append("AgentExecutor.log", Line("marker 1", writtenAt, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
|
||||
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
Assert.Equal(writtenAt, h.Tracker.LastMatchedLogTimestamp);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task PollGapBeyondFreshWindow_FallsBack()
|
||||
{
|
||||
// A stalled agent (GC, CPU starvation) re-polls after a long gap: the new bytes span
|
||||
// the whole gap and their age is unbounded, so the pass must not anchor — even
|
||||
// though most of its lines may in fact be recent. Conservative by design.
|
||||
using var h = new Harness();
|
||||
|
||||
h.Append("IntuneManagementExtension.log", Line("marker 1", h.Now, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
h.Now = T0.AddSeconds(10);
|
||||
h.Append("IntuneManagementExtension.log", Line("marker 2", h.Now, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
|
||||
// Gap far beyond FreshLineMaxAge.
|
||||
h.Now = h.Now + ImeLogTracker.FreshLineMaxAge + TimeSpan.FromSeconds(10);
|
||||
h.Append("IntuneManagementExtension.log", Line("marker 3", h.Now, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
|
||||
Assert.Equal(CmTraceOffsetOrigin.None, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
|
||||
// And the very next tight poll is fresh again.
|
||||
h.Now = h.Now.AddSeconds(1);
|
||||
h.Append("IntuneManagementExtension.log", Line("marker 4", h.Now, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RestartWithRestoredBookmark_FirstPassIsBacklog_SecondPassAnchors()
|
||||
{
|
||||
// The persisted position bookmark survives a restart; the freshness guarantee must
|
||||
// not — the first pass after a restart reads the downtime backlog, whose lines can
|
||||
// be arbitrarily old.
|
||||
using var stateDir = new TempDirectory();
|
||||
using var h = new Harness(stateDirectory: stateDir.Path);
|
||||
|
||||
h.Append("IntuneManagementExtension.log", Line("marker 1", h.Now, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
h.Tracker.SaveStateForTest();
|
||||
|
||||
// "Restart": a second tracker over the same folder restores the bookmark.
|
||||
using var h2 = new HarnessOver(h, stateDir.Path);
|
||||
h2.Tracker.LoadStateForTest();
|
||||
|
||||
// Backlog written during the downtime — 30 min old, exactly on the offset grid,
|
||||
// the nastiest possible backlog shape. Must NOT anchor.
|
||||
h2.Now = T0.AddMinutes(30);
|
||||
h2.Append(Line("marker 2", T0.AddSeconds(20), TimeSpan.FromHours(2)));
|
||||
await h2.Pass();
|
||||
Assert.Equal(CmTraceOffsetOrigin.None, h2.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
|
||||
// The next tight poll is fresh again.
|
||||
h2.Now = h2.Now.AddSeconds(1);
|
||||
var writtenAt = h2.Now.AddMilliseconds(-70);
|
||||
h2.Append(Line("marker 3", writtenAt, TimeSpan.FromHours(2)));
|
||||
await h2.Pass();
|
||||
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h2.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
Assert.Equal(writtenAt, h2.Tracker.LastMatchedLogTimestamp);
|
||||
}
|
||||
|
||||
/// <summary>Second tracker over the FIRST harness's log folder — the "restarted agent".</summary>
|
||||
private sealed class HarnessOver : IDisposable
|
||||
{
|
||||
private readonly string _logFolder;
|
||||
public ImeLogTracker Tracker { get; }
|
||||
public DateTime Now { get; set; }
|
||||
|
||||
public HarnessOver(Harness original, string stateDirectory)
|
||||
{
|
||||
_logFolder = Path.GetDirectoryName(original.PathOf("x"));
|
||||
Tracker = new ImeLogTracker(
|
||||
logFolder: _logFolder,
|
||||
patterns: Patterns(),
|
||||
logger: new AgentLogger(_logFolder, AgentLogLevel.Info),
|
||||
stateDirectory: stateDirectory);
|
||||
Tracker.UtcNowProvider = () => Now;
|
||||
}
|
||||
|
||||
public void Append(string line)
|
||||
=> File.AppendAllText(Path.Combine(_logFolder, "IntuneManagementExtension.log"), line + Environment.NewLine);
|
||||
|
||||
public Task Pass() => Tracker.CheckLogFilesAsync(CancellationToken.None);
|
||||
|
||||
public void Dispose() => Tracker.Dispose();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FreshlyWrittenReplayLine_OnExactGridAge_ResolvesToRoughlyNow()
|
||||
{
|
||||
// Characterization of the one accepted edge: a line freshly WRITTEN but carrying a
|
||||
// REPLAYED timestamp whose age is exactly a grid multiple (here 30 min) measures
|
||||
// that age as an "offset" and resolves to ~now instead of its replayed past. The
|
||||
// error is bounded by construction (an anchored line always lands at now ± 2 min),
|
||||
// and sub-24h replays are treated as current by the historic-replay guard anyway.
|
||||
using var h = new Harness();
|
||||
|
||||
h.Append("IntuneManagementExtension.log", Line("marker 1", h.Now, TimeSpan.Zero));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(10);
|
||||
var replayedInstant = h.Now.AddMinutes(-30);
|
||||
h.Append("IntuneManagementExtension.log", Line("marker 2", replayedInstant, TimeSpan.Zero));
|
||||
await h.Pass();
|
||||
|
||||
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
Assert.NotNull(h.Tracker.LastMatchedLogTimestamp);
|
||||
var error = (h.Tracker.LastMatchedLogTimestamp!.Value - h.Now).Duration();
|
||||
Assert.True(error <= CmTraceOffsetCalibrator.MaxGridResidual,
|
||||
$"anchored line must resolve to now ± residual, was off by {error}");
|
||||
}
|
||||
|
||||
// =====================================================================
|
||||
// The committed field fixture — session e9753578, ImeLogs/AgentExecutor.log.
|
||||
// One file, two writer eras (PDT hour-05 / CEST hour-14), interleaved, flipping
|
||||
// BOTH ways. Replayed here as the live tail the field agent saw: entry by entry,
|
||||
// with the clock following the true write time.
|
||||
// =====================================================================
|
||||
|
||||
private static string FixturePath()
|
||||
{
|
||||
var dir = AppContext.BaseDirectory;
|
||||
for (int i = 0; i < 12; i++)
|
||||
{
|
||||
if (File.Exists(Path.Combine(dir, "AutopilotMonitor.sln")))
|
||||
return Path.Combine(dir, "tests", "fixtures", "cmtrace-logs",
|
||||
"agentexecutor-two-writer-eras-v1.cmtrace");
|
||||
var parent = Directory.GetParent(dir)?.FullName;
|
||||
if (parent == null || parent == dir) break;
|
||||
dir = parent;
|
||||
}
|
||||
throw new DirectoryNotFoundException(
|
||||
"Could not locate repo root (AutopilotMonitor.sln) walking up from " + AppContext.BaseDirectory);
|
||||
}
|
||||
|
||||
/// <summary>Group the fixture's physical lines into complete CMTrace entries (multiline-safe).</summary>
|
||||
private static List<string> FixtureEntries()
|
||||
{
|
||||
var entries = new List<string>();
|
||||
string? pending = null;
|
||||
foreach (var rawWithBom in File.ReadAllLines(FixturePath()))
|
||||
{
|
||||
// The fixture keeps the original file's BOM verbatim — it sat at the START of
|
||||
// the real AgentExecutor.log, which is mid-file in this excerpt.
|
||||
var raw = rawWithBom.TrimStart('');
|
||||
if (pending == null)
|
||||
{
|
||||
if (raw.Length == 0 || raw[0] == '#') continue;
|
||||
if (!raw.StartsWith("<![LOG[", StringComparison.Ordinal)) continue;
|
||||
if (raw.Contains("]LOG]!>")) { entries.Add(raw); continue; }
|
||||
pending = raw;
|
||||
}
|
||||
else
|
||||
{
|
||||
pending += "\n" + raw;
|
||||
if (raw.Contains("]LOG]!>")) { entries.Add(pending!); pending = null; }
|
||||
}
|
||||
}
|
||||
return entries;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TwoWriterEraFixture_EveryLineOfBothErasResolvesToItsTrueInstant()
|
||||
{
|
||||
// The committed fixture is a curated, verbatim excerpt of the transition window
|
||||
// (commit policy: anonymized, versioned) — 20 entries covering both eras and the flip.
|
||||
var entries = FixtureEntries();
|
||||
Assert.True(entries.Count >= 15, $"fixture unexpectedly small: {entries.Count} entries");
|
||||
|
||||
using var h = new Harness();
|
||||
|
||||
// Match every fixture entry.
|
||||
h.Tracker.CompilePatterns(new List<ImeLogPattern>
|
||||
{
|
||||
new ImeLogPattern
|
||||
{
|
||||
PatternId = "T-ANY", Category = "always", Enabled = true,
|
||||
Pattern = ".+", Action = "noop",
|
||||
Parameters = new Dictionary<string, string>(),
|
||||
},
|
||||
});
|
||||
|
||||
// The file exists (empty) before its first content — first sight is the observation
|
||||
// baseline, everything after is a fresh tail, exactly like the field.
|
||||
File.WriteAllText(h.PathOf("AgentExecutor.log"), string.Empty);
|
||||
h.Now = new DateTime(2026, 8, 20, 12, 13, 30, DateTimeKind.Utc);
|
||||
await h.Pass();
|
||||
|
||||
var anchored = 0;
|
||||
var resolvedList = new List<(DateTime Resolved, DateTime Expected)>();
|
||||
Action<string> capture = _ =>
|
||||
{
|
||||
if (h.Tracker.LastMatchedSourceOffsetOrigin == CmTraceOffsetOrigin.LineAnchored)
|
||||
anchored++;
|
||||
if (h.Tracker.LastMatchedLogTimestamp.HasValue && _expectedForCurrentEntry.HasValue)
|
||||
resolvedList.Add((h.Tracker.LastMatchedLogTimestamp.Value, _expectedForCurrentEntry.Value));
|
||||
};
|
||||
h.Tracker.OnPatternMatched += capture;
|
||||
try
|
||||
{
|
||||
foreach (var entry in entries)
|
||||
{
|
||||
CmTraceLogEntry? parsed;
|
||||
Assert.True(CmTraceLogParser.TryParseLine(entry, out parsed), "fixture entry did not parse");
|
||||
|
||||
// Era by local hour: the fixture holds exactly two eras — PDT (UTC-7) lines
|
||||
// render at local hour 05, CEST (UTC+2) lines at hour 14 (fixture header).
|
||||
var eraOffset = parsed.LocalTimestamp.Hour == 14
|
||||
? TimeSpan.FromHours(2)
|
||||
: TimeSpan.FromHours(-7);
|
||||
var trueUtc = DateTime.SpecifyKind(parsed.LocalTimestamp - eraOffset, DateTimeKind.Utc);
|
||||
|
||||
// The clock follows the write; never move backwards (interleaved eras are
|
||||
// written in true-UTC order, but guard against sub-ms jitter).
|
||||
var readAt = trueUtc.AddMilliseconds(120);
|
||||
if (readAt > h.Now) h.Now = readAt;
|
||||
|
||||
// In production the 100 ms poll keeps running through write silences — every
|
||||
// empty look refreshes the freshness window. The fixture's transition window
|
||||
// spans multi-minute write gaps, so replay that cadence with one empty poll
|
||||
// just before each append; without it the NEXT pass would (correctly!)
|
||||
// classify the gap as a stall and refuse to anchor.
|
||||
h.Now = h.Now.AddMilliseconds(-100);
|
||||
await h.Pass();
|
||||
h.Now = readAt;
|
||||
|
||||
_expectedForCurrentEntry = trueUtc;
|
||||
File.AppendAllText(h.PathOf("AgentExecutor.log"), entry + Environment.NewLine);
|
||||
await h.Pass();
|
||||
_expectedForCurrentEntry = null;
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
h.Tracker.OnPatternMatched -= capture;
|
||||
}
|
||||
|
||||
// Every matched entry resolved to the instant its writer rendered — across BOTH
|
||||
// eras, including the flips. This is the exact property whose absence produced the
|
||||
// -9 h script durations.
|
||||
Assert.Equal(entries.Count, resolvedList.Count);
|
||||
Assert.All(resolvedList, r => Assert.Equal(r.Expected, r.Resolved));
|
||||
Assert.True(anchored == resolvedList.Count,
|
||||
$"expected every entry line-anchored, got {anchored}/{resolvedList.Count}");
|
||||
|
||||
// Both eras genuinely exercised: PDT entries render at local hour 05, CEST at 14.
|
||||
Assert.Contains(resolvedList, r => (r.Expected + TimeSpan.FromHours(-7)).Hour == 5);
|
||||
Assert.Contains(resolvedList, r => (r.Expected + TimeSpan.FromHours(2)).Hour == 14);
|
||||
}
|
||||
|
||||
private DateTime? _expectedForCurrentEntry;
|
||||
}
|
||||
}
|
||||
+275
@@ -0,0 +1,275 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
|
||||
{
|
||||
/// <summary>
|
||||
/// End-to-end offset calibration through a real read cycle.
|
||||
///
|
||||
/// <para>
|
||||
/// A CMTrace line states local time and nothing else, so its UTC value depends on which zone
|
||||
/// the WRITING process believed it was in. The tracker used to substitute its own belief,
|
||||
/// which is right only when the two happen to agree.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// These tests are deliberately independent of the machine they run on: the assertion is
|
||||
/// always "a line written at instant T resolves to T", for every writer belief. That single
|
||||
/// property covers both field cases at once — the writer disagreeing with us (the bug) and
|
||||
/// the writer agreeing with us (the majority, which a naive fix would have broken).
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class ImeLogTrackerOffsetCalibrationTests
|
||||
{
|
||||
private const string LogFileName = "IntuneManagementExtension.log";
|
||||
private static readonly DateTime T0 = new DateTime(2026, 8, 20, 7, 30, 0, DateTimeKind.Utc);
|
||||
|
||||
private static List<ImeLogPattern> Patterns() => new List<ImeLogPattern>
|
||||
{
|
||||
new ImeLogPattern
|
||||
{
|
||||
PatternId = "T-MARK", Category = "always", Enabled = true,
|
||||
Pattern = @"marker (?<n>\d+)",
|
||||
Action = "noop",
|
||||
Parameters = new Dictionary<string, string>(),
|
||||
},
|
||||
};
|
||||
|
||||
/// <summary>A CMTrace line as a writer holding <paramref name="writerOffset"/> renders instant <paramref name="utcInstant"/>.</summary>
|
||||
private static string Line(string message, DateTime utcInstant, TimeSpan writerOffset)
|
||||
{
|
||||
var local = utcInstant + writerOffset;
|
||||
return $"<![LOG[{message}]LOG]!><time=\"{local:HH:mm:ss.fffffff}\" date=\"{local:M-d-yyyy}\" " +
|
||||
"component=\"IntuneManagementExtension\" context=\"\" type=\"1\" thread=\"1\" file=\"\">";
|
||||
}
|
||||
|
||||
private sealed class Harness : IDisposable
|
||||
{
|
||||
private readonly TempDirectory _tmp = new TempDirectory();
|
||||
public ImeLogTracker Tracker { get; }
|
||||
public DateTime Now { get; set; } = T0;
|
||||
|
||||
public Harness()
|
||||
{
|
||||
Tracker = new ImeLogTracker(
|
||||
logFolder: _tmp.Path,
|
||||
patterns: Patterns(),
|
||||
logger: new AgentLogger(_tmp.Path, AgentLogLevel.Info));
|
||||
Tracker.UtcNowProvider = () => Now;
|
||||
}
|
||||
|
||||
public string LogPath => Path.Combine(_tmp.Path, LogFileName);
|
||||
|
||||
public void Append(string line) => File.AppendAllText(LogPath, line + Environment.NewLine);
|
||||
|
||||
public Task Pass() => Tracker.CheckLogFilesAsync(CancellationToken.None);
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
Tracker.Dispose();
|
||||
_tmp.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
[Theory]
|
||||
// The writer's belief. The MEASUREMENT must recover it exactly, whatever it is.
|
||||
[InlineData(2)] // CEST
|
||||
[InlineData(1)] // BST
|
||||
[InlineData(-7)] // PDT — the OOBE default behind the -17 h and -9 h field cases
|
||||
[InlineData(10)] // E. Australia
|
||||
[InlineData(0)] // UTC
|
||||
public async Task MeasuresTheWritersOffsetExactly_ForAnyBelief(int writerOffsetHours)
|
||||
{
|
||||
// NOTE: the per-FILE measurement asserted here is OBSERVATIONAL — its application was
|
||||
// reverted on 2026-08-20 (04b1a7c6) because one offset per file is wrong when a file
|
||||
// holds interleaved writer eras. Resolution now anchors per LINE instead (see
|
||||
// ImeLogTrackerLineAnchoringTests, which restores the "a line written at T resolves
|
||||
// to T" property era-safely). The per-file measurement stays for the Info log and
|
||||
// measuredWriterOffsetMinutes provenance.
|
||||
var writerOffset = TimeSpan.FromHours(writerOffsetHours);
|
||||
using var h = new Harness();
|
||||
|
||||
h.Append(Line("marker 1", h.Now, writerOffset));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(10);
|
||||
h.Append(Line("marker 2", h.Now, writerOffset));
|
||||
await h.Pass();
|
||||
|
||||
TimeSpan measured;
|
||||
Assert.True(h.Tracker.OffsetCalibrator.TryGetOffset(LogFileName, out measured));
|
||||
Assert.Equal(writerOffset, measured);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MeasuresTheWritersOffset_NotItsOwn()
|
||||
{
|
||||
using var h = new Harness();
|
||||
var writerOffset = TimeSpan.FromHours(2);
|
||||
|
||||
h.Append(Line("marker 1", h.Now, writerOffset));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(10);
|
||||
h.Append(Line("marker 2", h.Now, writerOffset));
|
||||
await h.Pass();
|
||||
|
||||
TimeSpan measured;
|
||||
Assert.True(h.Tracker.OffsetCalibrator.TryGetOffset(LogFileName, out measured));
|
||||
Assert.Equal(writerOffset, measured);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task DoesNotCalibrateOnFirstSightOfAFile()
|
||||
{
|
||||
// The first pass reads from position 0, so the newest line in the file may be hours
|
||||
// old. Anchoring on it would measure the line's age instead of the writer's offset.
|
||||
using var h = new Harness();
|
||||
|
||||
h.Append(Line("marker 1", T0.AddHours(-3), TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
|
||||
TimeSpan ignored;
|
||||
Assert.False(h.Tracker.OffsetCalibrator.TryGetOffset(LogFileName, out ignored));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FollowsTheWriterAcrossARestartIntoANewZone()
|
||||
{
|
||||
// IME restarts mid-enrollment and picks up the timezone the OS moved to. The
|
||||
// measurement has to follow, otherwise every later line is off by the difference.
|
||||
using var h = new Harness();
|
||||
|
||||
h.Append(Line("marker 1", h.Now, TimeSpan.FromHours(1)));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(10);
|
||||
h.Append(Line("marker 2", h.Now, TimeSpan.FromHours(1)));
|
||||
await h.Pass();
|
||||
|
||||
TimeSpan before;
|
||||
Assert.True(h.Tracker.OffsetCalibrator.TryGetOffset(LogFileName, out before));
|
||||
Assert.Equal(TimeSpan.FromHours(1), before);
|
||||
|
||||
// Writer restarts, now on +2.
|
||||
h.Now = T0.AddSeconds(20);
|
||||
h.Append(Line("marker 3", h.Now, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(30);
|
||||
var writtenAt = h.Now;
|
||||
h.Append(Line("marker 4", writtenAt, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
|
||||
TimeSpan after;
|
||||
Assert.True(h.Tracker.OffsetCalibrator.TryGetOffset(LogFileName, out after));
|
||||
Assert.Equal(TimeSpan.FromHours(2), after);
|
||||
// The measurement follows the restart. Applying it is what the 2026-08-20 revert
|
||||
// removed — that is exactly the case a file with two writer eras breaks.
|
||||
Assert.Equal(120, h.Tracker.LastMatchedMeasuredWriterOffsetMinutes);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task HonorsAWriterDeclaredBiasWithoutMeasuring()
|
||||
{
|
||||
// A line carrying its own bias states the offset outright; it must be used as-is and
|
||||
// must never become a calibration anchor.
|
||||
using var h = new Harness();
|
||||
|
||||
h.Append(Line("marker 1", h.Now, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(10);
|
||||
var writtenAt = h.Now;
|
||||
var local = writtenAt + TimeSpan.FromHours(-8); // a PST writer
|
||||
h.Append($"<![LOG[marker 2]LOG]!><time=\"{local:HH:mm:ss.fffffff}+480\" date=\"{local:M-d-yyyy}\" " +
|
||||
"component=\"IntuneManagementExtension\" context=\"\" type=\"1\" thread=\"1\" file=\"\">");
|
||||
await h.Pass();
|
||||
|
||||
Assert.True(h.Tracker.LastMatchedLogTimestamp.HasValue);
|
||||
Assert.Equal(writtenAt, h.Tracker.LastMatchedLogTimestamp.Value);
|
||||
|
||||
TimeSpan ignored;
|
||||
Assert.False(h.Tracker.OffsetCalibrator.TryGetOffset(LogFileName, out ignored));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RecordsMeasuredOffsetSeparatelyFromTheAppliedOne()
|
||||
{
|
||||
// The regression that forced the revert shipped an event tagged "calibrated" while
|
||||
// carrying an offset that did not hold for that line. Measured and applied are now
|
||||
// separate fields so a future reader cannot mistake one for the other.
|
||||
using var h = new Harness();
|
||||
var writerOffset = TimeSpan.FromHours(-7); // PDT, the OOBE default
|
||||
|
||||
h.Append(Line("marker 1", h.Now, writerOffset));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(10);
|
||||
h.Append(Line("marker 2", h.Now, writerOffset));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(20);
|
||||
h.Append(Line("marker 3", h.Now, writerOffset));
|
||||
await h.Pass();
|
||||
|
||||
// Measured: the writer's actual belief, per file.
|
||||
Assert.Equal(-420, h.Tracker.LastMatchedMeasuredWriterOffsetMinutes);
|
||||
|
||||
// Applied: the line's OWN anchor (fresh pass), never the per-file "calibrated" value —
|
||||
// the two must stay distinguishable even when they agree numerically.
|
||||
Assert.Equal(CmTraceOffsetOrigin.LineAnchored, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
Assert.NotEqual(CmTraceOffsetOrigin.Calibrated, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RecordsProvenance_AsFallback_BeforeAnyCalibration()
|
||||
{
|
||||
// The warm-up pass: the offset is not measured yet, so the reader zone is assumed and
|
||||
// the event has to say so rather than look source-grounded.
|
||||
using var h = new Harness();
|
||||
|
||||
h.Append(Line("marker 1", h.Now, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
|
||||
Assert.Equal(CmTraceOffsetOrigin.None, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
Assert.True(h.Tracker.LastMatchedSourceLocalTimestamp.HasValue);
|
||||
|
||||
// The APPLIED offset is reported even on the fallback path — an event that states no
|
||||
// offset at all cannot be recomputed later, which defeats the purpose of recording
|
||||
// provenance. Asserted against the running machine's zone so this stays portable.
|
||||
var readerZone = (int)TimeZoneInfo.Local
|
||||
.GetUtcOffset(h.Tracker.LastMatchedSourceLocalTimestamp!.Value).TotalMinutes;
|
||||
Assert.Equal(readerZone, h.Tracker.LastMatchedSourceOffsetMinutes);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RecordsProvenance_ForADeclaredBias()
|
||||
{
|
||||
using var h = new Harness();
|
||||
|
||||
h.Append(Line("marker 1", h.Now, TimeSpan.FromHours(2)));
|
||||
await h.Pass();
|
||||
|
||||
h.Now = T0.AddSeconds(10);
|
||||
var local = h.Now + TimeSpan.FromHours(-8);
|
||||
h.Append($"<![LOG[marker 2]LOG]!><time=\"{local:HH:mm:ss.fffffff}+480\" date=\"{local:M-d-yyyy}\" " +
|
||||
"component=\"IntuneManagementExtension\" context=\"\" type=\"1\" thread=\"1\" file=\"\">");
|
||||
await h.Pass();
|
||||
|
||||
Assert.Equal(CmTraceOffsetOrigin.Bias, h.Tracker.LastMatchedSourceOffsetOrigin);
|
||||
// Reported in the same sense as a measured offset (local = UTC + offset), so the two
|
||||
// origins are directly comparable; the wire bias uses the opposite convention.
|
||||
Assert.Equal(-480, h.Tracker.LastMatchedSourceOffsetMinutes);
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
+258
@@ -0,0 +1,258 @@
|
||||
#nullable enable
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
|
||||
using AutopilotMonitor.Agent.V2.Core.Runtime;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
|
||||
{
|
||||
/// <summary>
|
||||
/// Pure state machine behind the power watcher: AC↔battery edges, the 50/30/15 threshold
|
||||
/// ladder (latch once per lifetime, lowest-crossed-only on multi-level jumps, baseline
|
||||
/// coverage for sessions starting low), null-percent tolerance and the storm cap.
|
||||
/// </summary>
|
||||
public sealed class PowerStateTransitionTrackerTests
|
||||
{
|
||||
private static PowerStateResult Ac(int? percent = 80, bool charging = true) => new PowerStateResult
|
||||
{
|
||||
OnAcPower = true,
|
||||
HasBattery = true,
|
||||
BatteryPercent = percent,
|
||||
IsCharging = charging,
|
||||
BatteryLifeMinutes = null,
|
||||
};
|
||||
|
||||
private static PowerStateResult Battery(int? percent, int? lifeMinutes = 90) => new PowerStateResult
|
||||
{
|
||||
OnAcPower = false,
|
||||
HasBattery = true,
|
||||
BatteryPercent = percent,
|
||||
IsCharging = false,
|
||||
BatteryLifeMinutes = lifeMinutes,
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void Baseline_on_ac_is_silent()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
Assert.Empty(tracker.Baseline(Ac()));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Baseline_without_battery_or_with_probe_error_is_silent()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
Assert.Empty(tracker.Baseline(new PowerStateResult { OnAcPower = true, HasBattery = false }));
|
||||
Assert.Empty(tracker.Baseline(new PowerStateResult { ProbeError = "boom" }));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Baseline_on_battery_above_ladder_is_silent()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
Assert.Empty(tracker.Baseline(Battery(80)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Baseline_at_12_percent_emits_exactly_the_15_threshold()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
|
||||
var emissions = tracker.Baseline(Battery(12));
|
||||
|
||||
var e = Assert.Single(emissions);
|
||||
Assert.Equal(PowerStateTransitionTracker.TransitionThresholdCrossed, e.Transition);
|
||||
Assert.Equal(15, e.ThresholdPercent);
|
||||
Assert.Equal(12, e.BatteryPercent);
|
||||
Assert.Equal(EventSeverity.Error, e.Severity);
|
||||
Assert.True(e.ImmediateUpload);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Baseline_at_25_percent_emits_30_and_latches_50()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
|
||||
var baseline = tracker.Baseline(Battery(25));
|
||||
var e = Assert.Single(baseline);
|
||||
Assert.Equal(30, e.ThresholdPercent);
|
||||
Assert.Equal(EventSeverity.Warning, e.Severity);
|
||||
Assert.True(e.ImmediateUpload);
|
||||
|
||||
// 50 and 30 are latched by the baseline; draining to 40% stays silent, 14% emits 15.
|
||||
Assert.Empty(tracker.Evaluate(Battery(40)));
|
||||
var next = Assert.Single(tracker.Evaluate(Battery(14)));
|
||||
Assert.Equal(15, next.ThresholdPercent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Ac_to_battery_emits_warning_with_immediate_upload()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
tracker.Baseline(Ac(63));
|
||||
|
||||
var e = Assert.Single(tracker.Evaluate(Battery(63)));
|
||||
|
||||
Assert.Equal(PowerStateTransitionTracker.TransitionAcToBattery, e.Transition);
|
||||
Assert.Null(e.ThresholdPercent);
|
||||
Assert.Equal(EventSeverity.Warning, e.Severity);
|
||||
Assert.True(e.ImmediateUpload);
|
||||
Assert.False(e.OnAcPower);
|
||||
Assert.Equal(63, e.BatteryPercent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Battery_to_ac_emits_info_without_immediate_upload()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
tracker.Baseline(Battery(70));
|
||||
|
||||
var e = Assert.Single(tracker.Evaluate(Ac(70)));
|
||||
|
||||
Assert.Equal(PowerStateTransitionTracker.TransitionBatteryToAc, e.Transition);
|
||||
Assert.Equal(EventSeverity.Info, e.Severity);
|
||||
Assert.False(e.ImmediateUpload);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void No_diff_stays_silent()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
tracker.Baseline(Ac());
|
||||
|
||||
Assert.Empty(tracker.Evaluate(Ac()));
|
||||
Assert.Empty(tracker.Evaluate(Ac(79)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Unplug_below_threshold_emits_transition_and_threshold_together()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
tracker.Baseline(Ac(10));
|
||||
|
||||
var emissions = tracker.Evaluate(Battery(10));
|
||||
|
||||
Assert.Equal(2, emissions.Count);
|
||||
Assert.Equal(PowerStateTransitionTracker.TransitionAcToBattery, emissions[0].Transition);
|
||||
Assert.Equal(PowerStateTransitionTracker.TransitionThresholdCrossed, emissions[1].Transition);
|
||||
Assert.Equal(15, emissions[1].ThresholdPercent);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Multi_level_drop_emits_lowest_crossed_level_only()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
tracker.Baseline(Battery(60));
|
||||
|
||||
var e = Assert.Single(tracker.Evaluate(Battery(10)));
|
||||
|
||||
Assert.Equal(15, e.ThresholdPercent);
|
||||
// All levels are latched — recovering and re-draining emits nothing more.
|
||||
tracker.Evaluate(Ac(90));
|
||||
Assert.Single(tracker.Evaluate(Battery(45)), x => x.Transition == PowerStateTransitionTracker.TransitionAcToBattery);
|
||||
Assert.Empty(tracker.Evaluate(Battery(13)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Each_level_latches_once_per_lifetime()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
tracker.Baseline(Battery(70));
|
||||
|
||||
Assert.Single(tracker.Evaluate(Battery(48))); // 50
|
||||
Assert.Empty(tracker.Evaluate(Battery(47))); // still inside 50, no re-emit
|
||||
Assert.Single(tracker.Evaluate(Battery(29))); // 30
|
||||
Assert.Single(tracker.Evaluate(Battery(15))); // 15 (boundary: <= level)
|
||||
Assert.Empty(tracker.Evaluate(Battery(5))); // ladder exhausted
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Threshold_severity_ladder_is_info_warning_error()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
tracker.Baseline(Battery(70));
|
||||
|
||||
var fifty = Assert.Single(tracker.Evaluate(Battery(50)));
|
||||
Assert.Equal(EventSeverity.Info, fifty.Severity);
|
||||
Assert.False(fifty.ImmediateUpload);
|
||||
|
||||
var thirty = Assert.Single(tracker.Evaluate(Battery(30)));
|
||||
Assert.Equal(EventSeverity.Warning, thirty.Severity);
|
||||
Assert.True(thirty.ImmediateUpload);
|
||||
|
||||
var fifteen = Assert.Single(tracker.Evaluate(Battery(15)));
|
||||
Assert.Equal(EventSeverity.Error, fifteen.Severity);
|
||||
Assert.True(fifteen.ImmediateUpload);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Null_battery_percent_skips_thresholds_but_ac_diff_still_works()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
tracker.Baseline(Ac(null));
|
||||
|
||||
var e = Assert.Single(tracker.Evaluate(Battery(null, lifeMinutes: null)));
|
||||
|
||||
Assert.Equal(PowerStateTransitionTracker.TransitionAcToBattery, e.Transition);
|
||||
Assert.Null(e.BatteryPercent);
|
||||
Assert.Contains("battery level unknown", e.Message);
|
||||
Assert.Empty(tracker.Evaluate(Battery(null, lifeMinutes: null)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Probe_error_mid_run_preserves_state()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
tracker.Baseline(Ac());
|
||||
|
||||
Assert.Empty(tracker.Evaluate(new PowerStateResult { ProbeError = "transient" }));
|
||||
// State was not corrupted by the error — the real transition still diffs cleanly.
|
||||
Assert.Single(tracker.Evaluate(Battery(60)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Evaluate_without_baseline_latches_like_a_baseline()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
|
||||
var e = Assert.Single(tracker.Evaluate(Battery(12)));
|
||||
|
||||
// No phantom ac_to_battery from the implicit default state — only the threshold.
|
||||
Assert.Equal(PowerStateTransitionTracker.TransitionThresholdCrossed, e.Transition);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Emission_cap_suppresses_and_flips_flag_once()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker(maxEmissions: 4);
|
||||
tracker.Baseline(Ac());
|
||||
|
||||
// Flapping power source: each edge is one emission.
|
||||
for (var i = 0; i < 2; i++)
|
||||
{
|
||||
Assert.Single(tracker.Evaluate(Battery(80)));
|
||||
Assert.Single(tracker.Evaluate(Ac(80)));
|
||||
}
|
||||
Assert.False(tracker.EmissionCapReached);
|
||||
|
||||
Assert.Empty(tracker.Evaluate(Battery(80)));
|
||||
Assert.True(tracker.EmissionCapReached);
|
||||
Assert.Empty(tracker.Evaluate(Ac(80)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Message_carries_percent_and_transition_context()
|
||||
{
|
||||
var tracker = new PowerStateTransitionTracker();
|
||||
tracker.Baseline(Ac(62));
|
||||
|
||||
var toBattery = Assert.Single(tracker.Evaluate(Battery(62)));
|
||||
Assert.Contains("AC to battery", toBattery.Message);
|
||||
Assert.Contains("battery 62%", toBattery.Message);
|
||||
|
||||
var backToAc = Assert.Single(tracker.Evaluate(Ac(63)));
|
||||
Assert.Contains("back to AC", backToAc.Message);
|
||||
Assert.Contains("charging", backToAc.Message);
|
||||
}
|
||||
}
|
||||
}
|
||||
+162
@@ -0,0 +1,162 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Runtime;
|
||||
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
|
||||
using AutopilotMonitor.DecisionCore.Engine;
|
||||
using AutopilotMonitor.DecisionCore.Signals;
|
||||
using AutopilotMonitor.Shared;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
|
||||
{
|
||||
/// <summary>
|
||||
/// PowerStateWatcherHost — no-battery/probe-error devices never arm, a low-battery baseline
|
||||
/// emits its threshold event onto the single rail, and debounced ticks translate tracker
|
||||
/// emissions into <c>power_state_change</c> events. WMI arming itself stays untested
|
||||
/// (ConsoleBypassWatcher precedent); <c>StartCore(armWmi: false)</c> exercises everything else.
|
||||
/// </summary>
|
||||
public sealed class PowerStateWatcherHostTests : IDisposable
|
||||
{
|
||||
private static readonly DateTime At = new DateTime(2026, 8, 21, 9, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private readonly TempDirectory _tmp = new TempDirectory();
|
||||
private readonly FakeSignalIngressSink _sink = new FakeSignalIngressSink();
|
||||
private readonly AgentLogger _logger;
|
||||
private PowerStateResult _probeResult = new PowerStateResult { OnAcPower = true, HasBattery = true, BatteryPercent = 80 };
|
||||
|
||||
public PowerStateWatcherHostTests()
|
||||
{
|
||||
_logger = new AgentLogger(_tmp.Path, AgentLogLevel.Info);
|
||||
}
|
||||
|
||||
public void Dispose() => _tmp.Dispose();
|
||||
|
||||
private PowerStateWatcherHost Build() => new PowerStateWatcherHost(
|
||||
sessionId: "sess-power",
|
||||
tenantId: "tenant-power",
|
||||
ingress: _sink,
|
||||
clock: new VirtualClock(At),
|
||||
logger: _logger,
|
||||
probe: () => _probeResult);
|
||||
|
||||
private IReadOnlyList<FakeSignalIngressSink.PostedSignal> PowerEvents() =>
|
||||
_sink.Posted.Where(p =>
|
||||
p.Kind == DecisionSignalKind.InformationalEvent
|
||||
&& p.Payload != null
|
||||
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
|
||||
&& et == Constants.EventTypes.PowerStateChange).ToList();
|
||||
|
||||
[Fact]
|
||||
public void No_battery_emits_nothing()
|
||||
{
|
||||
_probeResult = new PowerStateResult { OnAcPower = true, HasBattery = false };
|
||||
|
||||
Build().StartCore(armWmi: false);
|
||||
|
||||
Assert.Empty(_sink.Posted);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Probe_error_emits_nothing()
|
||||
{
|
||||
_probeResult = new PowerStateResult { ProbeError = "GetSystemPowerStatus returned false" };
|
||||
|
||||
Build().StartCore(armWmi: false);
|
||||
|
||||
Assert.Empty(_sink.Posted);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Baseline_below_threshold_emits_threshold_event_with_immediate_upload()
|
||||
{
|
||||
_probeResult = new PowerStateResult { OnAcPower = false, HasBattery = true, BatteryPercent = 12, BatteryLifeMinutes = 34 };
|
||||
|
||||
Build().StartCore(armWmi: false);
|
||||
|
||||
var s = Assert.Single(PowerEvents());
|
||||
Assert.Equal("Error", s.Payload![SignalPayloadKeys.Severity]);
|
||||
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
|
||||
Assert.Contains("15%", s.Payload![SignalPayloadKeys.Message]);
|
||||
|
||||
var data = (IReadOnlyDictionary<string, object>)s.TypedPayload!;
|
||||
Assert.Equal("threshold_crossed", data["transition"]);
|
||||
Assert.Equal(15, data["thresholdPercent"]);
|
||||
Assert.Equal(12, data["batteryPercent"]);
|
||||
Assert.Equal(false, data["onAcPower"]);
|
||||
Assert.Equal(34, data["batteryLifeMinutes"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Tick_after_unplug_emits_ac_to_battery_warning()
|
||||
{
|
||||
var host = Build();
|
||||
host.StartCore(armWmi: false);
|
||||
Assert.Empty(PowerEvents());
|
||||
|
||||
_probeResult = new PowerStateResult { OnAcPower = false, HasBattery = true, BatteryPercent = 79 };
|
||||
host.TickForTest();
|
||||
|
||||
var s = Assert.Single(PowerEvents());
|
||||
Assert.Equal("Warning", s.Payload![SignalPayloadKeys.Severity]);
|
||||
Assert.Equal("true", s.Payload![SignalPayloadKeys.ImmediateUpload]);
|
||||
|
||||
var data = (IReadOnlyDictionary<string, object>)s.TypedPayload!;
|
||||
Assert.Equal("ac_to_battery", data["transition"]);
|
||||
Assert.Equal(79, data["batteryPercent"]);
|
||||
Assert.False(data.ContainsKey("thresholdPercent"));
|
||||
|
||||
// Replug: Info recovery event without immediate upload.
|
||||
_probeResult = new PowerStateResult { OnAcPower = true, HasBattery = true, BatteryPercent = 79, IsCharging = true };
|
||||
host.TickForTest();
|
||||
|
||||
Assert.Equal(2, PowerEvents().Count);
|
||||
var back = PowerEvents()[1];
|
||||
Assert.Equal("Info", back.Payload![SignalPayloadKeys.Severity]);
|
||||
Assert.Equal("false", back.Payload![SignalPayloadKeys.ImmediateUpload]);
|
||||
Assert.Equal("battery_to_ac", ((IReadOnlyDictionary<string, object>)back.TypedPayload!)["transition"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Tick_with_unchanged_state_emits_nothing()
|
||||
{
|
||||
var host = Build();
|
||||
host.StartCore(armWmi: false);
|
||||
|
||||
host.TickForTest();
|
||||
host.TickForTest();
|
||||
|
||||
Assert.Empty(PowerEvents());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Stopped_host_ignores_ticks()
|
||||
{
|
||||
var host = Build();
|
||||
host.StartCore(armWmi: false);
|
||||
host.Stop();
|
||||
|
||||
_probeResult = new PowerStateResult { OnAcPower = false, HasBattery = true, BatteryPercent = 50 };
|
||||
host.TickForTest();
|
||||
|
||||
Assert.Empty(PowerEvents());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Unknown_battery_percent_serializes_as_unknown()
|
||||
{
|
||||
var host = Build();
|
||||
host.StartCore(armWmi: false);
|
||||
|
||||
_probeResult = new PowerStateResult { OnAcPower = false, HasBattery = true, BatteryPercent = null, BatteryLifeMinutes = null };
|
||||
host.TickForTest();
|
||||
|
||||
var data = (IReadOnlyDictionary<string, object>)Assert.Single(PowerEvents()).TypedPayload!;
|
||||
Assert.Equal("unknown", data["batteryPercent"]);
|
||||
Assert.Equal("unknown", data["batteryLifeMinutes"]);
|
||||
}
|
||||
}
|
||||
}
|
||||
+2
@@ -35,6 +35,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
|
||||
// OS-eventlog forwarding — observability, not enrollment progress (session 8bc1180f:
|
||||
// an EventID-100 burst must not keep the periodic collectors alive).
|
||||
[InlineData("modern_deployment_log")]
|
||||
// Environment observation — battery drain on an idle device must not reset idle clocks.
|
||||
[InlineData("power_state_change")]
|
||||
// Agent health / control / transport — not device/enrollment progress (P2).
|
||||
[InlineData("collector_degraded")]
|
||||
[InlineData("telemetry_upload_poisoned")]
|
||||
|
||||
+11
-9
@@ -245,11 +245,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Terminal_to_active_retry_rearms_CompletedAt_and_final_terminal_covers_total_occupancy()
|
||||
public void Terminal_to_active_retry_rearms_BothEndpoints_and_final_terminal_covers_last_attempt()
|
||||
{
|
||||
// Audit Q1: the IME retry path (Error -> Downloading -> Installed is permitted by
|
||||
// AppPackageState.UpdateState). Pre-fix, CompletedAt froze at the first Error, so
|
||||
// the final Installed payload carried the first failure's completedAt/duration.
|
||||
// Audit Q1 + 2026-08 attempt-duration change: the IME retry path
|
||||
// (Error -> Downloading -> Installed is permitted by AppPackageState.UpdateState).
|
||||
// A terminal->active transition re-arms BOTH endpoints, so the final Installed
|
||||
// payload describes the retry ATTEMPT (matching the backend's AppInstallSummaries
|
||||
// semantics) — not the first-start -> last-terminal span across attempts.
|
||||
using var f = new ImeLogTrackerAdapterFixture(T0);
|
||||
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
|
||||
var app = new AppPackageState("app-R", 0);
|
||||
@@ -261,23 +263,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
|
||||
adapter.TriggerAppStateFromTest(app, AppInstallationState.Error, AppInstallationState.Downloading);
|
||||
|
||||
// Re-armed while active again: the retry's download event carries no completedAt,
|
||||
// and the FIRST start stamp is kept.
|
||||
// and the start stamp is the RETRY's re-entry time (fresh attempt window).
|
||||
Assert.Null(adapter.AppTimings["app-R"].CompletedAtUtc);
|
||||
var retryDownload = f.InfoEvent(SharedEventTypes.AppDownloadStarted);
|
||||
Assert.False(retryDownload.Payload!.ContainsKey("completedAt"));
|
||||
Assert.False(retryDownload.Payload.ContainsKey("durationSeconds"));
|
||||
Assert.Equal(T0.ToString("o"), retryDownload.Payload["startedAt"]);
|
||||
Assert.Equal(T0.AddSeconds(15).ToString("o"), retryDownload.Payload["startedAt"]);
|
||||
|
||||
f.Clock.Advance(TimeSpan.FromSeconds(20));
|
||||
adapter.TriggerAppStateFromTest(app, AppInstallationState.Downloading, AppInstallationState.Installed);
|
||||
|
||||
var timing = adapter.AppTimings["app-R"];
|
||||
Assert.Equal(T0, timing.StartedAtUtc);
|
||||
Assert.Equal(T0.AddSeconds(15), timing.StartedAtUtc);
|
||||
Assert.Equal(T0.AddSeconds(35), timing.CompletedAtUtc);
|
||||
Assert.Equal(35.0, timing.DurationSeconds);
|
||||
Assert.Equal(20.0, timing.DurationSeconds);
|
||||
|
||||
var completed = Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
|
||||
Assert.Equal("35.00", completed.Payload!["durationSeconds"]);
|
||||
Assert.Equal("20.00", completed.Payload!["durationSeconds"]);
|
||||
Assert.Equal(T0.AddSeconds(35).ToString("o"), completed.Payload["completedAt"]);
|
||||
}
|
||||
|
||||
|
||||
+25
@@ -107,6 +107,31 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Transport
|
||||
Assert.False(req.Headers.ContainsKey("X-Bootstrap-Token"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task UploadBatchAsync_stamps_send_time_header_per_attempt()
|
||||
{
|
||||
// P14: X-Send-Time-Utc carries the device clock at the SEND moment, round-trip
|
||||
// ISO-8601, one value per request. A retry is a new send moment — the header is
|
||||
// stamped inside UploadBatchAsync, so each attempt gets a fresh value.
|
||||
using var rig = new Rig();
|
||||
rig.Handler.QueueStatus(HttpStatusCode.ServiceUnavailable);
|
||||
rig.Handler.QueueStatus(HttpStatusCode.OK);
|
||||
|
||||
var before = DateTime.UtcNow;
|
||||
await rig.Sut.UploadBatchAsync(new[] { NewEventItem(1) }, CancellationToken.None);
|
||||
await rig.Sut.UploadBatchAsync(new[] { NewEventItem(1) }, CancellationToken.None);
|
||||
var after = DateTime.UtcNow;
|
||||
|
||||
Assert.Equal(2, rig.Handler.Captured.Count);
|
||||
foreach (var req in rig.Handler.Captured)
|
||||
{
|
||||
Assert.True(req.TryGetHeader("X-Send-Time-Utc", out var raw));
|
||||
var parsed = DateTime.Parse(raw, null, System.Globalization.DateTimeStyles.RoundtripKind);
|
||||
Assert.Equal(DateTimeKind.Utc, parsed.Kind);
|
||||
Assert.InRange(parsed, before.AddSeconds(-1), after.AddSeconds(1));
|
||||
}
|
||||
}
|
||||
|
||||
// ============================================================ Compression (bandwidth)
|
||||
|
||||
[Fact]
|
||||
|
||||
+7
-4
@@ -10,10 +10,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
/// the <c>app_tracking_summary</c> terminal event.
|
||||
/// <para>
|
||||
/// <b>StartedAtUtc</b> is recorded on the first lifecycle transition into Downloading /
|
||||
/// Installing / InProgress and never overwritten. <b>CompletedAtUtc</b> is recorded on a
|
||||
/// transition to a terminal state (Installed / Skipped / Postponed / Error) and re-armed
|
||||
/// to <c>null</c> when the app goes terminal → active again (IME retry / re-evaluation,
|
||||
/// audit Q1) — so the final terminal event carries first-start → last-terminal.
|
||||
/// Installing / InProgress. <b>CompletedAtUtc</b> is recorded on a transition to a
|
||||
/// terminal state (Installed / Skipped / Postponed / Error). When the app goes
|
||||
/// terminal → active again (IME retry / re-evaluation pass, audit Q1) BOTH endpoints
|
||||
/// re-arm — CompletedAtUtc to <c>null</c>, StartedAtUtc to the re-entry time — so the
|
||||
/// final terminal event carries the status-defining ATTEMPT's window (2026-08
|
||||
/// attempt-duration change, matching the backend's AppInstallSummaries semantics),
|
||||
/// never the first-start → last-terminal span across install passes.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class AppInstallTiming
|
||||
|
||||
+215
@@ -0,0 +1,215 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
{
|
||||
/// <summary>How the UTC value of a CMTrace line was arrived at. Recorded as evidence.</summary>
|
||||
public enum CmTraceOffsetOrigin
|
||||
{
|
||||
/// <summary>No offset known — the caller fell back to its own clock.</summary>
|
||||
None = 0,
|
||||
|
||||
/// <summary>The writer declared its offset in the line ("+480"). Authoritative.</summary>
|
||||
Bias = 1,
|
||||
|
||||
/// <summary>
|
||||
/// A per-FILE measured offset was applied. Retired by the 2026-08-20 revert (04b1a7c6):
|
||||
/// one offset per file is wrong when a file holds lines from two writer eras. Kept so
|
||||
/// stored events from agent 2.0.1410 remain interpretable.
|
||||
/// </summary>
|
||||
Calibrated = 2,
|
||||
|
||||
/// <summary>
|
||||
/// The line anchored ITSELF: it was read in a pass whose lines are provably fresh
|
||||
/// (written since the previous poll), so its own distance to the agent clock, rounded
|
||||
/// to the 15-minute offset grid, IS the writer's offset — no cross-line state involved.
|
||||
/// Immune to interleaved writer eras by construction. See
|
||||
/// <see cref="CmTraceOffsetCalibrator.TryMeasureOffset"/>.
|
||||
/// </summary>
|
||||
LineAnchored = 3,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Measures the UTC offset that the process WRITING a CMTrace log believes it is in.
|
||||
///
|
||||
/// <para>
|
||||
/// A CMTrace line carries local time and no offset — IME 1.104 writes
|
||||
/// <c>DateTime.Now.TimeOfDay</c> in both of its trace listeners. Converting such a line with
|
||||
/// the agent's own <c>TimeZoneInfo.Local</c> assumes the two processes agree, and they do not:
|
||||
/// both cache their zone at process start and neither follows a later <c>tzutil</c> or a
|
||||
/// Windows auto-timezone change. The resulting error is
|
||||
/// <c>Offset_writer - Offset_agent</c> and is zero only by coincidence. Field measurement over
|
||||
/// 11,068 sessions found 26 sessions carrying a real zone-offset error, from +1 h
|
||||
/// (GMT to W. Europe) to -17 h (a writer still on the Pacific OOBE default while the agent had
|
||||
/// already moved to E. Australia).
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// So the offset is MEASURED, not assumed. The tracker tails the log on a 100 ms poll, so a
|
||||
/// line that appeared since the previous pass was written essentially "now":
|
||||
/// <code>
|
||||
/// candidate = lineLocalTime - agentUtcNow
|
||||
/// offset = round(candidate / 15 min) * 15 min
|
||||
/// lineUtc = lineLocalTime - offset
|
||||
/// </code>
|
||||
/// Every real UTC offset is a whole multiple of 15 minutes, so the rounding absorbs poll and
|
||||
/// flush latency without ambiguity.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// THE FRESHNESS REQUIREMENT IS WHAT MAKES THIS CORRECT. An anchor must be the newest line of
|
||||
/// a pass in which the file actually GREW. Without that rule a backlog line that happens to be
|
||||
/// an exact multiple of 15 minutes old (30 min, 45 min, ...) rounds to a clean value with zero
|
||||
/// residual and calibrates the offset to a wrong figure — the residual guard alone cannot
|
||||
/// catch it. Callers must only offer anchors they know to be fresh.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// Calibration is kept per source file: IntuneManagementExtension.log and AppWorkload.log are
|
||||
/// written by different processes and can hold different beliefs. Re-calibrating on every
|
||||
/// fresh anchor makes this self-healing — if the writing process restarts and picks up the
|
||||
/// current zone, the measurement follows within one poll.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class CmTraceOffsetCalibrator
|
||||
{
|
||||
/// <summary>Grid that every real UTC offset sits on.</summary>
|
||||
internal static readonly TimeSpan OffsetGrid = TimeSpan.FromMinutes(15);
|
||||
|
||||
/// <summary>
|
||||
/// How far a candidate may sit off the grid before it is rejected. Poll latency (100 ms)
|
||||
/// plus the writer's flush latency are far below this; anything larger means the anchor
|
||||
/// was not actually fresh.
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan MaxGridResidual = TimeSpan.FromMinutes(2);
|
||||
|
||||
/// <summary>Real UTC offsets span UTC-12 (Dateline) to UTC+14 (Line Islands).</summary>
|
||||
internal static readonly TimeSpan MinOffset = TimeSpan.FromHours(-12);
|
||||
|
||||
/// <summary>Upper bound of a real UTC offset. See <see cref="MinOffset"/>.</summary>
|
||||
internal static readonly TimeSpan MaxOffset = TimeSpan.FromHours(14);
|
||||
|
||||
private sealed class Anchor
|
||||
{
|
||||
public TimeSpan Offset;
|
||||
public DateTime LocalTimestamp;
|
||||
}
|
||||
|
||||
private readonly object _lock = new object();
|
||||
|
||||
private readonly Dictionary<string, Anchor> _bySource =
|
||||
new Dictionary<string, Anchor>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
/// <summary>
|
||||
/// Offer a freshly appended line as a calibration anchor.
|
||||
/// <para>
|
||||
/// The caller MUST only pass a line it knows to be fresh — the newest line of a pass in
|
||||
/// which the file grew. See the freshness note on this class.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
/// <param name="sourceKey">Log file identity; the file name is enough.</param>
|
||||
/// <param name="localTimestamp">The line's local time exactly as written (Kind is ignored).</param>
|
||||
/// <param name="agentUtcNow">The agent's UTC clock at parse time.</param>
|
||||
/// <returns><c>true</c> when the anchor was accepted and the offset updated.</returns>
|
||||
/// <summary>
|
||||
/// The pure measurement: the distance between a FRESH line's local time and the agent's
|
||||
/// UTC clock, rounded to the 15-minute offset grid, is the writer's UTC offset.
|
||||
/// <para>
|
||||
/// Freshness is the caller's obligation and is what makes this correct — for a line
|
||||
/// written within the last poll interval, <c>local - now</c> differs from the writer's
|
||||
/// true offset only by seconds, which the grid rounding absorbs. Returns <c>false</c>
|
||||
/// when the residual exceeds <see cref="MaxGridResidual"/> (the line is not fresh, or
|
||||
/// the writer's clock is broken) or the rounded value is not a real timezone offset.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Known, accepted edge: a freshly WRITTEN line carrying an old REPLAYED timestamp whose
|
||||
/// age happens to sit within the residual of an exact grid multiple measures that age as
|
||||
/// an "offset" — the line then resolves to roughly the read time instead of its replayed
|
||||
/// past. The error is bounded by construction (an anchored line always resolves to
|
||||
/// now ± <see cref="MaxGridResidual"/>), the case needs an age of exactly N×15 min ± 2 min,
|
||||
/// and sub-24h replays are treated as current by the historic-replay guard anyway.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static bool TryMeasureOffset(DateTime localTimestamp, DateTime agentUtcNow, out TimeSpan offset)
|
||||
{
|
||||
var candidate = DateTime.SpecifyKind(localTimestamp, DateTimeKind.Unspecified)
|
||||
- DateTime.SpecifyKind(agentUtcNow, DateTimeKind.Unspecified);
|
||||
|
||||
var gridUnits = Math.Round(candidate.TotalMinutes / OffsetGrid.TotalMinutes,
|
||||
MidpointRounding.AwayFromZero);
|
||||
offset = TimeSpan.FromMinutes(gridUnits * OffsetGrid.TotalMinutes);
|
||||
|
||||
// Off-grid by more than the tolerance: the line was not fresh, or the writer's own
|
||||
// clock is broken. Either way it must not become an offset.
|
||||
if ((candidate - offset).Duration() > MaxGridResidual) return false;
|
||||
|
||||
// Not a value any real timezone can produce.
|
||||
if (offset < MinOffset || offset > MaxOffset) return false;
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
public bool TryCalibrate(string sourceKey, DateTime localTimestamp, DateTime agentUtcNow)
|
||||
{
|
||||
if (string.IsNullOrEmpty(sourceKey)) return false;
|
||||
|
||||
TimeSpan offset;
|
||||
if (!TryMeasureOffset(localTimestamp, agentUtcNow, out offset)) return false;
|
||||
|
||||
lock (_lock)
|
||||
{
|
||||
if (_bySource.TryGetValue(sourceKey, out var existing))
|
||||
{
|
||||
// Only ever move the anchor forward. A line older than the current anchor
|
||||
// carries nothing newer and could re-introduce a stale belief.
|
||||
if (localTimestamp <= existing.LocalTimestamp) return false;
|
||||
existing.Offset = offset;
|
||||
existing.LocalTimestamp = localTimestamp;
|
||||
}
|
||||
else
|
||||
{
|
||||
_bySource[sourceKey] = new Anchor
|
||||
{
|
||||
Offset = offset,
|
||||
LocalTimestamp = localTimestamp,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
/// <summary>The offset measured for this source, when one has been established.</summary>
|
||||
public bool TryGetOffset(string sourceKey, out TimeSpan offset)
|
||||
{
|
||||
offset = TimeSpan.Zero;
|
||||
if (string.IsNullOrEmpty(sourceKey)) return false;
|
||||
|
||||
lock (_lock)
|
||||
{
|
||||
Anchor anchor;
|
||||
if (!_bySource.TryGetValue(sourceKey, out anchor)) return false;
|
||||
offset = anchor.Offset;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Convert a line's local time to UTC using the measured offset. Returns <c>false</c> when
|
||||
/// this source has not been calibrated yet — the caller then picks its own fallback and
|
||||
/// should mark the result as not source-grounded.
|
||||
/// </summary>
|
||||
public bool TryResolveUtc(string sourceKey, DateTime localTimestamp, out DateTime utc)
|
||||
{
|
||||
utc = default(DateTime);
|
||||
|
||||
TimeSpan offset;
|
||||
if (!TryGetOffset(sourceKey, out offset)) return false;
|
||||
|
||||
utc = DateTime.SpecifyKind(
|
||||
DateTime.SpecifyKind(localTimestamp, DateTimeKind.Unspecified) - offset,
|
||||
DateTimeKind.Utc);
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
+231
-6
@@ -42,7 +42,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
return false;
|
||||
}
|
||||
|
||||
private async Task CheckLogFilesAsync(CancellationToken token)
|
||||
/// <remarks>internal (not private) as a test seam: the offset calibration depends on the
|
||||
/// pass structure — first observation versus a later pass over a grown file — which only
|
||||
/// a real read cycle exercises. Production still reaches it solely from the poll loop.</remarks>
|
||||
internal async Task CheckLogFilesAsync(CancellationToken token)
|
||||
{
|
||||
if (!Directory.Exists(_logFolder))
|
||||
return;
|
||||
@@ -74,7 +77,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
var fileInfo = new FileInfo(filePath);
|
||||
if (!fileInfo.Exists) continue;
|
||||
|
||||
// Captured BEFORE MarkChecked below stamps this pass. Growth measured
|
||||
// against a previously observed state is what makes this pass's lines valid
|
||||
// "written now" anchors — for the per-file measurement AND per-line anchoring.
|
||||
var hadPreviousObservation = _positionTracker.HasSeen(filePath);
|
||||
var lastCheckedUtc = _positionTracker.GetLastCheckedUtc(filePath);
|
||||
var passNowUtc = UtcNowProvider();
|
||||
|
||||
var startPos = _positionTracker.GetSafePosition(filePath, fileInfo.Length);
|
||||
|
||||
// Every look counts, including "no new data" and an empty first sight: the
|
||||
// NEXT pass's freshness window is measured from here. Restored bookmarks
|
||||
// deliberately carry no LastCheckedUtc — the first pass after a restart reads
|
||||
// downtime backlog and must never count as fresh.
|
||||
_positionTracker.MarkChecked(filePath, passNowUtc);
|
||||
|
||||
if (startPos >= fileInfo.Length)
|
||||
{
|
||||
// M2: guard the interpolated string so it isn't built every 100 ms tick
|
||||
@@ -86,6 +103,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
if (_logger.LogLevel >= AgentLogLevel.Trace)
|
||||
_logger.Trace($"ImeLogTracker: reading {Path.GetFileName(filePath)} from pos {startPos} (size={fileInfo.Length}, delta={fileInfo.Length - startPos})");
|
||||
|
||||
_currentSourceFileName = Path.GetFileName(filePath);
|
||||
_currentPassLinesAreFresh = hadPreviousObservation
|
||||
&& lastCheckedUtc > DateTime.MinValue
|
||||
&& (passNowUtc - lastCheckedUtc) <= FreshLineMaxAge;
|
||||
|
||||
// Newest bias-less line of this pass — the calibration anchor. Bias-carrying
|
||||
// lines are skipped: they already state the writer's offset, so they need no
|
||||
// measurement and must not overwrite one.
|
||||
CmTraceLogEntry calibrationAnchor = null;
|
||||
|
||||
using (var stream = new FileStream(filePath, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
|
||||
{
|
||||
stream.Seek(startPos, SeekOrigin.Begin);
|
||||
@@ -147,6 +174,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
if (CmTraceLogParser.TryParseLine(line, out entry))
|
||||
{
|
||||
messageToMatch = entry.Message;
|
||||
if (entry.HasTimestamp && !entry.BiasMinutes.HasValue)
|
||||
calibrationAnchor = entry;
|
||||
}
|
||||
else
|
||||
{
|
||||
@@ -160,7 +189,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
// Simulation mode delay
|
||||
if (SimulationMode && entry != null)
|
||||
{
|
||||
await ApplySimulationDelay(entry.Timestamp, token);
|
||||
await ApplySimulationDelay(ResolveEntryUtc(entry), token);
|
||||
}
|
||||
|
||||
// Match against active patterns
|
||||
@@ -185,6 +214,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
|
||||
_positionTracker.SetPosition(filePath, stream.Position);
|
||||
_stateDirty = true;
|
||||
|
||||
// Calibrate AFTER the pass: this pass's lines were resolved with the
|
||||
// offset established previously, at most one poll (100 ms) old. Buffering
|
||||
// the pass to calibrate first is not an option — the first pass of
|
||||
// AppWorkload.log can be hundreds of MB. The cost is a warm-up of one
|
||||
// growing pass, during which lines fall back to the reader zone and are
|
||||
// flagged as such.
|
||||
if (hadPreviousObservation && calibrationAnchor != null)
|
||||
CalibrateFrom(_currentSourceFileName, Path.GetFileName(filePath), calibrationAnchor);
|
||||
}
|
||||
}
|
||||
catch (FileNotFoundException) { }
|
||||
@@ -222,7 +260,32 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
private void HandlePatternMatch(CompiledPattern pattern, Match match, string message, CmTraceLogEntry entry)
|
||||
{
|
||||
LastMatchedPatternId = pattern.PatternId;
|
||||
LastMatchedLogTimestamp = entry?.Timestamp;
|
||||
if (entry == null)
|
||||
{
|
||||
LastMatchedLogTimestamp = null;
|
||||
LastMatchedSourceLocalTimestamp = null;
|
||||
LastMatchedSourceOffsetMinutes = null;
|
||||
LastMatchedSourceOffsetOrigin = CmTraceOffsetOrigin.None;
|
||||
LastMatchedMeasuredWriterOffsetMinutes = null;
|
||||
}
|
||||
else
|
||||
{
|
||||
CmTraceOffsetOrigin origin;
|
||||
int? offsetMinutes;
|
||||
LastMatchedLogTimestamp = ResolveEntryUtc(entry, out origin, out offsetMinutes);
|
||||
LastMatchedSourceLocalTimestamp = entry.HasTimestamp ? entry.LocalTimestamp : (DateTime?)null;
|
||||
LastMatchedSourceOffsetMinutes = offsetMinutes;
|
||||
LastMatchedSourceOffsetOrigin = origin;
|
||||
|
||||
// Observational: what the calibrator measured for this file, distinct from what
|
||||
// was applied above.
|
||||
TimeSpan measuredWriterOffset;
|
||||
LastMatchedMeasuredWriterOffsetMinutes =
|
||||
_currentSourceFileName != null
|
||||
&& OffsetCalibrator.TryGetOffset(_currentSourceFileName, out measuredWriterOffset)
|
||||
? (int)measuredWriterOffset.TotalMinutes
|
||||
: (int?)null;
|
||||
}
|
||||
|
||||
// Generic pattern-match hook (M4.4.4). Invoked before action-specific callbacks so
|
||||
// subscribers (e.g. ImeLogTrackerAdapter emitting WhiteGloveSealingPatternDetected)
|
||||
@@ -237,10 +300,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
// state nor the phase-change ignore list. SimulationMode (--replay-log-dir dev tool)
|
||||
// replays historic logs on purpose and bypasses the guard.
|
||||
var isStaleReplayLine = !SimulationMode && entry != null
|
||||
&& (UtcNowProvider() - NormalizeUtc(entry.Timestamp)) > HistoricReplayThreshold;
|
||||
&& (UtcNowProvider() - NormalizeUtc(ResolveEntryUtc(entry))) > HistoricReplayThreshold;
|
||||
if (isStaleReplayLine && AppMutatingActions.Contains(pattern.Action?.ToLower() ?? string.Empty))
|
||||
{
|
||||
_logger.Debug($"ImeLogTracker: skipped app action '{pattern.Action}' for historic line ({entry.Timestamp:o})");
|
||||
_logger.Debug($"ImeLogTracker: skipped app action '{pattern.Action}' for historic line ({ResolveEntryUtc(entry):o})");
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -524,8 +587,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
internal void ProcessLogMessageForTest(string message, DateTime? sourceTimestampUtc = null)
|
||||
{
|
||||
if (string.IsNullOrEmpty(message)) return;
|
||||
// The test seam hands in an already-resolved UTC instant, so it takes the same route a
|
||||
// writer-declared bias does: TimestampUtc set, no zone left to guess.
|
||||
var entry = sourceTimestampUtc.HasValue
|
||||
? new CmTraceLogEntry { Timestamp = sourceTimestampUtc.Value, Message = message }
|
||||
? new CmTraceLogEntry
|
||||
{
|
||||
TimestampUtc = sourceTimestampUtc.Value,
|
||||
LocalTimestamp = DateTime.SpecifyKind(sourceTimestampUtc.Value, DateTimeKind.Unspecified),
|
||||
HasTimestamp = true,
|
||||
Message = message,
|
||||
}
|
||||
: null;
|
||||
foreach (var pattern in _activePatterns)
|
||||
{
|
||||
@@ -543,5 +614,159 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
/// Parses [DO TEL] JSON and links it to the correct app via FileId.
|
||||
/// The FileId contains the app GUID in the format: ...intunewin-bin_{appGuid}_{number}
|
||||
/// </summary>
|
||||
|
||||
/// <summary>
|
||||
/// Resolve a parsed CMTrace entry to UTC.
|
||||
///
|
||||
/// <para>
|
||||
/// Order of preference: a writer-declared bias (authoritative) > this process's own zone
|
||||
/// (a fallback carrying a known defect, see
|
||||
/// <see cref="CmTraceLogParser.ResolveUtcAssumingReaderZone"/>) > the agent clock for a
|
||||
/// line with no parseable timestamp.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// The reader-zone step is where the tracker is still wrong when IME's process holds a
|
||||
/// different zone belief than ours. It is preserved verbatim here so that splitting the
|
||||
/// parser changes no behaviour; the measured offset from <c>CmTraceOffsetCalibrator</c>
|
||||
/// slots in at exactly this point in the follow-up change.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private DateTime ResolveEntryUtc(CmTraceLogEntry entry)
|
||||
{
|
||||
CmTraceOffsetOrigin origin;
|
||||
int? offsetMinutes;
|
||||
return ResolveEntryUtc(entry, out origin, out offsetMinutes);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resolve a parsed entry to UTC and report HOW the offset was obtained, so the emitting
|
||||
/// side can attach the evidence (P8). See <see cref="CmTraceOffsetOrigin"/>.
|
||||
/// </summary>
|
||||
private DateTime ResolveEntryUtc(CmTraceLogEntry entry, out CmTraceOffsetOrigin origin, out int? offsetMinutes)
|
||||
{
|
||||
origin = CmTraceOffsetOrigin.None;
|
||||
offsetMinutes = null;
|
||||
|
||||
if (entry == null) return UtcNowProvider();
|
||||
|
||||
// Writer declared its own offset — nothing left to measure.
|
||||
if (entry.TimestampUtc.HasValue)
|
||||
{
|
||||
origin = CmTraceOffsetOrigin.Bias;
|
||||
// Bias uses the GetTimeZoneInformation convention (UTC = local + bias); report the
|
||||
// value in the same sense as a measured offset (local = UTC + offset).
|
||||
offsetMinutes = entry.BiasMinutes.HasValue ? -entry.BiasMinutes.Value : (int?)null;
|
||||
return entry.TimestampUtc.Value;
|
||||
}
|
||||
|
||||
if (!entry.HasTimestamp) return UtcNowProvider();
|
||||
|
||||
// ERA-AWARE RESOLUTION (2026-08-20, second attempt): a FRESH line anchors ITSELF.
|
||||
//
|
||||
// The first attempt (e9dba11b, reverted in 04b1a7c6) applied one measured offset per
|
||||
// FILE — wrong, because a single log file holds lines from multiple writer eras:
|
||||
// AgentExecutor.log is written by short-lived child processes whose zone belief
|
||||
// flips per process, interleaved in one file (fixture
|
||||
// tests/fixtures/cmtrace-logs/agentexecutor-two-writer-eras-v1.cmtrace). Any
|
||||
// cross-line anchor — per file, or nearest-in-write-order — inherits that trap.
|
||||
//
|
||||
// Per-line self-anchoring does not: at a 100 ms poll every line of a growing pass
|
||||
// was written essentially "now", so its own distance to the agent clock IS its
|
||||
// writer's offset, era by era. The line's timestamp still contributes the sub-poll
|
||||
// precision and ordering (two lines 6 ms apart stay 6 ms apart — a plain
|
||||
// occurredAt=now could not do that), while the offset grid absorbs poll latency.
|
||||
// Freshness is load-bearing (see FreshLineMaxAge) — backlog passes, restart
|
||||
// catch-up and replay logs must never anchor, because an old line's AGE can round
|
||||
// onto the grid.
|
||||
if (_currentPassLinesAreFresh)
|
||||
{
|
||||
TimeSpan lineOffset;
|
||||
if (CmTraceOffsetCalibrator.TryMeasureOffset(entry.LocalTimestamp, UtcNowProvider(), out lineOffset))
|
||||
{
|
||||
origin = CmTraceOffsetOrigin.LineAnchored;
|
||||
offsetMinutes = (int)lineOffset.TotalMinutes;
|
||||
return DateTime.SpecifyKind(entry.LocalTimestamp - lineOffset, DateTimeKind.Utc);
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback for everything not provably fresh: assume the reader's zone, UNIFORMLY.
|
||||
// Wrong in absolute terms whenever the writer held a different belief, but
|
||||
// self-consistent — both ends of a duration are wrong by the same amount, so derived
|
||||
// durations stay right. That asymmetry is the lesson of the 04b1a7c6 revert:
|
||||
// partially corrected is strictly worse than uniformly wrong. Origin stays None so
|
||||
// the emitted event says so.
|
||||
var readerZoneOffset = TimeZoneInfo.Local.GetUtcOffset(entry.LocalTimestamp);
|
||||
offsetMinutes = (int)readerZoneOffset.TotalMinutes;
|
||||
return CmTraceLogParser.ResolveUtcAssumingReaderZone(entry.LocalTimestamp);
|
||||
}
|
||||
|
||||
// Field-forensics (tripwire, 2026-08-20): sessions e9753578 and 067797d8 both logged a
|
||||
// measurement whose LABEL and ANCHOR could not have come from the same file iteration —
|
||||
// 067797d8's anchor value (19:51:26.948) does not even exist in ANY log file on the
|
||||
// device, and IntuneManagementExtension.log never produced its expected first
|
||||
// measurement at all. Mechanism unexplained; resolution is immune (per-line anchoring),
|
||||
// but the observational layer must convict the culprit on the next occurrence. Hence:
|
||||
// the stream's file name travels alongside the label (a mismatch is THE smoking gun),
|
||||
// and the first rejected anchor per file becomes visible instead of silent.
|
||||
private readonly HashSet<string> _calibrationRejectLogged = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
/// <summary>
|
||||
/// Feed the pass's newest bias-less line to the calibrator and report a measured offset
|
||||
/// that disagrees with this process's own zone — that disagreement is precisely the
|
||||
/// condition that used to corrupt every IME-derived timestamp silently.
|
||||
/// </summary>
|
||||
/// <param name="sourceFileName">The calibration key (<c>_currentSourceFileName</c>).</param>
|
||||
/// <param name="streamFileName">The file the enclosing iteration actually read. Must equal
|
||||
/// <paramref name="sourceFileName"/> — logged loudly when it does not (see tripwire note).</param>
|
||||
private void CalibrateFrom(string sourceFileName, string streamFileName, CmTraceLogEntry anchor)
|
||||
{
|
||||
if (!string.Equals(sourceFileName, streamFileName, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
_logger?.Warning(
|
||||
$"ImeLogTracker: CALIBRATION LABEL MISMATCH — key='{sourceFileName}' but the iteration read '{streamFileName}' " +
|
||||
$"(anchor local={anchor.LocalTimestamp:yyyy-MM-ddTHH:mm:ss.fffffff}). Skipping this anchor; see tripwire note.");
|
||||
return;
|
||||
}
|
||||
|
||||
TimeSpan previous;
|
||||
var hadOffset = OffsetCalibrator.TryGetOffset(sourceFileName, out previous);
|
||||
|
||||
if (!OffsetCalibrator.TryCalibrate(sourceFileName, anchor.LocalTimestamp, UtcNowProvider()))
|
||||
{
|
||||
// Once per file: why a file that visibly grows never produces a measurement.
|
||||
// 067797d8 ran 24 s of growing IME.log passes with no measurement at all — this
|
||||
// silence is exactly what made the phenomenon undiagnosable from the log.
|
||||
if (_calibrationRejectLogged.Add(sourceFileName))
|
||||
{
|
||||
_logger?.Debug(
|
||||
$"ImeLogTracker: {sourceFileName} anchor rejected by calibrator " +
|
||||
$"(anchor local={anchor.LocalTimestamp:yyyy-MM-ddTHH:mm:ss.fffffff}, now={UtcNowProvider():HH:mm:ss.fff}Z) — first rejection for this file, further ones stay silent.");
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
TimeSpan measured;
|
||||
if (!OffsetCalibrator.TryGetOffset(sourceFileName, out measured)) return;
|
||||
if (hadOffset && measured == previous) return;
|
||||
|
||||
// The anchor's local timestamp is part of the record ON PURPOSE (tripwire, 2026-08-20):
|
||||
// session e9753578 logged "+02:00 measured for IntuneManagementExtension.log" although
|
||||
// that file provably held no +2-era line — an anchor/label pairing the committed code
|
||||
// could not be shown to produce. Should it ever happen again, the logged anchor value
|
||||
// itself will prove (or refute) the crossing without needing the device's log files.
|
||||
var readerZoneOffset = TimeZoneInfo.Local.GetUtcOffset(UtcNowProvider());
|
||||
if (measured == readerZoneOffset)
|
||||
{
|
||||
_logger?.Debug(
|
||||
$"ImeLogTracker: {sourceFileName} writer offset measured {measured} (matches this process, anchor local={anchor.LocalTimestamp:yyyy-MM-ddTHH:mm:ss.fff}).");
|
||||
return;
|
||||
}
|
||||
|
||||
_logger?.Info(
|
||||
$"ImeLogTracker: {sourceFileName} writer offset measured {measured}, this process believes {readerZoneOffset} " +
|
||||
$"(anchor local={anchor.LocalTimestamp:yyyy-MM-ddTHH:mm:ss.fff}). Measurement is observational — resolution anchors per line.");
|
||||
}
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
@@ -85,6 +85,42 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
/// Unspecified depending on the parser path. Normalize via SpecifyKind=Utc rather than
|
||||
/// ToUniversalTime() to avoid double-conversion when Kind is already UTC.
|
||||
/// </summary>
|
||||
/// <summary>
|
||||
/// Measures the UTC offset the IME process believes it is in, per log file.
|
||||
/// <para>
|
||||
/// Only this tracker may WRITE to it: calibration requires tailing with growth detection
|
||||
/// so the freshness rule can be honoured, and no other collector tails. Exposed so the
|
||||
/// snapshot-reading collectors can eventually READ from a shared instance
|
||||
/// (tasks/todo.md, P3b).
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal CmTraceOffsetCalibrator OffsetCalibrator { get; } = new CmTraceOffsetCalibrator();
|
||||
|
||||
/// <summary>
|
||||
/// File whose lines are currently being processed. The polling loop walks files strictly
|
||||
/// sequentially on a single task, so a field is sufficient to give <c>ResolveEntryUtc</c>
|
||||
/// and the handlers the calibration key without threading it through every signature.
|
||||
/// </summary>
|
||||
private string _currentSourceFileName;
|
||||
|
||||
/// <summary>
|
||||
/// Whether every line of the file iteration currently in flight is provably FRESH —
|
||||
/// written after the previous look at this file, which happened at most
|
||||
/// <see cref="FreshLineMaxAge"/> ago in THIS process. Only then may a line anchor its
|
||||
/// own UTC offset (<see cref="CmTraceOffsetCalibrator.TryMeasureOffset"/>): the grid
|
||||
/// rounding is safe when the line's age is bounded by seconds, and becomes a trap on
|
||||
/// backlog whose age can hit an exact grid multiple. Same single-task justification as
|
||||
/// <see cref="_currentSourceFileName"/>.
|
||||
/// </summary>
|
||||
private bool _currentPassLinesAreFresh;
|
||||
|
||||
/// <summary>
|
||||
/// Maximum gap between two looks at a file for its new bytes to count as fresh. Far
|
||||
/// above the 100 ms poll (rides out GC pauses and ESP CPU starvation), far below the
|
||||
/// 13-minute danger zone where a line's AGE could round onto the 15-minute offset grid.
|
||||
/// </summary>
|
||||
internal static readonly TimeSpan FreshLineMaxAge = TimeSpan.FromSeconds(30);
|
||||
|
||||
internal static DateTime NormalizeUtc(DateTime value) =>
|
||||
value.Kind == DateTimeKind.Utc ? value : DateTime.SpecifyKind(value, DateTimeKind.Utc);
|
||||
|
||||
@@ -191,6 +227,38 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
public string LastMatchedPatternId { get; internal set; }
|
||||
public DateTime? LastMatchedLogTimestamp { get; internal set; }
|
||||
|
||||
/// <summary>
|
||||
/// Raw local time of the last matched line, exactly as the writer wrote it — no zone
|
||||
/// attached, because the line carries none.
|
||||
/// <para>
|
||||
/// Recorded so an emitted event can carry the evidence for its own timestamp. Without it
|
||||
/// a conversion error is only provable from the diagnostics archive, and of the 5,386
|
||||
/// sessions that went through a timezone change essentially none had one — the field
|
||||
/// analysis had to fall back to median statistics over thousands of sessions to show
|
||||
/// what one stored value would have made obvious per session.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public DateTime? LastMatchedSourceLocalTimestamp { get; internal set; }
|
||||
|
||||
/// <summary>Offset applied to <see cref="LastMatchedSourceLocalTimestamp"/>, in minutes.</summary>
|
||||
public int? LastMatchedSourceOffsetMinutes { get; internal set; }
|
||||
|
||||
/// <summary>How that offset was arrived at — declared, measured, or neither.</summary>
|
||||
public CmTraceOffsetOrigin LastMatchedSourceOffsetOrigin { get; internal set; }
|
||||
|
||||
/// <summary>
|
||||
/// The offset the calibrator MEASURED for the source file, which is not necessarily the
|
||||
/// one applied.
|
||||
/// <para>
|
||||
/// Since the 2026-08-20 revert the measurement is observational only: it is recorded so the
|
||||
/// field tells us what an era-aware calibrator would have to work with, without acting on
|
||||
/// it. Reported separately from <see cref="LastMatchedSourceOffsetMinutes"/> precisely so
|
||||
/// the two can never be confused — that event said "calibrated" while carrying a wrong
|
||||
/// value, and an honest record has to distinguish measured from applied.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public int? LastMatchedMeasuredWriterOffsetMinutes { get; internal set; }
|
||||
|
||||
// Callbacks to EnrollmentTracker
|
||||
public Action<string> OnEspPhaseChanged { get; set; }
|
||||
public Action<string> OnImeAgentVersion { get; set; }
|
||||
|
||||
+62
-1
@@ -19,6 +19,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
/// has been rotated - resets to 0 to read from beginning.
|
||||
/// Returns 0 if no position has been recorded yet.
|
||||
/// </summary>
|
||||
/// <summary>
|
||||
/// Whether this file was already observed in an earlier pass.
|
||||
/// <para>
|
||||
/// The offset calibration depends on this: growth measured against a size we saw before
|
||||
/// means the new bytes were written within one poll interval, so the last line of that
|
||||
/// pass is a valid "written now" anchor. On the FIRST sight of a file we read it from
|
||||
/// position 0 and its last line may be arbitrarily old — anchoring on that would measure
|
||||
/// the line's age instead of the writer's timezone offset.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public bool HasSeen(string filePath) => _positions.ContainsKey(filePath);
|
||||
|
||||
public long GetSafePosition(string filePath, long currentFileSize)
|
||||
{
|
||||
FilePositionState state;
|
||||
@@ -53,6 +65,42 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
state.LastReadTime = DateTime.UtcNow;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Records that the file was looked at in a poll pass — whether or not it had new data.
|
||||
/// Creates the entry on first sight (position 0), so a file first seen EMPTY counts as
|
||||
/// observed and its first content is provably fresh.
|
||||
/// <para>
|
||||
/// This timestamp is the freshness reference for per-line offset anchoring: bytes found
|
||||
/// in the NEXT pass were written after this instant, so
|
||||
/// <c>now - LastCheckedUtc</c> bounds the age of every line in that pass's chunk.
|
||||
/// The caller supplies the clock (the tracker's <c>UtcNowProvider</c>) so replay/test
|
||||
/// clocks stay consistent with the freshness comparison.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public void MarkChecked(string filePath, DateTime nowUtc)
|
||||
{
|
||||
FilePositionState state;
|
||||
if (!_positions.TryGetValue(filePath, out state))
|
||||
{
|
||||
state = new FilePositionState();
|
||||
_positions[filePath] = state;
|
||||
}
|
||||
|
||||
state.LastCheckedUtc = nowUtc;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// When the file was last looked at, or <see cref="DateTime.MinValue"/> when it never
|
||||
/// was in THIS process — including entries restored from persisted state, whose
|
||||
/// bookmark survives an agent restart but whose freshness deliberately does not
|
||||
/// (the restart gap can hold arbitrarily old backlog).
|
||||
/// </summary>
|
||||
public DateTime GetLastCheckedUtc(string filePath)
|
||||
{
|
||||
FilePositionState state;
|
||||
return _positions.TryGetValue(filePath, out state) ? state.LastCheckedUtc : DateTime.MinValue;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Gets the stored position for a file, or 0 if not tracked.
|
||||
/// Does not perform rollover detection.
|
||||
@@ -77,6 +125,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
/// <summary>
|
||||
/// Restores a previously persisted position for a file.
|
||||
/// Used on agent restart to continue reading from the last known position.
|
||||
/// <para>
|
||||
/// <see cref="FilePositionState.LastCheckedUtc"/> is deliberately NOT restored: the
|
||||
/// bookmark survives the restart, the freshness guarantee does not — the first pass
|
||||
/// after a restart reads the whole downtime backlog and must never anchor per-line
|
||||
/// offsets on it.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public void RestorePosition(string filePath, long position, long lastKnownSize)
|
||||
{
|
||||
@@ -84,7 +138,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
{
|
||||
Position = position,
|
||||
LastKnownSize = lastKnownSize,
|
||||
LastReadTime = DateTime.UtcNow
|
||||
LastReadTime = DateTime.UtcNow,
|
||||
LastCheckedUtc = DateTime.MinValue
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -94,5 +149,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
|
||||
public long Position { get; set; }
|
||||
public long LastKnownSize { get; set; }
|
||||
public DateTime LastReadTime { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// When this file was last looked at by a poll pass IN THIS PROCESS. In-memory only —
|
||||
/// never persisted, so a restart resets it (see <see cref="LogFilePositionTracker.RestorePosition"/>).
|
||||
/// </summary>
|
||||
public DateTime LastCheckedUtc { get; set; }
|
||||
}
|
||||
}
|
||||
|
||||
+197
@@ -0,0 +1,197 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Runtime;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
{
|
||||
/// <summary>
|
||||
/// Pure state machine behind <c>PowerStateWatcherHost</c>: diffs successive
|
||||
/// <see cref="PowerStateResult"/> snapshots into <c>power_state_change</c> emissions.
|
||||
/// No WMI, no timers, no logging — fully deterministic and unit-testable.
|
||||
/// <para>
|
||||
/// Semantics:
|
||||
/// <list type="bullet">
|
||||
/// <item><b>AC↔battery transitions</b> — one emission per accepted edge
|
||||
/// (<c>ac_to_battery</c> Warning/immediate, <c>battery_to_ac</c> Info).</item>
|
||||
/// <item><b>Threshold ladder</b> (default 50/30/15 %) — evaluated only while ON battery;
|
||||
/// each level latches once per tracker lifetime (no re-emit when the device charges
|
||||
/// back up and drains again — avoids plug-cycle alarm spam). A multi-level jump
|
||||
/// emits only the LOWEST newly-crossed level. Entering battery power already below
|
||||
/// a level counts as crossing it (covers "unplugged at 12 %" and, via
|
||||
/// <see cref="Baseline"/>, "enrollment started at 12 %").</item>
|
||||
/// <item><b>Storm cap</b> — a lifetime emission cap (default 20) as backstop against a
|
||||
/// flapping dock / dying battery controller; <see cref="EmissionCapReached"/> flips
|
||||
/// once so the host can log a single warning.</item>
|
||||
/// </list>
|
||||
/// A snapshot with <see cref="PowerStateResult.ProbeError"/> is ignored (previous state
|
||||
/// preserved); a null <see cref="PowerStateResult.BatteryPercent"/> skips threshold
|
||||
/// evaluation but AC diffing still works.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal sealed class PowerStateTransitionTracker
|
||||
{
|
||||
public const string TransitionAcToBattery = "ac_to_battery";
|
||||
public const string TransitionBatteryToAc = "battery_to_ac";
|
||||
public const string TransitionThresholdCrossed = "threshold_crossed";
|
||||
|
||||
internal static readonly int[] DefaultThresholdLadder = { 50, 30, 15 };
|
||||
internal const int DefaultMaxEmissions = 20;
|
||||
|
||||
private readonly int[] _ladder; // strictly descending
|
||||
private readonly int _maxEmissions;
|
||||
private readonly HashSet<int> _latchedLevels = new HashSet<int>();
|
||||
|
||||
private bool _hasState;
|
||||
private bool _lastOnAc;
|
||||
private int _emitted;
|
||||
|
||||
public PowerStateTransitionTracker(IReadOnlyList<int>? thresholdLadder = null, int maxEmissions = DefaultMaxEmissions)
|
||||
{
|
||||
_ladder = (thresholdLadder ?? DefaultThresholdLadder).OrderByDescending(l => l).ToArray();
|
||||
_maxEmissions = maxEmissions;
|
||||
}
|
||||
|
||||
/// <summary>True once the lifetime emission cap suppressed at least one emission.</summary>
|
||||
public bool EmissionCapReached { get; private set; }
|
||||
|
||||
/// <summary>
|
||||
/// Latches the arm-time state silently — except when the device is already on battery
|
||||
/// at/below a ladder level, which yields one threshold emission for the lowest crossed
|
||||
/// level (an enrollment STARTING at 12 % must still produce the event the analyze rule
|
||||
/// matches on). No AC-transition emission is ever produced from a baseline.
|
||||
/// </summary>
|
||||
public IReadOnlyList<PowerStateEmission> Baseline(PowerStateResult probe)
|
||||
{
|
||||
if (probe == null || probe.ProbeError != null || !probe.HasBattery)
|
||||
return Array.Empty<PowerStateEmission>();
|
||||
|
||||
_hasState = true;
|
||||
_lastOnAc = probe.OnAcPower;
|
||||
|
||||
var emissions = new List<PowerStateEmission>();
|
||||
EvaluateThresholds(probe, emissions);
|
||||
return emissions;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Diffs a fresh (host-debounced) snapshot against the last accepted state and returns
|
||||
/// 0..n emissions — an AC transition and a threshold crossing can co-occur (unplugged
|
||||
/// at 10 % ⇒ <c>ac_to_battery</c> + <c>threshold_crossed:15</c>).
|
||||
/// </summary>
|
||||
public IReadOnlyList<PowerStateEmission> Evaluate(PowerStateResult probe)
|
||||
{
|
||||
if (probe == null || probe.ProbeError != null)
|
||||
return Array.Empty<PowerStateEmission>();
|
||||
if (!_hasState)
|
||||
return Baseline(probe);
|
||||
|
||||
var emissions = new List<PowerStateEmission>();
|
||||
|
||||
if (_lastOnAc && !probe.OnAcPower)
|
||||
{
|
||||
Emit(emissions, new PowerStateEmission(
|
||||
transition: TransitionAcToBattery,
|
||||
thresholdPercent: null,
|
||||
probe: probe,
|
||||
severity: EventSeverity.Warning,
|
||||
immediateUpload: true,
|
||||
message: $"Power source switched from AC to battery ({FormatPercent(probe)} remaining) during enrollment"));
|
||||
}
|
||||
else if (!_lastOnAc && probe.OnAcPower)
|
||||
{
|
||||
Emit(emissions, new PowerStateEmission(
|
||||
transition: TransitionBatteryToAc,
|
||||
thresholdPercent: null,
|
||||
probe: probe,
|
||||
severity: EventSeverity.Info,
|
||||
immediateUpload: false,
|
||||
message: $"Power source switched back to AC ({FormatPercent(probe)}{(probe.IsCharging ? ", charging" : string.Empty)})"));
|
||||
}
|
||||
|
||||
_lastOnAc = probe.OnAcPower;
|
||||
EvaluateThresholds(probe, emissions);
|
||||
return emissions;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// While on battery with a known percentage: emit the LOWEST ladder level at/above the
|
||||
/// current charge that is not yet latched, then latch every level at/above the charge
|
||||
/// (a 60 %→10 % jump emits threshold 15 only, latching 50/30/15).
|
||||
/// </summary>
|
||||
private void EvaluateThresholds(PowerStateResult probe, List<PowerStateEmission> emissions)
|
||||
{
|
||||
if (probe.OnAcPower || !probe.BatteryPercent.HasValue) return;
|
||||
var percent = probe.BatteryPercent.Value;
|
||||
|
||||
int? lowestCrossed = null;
|
||||
foreach (var level in _ladder) // descending — the last match is the lowest crossed level
|
||||
{
|
||||
if (percent <= level && !_latchedLevels.Contains(level))
|
||||
lowestCrossed = level;
|
||||
}
|
||||
if (!lowestCrossed.HasValue) return;
|
||||
|
||||
foreach (var level in _ladder)
|
||||
{
|
||||
if (percent <= level) _latchedLevels.Add(level);
|
||||
}
|
||||
|
||||
var threshold = lowestCrossed.Value;
|
||||
var severity = threshold <= 15 ? EventSeverity.Error
|
||||
: threshold <= 30 ? EventSeverity.Warning
|
||||
: EventSeverity.Info;
|
||||
Emit(emissions, new PowerStateEmission(
|
||||
transition: TransitionThresholdCrossed,
|
||||
thresholdPercent: threshold,
|
||||
probe: probe,
|
||||
severity: severity,
|
||||
immediateUpload: severity >= EventSeverity.Warning,
|
||||
message: $"Battery dropped below {threshold}% ({FormatPercent(probe)} remaining) while enrolling on battery power"));
|
||||
}
|
||||
|
||||
private void Emit(List<PowerStateEmission> emissions, PowerStateEmission emission)
|
||||
{
|
||||
if (_emitted >= _maxEmissions)
|
||||
{
|
||||
EmissionCapReached = true;
|
||||
return;
|
||||
}
|
||||
_emitted++;
|
||||
emissions.Add(emission);
|
||||
}
|
||||
|
||||
private static string FormatPercent(PowerStateResult probe)
|
||||
=> probe.BatteryPercent.HasValue ? $"battery {probe.BatteryPercent.Value}%" : "battery level unknown";
|
||||
}
|
||||
|
||||
/// <summary>One decided <c>power_state_change</c> emission — the host maps this 1:1 onto an event.</summary>
|
||||
internal sealed class PowerStateEmission
|
||||
{
|
||||
public PowerStateEmission(string transition, int? thresholdPercent, PowerStateResult probe,
|
||||
EventSeverity severity, bool immediateUpload, string message)
|
||||
{
|
||||
Transition = transition;
|
||||
ThresholdPercent = thresholdPercent;
|
||||
OnAcPower = probe.OnAcPower;
|
||||
BatteryPercent = probe.BatteryPercent;
|
||||
IsCharging = probe.IsCharging;
|
||||
BatteryLifeMinutes = probe.BatteryLifeMinutes;
|
||||
Severity = severity;
|
||||
ImmediateUpload = immediateUpload;
|
||||
Message = message;
|
||||
}
|
||||
|
||||
public string Transition { get; }
|
||||
public int? ThresholdPercent { get; }
|
||||
public bool OnAcPower { get; }
|
||||
public int? BatteryPercent { get; }
|
||||
public bool IsCharging { get; }
|
||||
public int? BatteryLifeMinutes { get; }
|
||||
public EventSeverity Severity { get; }
|
||||
public bool ImmediateUpload { get; }
|
||||
public string Message { get; }
|
||||
}
|
||||
}
|
||||
+16
-4
@@ -580,10 +580,22 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
|
||||
var scanText = rawLine;
|
||||
if (CmTraceLogParser.TryParseLine(rawLine, out var entry))
|
||||
{
|
||||
// Provably stale → not progress. (TryParseLine falls back to now on an
|
||||
// unparseable timestamp, which keeps the line fresh — the safe default.)
|
||||
if (entry.Timestamp < cutoffUtc)
|
||||
continue;
|
||||
// Provably stale → not progress. A line without a parseable timestamp counts
|
||||
// as fresh — the safe default, since treating it as stale would hide progress.
|
||||
//
|
||||
// This collector reads a file snapshot rather than tailing it, so it cannot
|
||||
// measure the writer's UTC offset itself and falls back to this process's zone
|
||||
// (see ResolveUtcAssumingReaderZone). That fallback is wrong by exactly the
|
||||
// gap between the two processes' zone beliefs. It is the pre-existing
|
||||
// behaviour, kept deliberately unchanged here; the tracker-owned calibrator is
|
||||
// wired in separately so this commit shifts no behaviour.
|
||||
if (entry.HasTimestamp)
|
||||
{
|
||||
var entryUtc = entry.TimestampUtc
|
||||
?? CmTraceLogParser.ResolveUtcAssumingReaderZone(entry.LocalTimestamp);
|
||||
if (entryUtc < cutoffUtc)
|
||||
continue;
|
||||
}
|
||||
scanText = entry.Message;
|
||||
}
|
||||
|
||||
|
||||
+11
-2
@@ -331,7 +331,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
|
||||
data[groupName] = group.Value;
|
||||
}
|
||||
|
||||
data["logTimestamp"] = entry.Timestamp.ToString("o");
|
||||
// Gather-rule logs are arbitrary files this collector does not
|
||||
// tail, so no writer offset can be measured for them; the
|
||||
// reader-zone fallback carries its known defect (see
|
||||
// ResolveUtcAssumingReaderZone). Behaviour unchanged from before
|
||||
// the parser split.
|
||||
var entryUtc = entry.TimestampUtc
|
||||
?? (entry.HasTimestamp
|
||||
? CmTraceLogParser.ResolveUtcAssumingReaderZone(entry.LocalTimestamp)
|
||||
: DateTime.UtcNow);
|
||||
data["logTimestamp"] = entryUtc.ToString("o");
|
||||
data["logComponent"] = entry.Component;
|
||||
data["logType"] = entry.Type;
|
||||
data["logMessage"] = TruncateMessage(entry.Message, 500);
|
||||
@@ -348,7 +357,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather.Collectors
|
||||
{
|
||||
SessionId = context.SessionId,
|
||||
TenantId = context.TenantId,
|
||||
Timestamp = entry.Timestamp,
|
||||
Timestamp = entryUtc,
|
||||
EventType = eventType,
|
||||
Severity = severity,
|
||||
Source = "GatherRuleExecutor",
|
||||
|
||||
@@ -595,6 +595,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
|
||||
logger: logger,
|
||||
apiBaseUrl: _agentConfig.ApiBaseUrl));
|
||||
|
||||
// Live AC/battery watcher — WMI Win32_PowerManagementEvent push (no polling), emits
|
||||
// power_state_change on AC↔battery transitions and 50/30/15 % downward threshold
|
||||
// crossings. Deliberately NOT under PeriodicCollectorLifecycleHost: it is push-based
|
||||
// (zero idle cost) and must stay alive precisely when the device sits idle on battery.
|
||||
// Never arms on devices without a battery. Kill-switchable via remote config.
|
||||
if (collectors.EnablePowerStateWatcher)
|
||||
{
|
||||
hosts.Add(new PowerStateWatcherHost(
|
||||
sessionId: sessionId,
|
||||
tenantId: tenantId,
|
||||
ingress: ingress,
|
||||
clock: clock,
|
||||
logger: logger));
|
||||
}
|
||||
|
||||
// Microsoft 365 Apps (Office C2R) install detector — event-driven (Rev 2): woken by a WMI
|
||||
// Win32_ProcessStartTrace push on OfficeC2RClient.exe (+ startup probe), progress via
|
||||
// RegNotifyChangeKeyValue, stop via Process.Exited. No idle polling. Constructed first so
|
||||
|
||||
+286
@@ -0,0 +1,286 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Management;
|
||||
using System.Threading;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
|
||||
using AutopilotMonitor.Agent.V2.Core.Runtime;
|
||||
using AutopilotMonitor.DecisionCore.Engine;
|
||||
using AutopilotMonitor.Shared;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Orchestration
|
||||
{
|
||||
/// <summary>
|
||||
/// Live AC/battery watcher: subscribes to the WMI push event
|
||||
/// <c>Win32_PowerManagementEvent WHERE EventCode = 10</c> (PBT_APMPOWERSTATUSCHANGE — fires on
|
||||
/// AC↔DC switches and battery-percentage steps; the event carries no payload, so each arrival
|
||||
/// re-probes via <see cref="PowerStateProbe"/>) and diffs the snapshots through the pure
|
||||
/// <see cref="PowerStateTransitionTracker"/> into <c>power_state_change</c> events. No polling:
|
||||
/// between pushes the host is completely idle. Complements the one-shot startup
|
||||
/// <c>power_state_check</c> (<c>StartupEnvironmentProbes</c>), which stays the baseline event.
|
||||
/// <para>
|
||||
/// Devices without a battery (or with a failing probe) never arm the WMI subscription — zero
|
||||
/// overhead on desktops/VMs. WMI arming runs on a background thread (WinMgmt can hang in OOBE;
|
||||
/// same rationale as <c>ConsoleBypassWatcher</c>), and an arm failure is surfaced once as
|
||||
/// <c>collector_degraded</c>. A 5 s trailing-edge debounce collapses docking flaps to a
|
||||
/// no-diff; the tracker's lifetime emission cap is the storm backstop.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal sealed class PowerStateWatcherHost : ICollectorHost
|
||||
{
|
||||
public string Name => "PowerStateWatcher";
|
||||
|
||||
private static readonly TimeSpan DebounceDelay = TimeSpan.FromSeconds(5);
|
||||
|
||||
private readonly string _sessionId;
|
||||
private readonly string _tenantId;
|
||||
private readonly AgentLogger _logger;
|
||||
private readonly InformationalEventPost _post;
|
||||
private readonly Func<PowerStateResult> _probe;
|
||||
private readonly PowerStateTransitionTracker _tracker = new PowerStateTransitionTracker();
|
||||
private readonly object _lock = new object();
|
||||
private readonly object _tickLock = new object();
|
||||
|
||||
private ManagementEventWatcher? _watcher;
|
||||
private Timer? _debounceTimer;
|
||||
private bool _started;
|
||||
private bool _disposed;
|
||||
private bool _capWarned;
|
||||
|
||||
/// <param name="probe">Power-state snapshot source; test seam, defaults to
|
||||
/// <see cref="PowerStateProbe.Probe"/>. The ctor never calls it (the factory constructs
|
||||
/// hosts eagerly; all probing happens in <see cref="Start"/>'s background thread).</param>
|
||||
public PowerStateWatcherHost(
|
||||
string sessionId,
|
||||
string tenantId,
|
||||
ISignalIngressSink ingress,
|
||||
IClock clock,
|
||||
AgentLogger logger,
|
||||
Func<PowerStateResult>? probe = null)
|
||||
{
|
||||
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
|
||||
if (clock == null) throw new ArgumentNullException(nameof(clock));
|
||||
_sessionId = sessionId;
|
||||
_tenantId = tenantId;
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_post = new InformationalEventPost(ingress, clock, logger);
|
||||
_probe = probe ?? PowerStateProbe.Probe;
|
||||
}
|
||||
|
||||
public void Start()
|
||||
{
|
||||
lock (_lock)
|
||||
{
|
||||
if (_started || _disposed) return;
|
||||
_started = true;
|
||||
}
|
||||
|
||||
// Baseline probe + WMI arming are synchronous WinMgmt-adjacent operations and WinMgmt
|
||||
// can hang during OOBE — never run them inline on the orchestrator's sequential
|
||||
// host-start loop (ConsoleBypassWatcher precedent). Disposal racing the background
|
||||
// start is handled by the _disposed re-checks inside StartCore.
|
||||
System.Threading.Tasks.Task.Run(() => StartCore(armWmi: true));
|
||||
}
|
||||
|
||||
/// <summary>Internal for tests: the synchronous start body; tests pass
|
||||
/// <paramref name="armWmi"/> = false so no real WMI subscription is created
|
||||
/// (precedent: ConsoleBypassWatcher leaves WMI arming untested).</summary>
|
||||
internal void StartCore(bool armWmi)
|
||||
{
|
||||
PowerStateResult baseline;
|
||||
try
|
||||
{
|
||||
baseline = _probe();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Warning($"[{Name}] baseline probe threw ({ex.Message}) — power-state watcher not armed");
|
||||
return;
|
||||
}
|
||||
|
||||
if (baseline.ProbeError != null)
|
||||
{
|
||||
_logger.Info($"[{Name}] power probe failed ({baseline.ProbeError}) — watcher not armed");
|
||||
return;
|
||||
}
|
||||
if (!baseline.HasBattery)
|
||||
{
|
||||
_logger.Info($"[{Name}] no battery present (desktop/VM) — watcher not armed");
|
||||
return;
|
||||
}
|
||||
|
||||
// Latch the baseline BEFORE arming so the first WMI push diffs against a real state.
|
||||
// A session already on battery below a ladder level emits its threshold event here.
|
||||
EmitAll(_tracker.Baseline(baseline));
|
||||
|
||||
if (!armWmi) return;
|
||||
|
||||
try
|
||||
{
|
||||
var query = new WqlEventQuery("SELECT * FROM Win32_PowerManagementEvent WHERE EventCode = 10");
|
||||
var watcher = new ManagementEventWatcher(query);
|
||||
watcher.EventArrived += OnPowerStatusChange;
|
||||
|
||||
lock (_lock)
|
||||
{
|
||||
if (_disposed)
|
||||
{
|
||||
// Dispose won the race against the background start — don't arm.
|
||||
try { watcher.EventArrived -= OnPowerStatusChange; } catch { }
|
||||
try { watcher.Dispose(); } catch { }
|
||||
return;
|
||||
}
|
||||
_watcher = watcher;
|
||||
}
|
||||
|
||||
watcher.Start();
|
||||
|
||||
// Re-check after arming: Dispose() may have run between the lock release above and
|
||||
// Start() — it then stopped a not-yet-armed watcher and this thread armed it right
|
||||
// back. If disposal won that race, tear the watcher down here.
|
||||
bool disposedDuringArm;
|
||||
lock (_lock)
|
||||
{
|
||||
disposedDuringArm = _disposed;
|
||||
if (disposedDuringArm) _watcher = null;
|
||||
}
|
||||
if (disposedDuringArm)
|
||||
{
|
||||
try { watcher.EventArrived -= OnPowerStatusChange; } catch { }
|
||||
try { watcher.Stop(); } catch { }
|
||||
try { watcher.Dispose(); } catch { }
|
||||
return;
|
||||
}
|
||||
|
||||
_logger.Info($"[{Name}] watching Win32_PowerManagementEvent (EventCode 10) — " +
|
||||
$"baseline: {(baseline.OnAcPower ? "AC" : "battery")}, {baseline.BatteryPercent?.ToString() ?? "?"}%");
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Warning($"[{Name}] could not arm WMI power watcher ({ex.GetType().Name}: {ex.Message}) — " +
|
||||
"live power-state transitions will not be observed this run");
|
||||
lock (_lock) { _watcher = null; }
|
||||
CollectorDegradationReporter.Report(_post, _sessionId, _tenantId, Name, "watcher_arm_failed", ex);
|
||||
}
|
||||
}
|
||||
|
||||
private void OnPowerStatusChange(object sender, EventArrivedEventArgs e)
|
||||
{
|
||||
// The WMI event carries no usable payload; it only tells us "power status changed".
|
||||
// Debounce, then re-probe — never call watcher.Stop() from inside this callback.
|
||||
try { ScheduleDebouncedTick(); }
|
||||
catch (Exception ex) { _logger.Debug($"[{Name}] power event handler error: {ex.Message}"); }
|
||||
}
|
||||
|
||||
private void ScheduleDebouncedTick()
|
||||
{
|
||||
lock (_lock) { if (_disposed) return; }
|
||||
// One trailing-edge tick per change burst (ImeRegistryAppStateHost pattern): a
|
||||
// dock-flap AC→battery→AC inside the window collapses to a no-diff in the tracker.
|
||||
var timer = _debounceTimer;
|
||||
if (timer == null)
|
||||
{
|
||||
var created = new Timer(_ => DebouncedTick(), null, DebounceDelay, Timeout.InfiniteTimeSpan);
|
||||
if (Interlocked.CompareExchange(ref _debounceTimer, created, null) != null)
|
||||
created.Dispose();
|
||||
else
|
||||
return;
|
||||
timer = _debounceTimer;
|
||||
}
|
||||
try { timer?.Change(DebounceDelay, Timeout.InfiniteTimeSpan); }
|
||||
catch (ObjectDisposedException) { /* stopping */ }
|
||||
}
|
||||
|
||||
/// <summary>Internal for tests: runs one debounced probe/diff/emit cycle synchronously,
|
||||
/// exactly what the trailing-edge timer callback executes after a WMI push.</summary>
|
||||
internal void TickForTest() => DebouncedTick();
|
||||
|
||||
private void DebouncedTick()
|
||||
{
|
||||
lock (_lock) { if (_disposed) return; }
|
||||
try
|
||||
{
|
||||
IReadOnlyList<PowerStateEmission> emissions;
|
||||
lock (_tickLock) // tracker is not thread-safe; ticks must not interleave
|
||||
{
|
||||
emissions = _tracker.Evaluate(_probe());
|
||||
}
|
||||
EmitAll(emissions);
|
||||
|
||||
if (_tracker.EmissionCapReached && !_capWarned)
|
||||
{
|
||||
_capWarned = true;
|
||||
_logger.Warning($"[{Name}] emission cap reached — further power_state_change events " +
|
||||
"suppressed this run (flapping power source or dying battery controller)");
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Debug($"[{Name}] debounced power tick failed: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
private void EmitAll(IReadOnlyList<PowerStateEmission> emissions)
|
||||
{
|
||||
foreach (var emission in emissions)
|
||||
{
|
||||
try
|
||||
{
|
||||
var data = new Dictionary<string, object>
|
||||
{
|
||||
{ "transition", emission.Transition },
|
||||
{ "onAcPower", emission.OnAcPower },
|
||||
{ "batteryPercent", emission.BatteryPercent.HasValue ? (object)emission.BatteryPercent.Value : "unknown" },
|
||||
{ "isCharging", emission.IsCharging },
|
||||
{ "batteryLifeMinutes", emission.BatteryLifeMinutes.HasValue ? (object)emission.BatteryLifeMinutes.Value : "unknown" },
|
||||
};
|
||||
if (emission.ThresholdPercent.HasValue)
|
||||
data["thresholdPercent"] = emission.ThresholdPercent.Value;
|
||||
|
||||
_post.Emit(new EnrollmentEvent
|
||||
{
|
||||
SessionId = _sessionId,
|
||||
TenantId = _tenantId,
|
||||
EventType = Constants.EventTypes.PowerStateChange,
|
||||
Severity = emission.Severity,
|
||||
Source = Name,
|
||||
Phase = EnrollmentPhase.Unknown,
|
||||
Message = emission.Message,
|
||||
ImmediateUpload = emission.ImmediateUpload,
|
||||
Data = data,
|
||||
});
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.Warning($"[{Name}] failed to emit power_state_change: {ex.Message}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
public void Stop()
|
||||
{
|
||||
ManagementEventWatcher? watcher;
|
||||
lock (_lock)
|
||||
{
|
||||
if (_disposed) return;
|
||||
_disposed = true;
|
||||
watcher = _watcher;
|
||||
_watcher = null;
|
||||
}
|
||||
|
||||
if (watcher != null)
|
||||
{
|
||||
try { watcher.EventArrived -= OnPowerStatusChange; } catch { }
|
||||
try { watcher.Stop(); } catch { }
|
||||
try { watcher.Dispose(); } catch { }
|
||||
}
|
||||
|
||||
_debounceTimer?.Dispose();
|
||||
_debounceTimer = null;
|
||||
}
|
||||
|
||||
public void Dispose() => Stop();
|
||||
}
|
||||
}
|
||||
@@ -56,6 +56,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
|
||||
// real activity (rare, and worth keeping diagnostics running for).
|
||||
SharedConstants.EventTypes.ModernDeploymentLog,
|
||||
|
||||
// Environment observation, not device/enrollment progress: an idle device draining
|
||||
// its battery must not keep the periodic collectors alive or push out stall
|
||||
// detection — the exact scenario the power watcher exists to observe.
|
||||
SharedConstants.EventTypes.PowerStateChange,
|
||||
|
||||
// Agent health / control / transport — not device/enrollment progress.
|
||||
SharedConstants.EventTypes.CollectorDegraded,
|
||||
SharedConstants.EventTypes.TelemetryUploadPoisoned,
|
||||
|
||||
@@ -124,6 +124,17 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
|
||||
// in ImeLogTracker so the layers can never drift apart).
|
||||
private static readonly TimeSpan StaleSourceThreshold = ImeLogTracker.HistoricReplayThreshold;
|
||||
|
||||
/// <summary>
|
||||
/// How far ahead of the agent clock a source timestamp may sit before it is rejected.
|
||||
/// <para>
|
||||
/// Writer and reader share one system clock, so a correctly converted line is always at
|
||||
/// or behind "now"; this is pure jitter tolerance, not a skew allowance. Deliberately far
|
||||
/// below one hour — the value that a timezone mistake produces — so that such a mistake
|
||||
/// can actually trip it.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private static readonly TimeSpan MaxFutureSourceSkew = TimeSpan.FromMinutes(2);
|
||||
|
||||
// Matches the deterministic marker line the bootstrap script writes via Write-Log,
|
||||
// e.g. "Bootstrap script version: v2.0" — see scripts/Bootstrap/Install-AutopilotMonitor.ps1.
|
||||
// Same shape as the web-side regex in utils/bootstrapVersion.ts; kept in sync intentionally.
|
||||
@@ -329,12 +340,28 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
|
||||
|
||||
var clockNow = _clock.UtcNow;
|
||||
var ageHours = (clockNow - asUtc).TotalHours;
|
||||
if (ageHours > StaleSourceThreshold.TotalHours || ageHours < -1.0)
|
||||
|
||||
// Future guard. A source timestamp can never legitimately be ahead of the agent
|
||||
// clock: the writer stamps it from the SAME system clock, and the tracker only
|
||||
// sees the line after it was written. Anything ahead means the local-to-UTC
|
||||
// conversion was wrong.
|
||||
//
|
||||
// The previous threshold was ageHours < -1.0, which could not fire for the single
|
||||
// most common timezone error: with a one-hour offset mistake the real value is
|
||||
// parseLag - 3600 s, i.e. just ABOVE -1.0. It sat exactly on the boundary of the
|
||||
// case it was meant to catch. It was also asymmetric — a past-shifted error had to
|
||||
// exceed 24 h before anything noticed, while the measured field errors reached
|
||||
// -17 h. The bound is now a small jitter tolerance in the direction that is
|
||||
// physically impossible.
|
||||
var isFutureSkewed = asUtc > clockNow + MaxFutureSourceSkew;
|
||||
|
||||
if (ageHours > StaleSourceThreshold.TotalHours || isFutureSkewed)
|
||||
{
|
||||
// Pathologically stale (>24h old) or in the future (>1h skew) — log,
|
||||
// fall back, and let the caller flag the event.
|
||||
var reason = isFutureSkewed
|
||||
? $"ahead of the agent clock by {(asUtc - clockNow).TotalMinutes:F1} min"
|
||||
: $"pathologically stale (ageHours={ageHours:F1})";
|
||||
_logger?.Warning(
|
||||
$"ImeAdapter: source timestamp {asUtc:o} rejected (ageHours={ageHours:F1}); falling back to clock");
|
||||
$"ImeAdapter: source timestamp {asUtc:o} rejected — {reason}; falling back to clock");
|
||||
derivedFromClock = true;
|
||||
return clockNow;
|
||||
}
|
||||
@@ -347,12 +374,69 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
|
||||
return _clock.UtcNow;
|
||||
}
|
||||
|
||||
private static void TagDerivedTimestamp(IDictionary<string, string> data, bool derivedFromClock, DateTime? rawSourceTs)
|
||||
/// <summary>
|
||||
/// Attach the evidence for this event's timestamp: the raw local time as the writer wrote
|
||||
/// it, the offset applied, and how that offset was obtained.
|
||||
///
|
||||
/// <para>
|
||||
/// Without this the only record of a conversion is the converted value itself, so a wrong
|
||||
/// offset is indistinguishable from a correct one. Proving the field defect required the
|
||||
/// diagnostics archive — and of the 5,386 sessions that went through a timezone change,
|
||||
/// essentially none had one; the analysis had to fall back to median statistics across
|
||||
/// thousands of sessions. These three fields make each session answer the question on its
|
||||
/// own, and make a stored timestamp recomputable after the fact without an agent release.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// Only emitted for events whose time came from a CMTrace line. Agent-native events have
|
||||
/// no second clock frame, so there is nothing to record.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private void TagDerivedTimestamp(IDictionary<string, string> data, bool derivedFromClock, DateTime? rawSourceTs)
|
||||
{
|
||||
var culture = System.Globalization.CultureInfo.InvariantCulture;
|
||||
|
||||
var sourceLocal = _tracker.LastMatchedSourceLocalTimestamp;
|
||||
if (sourceLocal.HasValue)
|
||||
{
|
||||
// No zone suffix: the value genuinely carries none, and formatting one in would
|
||||
// re-introduce the very assumption this whole change removes.
|
||||
data["sourceLocalTs"] = sourceLocal.Value.ToString("yyyy-MM-ddTHH:mm:ss.fffffff", culture);
|
||||
data["sourceOffsetOrigin"] = OriginTag(_tracker.LastMatchedSourceOffsetOrigin);
|
||||
|
||||
var offsetMinutes = _tracker.LastMatchedSourceOffsetMinutes;
|
||||
if (offsetMinutes.HasValue)
|
||||
data["sourceOffsetMinutes"] = offsetMinutes.Value.ToString(culture);
|
||||
|
||||
// Observational only — what the calibrator measured, which since the 2026-08-20
|
||||
// revert is deliberately NOT what was applied. Kept separate so the two can never
|
||||
// be read as the same thing.
|
||||
var measured = _tracker.LastMatchedMeasuredWriterOffsetMinutes;
|
||||
if (measured.HasValue)
|
||||
data["measuredWriterOffsetMinutes"] = measured.Value.ToString(culture);
|
||||
}
|
||||
|
||||
if (!derivedFromClock) return;
|
||||
data["derivedTimestamp"] = "true";
|
||||
if (rawSourceTs.HasValue)
|
||||
data["rejectedSourceTimestamp"] = rawSourceTs.Value.ToString("o", System.Globalization.CultureInfo.InvariantCulture);
|
||||
data["rejectedSourceTimestamp"] = rawSourceTs.Value.ToString("o", culture);
|
||||
}
|
||||
|
||||
private static string OriginTag(CmTraceOffsetOrigin origin)
|
||||
{
|
||||
switch (origin)
|
||||
{
|
||||
case CmTraceOffsetOrigin.Bias: return "bias";
|
||||
// Per-FILE offset application, retired by the 2026-08-20 revert (04b1a7c6).
|
||||
// Kept so stored 2.0.1410 events remain interpretable; no current path emits it.
|
||||
case CmTraceOffsetOrigin.Calibrated: return "calibrated";
|
||||
// The line anchored itself: read provably fresh, its own distance to the agent
|
||||
// clock (grid-rounded) was applied as the writer's offset. Era-safe per line.
|
||||
case CmTraceOffsetOrigin.LineAnchored: return "line-anchored";
|
||||
// The line was not provably fresh (backlog, restart catch-up, replay), so this
|
||||
// process's own zone was assumed — correct only if the writer happens to share it.
|
||||
default: return "reader-zone-fallback";
|
||||
}
|
||||
}
|
||||
|
||||
// CMTrace Kind normalization — shared implementation lives on the tracker (used by its
|
||||
@@ -720,12 +804,15 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
|
||||
/// <see cref="AppInstallTiming"/> so the emitter can inline the values on the
|
||||
/// current event's payload without a second dictionary lookup.
|
||||
/// <para>
|
||||
/// Audit Q1: a terminal→active transition (the IME retry path — <see
|
||||
/// cref="AppPackageState.UpdateState"/> permits e.g. Error→Downloading and
|
||||
/// Skipped→Downloading) re-arms <c>Completed</c> to <c>null</c>, so the NEXT terminal
|
||||
/// transition stamps a fresh CompletedAt. StartedAt keeps its first value — the final
|
||||
/// terminal payload then covers first-start → last-terminal (total occupancy across
|
||||
/// attempts) instead of freezing at the first failure. Observed pre-fix: a user-scope
|
||||
/// Audit Q1 + 2026-08 attempt-duration change: a terminal→active transition (the IME
|
||||
/// retry / re-evaluation path — <see cref="AppPackageState.UpdateState"/> permits e.g.
|
||||
/// Error→Downloading and Skipped→Downloading) re-arms BOTH endpoints: <c>Completed</c>
|
||||
/// to <c>null</c> (the NEXT terminal transition stamps fresh) and <c>Started</c> to
|
||||
/// the re-entry time. The final terminal payload therefore describes the
|
||||
/// status-defining ATTEMPT — matching the backend's AppInstallSummaries semantics —
|
||||
/// instead of first-start → last-terminal across passes: the IME processes the app
|
||||
/// list repeatedly (device-ESP evaluation pass, user-context pass), and the idle gap
|
||||
/// between passes is not install time. Observed pre-fix (old bug shape): a user-scope
|
||||
/// app Skipped early in DeviceSetup, retried after sign-in, emitted its final
|
||||
/// Installed event with the stale early CompletedAt (durationSeconds=-2342).
|
||||
/// </para>
|
||||
@@ -736,11 +823,13 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
|
||||
{
|
||||
_appTimings.TryGetValue(appId, out var current);
|
||||
|
||||
// Terminal→active (retry): the earlier terminal stamp no longer describes this
|
||||
// app's outcome — re-arm so the next terminal transition stamps fresh.
|
||||
// Terminal→active (retry / new pass): the earlier stamps no longer describe
|
||||
// this app's outcome — re-arm both endpoints so the next terminal transition
|
||||
// reports the fresh attempt's window.
|
||||
if (current.Completed != null && IsLifecycleActive(newState))
|
||||
{
|
||||
current.Completed = null;
|
||||
current.Started = now;
|
||||
}
|
||||
|
||||
// First lifecycle-active transition → record StartedAt.
|
||||
|
||||
+6
@@ -119,6 +119,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Transport.Telemetry
|
||||
AddSecurityHeaders(request);
|
||||
request.Headers.Add("X-Correlation-ID", correlationId);
|
||||
|
||||
// P14: device-clock send time, one value per request. Lets the backend separate
|
||||
// spool delay (SentAt − OccurredUtc, both device frame) from device-vs-server
|
||||
// clock offset (ReceivedAt − SentAt) — indistinguishable without this header.
|
||||
// Stamped per attempt on purpose: a retry after backoff is a NEW send moment.
|
||||
request.Headers.Add("X-Send-Time-Utc", DateTime.UtcNow.ToString("O"));
|
||||
|
||||
HttpResponseMessage response;
|
||||
try
|
||||
{
|
||||
|
||||
@@ -372,6 +372,9 @@ namespace AutopilotMonitor.DecisionCore.Tests
|
||||
// Before RealmJoinPhaseChanged became a typed signal, LastDeploymentPhase was only
|
||||
// written at detection time — every realmjoin_timeout claimed "last phase: 0" no
|
||||
// matter how far RJ actually got, hiding this whole failure class from ops.
|
||||
// Since the activity-based extension, the PhaseChanged at T0+6 counts as activity,
|
||||
// so the first fire (59 min later) extends once; the timeout lands on the re-armed
|
||||
// fire a full inactivity window after the last activity.
|
||||
var engine = new DecisionEngine();
|
||||
var state = PrimeClassicAwaitingDesktop(engine);
|
||||
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
|
||||
@@ -384,7 +387,13 @@ namespace AutopilotMonitor.DecisionCore.Tests
|
||||
})).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.DesktopArrived, T0.AddMinutes(7))).NewState;
|
||||
|
||||
var step = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, T0.AddMinutes(65),
|
||||
// First fire at the original due (T0+65): activity 59 min ago → extends to T0+66.
|
||||
state = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, T0.AddMinutes(65),
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout })).NewState;
|
||||
var rearmed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
Assert.Equal(T0.AddMinutes(66), rearmed.DueAtUtc);
|
||||
|
||||
var step = engine.Reduce(state, MakeSignal(9, DecisionSignalKind.DeadlineFired, rearmed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
|
||||
|
||||
var timeout = Assert.Single(step.Effects, e =>
|
||||
@@ -395,6 +404,248 @@ namespace AutopilotMonitor.DecisionCore.Tests
|
||||
Assert.Equal("101", timeout.Parameters["lastSeenPhase"]);
|
||||
}
|
||||
|
||||
// ============================================================== Activity-based extension (report 55e6afd61c9d)
|
||||
|
||||
[Fact]
|
||||
public void Timeout_fire_with_recent_first_deployment_activity_extends_instead_of_timing_out()
|
||||
{
|
||||
// Report 55e6afd61c9d (Douglas): timer armed at detection (phase 0, RJ agent MSI
|
||||
// install during DeviceSetup), first deployment only started 16 min later, Office
|
||||
// completed 3 s before the deadline — the hard cut truncated an actively working
|
||||
// deployment. The fire must now re-arm to lastActivity + 60 min instead.
|
||||
var engine = new DecisionEngine();
|
||||
var state = PrimeClassicAwaitingDesktop(engine);
|
||||
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
|
||||
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "0" })).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(21),
|
||||
new Dictionary<string, string>
|
||||
{
|
||||
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
|
||||
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "0",
|
||||
})).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.RealmJoinPackageCompleted, T0.AddMinutes(62),
|
||||
new Dictionary<string, string>
|
||||
{
|
||||
[DecisionEngine.RealmJoinPayloadKeys.PackageId] = "generic-microsoft-office-2016-proplus",
|
||||
[DecisionEngine.RealmJoinPayloadKeys.Scope] = RealmJoinPackageFact.ScopeMachine,
|
||||
[DecisionEngine.RealmJoinPayloadKeys.Success] = "true",
|
||||
[DecisionEngine.RealmJoinPayloadKeys.LastExitCode] = "0",
|
||||
})).NewState;
|
||||
|
||||
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
var step = engine.Reduce(state, MakeSignal(9, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
|
||||
|
||||
// No completion, no outcome — the gate stays closed and the window is re-armed.
|
||||
Assert.NotEqual(SessionStage.Finalizing, step.NewState.Stage);
|
||||
Assert.Null(step.NewState.RealmJoinFacts.Outcome);
|
||||
Assert.Equal($"DeadlineFired:{DeadlineNames.RealmJoinTimeout}:Extended", step.Transition.Trigger);
|
||||
|
||||
var rearmed = Assert.Single(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
Assert.Equal(T0.AddMinutes(62 + 60), rearmed.DueAtUtc); // lastActivity + inactivity window
|
||||
|
||||
Assert.Single(step.Effects, e =>
|
||||
e.Kind == DecisionEffectKind.ScheduleDeadline &&
|
||||
e.Deadline != null &&
|
||||
e.Deadline.Name == DeadlineNames.RealmJoinTimeout &&
|
||||
e.Deadline.DueAtUtc == rearmed.DueAtUtc);
|
||||
|
||||
var extended = Assert.Single(step.Effects, e =>
|
||||
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
|
||||
e.Parameters != null &&
|
||||
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout_extended");
|
||||
Assert.Equal("Info", extended.Parameters!["severity"]);
|
||||
Assert.Equal("101", extended.Parameters["deploymentPhase"]);
|
||||
Assert.DoesNotContain(step.Effects, e =>
|
||||
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
|
||||
e.Parameters != null &&
|
||||
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resolved_after_extension_completes_via_classic_path()
|
||||
{
|
||||
var engine = new DecisionEngine();
|
||||
var state = PrimeClassicAwaitingDesktop(engine);
|
||||
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
|
||||
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(40),
|
||||
new Dictionary<string, string>
|
||||
{
|
||||
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "100",
|
||||
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "101",
|
||||
})).NewState;
|
||||
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
state = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout })).NewState;
|
||||
Assert.Null(state.RealmJoinFacts.Outcome); // extended, not timed out
|
||||
|
||||
var step = engine.Reduce(state, MakeSignal(9, DecisionSignalKind.RealmJoinResolved, T0.AddMinutes(80),
|
||||
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "110" }));
|
||||
|
||||
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
|
||||
Assert.Equal("Resolved", step.NewState.RealmJoinFacts.Outcome!.Value);
|
||||
Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Rearmed_fire_after_quiet_window_times_out_with_inactivity_reason()
|
||||
{
|
||||
var engine = new DecisionEngine();
|
||||
var state = PrimeClassicAwaitingDesktop(engine);
|
||||
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
|
||||
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "0" })).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(30),
|
||||
new Dictionary<string, string>
|
||||
{
|
||||
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
|
||||
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "0",
|
||||
})).NewState;
|
||||
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
state = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout })).NewState;
|
||||
var rearmed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
Assert.Equal(T0.AddMinutes(90), rearmed.DueAtUtc); // lastActivity (T0+30) + 60 min
|
||||
|
||||
// No further activity — the re-armed fire must now time out for real.
|
||||
var step = engine.Reduce(state, MakeSignal(9, DecisionSignalKind.DeadlineFired, rearmed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
|
||||
|
||||
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
|
||||
Assert.Equal("Timeout", step.NewState.RealmJoinFacts.Outcome!.Value);
|
||||
var timeout = Assert.Single(step.Effects, e =>
|
||||
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
|
||||
e.Parameters != null &&
|
||||
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
|
||||
Assert.Equal("inactivity", timeout.Parameters!["reason"]);
|
||||
Assert.Contains("no deployment activity", timeout.Parameters["message"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Extension_is_capped_at_absolute_ceiling_and_cap_fire_times_out_despite_activity()
|
||||
{
|
||||
// Late activity pushes the sliding window beyond detection + 4 h — the re-arm must
|
||||
// clamp to the cap, and the cap fire must time out even though activity is recent.
|
||||
var engine = new DecisionEngine();
|
||||
var state = PrimeClassicAwaitingDesktop(engine);
|
||||
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
|
||||
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
|
||||
// Package completion whose sliding window (T0+230+60) would exceed the cap (T0+245).
|
||||
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPackageCompleted, T0.AddMinutes(230),
|
||||
new Dictionary<string, string>
|
||||
{
|
||||
[DecisionEngine.RealmJoinPayloadKeys.PackageId] = "generic-big-suite",
|
||||
[DecisionEngine.RealmJoinPayloadKeys.Scope] = RealmJoinPackageFact.ScopeMachine,
|
||||
[DecisionEngine.RealmJoinPayloadKeys.Success] = "true",
|
||||
[DecisionEngine.RealmJoinPayloadKeys.LastExitCode] = "0",
|
||||
})).NewState;
|
||||
|
||||
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
state = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout })).NewState;
|
||||
|
||||
var rearmed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
Assert.Equal(T0.AddMinutes(5).AddHours(4), rearmed.DueAtUtc); // clamped to detection + cap
|
||||
|
||||
var step = engine.Reduce(state, MakeSignal(9, DecisionSignalKind.DeadlineFired, rearmed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
|
||||
|
||||
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
|
||||
Assert.Equal("Timeout", step.NewState.RealmJoinFacts.Outcome!.Value);
|
||||
var timeout = Assert.Single(step.Effects, e =>
|
||||
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
|
||||
e.Parameters != null &&
|
||||
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
|
||||
Assert.Equal("absolute_cap", timeout.Parameters!["reason"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Stale_fire_of_old_incarnation_after_rearm_is_dead_end()
|
||||
{
|
||||
// Race: the OLD deadline incarnation's fire was already queued when the extension
|
||||
// re-armed. The armed deadline is due LATER than the stale fire — dead-end, no
|
||||
// second extension evaluation, no timeout.
|
||||
var engine = new DecisionEngine();
|
||||
var state = PrimeClassicAwaitingDesktop(engine);
|
||||
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
|
||||
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "0" })).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(30),
|
||||
new Dictionary<string, string>
|
||||
{
|
||||
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
|
||||
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "0",
|
||||
})).NewState;
|
||||
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout })).NewState;
|
||||
Assert.Null(state.RealmJoinFacts.Outcome); // extended
|
||||
|
||||
var step = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
|
||||
|
||||
Assert.False(step.Transition.Taken);
|
||||
Assert.Null(step.NewState.RealmJoinFacts.Outcome);
|
||||
Assert.Empty(step.Effects);
|
||||
var stillArmed = Assert.Single(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
Assert.Equal(T0.AddMinutes(90), stillArmed.DueAtUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Detection_phase_alone_is_not_activity_idle_case_times_out_at_60_min_unchanged()
|
||||
{
|
||||
// The phase captured AT detection must not count as activity — RJ detected in a
|
||||
// first-deployment phase that then never moves still times out after the original
|
||||
// 60 min (the extension exists for demonstrable progress, not for standing still).
|
||||
var engine = new DecisionEngine();
|
||||
var state = PrimeClassicAwaitingDesktop(engine);
|
||||
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
|
||||
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
|
||||
Assert.Null(state.RealmJoinFacts.LastActivityUtc);
|
||||
|
||||
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
|
||||
|
||||
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
|
||||
Assert.Equal("Timeout", step.NewState.RealmJoinFacts.Outcome!.Value);
|
||||
var timeout = Assert.Single(step.Effects, e =>
|
||||
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
|
||||
e.Parameters != null &&
|
||||
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
|
||||
Assert.Equal("hard_timeout", timeout.Parameters!["reason"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Regular_deployment_phase_with_activity_does_not_extend()
|
||||
{
|
||||
// Session 6f1959c0 shape: RJ stood at 210 when the agent booted and deploys
|
||||
// regular (non-first) packages afterwards. Phase 200/210 is outside the
|
||||
// first-deployment window — activity there must NOT hold the session hostage.
|
||||
var engine = new DecisionEngine();
|
||||
var state = PrimeClassicAwaitingDesktop(engine);
|
||||
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
|
||||
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "210" })).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
|
||||
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(40),
|
||||
new Dictionary<string, string>
|
||||
{
|
||||
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
|
||||
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "210",
|
||||
})).NewState;
|
||||
|
||||
var armed = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
|
||||
var step = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, armed.DueAtUtc,
|
||||
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
|
||||
|
||||
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
|
||||
Assert.Equal("Timeout", step.NewState.RealmJoinFacts.Outcome!.Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SelfDeploying_deferred_terminal_released_by_first_deployment_incomplete()
|
||||
{
|
||||
|
||||
+160
@@ -198,4 +198,164 @@ public class AppInstallTerminalStateAndReconcileTests
|
||||
// Column absent → Merge-mode preserves a prior batch's terminal state.
|
||||
Assert.False(empty.ContainsKey("TerminalState"));
|
||||
}
|
||||
|
||||
// ── attempt-duration semantics (2026-08): the field case that motivated it ──
|
||||
// Session 44862cd3, "Set TimeZone": device-ESP evaluation pass ends Skipped, the real
|
||||
// install happens ~66 min later in the user-context pass and takes 23 s. The old span
|
||||
// semantics reported 4002 s.
|
||||
|
||||
[Fact]
|
||||
public void TwoPasses_SingleBatch_DurationIsLastAttempt_NotSpan()
|
||||
{
|
||||
var s = Aggregate(
|
||||
AppEvent("app_install_started", T0),
|
||||
AppEvent("app_install_completed", T0.AddMinutes(6), "Skipped"),
|
||||
AppEvent("app_install_started", T0.AddMinutes(66)),
|
||||
AppEvent("app_install_completed", T0.AddMinutes(66).AddSeconds(23), "Installed"))["App"].Summary;
|
||||
|
||||
Assert.Equal(23, s.DurationSeconds); // last attempt, not 3983 s
|
||||
Assert.Equal(T0, s.StartedAt); // first-seen anchor untouched
|
||||
Assert.Equal(T0.AddMinutes(66), s.LastAttemptStartedAt);
|
||||
Assert.Equal(2, s.InstallPassCount);
|
||||
Assert.Equal("Installed", s.TerminalState);
|
||||
Assert.Equal(T0.AddMinutes(66).AddSeconds(23), s.CompletedAt);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WeakerTerminal_CompletedSkipped_NeverOverridesInstalled_InBatch()
|
||||
{
|
||||
// Re-evaluation after a real install reports Skipped (already present) — the row
|
||||
// keeps describing the install attempt, incl. CompletedAt and duration.
|
||||
var s = Aggregate(
|
||||
AppEvent("app_install_started", T0),
|
||||
AppEvent("app_install_completed", T0.AddSeconds(40), "Installed"),
|
||||
AppEvent("app_install_started", T0.AddMinutes(30)),
|
||||
AppEvent("app_install_completed", T0.AddMinutes(30).AddSeconds(2), "Skipped"))["App"].Summary;
|
||||
|
||||
Assert.Equal("Installed", s.TerminalState);
|
||||
Assert.Equal(T0.AddSeconds(40), s.CompletedAt);
|
||||
Assert.Equal(40, s.DurationSeconds);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StartInOneBatch_TerminalInNext_ReconcileAnchorsOnStoredAttempt()
|
||||
{
|
||||
// Pass 2 start landed in batch N (stored anchor); the terminal-only batch N+1 must
|
||||
// pair against it — not against the first-seen StartedAt an hour earlier.
|
||||
var existing = new TableEntity("tenant", "session_App")
|
||||
{
|
||||
["Status"] = "InProgress",
|
||||
["StartedAt"] = new DateTimeOffset(T0),
|
||||
["LastAttemptStartedAt"] = new DateTimeOffset(T0.AddMinutes(66)),
|
||||
["InstallPassCount"] = 2,
|
||||
};
|
||||
var summary = new AppInstallSummary
|
||||
{
|
||||
AppName = "App", SessionId = "session", TenantId = "tenant",
|
||||
Status = "Succeeded", TerminalState = "Installed",
|
||||
StartedAt = T0.AddMinutes(66).AddSeconds(23), // batch init stamp (terminal event)
|
||||
CompletedAt = T0.AddMinutes(66).AddSeconds(23),
|
||||
};
|
||||
|
||||
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
|
||||
|
||||
Assert.Equal(T0, summary.StartedAt); // earliest wins (window key)
|
||||
Assert.Equal(T0.AddMinutes(66), summary.LastAttemptStartedAt);
|
||||
Assert.Equal(23, summary.DurationSeconds); // attempt-anchored, not 3983 s
|
||||
Assert.Equal(2, summary.InstallPassCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reconcile_ReplayedOlderBatch_NeverRegressesAnchorOrDoubleCountsPasses()
|
||||
{
|
||||
var existing = new TableEntity("tenant", "session_App")
|
||||
{
|
||||
["StartedAt"] = new DateTimeOffset(T0),
|
||||
["LastAttemptStartedAt"] = new DateTimeOffset(T0.AddMinutes(66)),
|
||||
["InstallPassCount"] = 2,
|
||||
};
|
||||
// Replay of the pass-1 batch: its newest start is OLDER than the stored anchor.
|
||||
var summary = new AppInstallSummary
|
||||
{
|
||||
AppName = "App", Status = "InProgress",
|
||||
StartedAt = T0, LastAttemptStartedAt = T0, InstallPassCount = 1,
|
||||
};
|
||||
|
||||
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
|
||||
|
||||
Assert.Equal(T0.AddMinutes(66), summary.LastAttemptStartedAt); // max-fold: no regression
|
||||
Assert.Equal(2, summary.InstallPassCount); // not 3 — replay not re-counted
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reconcile_NewerAttemptBatch_AddsPassCount()
|
||||
{
|
||||
var existing = new TableEntity("tenant", "session_App")
|
||||
{
|
||||
["StartedAt"] = new DateTimeOffset(T0),
|
||||
["LastAttemptStartedAt"] = new DateTimeOffset(T0),
|
||||
["InstallPassCount"] = 1,
|
||||
};
|
||||
var summary = new AppInstallSummary
|
||||
{
|
||||
AppName = "App", Status = "InProgress",
|
||||
StartedAt = T0.AddMinutes(66), LastAttemptStartedAt = T0.AddMinutes(66), InstallPassCount = 1,
|
||||
};
|
||||
|
||||
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
|
||||
|
||||
Assert.Equal(T0.AddMinutes(66), summary.LastAttemptStartedAt);
|
||||
Assert.Equal(2, summary.InstallPassCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Reconcile_WeakerTerminalBatch_NeverOverridesStoredInstalled()
|
||||
{
|
||||
// Cross-batch mirror of the in-batch guard: a batch that only saw the Skipped
|
||||
// re-evaluation must not re-describe an Installed row.
|
||||
var existing = new TableEntity("tenant", "session_App")
|
||||
{
|
||||
["Status"] = "Succeeded",
|
||||
["TerminalState"] = "Installed",
|
||||
["StartedAt"] = new DateTimeOffset(T0),
|
||||
["CompletedAt"] = new DateTimeOffset(T0.AddSeconds(40)),
|
||||
["DurationSeconds"] = 40,
|
||||
};
|
||||
var summary = new AppInstallSummary
|
||||
{
|
||||
AppName = "App", Status = "Succeeded", TerminalState = "Skipped",
|
||||
StartedAt = T0.AddMinutes(30),
|
||||
LastAttemptStartedAt = T0.AddMinutes(30), InstallPassCount = 1,
|
||||
CompletedAt = T0.AddMinutes(30).AddSeconds(2), DurationSeconds = 2,
|
||||
};
|
||||
|
||||
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
|
||||
|
||||
Assert.Equal("Installed", summary.TerminalState);
|
||||
// The guard drops the skip-pass endpoints; the Q4 path then re-adopts the STORED
|
||||
// CompletedAt (the surviving install attempt's) — same value as on disk, and the
|
||||
// attempt anchor being newer keeps DurationSeconds at 0, so the omitted column
|
||||
// lets Merge preserve the stored 40 s.
|
||||
Assert.Equal(T0.AddSeconds(40), summary.CompletedAt);
|
||||
Assert.Equal(0, summary.DurationSeconds);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EntityBuilder_AttemptColumns_AreSentinelGated()
|
||||
{
|
||||
var with = new AppInstallSummary
|
||||
{
|
||||
AppName = "App",
|
||||
LastAttemptStartedAt = T0,
|
||||
InstallPassCount = 2,
|
||||
};
|
||||
var entity = TableStorageService.BuildAppInstallSummaryEntity(with, "rk");
|
||||
Assert.Equal(new DateTimeOffset(T0), entity.GetDateTimeOffset("LastAttemptStartedAt"));
|
||||
Assert.Equal(2, entity.GetInt32("InstallPassCount"));
|
||||
|
||||
var without = new AppInstallSummary { AppName = "App" };
|
||||
var empty = TableStorageService.BuildAppInstallSummaryEntity(without, "rk");
|
||||
Assert.False(empty.ContainsKey("LastAttemptStartedAt"));
|
||||
Assert.False(empty.ContainsKey("InstallPassCount"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,7 +36,8 @@ public class AppVersionRegressionRadarTests
|
||||
string appName = App,
|
||||
string status = "Succeeded",
|
||||
string terminalState = "Installed",
|
||||
bool collision = false)
|
||||
bool collision = false,
|
||||
bool attemptAnchored = true)
|
||||
=> new()
|
||||
{
|
||||
TenantId = TenantA,
|
||||
@@ -47,6 +48,9 @@ public class AppVersionRegressionRadarTests
|
||||
TerminalState = terminalState,
|
||||
DurationSeconds = durationSeconds,
|
||||
StartedAt = startedAt ?? Now.AddDays(-1),
|
||||
// Post-cutover rows carry the attempt anchor; the radar's cutover gate
|
||||
// excludes rows without it (attemptAnchored: false = legacy span row).
|
||||
LastAttemptStartedAt = attemptAnchored ? (startedAt ?? Now.AddDays(-1)) : null,
|
||||
AppIdCollision = collision,
|
||||
};
|
||||
|
||||
@@ -164,6 +168,26 @@ public class AppVersionRegressionRadarTests
|
||||
Assert.Equal(900, finding.CurrentMedianSeconds);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Evaluate_CutoverGate_LegacySpanRows_ExcludedOnBothSides()
|
||||
{
|
||||
// 2026-08 attempt-duration cutover: rows without LastAttemptStartedAt carry the old
|
||||
// full-span semantics and are excluded from BOTH comparison sides.
|
||||
|
||||
// Current side: 10 legacy span rows would fire (lift 3.0), but only 5 are
|
||||
// attempt-anchored → below MinMeasuredInstalls → silent.
|
||||
var rows = Batch("1.0", 10, 300, Now.AddDays(-20));
|
||||
rows.AddRange(Enumerable.Range(0, 10).Select(_ => Install("2.0", 900, Now.AddDays(-5), attemptAnchored: false)));
|
||||
rows.AddRange(Enumerable.Range(0, 5).Select(_ => Install("2.0", 900, Now.AddDays(-5))));
|
||||
Assert.Empty(AppVersionRegressionRadar.Evaluate(rows));
|
||||
|
||||
// Previous side entirely legacy → gated away → no comparison basis → silent,
|
||||
// even though the raw medians (300 → 900) would scream regression.
|
||||
var rows2 = Enumerable.Range(0, 10).Select(_ => Install("1.0", 300, Now.AddDays(-20), attemptAnchored: false)).ToList();
|
||||
rows2.AddRange(Batch("2.0", 10, 900, Now.AddDays(-5)));
|
||||
Assert.Empty(AppVersionRegressionRadar.Evaluate(rows2));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Evaluate_DeterministicOrdering_LiftDesc_ThenAppNameOrdinal()
|
||||
{
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins the shipped rule catalogs to the reserved built-in ID namespace from
|
||||
/// RuleIdPolicy: every built-in occupies (ANALYZE|GATHER)-<CATEGORY>-<NUMBER> with
|
||||
/// CATEGORY != CUSTOM, and IDs are unique (case-insensitively — case variants are
|
||||
/// indistinguishable to humans and to the policy regex). This is the backend-side
|
||||
/// enforcement of the "platform never ships a built-in in the CUSTOM namespace"
|
||||
/// commitment documented in RuleIdPolicy — without it, a shipped CUSTOM-ID rule
|
||||
/// would silently shadow every tenant's same-ID custom rule at merge time
|
||||
/// (global wins, tenant copy dropped). combine.js mirrors these checks at
|
||||
/// catalog-build time; this test guards the embedded resource actually deployed.
|
||||
/// </summary>
|
||||
public class BuiltInRuleCatalogPolicyTests
|
||||
{
|
||||
[Fact]
|
||||
public void BuiltIn_analyze_ids_are_reserved_namespace_and_unique()
|
||||
{
|
||||
var ids = BuiltInAnalyzeRules.GetAll().Select(r => r.RuleId).ToList();
|
||||
|
||||
Assert.NotEmpty(ids);
|
||||
foreach (var id in ids)
|
||||
{
|
||||
Assert.True(RuleIdPolicy.IsReservedBuiltInId(id),
|
||||
$"Built-in analyze rule '{id}' lies outside the reserved built-in namespace — " +
|
||||
"it would collide with the tenant custom namespace.");
|
||||
}
|
||||
|
||||
var duplicates = ids.GroupBy(id => id, StringComparer.OrdinalIgnoreCase)
|
||||
.Where(g => g.Count() > 1).Select(g => g.Key).ToList();
|
||||
Assert.True(duplicates.Count == 0,
|
||||
$"Duplicate built-in analyze rule IDs: {string.Join(", ", duplicates)}");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuiltIn_gather_ids_are_reserved_namespace_and_unique()
|
||||
{
|
||||
var ids = BuiltInGatherRules.GetAll().Select(r => r.RuleId).ToList();
|
||||
|
||||
Assert.NotEmpty(ids);
|
||||
foreach (var id in ids)
|
||||
{
|
||||
Assert.True(RuleIdPolicy.IsReservedBuiltInId(id),
|
||||
$"Built-in gather rule '{id}' lies outside the reserved built-in namespace — " +
|
||||
"it would collide with the tenant custom namespace.");
|
||||
}
|
||||
|
||||
var duplicates = ids.GroupBy(id => id, StringComparer.OrdinalIgnoreCase)
|
||||
.Where(g => g.Count() > 1).Select(g => g.Key).ToList();
|
||||
Assert.True(duplicates.Count == 0,
|
||||
$"Duplicate built-in gather rule IDs: {string.Join(", ", duplicates)}");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,152 @@
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins the CMTrace time-skew regression tripwire (CmTraceSkewTripwire.Evaluate): fires only
|
||||
/// when the IME-vs-other median delta divergence sits on the 15-minute offset grid (residual
|
||||
/// < 2 min) AND both sides have enough samples over enough distinct upload batches. The
|
||||
/// tripwire's goal state is to never fire, so the must-stay-silent directions are the primary
|
||||
/// cases: off-grid replays, sub-grid divergence, thin samples, batch-composition artifacts,
|
||||
/// and common-mode spool latency must all pass without a hit.
|
||||
/// </summary>
|
||||
public class CmTraceSkewTripwireTests
|
||||
{
|
||||
/// <summary>Scan with the given per-side medians, comfortably above all sample/batch minimums.</summary>
|
||||
private static SessionSkewScan MakeScan(
|
||||
double imeDeltaMinutes, double otherDeltaMinutes,
|
||||
int imeSamples = 30, int otherSamples = 30,
|
||||
int imeBatches = 5, int otherBatches = 5)
|
||||
{
|
||||
var scan = new SessionSkewScan
|
||||
{
|
||||
ImeDistinctBatchCount = imeBatches,
|
||||
OtherDistinctBatchCount = otherBatches,
|
||||
};
|
||||
for (int i = 0; i < imeSamples; i++)
|
||||
scan.ImeDeltaMinutes.Add(imeDeltaMinutes);
|
||||
for (int i = 0; i < otherSamples; i++)
|
||||
scan.OtherDeltaMinutes.Add(otherDeltaMinutes);
|
||||
return scan;
|
||||
}
|
||||
|
||||
// ── Must fire ──────────────────────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public void Fires_OnLargeNegativeGridDivergence()
|
||||
{
|
||||
// −540 min = the field-measured E.-Australia-style skew (−9 h), exactly 36 grid steps.
|
||||
// Both sides share 1.2 min upload latency — it cancels in the difference.
|
||||
var result = CmTraceSkewTripwire.Evaluate(MakeScan(imeDeltaMinutes: -538.8, otherDeltaMinutes: 1.2));
|
||||
|
||||
Assert.NotNull(result);
|
||||
Assert.Equal(-36, result!.GridSteps);
|
||||
Assert.Equal(-540.0, result.DiffMinutes, 3);
|
||||
Assert.True(result.ResidualMinutes < CmTraceSkewTripwire.ResidualToleranceMinutes);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Fires_OnSingleGridStepWithBoundaryResidual()
|
||||
{
|
||||
// +15 min divergence with residual 1.9 — just inside the 2-min tolerance.
|
||||
var result = CmTraceSkewTripwire.Evaluate(MakeScan(imeDeltaMinutes: 16.9, otherDeltaMinutes: 0.0));
|
||||
|
||||
Assert.NotNull(result);
|
||||
Assert.Equal(1, result!.GridSteps);
|
||||
Assert.Equal(1.9, result.ResidualMinutes, 3);
|
||||
}
|
||||
|
||||
// ── Must stay silent ───────────────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public void Silent_WhenResidualOffGrid()
|
||||
{
|
||||
// 17.5 min divergence: nearest grid step is 15, residual 2.5 ≥ tolerance — the replay
|
||||
// signature (old log lines re-read late land anywhere, not on the zone grid).
|
||||
Assert.Null(CmTraceSkewTripwire.Evaluate(MakeScan(imeDeltaMinutes: 17.5, otherDeltaMinutes: 0.0)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Silent_WhenDivergenceBelowOneGridStep()
|
||||
{
|
||||
// 9 min rounds to 1 step but with residual 6 — and 7 min rounds to 0 steps. Both silent.
|
||||
Assert.Null(CmTraceSkewTripwire.Evaluate(MakeScan(imeDeltaMinutes: 9.0, otherDeltaMinutes: 0.0)));
|
||||
Assert.Null(CmTraceSkewTripwire.Evaluate(MakeScan(imeDeltaMinutes: 7.0, otherDeltaMinutes: 0.0)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Silent_WhenImeSampleCountBelowMinimum()
|
||||
{
|
||||
var scan = MakeScan(imeDeltaMinutes: -540.0, otherDeltaMinutes: 0.0,
|
||||
imeSamples: CmTraceSkewTripwire.MinSamplesPerSide - 1);
|
||||
Assert.Null(CmTraceSkewTripwire.Evaluate(scan));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Silent_WhenDistinctBatchesBelowMinimum()
|
||||
{
|
||||
// A perfect grid divergence backed by only 2 upload batches on one side is a
|
||||
// batch-composition artifact (ReceivedAt is stamped per batch, not per event).
|
||||
var scan = MakeScan(imeDeltaMinutes: -540.0, otherDeltaMinutes: 0.0,
|
||||
otherBatches: CmTraceSkewTripwire.MinDistinctBatchesPerSide - 1);
|
||||
Assert.Null(CmTraceSkewTripwire.Evaluate(scan));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Silent_OnCommonModeSpoolLatency()
|
||||
{
|
||||
// Offline spool: BOTH sides uploaded 45 min late. The difference of medians is ~0 —
|
||||
// shared latency must cancel, never read as skew.
|
||||
Assert.Null(CmTraceSkewTripwire.Evaluate(MakeScan(imeDeltaMinutes: 45.3, otherDeltaMinutes: 45.1)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Silent_OnNullOrEmptyScan()
|
||||
{
|
||||
Assert.Null(CmTraceSkewTripwire.Evaluate(null));
|
||||
Assert.Null(CmTraceSkewTripwire.Evaluate(new SessionSkewScan()));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MedianIsRobustAgainstMinorityOutliers()
|
||||
{
|
||||
// A handful of replayed old lines (huge off-grid deltas) inside an otherwise healthy
|
||||
// IME stream must not drag the median onto the grid.
|
||||
var scan = MakeScan(imeDeltaMinutes: 0.5, otherDeltaMinutes: 0.4);
|
||||
scan.ImeDeltaMinutes.Add(-1327.0);
|
||||
scan.ImeDeltaMinutes.Add(-738.0);
|
||||
Assert.Null(CmTraceSkewTripwire.Evaluate(scan));
|
||||
}
|
||||
|
||||
// ── Bias suppression predicate ─────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public void BiasDominated_WhenBiasIsHalfOrMore()
|
||||
{
|
||||
Assert.True(CmTraceSkewTripwire.IsBiasDominated(new Dictionary<string, int>
|
||||
{
|
||||
["bias"] = 50,
|
||||
["line-anchored"] = 50,
|
||||
}));
|
||||
Assert.True(CmTraceSkewTripwire.IsBiasDominated(new Dictionary<string, int>
|
||||
{
|
||||
["Bias"] = 3, // case-insensitive
|
||||
}));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NotBiasDominated_WhenAnchoredMajorityOrNoData()
|
||||
{
|
||||
Assert.False(CmTraceSkewTripwire.IsBiasDominated(new Dictionary<string, int>
|
||||
{
|
||||
["bias"] = 10,
|
||||
["line-anchored"] = 80,
|
||||
["reader-zone-fallback"] = 11,
|
||||
}));
|
||||
// Empty/missing histogram fails open toward reporting — a storage error must not
|
||||
// silently swallow a genuine regression hit.
|
||||
Assert.False(CmTraceSkewTripwire.IsBiasDominated(new Dictionary<string, int>()));
|
||||
Assert.False(CmTraceSkewTripwire.IsBiasDominated(null));
|
||||
}
|
||||
}
|
||||
@@ -379,20 +379,50 @@ public class EventTimestampValidationTests
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RecalculateAppDurations_DownloadThenInstall_CorrectDownloadDuration()
|
||||
public void DownloadThenInstall_AttemptScopedPairing_DownloadAndInstallDurations()
|
||||
{
|
||||
// 2026-08 attempt-duration change: DownloadDurationSeconds pairs the latest download
|
||||
// start with the NEXT install start at event-fold time (no first→first span math),
|
||||
// and DurationSeconds anchors on the install attempt (download time is reported
|
||||
// separately, not folded into the install duration).
|
||||
var downloadStart = new DateTime(2026, 3, 30, 10, 0, 0, DateTimeKind.Utc);
|
||||
var installStart = downloadStart.AddSeconds(30);
|
||||
var completed = installStart.AddSeconds(45);
|
||||
|
||||
var summaries = new Dictionary<string, AppInstallAggregationState>(StringComparer.OrdinalIgnoreCase);
|
||||
foreach (var evt in new[]
|
||||
{
|
||||
AppInstallEvent("app_download_started", downloadStart),
|
||||
AppInstallEvent("app_install_started", installStart),
|
||||
AppInstallEvent("app_install_completed", completed, "Installed"),
|
||||
})
|
||||
{
|
||||
EventIngestProcessor.AggregateAppInstallEvent(evt, "tenant", "session", summaries);
|
||||
}
|
||||
|
||||
var summary = summaries["TestApp"].Summary;
|
||||
Assert.Equal(30, summary.DownloadDurationSeconds); // download→install gap
|
||||
Assert.Equal(45, summary.DurationSeconds); // install attempt only
|
||||
Assert.Equal(installStart, summary.LastAttemptStartedAt);
|
||||
Assert.Equal(downloadStart, summary.StartedAt); // first-seen anchor
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RecalculateAppDurations_AttemptAnchor_BeatsFirstSeenSpan()
|
||||
{
|
||||
// With an observed attempt start, the duration is the attempt — never the span
|
||||
// back to the first observation an hour earlier.
|
||||
var firstSeen = new DateTime(2026, 3, 30, 10, 0, 0, DateTimeKind.Utc);
|
||||
var attemptStart = firstSeen.AddMinutes(60);
|
||||
var completed = attemptStart.AddSeconds(45);
|
||||
|
||||
var state = new AppInstallAggregationState
|
||||
{
|
||||
DownloadStartedAt = downloadStart,
|
||||
InstallStartedAt = installStart,
|
||||
Summary = new AppInstallSummary
|
||||
{
|
||||
AppName = "TestApp",
|
||||
StartedAt = downloadStart,
|
||||
StartedAt = firstSeen,
|
||||
LastAttemptStartedAt = attemptStart,
|
||||
CompletedAt = completed,
|
||||
Status = "Succeeded"
|
||||
}
|
||||
@@ -400,8 +430,46 @@ public class EventTimestampValidationTests
|
||||
|
||||
EventIngestProcessor.RecalculateAppDurations(state);
|
||||
|
||||
Assert.Equal(30, state.Summary.DownloadDurationSeconds); // download→install gap
|
||||
Assert.Equal(75, state.Summary.DurationSeconds); // full duration
|
||||
Assert.Equal(45, state.Summary.DurationSeconds);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RecalculateAppDurations_AttemptNewerThanCompletion_KeepsRecordedDuration()
|
||||
{
|
||||
// A fresh pass is in flight (start after the last terminal): CompletedAt belongs to
|
||||
// the previous attempt — its recorded duration must stand.
|
||||
var attempt1 = new DateTime(2026, 3, 30, 10, 0, 0, DateTimeKind.Utc);
|
||||
var completed = attempt1.AddSeconds(40);
|
||||
var attempt2 = completed.AddMinutes(30);
|
||||
|
||||
var state = new AppInstallAggregationState
|
||||
{
|
||||
Summary = new AppInstallSummary
|
||||
{
|
||||
AppName = "TestApp",
|
||||
StartedAt = attempt1,
|
||||
LastAttemptStartedAt = attempt2,
|
||||
CompletedAt = completed,
|
||||
DurationSeconds = 40,
|
||||
Status = "Succeeded"
|
||||
}
|
||||
};
|
||||
|
||||
EventIngestProcessor.RecalculateAppDurations(state);
|
||||
|
||||
Assert.Equal(40, state.Summary.DurationSeconds);
|
||||
}
|
||||
|
||||
private static EnrollmentEvent AppInstallEvent(string eventType, DateTime ts, string? state = null)
|
||||
{
|
||||
var evt = new EnrollmentEvent
|
||||
{
|
||||
EventType = eventType,
|
||||
Timestamp = ts,
|
||||
Data = new Dictionary<string, object> { ["appName"] = "TestApp" },
|
||||
};
|
||||
if (state != null) evt.Data["state"] = state;
|
||||
return evt;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -46,6 +46,8 @@ public class FleetHealthPayloadTests
|
||||
Assert.Equal(0, payload.Stats.Total);
|
||||
Assert.Equal(0d, payload.Stats.SuccessRate);
|
||||
Assert.Equal(0, payload.Stats.AvgDurationMinutes);
|
||||
Assert.Equal(0, payload.Stats.MedianDurationMinutes);
|
||||
Assert.Equal(0, payload.Stats.P90DurationMinutes);
|
||||
Assert.Empty(payload.FailureReasons);
|
||||
Assert.Empty(payload.ModelHealth);
|
||||
Assert.Empty(payload.SlowestModels);
|
||||
@@ -109,6 +111,28 @@ public class FleetHealthPayloadTests
|
||||
|
||||
// (600 + 1200) / 2 / 60 = 15 min.
|
||||
Assert.Equal(15, payload.Stats.AvgDurationMinutes);
|
||||
// Same population feeds the percentile stats: sorted [600, 1200].
|
||||
Assert.Equal(10, payload.Stats.MedianDurationMinutes);
|
||||
Assert.Equal(20, payload.Stats.P90DurationMinutes);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MedianDuration_IsRobustAgainstOutliers_ThatDominateTheAverage()
|
||||
{
|
||||
// The card's raison d'être: 8 typical 10-min enrollments plus a 2 h and a 24 h
|
||||
// outlier (overnight ESP / WhiteGlove late user phase). The mean explodes to ~2.7 h
|
||||
// while the median still answers "how long does a typical enrollment take".
|
||||
var sessions = new List<SessionSummary>();
|
||||
for (int i = 0; i < 8; i++)
|
||||
sessions.Add(S(SessionStatus.Succeeded, durationSeconds: 600));
|
||||
sessions.Add(S(SessionStatus.Succeeded, durationSeconds: 7200));
|
||||
sessions.Add(S(SessionStatus.Succeeded, durationSeconds: 86400));
|
||||
|
||||
var payload = MetricsMath.BuildFleetHealthPayload(sessions, days: 7);
|
||||
|
||||
Assert.Equal(164, payload.Stats.AvgDurationMinutes); // (8*600 + 7200 + 86400) / 10 / 60
|
||||
Assert.Equal(10, payload.Stats.MedianDurationMinutes); // unmoved by the tail
|
||||
Assert.Equal(120, payload.Stats.P90DurationMinutes); // tail stays visible
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -33,6 +33,31 @@ public class IngestCriticalPathTests
|
||||
Assert.All(events, evt => Assert.Equal(receivedAt, evt.ReceivedAt));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StampServerFields_SetsSentAtOnAllEvents_AndOverridesAgentValues()
|
||||
{
|
||||
// P14: SentAt is request-level and server-stamped — a per-event value coming off the
|
||||
// wire (agent bug, forged payload) must never survive.
|
||||
var sentAt = DateTime.UtcNow.AddSeconds(-4);
|
||||
var events = MakeEvents(3);
|
||||
events[1].SentAt = DateTime.UtcNow.AddYears(1);
|
||||
|
||||
EventIngestProcessor.StampServerFields(events, ValidTenantId, ValidSessionId, DateTime.UtcNow, sentAt);
|
||||
|
||||
Assert.All(events, evt => Assert.Equal(sentAt, evt.SentAt));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StampServerFields_WithoutSendTimeHeader_LeavesSentAtNull()
|
||||
{
|
||||
var events = MakeEvents(2);
|
||||
events[0].SentAt = DateTime.UtcNow; // forged per-event value must be cleared, not kept
|
||||
|
||||
EventIngestProcessor.StampServerFields(events, ValidTenantId, ValidSessionId, DateTime.UtcNow);
|
||||
|
||||
Assert.All(events, evt => Assert.Null(evt.SentAt));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void StampServerFields_SetsTenantIdOnAllEvents()
|
||||
{
|
||||
|
||||
@@ -43,6 +43,45 @@ public class IngestTelemetryFunctionTests
|
||||
Assert.Equal(string.Empty, session);
|
||||
}
|
||||
|
||||
// ============================================================ X-Send-Time-Utc parsing (P14)
|
||||
|
||||
[Fact]
|
||||
public void ParseSendTimeHeader_RoundTripUtc_ParsesToUtc()
|
||||
{
|
||||
var parsed = IngestTelemetryFunction.ParseSendTimeHeader("2026-08-21T10:15:30.1234567Z");
|
||||
|
||||
Assert.Equal(new DateTime(2026, 8, 21, 10, 15, 30, DateTimeKind.Utc).AddTicks(1234567), parsed);
|
||||
Assert.Equal(DateTimeKind.Utc, parsed!.Value.Kind);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ParseSendTimeHeader_OffsetValue_NormalizedToUtc()
|
||||
{
|
||||
var parsed = IngestTelemetryFunction.ParseSendTimeHeader("2026-08-21T12:15:30+02:00");
|
||||
|
||||
Assert.Equal(new DateTime(2026, 8, 21, 10, 15, 30, DateTimeKind.Utc), parsed);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ParseSendTimeHeader_FutureDatedValue_KeptVerbatim()
|
||||
{
|
||||
// A future send time IS the device-clock-error measurement — no upper clamp.
|
||||
var parsed = IngestTelemetryFunction.ParseSendTimeHeader("2027-01-01T00:00:00Z");
|
||||
|
||||
Assert.Equal(new DateTime(2027, 1, 1, 0, 0, 0, DateTimeKind.Utc), parsed);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
[InlineData("not-a-date")]
|
||||
[InlineData("1999-12-31T23:59:59Z")] // pre-2000 can only be garbage (Table Storage floor guard)
|
||||
public void ParseSendTimeHeader_AbsentOrGarbage_ReturnsNull(string? headerValue)
|
||||
{
|
||||
Assert.Null(IngestTelemetryFunction.ParseSendTimeHeader(headerValue));
|
||||
}
|
||||
|
||||
// ============================================================ Payload size-cap guard
|
||||
|
||||
[Fact]
|
||||
|
||||
+75
@@ -0,0 +1,75 @@
|
||||
using System;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests
|
||||
{
|
||||
/// <summary>
|
||||
/// Pins the clock-frame contract of the agent-silence sweep.
|
||||
///
|
||||
/// <para>
|
||||
/// <c>LastEventAt</c> is the maximum AGENT-supplied event timestamp and therefore lives in the
|
||||
/// device clock frame — it carries the device's clock error and, for CMTrace-derived events,
|
||||
/// the writing process's timezone belief. Field measurement over 11k sessions found skews from
|
||||
/// -17 h to +1 h. Comparing that against a server-derived silence cutoff marked live agents
|
||||
/// Stalled ("Agent silent for 1020min") and left genuinely silent ones undetected for the
|
||||
/// duration of the skew. <c>LastIngestAt</c> is stamped from the server clock on every ingest
|
||||
/// and is the only value such a comparison may use.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class MaintenanceServerFrameLastContactTests
|
||||
{
|
||||
private static readonly DateTime Now = new(2026, 08, 20, 12, 00, 00, DateTimeKind.Utc);
|
||||
|
||||
private static SessionSummary Session(DateTime? lastIngestAt, DateTime? lastEventAt, DateTime startedAt)
|
||||
=> new() { LastIngestAt = lastIngestAt, LastEventAt = lastEventAt, StartedAt = startedAt };
|
||||
|
||||
[Fact]
|
||||
public void PrefersServerFrameIngestStamp_OverDeviceFrameEventStamp()
|
||||
{
|
||||
var s = Session(lastIngestAt: Now.AddMinutes(-1), lastEventAt: Now.AddHours(-17), startedAt: Now.AddHours(-2));
|
||||
Assert.Equal(Now.AddMinutes(-1), MaintenanceService.ServerFrameLastContact(s));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NegativeSkew_DoesNotLookSilent_WhenIngestIsFresh()
|
||||
{
|
||||
// The exact field shape: IME-derived events landed 17 h in the past, so LastEventAt is
|
||||
// far behind the 2 h silence cutoff while the agent is demonstrably still reporting.
|
||||
var silenceCutoff = Now.AddHours(-2);
|
||||
var s = Session(lastIngestAt: Now.AddSeconds(-30), lastEventAt: Now.AddHours(-17), startedAt: Now.AddHours(-1));
|
||||
|
||||
Assert.True(MaintenanceService.ServerFrameLastContact(s) > silenceCutoff,
|
||||
"a session whose ingest stamp is 30 s old must never be classified as agent-silent");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PositiveSkew_DoesNotMaskSilence_WhenIngestIsStale()
|
||||
{
|
||||
// Mirror case: a +1 h skew pushes LastEventAt into the future, which previously made the
|
||||
// sweep blind for an hour. The server frame still sees the silence.
|
||||
var silenceCutoff = Now.AddHours(-2);
|
||||
var s = Session(lastIngestAt: Now.AddHours(-3), lastEventAt: Now.AddHours(1), startedAt: Now.AddHours(-4));
|
||||
|
||||
Assert.True(MaintenanceService.ServerFrameLastContact(s) < silenceCutoff,
|
||||
"a session with no ingest for 3 h must be detected as silent regardless of a future-dated event stamp");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FallsBackToEventStamp_OnRowsPredatingTheIngestStamp()
|
||||
{
|
||||
// Rollout safety: rows written before LastIngestAt existed must keep the old behaviour
|
||||
// rather than dropping out of the sweep entirely.
|
||||
var s = Session(lastIngestAt: null, lastEventAt: Now.AddHours(-3), startedAt: Now.AddHours(-5));
|
||||
Assert.Equal(Now.AddHours(-3), MaintenanceService.ServerFrameLastContact(s));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FallsBackToStartedAt_WhenNoEventStampExists()
|
||||
{
|
||||
var s = Session(lastIngestAt: null, lastEventAt: null, startedAt: Now.AddHours(-4));
|
||||
Assert.Equal(Now.AddHours(-4), MaintenanceService.ServerFrameLastContact(s));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,222 @@
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Moq;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Power watcher analyze rules — pins the firing behaviour of the built-ins that consume the
|
||||
/// agent's live <c>power_state_change</c> events (PowerStateWatcherHost):
|
||||
/// - ANALYZE-DEV-009 (high): battery crossed the 15% threshold while enrolling on battery.
|
||||
/// - ANALYZE-DEV-010 (warning): the device switched from AC to battery mid-enrollment.
|
||||
/// Both rules are deliberately single-required-condition: event_data conditions scan events
|
||||
/// INDEPENDENTLY (no same-instance join), so a two-condition rule could false-positive across
|
||||
/// two different power events. DEV-009 matches the agent-stamped thresholdPercent (only present
|
||||
/// on threshold events, value pins the ladder level); DEV-010 matches transition=ac_to_battery.
|
||||
/// Also pins that the rule's string "15" compares equal against the agent's numeric payload.
|
||||
/// </summary>
|
||||
public class PowerStateRuleTests
|
||||
{
|
||||
private const string TenantId = "a1b2c3d4-e5f6-7890-abcd-ef1234567890";
|
||||
private const string SessionId = "11111111-2222-3333-4444-555555555555";
|
||||
|
||||
[Fact]
|
||||
public async Task ANALYZE_DEV_009_fires_high_on_the_15_percent_threshold_event()
|
||||
{
|
||||
var rule = BuiltInAnalyzeRules.GetAll().First(r => r.RuleId == "ANALYZE-DEV-009");
|
||||
Assert.True(rule.Enabled);
|
||||
Assert.Equal("high", rule.Severity);
|
||||
Assert.False(rule.MarkSessionAsFailedDefault); // environment risk, not a KO verdict
|
||||
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
ThresholdCrossed(thresholdPercent: 15, batteryPercent: 13),
|
||||
};
|
||||
|
||||
var outcome = await RunAsync(rule, events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
Assert.Equal("ANALYZE-DEV-009", result.RuleId);
|
||||
Assert.Equal("high", result.Severity);
|
||||
|
||||
// The numeric payload (int 15) must satisfy the rule's string value "15"; field/value on
|
||||
// the matched condition feed the {{thresholdPercent}} interpolation token.
|
||||
var matched = AsDict(result.MatchedConditions["battery_threshold_15"]);
|
||||
Assert.Equal("thresholdPercent", AsString(matched["field"]));
|
||||
Assert.Equal("15", AsString(matched["value"]));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(50)]
|
||||
[InlineData(30)]
|
||||
public async Task ANALYZE_DEV_009_does_not_fire_on_higher_threshold_levels(int level)
|
||||
{
|
||||
var rule = BuiltInAnalyzeRules.GetAll().First(r => r.RuleId == "ANALYZE-DEV-009");
|
||||
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
ThresholdCrossed(thresholdPercent: level, batteryPercent: level - 2),
|
||||
};
|
||||
|
||||
var outcome = await RunAsync(rule, events);
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ANALYZE_DEV_009_does_not_fire_on_a_low_battery_transition_event()
|
||||
{
|
||||
// An ac_to_battery event at 13% carries batteryPercent but NO thresholdPercent — the
|
||||
// agent emits the separate threshold event for that; the rule must not fire early here.
|
||||
var rule = BuiltInAnalyzeRules.GetAll().First(r => r.RuleId == "ANALYZE-DEV-009");
|
||||
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
Transition("ac_to_battery", onAcPower: false, batteryPercent: 13),
|
||||
};
|
||||
|
||||
var outcome = await RunAsync(rule, events);
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ANALYZE_DEV_010_fires_warning_on_ac_to_battery()
|
||||
{
|
||||
var rule = BuiltInAnalyzeRules.GetAll().First(r => r.RuleId == "ANALYZE-DEV-010");
|
||||
Assert.True(rule.Enabled);
|
||||
Assert.Equal("warning", rule.Severity);
|
||||
Assert.False(rule.MarkSessionAsFailedDefault);
|
||||
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
Transition("ac_to_battery", onAcPower: false, batteryPercent: 63),
|
||||
};
|
||||
|
||||
var outcome = await RunAsync(rule, events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
Assert.Equal("ANALYZE-DEV-010", result.RuleId);
|
||||
Assert.Equal("warning", result.Severity);
|
||||
|
||||
var matched = AsDict(result.MatchedConditions["switched_to_battery"]);
|
||||
Assert.Equal("transition", AsString(matched["field"]));
|
||||
Assert.Equal("ac_to_battery", AsString(matched["value"]));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("battery_to_ac")]
|
||||
[InlineData("threshold_crossed")]
|
||||
public async Task ANALYZE_DEV_010_does_not_fire_on_other_transitions(string transition)
|
||||
{
|
||||
var rule = BuiltInAnalyzeRules.GetAll().First(r => r.RuleId == "ANALYZE-DEV-010");
|
||||
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
Transition(transition, onAcPower: transition == "battery_to_ac", batteryPercent: 40),
|
||||
};
|
||||
|
||||
var outcome = await RunAsync(rule, events);
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Neither_rule_fires_on_the_startup_power_state_check()
|
||||
{
|
||||
// The one-shot startup probe event (power_state_check) is a different event type and must
|
||||
// not satisfy either rule, even when it reports a low battery on DC power.
|
||||
var rules = BuiltInAnalyzeRules.GetAll()
|
||||
.Where(r => r.RuleId is "ANALYZE-DEV-009" or "ANALYZE-DEV-010").ToList();
|
||||
Assert.Equal(2, rules.Count);
|
||||
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
new()
|
||||
{
|
||||
EventId = Guid.NewGuid().ToString(),
|
||||
TenantId = TenantId,
|
||||
SessionId = SessionId,
|
||||
EventType = "power_state_check",
|
||||
Timestamp = DateTime.UtcNow,
|
||||
Sequence = 5,
|
||||
Data = new Dictionary<string, object>
|
||||
{
|
||||
["onAcPower"] = false,
|
||||
["hasBattery"] = true,
|
||||
["batteryPercent"] = 12,
|
||||
["isCharging"] = false,
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
foreach (var rule in rules)
|
||||
{
|
||||
var outcome = await RunAsync(rule, events);
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
}
|
||||
|
||||
// ===== Event builders — mirror the PowerStateWatcherHost emit shape =====
|
||||
|
||||
private static EnrollmentEvent ThresholdCrossed(int thresholdPercent, int batteryPercent) => new()
|
||||
{
|
||||
EventId = Guid.NewGuid().ToString(),
|
||||
TenantId = TenantId,
|
||||
SessionId = SessionId,
|
||||
EventType = "power_state_change",
|
||||
Timestamp = DateTime.UtcNow,
|
||||
Sequence = 42,
|
||||
Data = new Dictionary<string, object>
|
||||
{
|
||||
["transition"] = "threshold_crossed",
|
||||
["thresholdPercent"] = thresholdPercent,
|
||||
["onAcPower"] = false,
|
||||
["batteryPercent"] = batteryPercent,
|
||||
["isCharging"] = false,
|
||||
["batteryLifeMinutes"] = 25,
|
||||
}
|
||||
};
|
||||
|
||||
private static EnrollmentEvent Transition(string transition, bool onAcPower, int batteryPercent) => new()
|
||||
{
|
||||
EventId = Guid.NewGuid().ToString(),
|
||||
TenantId = TenantId,
|
||||
SessionId = SessionId,
|
||||
EventType = "power_state_change",
|
||||
Timestamp = DateTime.UtcNow,
|
||||
Sequence = 43,
|
||||
Data = new Dictionary<string, object>
|
||||
{
|
||||
["transition"] = transition,
|
||||
["onAcPower"] = onAcPower,
|
||||
["batteryPercent"] = batteryPercent,
|
||||
["isCharging"] = onAcPower,
|
||||
["batteryLifeMinutes"] = 120,
|
||||
}
|
||||
};
|
||||
|
||||
private static Dictionary<string, object> AsDict(object o)
|
||||
{
|
||||
if (o is Dictionary<string, object> d) return d;
|
||||
throw new InvalidOperationException($"Expected Dictionary<string,object>, got {o?.GetType().Name ?? "null"}");
|
||||
}
|
||||
|
||||
private static string AsString(object o) => o?.ToString() ?? string.Empty;
|
||||
|
||||
private static async Task<AnalysisOutcome> RunAsync(AnalyzeRule rule, List<EnrollmentEvent> events)
|
||||
{
|
||||
var ruleRepo = new Mock<IRuleRepository>();
|
||||
ruleRepo.Setup(r => r.GetAnalyzeRulesAsync("global")).ReturnsAsync(new List<AnalyzeRule> { rule });
|
||||
ruleRepo.Setup(r => r.GetAnalyzeRulesAsync(TenantId)).ReturnsAsync(new List<AnalyzeRule>());
|
||||
ruleRepo.Setup(r => r.GetRuleStatesAsync(It.IsAny<string>())).ReturnsAsync(new Dictionary<string, RuleState>());
|
||||
ruleRepo.Setup(r => r.GetRuleResultsAsync(It.IsAny<string>(), It.IsAny<string>())).ReturnsAsync(new List<RuleResult>());
|
||||
|
||||
var sessionRepo = new Mock<ISessionRepository>();
|
||||
sessionRepo.Setup(s => s.GetSessionEventsStrictAsync(TenantId, SessionId, It.IsAny<int>())).ReturnsAsync(events);
|
||||
|
||||
var ruleService = new AnalyzeRuleService(ruleRepo.Object, NullLogger<AnalyzeRuleService>.Instance);
|
||||
var engine = new RuleEngine(ruleService, ruleRepo.Object, sessionRepo.Object, NullLogger<PowerStateRuleTests>.Instance);
|
||||
|
||||
return await engine.AnalyzeSessionAsync(TenantId, SessionId);
|
||||
}
|
||||
}
|
||||
@@ -200,4 +200,120 @@ public class ReportAgentErrorFunctionTests
|
||||
|
||||
Assert.Empty(h.OpsEvents);
|
||||
}
|
||||
|
||||
// =========================================================================
|
||||
// MaterializeIntegrityMismatchAsync — an IntegrityCheckFailed report becomes an
|
||||
// AgentBinaryIntegrityMismatch ops event, UNLESS it is the known release race
|
||||
// (agents comparing against hash snapshots that don't belong to their own binary;
|
||||
// AdminConfig cache TTL 5 min). e9753578 proved the race: a genuine 1410 exe was
|
||||
// compared against the cached 1409 hash and reported a "mismatch" that cost a day
|
||||
// of mis-attributed root-causing while sitting invisible in App Insights.
|
||||
// =========================================================================
|
||||
|
||||
private const string RunningExeSha = "5fca975e43fa5fca975e43fa5fca975e43fa5fca975e43fa5fca975e43fa0000";
|
||||
private const string PublishedExeSha = "8ebaa060ad348ebaa060ad348ebaa060ad348ebaa060ad348ebaa060ad340000";
|
||||
|
||||
private static AgentErrorReport IntegrityReport(
|
||||
string? sessionId = SessionId, string agentVersion = "2.0.1410", string? actualSha = null) => new()
|
||||
{
|
||||
SessionId = sessionId!,
|
||||
TenantId = TenantId,
|
||||
ErrorType = AgentErrorType.IntegrityCheckFailed,
|
||||
Message = $"Running exe SHA-256 differs from backend-advertised hash. actual={actualSha ?? RunningExeSha}, expected={PublishedExeSha}",
|
||||
AgentVersion = agentVersion,
|
||||
Timestamp = Now,
|
||||
};
|
||||
|
||||
private static Task RunIntegrityAsync(
|
||||
Harness h, AgentErrorReport report,
|
||||
string? latestVersion = "2.0.1410", string? latestExeSha = PublishedExeSha)
|
||||
=> ReportAgentErrorFunction.MaterializeIntegrityMismatchAsync(
|
||||
report, TenantId, h.OpsService, latestVersion, latestExeSha,
|
||||
NullLogger<ReportAgentErrorFunction>.Instance);
|
||||
|
||||
[Fact]
|
||||
public async Task SameVersion_with_unknown_binary_records_ops_event()
|
||||
{
|
||||
// The real signal: an agent claiming the LATEST version whose running exe is NOT
|
||||
// the published one — tamper, stale blob, or a build outside the release pipeline.
|
||||
var h = new Harness();
|
||||
|
||||
await RunIntegrityAsync(h, IntegrityReport());
|
||||
|
||||
var ops = Assert.Single(h.OpsEvents);
|
||||
Assert.Equal("AgentBinaryIntegrityMismatch", ops.EventType);
|
||||
Assert.Equal(TenantId, ops.TenantId);
|
||||
Assert.Contains("actual=5fca", ops.Message);
|
||||
Assert.Contains("2.0.1410", ops.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RunningHash_equal_to_published_exe_is_suppressed_as_stale_expectation()
|
||||
{
|
||||
// e9753578's exact shape: the device RUNS the published binary; only its cached
|
||||
// expectation was the previous release's hash. Self-heals — must not alert.
|
||||
var h = new Harness();
|
||||
|
||||
await RunIntegrityAsync(h, IntegrityReport(actualSha: PublishedExeSha));
|
||||
|
||||
Assert.Empty(h.OpsEvents);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Older_agent_version_than_latest_is_suppressed_as_update_race()
|
||||
{
|
||||
// The long-known direction: version N agent sees version N+1's hash right after a
|
||||
// release; the forced self-update heals it.
|
||||
var h = new Harness();
|
||||
|
||||
await RunIntegrityAsync(h, IntegrityReport(agentVersion: "2.0.1409+d43cfe2b"), latestVersion: "2.0.1410");
|
||||
|
||||
Assert.Empty(h.OpsEvents);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Version_with_commit_suffix_matches_latest_without_suffix()
|
||||
{
|
||||
// Agent versions carry "+<commit>", AdminConfiguration does not — the comparison
|
||||
// must normalize, otherwise every genuine mismatch would look like a version race.
|
||||
var h = new Harness();
|
||||
|
||||
await RunIntegrityAsync(h, IntegrityReport(agentVersion: "2.0.1410+3b59dab2df6c"));
|
||||
|
||||
Assert.Single(h.OpsEvents);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Missing_latest_config_fails_open_to_recording()
|
||||
{
|
||||
// No published-version info available → visibility wins over suppression.
|
||||
var h = new Harness();
|
||||
|
||||
await RunIntegrityAsync(h, IntegrityReport(), latestVersion: null, latestExeSha: null);
|
||||
|
||||
Assert.Single(h.OpsEvents);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Integrity_mismatch_without_session_id_still_records_ops_event()
|
||||
{
|
||||
// The report can arrive before session registration — the binary evidence must
|
||||
// never be dropped for lack of a session.
|
||||
var h = new Harness();
|
||||
|
||||
await RunIntegrityAsync(h, IntegrityReport(sessionId: null));
|
||||
|
||||
var ops = Assert.Single(h.OpsEvents);
|
||||
Assert.Equal("AgentBinaryIntegrityMismatch", ops.EventType);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Other_error_types_do_not_record_integrity_ops_event()
|
||||
{
|
||||
var h = new Harness();
|
||||
|
||||
await RunIntegrityAsync(h, Report(Now));
|
||||
|
||||
Assert.Empty(h.OpsEvents);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,450 @@
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Moq;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins the clock_skew condition source (EvaluateClockSkewCondition) behind ANALYZE-DEV-008.
|
||||
/// The evaluator measures d = Timestamp − ReceivedAt per event, reduces to per-upload-batch
|
||||
/// medians (wide-spread batches = spool backlogs are dropped), then detects either a persistent
|
||||
/// mid-session step between two flat plateaus (clock_jump) or a stable whole-session offset
|
||||
/// (sustained_offset). IME-log-derived events are excluded so a CMTrace anchoring regression
|
||||
/// (the e9753578 field incident) can never fire this customer-facing rule — that failure mode
|
||||
/// belongs to the operator-side CmTraceSkewTripwire.
|
||||
/// </summary>
|
||||
public class RuleEngineClockSkewTests
|
||||
{
|
||||
private const string TenantId = "a1b2c3d4-e5f6-7890-abcd-ef1234567890";
|
||||
private const string SessionId = "b2c3d4e5-f6a7-8901-bcde-f12345678901";
|
||||
|
||||
private static readonly DateTime T0 = new(2026, 8, 20, 10, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
// ── clock_jump: must fire ──────────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_ForwardStepBetweenStableFrames_Fires()
|
||||
{
|
||||
// 4 batches on frame 0, then 4 batches on frame +1800 s (+30 min) that hold to
|
||||
// session end — the canonical "clock corrected forward mid-session".
|
||||
var events = Batches(offsetsSeconds: new double[] { 0, 0, 0, 0, 1800, 1800, 1800, 1800 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
var evidence = Assert.IsAssignableFrom<IDictionary<string, object>>(result.MatchedConditions["clock_jump"]);
|
||||
Assert.Equal("forward", evidence["direction"]);
|
||||
Assert.Equal(1800, Convert.ToDouble(evidence["jumpSeconds"]), 0);
|
||||
Assert.Equal("skewSummary", evidence["field"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_BackwardStep_Fires()
|
||||
{
|
||||
var events = Batches(new double[] { 0, 0, 0, -1200, -1200, -1200 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
var evidence = Assert.IsAssignableFrom<IDictionary<string, object>>(result.MatchedConditions["clock_jump"]);
|
||||
Assert.Equal("backward", evidence["direction"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_WrongClockCorrectedMidSession_Fires()
|
||||
{
|
||||
// Device started 30 min behind, time sync fixed it: the CORRECTED frame is the
|
||||
// persistent change, so this must fire (direction forward).
|
||||
var events = Batches(new double[] { -1800, -1800, -1800, 0, 0, 0 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
var evidence = Assert.IsAssignableFrom<IDictionary<string, object>>(result.MatchedConditions["clock_jump"]);
|
||||
Assert.Equal("forward", evidence["direction"]);
|
||||
}
|
||||
|
||||
// ── clock_jump: must stay silent ───────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_ImeDerivedSkew_ExcludedFromMeasurement_Silent()
|
||||
{
|
||||
// e9753578 replica: IME-derived events carry grid-multiple wrong timestamps (−7 h /
|
||||
// +2 h writer eras) while the agent's own events are healthy. The IME events must be
|
||||
// excluded up front — a CMTrace anchoring regression may NEVER fire the customer rule.
|
||||
var events = Batches(new double[] { 0, 0, 0, 0, 0, 0 });
|
||||
var seq = 1000;
|
||||
foreach (var imeOffset in new double[] { -25200, -25200, 7200, 7200, -25200, 7200 })
|
||||
events.Add(MakeEvent(seq++, T0.AddMinutes(seq % 7), imeOffset, source: "ImeLogTracker"));
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_WideSpreadSpoolBatch_DroppedAndSilent()
|
||||
{
|
||||
// One upload batch whose events span 30 min of emission time (offline spool flushed
|
||||
// in a single request) — the internal spread disqualifies the batch entirely.
|
||||
var events = Batches(new double[] { 0, 0, 0, 0, 0, 0 });
|
||||
var flushReceivedAt = T0.AddMinutes(90);
|
||||
for (int i = 0; i < 20; i++)
|
||||
events.Add(MakeEvent(2000 + i, flushReceivedAt, offsetSeconds: -1800 + i * 90));
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_GradualBacklogDrainRamp_Silent()
|
||||
{
|
||||
// A chunked spool drain produces a RAMP of batch medians walking back to the live
|
||||
// frame. No flat pre/post plateau pair exists, so the step detector must reject every
|
||||
// candidate — this is a delivery artifact, not a clock change.
|
||||
var events = Batches(new double[] { -1800, -1500, -1200, -900, -600, -300, 0, 0 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_TransientExcursionReturnsToFrame_Silent()
|
||||
{
|
||||
// Frame dips for two batches and returns — whatever it was, it is not a SUSTAINED
|
||||
// clock change (end-state persistence fails).
|
||||
var events = Batches(new double[] { 0, 0, 0, -1800, -1800, 0, 0, 0 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_BelowThreshold_Silent()
|
||||
{
|
||||
var events = Batches(new double[] { 0, 0, 0, 240, 240, 240 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_TooFewBatches_Silent()
|
||||
{
|
||||
// 5 batches < 2×ConfirmBatches — not enough for two comparison windows.
|
||||
var events = Batches(new double[] { 0, 0, 1800, 1800, 1800 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_LegacyEventsWithoutReceivedAt_Silent()
|
||||
{
|
||||
var events = Batches(new double[] { 0, 0, 0, 1800, 1800, 1800 });
|
||||
foreach (var e in events)
|
||||
e.ReceivedAt = null;
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ClockJump_ClampedEventsExcluded_Silent()
|
||||
{
|
||||
// The jumped frame exists only on clamped events (their OccurredUtc is already
|
||||
// server-substituted in production) — excluding them leaves a single stable frame.
|
||||
var events = Batches(new double[] { 0, 0, 0, 0, 0, 0 });
|
||||
var seq = 3000;
|
||||
foreach (var offset in new double[] { 1800, 1800, 1800 })
|
||||
{
|
||||
var evt = MakeEvent(seq, T0.AddMinutes(120 + seq % 5), offset);
|
||||
evt.TimestampClamped = true;
|
||||
events.Add(evt);
|
||||
seq++;
|
||||
}
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
// ── sustained_offset ───────────────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public async Task SustainedOffset_ClockBehindWholeSession_Fires()
|
||||
{
|
||||
var events = Batches(new double[] { -1800, -1795, -1805, -1800, -1798 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("sustained_offset", "300"), events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
var evidence = Assert.IsAssignableFrom<IDictionary<string, object>>(result.MatchedConditions["sustained_offset"]);
|
||||
Assert.Equal("behind", evidence["direction"]);
|
||||
Assert.Equal(-30.0, Convert.ToDouble(evidence["offsetMinutes"]), 1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SustainedOffset_HealthyUploadLatencyBaseline_Silent()
|
||||
{
|
||||
// Healthy devices sit a few seconds NEGATIVE (upload latency) — far below threshold.
|
||||
var events = Batches(new double[] { -3, -5, -2, -4, -6 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("sustained_offset", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SustainedOffset_UnstableBatchOffsets_Silent()
|
||||
{
|
||||
// Medians all over the place: no honest single-offset statement exists, even though
|
||||
// the overall median magnitude clears the threshold.
|
||||
var events = Batches(new double[] { -2400, 900, -1800, 300, -3600, -600 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("sustained_offset", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
// ── P14: SentAt direct measurement (X-Send-Time-Utc batches) ───────────
|
||||
|
||||
[Fact]
|
||||
public async Task SentAt_SpoolFlushWithWideEventSpread_JumpStillMeasured_Fires()
|
||||
{
|
||||
// Every batch is a spool flush spanning 30 min of emission time — the legacy path
|
||||
// drops ALL of them via the spread cap and stays blind. SentAt measures the frame
|
||||
// crossing directly, so the +30 min step must fire regardless of event-timestamp
|
||||
// spread, and every batch counts as a SentAt batch.
|
||||
static double[] Wide(int _) => new double[] { -1800, -1350, -900, -450, 0 };
|
||||
var events = SentAtBatches(new double[] { 0, 0, 0, 0, 1800, 1800, 1800, 1800 }, Wide);
|
||||
|
||||
var outcome = await RunAsync(MakeRule("clock_jump", "300"), events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
var evidence = Assert.IsAssignableFrom<IDictionary<string, object>>(result.MatchedConditions["clock_jump"]);
|
||||
Assert.Equal("forward", evidence["direction"]);
|
||||
Assert.Equal(8, Convert.ToInt32(evidence["sentAtBatchCount"]));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SentAt_UniformBacklogBatches_ReadAsHealthy_Silent()
|
||||
{
|
||||
// EXACTLY the event shape of SustainedOffset_ClockBehindWholeSession_Fires (tight
|
||||
// −30 min Timestamp−ReceivedAt deltas), but the batches carry SentAt ≈ ReceivedAt:
|
||||
// the device clock is fine, the events are merely OLD (uniform spool backlog).
|
||||
// The legacy median cannot tell those apart — SentAt can, and must stay silent.
|
||||
static double[] Backlog(int b) => new double[] { -1800, -1795, -1805, -1800, -1798 };
|
||||
var events = SentAtBatches(new double[] { -3, -5, -2, -4, -6 }, Backlog);
|
||||
|
||||
var outcome = await RunAsync(MakeRule("sustained_offset", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SentAt_SustainedOffset_MeasuredFromSendTime_Fires()
|
||||
{
|
||||
var events = SentAtBatches(new double[] { -1800, -1795, -1805, -1800, -1798 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("sustained_offset", "300"), events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
var evidence = Assert.IsAssignableFrom<IDictionary<string, object>>(result.MatchedConditions["sustained_offset"]);
|
||||
Assert.Equal("behind", evidence["direction"]);
|
||||
Assert.Equal(-30.0, Convert.ToDouble(evidence["offsetMinutes"]), 1);
|
||||
Assert.Equal(5, Convert.ToInt32(evidence["sentAtBatchCount"]));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SentAt_ImeOnlyBatches_MeasurableViaSendTime_Fires()
|
||||
{
|
||||
// IME-source events are excluded from the LEGACY measurement (writer timezone
|
||||
// belief), but a SentAt batch never reads event timestamps — the device clock is
|
||||
// measured from the send moment, so IME-only sessions become measurable.
|
||||
var events = SentAtBatches(
|
||||
new double[] { -1800, -1800, -1800, -1800, -1800 }, source: "ImeLogTracker");
|
||||
|
||||
var outcome = await RunAsync(MakeRule("sustained_offset", "300"), events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
var evidence = Assert.IsAssignableFrom<IDictionary<string, object>>(result.MatchedConditions["sustained_offset"]);
|
||||
Assert.Equal("behind", evidence["direction"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SentAt_HealthyUploadLatencyBaseline_Silent()
|
||||
{
|
||||
var events = SentAtBatches(new double[] { -3, -5, -2, -4, -6 });
|
||||
|
||||
var outcome = await RunAsync(MakeRule("sustained_offset", "300"), events);
|
||||
|
||||
Assert.Empty(outcome.Results);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SentAt_MixedWithLegacyBatches_ConsistentFrame_Fires()
|
||||
{
|
||||
// Mid-session agent update: first half legacy (event-median), second half SentAt.
|
||||
// Both measure the same physical quantity, so a consistent −30 min frame across the
|
||||
// mix must still produce a sustained-offset verdict.
|
||||
var events = Batches(new double[] { -1800, -1795, -1805 });
|
||||
long seq = 1000;
|
||||
for (int b = 0; b < 3; b++)
|
||||
{
|
||||
var receivedAt = T0.AddMinutes(60 + b * 5);
|
||||
var sentAt = receivedAt.AddSeconds(-1800);
|
||||
for (int i = 0; i < 5; i++)
|
||||
{
|
||||
var evt = MakeEvent(seq++, receivedAt, -1800 + (i - 2));
|
||||
evt.SentAt = sentAt;
|
||||
events.Add(evt);
|
||||
}
|
||||
}
|
||||
|
||||
var outcome = await RunAsync(MakeRule("sustained_offset", "300"), events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
var evidence = Assert.IsAssignableFrom<IDictionary<string, object>>(result.MatchedConditions["sustained_offset"]);
|
||||
Assert.Equal(-30.0, Convert.ToDouble(evidence["offsetMinutes"]), 1);
|
||||
Assert.Equal(3, Convert.ToInt32(evidence["sentAtBatchCount"]));
|
||||
Assert.Equal(6, Convert.ToInt32(evidence["batchCount"]));
|
||||
}
|
||||
|
||||
// ── ANALYZE-DEV-008 shape: two optional conditions ─────────────────────
|
||||
|
||||
[Fact]
|
||||
public async Task Dev008Shape_JumpOnlySession_FiresWithSingleMatchedCondition()
|
||||
{
|
||||
// Mirrors the shipped rule: both conditions required:false, baseConfidence 60,
|
||||
// threshold 50 — one matching metric must be enough to fire.
|
||||
var rule = MakeRule("clock_jump", "300");
|
||||
rule.Conditions[0].Required = false;
|
||||
rule.Conditions.Add(new RuleCondition
|
||||
{
|
||||
Signal = "sustained_offset",
|
||||
Source = "clock_skew",
|
||||
SkewMetric = "sustained_offset",
|
||||
Operator = "gte",
|
||||
Value = "300",
|
||||
Required = false,
|
||||
});
|
||||
|
||||
var events = Batches(new double[] { 0, 0, 0, 0, 1800, 1800, 1800, 1800 });
|
||||
|
||||
var outcome = await RunAsync(rule, events);
|
||||
|
||||
var result = Assert.Single(outcome.Results);
|
||||
Assert.True(result.MatchedConditions.ContainsKey("clock_jump"));
|
||||
}
|
||||
|
||||
// ===== Helpers =====
|
||||
|
||||
private static AnalyzeRule MakeRule(string skewMetric, string thresholdSeconds) => new()
|
||||
{
|
||||
RuleId = "ANALYZE-TST-CLK",
|
||||
Title = "clock_skew test rule",
|
||||
Severity = "warning",
|
||||
Category = "device",
|
||||
Enabled = true,
|
||||
IsBuiltIn = false,
|
||||
BaseConfidence = 60,
|
||||
ConfidenceThreshold = 50,
|
||||
Conditions = new List<RuleCondition>
|
||||
{
|
||||
new()
|
||||
{
|
||||
Signal = skewMetric,
|
||||
Source = "clock_skew",
|
||||
SkewMetric = skewMetric,
|
||||
Operator = "gte",
|
||||
Value = thresholdSeconds,
|
||||
Required = true,
|
||||
}
|
||||
},
|
||||
Explanation = "test {{skewSummary}}",
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// One upload batch per entry: 5 events sharing the batch's ReceivedAt, each with
|
||||
/// Timestamp = ReceivedAt + offsetSeconds (±2 s in-batch jitter, well under the spread cap).
|
||||
/// </summary>
|
||||
private static List<EnrollmentEvent> Batches(double[] offsetsSeconds)
|
||||
{
|
||||
var events = new List<EnrollmentEvent>();
|
||||
long seq = 1;
|
||||
for (int b = 0; b < offsetsSeconds.Length; b++)
|
||||
{
|
||||
var receivedAt = T0.AddMinutes(b * 5);
|
||||
for (int i = 0; i < 5; i++)
|
||||
events.Add(MakeEvent(seq++, receivedAt, offsetsSeconds[b] + (i - 2)));
|
||||
}
|
||||
return events;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One SentAt-carrying upload batch per frame offset: SentAt = ReceivedAt + frameOffset
|
||||
/// (the P14 direct measurement), event timestamps default to healthy in-batch jitter but
|
||||
/// can be overridden per batch to simulate spool spreads or backlogs.
|
||||
/// </summary>
|
||||
private static List<EnrollmentEvent> SentAtBatches(
|
||||
double[] frameOffsetsSeconds,
|
||||
Func<int, double[]>? eventOffsetsForBatch = null,
|
||||
string source = "DecisionEngine")
|
||||
{
|
||||
var events = new List<EnrollmentEvent>();
|
||||
long seq = 1;
|
||||
for (int b = 0; b < frameOffsetsSeconds.Length; b++)
|
||||
{
|
||||
var receivedAt = T0.AddMinutes(b * 5);
|
||||
var sentAt = receivedAt.AddSeconds(frameOffsetsSeconds[b]);
|
||||
var offsets = eventOffsetsForBatch?.Invoke(b) ?? new double[] { -2, -1, 0, 1, 2 };
|
||||
foreach (var offset in offsets)
|
||||
{
|
||||
var evt = MakeEvent(seq++, receivedAt, offset, source);
|
||||
evt.SentAt = sentAt;
|
||||
events.Add(evt);
|
||||
}
|
||||
}
|
||||
return events;
|
||||
}
|
||||
|
||||
private static EnrollmentEvent MakeEvent(long sequence, DateTime receivedAt, double offsetSeconds,
|
||||
string source = "DecisionEngine") => new()
|
||||
{
|
||||
EventId = Guid.NewGuid().ToString(),
|
||||
TenantId = TenantId,
|
||||
SessionId = SessionId,
|
||||
EventType = "status_update",
|
||||
Source = source,
|
||||
Timestamp = receivedAt.AddSeconds(offsetSeconds),
|
||||
ReceivedAt = receivedAt,
|
||||
Sequence = sequence,
|
||||
Data = new Dictionary<string, object>(),
|
||||
};
|
||||
|
||||
private static async Task<AnalysisOutcome> RunAsync(AnalyzeRule rule, List<EnrollmentEvent> events)
|
||||
{
|
||||
var ruleRepo = new Mock<IRuleRepository>();
|
||||
ruleRepo.Setup(r => r.GetAnalyzeRulesAsync("global")).ReturnsAsync(new List<AnalyzeRule> { rule });
|
||||
ruleRepo.Setup(r => r.GetAnalyzeRulesAsync(TenantId)).ReturnsAsync(new List<AnalyzeRule>());
|
||||
ruleRepo.Setup(r => r.GetRuleStatesAsync(It.IsAny<string>())).ReturnsAsync(new Dictionary<string, RuleState>());
|
||||
ruleRepo.Setup(r => r.GetRuleResultsAsync(It.IsAny<string>(), It.IsAny<string>())).ReturnsAsync(new List<RuleResult>());
|
||||
|
||||
var sessionRepo = new Mock<ISessionRepository>();
|
||||
sessionRepo.Setup(s => s.GetSessionEventsStrictAsync(TenantId, SessionId, It.IsAny<int>())).ReturnsAsync(events);
|
||||
|
||||
var ruleService = new AnalyzeRuleService(ruleRepo.Object, NullLogger<AnalyzeRuleService>.Instance);
|
||||
var engine = new RuleEngine(ruleService, ruleRepo.Object, sessionRepo.Object, NullLogger<RuleEngineClockSkewTests>.Instance);
|
||||
|
||||
return await engine.AnalyzeSessionAsync(TenantId, SessionId);
|
||||
}
|
||||
}
|
||||
@@ -270,6 +270,59 @@ public class RuleEngineDryRunTests
|
||||
Assert.Contains(DryRunAnalyzeRuleFunction.ValidateDraftRule(rule), e => e.Contains("joinField"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validate_ClockSkewMissingOrUnknownMetric_Error()
|
||||
{
|
||||
var rule = MakeRule();
|
||||
rule.Conditions[0].Source = "clock_skew";
|
||||
rule.Conditions[0].SkewMetric = null;
|
||||
rule.Conditions[0].Operator = "gte";
|
||||
rule.Conditions[0].Value = "300";
|
||||
Assert.Contains(DryRunAnalyzeRuleFunction.ValidateDraftRule(rule), e => e.Contains("skewMetric"));
|
||||
|
||||
rule.Conditions[0].SkewMetric = "drift"; // not a known metric
|
||||
Assert.Contains(DryRunAnalyzeRuleFunction.ValidateDraftRule(rule), e => e.Contains("skewMetric"));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("lt")]
|
||||
[InlineData("equals")]
|
||||
public void Validate_ClockSkewUnsupportedOperator_Error(string op)
|
||||
{
|
||||
var rule = MakeRule();
|
||||
rule.Conditions[0].Source = "clock_skew";
|
||||
rule.Conditions[0].SkewMetric = "clock_jump";
|
||||
rule.Conditions[0].Operator = op;
|
||||
rule.Conditions[0].Value = "300";
|
||||
Assert.Contains(DryRunAnalyzeRuleFunction.ValidateDraftRule(rule), e => e.Contains("only gt/gte"));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("")]
|
||||
[InlineData("abc")]
|
||||
[InlineData("-300")]
|
||||
[InlineData("0")]
|
||||
public void Validate_ClockSkewNonPositiveThreshold_Error(string value)
|
||||
{
|
||||
var rule = MakeRule();
|
||||
rule.Conditions[0].Source = "clock_skew";
|
||||
rule.Conditions[0].SkewMetric = "sustained_offset";
|
||||
rule.Conditions[0].Operator = "gte";
|
||||
rule.Conditions[0].Value = value;
|
||||
Assert.Contains(DryRunAnalyzeRuleFunction.ValidateDraftRule(rule), e => e.Contains("positive numeric value"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validate_ClockSkewValidCondition_NoErrors()
|
||||
{
|
||||
var rule = MakeRule();
|
||||
rule.Conditions[0].Source = "clock_skew";
|
||||
rule.Conditions[0].SkewMetric = "clock_jump";
|
||||
rule.Conditions[0].Operator = "gte";
|
||||
rule.Conditions[0].Value = "300";
|
||||
Assert.Empty(DryRunAnalyzeRuleFunction.ValidateDraftRule(rule));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Validate_InvalidRegex_Error()
|
||||
{
|
||||
|
||||
@@ -90,6 +90,7 @@ public class SessionIndexFieldManifestTests
|
||||
["DiagnosticsBlobName"] = "diag-0001.zip",
|
||||
["DiagnosticsBlobDestination"] = "Hosted",
|
||||
["LastEventAt"] = new DateTimeOffset(StartedAt.AddMinutes(44)),
|
||||
["LastIngestAt"] = new DateTimeOffset(StartedAt.AddMinutes(46)),
|
||||
["ResumedAt"] = new DateTimeOffset(StartedAt.AddMinutes(10)),
|
||||
["StalledAt"] = new DateTimeOffset(StartedAt.AddMinutes(20)),
|
||||
["AvgApiLatencyMs"] = 513.5,
|
||||
|
||||
@@ -36,6 +36,8 @@ public class SessionStatsAggregationTests
|
||||
Assert.Equal(0, stats.FailedLastNDays);
|
||||
Assert.Equal(0, stats.SuccessRatePct);
|
||||
Assert.Equal(0, stats.AvgDurationMinutes);
|
||||
Assert.Equal(0, stats.MedianDurationMinutes);
|
||||
Assert.Equal(0, stats.P90DurationMinutes);
|
||||
Assert.Equal(0, stats.TotalToday);
|
||||
Assert.Equal(0, stats.FailedToday);
|
||||
}
|
||||
@@ -116,6 +118,29 @@ public class SessionStatsAggregationTests
|
||||
|
||||
// (30 + 10) / 2 = 20 minutes
|
||||
Assert.Equal(20, stats.AvgDurationMinutes);
|
||||
// Same Succeeded population feeds the percentile stats: sorted [600, 1800].
|
||||
Assert.Equal(10, stats.MedianDurationMinutes);
|
||||
Assert.Equal(30, stats.P90DurationMinutes);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Median_duration_is_robust_against_outliers_that_dominate_the_average()
|
||||
{
|
||||
// 8 typical 10-min enrollments plus a 2 h and a 24 h outlier: the mean explodes
|
||||
// to ~2.7 h while the median still answers "how long does a typical enrollment
|
||||
// take" — the reason the dashboard card leads with the median.
|
||||
var now = DateTime.UtcNow;
|
||||
var sessions = new List<SessionSummary>();
|
||||
for (int i = 0; i < 8; i++)
|
||||
sessions.Add(MakeSession(SessionStatus.Succeeded, now.AddHours(-1), 600));
|
||||
sessions.Add(MakeSession(SessionStatus.Succeeded, now.AddHours(-2), 7200));
|
||||
sessions.Add(MakeSession(SessionStatus.Succeeded, now.AddHours(-3), 86400));
|
||||
|
||||
var stats = TableStorageService.AggregateSessionStats(sessions, days: 7);
|
||||
|
||||
Assert.Equal(164, stats.AvgDurationMinutes); // (8*600 + 7200 + 86400) / 10 / 60
|
||||
Assert.Equal(10, stats.MedianDurationMinutes); // unmoved by the tail
|
||||
Assert.Equal(120, stats.P90DurationMinutes); // tail stays visible
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
+3
-1
@@ -172,7 +172,9 @@ public class SessionStatsProjectionEquivalenceTests
|
||||
Assert.Equal(full.SucceededLastNDays, projected.SucceededLastNDays);
|
||||
Assert.Equal(full.FailedLastNDays, projected.FailedLastNDays);
|
||||
Assert.Equal(full.SuccessRatePct, projected.SuccessRatePct);
|
||||
Assert.Equal(full.AvgDurationMinutes, projected.AvgDurationMinutes); // the duration-sensitive card
|
||||
Assert.Equal(full.AvgDurationMinutes, projected.AvgDurationMinutes); // duration-sensitive
|
||||
Assert.Equal(full.MedianDurationMinutes, projected.MedianDurationMinutes); // the headline card
|
||||
Assert.Equal(full.P90DurationMinutes, projected.P90DurationMinutes);
|
||||
Assert.Equal(full.TotalToday, projected.TotalToday);
|
||||
Assert.Equal(full.FailedToday, projected.FailedToday);
|
||||
}
|
||||
|
||||
@@ -404,10 +404,12 @@ public class TimeAttributionCalculatorTests
|
||||
Assert.Equal(0, b.BlockingAppCount);
|
||||
}
|
||||
|
||||
// ── retry: occupancy spans to the LAST terminal event (audit Q1) ────────
|
||||
// ── retry: every ATTEMPT counts as active time, idle gaps between them don't ──
|
||||
// (2026-08 attempt-duration change: per-pass segment pairing replaced the old
|
||||
// first-start → last-terminal hull.)
|
||||
|
||||
[Fact]
|
||||
public void Retry_IntervalRunsFromFirstStart_ToLastTerminalEvent()
|
||||
public void Retry_ActiveSegmentsSummed_GapBetweenAttemptsExcluded()
|
||||
{
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
@@ -426,10 +428,41 @@ public class TimeAttributionCalculatorTests
|
||||
|
||||
AssertExactPartition(b);
|
||||
var interval = Assert.Single(b.BlockingApps);
|
||||
// First started (+6m) → LAST terminal (+12m) = 360 s. The payload timing would have
|
||||
// frozen at the first failure (+8m) — event timestamps are the only valid source.
|
||||
Assert.Equal(360, interval.Seconds);
|
||||
Assert.Equal(360, b.EspAppsOccupancySeconds);
|
||||
// Failed attempt (+6m→+8m = 120 s) + retry (+9m→+12m = 180 s) = 300 s active; the
|
||||
// idle minute between the attempts is not this app's time. The failed attempt still
|
||||
// counts — the payload timing would have frozen at the first failure (audit Q1).
|
||||
Assert.Equal(300, interval.Seconds);
|
||||
// Hull endpoints stay first-start → last-terminal for display/what-if.
|
||||
Assert.Equal(T0.AddMinutes(6), interval.StartUtc);
|
||||
Assert.Equal(T0.AddMinutes(12), interval.EndUtc);
|
||||
Assert.Equal(300, b.EspAppsOccupancySeconds);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TwoPasses_OccupancyCountsActiveSegments_NotTheInterPassGap()
|
||||
{
|
||||
// Field case 44862cd3 ("Set TimeZone"): evaluation pass ends Skipped, the real
|
||||
// install runs much later. The app occupies the critical path only while the IME
|
||||
// actively processes it — the gap belongs to whatever installed in between.
|
||||
var events = new List<EnrollmentEvent>
|
||||
{
|
||||
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
|
||||
Evt(T0.AddMinutes(1), "esp_config_detected", data: EspLists(win32Ids: new[] { AppA })),
|
||||
Evt(T0.AddMinutes(2), "esp_phase_changed", EnrollmentPhase.AppsDevice),
|
||||
Evt(T0.AddMinutes(6), "app_install_started", data: AppData(AppA, "App A")),
|
||||
Evt(T0.AddMinutes(7), "app_install_completed", data: AppData(AppA, "App A", "Skipped")),
|
||||
Evt(T0.AddMinutes(40), "app_install_started", data: AppData(AppA, "App A")), // real install pass
|
||||
Evt(T0.AddMinutes(41), "app_install_completed", data: AppData(AppA, "App A", "Installed")),
|
||||
Evt(T0.AddMinutes(45), "enrollment_complete"),
|
||||
};
|
||||
|
||||
var b = TimeAttributionCalculator.Compute(
|
||||
Input(events, completedAt: T0.AddMinutes(45), durationSeconds: 2700))!;
|
||||
|
||||
AssertExactPartition(b);
|
||||
var interval = Assert.Single(b.BlockingApps);
|
||||
Assert.Equal(120, interval.Seconds); // 60 s eval + 60 s install — not 2100 s hull
|
||||
Assert.Equal(120, b.EspAppsOccupancySeconds); // gap not on the critical path either
|
||||
}
|
||||
|
||||
// ── reboots: spans from event gaps, never the detection event's stamp ───
|
||||
|
||||
+138
@@ -0,0 +1,138 @@
|
||||
using AutopilotMonitor.Functions.Services.Vulnerability;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins <see cref="VulnerabilityCorrelateHandler.BuildVulnerabilityReportEvent"/> — the
|
||||
/// backend-materialized <c>vulnerability_report</c> event. The load-bearing logic is the
|
||||
/// Sequence assignment: the correlation runs async off a queue, so without one the event
|
||||
/// defaulted to Sequence 0 and sorted to the TOP of the timeline (a −13m time-jump badge
|
||||
/// on a report that actually arrived last). Default is max+1 (sorts LAST, same contract as
|
||||
/// BuildSessionTimeoutEvent / BuildAgentEmergencyBreakEvent); WhiteGlove Part 1 instead
|
||||
/// anchors at the Part-1 terminal markers so a late Part-1 report can never cross the
|
||||
/// web timeline's sequence-based Pre-Provisioning / User Enrollment split and render as
|
||||
/// the start of User Enrollment.
|
||||
/// </summary>
|
||||
public class VulnerabilityReportEventSequenceTests
|
||||
{
|
||||
private const string TenantId = "a1b2c3d4-e5f6-7890-abcd-ef1234567890";
|
||||
private const string SessionId = "b2c3d4e5-f6a7-8901-bcde-f12345678901";
|
||||
|
||||
private static readonly DateTime NowUtc = new(2026, 8, 21, 13, 25, 46, DateTimeKind.Utc);
|
||||
|
||||
private static Dictionary<string, object> Report(string risk = "high") => new()
|
||||
{
|
||||
["scan_summary"] = new Dictionary<string, object> { ["overall_risk"] = risk },
|
||||
["findings"] = new List<Dictionary<string, object>>(),
|
||||
};
|
||||
|
||||
private static EnrollmentEvent Build(
|
||||
string? phaseLabel, IReadOnlyList<EnrollmentEvent> existing, Dictionary<string, object>? report = null)
|
||||
=> VulnerabilityCorrelateHandler.BuildVulnerabilityReportEvent(
|
||||
TenantId, SessionId, report ?? Report(), phaseLabel, existing, NowUtc);
|
||||
|
||||
[Fact]
|
||||
public void NonWhiteGlove_SortsOnePastTheSessionMax()
|
||||
{
|
||||
var existing = new List<EnrollmentEvent>
|
||||
{
|
||||
new() { Sequence = 3 },
|
||||
new() { Sequence = 41 }, // max — even though out of array order
|
||||
new() { Sequence = 17 },
|
||||
};
|
||||
|
||||
Assert.Equal(42, Build(phaseLabel: null, existing).Sequence);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sequence_IsOne_WhenNoPriorEvents()
|
||||
{
|
||||
Assert.Equal(1, Build(null, new List<EnrollmentEvent>()).Sequence);
|
||||
Assert.Equal(1, Build(null, null!).Sequence);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WhiteGlovePart1_AnchorsAtPart1Markers_NotPastThePart2Boot()
|
||||
{
|
||||
// Part 2 already running when the async Part-1 scan lands. The web split is
|
||||
// agent_started(50) - 1 = 49; max+1 would be 61 and cross it.
|
||||
var existing = new List<EnrollmentEvent>
|
||||
{
|
||||
new() { Sequence = 10, EventType = "esp_phase_changed" },
|
||||
new() { Sequence = 20, EventType = "whiteglove_complete" },
|
||||
new() { Sequence = 30, EventType = "software_inventory_analysis" },
|
||||
new() { Sequence = 40, EventType = "whiteglove_part1_complete" },
|
||||
new() { Sequence = 50, EventType = "agent_started" }, // Part-2 boot
|
||||
new() { Sequence = 60, EventType = "whiteglove_resumed" },
|
||||
};
|
||||
|
||||
var evt = Build("device_setup", existing);
|
||||
|
||||
// Anchored at the highest Part-1 terminal marker; the timestamp tie-break in the
|
||||
// web sort places it right behind that marker, still inside Pre-Provisioning.
|
||||
Assert.Equal(40, evt.Sequence);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WhiteGlovePart1_UsesWhiteGloveComplete_WhenPart1CompleteMarkerMissing()
|
||||
{
|
||||
var existing = new List<EnrollmentEvent>
|
||||
{
|
||||
new() { Sequence = 20, EventType = "whiteglove_complete" },
|
||||
new() { Sequence = 50, EventType = "agent_started" },
|
||||
};
|
||||
|
||||
Assert.Equal(20, Build("device_setup", existing).Sequence);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WhiteGlovePart1_FallsBackToMaxPlusOne_WithoutAnyMarker()
|
||||
{
|
||||
// No marker ⇒ the web derives no split either, so sorting last is safe.
|
||||
var existing = new List<EnrollmentEvent> { new() { Sequence = 7, EventType = "agent_started" } };
|
||||
|
||||
Assert.Equal(8, Build("device_setup", existing).Sequence);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WhiteGlovePart2_SortsOnePastTheSessionMax()
|
||||
{
|
||||
var existing = new List<EnrollmentEvent>
|
||||
{
|
||||
new() { Sequence = 40, EventType = "whiteglove_part1_complete" },
|
||||
new() { Sequence = 90, EventType = "esp_phase_changed" },
|
||||
};
|
||||
|
||||
Assert.Equal(91, Build("user_enrollment", existing).Sequence);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Shape_IsUnchangedByTheSequenceFix()
|
||||
{
|
||||
var evt = Build(null, new List<EnrollmentEvent>());
|
||||
|
||||
Assert.Equal("vulnerability_report", evt.EventType);
|
||||
Assert.Equal(AutopilotMonitor.Shared.Constants.EventTypes.VulnerabilityReport, evt.EventType);
|
||||
Assert.Equal("VulnerabilityCorrelationService", evt.Source);
|
||||
Assert.Equal(EventSeverity.Error, evt.Severity); // risk=high → Error
|
||||
// Non-phase event: must stay Unknown so the UI timeline doesn't render a phantom phase boundary.
|
||||
Assert.Equal(EnrollmentPhase.Unknown, evt.Phase);
|
||||
Assert.Equal(NowUtc, evt.Timestamp);
|
||||
Assert.Equal(TenantId, evt.TenantId);
|
||||
Assert.Equal(SessionId, evt.SessionId);
|
||||
Assert.Equal("Vulnerability scan complete (risk=high).", evt.Message);
|
||||
Assert.Equal(true, evt.Data["reportAvailable"]);
|
||||
Assert.Equal("single", evt.Data["phase"]);
|
||||
Assert.IsType<Dictionary<string, object>>(evt.Data["scan_summary"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Data_Phase_CarriesTheLabel()
|
||||
{
|
||||
var existing = new List<EnrollmentEvent> { new() { Sequence = 20, EventType = "whiteglove_complete" } };
|
||||
|
||||
Assert.Equal("device_setup", Build("device_setup", existing).Data["phase"]);
|
||||
Assert.Equal("user_enrollment", Build("user_enrollment", existing).Data["phase"]);
|
||||
}
|
||||
}
|
||||
+4
-1
@@ -131,9 +131,12 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
earliestEventTimestamp, latestEventTimestamp, currentPhase,
|
||||
platformScriptIncrement, remediationScriptIncrement, rebootIncrement);
|
||||
|
||||
public Task ReconcileSessionCountersAsync(string tenantId, string sessionId)
|
||||
public Task<SessionSkewScan?> ReconcileSessionCountersAsync(string tenantId, string sessionId)
|
||||
=> _storage.ReconcileSessionCountersAsync(tenantId, sessionId);
|
||||
|
||||
public Task<Dictionary<string, int>> GetImeOffsetOriginHistogramAsync(string tenantId, string sessionId, int maxRows = 500)
|
||||
=> _storage.GetImeOffsetOriginHistogramAsync(tenantId, sessionId, maxRows);
|
||||
|
||||
public Task UpdateSessionDiagnosticsBlobAsync(
|
||||
string tenantId, string sessionId, string blobName, string? destination = null)
|
||||
=> _storage.UpdateSessionDiagnosticsBlobAsync(tenantId, sessionId, blobName, destination);
|
||||
|
||||
@@ -137,6 +137,18 @@ namespace AutopilotMonitor.Functions.Functions.Admin
|
||||
|
||||
private async Task<(int deleted, int written, int orphanStatesGcd, int sunsetSkipped)> ReseedGatherAsync(List<AutopilotMonitor.Shared.Models.GatherRule> rules)
|
||||
{
|
||||
// Global-partition rules must stay inside the reserved built-in namespace — an ID
|
||||
// outside it (e.g. GATHER-CUSTOM-*) would silently shadow every tenant's same-ID
|
||||
// custom rule at merge time (global wins). Reject instead of import.
|
||||
var rejectedGather = rules.Where(r => !AutopilotMonitor.Shared.Models.RuleIdPolicy.IsReservedBuiltInId(r.RuleId)).ToList();
|
||||
if (rejectedGather.Count > 0)
|
||||
{
|
||||
_logger.LogError(
|
||||
"GitHub reseed gather: rejected {Count} rule(s) outside the reserved built-in namespace: {Ids}",
|
||||
rejectedGather.Count, string.Join(", ", rejectedGather.Select(r => r.RuleId)));
|
||||
rules = rules.Except(rejectedGather).ToList();
|
||||
}
|
||||
|
||||
var existing = await _ruleRepo.GetGatherRulesAsync("global");
|
||||
var newCatalogIds = rules.Select(r => r.RuleId).ToHashSet(System.StringComparer.Ordinal);
|
||||
// A rule the deployed binary already ships IDENTICALLY is embedded-provenance; one whose
|
||||
@@ -199,6 +211,18 @@ namespace AutopilotMonitor.Functions.Functions.Admin
|
||||
|
||||
private async Task<(int deleted, int written, int orphanStatesGcd, int sunsetSkipped)> ReseedAnalyzeAsync(List<AutopilotMonitor.Shared.Models.AnalyzeRule> rules)
|
||||
{
|
||||
// Global-partition rules must stay inside the reserved built-in namespace — an ID
|
||||
// outside it (e.g. ANALYZE-CUSTOM-*) would silently shadow every tenant's same-ID
|
||||
// custom rule at merge time (global wins). Reject instead of import.
|
||||
var rejectedAnalyze = rules.Where(r => !AutopilotMonitor.Shared.Models.RuleIdPolicy.IsReservedBuiltInId(r.RuleId)).ToList();
|
||||
if (rejectedAnalyze.Count > 0)
|
||||
{
|
||||
_logger.LogError(
|
||||
"GitHub reseed analyze: rejected {Count} rule(s) outside the reserved built-in namespace: {Ids}",
|
||||
rejectedAnalyze.Count, string.Join(", ", rejectedAnalyze.Select(r => r.RuleId)));
|
||||
rules = rules.Except(rejectedAnalyze).ToList();
|
||||
}
|
||||
|
||||
var existing = await _ruleRepo.GetAnalyzeRulesAsync("global");
|
||||
var newCatalogIds = rules.Select(r => r.RuleId).ToHashSet(System.StringComparer.Ordinal);
|
||||
// A rule the deployed binary already ships IDENTICALLY is embedded-provenance; one whose
|
||||
|
||||
@@ -513,7 +513,10 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
exitCode = s.ExitCode,
|
||||
attemptNumber = s.AttemptNumber,
|
||||
startedAt = s.StartedAt,
|
||||
durationSeconds = s.DurationSeconds
|
||||
durationSeconds = s.DurationSeconds,
|
||||
// 2+ = the IME processed this app in multiple passes (device-ESP
|
||||
// evaluation + real install) — explains a completion far after startedAt.
|
||||
installPassCount = s.InstallPassCount
|
||||
};
|
||||
}).ToList();
|
||||
|
||||
@@ -606,16 +609,14 @@ namespace AutopilotMonitor.Functions.Functions.Apps
|
||||
return date.AddDays(-diff);
|
||||
}
|
||||
|
||||
// Internal so AppVersionRegressionRadar shares the same nearest-rank convention
|
||||
// (a third percentile implementation would invite subtle median drift).
|
||||
// Internal so AppVersionRegressionRadar shares the same nearest-rank convention.
|
||||
// Thin delegation onto the single shared implementation (PercentileMath) — parallel
|
||||
// implementations would invite subtle median drift.
|
||||
internal static int Percentile(List<int> values, double percentile)
|
||||
{
|
||||
if (values.Count == 0) return 0;
|
||||
var sorted = values.OrderBy(v => v).ToList();
|
||||
var rank = (int)Math.Ceiling(percentile * sorted.Count) - 1;
|
||||
if (rank < 0) rank = 0;
|
||||
if (rank >= sorted.Count) rank = sorted.Count - 1;
|
||||
return sorted[rank];
|
||||
var sorted = values.Select(v => (double)v).OrderBy(v => v).ToList();
|
||||
return (int)Helpers.PercentileMath.Percentile(sorted, percentile);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -274,7 +274,7 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new AgentConfigResponse
|
||||
{
|
||||
ConfigVersion = 37, // EnableEspContinueAnywayObservation (Continue-Anyway observation mode)
|
||||
ConfigVersion = 38, // Collectors.EnablePowerStateWatcher (live AC/battery watcher)
|
||||
UploadIntervalSeconds = Shared.Constants.DefaultUploadIntervalSeconds,
|
||||
SelfDestructOnComplete = tenantConfig.SelfDestructOnComplete ?? true,
|
||||
KeepLogFile = tenantConfig.KeepLogFile ?? false,
|
||||
|
||||
@@ -221,10 +221,17 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
|
||||
// EventIngestProcessor, saving one Sessions point-read per batch.
|
||||
var preFetchedStatus = TryReadSessionStatus(guardSessionRow);
|
||||
|
||||
// P14: device-clock send time of this upload (one value per request). Separates
|
||||
// spool delay from device-vs-server clock offset downstream; absent/garbage → null.
|
||||
var sentAtHeader = req.Headers.Contains("X-Send-Time-Utc")
|
||||
? req.Headers.GetValues("X-Send-Time-Utc").FirstOrDefault()
|
||||
: null;
|
||||
var sentAt = ParseSendTimeHeader(sentAtHeader);
|
||||
|
||||
// Partition + persist. Events are routed through EventIngestProcessor which runs the
|
||||
// full pipeline (rule engine / app-install aggregation / SignalR / webhooks / ...);
|
||||
// Signal + Transition go straight to their repositories.
|
||||
var outcome = await PersistItemsAsync(items, bodyTenantId, sessionId, validation, preFetchedStatus);
|
||||
var outcome = await PersistItemsAsync(items, bodyTenantId, sessionId, validation, preFetchedStatus, sentAt);
|
||||
|
||||
_logger.LogInformation(
|
||||
"IngestTelemetry: tenant={Tenant} session={Session} events={E} signals={S} transitions={T} unknown={U}",
|
||||
@@ -349,7 +356,8 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
|
||||
string tenantId,
|
||||
string sessionId,
|
||||
SecurityValidationResult validation,
|
||||
SessionStatus? preFetchedStatus)
|
||||
SessionStatus? preFetchedStatus,
|
||||
DateTime? sentAt)
|
||||
{
|
||||
var events = new List<EnrollmentEvent>();
|
||||
var signals = new List<SignalRecord>();
|
||||
@@ -435,6 +443,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
|
||||
SessionId = sessionId,
|
||||
TenantId = tenantId,
|
||||
Events = events,
|
||||
SentAt = sentAt,
|
||||
};
|
||||
var processed = await _eventProcessor.ProcessEventsAsync(eventRequest, validation, preFetchedStatus);
|
||||
|
||||
@@ -570,6 +579,29 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
|
||||
return false;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Parses the <c>X-Send-Time-Utc</c> header (agent device-clock send time, ISO-8601
|
||||
/// round-trip). Returns null for absent/unparseable values and for anything before
|
||||
/// year 2000 — Table Storage rejects pre-1601 DateTimes outright, and a sub-2000
|
||||
/// value can only be garbage, never a measurable clock offset. Deliberately NO upper
|
||||
/// clamp: a future-dated send time IS the device-clock-error measurement this field
|
||||
/// exists to capture.
|
||||
/// </summary>
|
||||
internal static DateTime? ParseSendTimeHeader(string? headerValue)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(headerValue)) return null;
|
||||
|
||||
if (!DateTimeOffset.TryParse(
|
||||
headerValue,
|
||||
System.Globalization.CultureInfo.InvariantCulture,
|
||||
System.Globalization.DateTimeStyles.RoundtripKind,
|
||||
out var parsed))
|
||||
return null;
|
||||
|
||||
var utc = parsed.UtcDateTime;
|
||||
return utc.Year >= 2000 ? utc : (DateTime?)null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// PartitionKey convention is <c>{tenantId}_{sessionId}</c> — both GUIDs with dashes, so
|
||||
/// splitting on the single underscore between them is unambiguous. Returns false for any
|
||||
|
||||
@@ -178,9 +178,90 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
|
||||
|
||||
await MaterializeEmergencyBreakArtifactsAsync(report, tenantId, _sessionRepo, _opsEventService, _logger);
|
||||
|
||||
var adminConfig = await _adminConfigService.GetConfigurationAsync();
|
||||
await MaterializeIntegrityMismatchAsync(
|
||||
report, tenantId, _opsEventService,
|
||||
adminConfig?.LatestAgentV2Version, adminConfig?.LatestAgentV2ExeSha256, _logger);
|
||||
|
||||
return req.CreateResponse(HttpStatusCode.OK);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Regex over OUR OWN report format (AgentRuntimeConfig.VerifyBinaryIntegrity):
|
||||
/// "... actual=<sha256>, expected=<sha256>". Parsed leniently — an unparseable
|
||||
/// message records rather than suppresses.
|
||||
/// </summary>
|
||||
private static readonly System.Text.RegularExpressions.Regex ActualHashRegex =
|
||||
new(@"actual=([0-9a-fA-F]{64})", System.Text.RegularExpressions.RegexOptions.Compiled);
|
||||
|
||||
/// <summary>
|
||||
/// Surfaces an <c>IntegrityCheckFailed</c> report as an <c>AgentBinaryIntegrityMismatch</c>
|
||||
/// ops event. Before 2026-08-20 this report went to App Insights only — session e9753578's
|
||||
/// agent reported a hash mismatch and no product surface showed it, which cost a day of
|
||||
/// mis-attributed root-causing.
|
||||
///
|
||||
/// <para>
|
||||
/// KNOWN, ACCEPTED RACE (must not spam the ops feed): around every release, agents compare
|
||||
/// against a hash snapshot that does not belong to their own binary — a freshly installed
|
||||
/// latest agent can receive the PREVIOUS release's hashes from the 5-minute
|
||||
/// AdminConfiguration cache (e9753578: genuine 1410 exe vs cached 1409 hash), and an
|
||||
/// older agent's next config fetch sees the NEW release's hash. Both self-heal via the
|
||||
/// forced self-update. Suppression, evaluated against the CURRENT config at receive time:
|
||||
/// the reported RUNNING hash equals the published exe (stale expectation), or the
|
||||
/// reporting agent's version is not the latest (update in flight). What remains — an agent
|
||||
/// claiming the latest version whose binary is NOT the published one — is the real signal:
|
||||
/// tamper, stale blob, or a build that never came from the release pipeline.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// Ops event only, no timeline event: the report is about the BINARY, not the enrollment,
|
||||
/// and may arrive before the session is registered. No idempotency read: the agent-side
|
||||
/// trigger is single-shot per process. Best-effort — a failure here must never turn the
|
||||
/// always-200 channel into a retry loop.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal static async Task MaterializeIntegrityMismatchAsync(
|
||||
AgentErrorReport report,
|
||||
string tenantId,
|
||||
OpsEventService opsEventService,
|
||||
string? latestAgentVersion,
|
||||
string? latestAgentExeSha256,
|
||||
ILogger logger)
|
||||
{
|
||||
if (report.ErrorType != AgentErrorType.IntegrityCheckFailed)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
// "2.0.1410+3b59dab2..." → "2.0.1410" (the manifest/AdminConfig carry no +commit).
|
||||
var reportedVersion = (report.AgentVersion ?? string.Empty).Split('+')[0];
|
||||
if (!string.IsNullOrEmpty(latestAgentVersion)
|
||||
&& !string.IsNullOrEmpty(reportedVersion)
|
||||
&& !string.Equals(reportedVersion, latestAgentVersion, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return; // older agent vs newer release — the forced self-update heals this.
|
||||
}
|
||||
|
||||
var actualMatch = ActualHashRegex.Match(report.Message ?? string.Empty);
|
||||
if (actualMatch.Success
|
||||
&& !string.IsNullOrEmpty(latestAgentExeSha256)
|
||||
&& string.Equals(actualMatch.Groups[1].Value, latestAgentExeSha256, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
return; // the running binary IS the published one — the agent's expectation was stale.
|
||||
}
|
||||
|
||||
await opsEventService.RecordAgentBinaryIntegrityMismatchAsync(
|
||||
tenantId, report.SessionId, report.AgentVersion, report.Message ?? string.Empty);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.LogWarning(ex,
|
||||
"ReportAgentError: failed to record AgentBinaryIntegrityMismatch ops event for session {SessionId}", report.SessionId);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Materializes the agent's silent 48h emergency break as (1) a timeline event so it shows
|
||||
/// in the session and the timeout classifier can see it, (2) the cross-session EventType
|
||||
|
||||
+18
-1
@@ -128,7 +128,12 @@ namespace AutopilotMonitor.Functions.Functions.Rules
|
||||
internal static readonly HashSet<string> KnownSources = new(StringComparer.Ordinal)
|
||||
{
|
||||
"event_type", "event_data", "event_data_array", "event_count",
|
||||
"phase_duration", "app_install_duration", "event_correlation",
|
||||
"phase_duration", "app_install_duration", "event_correlation", "clock_skew",
|
||||
};
|
||||
|
||||
internal static readonly HashSet<string> KnownSkewMetrics = new(StringComparer.Ordinal)
|
||||
{
|
||||
"clock_jump", "sustained_offset",
|
||||
};
|
||||
|
||||
internal static readonly HashSet<string> KnownOperators = new(StringComparer.OrdinalIgnoreCase)
|
||||
@@ -196,6 +201,18 @@ namespace AutopilotMonitor.Functions.Functions.Rules
|
||||
errors.Add($"{label}: event_correlation requires joinField");
|
||||
}
|
||||
|
||||
if (string.Equals(c.Source, "clock_skew", StringComparison.Ordinal))
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(c.SkewMetric) || !KnownSkewMetrics.Contains(c.SkewMetric))
|
||||
errors.Add($"{label}: clock_skew requires skewMetric (one of: {string.Join(", ", KnownSkewMetrics)})");
|
||||
if (!string.Equals(c.Operator, "gt", StringComparison.OrdinalIgnoreCase)
|
||||
&& !string.Equals(c.Operator, "gte", StringComparison.OrdinalIgnoreCase))
|
||||
errors.Add($"{label}: clock_skew supports only gt/gte (magnitude vs threshold)");
|
||||
if (!double.TryParse(c.Value, System.Globalization.NumberStyles.Float,
|
||||
System.Globalization.CultureInfo.InvariantCulture, out var skewThreshold) || skewThreshold <= 0)
|
||||
errors.Add($"{label}: clock_skew requires a positive numeric value (threshold in seconds)");
|
||||
}
|
||||
|
||||
if ((string.Equals(c.Operator, "regex", StringComparison.OrdinalIgnoreCase)
|
||||
|| string.Equals(c.Operator, "not_regex", StringComparison.OrdinalIgnoreCase))
|
||||
&& !string.IsNullOrEmpty(c.Value))
|
||||
|
||||
@@ -35,7 +35,12 @@ public sealed class AppVersionDurationRegressionFinding
|
||||
/// </list>
|
||||
/// Measured population: Succeeded, non-skip terminal state, plausible observed duration
|
||||
/// (<see cref="MetricsMath.HasMeasuredDuration"/>), no AppId collision, non-empty
|
||||
/// AppVersion. Version ORDER comes from first-seen install time (min StartedAt) — version
|
||||
/// AppVersion, and — cutover gate of the 2026-08 attempt-duration change — an
|
||||
/// attempt-anchored duration (<c>LastAttemptStartedAt</c> present). Rows from before the
|
||||
/// change carry the old full-span semantics; a median over a mixed population would both
|
||||
/// mask real regressions (inflated previous vs. attempt-short current) and falsely re-arm
|
||||
/// active episodes, so pre-cutover rows are excluded on BOTH sides of every comparison.
|
||||
/// Version ORDER comes from first-seen install time (min StartedAt) — version
|
||||
/// strings are never sorted lexically ("9.1" vs "2024.10" would lie). "Previous" is the
|
||||
/// version with the latest first-seen strictly before the current version's first-seen;
|
||||
/// with parallel ring rollouts that can be a concurrently-deployed version — accepted for
|
||||
@@ -149,6 +154,9 @@ public static class AppVersionRegressionRadar
|
||||
if (summary.Status != "Succeeded") continue;
|
||||
if (MetricsMath.IsSkipTerminalState(summary)) continue;
|
||||
if (!MetricsMath.HasMeasuredDuration(summary)) continue;
|
||||
// Attempt-semantics cutover gate (see class doc): only attempt-anchored rows
|
||||
// enter the duration population, on both sides of every comparison.
|
||||
if (!summary.LastAttemptStartedAt.HasValue) continue;
|
||||
|
||||
if (!groups.TryGetValue(summary.AppName, out var list))
|
||||
{
|
||||
|
||||
@@ -63,7 +63,7 @@ namespace AutopilotMonitor.Functions.Helpers
|
||||
private static readonly HashSet<string> KnownFieldKeys = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
"eventId", "sessionId", "tenantId", "eventType", "severity", "source", "phase",
|
||||
"phaseName", "timestamp", "receivedAt", "message", "sequence", "rowKey",
|
||||
"phaseName", "timestamp", "receivedAt", "sentAt", "message", "sequence", "rowKey",
|
||||
"originalTimestamp", "timestampClamped", "causedByTransitionStepIndex",
|
||||
"causedBySignalOrdinal", "data",
|
||||
};
|
||||
@@ -86,6 +86,7 @@ namespace AutopilotMonitor.Functions.Helpers
|
||||
if (fields.Contains("phaseName")) dict["phaseName"] = e.PhaseName;
|
||||
if (fields.Contains("timestamp")) dict["timestamp"] = e.Timestamp;
|
||||
if (fields.Contains("receivedAt")) dict["receivedAt"] = e.ReceivedAt;
|
||||
if (fields.Contains("sentAt")) dict["sentAt"] = e.SentAt;
|
||||
if (fields.Contains("message")) dict["message"] = e.Message;
|
||||
if (fields.Contains("sequence")) dict["sequence"] = e.Sequence;
|
||||
if (fields.Contains("rowKey")) dict["rowKey"] = e.RowKey;
|
||||
|
||||
@@ -35,11 +35,14 @@ public static class MetricsMath
|
||||
/// <summary>
|
||||
/// Upper plausibility bound for an observed install duration: 6 h, the agent's default
|
||||
/// max observation lifetime (AgentMaxLifetimeMinutes = 360) — nothing longer can have
|
||||
/// been continuously watched. Rows above it are emission artifacts, not slow installs:
|
||||
/// a batch emission at session end pins CompletedAt far past the real finish (verified
|
||||
/// in production 2026-07-27: app rows carrying 60 175 s in a session whose own wall
|
||||
/// clock was 2 697 s; 490 rows > 6 h overall, worst 117 days — dominating the
|
||||
/// slowest-apps ranking with six-figure averages).
|
||||
/// been continuously watched. Historically this mostly caught full-span pathologies
|
||||
/// (pre-2026-08 rows measured first observation → last terminal across all install
|
||||
/// passes; verified in production 2026-07-27: app rows carrying 60 175 s in a session
|
||||
/// whose own wall clock was 2 697 s; 490 rows > 6 h overall, worst 117 days). Since
|
||||
/// the 2026-08 attempt-duration change DurationSeconds measures the last attempt, so
|
||||
/// far fewer rows trip this bound — it remains as the guard against back-stamped
|
||||
/// completions, where a single ATTEMPT longer than the agent's own lifetime is still
|
||||
/// impossible to have observed.
|
||||
/// </summary>
|
||||
public const int MaxPlausibleInstallDurationSeconds = 6 * 3600;
|
||||
|
||||
@@ -168,14 +171,15 @@ public static class MetricsMath
|
||||
/// drained up to 200k raw sessions into the browser and ran these aggregations on the main
|
||||
/// thread. Success rate follows the SLA convention: Succeeded / (Succeeded + Failed) — finished
|
||||
/// enrollments only, so in-flight sessions and Incomplete (terminal, non-failure) never dilute
|
||||
/// it. Average duration counts every non-in-progress session that carries a positive duration
|
||||
/// (failures included).
|
||||
/// it. Duration stats (average, median, P90) count every non-in-progress session that carries a
|
||||
/// positive duration (failures included); the cards lead with the median because enrollment
|
||||
/// durations are heavily right-skewed and a handful of multi-hour outliers make the mean
|
||||
/// meaningless as a "typical enrollment" answer.
|
||||
/// </summary>
|
||||
public static FleetHealthMetrics BuildFleetHealthPayload(IReadOnlyList<SessionSummary> sessions, int days)
|
||||
{
|
||||
int succeeded = 0, failed = 0, inProgress = 0, incomplete = 0;
|
||||
long completedDurationSeconds = 0;
|
||||
int completedWithDurationCount = 0;
|
||||
var completedDurationSeconds = new List<double>();
|
||||
|
||||
foreach (var s in sessions)
|
||||
{
|
||||
@@ -188,12 +192,10 @@ public static class MetricsMath
|
||||
}
|
||||
|
||||
if (s.Status != SessionStatus.InProgress && s.DurationSeconds is int d && d > 0)
|
||||
{
|
||||
completedDurationSeconds += d;
|
||||
completedWithDurationCount++;
|
||||
}
|
||||
completedDurationSeconds.Add(d);
|
||||
}
|
||||
|
||||
completedDurationSeconds.Sort(); // Percentile() expects ascending order
|
||||
int total = sessions.Count;
|
||||
int finished = succeeded + failed;
|
||||
var stats = new FleetHealthStats
|
||||
@@ -204,9 +206,11 @@ public static class MetricsMath
|
||||
InProgress = inProgress,
|
||||
Incomplete = incomplete,
|
||||
SuccessRate = finished > 0 ? Math.Round((double)succeeded / finished * 100, 1) : 0,
|
||||
AvgDurationMinutes = completedWithDurationCount > 0
|
||||
? (int)Math.Round((double)completedDurationSeconds / completedWithDurationCount / 60.0, MidpointRounding.AwayFromZero)
|
||||
AvgDurationMinutes = completedDurationSeconds.Count > 0
|
||||
? (int)Math.Round(completedDurationSeconds.Average() / 60.0, MidpointRounding.AwayFromZero)
|
||||
: 0,
|
||||
MedianDurationMinutes = SecondsToMinutes(Percentile(completedDurationSeconds, 50)),
|
||||
P90DurationMinutes = SecondsToMinutes(Percentile(completedDurationSeconds, 90)),
|
||||
};
|
||||
|
||||
return new FleetHealthMetrics
|
||||
@@ -369,6 +373,10 @@ public static class MetricsMath
|
||||
/// rounded to one decimal place. Callers MUST pass values pre-sorted ascending.
|
||||
/// Returns 0 for an empty list.
|
||||
/// </summary>
|
||||
/// <summary>Rounds a duration in seconds to whole minutes (away from zero, matching the avg).</summary>
|
||||
public static int SecondsToMinutes(double seconds)
|
||||
=> (int)Math.Round(seconds / 60.0, MidpointRounding.AwayFromZero);
|
||||
|
||||
public static double Percentile(List<double> sortedValues, int percentile)
|
||||
{
|
||||
if (sortedValues.Count == 0) return 0;
|
||||
|
||||
@@ -0,0 +1,30 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Helpers
|
||||
{
|
||||
/// <summary>
|
||||
/// The single nearest-rank percentile implementation (rank = ceil(p·n) − 1, clamped).
|
||||
/// AppsAnalyticsHelper.Percentile and PlatformMetricsService.Percentile delegate here;
|
||||
/// keep it that way — parallel implementations invite subtle median drift between
|
||||
/// surfaces that report the "same" statistic.
|
||||
/// </summary>
|
||||
internal static class PercentileMath
|
||||
{
|
||||
/// <summary>Nearest-rank percentile over an ascending-sorted list.</summary>
|
||||
internal static double Percentile(IReadOnlyList<double> sortedValues, double percentile)
|
||||
{
|
||||
if (sortedValues.Count == 0) return 0;
|
||||
var rank = (int)Math.Ceiling(percentile * sortedValues.Count) - 1;
|
||||
return sortedValues[Math.Max(0, Math.Min(rank, sortedValues.Count - 1))];
|
||||
}
|
||||
|
||||
/// <summary>Median (p50) over unsorted values.</summary>
|
||||
internal static double Median(IEnumerable<double> values)
|
||||
{
|
||||
var sorted = values.OrderBy(v => v).ToList();
|
||||
return Percentile(sorted, 0.50);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -111,6 +111,10 @@ public static class TimeAttributionCalculator
|
||||
public string AppName = string.Empty;
|
||||
public DateTime? FirstStart;
|
||||
public DateTime? LastTerminal;
|
||||
/// <summary>Start of the currently open activity segment (start seen, terminal pending).</summary>
|
||||
public DateTime? OpenStart;
|
||||
/// <summary>Closed activity segments: one per fully-observed install pass/attempt (start → terminal).</summary>
|
||||
public List<(DateTime Start, DateTime End)> Segments = new();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -152,13 +156,15 @@ public static class TimeAttributionCalculator
|
||||
else if (blockingSets.IsTruncated)
|
||||
flags |= TimeAttributionFlags.BlockingSetTruncated;
|
||||
|
||||
var (blockingApps, blockingAppCount) = BuildBlockingAppIntervals(events, blockingSets, windows, ref flags);
|
||||
var (blockingApps, blockingAppCount, blockingAppSegments) = BuildBlockingAppIntervals(events, blockingSets, windows, ref flags);
|
||||
|
||||
int? occupancy = null;
|
||||
if (blockingSets != null && blockingSets.ListedCount > 0)
|
||||
{
|
||||
var espAppsSpans = spans.Where(s => s.SegmentKey == TimeAttributionSegments.EspApps).ToList();
|
||||
occupancy = ComputeOccupancySeconds(blockingApps, espAppsSpans);
|
||||
// Occupancy merges the per-pass SEGMENTS, not the per-app hulls — a hull spans
|
||||
// the idle gap between install passes, which is not occupied by this app.
|
||||
occupancy = ComputeOccupancySeconds(blockingAppSegments, espAppsSpans);
|
||||
}
|
||||
|
||||
var rebootSpans = BuildRebootSpans(events, windows, spans);
|
||||
@@ -403,12 +409,12 @@ public static class TimeAttributionCalculator
|
||||
|
||||
// ── blocking-app intervals + occupancy ──────────────────────────────────
|
||||
|
||||
private static (List<BlockingAppInterval> Intervals, int MatchedCount) BuildBlockingAppIntervals(
|
||||
private static (List<BlockingAppInterval> Intervals, int MatchedCount, List<BlockingAppInterval> OccupancySegments) BuildBlockingAppIntervals(
|
||||
List<EnrollmentEvent> events, EspBlockingSets? sets, List<Window> windows, ref TimeAttributionFlags flags)
|
||||
{
|
||||
var intervals = new List<BlockingAppInterval>();
|
||||
if (sets == null || sets.ListedCount == 0)
|
||||
return (intervals, 0);
|
||||
return (intervals, 0, intervals);
|
||||
|
||||
// Fold per-app activity from EVENT timestamps (never payload timing — audit Q1).
|
||||
var apps = new Dictionary<string, AppActivity>(StringComparer.OrdinalIgnoreCase);
|
||||
@@ -436,36 +442,75 @@ public static class TimeAttributionCalculator
|
||||
activity.AppName = nameObj?.ToString()?.Trim() ?? string.Empty;
|
||||
}
|
||||
|
||||
if (isStart && (!activity.FirstStart.HasValue || evt.Timestamp < activity.FirstStart.Value))
|
||||
activity.FirstStart = evt.Timestamp;
|
||||
// LAST terminal wins — an IME retry (Error → retry → Installed) extends the
|
||||
// occupancy to the final outcome instead of freezing at the first failure.
|
||||
if (isTerminal && (!activity.LastTerminal.HasValue || evt.Timestamp > activity.LastTerminal.Value))
|
||||
activity.LastTerminal = evt.Timestamp;
|
||||
if (isStart)
|
||||
{
|
||||
if (!activity.FirstStart.HasValue || evt.Timestamp < activity.FirstStart.Value)
|
||||
activity.FirstStart = evt.Timestamp;
|
||||
// A download start followed by an install start is ONE active window —
|
||||
// the earliest open start survives until a terminal closes the segment.
|
||||
activity.OpenStart ??= evt.Timestamp;
|
||||
}
|
||||
if (isTerminal)
|
||||
{
|
||||
if (!activity.LastTerminal.HasValue || evt.Timestamp > activity.LastTerminal.Value)
|
||||
activity.LastTerminal = evt.Timestamp;
|
||||
// Segment pairing (2026-08 attempt-duration change): each fully-observed
|
||||
// pass/attempt (start → terminal, sequence-ordered) is its own activity
|
||||
// segment. The IME processes the app list in multiple passes; the idle gap
|
||||
// between passes belongs to the apps actually running then, not to this one.
|
||||
// A failed attempt still counts as active time — the retry opens a new
|
||||
// segment, so the old "LAST terminal wins" retry intent is preserved.
|
||||
if (activity.OpenStart.HasValue)
|
||||
{
|
||||
if (evt.Timestamp >= activity.OpenStart.Value)
|
||||
activity.Segments.Add((activity.OpenStart.Value, evt.Timestamp));
|
||||
else
|
||||
flags |= TimeAttributionFlags.ClockSkewDropped;
|
||||
activity.OpenStart = null;
|
||||
}
|
||||
// Terminal without an observed start: no segment claim (attach window,
|
||||
// audit §0.5) — same rule as before.
|
||||
}
|
||||
}
|
||||
|
||||
var matched = 0;
|
||||
var occupancySegments = new List<BlockingAppInterval>();
|
||||
foreach (var activity in apps.Values)
|
||||
{
|
||||
if (!sets.Contains(activity.AppId)) continue; // absent ⇒ unknown, never "not blocking"
|
||||
matched++;
|
||||
|
||||
if (!activity.FirstStart.HasValue || !activity.LastTerminal.HasValue)
|
||||
continue; // unobserved endpoint → no interval claim (attach window, audit §0.5)
|
||||
if (activity.LastTerminal.Value < activity.FirstStart.Value)
|
||||
{
|
||||
flags |= TimeAttributionFlags.ClockSkewDropped;
|
||||
continue;
|
||||
}
|
||||
if (activity.Segments.Count == 0)
|
||||
continue; // no fully-observed attempt → no interval claim (attach window, audit §0.5)
|
||||
|
||||
var clamped = ClampToWindows(activity.FirstStart.Value, activity.LastTerminal.Value, windows);
|
||||
if (!clamped.HasValue)
|
||||
// Clamp each segment; the persisted per-app interval is the hull of the clamped
|
||||
// segments with Seconds = the SUM of in-window active time (not the hull span —
|
||||
// the inter-pass idle gap is not this app's time).
|
||||
var total = 0;
|
||||
DateTime? hullStart = null, hullEnd = null;
|
||||
foreach (var segment in activity.Segments)
|
||||
{
|
||||
// Outside the hull entirely → pathological timestamps (flagged, as before).
|
||||
// Inside the hull but touching no window (wholly inside the WhiteGlove pause):
|
||||
// no in-window observation → no interval claim, and nothing wrong with clocks.
|
||||
if (activity.LastTerminal.Value < windows[0].Start ||
|
||||
activity.FirstStart.Value > windows[windows.Count - 1].End)
|
||||
var clamped = ClampToWindows(segment.Start, segment.End, windows);
|
||||
if (!clamped.HasValue) continue;
|
||||
total += clamped.Value.Seconds;
|
||||
if (hullStart == null || clamped.Value.Start < hullStart.Value) hullStart = clamped.Value.Start;
|
||||
if (hullEnd == null || clamped.Value.End > hullEnd.Value) hullEnd = clamped.Value.End;
|
||||
occupancySegments.Add(new BlockingAppInterval
|
||||
{
|
||||
AppId = activity.AppId,
|
||||
AppName = activity.AppName,
|
||||
StartUtc = clamped.Value.Start,
|
||||
EndUtc = clamped.Value.End,
|
||||
Seconds = clamped.Value.Seconds,
|
||||
});
|
||||
}
|
||||
if (hullStart == null || hullEnd == null)
|
||||
{
|
||||
// No segment overlapped any window. Outside the hull entirely →
|
||||
// pathological timestamps (flagged, as before). Wholly inside the
|
||||
// WhiteGlove pause → no in-window observation, nothing wrong with clocks.
|
||||
if (activity.LastTerminal!.Value < windows[0].Start ||
|
||||
activity.FirstStart!.Value > windows[windows.Count - 1].End)
|
||||
{
|
||||
flags |= TimeAttributionFlags.ClockSkewDropped;
|
||||
}
|
||||
@@ -476,14 +521,13 @@ public static class TimeAttributionCalculator
|
||||
{
|
||||
AppId = activity.AppId,
|
||||
AppName = activity.AppName,
|
||||
StartUtc = clamped.Value.Start,
|
||||
EndUtc = clamped.Value.End,
|
||||
// In-window seconds — a WG-pause-straddling interval never counts the pause.
|
||||
Seconds = clamped.Value.Seconds,
|
||||
StartUtc = hullStart.Value,
|
||||
EndUtc = hullEnd.Value,
|
||||
Seconds = total,
|
||||
});
|
||||
}
|
||||
|
||||
return (intervals, matched);
|
||||
return (intervals, matched, occupancySegments);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
+10
-4
@@ -289,10 +289,16 @@ namespace AutopilotMonitor.Functions.Services.Analyze
|
||||
rule.Title, rule.Category, rule.Severity,
|
||||
fired, confidence).ConfigureAwait(false);
|
||||
|
||||
await _metricsRepo.IncrementRuleStatAsync(
|
||||
today, "global", rule.RuleId, "analyze",
|
||||
rule.Title, rule.Category, rule.Severity,
|
||||
fired, confidence).ConfigureAwait(false);
|
||||
// Global aggregate row is catalog-only: custom-rule IDs are tenant-chosen
|
||||
// and not unique across tenants, so a shared "global_{ruleId}" row would
|
||||
// sum unrelated tenants' counters (title/severity last-writer-wins).
|
||||
if (rule.IsBuiltIn || rule.IsCommunity)
|
||||
{
|
||||
await _metricsRepo.IncrementRuleStatAsync(
|
||||
today, "global", rule.RuleId, "analyze",
|
||||
rule.Title, rule.Category, rule.Severity,
|
||||
fired, confidence).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
|
||||
@@ -0,0 +1,107 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Services
|
||||
{
|
||||
/// <summary>
|
||||
/// Regression tripwire for the agent's CMTrace time resolution (per-line self-anchoring,
|
||||
/// docs/agent/cmtrace-time-resolution.md). Runs once per session on the terminal ingest
|
||||
/// batch, over samples the counter reconcile collected in its existing partition scan.
|
||||
///
|
||||
/// Detection: diff = median(Δ_IME) − median(Δ_other), where Δ = ReceivedAt − OccurredUtc.
|
||||
/// Shared upload/spool latency is common-mode and cancels in the difference; a timezone
|
||||
/// mis-conversion shifts only the IME side and lands on the 15-minute offset grid.
|
||||
/// Replayed old log lines produce large OFF-grid deltas — the grid-residual test is what
|
||||
/// separates the two (field measurement 2026-08-20: without it, 48 instead of 26 true hits).
|
||||
///
|
||||
/// Goal state: this never fires. Any hit is either a case the per-line anchoring does not
|
||||
/// cover or a detector bug — both are actionable findings, never noise to tune away.
|
||||
/// </summary>
|
||||
internal static class CmTraceSkewTripwire
|
||||
{
|
||||
// Same grid the agent anchors on and the same residual tolerance as its own grid
|
||||
// guard (docs/agent/cmtrace-time-resolution.md) — keep these in lockstep with the
|
||||
// agent so a future anchoring change stays traceable to this detector.
|
||||
internal const int GridMinutes = 15;
|
||||
internal const double ResidualToleranceMinutes = 2.0;
|
||||
internal const int MinSamplesPerSide = 20;
|
||||
// ReceivedAt is stamped per upload batch, not per event; a side backed by fewer
|
||||
// distinct batches yields a batch-composition artifact, not a session property.
|
||||
internal const int MinDistinctBatchesPerSide = 3;
|
||||
|
||||
internal sealed class Result
|
||||
{
|
||||
public double MedianImeDeltaMinutes { get; set; }
|
||||
public double MedianOtherDeltaMinutes { get; set; }
|
||||
public double DiffMinutes { get; set; }
|
||||
public int GridSteps { get; set; }
|
||||
public double ResidualMinutes { get; set; }
|
||||
public int ImeSampleCount { get; set; }
|
||||
public int OtherSampleCount { get; set; }
|
||||
public int ImeBatchCount { get; set; }
|
||||
public int OtherBatchCount { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pure detector: non-null exactly when the session shows a grid-aligned divergence
|
||||
/// between IME-derived and other event timestamps. All suppression inputs live in the
|
||||
/// scan itself so the decision is fully unit-testable.
|
||||
/// </summary>
|
||||
internal static Result? Evaluate(SessionSkewScan? scan)
|
||||
{
|
||||
if (scan == null)
|
||||
return null;
|
||||
if (scan.ImeDeltaMinutes.Count < MinSamplesPerSide || scan.OtherDeltaMinutes.Count < MinSamplesPerSide)
|
||||
return null;
|
||||
if (scan.ImeDistinctBatchCount < MinDistinctBatchesPerSide || scan.OtherDistinctBatchCount < MinDistinctBatchesPerSide)
|
||||
return null;
|
||||
|
||||
var medianIme = PercentileMath.Median(scan.ImeDeltaMinutes);
|
||||
var medianOther = PercentileMath.Median(scan.OtherDeltaMinutes);
|
||||
var diff = medianIme - medianOther;
|
||||
var steps = (int)Math.Round(diff / GridMinutes, MidpointRounding.AwayFromZero);
|
||||
var residual = Math.Abs(diff - steps * (double)GridMinutes);
|
||||
|
||||
if (Math.Abs(steps) < 1 || residual >= ResidualToleranceMinutes)
|
||||
return null;
|
||||
|
||||
return new Result
|
||||
{
|
||||
MedianImeDeltaMinutes = medianIme,
|
||||
MedianOtherDeltaMinutes = medianOther,
|
||||
DiffMinutes = diff,
|
||||
GridSteps = steps,
|
||||
ResidualMinutes = residual,
|
||||
ImeSampleCount = scan.ImeDeltaMinutes.Count,
|
||||
OtherSampleCount = scan.OtherDeltaMinutes.Count,
|
||||
ImeBatchCount = scan.ImeDistinctBatchCount,
|
||||
OtherBatchCount = scan.OtherDistinctBatchCount,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// True when the session's IME events are dominated by writer-declared ("bias")
|
||||
/// offsets — a bias offset comes verbatim from the log line's own zone marker and
|
||||
/// cannot be an anchoring regression, so a grid hit there is not ours to alarm on.
|
||||
/// measuredWriterOffsetMinutes is deliberately NOT consulted: the per-file
|
||||
/// observation sticks to a stale value after an era flip-back (monotonicity guard),
|
||||
/// making it non-authoritative for exactly the sessions this tripwire inspects.
|
||||
/// </summary>
|
||||
internal static bool IsBiasDominated(IReadOnlyDictionary<string, int>? originHistogram)
|
||||
{
|
||||
if (originHistogram == null || originHistogram.Count == 0)
|
||||
return false;
|
||||
|
||||
int total = 0, bias = 0;
|
||||
foreach (var kv in originHistogram)
|
||||
{
|
||||
total += kv.Value;
|
||||
if (string.Equals(kv.Key, "bias", StringComparison.OrdinalIgnoreCase))
|
||||
bias += kv.Value;
|
||||
}
|
||||
return total > 0 && bias * 2 >= total;
|
||||
}
|
||||
}
|
||||
}
|
||||
+62
-20
@@ -112,6 +112,22 @@ namespace AutopilotMonitor.Functions.Services
|
||||
case "app_install_start":
|
||||
if (!state.InstallStartedAt.HasValue || evt.Timestamp < state.InstallStartedAt.Value)
|
||||
state.InstallStartedAt = evt.Timestamp;
|
||||
// Attempt anchor: the LATEST start wins (max-fold). DurationSeconds
|
||||
// measures the status-defining attempt, not the session-wide span — the
|
||||
// IME processes the app list in multiple passes (device-ESP evaluation,
|
||||
// user-context install, retries), often an hour apart.
|
||||
if (!summary.LastAttemptStartedAt.HasValue || evt.Timestamp > summary.LastAttemptStartedAt.Value)
|
||||
summary.LastAttemptStartedAt = evt.Timestamp;
|
||||
summary.InstallPassCount++;
|
||||
// Attempt-scoped download pairing: a pending download start is consumed
|
||||
// by the next install start of the same app.
|
||||
if (state.PendingDownloadStartedAt.HasValue && evt.Timestamp >= state.PendingDownloadStartedAt.Value)
|
||||
{
|
||||
var pairedDl = EventTimestampValidator.SafeDurationSeconds(state.PendingDownloadStartedAt.Value, evt.Timestamp);
|
||||
if (pairedDl > 0)
|
||||
summary.DownloadDurationSeconds = pairedDl;
|
||||
state.PendingDownloadStartedAt = null;
|
||||
}
|
||||
if (summary.Status == "InProgress" || summary.Status == string.Empty)
|
||||
summary.Status = "InProgress";
|
||||
break;
|
||||
@@ -119,34 +135,48 @@ namespace AutopilotMonitor.Functions.Services
|
||||
case "app_download_started":
|
||||
if (!state.DownloadStartedAt.HasValue || evt.Timestamp < state.DownloadStartedAt.Value)
|
||||
state.DownloadStartedAt = evt.Timestamp;
|
||||
// Latest unconsumed download start — pairs with the NEXT install start.
|
||||
if (!state.PendingDownloadStartedAt.HasValue || evt.Timestamp > state.PendingDownloadStartedAt.Value)
|
||||
state.PendingDownloadStartedAt = evt.Timestamp;
|
||||
if (summary.Status == "InProgress" || summary.Status == string.Empty)
|
||||
summary.Status = "InProgress";
|
||||
break;
|
||||
|
||||
case "app_install_completed":
|
||||
case "app_install_complete":
|
||||
// A weaker terminal (Skipped/Postponed re-evaluation pass) never overrides
|
||||
// an Installed/Error already observed — mirrors the app_install_skipped
|
||||
// guard below. It must not move CompletedAt/DurationSeconds either: the
|
||||
// row keeps describing the attempt that produced the surviving terminal.
|
||||
var incomingTerminalState = evt.Data?.ContainsKey("state") == true
|
||||
? evt.Data["state"]?.ToString() : null;
|
||||
if ((summary.TerminalState == "Installed" || summary.TerminalState == "Error") &&
|
||||
(incomingTerminalState == "Skipped" || incomingTerminalState == "Postponed"))
|
||||
break;
|
||||
summary.Status = "Succeeded";
|
||||
summary.CompletedAt = evt.Timestamp;
|
||||
if (summary.StartedAt != DateTime.MinValue)
|
||||
summary.DurationSeconds = Math.Max(1, EventTimestampValidator.SafeDurationSeconds(summary.StartedAt, evt.Timestamp));
|
||||
var attemptStart = summary.LastAttemptStartedAt
|
||||
?? (summary.StartedAt != DateTime.MinValue ? summary.StartedAt : (DateTime?)null);
|
||||
if (attemptStart.HasValue)
|
||||
summary.DurationSeconds = Math.Max(1, EventTimestampValidator.SafeDurationSeconds(attemptStart.Value, evt.Timestamp));
|
||||
// PR0 (2026-07-26): the agent emits app_install_completed for EVERY terminal
|
||||
// transition — including Skipped/Postponed (V1 wire parity). The payload's
|
||||
// `state` field is the only way to tell a real install apart from a no-op
|
||||
// (e.g. WinGet "Update for X" policies that were not applicable). Persist it
|
||||
// so metrics can exclude skips from durations and rates. Unknown/absent state
|
||||
// leaves the sentinel empty — never guessed.
|
||||
var terminalStateRaw = evt.Data?.ContainsKey("state") == true
|
||||
? evt.Data["state"]?.ToString() : null;
|
||||
if (terminalStateRaw == "Installed" || terminalStateRaw == "Skipped" || terminalStateRaw == "Postponed")
|
||||
summary.TerminalState = terminalStateRaw;
|
||||
if (incomingTerminalState == "Installed" || incomingTerminalState == "Skipped" || incomingTerminalState == "Postponed")
|
||||
summary.TerminalState = incomingTerminalState;
|
||||
break;
|
||||
|
||||
case "app_install_failed":
|
||||
summary.Status = "Failed";
|
||||
summary.TerminalState = "Error";
|
||||
summary.CompletedAt = evt.Timestamp;
|
||||
if (summary.StartedAt != DateTime.MinValue)
|
||||
summary.DurationSeconds = Math.Max(1, EventTimestampValidator.SafeDurationSeconds(summary.StartedAt, evt.Timestamp));
|
||||
var failedAttemptStart = summary.LastAttemptStartedAt
|
||||
?? (summary.StartedAt != DateTime.MinValue ? summary.StartedAt : (DateTime?)null);
|
||||
if (failedAttemptStart.HasValue)
|
||||
summary.DurationSeconds = Math.Max(1, EventTimestampValidator.SafeDurationSeconds(failedAttemptStart.Value, evt.Timestamp));
|
||||
// FailureCode preference: canonical `failureType` > raw `errorCode` > empty.
|
||||
// c117946b debrief (2026-05-12): the V2 termination handler tags promoted
|
||||
// "likely stuck" apps with failureType=esp_apps_timeout so the UI can
|
||||
@@ -310,20 +340,30 @@ namespace AutopilotMonitor.Functions.Services
|
||||
summary.StartedAt = effectiveStart;
|
||||
}
|
||||
|
||||
// Download duration: from first download start to first install start.
|
||||
if (state.DownloadStartedAt.HasValue && state.InstallStartedAt.HasValue &&
|
||||
state.InstallStartedAt.Value >= state.DownloadStartedAt.Value)
|
||||
{
|
||||
summary.DownloadDurationSeconds = EventTimestampValidator.SafeDurationSeconds(
|
||||
state.DownloadStartedAt.Value, state.InstallStartedAt.Value);
|
||||
}
|
||||
// Download duration is paired attempt-scoped at the app_install_started event
|
||||
// (latest download start → next install start) — no span computation here.
|
||||
|
||||
// Full duration: from effective start to completion/failure.
|
||||
if (summary.CompletedAt.HasValue && summary.StartedAt != DateTime.MinValue &&
|
||||
summary.CompletedAt.Value >= summary.StartedAt)
|
||||
// Full duration: the status-defining attempt. CompletedAt pairs with the latest
|
||||
// attempt start at or before it. A LastAttemptStartedAt NEWER than CompletedAt
|
||||
// means a fresh pass is in flight — the recorded duration of the completed
|
||||
// attempt stands. Rows without any observed start (legacy agents, terminal-only
|
||||
// batches) keep the historical StartedAt (span) fallback.
|
||||
if (summary.CompletedAt.HasValue)
|
||||
{
|
||||
summary.DurationSeconds = EventTimestampValidator.SafeDurationSeconds(
|
||||
summary.StartedAt, summary.CompletedAt.Value);
|
||||
if (summary.LastAttemptStartedAt.HasValue)
|
||||
{
|
||||
if (summary.LastAttemptStartedAt.Value <= summary.CompletedAt.Value)
|
||||
{
|
||||
summary.DurationSeconds = EventTimestampValidator.SafeDurationSeconds(
|
||||
summary.LastAttemptStartedAt.Value, summary.CompletedAt.Value);
|
||||
}
|
||||
}
|
||||
else if (summary.StartedAt != DateTime.MinValue &&
|
||||
summary.CompletedAt.Value >= summary.StartedAt)
|
||||
{
|
||||
summary.DurationSeconds = EventTimestampValidator.SafeDurationSeconds(
|
||||
summary.StartedAt, summary.CompletedAt.Value);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -333,5 +373,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
public AppInstallSummary Summary { get; set; } = new();
|
||||
public DateTime? DownloadStartedAt { get; set; }
|
||||
public DateTime? InstallStartedAt { get; set; }
|
||||
/// <summary>Latest download start not yet consumed by an install start (attempt-scoped download-duration pairing).</summary>
|
||||
public DateTime? PendingDownloadStartedAt { get; set; }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -378,6 +378,9 @@ namespace AutopilotMonitor.Functions.Services
|
||||
"stall_probe_result" => true,
|
||||
"session_stalled" => true,
|
||||
"modern_deployment_log" => true,
|
||||
// Environment observation (battery drain / AC flap), not enrollment progress —
|
||||
// must not count as "real activity" that heals a Stalled session.
|
||||
"power_state_change" => true,
|
||||
_ => false
|
||||
};
|
||||
|
||||
|
||||
@@ -36,10 +36,18 @@ namespace AutopilotMonitor.Functions.Services
|
||||
ruleTitle, "", "",
|
||||
fired: true, confidenceScore: null);
|
||||
|
||||
await _metricsRepo.IncrementRuleStatAsync(
|
||||
today, "global", ruleId, "gather",
|
||||
ruleTitle, "", "",
|
||||
fired: true, confidenceScore: null);
|
||||
// Global aggregate row is catalog-only. The agent event carries no
|
||||
// IsBuiltIn flag, but custom rules can never occupy the reserved
|
||||
// built-in namespace (create/update guard), so the ID pattern is the
|
||||
// discriminator. Custom IDs are tenant-chosen and not unique across
|
||||
// tenants — a shared "global_{ruleId}" row would mix their counters.
|
||||
if (RuleIdPolicy.IsReservedBuiltInId(ruleId))
|
||||
{
|
||||
await _metricsRepo.IncrementRuleStatAsync(
|
||||
today, "global", ruleId, "gather",
|
||||
ruleTitle, "", "",
|
||||
fired: true, confidenceScore: null);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
|
||||
@@ -6,6 +6,7 @@ using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.ApplicationInsights;
|
||||
using Microsoft.Azure.Functions.Worker;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Services
|
||||
@@ -44,6 +45,10 @@ namespace AutopilotMonitor.Functions.Services
|
||||
private readonly AutopilotMonitor.Functions.Services.Analyze.InterimTriggerRegistry _interimTriggerRegistry;
|
||||
private readonly IVulnerabilityCorrelateProducer _vulnProducer;
|
||||
private readonly Ime.IImeMsiArchiveProducer _imeMsiArchiveProducer;
|
||||
private readonly IConfiguration _configuration;
|
||||
|
||||
/// <summary>App setting kill switch: set to "true" to skip the CMTrace skew tripwire entirely. Fail-open — the tripwire only notifies, it never mutates data.</summary>
|
||||
internal const string CmTraceSkewTripwireKillSwitchSetting = "CmTraceSkewTripwireDisabled";
|
||||
|
||||
public EventIngestProcessor(
|
||||
ILogger<EventIngestProcessor> logger,
|
||||
@@ -60,7 +65,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
AutopilotMonitor.Functions.Services.Analyze.IAnalyzeOnEnrollmentEndProducer analyzeProducer,
|
||||
AutopilotMonitor.Functions.Services.Analyze.InterimTriggerRegistry interimTriggerRegistry,
|
||||
IVulnerabilityCorrelateProducer vulnProducer,
|
||||
Ime.IImeMsiArchiveProducer imeMsiArchiveProducer)
|
||||
Ime.IImeMsiArchiveProducer imeMsiArchiveProducer,
|
||||
IConfiguration configuration)
|
||||
{
|
||||
_logger = logger;
|
||||
_sessionRepo = sessionRepo;
|
||||
@@ -77,6 +83,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
_interimTriggerRegistry = interimTriggerRegistry;
|
||||
_vulnProducer = vulnProducer;
|
||||
_imeMsiArchiveProducer = imeMsiArchiveProducer;
|
||||
_configuration = configuration;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -106,7 +113,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
validation.RateLimitResult?.MaxRequests);
|
||||
|
||||
var receivedAt = DateTime.UtcNow;
|
||||
StampServerFields(request.Events, request.TenantId, request.SessionId, receivedAt);
|
||||
StampServerFields(request.Events, request.TenantId, request.SessionId, receivedAt, request.SentAt);
|
||||
SanitizeEventTimestamps(request.Events, receivedAt, _logger);
|
||||
|
||||
var storedEvents = await _sessionRepo.StoreEventsBatchAsync(request.Events);
|
||||
@@ -247,7 +254,17 @@ namespace AutopilotMonitor.Functions.Services
|
||||
// replays where UpdateSessionStatusAsync no-ops.
|
||||
// Idempotent (no-ops when already correct) and fail-soft.
|
||||
if (isTerminalBatch)
|
||||
await _sessionRepo.ReconcileSessionCountersAsync(request.TenantId, request.SessionId);
|
||||
{
|
||||
var skewScan = await _sessionRepo.ReconcileSessionCountersAsync(request.TenantId, request.SessionId);
|
||||
|
||||
// CMTrace time-skew tripwire (goal state: never fires). Gated on the actual
|
||||
// status transition so terminal-batch REPLAYS (statusTransitioned == false)
|
||||
// cannot re-emit — the reconcile above runs on replays by design, the
|
||||
// tripwire must not. Fail-soft: ingest stays a 200 no matter what.
|
||||
if (skewScan != null && (statusTransitioned || whiteGloveStatusTransitioned))
|
||||
await TryFireCmTraceSkewTripwireAsync(request.TenantId, request.SessionId,
|
||||
updatedSession?.AgentVersion, skewScan);
|
||||
}
|
||||
|
||||
// Auto-analyze fan-out: enqueue a queue message instead of running fire-and-forget
|
||||
// Task.Run inside the function. The previous in-function approach could be killed
|
||||
@@ -545,18 +562,83 @@ namespace AutopilotMonitor.Functions.Services
|
||||
System.Globalization.CultureInfo.InvariantCulture, out value);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// CMTrace time-skew regression tripwire. Evaluates the per-source timestamp-delta
|
||||
/// samples the counter reconcile collected and emits a CmTraceTimeSkewRegression ops
|
||||
/// event when the IME-derived events diverge from the other sources by a clean
|
||||
/// 15-minute-grid multiple (see <see cref="CmTraceSkewTripwire"/>). One extra bounded
|
||||
/// storage read happens ONLY on the suspicion path (≈ never): the sourceOffsetOrigin
|
||||
/// histogram that rules out writer-declared ("bias") offsets, which come verbatim
|
||||
/// from the log line and cannot be an anchoring regression.
|
||||
/// </summary>
|
||||
private async Task TryFireCmTraceSkewTripwireAsync(
|
||||
string tenantId, string sessionId, string? agentVersion, SessionSkewScan skewScan)
|
||||
{
|
||||
try
|
||||
{
|
||||
if (string.Equals(_configuration[CmTraceSkewTripwireKillSwitchSetting], "true", StringComparison.OrdinalIgnoreCase))
|
||||
return;
|
||||
|
||||
var result = CmTraceSkewTripwire.Evaluate(skewScan);
|
||||
if (result == null)
|
||||
return;
|
||||
|
||||
var histogram = await _sessionRepo.GetImeOffsetOriginHistogramAsync(tenantId, sessionId);
|
||||
if (CmTraceSkewTripwire.IsBiasDominated(histogram))
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"CMTrace skew tripwire suppressed for session {SessionId}: grid divergence {DiffMinutes:+0.0;-0.0} min, but IME offsets are bias-dominated (writer-declared)",
|
||||
sessionId, result.DiffMinutes);
|
||||
return;
|
||||
}
|
||||
|
||||
var origins = string.Join(", ",
|
||||
histogram.OrderByDescending(kv => kv.Value).Select(kv => $"{kv.Key} {kv.Value}"));
|
||||
var message =
|
||||
$"IME-derived events skewed {result.DiffMinutes:+0.0;-0.0} min vs other sources " +
|
||||
$"({result.GridSteps}x15min grid, residual {result.ResidualMinutes:0.0} min; " +
|
||||
$"median dIME {result.MedianImeDeltaMinutes:0.0} / dOther {result.MedianOtherDeltaMinutes:0.0} min; " +
|
||||
$"{result.ImeSampleCount}/{result.OtherSampleCount} samples over {result.ImeBatchCount}/{result.OtherBatchCount} batches; " +
|
||||
$"origins: {(origins.Length > 0 ? origins : "none")})";
|
||||
|
||||
await _opsEventService.RecordCmTraceTimeSkewRegressionAsync(tenantId, sessionId, agentVersion, message, new
|
||||
{
|
||||
sessionId,
|
||||
agentVersion,
|
||||
diffMinutes = Math.Round(result.DiffMinutes, 2),
|
||||
gridSteps = result.GridSteps,
|
||||
residualMinutes = Math.Round(result.ResidualMinutes, 2),
|
||||
medianImeDeltaMinutes = Math.Round(result.MedianImeDeltaMinutes, 2),
|
||||
medianOtherDeltaMinutes = Math.Round(result.MedianOtherDeltaMinutes, 2),
|
||||
imeSampleCount = result.ImeSampleCount,
|
||||
otherSampleCount = result.OtherSampleCount,
|
||||
imeBatchCount = result.ImeBatchCount,
|
||||
otherBatchCount = result.OtherBatchCount,
|
||||
sourceOffsetOrigins = histogram,
|
||||
});
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "CMTrace skew tripwire failed for session {SessionId}", sessionId);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Stamps authoritative server-side fields onto all events before storage.
|
||||
/// TenantId and SessionId always come from the validated request metadata,
|
||||
/// overriding any values the agent may have sent per-event.
|
||||
/// overriding any values the agent may have sent per-event. SentAt is the
|
||||
/// request-level device-clock send time (P14) — one value for every event of
|
||||
/// the batch, null for agents that pre-date the X-Send-Time-Utc header.
|
||||
/// Exposed as internal for unit testing.
|
||||
/// </summary>
|
||||
internal static void StampServerFields(
|
||||
List<EnrollmentEvent> events, string tenantId, string sessionId, DateTime receivedAt)
|
||||
List<EnrollmentEvent> events, string tenantId, string sessionId, DateTime receivedAt,
|
||||
DateTime? sentAt = null)
|
||||
{
|
||||
foreach (var evt in events)
|
||||
{
|
||||
evt.ReceivedAt = receivedAt;
|
||||
evt.SentAt = sentAt;
|
||||
evt.TenantId = tenantId;
|
||||
evt.SessionId = sessionId;
|
||||
}
|
||||
|
||||
@@ -351,8 +351,16 @@ namespace AutopilotMonitor.Functions.Services
|
||||
int silentMarked = 0;
|
||||
foreach (var silent in silentSessions)
|
||||
{
|
||||
var lastEventAt = silent.LastEventAt ?? silent.StartedAt;
|
||||
var silentMinutes = (int)(now - lastEventAt).TotalMinutes;
|
||||
// The query is a permissive two-frame pre-filter; the decision is made
|
||||
// here on the SERVER frame. LastEventAt carries the device clock and,
|
||||
// for IME-derived events, a timezone skew (field-measured: -17 h to
|
||||
// +1 h). Trusting it marks a live agent Stalled — and puts a bogus
|
||||
// "silent for 1020min" in front of the customer.
|
||||
var lastContactAt = ServerFrameLastContact(silent);
|
||||
if (lastContactAt > silenceCutoff)
|
||||
continue; // provably alive on the server clock
|
||||
|
||||
var silentMinutes = (int)(now - lastContactAt).TotalMinutes;
|
||||
await _sessionRepo.UpdateSessionStatusAsync(
|
||||
silent.TenantId,
|
||||
silent.SessionId,
|
||||
@@ -409,7 +417,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
// legitimately exceed SessionTimeoutHours) and must never be terminalized here
|
||||
// — without this, an actively-installing 6h enrollment could be classified
|
||||
// Incomplete mid-run purely because it outlived the 5h window.
|
||||
if (session.LastEventAt.HasValue && session.LastEventAt.Value > silenceCutoff)
|
||||
if (ServerFrameLastContact(session) > silenceCutoff)
|
||||
continue;
|
||||
|
||||
// Fast path: an AwaitingUser session still inside the grace window needs no
|
||||
@@ -774,5 +782,26 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// Aggregates metrics for any missed days in the last 7 days, plus yesterday.
|
||||
/// Checks the UsageMetrics table for existing snapshots to avoid re-aggregation.
|
||||
/// </summary>
|
||||
|
||||
/// <summary>
|
||||
/// Last contact with the agent, expressed in the SERVER clock frame — the only value a
|
||||
/// server-derived cutoff may legitimately be compared against.
|
||||
/// <para>
|
||||
/// Preference order:
|
||||
/// <list type="number">
|
||||
/// <item><c>LastIngestAt</c> — stamped from the server clock on every ingest. Authoritative.</item>
|
||||
/// <item><c>LastEventAt</c> — device frame; used only for rows written before
|
||||
/// <c>LastIngestAt</c> existed, so the rollout has no blind window. Carries the device's
|
||||
/// clock error and any CMTrace timezone skew.</item>
|
||||
/// <item><c>StartedAt</c> — last resort when a session has no event stamp at all.</item>
|
||||
/// </list>
|
||||
/// </para>
|
||||
/// Mixing the two frames is what let a live agent be marked Stalled: a session whose
|
||||
/// IME-derived events were skewed -17 h looked silent for 17 hours the moment it started.
|
||||
/// </summary>
|
||||
/// <remarks>internal (not private) purely as a unit-test seam — it is a pure function
|
||||
/// over its argument, pinned by <c>MaintenanceServerFrameLastContactTests</c>.</remarks>
|
||||
internal static DateTime ServerFrameLastContact(SessionSummary session)
|
||||
=> session.LastIngestAt ?? session.LastEventAt ?? session.StartedAt;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -608,6 +608,36 @@ namespace AutopilotMonitor.Functions.Services
|
||||
$"Agent emergency break on session {sessionId} (agent {agentVersion ?? "?"}): {message}",
|
||||
tenantId, "System.EmergencyChannel", new { sessionId, agentVersion });
|
||||
|
||||
/// <summary>
|
||||
/// An agent reported that its RUNNING exe's SHA-256 differs from the hash the backend
|
||||
/// advertises for its version — the binary in the field is not the published build
|
||||
/// (tamper, stale blob, or a build that never came from a committed tree). Session
|
||||
/// e9753578 (2026-08-20) carried exactly this report, but it landed only in App
|
||||
/// Insights and was invisible to every product surface while the field damage was
|
||||
/// mis-attributed to committed code for hours. Warning-tier so operators can wire a
|
||||
/// Telegram rule; the message carries both hashes verbatim.
|
||||
/// </summary>
|
||||
public Task RecordAgentBinaryIntegrityMismatchAsync(string tenantId, string? sessionId, string? agentVersion, string message)
|
||||
=> WriteAsync(OpsEventCategory.Agent, "AgentBinaryIntegrityMismatch", OpsEventSeverity.Warning,
|
||||
$"Agent binary integrity mismatch on session {sessionId ?? "?"} (agent {agentVersion ?? "?"}): {message}",
|
||||
tenantId, "System.EmergencyChannel", new { sessionId, agentVersion });
|
||||
|
||||
/// <summary>
|
||||
/// The CMTrace time-skew tripwire fired: a terminal session's IME-derived event
|
||||
/// timestamps diverge from its other events by a clean 15-minute-grid multiple —
|
||||
/// the signature of a timezone mis-conversion the per-line self-anchoring
|
||||
/// (docs/agent/cmtrace-time-resolution.md) failed to catch. Goal state: this event
|
||||
/// NEVER fires; any occurrence is a real anchoring regression or a detector bug,
|
||||
/// both actionable. Emitted by <see cref="EventIngestProcessor"/> once per session
|
||||
/// (gated on the terminal status transition). The message carries the full numbers
|
||||
/// so the event is verifiable without a portal round-trip.
|
||||
/// Dual-register per memory feedback_ops_event_types_dual_register.
|
||||
/// </summary>
|
||||
public Task RecordCmTraceTimeSkewRegressionAsync(string tenantId, string sessionId, string? agentVersion, string message, object details)
|
||||
=> WriteAsync(OpsEventCategory.Agent, "CmTraceTimeSkewRegression", OpsEventSeverity.Warning,
|
||||
$"CMTrace time-skew tripwire on session {sessionId} (agent {agentVersion ?? "?"}): {message}",
|
||||
tenantId, "System.Ingest", details);
|
||||
|
||||
public Task RecordExcessiveSessionEventsAsync(string tenantId, string sessionId, int eventCount, int threshold)
|
||||
=> WriteAsync(OpsEventCategory.Agent, "ExcessiveSessionEvents", OpsEventSeverity.Warning,
|
||||
$"Session {sessionId} has {eventCount} events (threshold {threshold}) — likely agent loop bug",
|
||||
|
||||
@@ -312,12 +312,9 @@ namespace AutopilotMonitor.Functions.Services
|
||||
};
|
||||
}
|
||||
|
||||
// Thin delegation onto the single shared nearest-rank implementation (PercentileMath).
|
||||
private static double Percentile(List<double> sortedValues, double percentile)
|
||||
{
|
||||
if (sortedValues.Count == 0) return 0;
|
||||
var index = (int)Math.Ceiling(percentile * sortedValues.Count) - 1;
|
||||
return sortedValues[Math.Max(0, Math.Min(index, sortedValues.Count - 1))];
|
||||
}
|
||||
=> Helpers.PercentileMath.Percentile(sortedValues, percentile);
|
||||
|
||||
// Payload getters live in AgentMetricsAggregation (shared with the efficiency
|
||||
// service); these thin wrappers keep the call sites above unchanged.
|
||||
|
||||
@@ -98,6 +98,9 @@ namespace AutopilotMonitor.Functions.Services
|
||||
case "event_correlation":
|
||||
return EvaluateEventCorrelationCondition(condition, events);
|
||||
|
||||
case "clock_skew":
|
||||
return EvaluateClockSkewCondition(condition, events);
|
||||
|
||||
default:
|
||||
return (false, "unknown source");
|
||||
}
|
||||
@@ -752,7 +755,12 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
private (bool matched, object evidence) EvaluateAppInstallDurationCondition(RuleCondition condition, List<EnrollmentEvent> events)
|
||||
{
|
||||
var sortedEvents = events.OrderBy(e => e.Timestamp).ThenBy(e => e.Sequence).ToList();
|
||||
// Ordering key is Sequence, never Timestamp: Sequence is a persisted, strictly
|
||||
// monotonic per-session counter assigned at emit time and is immune to every clock
|
||||
// problem, whereas Timestamp carries the device clock frame plus — for CMTrace-derived
|
||||
// events — the writer process's timezone belief. Timestamp stays in use below only
|
||||
// where an actual ELAPSED DURATION is meant, which is what it is for.
|
||||
var sortedEvents = events.OrderBy(e => e.Sequence).ToList();
|
||||
|
||||
var completionEventTypes = string.IsNullOrWhiteSpace(condition.EventType)
|
||||
? new HashSet<string>(new[] { "app_install_completed", "app_install_failed" }, StringComparer.OrdinalIgnoreCase)
|
||||
@@ -774,7 +782,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var startEvent = sortedEvents.LastOrDefault(e =>
|
||||
(string.Equals(e.EventType, "app_install_started", StringComparison.OrdinalIgnoreCase) ||
|
||||
string.Equals(e.EventType, "app_install_start", StringComparison.OrdinalIgnoreCase)) &&
|
||||
e.Timestamp <= completionEvent.Timestamp &&
|
||||
e.Sequence <= completionEvent.Sequence &&
|
||||
string.Equals(GetDataFieldValue(e, "appId") ?? GetDataFieldValue(e, "appName"), appKey, StringComparison.OrdinalIgnoreCase));
|
||||
|
||||
if (startEvent == null)
|
||||
@@ -803,6 +811,209 @@ namespace AutopilotMonitor.Functions.Services
|
||||
return (false, "no app install duration matched");
|
||||
}
|
||||
|
||||
// ===== clock_skew constants =====
|
||||
// Named constants, deliberately NOT condition fields: promoting a constant to a field
|
||||
// later is non-breaking, the reverse is not.
|
||||
/// <summary>Frame windows (in upload batches) that must agree before/after a step for it to count as a clock jump.</summary>
|
||||
internal const int ClockSkewConfirmBatches = 3;
|
||||
/// <summary>Floor for the per-batch internal-spread cap, seconds.</summary>
|
||||
internal const double ClockSkewSpoolSpreadFloorSeconds = 60;
|
||||
/// <summary>Per-batch internal-spread cap as a fraction of the condition threshold.</summary>
|
||||
internal const double ClockSkewSpoolSpreadCapFraction = 0.5;
|
||||
/// <summary>Fraction of surviving batches that must sit near the overall median for a sustained-offset verdict.</summary>
|
||||
internal const double ClockSkewStableBatchFraction = 0.75;
|
||||
|
||||
/// <summary>
|
||||
/// Detects DEVICE clock problems from a per-upload-batch offset between the device's
|
||||
/// clock frame and the server's receive frame (seconds; healthy baseline ≈
|
||||
/// −upload-latency, i.e. a few seconds negative).
|
||||
/// skewMetric "clock_jump": a persistent step in the device's clock frame mid-session
|
||||
/// (time sync correcting a wrong RTC, Windows auto-timezone, manual change).
|
||||
/// skewMetric "sustained_offset": the whole session ran on a clock off by ≥ threshold
|
||||
/// — the during-enrollment complement of ANALYZE-DEV-007's one-shot NTP check.
|
||||
///
|
||||
/// Batch offset, two measurement paths (P14):
|
||||
/// - Events carrying SentAt (X-Send-Time-Utc header, one device-clock value per ingest
|
||||
/// request): offset = SentAt − ReceivedAt, a DIRECT frame-crossing measurement. No
|
||||
/// event timestamps involved → immune to spool backlog, CMTrace writer beliefs and
|
||||
/// clamping by construction, so no spread filter is needed and every batch counts.
|
||||
/// - Legacy batches (agents without the header): median of per-event Timestamp −
|
||||
/// ReceivedAt with a spread cap that drops replayed spool backlogs, excluding
|
||||
/// CMTrace-derived events (Source == "ImeLogTracker") — their Timestamp carries the
|
||||
/// log WRITER's timezone belief, so an anchoring regression must fire the
|
||||
/// operator-side CmTraceSkewTripwire, never this customer-facing rule.
|
||||
///
|
||||
/// Ordering key is Sequence, never Timestamp (see EvaluateAppInstallDurationCondition).
|
||||
/// The legacy path then inspects Timestamp AGAINST that Sequence order ON PURPOSE — a
|
||||
/// device-clock problem is precisely a Timestamp anomaly relative to the clock-immune
|
||||
/// Sequence/ReceivedAt frame. Timestamp is a MEASUREMENT here, never a sort key.
|
||||
/// </summary>
|
||||
private (bool matched, object evidence) EvaluateClockSkewCondition(RuleCondition condition, List<EnrollmentEvent> events)
|
||||
{
|
||||
var metric = condition.SkewMetric ?? string.Empty;
|
||||
if (!double.TryParse(condition.Value, System.Globalization.NumberStyles.Float,
|
||||
System.Globalization.CultureInfo.InvariantCulture, out var thresholdSeconds) || thresholdSeconds <= 0)
|
||||
return (false, "clock_skew requires a positive numeric value (threshold in seconds)");
|
||||
|
||||
var usable = events
|
||||
.Where(e => e.ReceivedAt.HasValue)
|
||||
.OrderBy(e => e.Sequence)
|
||||
.ToList();
|
||||
if (usable.Count == 0)
|
||||
return (false, "no events with a server receive frame");
|
||||
|
||||
// Group into upload batches (ReceivedAt is stamped once per ingest request). Legacy
|
||||
// rationale for the spread cap: a clock offset shifts a whole batch uniformly
|
||||
// (internal spread ≈ upload debounce, seconds); a spool-backlog batch spans the
|
||||
// offline period (spread = minutes–hours). Dropping wide-spread batches is what
|
||||
// makes replayed backlogs inert on the legacy path.
|
||||
var spreadCap = Math.Max(ClockSkewSpoolSpreadFloorSeconds, thresholdSeconds * ClockSkewSpoolSpreadCapFraction);
|
||||
var batches = new List<(double MedianOffsetSeconds, EnrollmentEvent FirstEvent)>();
|
||||
int usedSamples = 0;
|
||||
int sentAtBatchCount = 0;
|
||||
foreach (var batch in usable.GroupBy(e => e.ReceivedAt!.Value))
|
||||
{
|
||||
// P14 direct path — SentAt is request-level, so any event's value speaks for
|
||||
// the whole batch.
|
||||
var sentAt = batch.Select(e => e.SentAt).FirstOrDefault(s => s.HasValue);
|
||||
if (sentAt.HasValue)
|
||||
{
|
||||
sentAtBatchCount++;
|
||||
usedSamples += batch.Count();
|
||||
batches.Add(((sentAt.Value - batch.Key).TotalSeconds, batch.First()));
|
||||
continue;
|
||||
}
|
||||
|
||||
// Legacy path — see the method doc for the IME/clamped exclusions.
|
||||
var measurable = batch
|
||||
.Where(e => !e.TimestampClamped
|
||||
&& !string.Equals(e.Source, "ImeLogTracker", StringComparison.OrdinalIgnoreCase))
|
||||
.ToList();
|
||||
if (measurable.Count == 0)
|
||||
continue;
|
||||
|
||||
var offsets = measurable.Select(e => (e.Timestamp - batch.Key).TotalSeconds).OrderBy(v => v).ToList();
|
||||
if (offsets[offsets.Count - 1] - offsets[0] > spreadCap)
|
||||
continue;
|
||||
usedSamples += offsets.Count;
|
||||
batches.Add((Helpers.PercentileMath.Percentile(offsets, 0.50), measurable[0]));
|
||||
}
|
||||
|
||||
switch (metric)
|
||||
{
|
||||
case "sustained_offset":
|
||||
{
|
||||
if (batches.Count < 2)
|
||||
return (false, "insufficient upload batches for an offset measurement");
|
||||
|
||||
var batchMedians = batches.Select(b => b.MedianOffsetSeconds).ToList();
|
||||
var overall = Helpers.PercentileMath.Median(batchMedians);
|
||||
|
||||
// Constant offset, not drift chaos: most batches must individually sit near
|
||||
// the session median, otherwise no single "the clock was off by X" statement
|
||||
// is honest.
|
||||
var stable = batchMedians.Count(m => Math.Abs(m - overall) <= spreadCap);
|
||||
if (stable < Math.Max(2, (int)Math.Ceiling(batchMedians.Count * ClockSkewStableBatchFraction)))
|
||||
return (false, "batch offsets too unstable for a sustained-offset verdict");
|
||||
|
||||
if (!CompareClockSkewThreshold(Math.Abs(overall), condition.Operator, thresholdSeconds))
|
||||
return (false, $"sustained offset {overall:0} s below threshold");
|
||||
|
||||
var first = batches[0].FirstEvent;
|
||||
var offsetMinutes = Math.Round(overall / 60.0, 1);
|
||||
var direction = overall > 0 ? "ahead" : "behind";
|
||||
return (true, new Dictionary<string, object>
|
||||
{
|
||||
// field/value pair drives {{skewSummary}} template interpolation
|
||||
// (lib/interpolateRuleTemplate.ts resolves field→value first).
|
||||
["field"] = "skewSummary",
|
||||
["value"] = $"clock ran {Math.Abs(offsetMinutes)} min {direction} of server time throughout the session",
|
||||
["offsetSeconds"] = Math.Round(overall, 0),
|
||||
["offsetMinutes"] = offsetMinutes,
|
||||
["direction"] = direction,
|
||||
["eventId"] = first.EventId,
|
||||
["sequence"] = first.Sequence,
|
||||
["timestamp"] = first.Timestamp,
|
||||
["sampleCount"] = usedSamples,
|
||||
["batchCount"] = batches.Count,
|
||||
["sentAtBatchCount"] = sentAtBatchCount,
|
||||
});
|
||||
}
|
||||
|
||||
case "clock_jump":
|
||||
{
|
||||
const int w = ClockSkewConfirmBatches;
|
||||
if (batches.Count < 2 * w)
|
||||
return (false, "insufficient upload batches for a frame comparison");
|
||||
|
||||
var medians = batches.Select(b => b.MedianOffsetSeconds).ToList();
|
||||
(int Index, double Pre, double Post, double Jump)? best = null;
|
||||
for (int j = w - 1; j <= batches.Count - 1 - w; j++)
|
||||
{
|
||||
var preWindow = medians.Skip(j - w + 1).Take(w).ToList();
|
||||
var postWindow = medians.Skip(j + 1).Take(w).ToList();
|
||||
|
||||
// Both comparison windows must be flat plateaus: a genuine clock change
|
||||
// is a step between two stable frames, while a gradually draining spool
|
||||
// backlog produces a RAMP of batch medians — sloping windows would let
|
||||
// any pre/post pair straddling the ramp fake a step.
|
||||
if (preWindow.Max() - preWindow.Min() > spreadCap || postWindow.Max() - postWindow.Min() > spreadCap)
|
||||
continue;
|
||||
|
||||
var pre = Helpers.PercentileMath.Median(preWindow);
|
||||
var post = Helpers.PercentileMath.Median(postWindow);
|
||||
var jump = post - pre;
|
||||
if (CompareClockSkewThreshold(Math.Abs(jump), condition.Operator, thresholdSeconds)
|
||||
&& (best == null || Math.Abs(jump) > Math.Abs(best.Value.Jump)))
|
||||
best = (j, pre, post, jump);
|
||||
}
|
||||
if (best == null)
|
||||
return (false, "no persistent clock step");
|
||||
|
||||
// End-state persistence: a transient excursion that returns to the original
|
||||
// frame (spool backlog draining, historic replay) must stay silent; a genuine
|
||||
// clock change — including "wrong clock corrected by time sync mid-session" —
|
||||
// persists to session end, because the corrected frame IS the change.
|
||||
var tail = Helpers.PercentileMath.Median(medians.Skip(medians.Count - w).Take(w));
|
||||
var tailShift = tail - best.Value.Pre;
|
||||
if (Math.Abs(tailShift) < thresholdSeconds || Math.Sign(tailShift) != Math.Sign(best.Value.Jump))
|
||||
return (false, "transient timestamp excursion, not a sustained clock change");
|
||||
|
||||
var boundary = batches[best.Value.Index + 1].FirstEvent;
|
||||
var jumpMinutes = Math.Round(best.Value.Jump / 60.0, 1);
|
||||
var direction = best.Value.Jump > 0 ? "forward" : "backward";
|
||||
return (true, new Dictionary<string, object>
|
||||
{
|
||||
// field/value pair drives {{skewSummary}} template interpolation
|
||||
// (lib/interpolateRuleTemplate.ts resolves field→value first).
|
||||
["field"] = "skewSummary",
|
||||
["value"] = $"clock jumped {Math.Abs(jumpMinutes)} min {direction} mid-session",
|
||||
["jumpSeconds"] = Math.Round(best.Value.Jump, 0),
|
||||
["jumpMinutes"] = jumpMinutes,
|
||||
["direction"] = direction,
|
||||
["beforeOffsetMinutes"] = Math.Round(best.Value.Pre / 60.0, 1),
|
||||
["afterOffsetMinutes"] = Math.Round(best.Value.Post / 60.0, 1),
|
||||
["eventId"] = boundary.EventId,
|
||||
["sequence"] = boundary.Sequence,
|
||||
["timestamp"] = boundary.Timestamp,
|
||||
["eventType"] = boundary.EventType,
|
||||
["sampleCount"] = usedSamples,
|
||||
["batchCount"] = batches.Count,
|
||||
["sentAtBatchCount"] = sentAtBatchCount,
|
||||
});
|
||||
}
|
||||
|
||||
default:
|
||||
return (false, $"unknown skewMetric '{metric}' (expected clock_jump or sustained_offset)");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>gt/gte magnitude comparison for clock_skew; unknown operators behave as gte (validation rejects them upstream).</summary>
|
||||
private static bool CompareClockSkewThreshold(double magnitude, string? op, double thresholdSeconds)
|
||||
=> string.Equals(op, "gt", StringComparison.OrdinalIgnoreCase)
|
||||
? magnitude > thresholdSeconds
|
||||
: magnitude >= thresholdSeconds;
|
||||
|
||||
/// <summary>
|
||||
/// Generic event correlation: finds pairs of events (A before B) sharing
|
||||
/// the same value in a join field, with optional filters on each event
|
||||
@@ -826,7 +1037,12 @@ namespace AutopilotMonitor.Functions.Services
|
||||
return (false, "event_correlation requires EventType, CorrelateEventType, and JoinField");
|
||||
}
|
||||
|
||||
var sortedEvents = events.OrderBy(e => e.Timestamp).ThenBy(e => e.Sequence).ToList();
|
||||
// Ordering key is Sequence, never Timestamp: Sequence is a persisted, strictly
|
||||
// monotonic per-session counter assigned at emit time and is immune to every clock
|
||||
// problem, whereas Timestamp carries the device clock frame plus — for CMTrace-derived
|
||||
// events — the writer process's timezone belief. Timestamp stays in use below only
|
||||
// where an actual ELAPSED DURATION is meant, which is what it is for.
|
||||
var sortedEvents = events.OrderBy(e => e.Sequence).ToList();
|
||||
|
||||
// Collect Event A candidates with optional filter
|
||||
var eventAList = sortedEvents
|
||||
@@ -874,16 +1090,14 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
// Find the latest Event A that occurred before Event B
|
||||
var matchingA = eventAByJoinKey[bKey]
|
||||
.Where(a => a.Timestamp < eventB.Timestamp ||
|
||||
(a.Timestamp == eventB.Timestamp && a.Sequence < eventB.Sequence))
|
||||
.Where(a => a.Sequence < eventB.Sequence)
|
||||
.Where(a =>
|
||||
{
|
||||
if (condition.TimeWindowSeconds == null || condition.TimeWindowSeconds <= 0)
|
||||
return true;
|
||||
return (eventB.Timestamp - a.Timestamp).TotalSeconds <= condition.TimeWindowSeconds.Value;
|
||||
})
|
||||
.OrderByDescending(a => a.Timestamp)
|
||||
.ThenByDescending(a => a.Sequence)
|
||||
.OrderByDescending(a => a.Sequence)
|
||||
.FirstOrDefault();
|
||||
|
||||
if (matchingA == null)
|
||||
|
||||
@@ -78,6 +78,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
"DiagnosticsBlobName",
|
||||
"DiagnosticsBlobDestination",
|
||||
"LastEventAt",
|
||||
"LastIngestAt",
|
||||
"ResumedAt",
|
||||
"StalledAt",
|
||||
"AvgApiLatencyMs",
|
||||
|
||||
@@ -698,9 +698,19 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
var silenceCutoffStr = silenceCutoff.ToString("yyyy-MM-ddTHH:mm:ss");
|
||||
var hardCutoffStr = hardCutoff.ToString("yyyy-MM-ddTHH:mm:ss");
|
||||
|
||||
// Candidate pre-filter across BOTH clock frames, deliberately permissive:
|
||||
// LastIngestAt is the server frame (authoritative, but absent on rows written
|
||||
// before it existed), LastEventAt the device frame (always present, but carries
|
||||
// the device's clock error and any CMTrace timezone skew). An OR keeps a session
|
||||
// in the candidate set when EITHER frame looks silent; the caller then makes the
|
||||
// authoritative call on the server frame. Filtering on LastIngestAt alone would
|
||||
// silently drop every pre-rollout row, since Table Storage does not match a
|
||||
// filter against a missing property.
|
||||
var filter = $"PartitionKey eq '{tenantId}' " +
|
||||
$"and Status eq 'InProgress' " +
|
||||
$"and LastEventAt lt datetime'{silenceCutoffStr}Z' " +
|
||||
$"and (LastIngestAt lt datetime'{silenceCutoffStr}Z' " +
|
||||
$"or LastEventAt lt datetime'{silenceCutoffStr}Z') " +
|
||||
$"and (StartedAt ge datetime'{hardCutoffStr}Z' or ResumedAt ge datetime'{hardCutoffStr}Z')";
|
||||
|
||||
var query = tableClient.QueryAsync<TableEntity>(filter: filter);
|
||||
@@ -772,7 +782,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var select = new[]
|
||||
{
|
||||
"PartitionKey", "RowKey", "Status", "StartedAt", "ResumedAt", "LastEventAt",
|
||||
"SerialNumber", "IsPreProvisioned"
|
||||
"LastIngestAt", "SerialNumber", "IsPreProvisioned"
|
||||
};
|
||||
|
||||
var sessions = new List<SessionSummary>();
|
||||
|
||||
@@ -260,29 +260,75 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var existingStartedAt = existing.GetDateTimeOffset("StartedAt")?.UtcDateTime;
|
||||
if (existingStartedAt.HasValue && existingStartedAt.Value != DateTime.MinValue)
|
||||
{
|
||||
// Keep the earlier StartedAt; recalculate duration if CompletedAt is now known
|
||||
// Keep the earlier StartedAt — it is the window/bucket filter column and
|
||||
// the radar's version-ordering key, never re-anchored to a later attempt.
|
||||
// Duration recompute happens attempt-aware further down.
|
||||
if (summary.StartedAt == DateTime.MinValue || existingStartedAt.Value < summary.StartedAt)
|
||||
{
|
||||
summary.StartedAt = existingStartedAt.Value;
|
||||
if (summary.CompletedAt.HasValue && summary.CompletedAt.Value >= summary.StartedAt)
|
||||
{
|
||||
summary.DurationSeconds = (int)(summary.CompletedAt.Value - summary.StartedAt).TotalSeconds;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Attempt anchor: the LATEST start wins across batches (mirror of the
|
||||
// in-batch max-fold — a replayed older batch can never regress it), and the
|
||||
// pass counter only grows when the batch carries genuinely newer starts
|
||||
// (replay guard: a batch whose newest start is not newer than the stored
|
||||
// anchor was already counted).
|
||||
var existingLastAttempt = existing.GetDateTimeOffset("LastAttemptStartedAt")?.UtcDateTime;
|
||||
var existingPassCount = existing.GetInt32("InstallPassCount") ?? 0;
|
||||
if (summary.LastAttemptStartedAt.HasValue &&
|
||||
(!existingLastAttempt.HasValue || summary.LastAttemptStartedAt.Value > existingLastAttempt.Value))
|
||||
{
|
||||
summary.InstallPassCount += existingPassCount;
|
||||
}
|
||||
else
|
||||
{
|
||||
summary.InstallPassCount = Math.Max(summary.InstallPassCount, existingPassCount);
|
||||
if (existingLastAttempt.HasValue)
|
||||
summary.LastAttemptStartedAt = existingLastAttempt;
|
||||
}
|
||||
|
||||
// Cross-batch mirror of the weaker-terminal guard: a batch that only saw a
|
||||
// Skipped/Postponed re-evaluation pass never overrides a stored
|
||||
// Installed/Error, and its CompletedAt/DurationSeconds must not re-describe
|
||||
// the row (nulling them omits the columns → Merge preserves the stored
|
||||
// values of the surviving attempt).
|
||||
var existingTerminalStateForGuard = existing.GetString("TerminalState");
|
||||
if ((existingTerminalStateForGuard == "Installed" || existingTerminalStateForGuard == "Error") &&
|
||||
(summary.TerminalState == "Skipped" || summary.TerminalState == "Postponed"))
|
||||
{
|
||||
summary.TerminalState = existingTerminalStateForGuard!;
|
||||
if (existingStatus == "Succeeded" || existingStatus == "Failed")
|
||||
summary.Status = existingStatus!;
|
||||
summary.CompletedAt = null;
|
||||
summary.DurationSeconds = 0;
|
||||
}
|
||||
|
||||
// Q4 (source-data audit 2026-07-26): out-of-order arrival — the terminal batch
|
||||
// landed FIRST (row already carries CompletedAt), the started batch arrives now
|
||||
// with the earlier StartedAt. Neither batch alone could compute the duration;
|
||||
// recompute it here against the stored CompletedAt so the row doesn't keep a
|
||||
// permanently unset duration despite both endpoints being known.
|
||||
// landed FIRST (row already carries CompletedAt), the started batch arrives now.
|
||||
// Neither batch alone could compute the duration; adopt the stored CompletedAt
|
||||
// so the attempt-aware recompute below can pair both endpoints.
|
||||
if (!summary.CompletedAt.HasValue && summary.StartedAt != DateTime.MinValue)
|
||||
{
|
||||
var existingCompletedAt = existing.GetDateTimeOffset("CompletedAt")?.UtcDateTime;
|
||||
if (existingCompletedAt.HasValue && existingCompletedAt.Value >= summary.StartedAt)
|
||||
{
|
||||
summary.CompletedAt = existingCompletedAt.Value;
|
||||
summary.DurationSeconds = (int)(existingCompletedAt.Value - summary.StartedAt).TotalSeconds;
|
||||
}
|
||||
|
||||
// Attempt-aware duration recompute — single site for both endpoints-known
|
||||
// paths (in-order and Q4 out-of-order). Anchor = latest attempt start at or
|
||||
// before CompletedAt; rows without an observed start keep the historical
|
||||
// StartedAt (span) fallback. A LastAttemptStartedAt NEWER than CompletedAt
|
||||
// means a fresh pass is in flight — the stored duration of the completed
|
||||
// attempt stands (DurationSeconds 0 omits the column, Merge preserves it).
|
||||
if (summary.CompletedAt.HasValue)
|
||||
{
|
||||
if (summary.LastAttemptStartedAt.HasValue)
|
||||
{
|
||||
if (summary.LastAttemptStartedAt.Value <= summary.CompletedAt.Value)
|
||||
summary.DurationSeconds = (int)(summary.CompletedAt.Value - summary.LastAttemptStartedAt.Value).TotalSeconds;
|
||||
}
|
||||
else if (summary.StartedAt != DateTime.MinValue && summary.CompletedAt.Value >= summary.StartedAt)
|
||||
{
|
||||
summary.DurationSeconds = (int)(summary.CompletedAt.Value - summary.StartedAt).TotalSeconds;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -571,7 +617,10 @@ namespace AutopilotMonitor.Functions.Services
|
||||
"PartitionKey", "RowKey", "TenantId", "SessionId", "AppName", "AppType", "AppVersion",
|
||||
"Status", "TerminalState", "StartedAt", "CompletedAt", "DurationSeconds", "DownloadBytes",
|
||||
"AttemptNumber", "InstallerPhase", "FailureCode", "FailureMessage", "ExitCode", "DetectionResult",
|
||||
"AppId", "EspBlocking", "AppIdCollision"
|
||||
"AppId", "EspBlocking", "AppIdCollision",
|
||||
// Attempt-duration columns (2026-08): the radar's cutover gate reads the anchor,
|
||||
// the dashboard sessions table shows the pass count.
|
||||
"LastAttemptStartedAt", "InstallPassCount"
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
@@ -671,6 +720,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
FailureMessage = entity.GetString("FailureMessage") ?? string.Empty,
|
||||
StartedAt = entity.GetDateTimeOffset("StartedAt")?.UtcDateTime ?? DateTime.MinValue,
|
||||
CompletedAt = entity.GetDateTimeOffset("CompletedAt")?.UtcDateTime,
|
||||
LastAttemptStartedAt = entity.GetDateTimeOffset("LastAttemptStartedAt")?.UtcDateTime,
|
||||
InstallPassCount = entity.GetInt32("InstallPassCount") ?? 0,
|
||||
// Delivery Optimization telemetry
|
||||
DoFileSize = entity.GetInt64("DoFileSize") ?? 0,
|
||||
DoTotalBytesDownloaded = entity.GetInt64("DoTotalBytesDownloaded") ?? 0,
|
||||
@@ -1563,6 +1614,12 @@ namespace AutopilotMonitor.Functions.Services
|
||||
entity["DurationSeconds"] = summary.DurationSeconds;
|
||||
if (summary.CompletedAt.HasValue)
|
||||
entity["CompletedAt"] = EnsureUtc(summary.CompletedAt.Value);
|
||||
// Attempt-duration columns (2026-08): both sentinel-gated — a batch that observed
|
||||
// no install start must not clobber the stored anchor/counter under Merge-mode.
|
||||
if (summary.LastAttemptStartedAt.HasValue)
|
||||
entity["LastAttemptStartedAt"] = EnsureUtc(summary.LastAttemptStartedAt.Value);
|
||||
if (summary.InstallPassCount > 0)
|
||||
entity["InstallPassCount"] = summary.InstallPassCount;
|
||||
if (!string.IsNullOrEmpty(summary.FailureCode))
|
||||
entity["FailureCode"] = summary.FailureCode;
|
||||
if (!string.IsNullOrEmpty(summary.FailureMessage))
|
||||
|
||||
@@ -442,6 +442,20 @@ namespace AutopilotMonitor.Functions.Services
|
||||
public async Task<(int deleted, int failed)> DeleteRuleStatesForRuleIdAcrossTenantsAsync(string ruleId)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(ruleId)) return (0, 0);
|
||||
|
||||
// This is a cross-partition delete keyed on a bare RowKey string, so only
|
||||
// catalog (built-in/community) IDs may pass: custom-rule IDs are tenant-chosen
|
||||
// and not unique across tenants, and a call with one would delete state rows
|
||||
// of unrelated tenants. failed=-1 reuses the enumeration-failure contract —
|
||||
// the caller then refuses the follow-up global catalog delete.
|
||||
if (!RuleIdPolicy.IsReservedBuiltInId(ruleId))
|
||||
{
|
||||
_logger.LogError(
|
||||
"Refusing cross-tenant RuleState GC for '{RuleId}': ID is outside the reserved built-in namespace",
|
||||
ruleId);
|
||||
return (0, -1);
|
||||
}
|
||||
|
||||
try
|
||||
{
|
||||
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.RuleStates);
|
||||
|
||||
@@ -225,6 +225,13 @@ namespace AutopilotMonitor.Functions.Services
|
||||
if (lastEventAt.HasValue)
|
||||
indexEntity["LastEventAt"] = EnsureUtc(lastEventAt.Value);
|
||||
|
||||
// Server-frame ingest stamp (IncrementSessionEventCountAsync merges this on every
|
||||
// batch, so the index already carries it) — must be in this superset or a
|
||||
// StartedAt-shift full upsert drops it and index-served reads lose the field.
|
||||
var lastIngestAt = sessionEntity.GetDateTimeOffset("LastIngestAt")?.UtcDateTime;
|
||||
if (lastIngestAt.HasValue)
|
||||
indexEntity["LastIngestAt"] = EnsureUtc(lastIngestAt.Value);
|
||||
|
||||
var resumedAt = sessionEntity.GetDateTimeOffset("ResumedAt")?.UtcDateTime;
|
||||
if (resumedAt.HasValue)
|
||||
indexEntity["ResumedAt"] = EnsureUtc(resumedAt.Value);
|
||||
@@ -815,6 +822,11 @@ namespace AutopilotMonitor.Functions.Services
|
||||
["TimestampClamped"] = evt.TimestampClamped
|
||||
};
|
||||
|
||||
// P14: request-level device-clock send time; column stays unset for
|
||||
// events from agents that pre-date the X-Send-Time-Utc header.
|
||||
if (evt.SentAt.HasValue)
|
||||
entity["SentAt"] = EnsureUtc(evt.SentAt.Value);
|
||||
|
||||
if (evt.OriginalTimestamp.HasValue)
|
||||
entity["OriginalTimestamp"] = EnsureUtc(evt.OriginalTimestamp.Value);
|
||||
|
||||
@@ -912,6 +924,10 @@ namespace AutopilotMonitor.Functions.Services
|
||||
["TimestampClamped"] = evt.TimestampClamped
|
||||
};
|
||||
|
||||
// P14: see StoreEventAsync.
|
||||
if (evt.SentAt.HasValue)
|
||||
entity["SentAt"] = EnsureUtc(evt.SentAt.Value);
|
||||
|
||||
if (evt.OriginalTimestamp.HasValue)
|
||||
entity["OriginalTimestamp"] = EnsureUtc(evt.OriginalTimestamp.Value);
|
||||
|
||||
@@ -1214,8 +1230,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
int succeeded = 0;
|
||||
int failed = 0;
|
||||
int incomplete = 0;
|
||||
long succeededDurationSeconds = 0;
|
||||
int succeededWithDurationCount = 0;
|
||||
var succeededDurationSeconds = new List<double>();
|
||||
int totalToday = 0;
|
||||
int failedToday = 0;
|
||||
|
||||
@@ -1234,10 +1249,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
case SessionStatus.Succeeded:
|
||||
succeeded++;
|
||||
if (s.DurationSeconds is int d && d > 0)
|
||||
{
|
||||
succeededDurationSeconds += d;
|
||||
succeededWithDurationCount++;
|
||||
}
|
||||
succeededDurationSeconds.Add(d);
|
||||
break;
|
||||
case SessionStatus.Failed:
|
||||
failed++;
|
||||
@@ -1259,8 +1271,9 @@ namespace AutopilotMonitor.Functions.Services
|
||||
int successRatePct = terminal > 0
|
||||
? (int)Math.Round((double)succeeded / terminal * 100.0)
|
||||
: 0;
|
||||
int avgDurationMinutes = succeededWithDurationCount > 0
|
||||
? (int)Math.Round((double)succeededDurationSeconds / succeededWithDurationCount / 60.0)
|
||||
succeededDurationSeconds.Sort(); // MetricsMath.Percentile expects ascending order
|
||||
int avgDurationMinutes = succeededDurationSeconds.Count > 0
|
||||
? (int)Math.Round(succeededDurationSeconds.Average() / 60.0)
|
||||
: 0;
|
||||
|
||||
return new SessionStats
|
||||
@@ -1273,6 +1286,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
IncompleteLastNDays = incomplete,
|
||||
SuccessRatePct = successRatePct,
|
||||
AvgDurationMinutes = avgDurationMinutes,
|
||||
MedianDurationMinutes = MetricsMath.SecondsToMinutes(MetricsMath.Percentile(succeededDurationSeconds, 50)),
|
||||
P90DurationMinutes = MetricsMath.SecondsToMinutes(MetricsMath.Percentile(succeededDurationSeconds, 90)),
|
||||
TotalToday = totalToday,
|
||||
FailedToday = failedToday,
|
||||
ComputedAt = DateTime.UtcNow,
|
||||
@@ -2338,6 +2353,14 @@ namespace AutopilotMonitor.Functions.Services
|
||||
update["LastEventAt"] = EnsureUtc(latestEventTimestamp.Value);
|
||||
}
|
||||
|
||||
// Server-frame ingest stamp. Reaching this method means a batch was just
|
||||
// received, so "now" on the SERVER clock is the honest answer to "when did we
|
||||
// last hear from this agent". LastEventAt above cannot answer that — it carries
|
||||
// the device's clock frame and its timezone skew. Written unconditionally (not
|
||||
// guarded by latestEventTimestamp) because an empty-timestamp batch is still
|
||||
// contact from the agent.
|
||||
update["LastIngestAt"] = EnsureUtc(DateTime.UtcNow);
|
||||
|
||||
// Increment script execution counters
|
||||
if (platformScriptIncrement > 0)
|
||||
{
|
||||
@@ -2410,17 +2433,20 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// are left untouched rather than being zeroed. Idempotent: no-ops when the row is
|
||||
/// already correct. PlatformScriptCount/RemediationScriptCount stay increment-only (their
|
||||
/// drift is bounded and they feed no enforcement heuristics; EventCount does).
|
||||
/// Returns the skew-scan samples collected during the authoritative count's partition
|
||||
/// scan (CmTraceSkewTripwire input), or null when the scan itself failed — the counter
|
||||
/// reconcile outcome does not affect the returned scan.
|
||||
/// </summary>
|
||||
public async Task ReconcileSessionCountersAsync(string tenantId, string sessionId)
|
||||
public async Task<SessionSkewScan?> ReconcileSessionCountersAsync(string tenantId, string sessionId)
|
||||
{
|
||||
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
|
||||
SecurityValidator.EnsureValidGuid(sessionId, nameof(sessionId));
|
||||
|
||||
var authoritative = await CountSessionEventRowsAsync(tenantId, sessionId);
|
||||
if (!authoritative.HasValue)
|
||||
return; // storage error — keep the incremental live values, do not zero them
|
||||
return null; // storage error — keep the incremental live values, do not zero them
|
||||
|
||||
var (totalEvents, rebootEvents) = authoritative.Value;
|
||||
var (totalEvents, rebootEvents, skewScan) = authoritative.Value;
|
||||
|
||||
const int maxRetries = 5;
|
||||
int retryCount = 0;
|
||||
@@ -2432,7 +2458,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.Sessions);
|
||||
var entityResponse = await tableClient.GetEntityIfExistsAsync<TableEntity>(tenantId, sessionId);
|
||||
if (!entityResponse.HasValue)
|
||||
return;
|
||||
return skewScan;
|
||||
var entity = entityResponse.Value!;
|
||||
|
||||
// No-op when already correct — avoids a needless write + index sync on the
|
||||
@@ -2451,7 +2477,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
drifted = true;
|
||||
}
|
||||
if (!drifted)
|
||||
return;
|
||||
return skewScan;
|
||||
|
||||
var currentStartedAt = entity.GetDateTimeOffset("StartedAt")?.UtcDateTime ?? DateTime.MaxValue;
|
||||
|
||||
@@ -2460,7 +2486,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
// Dual-write: keep SessionsIndex in sync (merge — no StartedAt shift here).
|
||||
await SyncSessionIndexAsync(tenantId, sessionId, entity, update, currentStartedAt, null);
|
||||
|
||||
return;
|
||||
return skewScan;
|
||||
}
|
||||
catch (Azure.RequestFailedException ex) when (ex.Status == 412)
|
||||
{
|
||||
@@ -2468,7 +2494,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
if (retryCount >= maxRetries)
|
||||
{
|
||||
_logger.LogWarning($"Failed to reconcile session counters for session {sessionId} after {maxRetries} retries due to ETag conflicts");
|
||||
return;
|
||||
return skewScan;
|
||||
}
|
||||
var baseDelay = 50 * (int)Math.Pow(2, retryCount - 1);
|
||||
await Task.Delay(baseDelay + Random.Shared.Next(0, baseDelay));
|
||||
@@ -2476,9 +2502,11 @@ namespace AutopilotMonitor.Functions.Services
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, $"Failed to reconcile session counters for session {sessionId}");
|
||||
return;
|
||||
return skewScan;
|
||||
}
|
||||
}
|
||||
|
||||
return skewScan;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -3075,6 +3103,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
Data = DeserializeEventData(entity.GetString("DataJson")),
|
||||
RowKey = entity.RowKey,
|
||||
ReceivedAt = entity.GetDateTimeOffset("ReceivedAt")?.UtcDateTime,
|
||||
SentAt = entity.GetDateTimeOffset("SentAt")?.UtcDateTime,
|
||||
OriginalTimestamp = entity.GetDateTimeOffset("OriginalTimestamp")?.UtcDateTime,
|
||||
TimestampClamped = entity.GetBoolean("TimestampClamped") ?? false,
|
||||
// Codex follow-up #3 — forward-link columns (nullable for legacy rows).
|
||||
@@ -3147,6 +3176,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
// null as "CustomerSas" for back-compat.
|
||||
DiagnosticsBlobDestination = entity.GetString("DiagnosticsBlobDestination"),
|
||||
LastEventAt = SafeGetDateTime(entity, "LastEventAt"),
|
||||
LastIngestAt = SafeGetDateTime(entity, "LastIngestAt"),
|
||||
IsPreProvisioned = entity.GetBoolean("IsPreProvisioned") ?? false,
|
||||
IsHybridJoin = entity.GetBoolean("IsHybridJoin") ?? false,
|
||||
IsSelfDeployingProfile = entity.GetBoolean("IsSelfDeployingProfile") ?? false,
|
||||
@@ -3278,9 +3308,10 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// and a unique RowKey per event, so the number of stored rows equals the true distinct
|
||||
/// count — making this an idempotent, retry-immune authoritative count (used to reconcile
|
||||
/// EventCount + RebootCount at the terminal transition). Single partition scan with a
|
||||
/// RowKey+EventType projection so both counts cost one query.
|
||||
/// slim projection so both counts cost one query. The CmTraceSkewTripwire's per-source
|
||||
/// timestamp-delta samples piggyback on the same enumeration (zero extra reads).
|
||||
/// </summary>
|
||||
private async Task<(int Total, int Reboots)?> CountSessionEventRowsAsync(string tenantId, string sessionId)
|
||||
private async Task<(int Total, int Reboots, SessionSkewScan Skew)?> CountSessionEventRowsAsync(string tenantId, string sessionId)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -3288,17 +3319,38 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var partitionKey = $"{tenantId}_{sessionId}";
|
||||
var query = tableClient.QueryAsync<TableEntity>(
|
||||
filter: $"PartitionKey eq '{partitionKey}'",
|
||||
select: new[] { "RowKey", "EventType" }
|
||||
select: new[] { "RowKey", "EventType", "Source", "ReceivedAt", BusinessTimestamp.OccurredUtcColumn, "TimestampClamped" }
|
||||
);
|
||||
|
||||
int total = 0, reboots = 0;
|
||||
var skew = new SessionSkewScan();
|
||||
var imeBatches = new HashSet<DateTime>();
|
||||
var otherBatches = new HashSet<DateTime>();
|
||||
const int maxSkewSamplesPerSide = 20_000;
|
||||
await foreach (var entity in query)
|
||||
{
|
||||
total++;
|
||||
if (entity.GetString("EventType") == Constants.EventTypes.SystemRebootDetected)
|
||||
reboots++;
|
||||
|
||||
// Skew sample. Clamped events carry a server-substituted OccurredUtc
|
||||
// (Δ ≈ 0) and would dilute the medians; legacy rows without ReceivedAt
|
||||
// have no server frame at all — both are skipped, never zero-filled.
|
||||
if (entity.GetBoolean("TimestampClamped") == true)
|
||||
continue;
|
||||
var receivedAt = entity.GetDateTimeOffset("ReceivedAt")?.UtcDateTime;
|
||||
var occurredAt = ResolveEventTimestampOrNull(entity);
|
||||
if (!receivedAt.HasValue || !occurredAt.HasValue)
|
||||
continue;
|
||||
var isIme = string.Equals(entity.GetString("Source"), "ImeLogTracker", StringComparison.OrdinalIgnoreCase);
|
||||
var side = isIme ? skew.ImeDeltaMinutes : skew.OtherDeltaMinutes;
|
||||
if (side.Count < maxSkewSamplesPerSide)
|
||||
side.Add((receivedAt.Value - occurredAt.Value).TotalMinutes);
|
||||
(isIme ? imeBatches : otherBatches).Add(receivedAt.Value);
|
||||
}
|
||||
return (total, reboots);
|
||||
skew.ImeDistinctBatchCount = imeBatches.Count;
|
||||
skew.OtherDistinctBatchCount = otherBatches.Count;
|
||||
return (total, reboots, skew);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -3307,6 +3359,49 @@ namespace AutopilotMonitor.Functions.Services
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Bounded histogram of the <c>sourceOffsetOrigin</c> provenance values ("bias",
|
||||
/// "line-anchored", "reader-zone-fallback", retired "calibrated") over a session's
|
||||
/// IME-derived events. Only read on the CmTraceSkewTripwire suspicion path (≈ never),
|
||||
/// so the row cap bounds the worst case rather than tuning the common one. Fail-soft:
|
||||
/// storage errors return an empty map, which the caller treats as "not bias-dominated"
|
||||
/// — the tripwire fails open toward reporting.
|
||||
/// </summary>
|
||||
public async Task<Dictionary<string, int>> GetImeOffsetOriginHistogramAsync(string tenantId, string sessionId, int maxRows = 500)
|
||||
{
|
||||
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
|
||||
SecurityValidator.EnsureValidGuid(sessionId, nameof(sessionId));
|
||||
|
||||
var histogram = new Dictionary<string, int>(StringComparer.OrdinalIgnoreCase);
|
||||
try
|
||||
{
|
||||
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.Events);
|
||||
var partitionKey = $"{tenantId}_{sessionId}";
|
||||
var query = tableClient.QueryAsync<TableEntity>(
|
||||
filter: $"PartitionKey eq '{partitionKey}' and Source eq 'ImeLogTracker'",
|
||||
select: new[] { "RowKey", "DataJson" });
|
||||
|
||||
int rows = 0;
|
||||
await foreach (var entity in query)
|
||||
{
|
||||
if (++rows > maxRows)
|
||||
break;
|
||||
var data = DeserializeEventData(entity.GetString("DataJson"));
|
||||
if (data != null && data.TryGetValue("sourceOffsetOrigin", out var origin))
|
||||
{
|
||||
var key = origin?.ToString();
|
||||
if (!string.IsNullOrEmpty(key))
|
||||
histogram[key!] = histogram.TryGetValue(key!, out var count) ? count + 1 : 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogDebug(ex, $"Could not read IME offset-origin histogram for session {sessionId}");
|
||||
}
|
||||
return histogram;
|
||||
}
|
||||
|
||||
#region IME Version History
|
||||
|
||||
/// <summary>
|
||||
|
||||
+93
-27
@@ -4,6 +4,7 @@ using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Functions.Services.Analyze;
|
||||
using AutopilotMonitor.Shared;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Extensions.Logging;
|
||||
@@ -255,36 +256,23 @@ namespace AutopilotMonitor.Functions.Services.Vulnerability
|
||||
{
|
||||
try
|
||||
{
|
||||
var scanSummary = reportData.TryGetValue("scan_summary", out var summaryObj)
|
||||
&& summaryObj is Dictionary<string, object> summaryDict
|
||||
? summaryDict
|
||||
: new Dictionary<string, object>();
|
||||
// Soft read: a storage hiccup degrades the event to Sequence 1 instead of
|
||||
// suppressing it entirely — the downstream re-analyze (ANALYZE-ID-003) needs
|
||||
// the event to exist more than it needs a perfect timeline position.
|
||||
var existingEvents = await _sessionRepo
|
||||
.GetSessionEventsAsync(tenantId, sessionId).ConfigureAwait(false);
|
||||
|
||||
var overallRisk = scanSummary.TryGetValue("overall_risk", out var riskObj)
|
||||
? riskObj?.ToString()
|
||||
: null;
|
||||
|
||||
var data = new Dictionary<string, object>(StringComparer.Ordinal)
|
||||
{
|
||||
["scan_summary"] = scanSummary,
|
||||
["reportAvailable"] = true,
|
||||
["phase"] = phaseLabel ?? "single",
|
||||
};
|
||||
|
||||
var evt = new EnrollmentEvent
|
||||
{
|
||||
TenantId = tenantId,
|
||||
SessionId = sessionId,
|
||||
EventType = "vulnerability_report",
|
||||
Severity = MapVulnerabilityRiskToSeverity(overallRisk),
|
||||
Source = "VulnerabilityCorrelationService",
|
||||
Phase = EnrollmentPhase.Unknown,
|
||||
Message = $"Vulnerability scan complete (risk={overallRisk ?? "unknown"}).",
|
||||
Timestamp = DateTime.UtcNow,
|
||||
Data = data,
|
||||
};
|
||||
var evt = BuildVulnerabilityReportEvent(
|
||||
tenantId, sessionId, reportData, phaseLabel, existingEvents, DateTime.UtcNow);
|
||||
|
||||
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt }).ConfigureAwait(false);
|
||||
|
||||
// StoreEventsBatchAsync writes the Events partition only; the cross-session
|
||||
// EventType index is normally written by EventIngestProcessor, which this
|
||||
// backend-materialized event never passes through (see the identical upsert in
|
||||
// ReportAgentErrorFunction for the incident that established the pattern).
|
||||
await _sessionRepo.UpsertEventTypeIndexBatchAsync(
|
||||
tenantId, sessionId, new List<EnrollmentEvent> { evt }).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
@@ -293,6 +281,84 @@ namespace AutopilotMonitor.Functions.Services.Vulnerability
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the synthetic <c>vulnerability_report</c> timeline event. Static + pure
|
||||
/// (analog to <c>ReportAgentErrorFunction.BuildAgentEmergencyBreakEvent</c>) so the
|
||||
/// field shape and the Sequence assignment are unit-testable.
|
||||
/// <para>
|
||||
/// <b>Sequence:</b> the correlation runs async off a queue, so the event arrives after
|
||||
/// the agent stream. Default is one past the session's max Sequence so it sorts LAST.
|
||||
/// Exception — WhiteGlove Part 1 (<c>phaseLabel == "device_setup"</c>): the web timeline
|
||||
/// splits Pre-Provisioning from User Enrollment strictly by Sequence
|
||||
/// (<c>computeWhiteGloveSplitSequence</c> in eventHelpers.ts), and if Part 2 is already
|
||||
/// running when the Part-1 scan lands, max+1 would push the device_setup report across
|
||||
/// the split and render it as the start of User Enrollment. The Part-1 report therefore
|
||||
/// anchors AT the max Sequence of the Part-1 terminal markers
|
||||
/// (<c>whiteglove_part1_complete</c> / <c>whiteglove_complete</c>) — provably ≤ the web
|
||||
/// split in every split branch, with the timestamp tie-break sorting it right behind the
|
||||
/// marker. Falls back to max+1 when no marker exists (then the web derives no split either).
|
||||
/// </para>
|
||||
/// </summary>
|
||||
internal static EnrollmentEvent BuildVulnerabilityReportEvent(
|
||||
string tenantId,
|
||||
string sessionId,
|
||||
Dictionary<string, object> reportData,
|
||||
string? phaseLabel,
|
||||
IReadOnlyList<EnrollmentEvent> existingEvents,
|
||||
DateTime nowUtc)
|
||||
{
|
||||
var scanSummary = reportData.TryGetValue("scan_summary", out var summaryObj)
|
||||
&& summaryObj is Dictionary<string, object> summaryDict
|
||||
? summaryDict
|
||||
: new Dictionary<string, object>();
|
||||
|
||||
var overallRisk = scanSummary.TryGetValue("overall_risk", out var riskObj)
|
||||
? riskObj?.ToString()
|
||||
: null;
|
||||
|
||||
var data = new Dictionary<string, object>(StringComparer.Ordinal)
|
||||
{
|
||||
["scan_summary"] = scanSummary,
|
||||
["reportAvailable"] = true,
|
||||
["phase"] = phaseLabel ?? "single",
|
||||
};
|
||||
|
||||
return new EnrollmentEvent
|
||||
{
|
||||
TenantId = tenantId,
|
||||
SessionId = sessionId,
|
||||
EventType = Constants.EventTypes.VulnerabilityReport,
|
||||
Severity = MapVulnerabilityRiskToSeverity(overallRisk),
|
||||
Source = "VulnerabilityCorrelationService",
|
||||
Phase = EnrollmentPhase.Unknown,
|
||||
Message = $"Vulnerability scan complete (risk={overallRisk ?? "unknown"}).",
|
||||
Timestamp = nowUtc,
|
||||
Sequence = ComputeReportEventSequence(phaseLabel, existingEvents),
|
||||
Data = data,
|
||||
};
|
||||
}
|
||||
|
||||
private static long ComputeReportEventSequence(
|
||||
string? phaseLabel, IReadOnlyList<EnrollmentEvent> existingEvents)
|
||||
{
|
||||
if (existingEvents == null || existingEvents.Count == 0)
|
||||
return 1L;
|
||||
|
||||
if (string.Equals(phaseLabel, "device_setup", StringComparison.Ordinal))
|
||||
{
|
||||
var anchor = existingEvents
|
||||
.Where(e =>
|
||||
string.Equals(e.EventType, Constants.EventTypes.WhiteGlovePart1Complete, StringComparison.OrdinalIgnoreCase)
|
||||
|| string.Equals(e.EventType, Constants.EventTypes.WhiteGloveComplete, StringComparison.OrdinalIgnoreCase))
|
||||
.Select(e => (long?)e.Sequence)
|
||||
.Max();
|
||||
if (anchor.HasValue)
|
||||
return anchor.Value;
|
||||
}
|
||||
|
||||
return existingEvents.Max(e => e.Sequence) + 1;
|
||||
}
|
||||
|
||||
private async Task SafeUpsertCveIndexAsync(
|
||||
VulnerabilityCorrelateEnvelope envelope,
|
||||
string sessionPrefix,
|
||||
|
||||
@@ -361,6 +361,41 @@ describe('analyze semantic lint', () => {
|
||||
expect(errors(validateRuleDraft(corr).findings).some((m) => m.includes('joinField'))).toBe(true);
|
||||
});
|
||||
|
||||
it('clock_skew: missing/unknown skewMetric, bad operator, bad threshold are errors; {{skewSummary}} resolves', () => {
|
||||
const missingMetric = validAnalyze();
|
||||
(missingMetric.conditions as Array<Record<string, unknown>>)[0] = {
|
||||
signal: 'clock_jump', source: 'clock_skew', operator: 'gte', value: '300', required: true,
|
||||
};
|
||||
expect(errors(validateRuleDraft(missingMetric).findings).some((m) => m.includes('skewMetric'))).toBe(true);
|
||||
|
||||
const badOperator = validAnalyze();
|
||||
(badOperator.conditions as Array<Record<string, unknown>>)[0] = {
|
||||
signal: 'clock_jump', source: 'clock_skew', skewMetric: 'clock_jump', operator: 'lt', value: '300', required: true,
|
||||
};
|
||||
expect(errors(validateRuleDraft(badOperator).findings).some((m) => m.includes('gt/gte'))).toBe(true);
|
||||
|
||||
const badThreshold = validAnalyze();
|
||||
(badThreshold.conditions as Array<Record<string, unknown>>)[0] = {
|
||||
signal: 'clock_jump', source: 'clock_skew', skewMetric: 'clock_jump', operator: 'gte', value: '-5', required: true,
|
||||
};
|
||||
expect(errors(validateRuleDraft(badThreshold).findings).some((m) => m.includes('positive numeric'))).toBe(true);
|
||||
|
||||
const noisy = validAnalyze();
|
||||
(noisy.conditions as Array<Record<string, unknown>>)[0] = {
|
||||
signal: 'clock_jump', source: 'clock_skew', skewMetric: 'clock_jump', operator: 'gte', value: '30', required: true,
|
||||
};
|
||||
expect(warnings(validateRuleDraft(noisy).findings).some((m) => m.includes('upload-latency noise'))).toBe(true);
|
||||
|
||||
const good = validAnalyze();
|
||||
(good.conditions as Array<Record<string, unknown>>)[0] = {
|
||||
signal: 'clock_jump', source: 'clock_skew', skewMetric: 'clock_jump', operator: 'gte', value: '300', required: true,
|
||||
};
|
||||
good.explanation = 'This device {{skewSummary}}.';
|
||||
const r = validateRuleDraft(good);
|
||||
expect(errors(r.findings)).toEqual([]);
|
||||
expect(warnings(r.findings).filter((m) => m.includes('{{'))).toEqual([]);
|
||||
});
|
||||
|
||||
it('invalid regex is an error; unanchored not_regex warns', () => {
|
||||
const bad = validAnalyze();
|
||||
(bad.conditions as Array<Record<string, unknown>>)[0] = {
|
||||
|
||||
@@ -110,6 +110,7 @@ export const EVENT_TYPES_CATALOG = {
|
||||
'timezone_auto_set',
|
||||
'ntp_time_check',
|
||||
'power_state_check',
|
||||
'power_state_change',
|
||||
],
|
||||
identity_events: [
|
||||
'aad_placeholder_user_detected',
|
||||
@@ -121,6 +122,7 @@ export const EVENT_TYPES_CATALOG = {
|
||||
'realmjoin_phase_changed',
|
||||
'realmjoin_resolved',
|
||||
'realmjoin_timeout',
|
||||
'realmjoin_timeout_extended',
|
||||
'realmjoin_first_deployment_incomplete',
|
||||
'realmjoin_package_started',
|
||||
'realmjoin_package_completed',
|
||||
|
||||
@@ -160,11 +160,22 @@ export const RULE_AUTHORING_GUIDE = {
|
||||
'Match a phase (eventType=esp_phase_changed, dataField=espPhase, operator=equals, ' +
|
||||
'value=<phase>); the evidence carries durationSeconds. Combine with a confidenceFactor ' +
|
||||
'"phase_duration > N" for the actual duration check.',
|
||||
app_install_duration: 'Compare app install duration in seconds (operator gt/lt/gte/lte, value=seconds).',
|
||||
app_install_duration:
|
||||
'Compare app install duration in seconds (operator gt/lt/gte/lte, value=seconds). ' +
|
||||
'Attempt-scoped: pairs each completion with the LATEST app_install_started before it ' +
|
||||
'(sequence order), so an IME re-evaluation pass an hour earlier does not inflate the ' +
|
||||
'value — same semantics as the apps dashboard since 2026-08.',
|
||||
event_correlation:
|
||||
'Join two event types over a shared field: eventType + correlateEventType + joinField, ' +
|
||||
'optional timeWindowSeconds, optional eventAFilterField/-Operator/-Value to pre-filter the ' +
|
||||
'first event, optional dataField+operator+value tested on the correlated pair.',
|
||||
clock_skew:
|
||||
'Device-clock check against the server receive frame: skewMetric=clock_jump (persistent ' +
|
||||
'mid-session step) or sustained_offset (whole session off by >= value). value = threshold ' +
|
||||
'seconds, operator gt/gte on the magnitude. No eventType/dataField; IME-log-derived events ' +
|
||||
'are excluded from the measurement. Batches from agents that send X-Send-Time-Utc are ' +
|
||||
'measured directly from the send time (spool-immune; evidence sentAtBatchCount counts them), ' +
|
||||
'older agents fall back to per-event timestamp medians.',
|
||||
},
|
||||
caseSensitivity:
|
||||
'source values are matched CASE-SENSITIVELY by the engine — write them exactly as listed ' +
|
||||
@@ -270,7 +281,7 @@ export const RULE_AUTHORING_GUIDE = {
|
||||
gatherOutputSeverities: ['Info', 'Warning', 'Error', 'Critical'],
|
||||
gatherCollectorTypes: ['registry', 'eventlog', 'wmi', 'file', 'json', 'xml', 'command_allowlisted', 'logparser'],
|
||||
gatherTriggers: ['startup', 'phase_change', 'phase_exit', 'interval', 'on_event'],
|
||||
analyzeConditionSources: ['event_type', 'event_data', 'event_data_array', 'phase_duration', 'event_count', 'app_install_duration', 'event_correlation'],
|
||||
analyzeConditionSources: ['event_type', 'event_data', 'event_data_array', 'phase_duration', 'event_count', 'app_install_duration', 'event_correlation', 'clock_skew'],
|
||||
analyzeEvaluateOnTriggers: ['enrollment_end', 'whiteglove_sealed', 'on_event:<eventType>'],
|
||||
},
|
||||
|
||||
|
||||
@@ -439,7 +439,16 @@ export const ANALYZE_RULE_SCHEMA: Record<string, unknown> = {
|
||||
"phase_duration",
|
||||
"event_count",
|
||||
"app_install_duration",
|
||||
"event_correlation"
|
||||
"event_correlation",
|
||||
"clock_skew"
|
||||
]
|
||||
},
|
||||
"skewMetric": {
|
||||
"type": "string",
|
||||
"description": "For clock_skew only: which device-clock metric to evaluate. clock_jump = persistent step in the device's clock frame mid-session; sustained_offset = whole session ran on a clock off by at least the threshold. value is the threshold in seconds; operator is limited to gt/gte on the magnitude. IME-log-derived events are excluded from the measurement.",
|
||||
"enum": [
|
||||
"clock_jump",
|
||||
"sustained_offset"
|
||||
]
|
||||
},
|
||||
"eventType": {
|
||||
|
||||
@@ -238,6 +238,7 @@ interface DraftCondition {
|
||||
required?: boolean;
|
||||
correlateEventType?: string;
|
||||
joinField?: string;
|
||||
skewMetric?: string;
|
||||
}
|
||||
|
||||
function checkEventTypeKnown(eventType: string | undefined, where: string): ValidationFinding[] {
|
||||
@@ -293,6 +294,24 @@ function lintAnalyzeRule(rule: Record<string, unknown>): ValidationFinding[] {
|
||||
findings.push(...checkEventTypeKnown(c.correlateEventType, `${label}.correlateEventType`));
|
||||
}
|
||||
|
||||
if (c.source === 'clock_skew') {
|
||||
if (c.skewMetric !== 'clock_jump' && c.skewMetric !== 'sustained_offset') {
|
||||
findings.push({ level: 'error', message: `${label}: clock_skew requires skewMetric ("clock_jump" or "sustained_offset").` });
|
||||
}
|
||||
if (c.operator !== 'gt' && c.operator !== 'gte') {
|
||||
findings.push({ level: 'error', message: `${label}: clock_skew supports only gt/gte — the threshold is compared against the skew MAGNITUDE.` });
|
||||
}
|
||||
const threshold = Number(c.value);
|
||||
if (!c.value || !Number.isFinite(threshold) || threshold <= 0) {
|
||||
findings.push({ level: 'error', message: `${label}: clock_skew requires a positive numeric value (threshold in seconds).` });
|
||||
} else if (threshold < 60) {
|
||||
findings.push({ level: 'warning', message: `${label}: clock_skew threshold below 60 s sits inside normal upload-latency noise — expect false positives.` });
|
||||
}
|
||||
if (c.eventType || c.dataField) {
|
||||
findings.push({ level: 'warning', message: `${label}: clock_skew ignores eventType/dataField — it measures Timestamp against ReceivedAt across the whole session.` });
|
||||
}
|
||||
}
|
||||
|
||||
findings.push(...checkEventTypeKnown(c.eventType, label));
|
||||
|
||||
if ((c.operator === 'regex' || c.operator === 'not_regex') && c.value) {
|
||||
@@ -364,6 +383,9 @@ function lintAnalyzeRule(rule: Record<string, unknown>): ValidationFinding[] {
|
||||
// event_data_array evidence carries the matched element under field=itemField,
|
||||
// so {{itemField}} interpolates at runtime (see EvaluateEventDataArrayCondition).
|
||||
if (c.itemField) resolvable.add(c.itemField);
|
||||
// clock_skew evidence carries a preformatted summary under field=skewSummary,
|
||||
// so {{skewSummary}} interpolates at runtime (see EvaluateClockSkewCondition).
|
||||
if (c.source === 'clock_skew') resolvable.add('skewSummary');
|
||||
}
|
||||
for (const token of extractTokens(rule)) {
|
||||
if (!resolvable.has(token)) {
|
||||
|
||||
@@ -1483,7 +1483,10 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
|
||||
description:
|
||||
'Get aggregated app-install health for Autopilot enrollments over a time window: the top failing apps ' +
|
||||
'(with failure counts, failure rate, and their most common failure codes), the slowest apps by average ' +
|
||||
'install duration, and a fleet "deliveryOptimization" rollup — total bytes downloaded and how much came ' +
|
||||
'install duration (durations measure the FINAL install attempt — the IME processes the app list in ' +
|
||||
'multiple passes, and evaluation passes / time queued behind other apps are not counted; rows written ' +
|
||||
'before 2026-08 carry the old first-observation-to-last-terminal span), and a fleet "deliveryOptimization" ' +
|
||||
'rollup — total bytes downloaded and how much came ' +
|
||||
'from peers / Microsoft Connected Cache (MCC) vs. the CDN, plus a peerOffloadPercent (bandwidth saved by ' +
|
||||
'not pulling from the internet). Use this to answer "which app breaks or slows down my enrollments?" and ' +
|
||||
'"how much install bandwidth is served locally?". ' +
|
||||
@@ -1528,7 +1531,10 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
|
||||
'time breakdown — wall-clock partition into device_prep / esp_apps / identity_hello / user_esp / ' +
|
||||
'desktop_handoff spans plus an EXPLICIT unattributed remainder (sums are exact against the session\'s ' +
|
||||
'authoritative duration; the WhiteGlove pause is excluded by design), the ESP-blocking app install ' +
|
||||
'intervals with critical-path occupancy, reboot outage spans, and data-quality flags. Omit sessionId ' +
|
||||
'intervals with critical-path occupancy (an app\'s seconds = its ACTIVE install time summed across all ' +
|
||||
'IME passes/attempts — deliberately broader than get_app_install_metrics\' final-attempt duration, ' +
|
||||
'because a failed attempt and an evaluation pass still occupied the critical path), reboot outage ' +
|
||||
'spans, and data-quality flags. Omit sessionId ' +
|
||||
'for the fleet rollup: rolling 30-day median/p75/p90 per segment per enrollment class (classes are ' +
|
||||
'never mixed) plus the top time-consuming ESP-blocking apps with "up to" what-if savings bounds. ' +
|
||||
'breakdown=null means the session has no computable attribution (pre-feature, non-terminal, or ' +
|
||||
|
||||
@@ -411,7 +411,7 @@ export function registerSessionTools(server: McpServer, ga: boolean, delegated:
|
||||
severity: z.enum(['Info', 'Warning', 'Error', 'Critical']).optional(),
|
||||
source: z.string().optional().describe('Filter by event source/app name (e.g. "MicrosoftTeams")'),
|
||||
fields: z.string().optional()
|
||||
.describe('Comma-separated lean projection (e.g. "eventType,severity,timestamp,message"). Drops the heavy "data" payload unless "data" is listed. Valid keys: eventId, sessionId, tenantId, eventType, severity, source, phase, phaseName, timestamp, receivedAt, message, sequence, rowKey, originalTimestamp, timestampClamped, causedByTransitionStepIndex, causedBySignalOrdinal, data.'),
|
||||
.describe('Comma-separated lean projection (e.g. "eventType,severity,timestamp,message"). Drops the heavy "data" payload unless "data" is listed. Valid keys: eventId, sessionId, tenantId, eventType, severity, source, phase, phaseName, timestamp, receivedAt, sentAt, message, sequence, rowKey, originalTimestamp, timestampClamped, causedByTransitionStepIndex, causedBySignalOrdinal, data.'),
|
||||
pageSize: z.coerce.number().int().min(1).max(1000).optional().default(200)
|
||||
.describe('Page size (1-1000, default 200). The endpoint returns this many events per call; follow nextLink to fetch more.'),
|
||||
continuation: z.string().optional()
|
||||
|
||||
@@ -151,6 +151,10 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
{
|
||||
data["realmjoinReleaseChannel"] = postState.RealmJoinFacts.ReleaseChannel.Value;
|
||||
}
|
||||
if (postState.RealmJoinFacts.LastActivityUtc != null)
|
||||
{
|
||||
data["realmjoinLastActivityUtc"] = FormatUtc(postState.RealmJoinFacts.LastActivityUtc.Value);
|
||||
}
|
||||
data["realmjoinPackages"] = BuildRealmJoinPackageSummary(postState.RealmJoinFacts.Packages);
|
||||
}
|
||||
|
||||
|
||||
@@ -11,16 +11,30 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
// RealmJoin (RJ) deployment-tracking handlers. Plan: tasks/zany-gathering-oasis plan.
|
||||
// The agent posts six DecisionSignalKinds when it observes the
|
||||
// HKLM\SYSTEM\CurrentControlSet\Services\realmjoin\Parameters key + HKLM\SOFTWARE\RealmJoin\Packages
|
||||
// (and the HKU\<sid>\... user-scope counterpart). Detection arms a 60-min hard timeout
|
||||
// (and the HKU\<sid>\... user-scope counterpart). Detection arms a 60-min timeout
|
||||
// and gates the enrollment-completion AND-gate. Resolution (phase 110), the
|
||||
// aborted-first-deployment release (phase left 100/101 for 200/210 without 110 —
|
||||
// session 224b2087) or the timeout releases the gate so the session can complete.
|
||||
// When the timeout fires while the first deployment is demonstrably still active
|
||||
// (phase 100/101 with deployment activity inside the last 60 min), the deadline is
|
||||
// re-armed instead — up to an absolute 4-h cap from detection (report 55e6afd61c9d:
|
||||
// a large package catalog outlived the 60-min window mid-install).
|
||||
public sealed partial class DecisionEngine
|
||||
{
|
||||
// Hard 60-minute deadline from RJ-detected. Not configurable (per design choice —
|
||||
// the trigger is intentionally aggressive, the timeout has to bound it).
|
||||
// 60-minute deadline from RJ-detected. Not configurable (per design choice —
|
||||
// the trigger is intentionally aggressive, the timeout has to bound it). Doubles
|
||||
// as the inactivity window for the activity-based extension: RJ package writes are
|
||||
// only observable at package COMPLETION (RJ creates the registry subkey when the
|
||||
// install finishes), so a single long install is registry-silent for its whole
|
||||
// duration and the window must not be tightened below the original 60 min.
|
||||
private static readonly TimeSpan s_realmJoinHardTimeout = TimeSpan.FromMinutes(60);
|
||||
|
||||
// Absolute ceiling for activity-based extensions, measured from RJ detection.
|
||||
// Preserves the original design intent (the aggressive detection trigger stays
|
||||
// bounded) and sits safely inside AgentMaxLifetimeMinutes (360) — the agent's own
|
||||
// lifetime watchdog would otherwise be the only backstop.
|
||||
private static readonly TimeSpan s_realmJoinAbsoluteTimeout = TimeSpan.FromHours(4);
|
||||
|
||||
// Payload keys consumed by the new handlers (set by the agent's RealmJoinWatcherAdapter).
|
||||
// Public because the V2.Core agent adapter assembles signal payloads with these keys.
|
||||
public static class RealmJoinPayloadKeys
|
||||
@@ -64,14 +78,16 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
|| facts.Outcome != null;
|
||||
|
||||
/// <summary>
|
||||
/// Build the 60-min RJ hard-timeout deadline. Floored at <see cref="DecisionState.AgentBootUtc"/>
|
||||
/// via <see cref="EffectiveDeadlineBase"/> so a replayed RealmJoinDetected signal cannot
|
||||
/// collapse the timer into immediate-fire at boot.
|
||||
/// Build the RJ timeout deadline for an explicit due time. Initial arm: detection
|
||||
/// (floored at <see cref="DecisionState.AgentBootUtc"/> via <see cref="EffectiveDeadlineBase"/>
|
||||
/// so a replayed RealmJoinDetected signal cannot collapse the timer into immediate-fire
|
||||
/// at boot) + 60 min. Re-arm (activity-based extension): last activity + 60 min,
|
||||
/// capped at detection + 4 h.
|
||||
/// </summary>
|
||||
private static ActiveDeadline BuildRealmJoinTimeoutDeadline(DateTime fromUtc) =>
|
||||
private static ActiveDeadline BuildRealmJoinTimeoutDeadline(DateTime dueAtUtc) =>
|
||||
new ActiveDeadline(
|
||||
name: DeadlineNames.RealmJoinTimeout,
|
||||
dueAtUtc: fromUtc.Add(s_realmJoinHardTimeout),
|
||||
dueAtUtc: dueAtUtc,
|
||||
firesSignalKind: DecisionSignalKind.DeadlineFired,
|
||||
firesPayload: new Dictionary<string, string>
|
||||
{
|
||||
@@ -134,7 +150,8 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
var effects = Array.Empty<DecisionEffect>();
|
||||
if (!alreadyDetected)
|
||||
{
|
||||
var deadline = BuildRealmJoinTimeoutDeadline(EffectiveDeadlineBase(state, signal));
|
||||
var deadline = BuildRealmJoinTimeoutDeadline(
|
||||
EffectiveDeadlineBase(state, signal).Add(s_realmJoinHardTimeout));
|
||||
builder.AddDeadline(deadline);
|
||||
effects = new[]
|
||||
{
|
||||
@@ -224,13 +241,20 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
private DecisionStep HandleRealmJoinTimeoutDeadlineFired(DecisionState state, DecisionSignal signal)
|
||||
{
|
||||
var alreadyResolvedOrTimedOut = state.RealmJoinFacts.Outcome != null;
|
||||
var deadlineStillArmed = false;
|
||||
ActiveDeadline? armedDeadline = null;
|
||||
foreach (var d in state.Deadlines)
|
||||
{
|
||||
if (d.Name == DeadlineNames.RealmJoinTimeout) { deadlineStillArmed = true; break; }
|
||||
if (d.Name == DeadlineNames.RealmJoinTimeout) { armedDeadline = d; break; }
|
||||
}
|
||||
|
||||
if (alreadyResolvedOrTimedOut || !deadlineStillArmed)
|
||||
// Third staleness shape (introduced with the activity-based extension): the fire
|
||||
// belongs to an OLDER deadline incarnation that a re-arm has since replaced —
|
||||
// recognizable because the armed deadline is due LATER than this fire
|
||||
// (OccurredAtUtc = DueAtUtc per the scheduler contract). Without this guard the
|
||||
// stale fire would evaluate the timeout/extension decision ahead of schedule.
|
||||
var supersededByRearm = armedDeadline != null && armedDeadline.DueAtUtc > signal.OccurredAtUtc;
|
||||
|
||||
if (alreadyResolvedOrTimedOut || armedDeadline == null || supersededByRearm)
|
||||
{
|
||||
var bookkept = BumpStepBookkeeping(state, signal);
|
||||
var staleTransition = BuildDeadEndTransition(
|
||||
@@ -240,12 +264,60 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
trigger: $"DeadlineFired:{DeadlineNames.RealmJoinTimeout}",
|
||||
deadEndReason: alreadyResolvedOrTimedOut
|
||||
? "realmjoin_timeout_stale_outcome_already_set"
|
||||
: "realmjoin_timeout_stale_deadline_not_armed");
|
||||
: armedDeadline == null
|
||||
? "realmjoin_timeout_stale_deadline_not_armed"
|
||||
: "realmjoin_timeout_stale_superseded_by_rearm");
|
||||
return new DecisionStep(bookkept, staleTransition, Array.Empty<DecisionEffect>());
|
||||
}
|
||||
|
||||
var nextStep = state.StepIndex + 1;
|
||||
|
||||
// Activity-based extension (report 55e6afd61c9d): when the first deployment is
|
||||
// demonstrably still active — phase 100/101 with deployment activity (phase change
|
||||
// or package observation) inside the last 60 min — re-arm instead of cutting off
|
||||
// mid-install. Bounded by an absolute 4-h cap from detection so the original
|
||||
// "aggressive trigger stays bounded" design intent survives. The idle case
|
||||
// (detected, but the first deployment never produced any activity) is unchanged:
|
||||
// LastActivityUtc stays null and the 60-min timeout fires exactly as before.
|
||||
var facts = state.RealmJoinFacts;
|
||||
var now = signal.OccurredAtUtc;
|
||||
var lastActivity = facts.LastActivityUtc?.Value;
|
||||
var detectedUtc = facts.DetectedUtc?.Value;
|
||||
var inFirstDeployment = facts.LastDeploymentPhase != null
|
||||
&& IsRealmJoinFirstDeploymentPhase(facts.LastDeploymentPhase.Value);
|
||||
var recentActivity = lastActivity != null && now - lastActivity.Value < s_realmJoinHardTimeout;
|
||||
var withinCap = detectedUtc != null && now - detectedUtc.Value < s_realmJoinAbsoluteTimeout;
|
||||
|
||||
if (inFirstDeployment && recentActivity && withinCap)
|
||||
{
|
||||
var windowDue = lastActivity!.Value.Add(s_realmJoinHardTimeout);
|
||||
var capDue = detectedUtc!.Value.Add(s_realmJoinAbsoluteTimeout);
|
||||
var newDue = windowDue < capDue ? windowDue : capDue;
|
||||
|
||||
var rearm = BuildRealmJoinTimeoutDeadline(newDue);
|
||||
var rearmBuilder = state.ToBuilder()
|
||||
.WithStepIndex(nextStep)
|
||||
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal)
|
||||
.CancelDeadline(DeadlineNames.RealmJoinTimeout)
|
||||
.AddDeadline(rearm);
|
||||
|
||||
var rearmState = rearmBuilder.Build();
|
||||
var rearmTransition = BuildTakenTransition(
|
||||
before: state,
|
||||
signal: signal,
|
||||
toStage: state.Stage,
|
||||
nextStepIndex: nextStep,
|
||||
trigger: $"DeadlineFired:{DeadlineNames.RealmJoinTimeout}:Extended");
|
||||
|
||||
var rearmEffects = new[]
|
||||
{
|
||||
new DecisionEffect(DecisionEffectKind.ScheduleDeadline, deadline: rearm),
|
||||
BuildRealmJoinTimeoutExtendedEvent(facts, lastActivity.Value, newDue),
|
||||
};
|
||||
|
||||
return new DecisionStep(rearmState, rearmTransition, rearmEffects);
|
||||
}
|
||||
|
||||
var builder = state.ToBuilder()
|
||||
.WithStepIndex(nextStep)
|
||||
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal)
|
||||
@@ -253,7 +325,15 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
|
||||
builder.RealmJoinFacts = state.RealmJoinFacts.WithTimeoutOutcome(signal.SessionSignalOrdinal);
|
||||
|
||||
var timeoutEffect = BuildRealmJoinTimeoutEvent(state);
|
||||
// Message differentiates WHY the gate gave up: absolute cap exhausted while still
|
||||
// active, activity went quiet, or the original hard timeout (no activity ever seen).
|
||||
var reason = inFirstDeployment && recentActivity && !withinCap
|
||||
? RealmJoinTimeoutReasonAbsoluteCap
|
||||
: inFirstDeployment && lastActivity != null
|
||||
? RealmJoinTimeoutReasonInactivity
|
||||
: RealmJoinTimeoutReasonHardTimeout;
|
||||
|
||||
var timeoutEffect = BuildRealmJoinTimeoutEvent(state, reason);
|
||||
|
||||
return CompleteIfDeferredOrBookkeep(
|
||||
state: state,
|
||||
@@ -331,7 +411,9 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
var builder = state.ToBuilder()
|
||||
.WithStepIndex(nextStep)
|
||||
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
|
||||
builder.RealmJoinFacts = state.RealmJoinFacts.WithLastPhase(currentPhase.Value, signal.SessionSignalOrdinal);
|
||||
builder.RealmJoinFacts = state.RealmJoinFacts
|
||||
.WithLastPhase(currentPhase.Value, signal.SessionSignalOrdinal)
|
||||
.WithActivity(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
|
||||
|
||||
if (!IsFirstDeploymentAbortTransition(state.RealmJoinFacts, previousPhase, currentPhase.Value))
|
||||
{
|
||||
@@ -442,12 +524,14 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
var builder = state.ToBuilder()
|
||||
.WithStepIndex(nextStep)
|
||||
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
|
||||
builder.RealmJoinFacts = state.RealmJoinFacts.WithPackageStarted(
|
||||
packageId: packageId!,
|
||||
displayName: displayName,
|
||||
version: version,
|
||||
scope: scope,
|
||||
startedUtc: signal.OccurredAtUtc);
|
||||
builder.RealmJoinFacts = state.RealmJoinFacts
|
||||
.WithPackageStarted(
|
||||
packageId: packageId!,
|
||||
displayName: displayName,
|
||||
version: version,
|
||||
scope: scope,
|
||||
startedUtc: signal.OccurredAtUtc)
|
||||
.WithActivity(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
|
||||
|
||||
var newState = builder.Build();
|
||||
var transition = BuildTakenTransition(
|
||||
@@ -490,14 +574,16 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
var builder = state.ToBuilder()
|
||||
.WithStepIndex(nextStep)
|
||||
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
|
||||
builder.RealmJoinFacts = state.RealmJoinFacts.WithPackageCompleted(
|
||||
packageId: packageId!,
|
||||
displayName: displayName,
|
||||
version: version,
|
||||
scope: scope,
|
||||
completedUtc: signal.OccurredAtUtc,
|
||||
success: success,
|
||||
lastExitCode: lastExitCode);
|
||||
builder.RealmJoinFacts = state.RealmJoinFacts
|
||||
.WithPackageCompleted(
|
||||
packageId: packageId!,
|
||||
displayName: displayName,
|
||||
version: version,
|
||||
scope: scope,
|
||||
completedUtc: signal.OccurredAtUtc,
|
||||
success: success,
|
||||
lastExitCode: lastExitCode)
|
||||
.WithActivity(signal.OccurredAtUtc, signal.SessionSignalOrdinal);
|
||||
|
||||
var newState = builder.Build();
|
||||
var transition = BuildTakenTransition(
|
||||
@@ -654,6 +740,15 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
return null;
|
||||
}
|
||||
|
||||
// Reason strings carried in the realmjoin_timeout event payload ("reason") and
|
||||
// reflected in its message. HardTimeout = original semantics (no deployment activity
|
||||
// ever observed, 60 min from detection); Inactivity = the first deployment produced
|
||||
// activity at some point but went quiet for a full 60-min window; AbsoluteCap = still
|
||||
// active but the 4-h extension ceiling from detection is exhausted.
|
||||
internal const string RealmJoinTimeoutReasonHardTimeout = "hard_timeout";
|
||||
internal const string RealmJoinTimeoutReasonInactivity = "inactivity";
|
||||
internal const string RealmJoinTimeoutReasonAbsoluteCap = "absolute_cap";
|
||||
|
||||
/// <summary>
|
||||
/// Build the <c>realmjoin_timeout</c> timeline-entry effect emitted by the deadline-
|
||||
/// fired handler. Unlike <see cref="DecisionSignalKind.RealmJoinDetected"/> /
|
||||
@@ -661,7 +756,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
/// InformationalEvent), the timeout is a synthetic deadline so the reducer owns its
|
||||
/// timeline visibility.
|
||||
/// </summary>
|
||||
private static DecisionEffect BuildRealmJoinTimeoutEvent(DecisionState state)
|
||||
private static DecisionEffect BuildRealmJoinTimeoutEvent(DecisionState state, string reason)
|
||||
{
|
||||
var facts = state.RealmJoinFacts;
|
||||
var lastPhase = facts.LastDeploymentPhase?.Value ?? 0;
|
||||
@@ -672,17 +767,60 @@ namespace AutopilotMonitor.DecisionCore.Engine
|
||||
if (facts.Packages[i].CompletedUtc != null) completed++;
|
||||
}
|
||||
|
||||
var message = reason == RealmJoinTimeoutReasonAbsoluteCap
|
||||
? $"RealmJoin did not reach phase 110 within {(int)s_realmJoinAbsoluteTimeout.TotalHours} h of detection (last phase: {lastPhase}) — monitoring window exhausted despite recent deployment activity."
|
||||
: reason == RealmJoinTimeoutReasonInactivity
|
||||
? $"RealmJoin did not reach phase 110 — no deployment activity for {(int)s_realmJoinHardTimeout.TotalMinutes} min (last phase: {lastPhase})."
|
||||
: $"RealmJoin did not reach phase 110 within {(int)s_realmJoinHardTimeout.TotalMinutes} min (last phase: {lastPhase}).";
|
||||
|
||||
return new DecisionEffect(
|
||||
kind: DecisionEffectKind.EmitEventTimelineEntry,
|
||||
parameters: new Dictionary<string, string>(StringComparer.Ordinal)
|
||||
{
|
||||
[SignalPayloadKeys.EventType] = SharedConstants.EventTypes.RealmJoinTimeout,
|
||||
[SignalPayloadKeys.Source] = "DecisionEngine",
|
||||
[SignalPayloadKeys.Severity] = "Warning",
|
||||
[SignalPayloadKeys.Message] = message,
|
||||
["lastSeenPhase"] = lastPhase.ToString(CultureInfo.InvariantCulture),
|
||||
["packagesTracked"] = tracked.ToString(CultureInfo.InvariantCulture),
|
||||
["packagesCompleted"] = completed.ToString(CultureInfo.InvariantCulture),
|
||||
["reason"] = reason,
|
||||
});
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Build the <c>realmjoin_timeout_extended</c> timeline-entry effect emitted when the
|
||||
/// timeout deadline fires but the first deployment is demonstrably still active and
|
||||
/// the monitoring window is re-armed instead. Reducer-owned like
|
||||
/// <see cref="BuildRealmJoinTimeoutEvent"/> (synthetic deadline, no agent dual-emit).
|
||||
/// </summary>
|
||||
private static DecisionEffect BuildRealmJoinTimeoutExtendedEvent(
|
||||
RealmJoinFacts facts,
|
||||
DateTime lastActivityUtc,
|
||||
DateTime newDueUtc)
|
||||
{
|
||||
var lastPhase = facts.LastDeploymentPhase?.Value ?? 0;
|
||||
var tracked = facts.Packages.Count;
|
||||
var completed = 0;
|
||||
for (var i = 0; i < facts.Packages.Count; i++)
|
||||
{
|
||||
if (facts.Packages[i].CompletedUtc != null) completed++;
|
||||
}
|
||||
|
||||
return new DecisionEffect(
|
||||
kind: DecisionEffectKind.EmitEventTimelineEntry,
|
||||
parameters: new Dictionary<string, string>(StringComparer.Ordinal)
|
||||
{
|
||||
[SignalPayloadKeys.EventType] = SharedConstants.EventTypes.RealmJoinTimeout,
|
||||
[SignalPayloadKeys.EventType] = SharedConstants.EventTypes.RealmJoinTimeoutExtended,
|
||||
[SignalPayloadKeys.Source] = "DecisionEngine",
|
||||
[SignalPayloadKeys.Severity] = "Warning",
|
||||
[SignalPayloadKeys.Message] = $"RealmJoin did not reach phase 110 within 60 min (last phase: {lastPhase}).",
|
||||
["lastSeenPhase"] = lastPhase.ToString(CultureInfo.InvariantCulture),
|
||||
[SignalPayloadKeys.Severity] = "Info",
|
||||
[SignalPayloadKeys.Message] =
|
||||
$"RealmJoin first deployment still active (phase {lastPhase}, {completed}/{tracked} packages completed, last activity {lastActivityUtc:HH:mm:ss} UTC) — extending monitoring window to {newDueUtc:HH:mm:ss} UTC (cap {(int)s_realmJoinAbsoluteTimeout.TotalHours} h after detection).",
|
||||
["deploymentPhase"] = lastPhase.ToString(CultureInfo.InvariantCulture),
|
||||
["packagesTracked"] = tracked.ToString(CultureInfo.InvariantCulture),
|
||||
["packagesCompleted"] = completed.ToString(CultureInfo.InvariantCulture),
|
||||
["lastActivityUtc"] = lastActivityUtc.ToString("O", CultureInfo.InvariantCulture),
|
||||
["extendedUntilUtc"] = newDueUtc.ToString("O", CultureInfo.InvariantCulture),
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -39,7 +39,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: null,
|
||||
productVersion: null,
|
||||
releaseChannel: null,
|
||||
packages: Array.Empty<RealmJoinPackageFact>());
|
||||
packages: Array.Empty<RealmJoinPackageFact>(),
|
||||
lastActivityUtc: null);
|
||||
|
||||
public RealmJoinFacts(
|
||||
SignalFact<DateTime>? detectedUtc,
|
||||
@@ -49,7 +50,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
SignalFact<bool>? selfDeployingDeferredCompletion,
|
||||
SignalFact<string>? productVersion,
|
||||
SignalFact<string>? releaseChannel,
|
||||
IReadOnlyList<RealmJoinPackageFact> packages)
|
||||
IReadOnlyList<RealmJoinPackageFact> packages,
|
||||
SignalFact<DateTime>? lastActivityUtc = null)
|
||||
{
|
||||
DetectedUtc = detectedUtc;
|
||||
ResolvedUtc = resolvedUtc;
|
||||
@@ -59,6 +61,7 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
ProductVersion = productVersion;
|
||||
ReleaseChannel = releaseChannel;
|
||||
Packages = packages ?? Array.Empty<RealmJoinPackageFact>();
|
||||
LastActivityUtc = lastActivityUtc;
|
||||
}
|
||||
|
||||
/// <summary>First-observation timestamp of the <c>HKLM\SYSTEM\...\realmjoin\Parameters</c> key.</summary>
|
||||
@@ -98,6 +101,15 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
/// <summary>Tracked per-package install rows (machine + user scope combined).</summary>
|
||||
public IReadOnlyList<RealmJoinPackageFact> Packages { get; }
|
||||
|
||||
/// <summary>
|
||||
/// Most recent observed RJ deployment activity: a <c>RealmJoinPhaseChanged</c> or a
|
||||
/// per-package started/completed observation. Deliberately NOT set at detection time —
|
||||
/// the RJ-detected-but-never-deploying idle case must still hit the 60-min hard
|
||||
/// timeout unchanged. Drives the activity-based timeout extension (report 55e6afd61c9d:
|
||||
/// the hard timeout cut off a demonstrably active first deployment mid-install).
|
||||
/// </summary>
|
||||
public SignalFact<DateTime>? LastActivityUtc { get; }
|
||||
|
||||
public RealmJoinFacts WithDetected(DateTime utc, long sourceSignalOrdinal)
|
||||
{
|
||||
if (DetectedUtc != null) return this; // set-once
|
||||
@@ -109,7 +121,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: Packages);
|
||||
packages: Packages,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -129,7 +142,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
|
||||
productVersion: new SignalFact<string>(productVersion, sourceSignalOrdinal),
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: Packages);
|
||||
packages: Packages,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -149,7 +163,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: new SignalFact<string>(releaseChannel, sourceSignalOrdinal),
|
||||
packages: Packages);
|
||||
packages: Packages,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
|
||||
public RealmJoinFacts WithResolved(DateTime utc, int phase, long sourceSignalOrdinal)
|
||||
@@ -163,7 +178,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: Packages);
|
||||
packages: Packages,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -184,7 +200,28 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: Packages);
|
||||
packages: Packages,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Record an RJ deployment-activity observation (phase change or package event).
|
||||
/// Monotonic — an earlier or equal timestamp (replayed signal) is a no-op, so the
|
||||
/// activity fact can never move backwards.
|
||||
/// </summary>
|
||||
public RealmJoinFacts WithActivity(DateTime utc, long sourceSignalOrdinal)
|
||||
{
|
||||
if (LastActivityUtc != null && LastActivityUtc.Value >= utc) return this;
|
||||
return new RealmJoinFacts(
|
||||
detectedUtc: DetectedUtc,
|
||||
resolvedUtc: ResolvedUtc,
|
||||
lastDeploymentPhase: LastDeploymentPhase,
|
||||
outcome: Outcome,
|
||||
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: Packages,
|
||||
lastActivityUtc: new SignalFact<DateTime>(utc, sourceSignalOrdinal));
|
||||
}
|
||||
|
||||
public RealmJoinFacts WithLastPhase(int phase, long sourceSignalOrdinal)
|
||||
@@ -198,7 +235,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: Packages);
|
||||
packages: Packages,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
|
||||
public RealmJoinFacts WithTimeoutOutcome(long sourceSignalOrdinal)
|
||||
@@ -212,7 +250,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: Packages);
|
||||
packages: Packages,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
|
||||
public RealmJoinFacts WithSelfDeployingDeferred(long sourceSignalOrdinal)
|
||||
@@ -226,7 +265,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: new SignalFact<bool>(true, sourceSignalOrdinal),
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: Packages);
|
||||
packages: Packages,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -246,7 +286,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: null,
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: Packages);
|
||||
packages: Packages,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -301,7 +342,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: copy);
|
||||
packages: copy,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -370,7 +412,8 @@ namespace AutopilotMonitor.DecisionCore.State
|
||||
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
|
||||
productVersion: ProductVersion,
|
||||
releaseChannel: ReleaseChannel,
|
||||
packages: copy);
|
||||
packages: copy,
|
||||
lastActivityUtc: LastActivityUtc);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -512,12 +512,17 @@ namespace AutopilotMonitor.Shared
|
||||
// RealmJoin (RJ) deployment tracking — driven by HKLM\SYSTEM\...\realmjoin\Parameters
|
||||
// + HKLM/HKU \SOFTWARE\RealmJoin\Packages\<id>. Detection extends the V2 enrollment
|
||||
// session lifetime until DeploymentPhase reaches CompletedFirstDeployment (110) or
|
||||
// the 60-min hard timeout fires. Per-package events surface install lifecycle in
|
||||
// the 60-min timeout fires. Per-package events surface install lifecycle in
|
||||
// the same shape as IME app_install_started / app_install_completed.
|
||||
public const string RealmJoinDetected = "realmjoin_detected";
|
||||
public const string RealmJoinPhaseChanged = "realmjoin_phase_changed";
|
||||
public const string RealmJoinResolved = "realmjoin_resolved";
|
||||
public const string RealmJoinTimeout = "realmjoin_timeout";
|
||||
// Info: the timeout deadline fired while the first deployment was demonstrably
|
||||
// still active (phase 100/101, deployment activity inside the last 60 min) —
|
||||
// the monitoring window was re-armed instead of timing out (capped at 4 h from
|
||||
// detection). Report 55e6afd61c9d.
|
||||
public const string RealmJoinTimeoutExtended = "realmjoin_timeout_extended";
|
||||
// Warning: DeploymentPhase left the first-deployment window (100/101) for 200/210
|
||||
// without CompletedFirstDeployment (110) — the RJ ESP was aborted (typically an
|
||||
// interactive logon during the first deployment reclassified the run as
|
||||
@@ -557,6 +562,7 @@ namespace AutopilotMonitor.Shared
|
||||
public const string TimezoneAutoSet = "timezone_auto_set";
|
||||
public const string NtpTimeCheck = "ntp_time_check";
|
||||
public const string PowerStateCheck = "power_state_check";
|
||||
public const string PowerStateChange = "power_state_change";
|
||||
public const string AgentTrace = "agent_trace";
|
||||
|
||||
// Crash / process lifecycle
|
||||
|
||||
@@ -109,8 +109,17 @@ namespace AutopilotMonitor.Shared.DataAccess
|
||||
/// Reconciles the stored EventCount and RebootCount with the authoritative row counts
|
||||
/// from the Events table (rows dedupe on deterministic RowKeys; the read-modify-write
|
||||
/// increments do not). Call as the last counter write on terminal ingest batches.
|
||||
/// Returns the per-source timestamp-delta samples collected during the same partition
|
||||
/// scan (input to the CMTrace time-skew tripwire), or null when the scan failed.
|
||||
/// </summary>
|
||||
Task ReconcileSessionCountersAsync(string tenantId, string sessionId);
|
||||
Task<SessionSkewScan?> ReconcileSessionCountersAsync(string tenantId, string sessionId);
|
||||
|
||||
/// <summary>
|
||||
/// Bounded histogram of the sourceOffsetOrigin provenance values over a session's
|
||||
/// IME-derived events ("bias" / "line-anchored" / "reader-zone-fallback"). Read only
|
||||
/// on the CMTrace skew tripwire's suspicion path; fail-soft (empty map on error).
|
||||
/// </summary>
|
||||
Task<Dictionary<string, int>> GetImeOffsetOriginHistogramAsync(string tenantId, string sessionId, int maxRows = 500);
|
||||
Task UpdateSessionDiagnosticsBlobAsync(
|
||||
string tenantId, string sessionId, string blobName, string? destination = null);
|
||||
Task SetSessionPreProvisionedAsync(string tenantId, string sessionId, bool isPreProvisioned,
|
||||
|
||||
@@ -6,11 +6,41 @@ using System.Text.RegularExpressions;
|
||||
namespace AutopilotMonitor.Shared.Logging
|
||||
{
|
||||
/// <summary>
|
||||
/// Represents a single parsed line from a CMTrace-format log file
|
||||
/// Represents a single parsed line from a CMTrace-format log file.
|
||||
/// <para>
|
||||
/// Timestamps are deliberately NOT resolved to UTC here unless the writer declared its own
|
||||
/// offset. See <see cref="CmTraceLogParser"/> for why.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class CmTraceLogEntry
|
||||
{
|
||||
public DateTime Timestamp { get; set; }
|
||||
/// <summary>
|
||||
/// The line's local time exactly as the writing process wrote it. Kind is
|
||||
/// <see cref="DateTimeKind.Unspecified"/> on purpose — this value is meaningless without
|
||||
/// knowing which zone the WRITER believed it was in, which the line itself does not say.
|
||||
/// </summary>
|
||||
public DateTime LocalTimestamp { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// The writer-declared UTC bias in minutes, when the line carried one ("+480").
|
||||
/// <c>null</c> for the far more common bias-less form.
|
||||
/// </summary>
|
||||
public int? BiasMinutes { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// UTC — populated ONLY when <see cref="BiasMinutes"/> was present, because a
|
||||
/// writer-declared offset is authoritative. <c>null</c> otherwise: the caller must resolve
|
||||
/// <see cref="LocalTimestamp"/> itself, and is the only party able to do so correctly.
|
||||
/// </summary>
|
||||
public DateTime? TimestampUtc { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Whether the line carried a parseable timestamp at all. When <c>false</c>,
|
||||
/// <see cref="LocalTimestamp"/> is <c>default</c> and callers must fall back to their own
|
||||
/// clock rather than treat the value as a real time.
|
||||
/// </summary>
|
||||
public bool HasTimestamp { get; set; }
|
||||
|
||||
public string Message { get; set; } = string.Empty;
|
||||
public string Component { get; set; } = string.Empty;
|
||||
public int Type { get; set; } // 1=Info, 2=Warning, 3=Error
|
||||
@@ -21,12 +51,25 @@ namespace AutopilotMonitor.Shared.Logging
|
||||
/// Parses CMTrace/SCCM log format used by IME and other Microsoft components.
|
||||
/// Format: <![LOG[{message}]LOG]!><time="{time}" date="{date}" component="{comp}" context="" type="{type}" thread="{thread}" file="">
|
||||
///
|
||||
/// Lives in Shared because it is the SINGLE parsing implementation for both sides:
|
||||
/// the agent's IME tracker / logparser gather collector (on-device) and the backend's
|
||||
/// rules/gather/test-pattern endpoint, which must reproduce the agent's matching
|
||||
/// semantics exactly so authors can test patterns without a device. Timestamp note:
|
||||
/// bias-less lines parse with DateTimeStyles.AssumeLocal — on the backend that is the
|
||||
/// SERVER's timezone, so the test endpoint reports match/parse results, not timestamps.
|
||||
/// <para>
|
||||
/// Lives in Shared because it is the SINGLE parsing implementation for both sides: the
|
||||
/// agent's IME tracker / logparser gather collector (on-device) and the backend's
|
||||
/// rules/gather/test-pattern endpoint, which must reproduce the agent's matching semantics
|
||||
/// exactly so authors can test patterns without a device.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// <b>The parser does not guess timezones.</b> A bias-less CMTrace line carries local time
|
||||
/// and nothing else — IME 1.104 writes <c>DateTime.Now.TimeOfDay</c> in both of its trace
|
||||
/// listeners. Resolving that with the READER's <c>TimeZoneInfo.Local</c> silently assumes the
|
||||
/// writing and reading processes agree on the zone. They do not: each caches its zone at
|
||||
/// process start and neither follows a later <c>tzutil</c> or a Windows auto-timezone change.
|
||||
/// Field measurement over 11,068 sessions found 26 sessions where the two beliefs differed,
|
||||
/// from +1 h to -17 h, silently shifting every IME-derived event by that amount. So this
|
||||
/// parser hands back <see cref="CmTraceLogEntry.LocalTimestamp"/> plus
|
||||
/// <see cref="CmTraceLogEntry.BiasMinutes"/> and lets the caller — which alone can measure
|
||||
/// the writer's actual offset — do the conversion.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class CmTraceLogParser
|
||||
{
|
||||
@@ -63,11 +106,24 @@ namespace AutopilotMonitor.Shared.Logging
|
||||
var typeStr = match.Groups["type"].Value;
|
||||
var thread = match.Groups["thread"].Value;
|
||||
|
||||
// Parse timestamp: date is "M-d-yyyy", time is "HH:mm:ss.ticks"
|
||||
DateTime timestamp;
|
||||
if (!TryParseTimestamp(dateStr, timeStr, biasStr, out timestamp))
|
||||
DateTime localTimestamp;
|
||||
var hasTimestamp = TryParseLocalTimestamp(dateStr, timeStr, out localTimestamp);
|
||||
|
||||
int? biasMinutes = null;
|
||||
if (!string.IsNullOrEmpty(biasStr)
|
||||
&& int.TryParse(biasStr, NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture, out var parsedBias))
|
||||
{
|
||||
timestamp = DateTime.UtcNow;
|
||||
biasMinutes = parsedBias;
|
||||
}
|
||||
|
||||
// A writer-declared bias is authoritative — it is the writer telling us its own
|
||||
// offset, which is exactly the fact we otherwise have to measure.
|
||||
// GetTimeZoneInformation convention: UTC = local + bias.
|
||||
DateTime? timestampUtc = null;
|
||||
if (hasTimestamp && biasMinutes.HasValue)
|
||||
{
|
||||
timestampUtc = DateTime.SpecifyKind(
|
||||
localTimestamp.AddMinutes(biasMinutes.Value), DateTimeKind.Utc);
|
||||
}
|
||||
|
||||
int type;
|
||||
@@ -75,7 +131,10 @@ namespace AutopilotMonitor.Shared.Logging
|
||||
|
||||
entry = new CmTraceLogEntry
|
||||
{
|
||||
Timestamp = timestamp,
|
||||
LocalTimestamp = localTimestamp,
|
||||
BiasMinutes = biasMinutes,
|
||||
TimestampUtc = timestampUtc,
|
||||
HasTimestamp = hasTimestamp,
|
||||
Message = message,
|
||||
Component = component,
|
||||
Type = type,
|
||||
@@ -85,7 +144,26 @@ namespace AutopilotMonitor.Shared.Logging
|
||||
return true;
|
||||
}
|
||||
|
||||
private static bool TryParseTimestamp(string dateStr, string timeStr, string biasStr, out DateTime result)
|
||||
/// <summary>
|
||||
/// Resolves a bias-less CMTrace local timestamp using the READING process's own timezone.
|
||||
///
|
||||
/// <para>
|
||||
/// <b>This is a fallback with a known defect, not a correct conversion.</b> It is right
|
||||
/// only when the writing process happens to hold the same zone belief as this one. Use it
|
||||
/// solely where no better information exists — a log this process does not tail and
|
||||
/// therefore cannot measure an offset against — and mark the resulting value as not
|
||||
/// source-grounded so the inaccuracy stays visible.
|
||||
/// </para>
|
||||
///
|
||||
/// <para>
|
||||
/// Callers that DO tail their log must instead measure the writer's offset from a freshly
|
||||
/// appended line (see the agent's <c>CmTraceOffsetCalibrator</c>).
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static DateTime ResolveUtcAssumingReaderZone(DateTime localTimestamp)
|
||||
=> DateTime.SpecifyKind(localTimestamp, DateTimeKind.Local).ToUniversalTime();
|
||||
|
||||
private static bool TryParseLocalTimestamp(string dateStr, string timeStr, out DateTime result)
|
||||
{
|
||||
result = DateTime.MinValue;
|
||||
|
||||
@@ -129,29 +207,15 @@ namespace AutopilotMonitor.Shared.Logging
|
||||
"M-d-yyyy HH:mm:ss"
|
||||
};
|
||||
|
||||
// Bias-suffixed time ("+480"): the writer declared its own UTC offset in minutes
|
||||
// (GetTimeZoneInformation convention: UTC = written time + bias). Honor it instead
|
||||
// of assuming the agent's local timezone — writer and agent can disagree (mixed
|
||||
// components, mid-enrollment timezone change; session df1fcf47).
|
||||
if (!string.IsNullOrEmpty(biasStr)
|
||||
&& int.TryParse(biasStr, NumberStyles.AllowLeadingSign, CultureInfo.InvariantCulture, out var biasMinutes))
|
||||
{
|
||||
if (DateTime.TryParseExact(combined, formats, CultureInfo.InvariantCulture,
|
||||
DateTimeStyles.None, out result))
|
||||
{
|
||||
result = DateTime.SpecifyKind(result.AddMinutes(biasMinutes), DateTimeKind.Utc);
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
// No bias: CMTrace timestamps are in LOCAL time. Parse as local, then convert to UTC.
|
||||
// DateTimeStyles.None: the value stays Unspecified. Attaching Local or Utc here would
|
||||
// be the very guess this parser refuses to make.
|
||||
if (DateTime.TryParseExact(combined, formats, CultureInfo.InvariantCulture,
|
||||
DateTimeStyles.AssumeLocal, out result))
|
||||
DateTimeStyles.None, out result))
|
||||
{
|
||||
result = result.ToUniversalTime();
|
||||
return true;
|
||||
}
|
||||
|
||||
result = default(DateTime);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user