Compare commits

...

29 Commits

Author SHA1 Message Date
Oliver Kieselbach b0bb7233e4 chore: drop Outbound IP row from Connectivity section
The outbound_ip trace event stays available for analysis (timeline,
MCP); only the UI row is removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:20:35 +02:00
Oliver Kieselbach 6758dc59e4 refactor: move API latency into Device Details "Connectivity" section
The extra grid cell in the Session Info card forced a fourth row once
Device Validation was present. The latency reading is a "when needed"
detail, so it moves into the collapsed Device Details card as a new
Connectivity section instead — together with values that were collected
but never displayed anywhere: the estimated internet bandwidth
(network_bandwidth_estimate incl. DO-throttle hint), total telemetry
upload volume (agent_metrics_snapshot), and the outbound IP.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 12:12:08 +02:00
Oliver Kieselbach ffd70e0383 feat: geographic API-latency view (per-location weighted avg by geo)
Surfaces the persisted per-session AvgApiLatencyMs in the geographic
views so operators can compare regions when deciding on new backend
locations (Asia/US):

- ComputeGeographicMetrics: per-location and global request-weighted
  latency (sum(avg*requests)/sum(requests)) + ApiLatencyVsGlobalPct;
  serves both the tenant and the GA cross-tenant endpoint
- GeoMetricsSessionProjection extended (column-projected scan would
  otherwise drop the new fields); equivalence test pins it
- Drilldown: LocationSessionRow carries the per-session values; sessions
  table + summary tile show them
- geographic-performance page: sortable "API Latency" column with
  absolute color buckets (<250 green .. >=800 red), global benchmark
- PlatformMetricsService: session AvgLatency now exact from cumulative
  counters when present (fallback: mean of interval averages)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:58:53 +02:00
Oliver Kieselbach c18154d76d feat: persist per-session agent-to-backend API latency (AvgApiLatencyMs)
The V2 agent already measures HTTP round-trip latency per request
(NetworkMetrics, cumulative _totalLatencyMs) but only emitted the
per-interval delta average. Operators deciding on new backend regions
(Asia/US) need the session-wide average, aggregatable by GeoCountry.
Measured baseline: Jakarta sessions average ~600-1050 ms client-side vs
94 ms server-side processing — the gap is pure network distance to GWC.

- Agent: agent_metrics_snapshot now carries cumulative net_total_latency_ms
  (total/requests = whole-session average; idempotent by construction)
- Ingest: last matching snapshot of a batch projects AvgApiLatencyMs +
  ApiRequestCount onto the Session row (fire-and-forget, fail-soft,
  last-write-wins; agents predating the field simply never match)
- SessionsIndex: both fields mirrored (merge + BuildSessionIndexEntity
  superset) so list/GA views can aggregate latency per geo
- Web: session detail shows "API Latency" when present

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-28 11:44:46 +02:00
Oliver Kieselbach f524473305 Merge branch 'main' of https://github.com/okieselbach/Autopilot-Monitor 2026-07-28 09:54:07 +02:00
Oliver Kieselbach 8fe420b902 updated commands 2026-07-28 09:53:57 +02:00
dependabot[bot] 2e8c2cdb2f chore(deps): bump @modelcontextprotocol/sdk (#110)
Bumps the mcp group in /src/McpServer/autopilot-monitor-mcp with 1 update: [@modelcontextprotocol/sdk](https://github.com/modelcontextprotocol/typescript-sdk).


Updates `@modelcontextprotocol/sdk` from 1.29.0 to 1.30.0
- [Release notes](https://github.com/modelcontextprotocol/typescript-sdk/releases)
- [Commits](https://github.com/modelcontextprotocol/typescript-sdk/compare/v1.29.0...1.30.0)

---
updated-dependencies:
- dependency-name: "@modelcontextprotocol/sdk"
  dependency-version: 1.30.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: mcp
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-28 09:44:13 +02:00
dependabot[bot] e04b75dd53 chore(deps): bump the web group (#111)
Bumps the web group in /src/Web/autopilot-monitor-web with 3 updates: [next](https://github.com/vercel/next.js), [@next/bundle-analyzer](https://github.com/vercel/next.js/tree/HEAD/packages/next-bundle-analyzer) and [eslint-config-next](https://github.com/vercel/next.js/tree/HEAD/packages/eslint-config-next).


Updates `next` from 15.5.21 to 15.5.22
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/compare/v15.5.21...v15.5.22)

Updates `@next/bundle-analyzer` from 16.2.11 to 16.2.12
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/commits/v16.2.12/packages/next-bundle-analyzer)

Updates `eslint-config-next` from 15.5.21 to 15.5.22
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/commits/v15.5.22/packages/eslint-config-next)

---
updated-dependencies:
- dependency-name: next
  dependency-version: 15.5.22
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: "@next/bundle-analyzer"
  dependency-version: 16.2.12
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: eslint-config-next
  dependency-version: 15.5.22
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-28 09:41:30 +02:00
Oliver Kieselbach d55b0ff864 test(backend): pin the cross-tenant redaction on metrics/ime-versions
GET metrics/ime-versions is MemberRead and the archive is one global
partition, so every tenant member reads the same platform-wide rows. The only
thing between an ordinary tenant user and another tenant's identifiers was an
inline projection with no test behind it.

There is no HTTP-level harness in this suite - nothing constructs an
HttpRequestData with a RequestContext - so the projection was unreachable from
a test while it lived inside Run. Extracted it to a static
BuildResponsePayload(versions, hasGlobalScope), matching the pattern
GetTpmPssUnsupportedFunction already uses for the same reason. Behaviour is
unchanged: the payload is returned as `object`, and System.Text.Json resolves
an `object` declared type via the runtime type, so both branches keep the wire
format they had inline. A test asserts that rather than trusting it.

10 tests, asserting on serialized JSON in camelCase (Program.cs's naming
policy) because a property dropped from an anonymous type is only a real
redaction once it is absent from the wire:

  - non-global: firstSeenTenantId / firstSeenSessionId absent, and their GUID
    VALUES absent anywhere in the payload (a model rename must not let the
    identifier survive under another key)
  - non-global: exactly {version, firstSeenAt, lastSeenAt, sessionCount}
  - non-global: sessionCount survives - the counts are platform-wide by
    decision, pinned so it cannot drift either way unnoticed
  - global (Admin or Reader): full entry including both identifiers
  - both scopes: array shape, empty archive, row order preserved

Mutation-checked both directions: forcing the global branch fails 3 tests,
dropping sessionCount from the projection fails 2. Full suite 3521 passed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:33:56 +02:00
Oliver Kieselbach 123ba0e55e fix(web): fleet-health crash in GA all-tenants scope - repeatDevices absent
The worker serializer omits null fields (WhenWritingNull), so the
aggregated global/metrics/device-journeys response carries no
repeatDevices at all. The strict `=== null` check missed undefined and
fell through to repeatDevices.length, crashing the page. Verified
against the deployed chunk: crash column 11464 is the `0===o.length`
read behind the `null===o` guard.

Loose `== null` check + optional DTO field; absent and null both render
the "select a tenant" hint.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 19:28:14 +02:00
Oliver Kieselbach c8175ff7b1 fix(mcp): drop npm from the runtime image, bump tar to 7.5.22
Defender for Cloud reports CVEs against the mcp container image that the
repo scanners cannot see. The subassessment evidence paths split them in two:

  /usr/local/lib/node_modules/npm/**  - the npm 10.9.8 that node:22-slim
  bundles, vendoring @sigstore/core 2.0.0, brace-expansion 2.0.2, tar 7.5.11
  (critical), sigstore, picomatch, ip-address, plus npm itself (high, no fix
  available). Present in all 20 images including the current one. Dependabot
  reads package-lock.json, so none of this is visible to it.

  /app/node_modules/**  - body-parser 2.2.2, fast-uri 3.1.2, hono, protobufjs,
  tar 7.5.16, all confined to images from 2026-07-21 and earlier and already
  fixed by 7df208fa/c3e9621a. Those findings age out with the old ACR tags.

npm is build-time only here: CMD is `node dist/index.js`, the HEALTHCHECK is
`node -e`, the container app spec overrides no command and probes over HTTP,
and src/ contains no child_process call. So remove npm rather than upgrade it
- npm 11.x still vendors @sigstore/core 3.1.0, itself inside the vulnerable
<= 3.2.0 range, and would clear nothing.

Separately, tar 7.5.20 -> 7.5.22 in our own tree (GHSA-r292-9mhp-454m, via
@huggingface/transformers -> onnxruntime-node). 7.5.22 satisfies the declared
^7.0.1, so no override is needed. fast-uri deliberately stays on 3.1.4: it is
patched against all four fast-uri advisories, and ajv 8.18.0 declares ^3.0.1,
so forcing 4.x would be an override against a major range.

npm audit --omit=dev: 0 vulnerabilities. tsc clean, 455 tests pass.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 19:00:07 +02:00
Oliver Kieselbach 0efeb0f734 fix(backend): alert-webhook 500 on non-seekable body - bounded read replaces Length check
With ASP.NET Core integration the HttpRequestData body is a
forward-only stream whose Length property throws NotSupportedException
(verified in prod: 500 + System.NotSupportedException on the first
POST). The size cap is now enforced by reading at most cap+1 chars via
ReadBlockAsync instead of asking the stream for its length.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:52:59 +02:00
Oliver Kieselbach 7b975d46b6 feat(backend): Azure Monitor alert webhook - platform alerts join the ops-event bus
New anonymous POST ops/alert-webhook receives Common Alert Schema
notifications from an Action Group webhook receiver and re-emits them
as ops events (category Platform, type AzureMonitorAlert), so Azure
Monitor alerts flow through the existing OpsEventService ->
OpsAlertDispatchService fan-out (Telegram/Teams/Slack + portal
ops-events list) like every internally raised operational event. The
delivery path is independent of the portal: a SWA outage cannot
suppress the notification; the action group's e-mail receiver stays as
the backend-independent fallback.

Security: action-group webhooks cannot send custom headers, so the
shared secret travels as ?secret= and is compared in constant time
(FixedTimeEquals) against the OpsAlertWebhookSecret app setting -
fail-closed while unset. 64 KB payload cap, defensive essentials-only
parsing (only alertRule is mandatory), severity mapped Sev0->Critical,
Sev1->Error, Sev2->Warning, Sev3/4->Info with unknown values falling
back to Info so schema drift can never escalate.

Dual-registered per convention: EndpointAccessPolicyCatalog entry
(PublicAnonymous), OPS_EVENT_TYPES in OpsAlertRulesSection (rule
wiring) and Platform category in OpsEventsSection (badge + filter).

Backend suite 3511 green (incl. new helper tests + catalog
completeness), web tsc + vitest 651 green.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:39:22 +02:00
Oliver Kieselbach 7680bdff7b fix(web): disable RSC prefetch on nav links - portal freeze root cause
Navbar + GlobalSidebar render the full navigation on every page, so the
App Router's default viewport prefetch fired one ?_rsc= request per
visible link against the SWA runtime (~10.4k requests / 14 days). That
origin serves document, chunks and RSC payloads over HTTP/1.1 (6
connections per origin); when the runtime queued the prefetches (p95
896ms, tail up to 234s), they held every socket and the portal froze
until the queue drained. Measured on the 2026-07-27 17:25 incident:
/admin/presence?_rsc= at 151s while the backend API answered in <2.2s.

All 16 nav links now set prefetch={false}. Every target route is a
client component behind ProtectedRoute that fetches its own data after
mount, so the prefetched RSC payload carried no benefit.

New OKF doc docs/web/portal-navigation-prefetch.md records the full
evidence chain and where browser telemetry lives (separate App Insights
resource, not reachable via MCP query_backend_logs).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 18:27:24 +02:00
Oliver Kieselbach 93094653a2 fix(web): tenant-scope seed races on usage-metrics/sla - own-tenant stats shown for override
A GA with a persisted tenant override opening /usage-metrics (or /sla) saw
their OWN tenant's stats until a manual refresh. useGlobalAdminScope seeded
the persisted selection in a post-mount effect, so the page effect first saw
effectiveTenantId = own tenant and fired an own-tenant fetch immediately
followed by the override fetch; with raw authenticatedFetch (no latest-wins
guard, unlike useAuthenticatedFetch since 82934a37) the last-RESOLVED
response won.

- useGlobalAdminScope: seed the GA selection render-time (same pattern as
  the delegated seed) so consumers never observe a transient own-tenant
  effectiveTenantId - the wrong fetch no longer starts at all.
- usage-metrics/sla pages: latest-wins sequence guard in fetchMetrics so a
  superseded request can neither write data/notifications nor clear the
  newer request's loading state (covers rapid selector switches).

Verified: tsc 0 errors, vitest 651/651.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 16:08:12 +02:00
Oliver Kieselbach ba7d14b070 fix(insights): post-deploy MCP findings - projection drop, aggregate starvation, duration artifacts
Three defects found by measuring production after the 1.5.1012 deploy:

- AppMetricsProjection dropped TerminalState, so the PR0 skip
  classification never reached metrics/app (totalSkipped stuck at 0).
  Column added; equivalence fixture now populates TerminalState/AppId/
  EspBlocking/AppIdCollision and asserts them (proven: test fails
  without the projection fix).
- BuildTimeAttributionAggregates excluded breakdowns on ANY quality
  flag; 903/1000 carry BlockingSetUnknown, starving fleet stats
  (Clean=12 vs 331 excluded on the largest tenant). New
  TimeAttributionFlagQuality.DurationCriticalFlags mask - only
  ClockSkew/PartialObservation/WGAnchors exclude; blocking-set flags
  keep their measured spans (positive-evidence semantics unchanged).
- HasMeasuredDuration gets an upper plausibility bound (6h = default
  AgentMaxLifetimeMinutes): 490 rows carried back-stamped completions
  (up to 117 days; 60175s app rows in a 2697s session) that dominated
  slowestApps with six-figure averages. Read-side, heals old + new rows.

Backend 3484/3484 green; solution build clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 14:50:55 +02:00
Oliver Kieselbach 7f7330e9e5 feat(web): auto-recover from post-deploy stale-chunk failures (one reload, loop-guarded)
A tab still running the previous deployment's bundle requests lazily-loaded
chunks (next/dynamic charts, route chunks) whose content-hashed filenames no
longer exist after the SWA swap - the user is stuck on "Loading chart..." or a
dead navigation until a manual refresh. Standard remedy: reload once so the
fresh document references the new bundle.

- utils/chunkReloadRecovery.ts: chunk-error detection (webpack ChunkLoadError +
  Chromium/Firefox/Safari dynamic-import variants) and a sessionStorage guard -
  at most one auto-reload per tab per 5 minutes; without working storage no
  auto-reload at all, so a reload loop is structurally impossible. Emits a
  ChunkReloadRecovery App Insights event before reloading.
- components/ChunkReloadRecovery.tsx (root layout): catches rejected dynamic
  imports via unhandledrejection and failed /_next/ script/link tags via a
  capture-phase error listener (third-party scripts never trigger a reload).
- app/error.tsx: a chunk error reaching the boundary reloads once instead of
  showing the error card; inside the guard window it falls through to the card.

Verified: tsc clean, vitest 651/651 (8 new tests).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 12:47:34 +02:00
Oliver Kieselbach 82934a3719 fix(web): tenant-aware unrestricted badge + latest-wins fetch guard on rules pages
GA override on gather-rules called GET /api/global/config/{tenantId}, a route
that does not exist in the backend; the swallowed 404 left the unrestricted
badge showing the GA's HOME tenant's mode for every foreign tenant. Both paths
now read the member-readable config/{tenantId}/feature-flags (MemberRead has a
global-scope bypass and the route is RouteParam-scoped), with a reset-on-switch
so a failed fetch never carries the previous tenant's mode over. The dead
globalConfig.tenant entry is removed from the URL register.

useAuthenticatedFetch now enforces latest-wins: the scope hook seeds a
persisted foreign tenant only after mount, so an own-tenant fetch races the
override fetch and the last-RESOLVED response used to win, leaving the rules
list on the home tenant while the selector showed the override. Stale requests
no longer write data/error/loading; the spinner is owned by the last-started
non-silent request.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 12:26:01 +02:00
Oliver Kieselbach 372f0c7c98 feat(web): source pill for non-Intune install rows + per-source row key
Customers who additionally deploy Microsoft 365 Apps as their own Win32 app saw two
rows in Install Progress (the IME view and the Office C2R lifecycle) with nothing
telling them apart, which read as a duplicate.

- New lib/installProgress.ts: buildInstallItems() extracted from the component so the
  event-folding rules are unit-testable; component is now render-only.
- InstallItem gains `source` (ime | office-c2r | realmjoin) and a source-scoped `key`.
  Rows were keyed by app name alone, so a Win32 app named exactly "Microsoft 365 Apps"
  (= OfficeInstallDetector.AppName) collided with the C2R row and overwrote its state.
- Pill only for non-default sources: "Click-to-Run" / "RealmJoin", neutral grey so it
  does not compete with the state badge, with a tooltip explaining what the row measures.
  Plain Intune app rows stay unlabelled.
- Drops the "RJ: " name prefix, which broke sorting and truncation.

Verified: tsc clean, vitest 643/643 (16 new).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-27 11:24:53 +02:00
Oliver Kieselbach 8057b44e48 fix(agent): serialize ESP-config collects so a later emission never carries an older snapshot
Codex review (P2): the ESP-config refresh triggers each run on their own
Task.Run (as do CollectAll's startup sweep and CollectAtEnrollmentStart),
so two concurrent collects could interleave read-before-emit - the LATER
emission then carried the OLDER (smaller) registry snapshot. The backend
uses the latest list-bearing esp_config_detected emission as positive
blocking evidence, so an out-of-order emission would silently shrink it.

CollectEspConfiguration now runs its read->emit sequence inside a
dedicated lock covering ALL callers; a later emission always carries an
equal-or-newer snapshot. The race predates the Q2 refresh triggers (they
only raised its frequency).

Deterministic concurrency test: 12 parallel refreshes against a probe
whose list grows per read - the emission sequence must be exactly 1..12.

Verified: Agent V2 Core 2059/2059.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 10:52:56 +02:00
Oliver Kieselbach 7842d121a0 fix(insights): codex review findings - WG pause clamping, late-event healing, stale buckets, date windows, banner count
All six review findings verified against code and real; five fixed here,
the agent-side race lands in the companion agent commit.

[P1] WhiteGlove pause counted as time: ClampToWindows clipped intervals
against the HULL of both WG windows, but the excluded pause lies inside
the hull - a reboot gap or app interval straddling the pause counted the
entire pause (a ~2-day "reboot" in a 30-minute enrollment was possible).
Seconds are now the per-window intersection sum (chronological endpoints
kept); an interval wholly inside the pause makes no claim and no longer
raises ClockSkewDropped. What-if savings are additionally capped at the
session's WallClockSeconds - endpoint deltas can still straddle the
pause, and no removal can save more than the enrollment took.

[P1] Late-arriving events never updated breakdowns: the terminal-seam
compute is one-shot and the sweep only healed missing/version-stale rows.
SessionTimeBreakdown now stores EventCountAtCompute (0 = pre-fix row,
recomputed once); the sweep recomputes on EventCount drift and then also
re-runs ResolveEspBlockingForSessionAsync (newly exposed on
IMetricsRepository) so late esp_config_detected re-emits and app batches
reach both the breakdown and the blocking evidence.

[P2] Stale aggregate buckets: the sweeps now reconcile every partition
they wrote - buckets that existed but were not regenerated (sessions
deleted, class left the window) are deleted via new targeted deletes;
rolling30 rows additionally get a 48h ComputedAt freshness filter at
read time, so a fully quiet tenant stops serving year-old numbers as
"last 30 days".

[P2] Date windows included an extra calendar day: both inclusive range
ends meant days=30 returned 31 keys and days=1 summed yesterday AND
today. InclusiveWindowStart(today, days) = today-(days-1) in both
response builders; the sweep's rolling30 source is filtered to the same
30 calendar days (its query window stays one day wider for backfill).

[P3] Device-history banner counted later sessions as "previous
enrollments" when viewing an older session: selectPriorSessions
(deviceHistoryPrior.ts) anchors on the session's own chain entry (live
sessions: new sessionStartedAt prop), sessionId tie-break mirroring the
server chain sort.

Verified: solution build clean, Backend 3480/3480 (+6: three WG-pause
golden fixtures, window-start facts, RK builder/delete consistency,
EventCountAtCompute roundtrip), Web tsc + vitest 633/633 (+6).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 10:52:39 +02:00
Oliver Kieselbach 4bc2435ec5 fix(agent): audit Q1/Q2/Q5 source fixes - timing re-arm, esp_config re-emits, serial trim
Q1 (AppInstallTiming): a terminal->active transition (IME retry path,
e.g. Error->Downloading or Skipped->Downloading) re-arms CompletedAtUtc
so the final terminal payload covers first-start -> last-terminal instead
of freezing at the first terminal state. Plus a negative-duration guard
in AppInstallTiming.DurationSeconds (inverted endpoints -> null, field
omitted) - observed in production as durationSeconds=-2342 on a
Skipped-then-retried user-scope app (session 66f1c5e1).

Q2 (esp_config_detected): the ESP blocking lists are read too early -
device lists grow progressively, user-scope S-<SID> lists appear only
after sign-in (88 of 89 prod sessions had an empty user list at last
emission). DeviceInfoHost now re-collects the ESP configuration as
phase-driven one-shots: apps sub-phase declaration (AppsDevice/AppsUser
phase_transition = first app activity) and AccountSetup detection.
The StartupEventGate keeps unchanged payloads silent (fingerprint covers
list contents), so only grown lists actually re-emit. F1 EspBlocking
resolution (latest list-bearing emission) benefits without backend change.

Q5 (HardwareInfo): WMI values (serial, manufacturer, model) trimmed at
the source via NormalizeWmiValue; whitespace-only collapses to the
"Unknown" sentinel the retry logic already treats as unresolved.

Attach-window investigation (11a3b67a): NO agent fix - hypothesis
disproved against two real agent logs (diag bundles db5ce007/2.0.1337,
4a9e9273/2.0.1379). "completed without started" first apps were already
installed (WinGet detection-only, zero install lines in AppWorkload.log);
pre-attach replay provably works (pattern-match log). PR0's
HasMeasuredDuration already handles the class server-side.
Duplicate app_install_completed (66f1c5e1): mechanism narrowed to an
in-process re-read of exactly one AppWorkload rotation window (<24h-old
lines, historic suppression not applicable) - documented as follow-up,
no fix without a bundle-bearing repro per lessons.md.

Verified: solution build clean, Agent V2 Core 2058/2058 (14 new tests),
DecisionCore 493/493. No ConfigVersion bump, no new event types,
decision rail untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:59:20 +02:00
Oliver Kieselbach 1bf37f6761 feat(insights): F3 regression radar - Wilson-gated rule hit-rate drift detection (PR6)
Analyze rules only (gather stats stay per-batch-deduped until Q6 is
fixed - never silently included). Per (tenant, rule) the radar compares
the trailing 7-day hit rate (FireCount/SessionsEvaluated from the daily
RuleStats rows) against the prior 28-day baseline and fires only when
the window has >=5 hit sessions and >=20 evaluated sessions, the rate
is >=2x baseline AND the Wilson 95% intervals are disjoint
(MetricsMath.WilsonInterval/RateIncreaseSeparated, hand-computed test
vectors). Suppressed for deleted rules, rules inside the CreatedAt
grace period (<35d), rules edited inside the window, empty baselines,
and the RuleRegressionRadarDisabled app-setting kill switch.

Alert episodes live as ruleregression|{ruleId} rows in the notification
tracker (third keyspace): AddEntity-409 dedup = one bell + one
RuleFrequencyRegression ops event (category Tenant, Warning, full
numbers, dual-registered in the web OPS_EVENT_TYPES) per episode;
numbers refresh while active without moving FirstNotifiedAt; episodes
re-arm under 1.5x baseline, on fire-stop, or via the tracker's 30d
retention prune (spec-intended reminder). On-fire dimension correlation
(osBuild/model/agentVersion/imeVersion; hit set from a RuleResults
range scan intersected with the window sessions) reports both shares
with the "correlated, not necessarily causal" wording contract or
honestly "no clear dimension concentration". Radar runs in the daily
maintenance pass after the rule-stats aggregation, anchored on
yesterday (whole days only).

Surfaces: regressions[] in the rule-stats response (MCP get_rule_stats
inherits it), tenant bell deep-linking to the rule card, and the
analyze-rules page gains a densified 30-day fire sparkline plus a
regression badge while an episode is active. OKF doc
backend/rule-regression-radar.md + index/log.

Backend 3474/3474 (27 new: Wilson vectors, every suppression branch,
window boundary dates, re-arm matrix, dimension gates, tracker
round-trips incl. tri-states, keyspace dedup/prune, wording pins),
web tsc + vitest 627/627, MCP tsc + 455 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 09:10:52 +02:00
Oliver Kieselbach 88da317bf4 feat(insights): F2 surfaces - device-history banner, fleet FTR panel, API routes, MCP tool (PR5)
Three read routes over the PR4 data: metrics/device-history
(MemberRead + tenant query param - one route serves the web banner AND
MCP, GA cross-tenant via ?tenantId= with a sessionId-based owner
fallback; history:null is a normal outcome), metrics/device-journeys
and global/metrics/device-journeys (days selector honored because FTR
counts are additive - window rate = ratio of summed daily rows; the
tenant-less global variant serves the "global" partition and discloses
repeatDevices:null). Attempt numbers are server-computed
(DeviceJourneyCalculator.ComputeAttemptNumber: chain position within
the journey; live sessions via a virtual non-successful attempt so the
redeploy and 30d-gap rules apply) - no consumer re-derives journey
semantics. Repeat-devices violator list: last journey >=2 attempts with
newest terminal ref in-window, attempts desc then recency, cap 10,
failure reason from the newest failed attempt (bounded point-reads).

Web: DeviceHistoryBanner on session detail ("Attempt N for this device
- View history", expandable terminal chain, durations strictly the
authoritative durationSeconds, WG/DevPrep/admin-marked badges) and the
Fleet Health "First-time-right" section (rate gated at >=20 completed
journeys with disclosed n, junk-serial exclusion note, Monday-UTC
weekly trend, attempt histogram, repeat devices) following the 7/30/90
selector. MCP: get_device_history (serial mode / fleet FTR mode,
role-aware routing, all roles - GA_FULL privilege snapshot extended
deliberately). Docs: OKF backend/device-journeys.md + index/log.

Backend 3447/3447 (14 new: attempt-number vectors incl. gap boundaries
and live sessions, additive sums, days clamping, repeat-device
gate/sort/cap), web tsc + vitest 627/627, MCP tsc + 455 passed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 08:38:17 +02:00
Oliver Kieselbach 1e6de47433 feat(insights): F2 journey grouping + DeviceHistories chains + daily FTR aggregates (PR4)
Device-key normalization (trim+casefold, junk-serial exclusion incl. the
agent's "Unknown" WMI sentinel), pure journey grouping (terminal set
Succeeded/Failed/Incomplete; journey ends at first success; >30d gap from
the previous attempt's CompletedAt starts a new journey; WG Pending stays
an open session, never an attempt), and the DeviceHistories table
(PK tenant, RK encoded normalized serial): terminal-only session-ref
chains capped at 20 with derived journey counts, written inline at the
session-terminal seam (all three terminal statuses, DeletionState-guarded)
and healed by a rolling 30d maintenance sweep. The sweep prunes chain refs
of deleted sessions tombstone-driven (SessionTombstones cover cascade AND
retention deletions), deletes emptied rows, and recomputes daily FTR
aggregates (DeviceJourneyAggregates, PK tenant/"global", RK date):
completed journeys, first-time-right count, attempt histogram and
junk-serial disclosure — additive counts, bucketed on the completing
success session's StartedAt date. Chain refs carry the authoritative
DurationSeconds verbatim (never CompletedAt-StartedAt) plus an
adminMarked flag. Both tables registered in Constants.All + offboarding
TenantPartitionTables; aggregates under the 180d retention block;
deliberately NO per-session deletion-manifest step for DeviceHistories.

Backend 3433/3433 (51 new tests: junk/normalization vectors, terminal
set, exact 30d gap boundaries, WG one-row-one-attempt, redeploy split,
deletion pruning, entity round-trips incl. row-key escaping, FTR
aggregation core), Agent 2044+493 green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 08:14:13 +02:00
Oliver Kieselbach 6de21e821a feat(insights): F1 surfaces — session lane, fleet panel, API routes, MCP tool (PR3)
Insights expansion PR3 (tasks/insights-expansion-spec.md §F1 "Surfaces").
Completes F1: schema (PR1) -> persistence/aggregation (PR2) -> surfaces.

- Rolling range rows: the sweep now also writes "rolling30|{class}"
  aggregate rows — range statistics over the window's actual sessions. A
  median of per-day medians is not the range median, so the fleet panel
  reads these and is deliberately fixed to "last 30 days"; the daily rows
  stay in the payload for trends/MCP.
- Routes (all registered in EndpointAccessPolicyCatalog):
  GET sessions/{sessionId}/time-attribution (MemberRead + tenant query
  param, GA fallback via FindSessionTenantId; breakdown=null is SUCCESS —
  a missing lane is a normal state, not an error), GET
  metrics/time-attribution (MemberRead) and GET
  global/metrics/time-attribution (GlobalReadOrAdmin + tenantId filter),
  both returning { windowDays, classes (rolling), daily }.
- Web: TimeAttributionLane under the session-detail phase timeline —
  proportional stacked bar of segment totals incl. the explicit
  unattributed remainder (bar always sums to the session's authoritative
  duration), ESP-blocking apps with critical-path occupancy, reboot badge
  (cross-cutting, never a bar slice), quality-flag chips; fetched
  fail-soft for terminal sessions only. Fleet Health gains the "Time
  attribution" section: per-class stacked medians gated at n>=20
  ("insufficient data (n=...)" below), flagged/missing exclusions
  disclosed, and a top-blocking-apps table with class-scoped rows and
  "up to X (p75 Y)" what-if wording. Deliberate deviation: the spec's
  per-app trend column is omitted — per-day app rows are too sparse under
  the daily n>=5 gate; revisit once rolling history exists.
- MCP: get_time_attribution (one tool, session + fleet mode, role-aware
  routing like get_app_install_metrics, registered for all roles) — the
  privilege-leak-guard catalog snapshot extended deliberately.
- Docs: OKF concept doc backend/time-attribution.md (invariant, windows,
  segment mapping, positive-evidence blocking, Q7 reboot spans,
  persistence + surfaces) registered in index.md + log.md.

Backend 3382/3382 (+1 rolling-aggregate test), web tsc clean + vitest
627/627, MCP tsc clean + 455 passed. Post-deploy: web deploy +
MCP redeploy (new tool + docs corpus).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 18:33:16 +02:00
Oliver Kieselbach 55c898bb9c feat(insights): F1 breakdown persistence + daily attribution aggregates (PR2)
Insights expansion PR2 (tasks/insights-expansion-spec.md §F1 "Data & compute
changes"):

- SessionTimeBreakdowns table (PK=TenantId, RK=SessionId): span/reboot/
  blocking-app JSON columns, QualityFlags, AttributionVersion,
  EspAppsOccupancySeconds kept tri-state (absent column = unknown, never 0).
  Written once at the terminal single-writer seam (TableSessionRepository,
  next to the counter reconcile + EspBlocking resolution; Succeeded/Failed
  only — Incomplete deliberately stores no duration). Deleted with its
  session: new deletion-manifest step 16 (PkRkExact; tombstone moves to 19)
  plus tenant-offboarding partition wipe — no separate age sweep.
- Self-healing 30d maintenance sweep instead of a one-off backfill (rides
  the existing 2h tick + manual parity): backfills pre-deploy sessions,
  heals inline-compute misses, folds late-terminating sessions into their
  StartedAt-date aggregates, and enforces AttributionVersion purity — a
  stale-version breakdown whose events aged out counts as missing rather
  than mixing algorithm semantics (truthfulness rule 8). Skips sessions
  with a deletion cascade in flight.
- TimeAttributionAggregates table (PK=TenantId or "global",
  RK="{date}|{class}"): per enrollment class (whiteglove /
  device_preparation / self_deploying / user_driven — never mixed),
  median/p75/p90 per segment over the fixed six-segment stack (incl.
  unattributed; absent segment = honest 0), clean/flagged/missing counts
  disclosed (rule 7), rows written below the n>=20 UI gate on purpose
  (the UI needs the n — rule 4). 180d retention alongside UsageMetrics.
- What-if bound: WhatIfSavingSeconds implements the spec formula
  max(0, cpEnd - cpEndWithoutX) including the idle gap before X (the ESP
  waits for X to start, too); sole-interval fallback = own start. Per-app
  aggregate rows gate at >=5 sessions, top 20 by median interval, carry
  median/p75 of interval and saving ("up to" wording is the consumer
  contract).

Backend 3381/3381 (13 new tests: entity round-trips incl. tri-state,
aggregation bucketing/disclosure/gates, what-if vectors, class precedence,
manifest step-count pins 17/19), solution build clean, web untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 18:03:14 +02:00
Oliver Kieselbach e637317bdb feat(insights): F1 schema + time attribution calculator (PR1)
Insights expansion PR1 (tasks/insights-expansion-spec.md §F1, audit §0.5):

- AppInstallSummary + AppId / EspBlocking (bool?, tri-state) / AppIdCollision:
  sentinel-gated columns + mapper + both projections. appId adopted from app
  events (first-seen wins); a second appId under the same display name flags
  the name-keyed row as a collision (audit Q3) — in-batch and cross-batch
  (sticky in the reconcile). Collision rows leave every per-app fleet
  aggregate (app metrics payload, apps list, app analytics) with a disclosed
  exclusion count; the fleet DO rollup and per-app session drill-down keep
  them.
- EspBlocking resolution at the session-terminal single-writer seam
  (TableSessionRepository.UpdateSessionStatusAsync, next to the counter
  reconcile — covers ingest, admin marks, maintenance sweep, rule engine):
  joins the session's app rows against the LATEST list-carrying
  esp_config_detected emission as a positive-evidence set (audit Q2) —
  listed => true, absent => stays unknown, never false. EspBlockingSets
  parser handles truncation via the uncapped espTracked*Count fields.
- TimeAttributionCalculator (pure, AttributionVersion 1) + SessionTimeBreakdown
  models: exact partition of the session's authoritative DurationSeconds into
  device_prep / esp_apps / identity_hello / user_esp / desktop_handoff spans
  plus an explicit unattributed remainder — invariant sum(spans)+unattributed
  == wall clock enforced in every test. WhiteGlove = two observation windows
  in ONE session (part1 anchored on whiteglove_part1_complete, pause never
  attributed). Per-app blocking intervals from EVENT timestamps (first start
  -> LAST terminal, audit Q1 retry freeze), occupancy = merged union clipped
  to esp_apps. Reboot spans from event gaps around lastBootUtc (audit Q7),
  deduped per boot. QualityFlags: ClockSkewDropped, PartialObservation,
  BlockingSetUnknown, BlockingSetTruncated, WhiteGloveAnchorsIncomplete.
  Persistence (SessionTimeBreakdowns) and surfaces follow in PR2/PR3.

Backend 3368/3368 (35 new tests: 19 calculator golden fixtures + 16
identity/blocking/aggregate-exclusion), solution build clean, web untouched
(new response fields are additive).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 17:29:36 +02:00
Oliver Kieselbach 4eac7d042c fix(metrics): app duration/rate stats over real installs only — TerminalState column + out-of-order reconcile
Source-data audit (tasks/insights-expansion-spec.md §0.5) verified against
production: 17.9% of Succeeded AppInstallSummaries rows carry no duration —
78% are genuine skips (app_install_completed with payload state=Skipped,
"Update for X" WinGet policies not applicable) whose state the ingest
discarded, the rest are real installs whose start the agent never observed
(attach window). Averaging them as 0s understated avg install duration by
18% and distorted the slowest-apps ranking.

- AppInstallSummary.TerminalState (Installed/Skipped/Postponed/Error) from
  the closing event's payload state; sentinel-gated column + mapper +
  AppsDashboardProjection. Status stays "Succeeded" for compatibility.
- MetricsMath.IsSkipTerminalState/HasMeasuredDuration: durations computed
  over measured rows only; skips leave the failure/success rate (decision
  2026-07-26 — a skip is not an install attempt); per-app skipped/unmeasured
  counts + totals exposed; slowest ranking gates on measured installs.
  Applied consistently: app metrics payload, apps dashboard (list/analytics/
  time series/trend/versions/device models), SLA weekly/snapshot/breach.
- StoreAppInstallSummaryAsync: reconcile extracted to internal static
  ReconcileAppInstallSummaryWithExisting (test seam). Fixes Q4 (terminal
  batch before started batch left duration permanently unset) and hardens
  terminal status against late/replayed InProgress batches.
- Web: measured-only hints on app detail cards, skip suffix in apps list,
  fleet-health "measured installs" caption.

Backend 3333/3333 (20 new tests), web tsc clean, vitest 627/627.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-26 16:24:45 +02:00
131 changed files with 12258 additions and 485 deletions
+122
View File
@@ -0,0 +1,122 @@
---
type: Concept
title: Device Journeys & First-Time-Right (F2)
description: How a device's terminal enrollment attempts are grouped into journeys (device key, junk-serial exclusion, first-success end, 30-day gap), where the DeviceHistories chain and the daily FTR aggregates are computed, stored and pruned, and which surfaces serve them.
resource: src/Backend/AutopilotMonitor.Functions/Helpers/DeviceJourneyCalculator.cs
tags:
- backend
- metrics
- device-history
- first-time-right
- insights
timestamp: 2026-07-27T12:00:00+02:00
---
# Device Journeys & First-Time-Right (F2)
Answers "how many devices enrolled right on the first try?" and "show me every attempt
this device made". Session success rate hides wipe-and-retry loops — every retry that
eventually succeeds counts as a success; First-Time-Right (FTR) exposes the real pain.
All grouping semantics live server-side in `DeviceJourneyCalculator` (JourneyVersion 1);
no consumer re-derives them.
# Schema
## Device key
`(TenantId, normalized serial)` — normalization is trim + lower-case. Junk/placeholder
serials are excluded entirely (never a chain, never in FTR, disclosed as a per-day
exclusion count): `System Serial Number`, `To Be Filled By O.E.M.`, `Default string`,
`0`, `None`, `INVALID`, `Unknown` (the agent's WMI-failure sentinel), empty, or shorter
than 4 characters. A motherboard swap changes the serial and therefore the device key;
re-enrollment under a different tenant is a separate key by design.
## Journey
Terminal sessions of a device key ordered by StartedAt. The terminal set is
**Succeeded / Failed / Incomplete** — `Pending` (a sealed WhiteGlove part 1),
`AwaitingUser`, `Stalled` and `InProgress` are OPEN sessions and never attempts.
* A journey **ends** with the first terminal success; the next session starts a new
journey (redeployment).
* A gap of **more than 30 days** since the previous attempt's terminal moment
(`CompletedAt`, fallback StartedAt) starts a new journey even without a success
(device shelved/repurposed). Constant, not a setting — production gaps: median 4.9 h,
92 % < 7 d.
* WhiteGlove part 1 + part 2 share ONE session row, so a completed WG enrollment is ONE
attempt; a WG device still waiting for its user session leaves the journey open —
never a failed attempt.
* **Attempt count** = terminal sessions in the journey (`Failed` and `Incomplete` are
non-successful attempts).
* **FTR rate** = completed journeys with attempt count 1 ÷ all completed journeys.
Open and gap-abandoned journeys never enter numerator or denominator.
Durations shown for attempts are the sessions' authoritative `DurationSeconds` verbatim
(WhiteGlove pause excluded; `Incomplete` deliberately has none) — never
`CompletedAt − StartedAt`, which is later in ~25 % of terminal sessions.
## Persistence & compute
* `DeviceHistories` (PK = TenantId, RK = percent-encoded normalized serial): the chain of
the device's terminal session refs `{sessionId, startedAt, completedAt, status,
enrollmentType, isPreProvisioned, durationSeconds, adminMarked}` capped at the 20 most
recent, plus derived `CurrentJourneyAttempts` (the LAST journey's attempts — the
banner's "Attempt N"), `JourneyCount` (chain-scoped) and `JourneyVersion`. Written
inline at the session-terminal single-writer seam
(`TableSessionRepository.UpdateSessionStatusAsync`, all three terminal statuses,
DeletionState-guarded) so the session banner is fresh; healed by the maintenance sweep.
Deliberately NOT part of the per-session deletion manifest — the row aggregates many
sessions. Wiped on tenant offboarding.
* Sweep (`SweepDeviceJourneysAsync`, 30-day rolling window, 2h tick): (1) drops chain
refs of deleted sessions **tombstone-driven** — every session deletion (cascade and
retention both run through the cascade worker) leaves a `SessionTombstones` marker for
~7 days, far longer than the sweep cadence, and even an expired-but-unpruned marker is
valid deletion evidence; a chain left empty deletes its row. (2) Merges the window's
terminal sessions into their chains (backfill, inline-miss healing, re-terminal
reclassifications). (3) Recomputes the daily FTR aggregates idempotently.
* `DeviceJourneyAggregates` (PK = TenantId or `global`, RK = `{yyyy-MM-dd}`): per day —
completed journeys, first-time-right count, attempt histogram, junk-serial exclusion
count, `JourneyVersion`. A journey buckets on the StartedAt date of its completing
success session (the platform's StartedAt-date convention; late terminals converge via
the sweep). All counts are **additive**, so a window rate is the ratio of summed daily
rows — there is no rolling-window row (unlike the median-based time-attribution
aggregates, which cannot merge). 180-day retention. Rows exist below the ≥20 UI gate
on purpose (the UI needs the n for "insufficient data (n=…)").
# Surfaces
* `GET api/metrics/device-history?serialNumber=&sessionId=` (MemberRead + tenant query
param — the same route serves the web banner and MCP; a Global Admin passes `tenantId`)
— the chain plus, with `sessionId`, that session's server-computed attempt number
(live sessions get their would-be position via a virtual non-successful attempt, so the
redeploy and gap rules apply). `history: null` = no recorded history.
* `GET api/metrics/device-journeys?days=` (MemberRead) /
`GET api/global/metrics/device-journeys?days=&tenantId=` (GlobalReadOrAdmin) —
`{ windowDays, totals, daily, repeatDevices }`. The days selector is honored because
counts are additive. Without `tenantId` the global variant serves the cross-tenant
aggregate rows and `repeatDevices: null` — a per-device list would require scanning
every tenant's partition.
* Repeat devices (violator list, capped 10): devices whose LAST journey took ≥2 attempts
and whose newest terminal session falls inside the window, ordered by attempts then
recency; the failure reason comes from the newest failed attempt (bounded point-reads).
* Web: "Attempt N for this device · View history" banner on session detail (expandable
chain); "First-time-right" section on Fleet Health (rate + weekly trend + attempt
histogram + repeat devices). MCP: `get_device_history` (serial mode / fleet FTR mode).
# Examples
A device fails ESP twice and succeeds on the third try within one week: one journey,
three attempts, completed — counts once in the FTR denominator, not in the numerator,
and the histogram's bucket `3` gains one. The same device redeployed a month later and
succeeding immediately starts journey 2 with attempt count 1 — first-time-right. A
device with `serialNumber = "Unknown"` (WMI failure) never gets a chain; its terminal
sessions surface only as that day's disclosed exclusion count.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/DeviceJourneyCalculator.cs` — device key, junk list, grouping, attempt numbers (JourneyVersion 1).
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.DeviceHistory.cs` — persistence + inline seam entry.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.DeviceJourneys.cs` — sweep, tombstone pruning, daily aggregation.
* `src/Backend/AutopilotMonitor.Functions/Functions/Metrics/GetDeviceJourneyFunctions.cs` — read surfaces.
* `src/Backend/AutopilotMonitor.Functions.Tests/DeviceJourneyAndFtrTests.cs` — table-driven grouping/aggregation/attempt-number tests.
* `tasks/insights-expansion-spec.md` — F2 specification and source-data audit (§0.5).
+112
View File
@@ -0,0 +1,112 @@
---
type: Concept
title: Rule Regression Radar (F3)
description: How the radar detects analyze rules whose hit rate suddenly rose (7-day window vs 28-day baseline, Wilson-separated), which suppressions keep it quiet, how alert episodes live in the notification tracker, and which surfaces show them.
resource: src/Backend/AutopilotMonitor.Functions/Helpers/RuleRegressionRadar.cs
tags:
- backend
- rules
- regression-radar
- insights
timestamp: 2026-07-27T16:00:00+02:00
---
# Rule Regression Radar (F3)
Detects "this analyze rule suddenly fires much more often" before helpdesk volume does, and
says what the spike correlates with — with honest statistics and honest wording. Analyze
rules only in v1: their daily stats are audit-verified session-deduplicated
(`FireCount` ≈ distinct sessions with a hit, `SessionsEvaluated` the same-source
denominator). Gather rules are excluded until their per-batch stats dedup is fixed —
never silently included with bad math.
# Schema
## Signal & detection
Per (tenant, ruleId), daily hit rate = `FireCount ÷ SessionsEvaluated` from the
`RuleStats` rows — RATES, not counts (absolute counts spike with every rollout wave).
A regression fires only when ALL hold for the trailing 7-day window vs. the prior
28-day baseline:
1. window has ≥ 5 hit sessions AND ≥ 20 evaluated sessions;
2. window rate ≥ 2× baseline rate;
3. the Wilson 95 % intervals are disjoint in the increase direction
(`MetricsMath.WilsonInterval` / `RateIncreaseSeparated` — deterministic, unit-pinned),
so small-n noise never alerts.
## Suppression (false positives are trust damage)
* Rule entity missing (deleted) — never alerts.
* Rule younger than baseline + window (`CreatedAt` within 35 days — grace period for new
rules; the entity timestamp is the honest gate, a stats-row count would fail on
low-volume tenants whose quiet days simply have no rows).
* Rule edited inside the window (`UpdatedAt` ≥ window start) — edits legitimately change
hit rates.
* Empty baseline denominator (rule disabled throughout the baseline).
* Platform kill switch: app setting `RuleRegressionRadarDisabled=true` skips the radar
(fail-open — it only notifies, never mutates data).
## Alert episodes (tracker keyspace)
An episode lives as `ruleregression|{ruleId}` in the notification-tracker table
(the `tpmpss|` pattern): the row IS the dedup (bell + ops event fire exactly once per
episode), the badge state and the `regressions[]` payload — window/baseline counts,
rates, lift (null for a zero-baseline "new signal" — no finite lift is invented), window
dates, and the dimension concentration captured at first fire. Numbers refresh on every
radar pass while the episode stays active; `FirstNotifiedAt` never moves. The episode
closes (badge clears, dedup re-arms) when the window rate falls back under 1.5× baseline
or fires stop entirely; the tracker's 30-day retention sweep also re-arms, so a
month-old still-burning regression rings once more by design.
## Dimension correlation (on fire only)
Hit sessions (RuleResults scan: tenant partition range + RowKey = ruleId + DetectedAt in
window, intersected with the window's sessions) are compared against all window sessions
across `osBuild`, `model` (manufacturer + model), `agentVersion` and `imeVersion`. The
top value by lift is reported with BOTH shares — wording contract: "X % of affected
sessions are on … vs Y % of all sessions — correlated, not necessarily causal". No value
with ≥ 5 hit sessions at ≥ 2× lift → "no clear dimension concentration"; a correlation
failure also yields no claim. App-version correlation for `ANALYZE-APP-*` rules is a
follow-up (needs rule→app matching semantics), tracked with the gather-rule fix.
## Compute & alert path
Runs in the daily maintenance pass right after the rule-stats aggregation
(`RunRuleRegressionRadarAsync`), anchored on YESTERDAY — whole days only, a partial
today would understate the window rate. One 35-day stats read covers all tenants;
evaluation is the pure `RuleRegressionRadar` core. On a NEW episode:
1. `RuleFrequencyRegression` ops event (category Tenant, Warning) with the full numbers
— dual-registered in the web `OPS_EVENT_TYPES` catalog for alert-rule routing.
2. Tenant bell notification (type `rule_frequency_regression`, Admin audience) with the
same numbers and a deep link to the rule's card (`/analyze-rules#rule-card-{ruleId}`).
A sessions-list pre-filtered to rule hits needs a rule filter the session search does
not have yet — follow-up alongside the gather-rule work.
3. v1 channel: bell + ops event only. Provider fan-out (webhook/Telegram) follows once
alert quality is proven — a noisy first version burns trust permanently.
# Surfaces
* `GET api/metrics/rule-stats` (and the global variant with `tenantId`) carries
`regressions[]` — the active episodes. MCP `get_rule_stats` inherits the block.
* Analyze-rules page: per-rule 30-day fire-count sparkline (densified from the `trend`
rows the endpoint already served) and a red "↑ Regression" badge while an episode is
active, with the full numbers in the tooltip.
# Examples
A rule firing on 2 % of sessions for a month jumps to 15 % over a week (15/100 vs
10/500): lift 7.5×, Wilson-separated → one bell, one ops event, badge until the rate
falls back under 3 %. The same jump on 5/20 sessions with a 12 % baseline stays silent —
the intervals overlap, so the spike is not statistically real yet. A rule that never
fired in the baseline and now hits 15 % alerts as a "new signal" with no lift figure.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/RuleRegressionRadar.cs` — gates, suppression, re-arm, dimension concentration.
* `src/Backend/AutopilotMonitor.Functions/Helpers/MetricsMath.cs` — Wilson interval + separation primitive.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.RuleRegression.cs` — sweep, episode reconcile, bell/ops wiring.
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableHardwareRejectionNotificationTracker.cs` — episode keyspace.
* `src/Backend/AutopilotMonitor.Functions.Tests/RuleRegressionRadarTests.cs` — hand-computed Wilson vectors + every suppression branch.
* `tasks/insights-expansion-spec.md` — F3 specification and source-data audit (§0.5).
+124
View File
@@ -0,0 +1,124 @@
---
type: Concept
title: Enrollment Time Attribution (F1)
description: How a terminal session's wall clock is partitioned into named segments with an explicit unattributed remainder, how ESP-blocking apps and reboots are attributed, and where breakdowns and fleet aggregates are computed, stored and served.
resource: src/Backend/AutopilotMonitor.Functions/Helpers/TimeAttributionCalculator.cs
tags:
- backend
- metrics
- time-attribution
- insights
timestamp: 2026-07-26T20:00:00+02:00
---
# Enrollment Time Attribution (F1)
Answers "where did the 47 minutes go?" per session and per fleet. Every number is
traceable to observed events; a missing signal produces "unknown" or an explicit
unattributed remainder — never a guess, and the partition is never normalized to 100 %.
# Schema
## Invariant
For every computed breakdown:
```
sum(span seconds) + UnattributedSeconds == WallClockSeconds == session DurationSeconds
```
`DurationSeconds` is the authoritative wall clock — for WhiteGlove it is part 1 + part 2
WITHOUT the pause, and re-terminal stamps move `CompletedAt`, so `CompletedAt − StartedAt`
is never used (it diverges in ~25 % of terminal sessions). Span seconds are floored;
sub-second dust lands in the unattributed remainder.
## Observation windows
Windows are end-anchored on authoritative timestamps and constructed so their lengths sum
exactly to `DurationSeconds`:
* Non-WhiteGlove: one window `[CompletedAt − DurationSeconds, CompletedAt]`.
* WhiteGlove (ONE session row, two windows): part 2 = `[ResumedAt, CompletedAt]`; part 1 =
`[part1End − (DurationSeconds − part2), part1End]` where `part1End` is anchored on the
`whiteglove_part1_complete` event. The pause between the windows is never attributed.
Fallback anchors set the `WhiteGloveAnchorsIncomplete` quality flag.
## Segments
Spans are sliced at phase-declaration events (only declaration events carry
`Phase != Unknown`) plus `desktop_arrived`, in canonical Sequence order; anchors whose
timestamp runs backward are dropped (`ClockSkewDropped`).
| EnrollmentPhase | Segment key |
|---|---|
| Start / DevicePreparation / DeviceSetup | `device_prep` |
| AppsDevice | `esp_apps` |
| AccountSetup / FinalizingSetup | `identity_hello` |
| AppsUser | `user_esp` |
| Complete, `desktop_arrived` event | `desktop_handoff` |
| Failed | attribution ends — tail stays unattributed |
## ESP-blocking apps (positive evidence only)
Per-app intervals use EVENT timestamps (first started/download event → LAST terminal
event, covering IME retries) — never the agent payload timing, which freezes at the first
terminal transition. Blocking membership joins the app id against the session's latest
list-carrying `esp_config_detected` emission: listed ⇒ blocking; absent ⇒ unknown, never
false (`BlockingSetUnknown` / `BlockingSetTruncated` flags). Critical-path occupancy =
overlap-merged union of blocking intervals clipped to the `esp_apps` spans;
`EspAppsOccupancySeconds` is null (unknown) when no lists were observed.
The fleet what-if bound per app X is `max(0, cpEnd − cpEndWithoutX)` (latest interval end
with vs. without X; the idle gap before X counts — the ESP also waits for X to start).
It is an upper bound by construction; every consumer says "up to", never "you will save".
## Reboots
`system_reboot_detected` is detection-time-stamped by the NEXT agent run; the real boot
moment is its `lastBootUtc` payload. Reboot spans are therefore the event-stream gap
bracketing `lastBootUtc`, deduped per boot, clipped to the observation windows, and carried
as a cross-cutting annotation (they overlap the segment they started in — never a slice of
the partition).
## Persistence & compute
* `SessionTimeBreakdowns` (PK = TenantId, RK = SessionId): spans, reboot spans, blocking
intervals (top 20 + uncapped count), quality flags, `AttributionVersion`. Written once at
the session-terminal single-writer seam (`TableSessionRepository.UpdateSessionStatusAsync`,
Succeeded/Failed only — Incomplete deliberately has no duration). Deleted with its session
(deletion-manifest step) and on tenant offboarding.
* Self-healing 30-day maintenance sweep (2h tick): backfills missing breakdowns, folds
late-terminating sessions into their StartedAt-date aggregates, and enforces version
purity — a stale-version row whose events aged out counts as missing, never mixed.
* `TimeAttributionAggregates` (PK = TenantId or `global`, RK = `{yyyy-MM-dd}|{class}` and
`rolling30|{class}`): per enrollment class (`user_driven`, `whiteglove`, `self_deploying`,
`device_preparation` — never mixed) median/p75/p90 per segment over the fixed six-segment
stack (incl. `unattributed`; a session without a segment contributes 0), plus top blocking
apps (≥5 sessions per app row). Clean/flagged/missing counts are part of every row; rows
exist below the ≥20 UI gate on purpose (the UI needs the n for "insufficient data (n=…)").
Daily rows retain 180 days; `rolling30` rows are refreshed whole (a median of per-day
medians is not the range median — the rolling rows are the honest range statistics).
## Surfaces
* `GET api/sessions/{sessionId}/time-attribution` (MemberRead + tenant query param) —
breakdown row or `breakdown: null` (pre-feature / non-terminal / Incomplete = no lane).
* `GET api/metrics/time-attribution` (MemberRead) / `GET api/global/metrics/time-attribution`
(GlobalReadOrAdmin, optional `tenantId`) — `{ windowDays, classes (rolling), daily }`.
* Web: attribution lane under the session-detail phase timeline; "Time attribution" section
on Fleet Health. MCP: `get_time_attribution` (sessionId mode / fleet mode).
# Examples
A 30-minute user-driven session: `device_prep` 5m → `esp_apps` 10m (occupancy 4m by one
blocking app) → `identity_hello` 8m → `user_esp` 5m → `desktop_handoff` 2m, unattributed 0.
A WhiteGlove session with a 2-day pause reports wall clock = part1 + part2 only; the resume
gap before the first part-2 phase declaration is unattributed, not guessed.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Helpers/TimeAttributionCalculator.cs` — calculator + semantics (AttributionVersion 1).
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.TimeAttribution.cs` — persistence.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.Aggregation.cs` — sweep + aggregation.
* `src/Backend/AutopilotMonitor.Functions.Tests/TimeAttributionCalculatorTests.cs` — golden fixtures pinning the invariant.
* `tasks/insights-expansion-spec.md` — F1 specification and source-data audit (§0.5).
+7
View File
@@ -32,12 +32,19 @@ https://docs.autopilotmonitor.com (separate repository).
# Backend
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
* [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract.
* [ServerActions & On-Demand Diagnostics](backend/server-actions-on-demand-diagnostics.md) - The queue-on-session, deliver-on-ingest action channel, the TenantAdminOrOperator tier with in-function per-type re-gating, the portal's Collect Logs flow (quick-config orders rotate_config before request_diagnostics), and the hosted-prefix sweep in the offboarding cascade.
# MCP Server
* [MCP Docs Corpus](mcp/docs-corpus.md) - How the published customer documentation becomes the `search_docs` corpus: CRLF-safe heading chunking, build-time embedding as a third precomputed section, and why a docs edit needs an MCP redeploy.
# Web Portal
* [Portal Navigation Prefetch & the SWA Runtime Bottleneck](web/portal-navigation-prefetch.md) - Why every navigation Link sets `prefetch={false}`: the App Router's viewport prefetch fans out one RSC request per visible link against an HTTP/1.1 Static Web Apps runtime, and its latency tail starves the six connections that also serve the document and the route chunks. Includes where browser telemetry actually lives.
# Security & Identity
* [MCP OAuth Flow](mcp-oauth-flow.md) - Who authenticates where when connecting an AI client to the MCP server; two identities, three parties plus Entra ID.
+12
View File
@@ -1,5 +1,13 @@
# Log
## 2026-07-27
* **Creation**: Added `web/portal-navigation-prefetch.md` (and a `Web Portal` section to the index) — root cause of the minute-long portal freezes. `GlobalSidebar` and `Navbar` render the whole navigation on every page, so the App Router's default viewport prefetch issued one `?_rsc=` request per visible link against `portal.autopilotmonitor.com` (~10,400 in 14 days). That origin serves the document, the route chunks and the RSC payloads over **HTTP/1.1**, so the six-connection-per-origin cap applies; when the Static Web Apps managed runtime queued the prefetches (p50 91 ms, p95 896 ms, but max 234 s / 21 samples over 30 s), the in-flight prefetches held every socket and the navigation the user wanted could not get one. Measured on the 2026-07-27 17:25 CEST incident: `/admin/presence?_rsc=` 151 s, three siblings finishing within 340 ms of each other at ~37.5 s, all `200`, while the backend API answered in 1.6–2.2 s and `browserTimings` showed `networkDuration` 16 ms against `processingDuration` 225 s. Every navigation Link now sets `prefetch={false}` — these routes are client components behind `ProtectedRoute` that fetch their own data, so the prefetched RSC payload carried no benefit. The doc also records that browser telemetry lives in a **separate** App Insights resource (`AUTOPILOT_MONITOR_WEB_APPINSIGHTS_ID`) that the MCP `query_backend_logs` tool cannot reach, which is why the freezes left no trace in the backend logs.
* **Creation**: Added `backend/rule-regression-radar.md` — F3 regression radar (analyze rules only; gather rules wait for their per-batch stats dedup fix): per (tenant, rule) daily hit rate from `RuleStats`, firing only when the trailing 7-day window has ≥5 hit sessions and ≥20 evaluated sessions, the rate is ≥2× the prior 28-day baseline AND the Wilson 95 % intervals are disjoint (deterministic `MetricsMath` primitive with hand-computed test vectors). Suppressed for deleted rules, rules younger than baseline+window (`CreatedAt` grace), rules edited inside the window (`UpdatedAt`), empty baselines, and the `RuleRegressionRadarDisabled` app-setting kill switch. Alert episodes live as `ruleregression|{ruleId}` tracker rows (one bell + one `RuleFrequencyRegression` ops event per episode; numbers refreshed while active; re-armed under 1.5× baseline, on fire-stop, or by the 30-day tracker retention). On-fire dimension correlation over osBuild/model/agentVersion/imeVersion with the "correlated, not necessarily causal" wording contract. Surfaces: `regressions[]` in the rule-stats response (MCP inherits), the analyze-rules page's 30-day sparkline and regression badge.
* **Creation**: Added `backend/device-journeys.md` — F2 device history / First-Time-Right: the device key (trim + case-fold serial, junk/placeholder exclusion incl. the agent's `Unknown` WMI sentinel), journey grouping over the terminal set Succeeded/Failed/Incomplete (first success ends a journey, a >30-day gap from the previous attempt's CompletedAt starts one, WhiteGlove's single session row is ONE attempt, open sessions are never attempts), the `DeviceHistories` chain (cap 20, written inline at the terminal seam, healed by the sweep, deleted-session refs pruned tombstone-driven, deliberately not in the per-session deletion manifest) and the additive `DeviceJourneyAggregates` daily rows (window rate = ratio of sums — no rolling row needed, 180-day retention). Surfaces: the session-detail "Attempt N" banner with server-computed attempt numbers (virtual-attempt rule for live sessions), the Fleet Health First-time-right section (≥20-journey gate, attempt histogram, repeat-devices violator list), and the MCP `get_device_history` tool (serial mode / fleet FTR mode).
## 2026-07-25
* **Update**: `agent/secureboot-ca2023-detection.md` — review follow-up on 90012d9 hardened the privilege scope, the repo's only privilege-adjustment code and therefore the precedent for any future one. The process token moved from a raw `IntPtr` to a `SafeHandle` assigned by the P/Invoke marshaler, so the CLR owns it from the moment `OpenProcessToken` returns and the critical finalizer releases it even when `Dispose` never runs — previously an async exception between the successful open and the caller's `using` leaked the handle *with the privilege enabled and no owner left to restore it*. `Dispose` became idempotent and now releases the handle in a `finally`, so a throwing restore cannot strand it; the restore obligation is recorded immediately after the successful `AdjustTokenPrivileges` with nothing throwable in between. The whole enable → read → restore sequence runs under a static gate because the privilege is adjusted on the *process* token: a concurrent reader would see "already enabled", correctly skip its own restore, and then lose the privilege mid-read when the first reader disposed, reporting `privilege_denied` on a machine that holds it.
@@ -52,6 +60,10 @@
* **Creation**: Added `agent/endpoint-migration.md` — config-channel agent re-home (`MigrateToApiBaseUrl` on `GET agent/config`, ConfigVersion 33): backend or per-tenant API-URL moves without DNS indirection (impossible due to mTLS/Flex-TLS). Live-fetch-only with cache strip, `AgentEndpointMigrationRules` allowlist validated on both sides, one hop, kill-switch wins. Built as the lesson from the 2026-07-18 WEU→GWC cutover, which hard-cut all in-flight agents.
## 2026-07-26
* **Creation**: Added `backend/time-attribution.md` — the F1 enrollment time attribution: exact-partition invariant against the session's authoritative DurationSeconds, two-window WhiteGlove model, segment mapping from phase-declaration events, positive-evidence ESP-blocking join with "up to" what-if bounds, reboot spans from event gaps around `lastBootUtc`, and the SessionTimeBreakdowns / TimeAttributionAggregates tables with their terminal-seam + maintenance-sweep compute paths and read surfaces (web lane, fleet panel, MCP `get_time_attribution`).
## 2026-07-17
* **Creation**: Added `agent/build-counter-blob.md` — agent build numbers now come from a shared public counter blob (`buildcounter-v{N}.txt`, reserve-before-build with ETag-CAS) so local `build.ps1` and CI `build-agent.yml` builds can never mint the same number. Documents the CI parity fixes (SummaryDialog.exe.config in the ZIP, EXE-vs-manifest version check, hash-oracle update gated on stable cutover) and the Sigstore provenance attestation on CI-built ZIPs.
+88
View File
@@ -0,0 +1,88 @@
---
type: Concept
title: Portal Navigation Prefetch & the SWA Runtime Bottleneck
description: Why every navigation Link in the portal sets prefetch={false} — the App Router's viewport prefetch fires one RSC request per visible link against an HTTP/1.1 Static Web Apps runtime, and its latency tail starves the connection pool that serves the document and the route chunks.
resource: src/Web/autopilot-monitor-web/components/GlobalSidebar.tsx
tags:
- web
- performance
- nextjs
- static-web-apps
- observability
timestamp: 2026-07-27T18:00:00+02:00
---
# Context
The portal is a Next.js 15 App Router application hosted on Azure Static Web Apps. Every page under
the authenticated shell is a client component behind `ProtectedRoute` that fetches its own data from
the Function App (`autopilotmonitor-api-eu`) after mount. The RSC payload a route returns on
navigation therefore carries no page data — only the shell.
`GlobalSidebar` and `Navbar` render the full navigation on every page. With the App Router's default
prefetch behaviour, each visible `<Link>` triggers a background `GET /<route>?_rsc=<hash>` as soon as
it enters the viewport.
# Mechanism
Three properties combine into a portal-wide freeze:
1. **Fan-out.** The navigation is fully rendered on each page load, so a single navigation issues one
RSC request per visible link — measured at ~10,400 requests over 14 days.
2. **Shared origin over HTTP/1.1.** `portal.autopilotmonitor.com` serves the document, the JS route
chunks *and* the RSC payloads, and it negotiates HTTP/1.1 — not HTTP/2. Browsers cap concurrent
connections per origin at six.
3. **A heavy latency tail on the SSR runtime.** RSC responses are prerendered and edge-cached
(`x-nextjs-cache: HIT`, `s-maxage=31536000`), so the median is fast. On a cache miss or cold start
the Static Web Apps managed runtime queues them.
When the tail hits, the in-flight prefetches hold all six connections. The document request and the
route chunks of the page the user actually wants then have no socket available, and the tab shows a
loading spinner or a blank page until the queue drains. Nothing fails — the requests eventually
return `200`, which is why the failure leaves no error trace anywhere.
# Evidence
Incident of 2026-07-27, 17:25–17:29 CEST. Browser-side dependencies during the freeze:
| Request | Duration | Status |
| --- | --- | --- |
| `/admin/presence?_rsc=…` | 151,461 ms | 200 |
| `/gather-rules?_rsc=…` | 52,283 ms | 200 |
| `/progress?_rsc=…` | 37,878 ms | 200 |
| `/analyze-rules?_rsc=…` | 37,559 ms | 200 |
| `/audit?_rsc=…` | 37,539 ms | 200 |
| Backend API, same window | 1,600–2,200 ms | 200 |
The three ~37.5 s responses complete within 340 ms of each other — a queue draining, not independent
slow requests. The corresponding `browserTimings` row for `/geographic-performance` reports
`networkDuration` = 16 ms against `processingDuration` = 225,095 ms: the network delivered the
document immediately and the browser then waited nearly four minutes for the `load` event.
Fourteen-day distribution of RSC prefetches: n = 10,428, p50 = 91 ms, p95 = 896 ms, max = 234,230 ms,
38 over 10 s, 21 over 30 s.
# Diagnosing this class of problem
The backend Application Insights resource is the wrong place to look — a freeze appears there as an
*absence* of requests, never as an error. Browser telemetry lives in a **separate** Application
Insights resource addressed by `AUTOPILOT_MONITOR_WEB_APPINSIGHTS_ID`, which the MCP
`query_backend_logs` tool does not reach. Query it through
`.claude/commands/scripts/query-appinsights.sh` with that ID exported, and read `dependencies`
(client-side fetches, including RSC prefetches) and `browserTimings` (the network/processing split of
a document load).
# Contract
Navigation `<Link>` elements in `GlobalSidebar` and `Navbar` set `prefetch={false}`. The prefetched
RSC payload buys nothing for these routes, so the only remaining cost of a click is the route's JS
chunk, which the browser caches after first use.
Do not remove the prop to "speed up navigation" without first re-measuring the tail in the web
Application Insights resource — the prefetch is cheap only while the runtime is warm.
# Citations
* `src/Web/autopilot-monitor-web/components/GlobalSidebar.tsx` — nav link rendering, rationale comment
* `src/Web/autopilot-monitor-web/components/Navbar.tsx` — top-bar and notification links
* [SWA and Oryx node pinning](../architecture.md) — hosting model for the web app
@@ -0,0 +1,157 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Persistence;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using Xunit;
using SharedEventTypes = AutopilotMonitor.Shared.Constants.EventTypes;
using DeviceInfoCollector = AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo.DeviceInfoCollector;
namespace AutopilotMonitor.Agent.V2.Core.Tests.DeviceInfo
{
/// <summary>
/// Audit Q2 — <see cref="DeviceInfoCollector.RefreshEspConfiguration"/>
/// re-reads the ESP blocking lists and re-emits <c>esp_config_detected</c> exactly when the
/// payload changed: the registry lists grow progressively (one timestamped subkey per CSP
/// status write) and the user-scope lists appear only after sign-in, so the early emissions
/// are structurally partial. The StartupEventGate keeps an unchanged re-read silent.
/// Probe overrides are static slots → serialized with the other probe-driven suites.
/// </summary>
[Collection("SerialThreading")]
public sealed class DeviceInfoCollectorEspRefreshTests
{
private static readonly DateTime T0 = new DateTime(2026, 7, 27, 10, 0, 0, DateTimeKind.Utc);
private const string DeviceApp1 = "11111111-1111-1111-1111-111111111111";
private const string DeviceApp2 = "22222222-2222-2222-2222-222222222222";
private const string UserApp1 = "33333333-3333-3333-3333-333333333333";
private static EspTrackingInfoSnapshot Snapshot(string[] deviceApps, string[] userApps)
{
var all = deviceApps.Concat(userApps).Distinct().ToArray();
return new EspTrackingInfoSnapshot(
msiProductCodes: Array.Empty<string>(),
modernAppPfns: Array.Empty<string>(),
win32AppIds: all,
userWin32AppIds: userApps,
msiCount: 0,
modernCount: 0,
win32Count: all.Length);
}
private static IReadOnlyList<FakeSignalIngressSink.PostedSignal> EspConfigEvents(FakeSignalIngressSink ingress) =>
ingress.Posted
.Where(p => p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == SharedEventTypes.EspConfigDetected)
.ToList();
private static IReadOnlyList<string> TypedList(FakeSignalIngressSink.PostedSignal signal, string key)
{
var data = Assert.IsAssignableFrom<IReadOnlyDictionary<string, object>>(signal.TypedPayload);
return Assert.IsAssignableFrom<IReadOnlyList<string>>(data[key]);
}
[Fact]
public void Refresh_reemits_when_lists_grew_and_stays_silent_when_unchanged()
{
using var tmp = new TempDirectory();
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
var ingress = new FakeSignalIngressSink();
var clock = new VirtualClock(T0);
var collector = new DeviceInfoCollector(
"session-1", "tenant-1",
new InformationalEventPost(ingress, clock), logger,
signalIngress: null, clock: clock,
startupGate: new StartupEventGate(tmp.Path, logger));
using var skip = new EspSkipConfigurationProbe.ScopedFullOverride(
_ => new EspFirstSyncSnapshot(
skipUser: false, skipDevice: false,
blockInStatusPage: null, syncFailureTimeoutMinutes: null));
// Early read (agent start / DeviceSetup detection): partial device list, empty user list.
using (new EspTrackingInfoProbe.ScopedOverride(_ => Snapshot(new[] { DeviceApp1 }, Array.Empty<string>())))
{
collector.CollectEspConfiguration();
}
var afterEarly = EspConfigEvents(ingress);
Assert.Single(afterEarly);
Assert.Empty(TypedList(afterEarly[0], "espTrackedUserWin32AppIds"));
// Apps sub-phase opened: the device list grew and sign-in populated the user list.
using (new EspTrackingInfoProbe.ScopedOverride(_ => Snapshot(new[] { DeviceApp1, DeviceApp2 }, new[] { UserApp1 })))
{
collector.RefreshEspConfiguration(DeviceInfoHost.EspConfigTriggerAppsUser);
}
var afterGrowth = EspConfigEvents(ingress);
Assert.Equal(2, afterGrowth.Count);
Assert.Contains(DeviceApp2, TypedList(afterGrowth[1], "espTrackedWin32AppIds"));
Assert.Equal(new[] { UserApp1 }, TypedList(afterGrowth[1], "espTrackedUserWin32AppIds"));
// A later trigger with an UNCHANGED registry state re-reads but emits nothing —
// the StartupEventGate suppresses the identical payload.
using (new EspTrackingInfoProbe.ScopedOverride(_ => Snapshot(new[] { DeviceApp1, DeviceApp2 }, new[] { UserApp1 })))
{
collector.RefreshEspConfiguration(DeviceInfoHost.EspConfigTriggerAccountSetup);
}
Assert.Equal(2, EspConfigEvents(ingress).Count);
}
[Fact]
public async Task Concurrent_refreshes_never_emit_an_older_snapshot_after_a_newer_one()
{
// Codex review: the refresh triggers each run on their own Task.Run — without the
// collector-side lock, two collects can interleave read-before-emit and the LATER
// emission carries the OLDER (smaller) list. The backend keeps the latest
// list-bearing emission as positive evidence, so that would silently shrink it.
// The probe returns a strictly growing list per read; with the read→emit critical
// section, emission order must equal read order — sizes strictly ascending.
using var tmp = new TempDirectory();
var logger = new AgentLogger(tmp.Path, AgentLogLevel.Info);
var ingress = new FakeSignalIngressSink();
var clock = new VirtualClock(T0);
var collector = new DeviceInfoCollector(
"session-1", "tenant-1",
new InformationalEventPost(ingress, clock), logger,
signalIngress: null, clock: clock,
startupGate: new StartupEventGate(tmp.Path, logger));
using var skip = new EspSkipConfigurationProbe.ScopedFullOverride(
_ => new EspFirstSyncSnapshot(
skipUser: false, skipDevice: false,
blockInStatusPage: null, syncFailureTimeoutMinutes: null));
var readCounter = 0;
using var tracking = new EspTrackingInfoProbe.ScopedOverride(_ =>
{
var n = Interlocked.Increment(ref readCounter);
var ids = Enumerable.Range(1, n)
.Select(i => $"{i:d8}-1111-2222-3333-444444444444")
.ToArray();
return Snapshot(ids, Array.Empty<string>());
});
var tasks = Enumerable.Range(0, 12)
.Select(i => Task.Run(() => collector.RefreshEspConfiguration($"stress-{i}")))
.ToArray();
await Task.WhenAll(tasks);
var sizes = EspConfigEvents(ingress)
.Select(e => TypedList(e, "espTrackedWin32AppIds").Count)
.ToList();
// Every refresh saw a strictly larger list (gate never suppresses) and each
// read→emit is atomic, so the emission sequence is exactly 1..12.
Assert.Equal(Enumerable.Range(1, 12), sizes);
}
}
}
@@ -37,5 +37,41 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
Assert.False(DeviceInfoHost.IsEndTrigger(DecisionSignalKind.EspPhaseChanged, null));
Assert.False(DeviceInfoHost.IsEndTrigger(DecisionSignalKind.InformationalEvent, Phase("FinalizingSetup")));
}
// Audit Q2 — the payload shape the adapter's apps sub-phase declaration produces on the
// ingress rail (InformationalEventPost: eventType + declared phase enum name).
private static IReadOnlyDictionary<string, string> SubPhaseDeclaration(string declaredPhase) =>
new Dictionary<string, string>
{
{ SignalPayloadKeys.EventType, AutopilotMonitor.Shared.Constants.EventTypes.PhaseTransition },
{ AutopilotMonitor.Agent.V2.Core.Telemetry.Events.EventTimelineEmitter.PhaseParamKey, declaredPhase },
};
[Fact]
public void Apps_subphase_declarations_classify_as_esp_config_refresh_triggers()
{
Assert.Equal(DeviceInfoHost.EspConfigTriggerAppsDevice,
DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent, SubPhaseDeclaration("AppsDevice")));
Assert.Equal(DeviceInfoHost.EspConfigTriggerAppsUser,
DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent, SubPhaseDeclaration("AppsUser")));
// Non-apps phase declarations (e.g. the reducer's Finalizing entry) never classify.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent, SubPhaseDeclaration("FinalizingSetup")));
// A phase_transition without a declared phase never classifies.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent,
new Dictionary<string, string> { { SignalPayloadKeys.EventType, AutopilotMonitor.Shared.Constants.EventTypes.PhaseTransition } }));
}
[Fact]
public void AccountSetup_phase_classifies_as_the_user_setup_refresh_trigger()
{
Assert.Equal(DeviceInfoHost.EspConfigTriggerAccountSetup,
DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.EspPhaseChanged, Phase("AccountSetup")));
// DeviceSetup is the enrollment-start trigger, not an ESP-config refresh.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.EspPhaseChanged, Phase("DeviceSetup")));
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.EspPhaseChanged, null));
// Unrelated informational events never classify.
Assert.Null(DeviceInfoHost.ClassifyEspConfigRefreshTrigger(DecisionSignalKind.InformationalEvent,
new Dictionary<string, string> { { SignalPayloadKeys.EventType, "esp_config_detected" } }));
}
}
}
@@ -0,0 +1,33 @@
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
{
/// <summary>
/// Audit Q5 — WMI hardware values are trimmed at the source. BIOS serials become device
/// identity keys downstream (F2 device history joins on TenantId + serial), so padding
/// must never leave the agent; whitespace-only values collapse to the "Unknown" sentinel
/// the retry logic already treats as unresolved.
/// </summary>
public sealed class HardwareInfoNormalizeTests
{
[Theory]
[InlineData(" PF60H941 ", "PF60H941")]
[InlineData("PF60H941", "PF60H941")]
[InlineData("\tLENOVO \r\n", "LENOVO")]
[InlineData("Unknown", "Unknown")]
public void Trims_whitespace_at_the_source(string raw, string expected)
=> Assert.Equal(expected, HardwareInfo.NormalizeWmiValue(raw));
[Theory]
[InlineData("")]
[InlineData(" ")]
[InlineData("\t\r\n")]
public void Empty_and_whitespace_only_collapse_to_the_Unknown_sentinel(string raw)
=> Assert.Equal("Unknown", HardwareInfo.NormalizeWmiValue(raw));
[Fact]
public void Null_collapses_to_the_Unknown_sentinel()
=> Assert.Equal("Unknown", HardwareInfo.NormalizeWmiValue(null));
}
}
@@ -243,5 +243,87 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
Assert.Equal(firstStamp, adapter.AppTimings["app-P"].StartedAtUtc);
}
[Fact]
public void Terminal_to_active_retry_rearms_CompletedAt_and_final_terminal_covers_total_occupancy()
{
// Audit Q1: the IME retry path (Error -> Downloading -> Installed is permitted by
// AppPackageState.UpdateState). Pre-fix, CompletedAt froze at the first Error, so
// the final Installed payload carried the first failure's completedAt/duration.
using var f = new ImeLogTrackerAdapterFixture(T0);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-R", 0);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Installing);
f.Clock.Advance(TimeSpan.FromSeconds(5));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Installing, AppInstallationState.Error);
f.Clock.Advance(TimeSpan.FromSeconds(10));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Error, AppInstallationState.Downloading);
// Re-armed while active again: the retry's download event carries no completedAt,
// and the FIRST start stamp is kept.
Assert.Null(adapter.AppTimings["app-R"].CompletedAtUtc);
var retryDownload = f.InfoEvent(SharedEventTypes.AppDownloadStarted);
Assert.False(retryDownload.Payload!.ContainsKey("completedAt"));
Assert.False(retryDownload.Payload.ContainsKey("durationSeconds"));
Assert.Equal(T0.ToString("o"), retryDownload.Payload["startedAt"]);
f.Clock.Advance(TimeSpan.FromSeconds(20));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Downloading, AppInstallationState.Installed);
var timing = adapter.AppTimings["app-R"];
Assert.Equal(T0, timing.StartedAtUtc);
Assert.Equal(T0.AddSeconds(35), timing.CompletedAtUtc);
Assert.Equal(35.0, timing.DurationSeconds);
var completed = Assert.Single(f.InfoEvents(SharedEventTypes.AppInstallComplete));
Assert.Equal("35.00", completed.Payload!["durationSeconds"]);
Assert.Equal(T0.AddSeconds(35).ToString("o"), completed.Payload["completedAt"]);
}
[Fact]
public void Skipped_then_retried_app_emits_positive_duration_from_retry_start()
{
// Production regression (session 66f1c5e1, audit Q1): a user-scope app evaluated
// Skipped during DeviceSetup (terminal, no start observed), retried after sign-in
// (Skipped -> Downloading is permitted), Installed later. Pre-fix payload:
// completedAt frozen at the early Skip + startedAt from the retry ->
// durationSeconds=-2342. Post-fix: retry re-arms, duration covers retry-start ->
// final terminal.
using var f = new ImeLogTrackerAdapterFixture(T0);
using var adapter = new ImeLogTrackerAdapter(f.Tracker, f.Ingress, f.Clock);
var app = new AppPackageState("app-S", 0);
adapter.TriggerAppStateFromTest(app, AppInstallationState.Unknown, AppInstallationState.Skipped);
Assert.Null(adapter.AppTimings["app-S"].StartedAtUtc);
Assert.Equal(T0, adapter.AppTimings["app-S"].CompletedAtUtc);
f.Clock.Advance(TimeSpan.FromSeconds(60));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Skipped, AppInstallationState.Downloading);
f.Clock.Advance(TimeSpan.FromSeconds(30));
adapter.TriggerAppStateFromTest(app, AppInstallationState.Downloading, AppInstallationState.Installed);
var timing = adapter.AppTimings["app-S"];
Assert.Equal(T0.AddSeconds(60), timing.StartedAtUtc);
Assert.Equal(T0.AddSeconds(90), timing.CompletedAtUtc);
Assert.Equal(30.0, timing.DurationSeconds);
var completedEvents = f.InfoEvents(SharedEventTypes.AppInstallComplete);
Assert.Equal(2, completedEvents.Count);
Assert.Equal("30.00", completedEvents[1].Payload!["durationSeconds"]);
}
[Fact]
public void Inverted_endpoints_yield_null_DurationSeconds()
{
// Negative guard (audit Q1): endpoints stamped from out-of-order source lines must
// never surface as a negative durationSeconds (observed -2342s in production). The
// payload emitter omits the field when DurationSeconds is null.
var inverted = new AppInstallTiming(startedAtUtc: T0.AddSeconds(100), completedAtUtc: T0);
Assert.Null(inverted.DurationSeconds);
var wellOrdered = new AppInstallTiming(startedAtUtc: T0, completedAtUtc: T0.AddSeconds(100));
Assert.Equal(100.0, wellOrdered.DurationSeconds);
}
}
}
@@ -10,9 +10,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// the <c>app_tracking_summary</c> terminal event.
/// <para>
/// <b>StartedAtUtc</b> is recorded on the first lifecycle transition into Downloading /
/// Installing / InProgress. <b>CompletedAtUtc</b> is recorded on the first transition to a
/// terminal state (Installed / Skipped / Postponed / Error). Both are "set-once" — subsequent
/// identical-state events do not overwrite an earlier stamp.
/// Installing / InProgress and never overwritten. <b>CompletedAtUtc</b> is recorded on a
/// transition to a terminal state (Installed / Skipped / Postponed / Error) and re-armed
/// to <c>null</c> when the app goes terminal → active again (IME retry / re-evaluation,
/// audit Q1) — so the final terminal event carries first-start → last-terminal.
/// </para>
/// </summary>
public sealed class AppInstallTiming
@@ -30,10 +31,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime
/// <summary>
/// Elapsed time from first Downloading/Installing transition to terminal state, in
/// seconds. <c>null</c> when either endpoint is missing (e.g. app still installing,
/// or the adapter missed the start event because it came online mid-install).
/// or the adapter missed the start event because it came online mid-install) — and
/// <c>null</c> when the endpoints are inverted (CompletedAt before StartedAt: log lines
/// from different source files can be processed out of chronological order; observed in
/// production as durationSeconds=-2342 on a Skipped-then-retried user-scope app,
/// audit Q1). A negative duration is never a real measurement, so it is not emitted.
/// </summary>
public double? DurationSeconds =>
StartedAtUtc.HasValue && CompletedAtUtc.HasValue
StartedAtUtc.HasValue && CompletedAtUtc.HasValue && CompletedAtUtc.Value >= StartedAtUtc.Value
? (CompletedAtUtc.Value - StartedAtUtc.Value).TotalSeconds
: (double?)null;
}
@@ -706,10 +706,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
/// the desktop via "Continue anyway".
/// </para>
/// </summary>
// Serializes the ESP-config read+emit across ALL callers (CollectAll on the startup
// ThreadPool task, CollectAtEnrollmentStart and RefreshEspConfiguration on their own
// Task.Run) — without it, two concurrent collects can interleave read-before-emit and
// the LATER emission can carry the OLDER (smaller) registry snapshot. The backend keeps
// the latest list-bearing emission as positive blocking evidence, so an out-of-order
// emission would silently shrink it (Codex review). Inside the lock each collect is
// read→emit atomic: a later emission always carries an equal-or-newer snapshot.
private readonly object _espConfigCollectLock = new object();
// Internal (not private) as a test seam: unlike CollectAll/CollectAtEnrollmentStart this
// method touches no live system state once both probes are overridden, so tests can
// drive the esp_config_detected event surface in isolation (InternalsVisibleTo).
internal (bool? skipUserStatusPage, bool? skipDeviceStatusPage) CollectEspConfiguration()
{
lock (_espConfigCollectLock)
{
return CollectEspConfigurationLocked();
}
}
private (bool? skipUserStatusPage, bool? skipDeviceStatusPage) CollectEspConfigurationLocked()
{
bool? skipUser = null;
bool? skipDevice = null;
@@ -114,6 +114,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
CollectTpmStatus();
}
/// <summary>
/// Re-reads the ESP configuration (skip flags + ESPTrackingInfo blocking lists) and
/// re-emits <c>esp_config_detected</c> when the payload changed. Audit Q2: the earlier
/// emissions (agent start, DeviceSetup detection) fire before the registry lists are
/// complete — the ESP writes one timestamped subkey per CSP status write, so the
/// device-scope list grows progressively and the user-scope <c>S-&lt;SID&gt;</c> lists
/// appear only after sign-in. Called by <see cref="Orchestration.DeviceInfoHost"/> as
/// phase-driven one-shots (apps sub-phase opened, AccountSetup detected) — never on a
/// timer. The StartupEventGate keeps an unchanged payload silent, so a re-collect that
/// found nothing new emits nothing.
/// </summary>
public void RefreshEspConfiguration(string trigger)
{
_logger.Info($"EnrollmentTracker: re-collecting ESP configuration (trigger={trigger})");
CollectEspConfiguration();
}
/// <summary>
/// Re-collects device info that may change during enrollment (e.g. BitLocker enabled via policy).
/// Called at enrollment complete / FinalizingSetup transition to capture final state.
@@ -77,11 +77,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
protected override void Collect()
{
// Full path writes 18 keys: agent_version + process metrics (5: cpu, ws, private,
// threads, handles) + spool stats (4) + network delta (8: requests, failures,
// bytes_up/down, avg_latency, total_up/down/requests). cap=18 → HashHelpers.GetPrime(18)=23
// buckets → no resize on the 18th key (cap=16 would land at 17 buckets and resize).
var data = new Dictionary<string, object>(capacity: 18, StringComparer.Ordinal)
// Full path writes 19 keys: agent_version + process metrics (5: cpu, ws, private,
// threads, handles) + spool stats (4) + network delta (9: requests, failures,
// bytes_up/down, avg_latency, total_up/down/requests/latency). cap=19 →
// HashHelpers.GetPrime(19)=23 buckets → no resize on the 19th key.
var data = new Dictionary<string, object>(capacity: 19, StringComparer.Ordinal)
{
{ "agent_version", _agentVersion }
};
@@ -164,6 +164,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
data["net_total_bytes_up"] = currentNet.TotalBytesUp;
data["net_total_bytes_down"] = currentNet.TotalBytesDown;
data["net_total_requests"] = currentNet.RequestCount;
// Cumulative latency sum: total/requests = session-wide average HTTP
// round-trip. The backend projects that average onto the Session row
// (AvgApiLatencyMs) — cumulative counters make the write idempotent.
data["net_total_latency_ms"] = currentNet.TotalLatencyMs;
}
catch (Exception ex)
{
@@ -36,6 +36,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
/// <item><c>CollectAtEnd</c> — once, on <c>FinalizingSetup</c> or desktop arrival
/// (whichever comes first; desktop arrival also covers no-ESP / WDP v2). Re-fetches
/// BitLocker (commonly enabled via policy DURING enrollment) + the active NIC.</item>
/// <item><c>RefreshEspConfiguration</c> — audit Q2, one-shot per trigger: the ESP
/// blocking lists in the registry grow progressively (device scope) and the user-scope
/// lists appear only after sign-in, so the early emissions are structurally partial.
/// Re-collected when the apps sub-phase opens (first app activity in DeviceSetup /
/// AccountSetup — the adapter's <c>phase_transition</c> declaration) and when the
/// AccountSetup ESP phase itself is detected. Phase-driven one-shots, never a timer
/// (no-heartbeat policy).</item>
/// </list>
/// <para>
/// Duplicate-event cost is zero: every re-collected event runs through the collector's
@@ -46,6 +53,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
{
public string Name => "DeviceInfoCollector";
// ESP-config refresh trigger labels (audit Q2) — also the values passed to
// DeviceInfoCollector.RefreshEspConfiguration for the agent-log line.
internal const string EspConfigTriggerAppsDevice = "apps_phase_device";
internal const string EspConfigTriggerAppsUser = "apps_phase_user";
internal const string EspConfigTriggerAccountSetup = "account_setup_detected";
private readonly Monitoring.Telemetry.DeviceInfo.DeviceInfoCollector _collector;
private readonly AgentLogger _logger;
@@ -58,6 +71,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
private int _endCollected;
private int _disposed;
// Audit Q2 — fire-once latches per ESP-config refresh trigger. Guarded by _espConfigLock
// instead of three Interlocked ints so the membership test + add stays one atomic step.
private readonly object _espConfigLock = new object();
private readonly HashSet<string> _espConfigRefreshesFired = new HashSet<string>(StringComparer.Ordinal);
public DeviceInfoHost(
string sessionId,
string tenantId,
@@ -100,6 +118,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
private void OnSignalPosted(DecisionSignalKind kind, IReadOnlyDictionary<string, string>? payload)
{
HandleEspConfigRefreshTrigger(kind, payload);
var startTrigger = IsEnrollmentStartTrigger(kind, payload);
var endTrigger = IsEndTrigger(kind, payload);
if (!startTrigger && !endTrigger) return;
@@ -111,10 +131,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
var runEndCollect = endTrigger && Interlocked.Exchange(ref _endCollected, 1) == 0;
if (!runStartRefresh && !runEndCollect) return;
// Both one-shots done → nothing left to observe.
if (Volatile.Read(ref _enrollmentStartCollected) == 1 && Volatile.Read(ref _endCollected) == 1)
Unsubscribe();
var trigger = kind == DecisionSignalKind.DesktopArrived ? "desktop_arrived" : "esp_phase_changed";
_logger.Info($"DeviceInfoHost: trigger '{trigger}' — scheduling re-collect (enrollmentStart={runStartRefresh}, end={runEndCollect}).");
@@ -134,6 +150,32 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
});
}
/// <summary>
/// Audit Q2 — ESP-config refresh one-shots. Each trigger label fires at most once per
/// agent run; the collector's StartupEventGate keeps an unchanged payload silent, so
/// the worst case is one silent registry re-read per trigger. The subscription stays
/// armed for the host's lifetime (unlike the start/end pair there is no "all triggers
/// seen" point that is guaranteed to arrive — SkipUser enrollments never enter
/// AccountSetup); the per-signal cost is a few dictionary lookups.
/// </summary>
private void HandleEspConfigRefreshTrigger(DecisionSignalKind kind, IReadOnlyDictionary<string, string>? payload)
{
var trigger = ClassifyEspConfigRefreshTrigger(kind, payload);
if (trigger == null) return;
lock (_espConfigLock)
{
if (!_espConfigRefreshesFired.Add(trigger)) return;
}
_logger.Info($"DeviceInfoHost: ESP-config refresh trigger '{trigger}' — scheduling re-collect.");
Task.Run(() =>
{
try { _collector.RefreshEspConfiguration(trigger); }
catch (Exception ex) { _logger.Warning($"DeviceInfoHost: ESP-config refresh threw: {ex.Message}"); }
});
}
/// <summary>First DeviceSetup phase signal — MDM enrollment has populated the registry surface.</summary>
internal static bool IsEnrollmentStartTrigger(DecisionSignalKind kind, IReadOnlyDictionary<string, string>? payload)
{
@@ -156,6 +198,45 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
&& string.Equals(phase, nameof(EnrollmentPhase.FinalizingSetup), StringComparison.Ordinal);
}
/// <summary>
/// Audit Q2 — maps a posted signal to an ESP-config refresh trigger label, or
/// <c>null</c> when the signal is not one. Two trigger families:
/// <list type="bullet">
/// <item>The adapter's apps sub-phase declaration (<c>phase_transition</c>
/// InformationalEvent with phase AppsDevice/AppsUser — fired on the first app
/// activity in an ESP phase): by then IME is actively processing the blocking set,
/// so the device-scope ESPTrackingInfo lists are as complete as they get.</item>
/// <item>The AccountSetup ESP phase (UserSetup detection): sign-in has happened, so
/// the user-scope <c>S-&lt;SID&gt;</c> lists exist now — production showed 88 of 89
/// sessions with an EMPTY user list at the last pre-fix emission.</item>
/// </list>
/// </summary>
internal static string? ClassifyEspConfigRefreshTrigger(DecisionSignalKind kind, IReadOnlyDictionary<string, string>? payload)
{
if (payload == null) return null;
if (kind == DecisionSignalKind.InformationalEvent
&& payload.TryGetValue(SignalPayloadKeys.EventType, out var eventType)
&& string.Equals(eventType, AutopilotMonitor.Shared.Constants.EventTypes.PhaseTransition, StringComparison.Ordinal)
&& payload.TryGetValue(Telemetry.Events.EventTimelineEmitter.PhaseParamKey, out var declaredPhase))
{
if (string.Equals(declaredPhase, nameof(EnrollmentPhase.AppsDevice), StringComparison.Ordinal))
return EspConfigTriggerAppsDevice;
if (string.Equals(declaredPhase, nameof(EnrollmentPhase.AppsUser), StringComparison.Ordinal))
return EspConfigTriggerAppsUser;
return null;
}
if (kind == DecisionSignalKind.EspPhaseChanged
&& payload.TryGetValue(SignalPayloadKeys.EspPhase, out var espPhase)
&& string.Equals(espPhase, nameof(EnrollmentPhase.AccountSetup), StringComparison.Ordinal))
{
return EspConfigTriggerAccountSetup;
}
return null;
}
public void Stop() => Unsubscribe();
public void Dispose()
@@ -93,7 +93,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
{
using (obj)
{
manufacturer = obj["Manufacturer"]?.ToString() ?? Unknown;
manufacturer = NormalizeWmiValue(obj["Manufacturer"]?.ToString());
if (manufacturer.IndexOf("lenovo", StringComparison.OrdinalIgnoreCase) >= 0)
{
using (var lenovoSearcher = new ManagementObjectSearcher("SELECT Version FROM Win32_ComputerSystemProduct"))
@@ -103,7 +103,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
{
using (lenovoObj)
{
model = lenovoObj["Version"]?.ToString() ?? Unknown;
model = NormalizeWmiValue(lenovoObj["Version"]?.ToString());
}
break;
}
@@ -111,7 +111,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
}
else
{
model = obj["Model"]?.ToString() ?? Unknown;
model = NormalizeWmiValue(obj["Model"]?.ToString());
}
}
}
@@ -132,7 +132,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
{
using (obj)
{
serialNumber = obj["SerialNumber"]?.ToString() ?? Unknown;
serialNumber = NormalizeWmiValue(obj["SerialNumber"]?.ToString());
}
}
}
@@ -154,5 +154,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
private static bool IsResolved(string value)
=> !string.IsNullOrWhiteSpace(value)
&& !string.Equals(value, Unknown, StringComparison.OrdinalIgnoreCase);
/// <summary>
/// Trims a raw WMI string value at the source (audit Q5: BIOS serials are used as device
/// identity keys downstream — F2 device history joins on them — so padding must never
/// leave the agent). Whitespace-only and empty values collapse to the "Unknown" sentinel,
/// matching the null fallback so the retry logic treats them as unresolved.
/// </summary>
internal static string NormalizeWmiValue(string raw)
{
if (string.IsNullOrWhiteSpace(raw)) return Unknown;
return raw.Trim();
}
}
}
@@ -132,7 +132,9 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
// Plan §5 Fix 4c — per-app install-lifecycle timing captured from state transitions.
// StartedAtUtc on first Downloading/Installing/InProgress transition (set-once);
// CompletedAtUtc on first terminal transition (Installed/Skipped/Postponed/Error).
// CompletedAtUtc on terminal transition (Installed/Skipped/Postponed/Error), re-armed
// to null on a terminal→active transition (IME retry, audit Q1) so the final terminal
// stamp reflects the last outcome instead of freezing at the first one.
// Surfaced via <see cref="AppTimings"/> to peripheral consumers (FinalStatusBuilder,
// app_tracking_summary emission in EnrollmentTerminationHandler).
private readonly Dictionary<string, (DateTime? Started, DateTime? Completed)> _appTimings =
@@ -691,10 +693,20 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
}
/// <summary>
/// Plan §5 Fix 4c — set-once per-app install-lifecycle timing. Called for every
/// state transition, no-op for duplicate stamps. Returns the up-to-date
/// Plan §5 Fix 4c — per-app install-lifecycle timing. Called for every state
/// transition, no-op for duplicate stamps. Returns the up-to-date
/// <see cref="AppInstallTiming"/> so the emitter can inline the values on the
/// current event's payload without a second dictionary lookup.
/// <para>
/// Audit Q1: a terminal→active transition (the IME retry path — <see
/// cref="AppPackageState.UpdateState"/> permits e.g. Error→Downloading and
/// Skipped→Downloading) re-arms <c>Completed</c> to <c>null</c>, so the NEXT terminal
/// transition stamps a fresh CompletedAt. StartedAt keeps its first value — the final
/// terminal payload then covers first-start → last-terminal (total occupancy across
/// attempts) instead of freezing at the first failure. Observed pre-fix: a user-scope
/// app Skipped early in DeviceSetup, retried after sign-in, emitted its final
/// Installed event with the stale early CompletedAt (durationSeconds=-2342).
/// </para>
/// </summary>
private AppInstallTiming UpdateAppTiming(string appId, AppInstallationState newState, DateTime now)
{
@@ -702,13 +714,20 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
{
_appTimings.TryGetValue(appId, out var current);
// Terminal→active (retry): the earlier terminal stamp no longer describes this
// app's outcome — re-arm so the next terminal transition stamps fresh.
if (current.Completed != null && IsLifecycleActive(newState))
{
current.Completed = null;
}
// First lifecycle-active transition → record StartedAt.
if (current.Started == null && IsLifecycleActive(newState))
{
current.Started = now;
}
// First terminal transition → record CompletedAt.
// Terminal transition → record CompletedAt (fresh after a re-arm).
if (current.Completed == null && IsCompletedState(newState))
{
current.Completed = now;
@@ -0,0 +1,333 @@
using System.Text.Json;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Functions.Apps;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// F1 PR1 (insights spec §F1, source-data audit Q2/Q3): app identity on the name-keyed
/// summary row and the ESP-blocking join.
///
/// Ingest side: <c>appId</c> is adopted from the app events; a second appId under the same
/// name flags the row (<see cref="AppInstallSummary.AppIdCollision"/>) — in-batch and
/// cross-batch. Store side: sentinel-gated columns + sticky collision. Join side:
/// <see cref="EspBlockingSets"/> is positive evidence only, and
/// <see cref="TableStorageService.ShouldStampEspBlocking"/> never stamps collision rows,
/// id-less rows, or non-members. Aggregate side: collision rows leave every per-app fleet
/// aggregate with a disclosed exclusion count.
/// </summary>
public class AppInstallIdentityAndEspBlockingTests
{
private static readonly DateTime T0 = new(2026, 7, 26, 10, 0, 0, DateTimeKind.Utc);
private const string IdA = "aaaaaaaa-1111-2222-3333-444444444444";
private const string IdB = "bbbbbbbb-1111-2222-3333-444444444444";
// ── ingest fold: appId adoption + in-batch collision ────────────────────
private static Dictionary<string, AppInstallAggregationState> Aggregate(params EnrollmentEvent[] events)
{
var summaries = new Dictionary<string, AppInstallAggregationState>(StringComparer.OrdinalIgnoreCase);
foreach (var evt in events)
EventIngestProcessor.AggregateAppInstallEvent(evt, "tenant", "session", summaries);
return summaries;
}
private static EnrollmentEvent AppEvent(string eventType, string? appId = null, string appName = "App")
{
var evt = new EnrollmentEvent
{
EventType = eventType,
Timestamp = T0,
Data = new Dictionary<string, object> { ["appName"] = appName },
};
if (appId != null) evt.Data["appId"] = appId;
return evt;
}
[Fact]
public void Ingest_AdoptsAppId_FromFirstObservingEvent()
{
var s = Aggregate(
AppEvent("app_install_started", IdA),
AppEvent("app_install_completed", IdA))["App"].Summary;
Assert.Equal(IdA, s.AppId);
Assert.False(s.AppIdCollision);
}
[Fact]
public void Ingest_SameName_DifferentAppId_FlagsCollision_FirstSeenIdWins()
{
// Device- + user-scope assignment of the same display name in one batch (audit Q3:
// observed in production as two "Company Portal" appIds in one session).
var s = Aggregate(
AppEvent("app_install_started", IdA),
AppEvent("app_install_started", IdB))["App"].Summary;
Assert.True(s.AppIdCollision);
Assert.Equal(IdA, s.AppId);
}
[Fact]
public void Ingest_AppIdCaseDrift_IsNotACollision()
{
var s = Aggregate(
AppEvent("app_install_started", IdA),
AppEvent("app_install_completed", IdA.ToUpperInvariant()))["App"].Summary;
Assert.False(s.AppIdCollision);
}
[Fact]
public void Ingest_EventsWithoutAppId_LeaveSentinelEmpty()
{
var s = Aggregate(AppEvent("app_install_completed"))["App"].Summary;
Assert.Equal(string.Empty, s.AppId);
Assert.False(s.AppIdCollision);
}
// ── reconcile: cross-batch identity ─────────────────────────────────────
[Fact]
public void Reconcile_AdoptsStoredAppId_WhenIncomingBatchHasNone()
{
var existing = new TableEntity("tenant", "session_App") { ["AppId"] = IdA };
var summary = new AppInstallSummary { AppName = "App", StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal(IdA, summary.AppId);
Assert.False(summary.AppIdCollision);
}
[Fact]
public void Reconcile_CrossBatchDifferentAppId_FlagsCollision_StoredIdWins()
{
var existing = new TableEntity("tenant", "session_App") { ["AppId"] = IdA };
var summary = new AppInstallSummary { AppName = "App", AppId = IdB, StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.True(summary.AppIdCollision);
Assert.Equal(IdA, summary.AppId); // first-seen (stored) identity wins, matching the fold rule
}
[Fact]
public void Reconcile_CollisionIsSticky_AndEspBlockingVerdictIsAdopted()
{
var existing = new TableEntity("tenant", "session_App")
{
["AppId"] = IdA,
["AppIdCollision"] = true,
["EspBlocking"] = true,
};
var summary = new AppInstallSummary { AppName = "App", AppId = IdA, StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.True(summary.AppIdCollision); // one-way sticky
Assert.True(summary.EspBlocking); // terminal-path verdict survives late batches
}
// ── entity builder: sentinel gating ─────────────────────────────────────
[Fact]
public void EntityBuilder_IdentityColumns_AreSentinelGated()
{
var empty = TableStorageService.BuildAppInstallSummaryEntity(
new AppInstallSummary { AppName = "App" }, "rk");
// Absent columns → Merge-mode preserves prior values / stays unknown.
Assert.False(empty.ContainsKey("AppId"));
Assert.False(empty.ContainsKey("EspBlocking"));
Assert.False(empty.ContainsKey("AppIdCollision"));
var full = TableStorageService.BuildAppInstallSummaryEntity(
new AppInstallSummary { AppName = "App", AppId = IdA, EspBlocking = true, AppIdCollision = true }, "rk");
Assert.Equal(IdA, full.GetString("AppId"));
Assert.True(full.GetBoolean("EspBlocking"));
Assert.True(full.GetBoolean("AppIdCollision"));
}
// ── EspBlockingSets: positive-evidence parsing ──────────────────────────
private static Dictionary<string, object> Lists(
object? win32 = null, object? userWin32 = null, object? msi = null, object? pfn = null,
long? win32Count = null)
{
var d = new Dictionary<string, object> { ["source"] = "registry_firstsync" };
if (win32 != null) d["espTrackedWin32AppIds"] = win32;
if (userWin32 != null) d["espTrackedUserWin32AppIds"] = userWin32;
if (msi != null) d["espTrackedMsiProductCodes"] = msi;
if (pfn != null) d["espTrackedModernAppPfns"] = pfn;
if (win32Count.HasValue) d["espTrackedWin32Count"] = win32Count.Value;
return d;
}
[Fact]
public void Sets_PayloadWithoutListKeys_YieldsNull_NoListsIsNotEmptyLists()
{
Assert.Null(EspBlockingSets.FromEventData(null));
Assert.Null(EspBlockingSets.FromEventData(Lists())); // probe found no Diagnostics key
}
[Fact]
public void Sets_MembershipIsCaseInsensitive_AcrossAllFourLists()
{
// Storage rehydration shape: List<object> (EventDataNormalizer).
var sets = EspBlockingSets.FromEventData(Lists(
win32: new List<object> { IdA },
msi: new List<object> { "{11111111-2222-3333-4444-555555555555}" },
pfn: new List<object> { "Contoso.App_abc123" }))!;
Assert.True(sets.Contains(IdA.ToUpperInvariant()));
Assert.True(sets.Contains("{11111111-2222-3333-4444-555555555555}"));
Assert.True(sets.Contains("Contoso.App_abc123"));
Assert.False(sets.Contains(IdB)); // absent ⇒ unknown at the caller, never false here
Assert.False(sets.Contains(null));
Assert.Equal(3, sets.ListedCount);
Assert.False(sets.IsTruncated);
}
[Fact]
public void Sets_EmittedCountAboveListLength_MeansTruncated()
{
var sets = EspBlockingSets.FromEventData(Lists(
win32: new List<object> { IdA }, win32Count: 60))!;
Assert.True(sets.IsTruncated);
}
[Fact]
public void Sets_UserScopeSubset_IsExposedForCoverageLabeling()
{
var sets = EspBlockingSets.FromEventData(Lists(
win32: new List<object> { IdA, IdB },
userWin32: new List<object> { IdB }))!;
Assert.Contains(IdB, sets.UserWin32AppIds);
Assert.DoesNotContain(IdA, sets.UserWin32AppIds);
}
// ── terminal-path stamping predicate ────────────────────────────────────
private static TableEntity Row(string? appId = null, bool? collision = null, bool? espBlocking = null)
{
var row = new TableEntity("tenant", "session_App");
if (appId != null) row["AppId"] = appId;
if (collision.HasValue) row["AppIdCollision"] = collision.Value;
if (espBlocking.HasValue) row["EspBlocking"] = espBlocking.Value;
return row;
}
[Fact]
public void Stamp_OnlyPositiveMembers_NeverIdlessCollidedOrAlreadyStamped()
{
var sets = EspBlockingSets.FromEventData(Lists(win32: new List<object> { IdA }))!;
Assert.True(TableStorageService.ShouldStampEspBlocking(Row(IdA), sets));
Assert.False(TableStorageService.ShouldStampEspBlocking(Row(), sets)); // no identity → unknown
Assert.False(TableStorageService.ShouldStampEspBlocking(Row(IdB), sets)); // absent ⇒ unknown, never false
Assert.False(TableStorageService.ShouldStampEspBlocking(Row(IdA, collision: true), sets)); // ambiguous identity
Assert.False(TableStorageService.ShouldStampEspBlocking(Row(IdA, espBlocking: true), sets)); // idempotent
}
// ── per-app fleet aggregates: collision rows leave, disclosed ───────────
private static AppInstallSummary Summary(
string appName, string status = "Succeeded", bool collision = false, int duration = 30)
=> new()
{
TenantId = "00000000-0000-0000-0000-000000000fa1",
SessionId = "s-1",
AppName = appName,
Status = status,
TerminalState = status == "Succeeded" ? "Installed" : "Error",
DurationSeconds = duration,
StartedAt = DateTime.UtcNow.AddHours(-2),
AppIdCollision = collision,
};
[Fact]
public void AppMetricsPayload_ExcludesCollisionRowsFromGroups_DisclosesCount()
{
var summaries = new List<AppInstallSummary>
{
Summary("Clean App"),
Summary("Clean App"),
Summary("Company Portal", collision: true),
Summary("Company Portal", status: "Failed", collision: true),
};
var root = JsonSerializer.SerializeToElement(MetricsMath.BuildAppMetricsPayload(summaries));
Assert.Equal(1, root.GetProperty("totalApps").GetInt32()); // collided name forms no group
Assert.Equal(4, root.GetProperty("totalInstalls").GetInt32()); // raw row count unchanged
Assert.Equal(2, root.GetProperty("totalCollisionExcluded").GetInt32());
Assert.Empty(root.GetProperty("topFailingApps").EnumerateArray()); // the collided Failed row ranks nowhere
}
[Fact]
public void AppsList_ExcludesCollisionRows_DisclosesCount()
{
var summaries = new List<AppInstallSummary>
{
Summary("Clean App"),
Summary("Company Portal", collision: true),
};
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
var apps = root.GetProperty("apps").EnumerateArray().ToList();
Assert.Single(apps);
Assert.Equal("Clean App", apps[0].GetProperty("appName").GetString());
Assert.Equal(2, root.GetProperty("totalInstalls").GetInt32());
Assert.Equal(1, root.GetProperty("collisionExcluded").GetInt32());
}
[Fact]
public async Task AppAnalytics_ExcludesCollisionRows_DisclosesCount()
{
var repo = new Mock<ISessionRepository>();
repo.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync((SessionSummary?)null);
var summaries = new List<AppInstallSummary>
{
Summary("Company Portal"),
Summary("Company Portal", status: "Failed", collision: true), // the second identity's failure
};
var root = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, repo.Object, "Company Portal", days: 30));
Assert.Equal(1, root.GetProperty("collisionExcluded").GetInt32());
var stats = root.GetProperty("summary");
Assert.Equal(1, stats.GetProperty("totalInstalls").GetInt32());
Assert.Equal(0, stats.GetProperty("failed").GetInt32()); // ambiguous failure not attributed
Assert.Equal(0d, stats.GetProperty("failureRate").GetDouble());
}
[Fact]
public async Task AppAnalytics_AllRowsCollided_ReturnsEmptyShape_WithDisclosure()
{
var repo = new Mock<ISessionRepository>();
repo.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync((SessionSummary?)null);
var summaries = new List<AppInstallSummary> { Summary("Company Portal", collision: true) };
var root = JsonSerializer.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, repo.Object, "Company Portal", days: 30));
Assert.Equal(1, root.GetProperty("collisionExcluded").GetInt32());
Assert.Equal(0, root.GetProperty("summary").GetProperty("totalInstalls").GetInt32());
}
}
@@ -0,0 +1,201 @@
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// PR0 (2026-07-26, source-data audit §0.5 of tasks/insights-expansion-spec.md).
///
/// Ingest side: the agent emits <c>app_install_completed</c> for EVERY terminal transition —
/// including Skipped/Postponed no-ops (V1 wire parity; production: 78 % of zero-duration rows
/// were skips). The payload's <c>state</c> field must be persisted as
/// <see cref="AppInstallSummary.TerminalState"/> so metrics can exclude skips from durations
/// and rates.
///
/// Store side: <see cref="TableStorageService.ReconcileAppInstallSummaryWithExisting"/> pins the
/// cross-batch contract — earliest StartedAt wins, terminal status is sticky, and the Q4
/// out-of-order case (terminal batch first, started batch later) recomputes the duration.
/// </summary>
public class AppInstallTerminalStateAndReconcileTests
{
private static readonly DateTime T0 = new(2026, 7, 26, 10, 0, 0, DateTimeKind.Utc);
// ── ingest: TerminalState fold ──────────────────────────────────────────
private static Dictionary<string, AppInstallAggregationState> Aggregate(params EnrollmentEvent[] events)
{
var summaries = new Dictionary<string, AppInstallAggregationState>(StringComparer.OrdinalIgnoreCase);
foreach (var evt in events)
EventIngestProcessor.AggregateAppInstallEvent(evt, "tenant", "session", summaries);
return summaries;
}
private static EnrollmentEvent AppEvent(string eventType, DateTime ts, string? state = null, string appName = "App")
{
var evt = new EnrollmentEvent
{
EventType = eventType,
Timestamp = ts,
Data = new Dictionary<string, object> { ["appName"] = appName },
};
if (state != null) evt.Data["state"] = state;
return evt;
}
[Theory]
[InlineData("Installed")]
[InlineData("Skipped")]
[InlineData("Postponed")]
public void Completed_PersistsPayloadState(string state)
{
var s = Aggregate(AppEvent("app_install_completed", T0, state))["App"].Summary;
Assert.Equal("Succeeded", s.Status); // backward-compatible status
Assert.Equal(state, s.TerminalState); // new precision column
}
[Fact]
public void Completed_WithoutState_LeavesSentinelEmpty_NeverGuessed()
{
var s = Aggregate(AppEvent("app_install_completed", T0))["App"].Summary;
Assert.Equal(string.Empty, s.TerminalState);
}
[Fact]
public void Completed_WithNonTerminalOrGarbageState_LeavesSentinelEmpty()
{
// A payload state outside the terminal set (bug or drift) must not be persisted.
var s = Aggregate(AppEvent("app_install_completed", T0, "Downloading"))["App"].Summary;
Assert.Equal(string.Empty, s.TerminalState);
}
[Fact]
public void Failed_SetsError()
{
var s = Aggregate(AppEvent("app_install_failed", T0))["App"].Summary;
Assert.Equal("Failed", s.Status);
Assert.Equal("Error", s.TerminalState);
}
[Fact]
public void FailedThenCompleted_RetrySucceeds_LastTerminalWins()
{
// IME retry path (audit Q1): Error → retry → Installed. The later completed event is
// the authoritative outcome for Status AND TerminalState.
var s = Aggregate(
AppEvent("app_install_failed", T0),
AppEvent("app_install_completed", T0.AddMinutes(10), "Installed"))["App"].Summary;
Assert.Equal("Succeeded", s.Status);
Assert.Equal("Installed", s.TerminalState);
}
[Fact]
public void SkippedEvent_SetsSkipped_ButNeverOverridesStrongerTerminal()
{
var fresh = Aggregate(AppEvent("app_install_skipped", T0))["App"].Summary;
Assert.Equal("Succeeded", fresh.Status);
Assert.Equal("Skipped", fresh.TerminalState);
var afterFailure = Aggregate(
AppEvent("app_install_failed", T0),
AppEvent("app_install_skipped", T0.AddSeconds(5)))["App"].Summary;
Assert.Equal("Error", afterFailure.TerminalState);
}
// ── store: cross-batch reconcile ────────────────────────────────────────
[Fact]
public void Reconcile_Q4_OutOfOrder_RecomputesDurationFromStoredCompletedAt()
{
// Terminal batch landed first: row has CompletedAt but no usable duration.
var existing = new TableEntity("tenant", "session_App")
{
["Status"] = "Succeeded",
["StartedAt"] = new DateTimeOffset(T0.AddMinutes(5)), // late (completed-event) stamp
["CompletedAt"] = new DateTimeOffset(T0.AddMinutes(5)),
};
// Started batch arrives now with the true (earlier) start.
var summary = new AppInstallSummary
{
AppName = "App", SessionId = "session", TenantId = "tenant",
Status = "InProgress", StartedAt = T0,
};
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal(T0, summary.StartedAt); // earlier start kept
Assert.Equal(T0.AddMinutes(5), summary.CompletedAt); // endpoint adopted from the row
Assert.Equal(300, summary.DurationSeconds); // recomputed — was permanently unset before PR0
Assert.Equal("Succeeded", summary.Status); // sticky terminal (below)
}
[Fact]
public void Reconcile_TerminalStatusIsSticky_AgainstLateInProgressBatch()
{
var existing = new TableEntity("tenant", "session_App") { ["Status"] = "Failed" };
var summary = new AppInstallSummary { AppName = "App", Status = "InProgress", StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal("Failed", summary.Status);
}
[Fact]
public void Reconcile_TerminalToTerminal_StaysWithIncomingEvidence()
{
// Failed row + succeeded batch (successful retry) → the new terminal wins.
var existing = new TableEntity("tenant", "session_App") { ["Status"] = "Failed" };
var summary = new AppInstallSummary { AppName = "App", Status = "Succeeded", StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal("Succeeded", summary.Status);
}
[Fact]
public void Reconcile_Q4Guard_DoesNotFireWithoutStoredCompletedAt()
{
var existing = new TableEntity("tenant", "session_App") { ["Status"] = "InProgress" };
var summary = new AppInstallSummary { AppName = "App", Status = "InProgress", StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Null(summary.CompletedAt);
Assert.Equal(0, summary.DurationSeconds);
}
[Fact]
public void Reconcile_Q4Guard_IgnoresStoredCompletedAtBeforeIncomingStart()
{
// Negative interval (clock mess) must not produce a negative duration.
var existing = new TableEntity("tenant", "session_App")
{
["Status"] = "Succeeded",
["CompletedAt"] = new DateTimeOffset(T0.AddMinutes(-10)),
};
var summary = new AppInstallSummary { AppName = "App", Status = "InProgress", StartedAt = T0 };
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Null(summary.CompletedAt);
Assert.Equal(0, summary.DurationSeconds);
}
// ── entity builder: sentinel gating for the new column ─────────────────
[Fact]
public void EntityBuilder_TerminalState_IsSentinelGated()
{
var withState = new AppInstallSummary { AppName = "App", TerminalState = "Skipped" };
var entity = TableStorageService.BuildAppInstallSummaryEntity(withState, "rk");
Assert.Equal("Skipped", entity.GetString("TerminalState"));
var withoutState = new AppInstallSummary { AppName = "App" };
var empty = TableStorageService.BuildAppInstallSummaryEntity(withoutState, "rk");
// Column absent → Merge-mode preserves a prior batch's terminal state.
Assert.False(empty.ContainsKey("TerminalState"));
}
}
@@ -162,6 +162,58 @@ public class AppsAnalyticsFailureRateTests
Assert.Equal(50d, todayBucket.GetProperty("failureRate").GetDouble());
}
// ── PR0 (2026-07-26): skip / unmeasured classification ──────────────────
[Fact]
public void List_Skips_LeaveRateAndDurations_AndAreReportedSeparately()
{
var skip = A("Succeeded");
skip.TerminalState = "Skipped";
skip.DurationSeconds = 0;
var unmeasured = A("Succeeded");
unmeasured.TerminalState = "Installed";
unmeasured.DurationSeconds = 0;
var summaries = new List<AppInstallSummary> { A("Failed"), A("Succeeded"), skip, unmeasured };
var root = JsonSerializer.SerializeToElement(
AppsAnalyticsHelper.BuildAppsListResponse(summaries, days: 30));
var app = root.GetProperty("apps")[0];
Assert.Equal(4, app.GetProperty("totalInstalls").GetInt32());
Assert.Equal(2, app.GetProperty("succeeded").GetInt32()); // real install + unmeasured
Assert.Equal(1, app.GetProperty("skipped").GetInt32());
Assert.Equal(1, app.GetProperty("unmeasured").GetInt32());
// 1 / (1 + 2) — the skip is not an attempt.
Assert.Equal(33.3d, app.GetProperty("failureRate").GetDouble());
// avg over the single measured row (30s) — the zero must not drag it to 15.
Assert.Equal(30d, app.GetProperty("avgDurationSeconds").GetDouble());
}
[Fact]
public async Task Analytics_Summary_CarriesSkippedAndUnmeasured_AndP95OverMeasuredOnly()
{
var skip = A("Succeeded");
skip.TerminalState = "Skipped";
skip.DurationSeconds = 0;
var unmeasured = A("Succeeded");
unmeasured.TerminalState = "Installed";
unmeasured.DurationSeconds = 0;
var measured = A("Succeeded");
measured.TerminalState = "Installed";
measured.DurationSeconds = 100;
var root = await BuildAnalyticsAsync(new List<AppInstallSummary> { skip, unmeasured, measured });
var summary = root.GetProperty("summary");
Assert.Equal(2, summary.GetProperty("succeeded").GetInt32());
Assert.Equal(1, summary.GetProperty("skipped").GetInt32());
Assert.Equal(1, summary.GetProperty("unmeasured").GetInt32());
Assert.Equal(100d, summary.GetProperty("avgDurationSeconds").GetDouble());
Assert.Equal(100, summary.GetProperty("p95DurationSeconds").GetInt32());
Assert.Equal(0d, summary.GetProperty("failureRate").GetDouble());
}
[Fact]
public async Task Analytics_DeviceModelBreakdown_RatesAndSampleFloor_UseFinishedInstalls()
{
@@ -34,6 +34,14 @@ public class AppsProjectionEquivalenceTests
["AppType"] = "Win32",
["AppVersion"] = "2.1.0",
["Status"] = "Failed",
// PR0/PR1 classification columns. The fixture MUST populate every column a Build*
// method reads: a column absent here passes the equivalence assert trivially (both
// sides map to default) — exactly how the missing TerminalState projection slipped
// through and zeroed totalSkipped in production (2026-07-27).
["TerminalState"] = "Skipped",
["AppId"] = "9e8d7c6b-5a49-4832-9182-736451234567",
["EspBlocking"] = true,
["AppIdCollision"] = true,
["StartedAt"] = new DateTimeOffset(DateTime.UtcNow.AddHours(-6)),
["CompletedAt"] = new DateTimeOffset(DateTime.UtcNow.AddHours(-5)),
["DurationSeconds"] = 340,
@@ -83,6 +91,9 @@ public class AppsProjectionEquivalenceTests
Assert.Equal(fromFull.AppName, fromProjected.AppName);
Assert.Equal(fromFull.Status, fromProjected.Status);
// IsSkipTerminalState / skip classification input — see the fixture comment.
Assert.Equal(fromFull.TerminalState, fromProjected.TerminalState);
Assert.Equal(fromFull.AppIdCollision, fromProjected.AppIdCollision);
Assert.Equal(fromFull.StartedAt, fromProjected.StartedAt);
Assert.Equal(fromFull.DurationSeconds, fromProjected.DurationSeconds);
Assert.Equal(fromFull.DownloadBytes, fromProjected.DownloadBytes);
@@ -122,6 +133,10 @@ public class AppsProjectionEquivalenceTests
Assert.Equal(fromFull.FailureMessage, fromProjected.FailureMessage);
Assert.Equal(fromFull.ExitCode, fromProjected.ExitCode);
Assert.Equal(fromFull.DetectionResult, fromProjected.DetectionResult);
Assert.Equal(fromFull.TerminalState, fromProjected.TerminalState);
Assert.Equal(fromFull.AppId, fromProjected.AppId);
Assert.Equal(fromFull.EspBlocking, fromProjected.EspBlocking);
Assert.Equal(fromFull.AppIdCollision, fromProjected.AppIdCollision);
}
[Fact]
@@ -0,0 +1,172 @@
using AutopilotMonitor.Functions.Functions.Infrastructure;
using AutopilotMonitor.Shared.DataAccess;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the pure helpers of <see cref="AzureMonitorAlertWebhookFunction"/> — the anonymous,
/// secret-gated bridge that turns Azure Monitor Common Alert Schema notifications into ops
/// events. Load-bearing bits: the secret gate is fail-closed (an unset app setting rejects
/// everything), severity mapping can never escalate on schema drift (unknown → Info), and
/// parsing is defensive (only the alert rule name is mandatory).
/// </summary>
public class AzureMonitorAlertWebhookFunctionTests
{
// ── SecretMatches ──────────────────────────────────────────────────────
[Fact]
public void SecretMatches_accepts_equal_secrets()
{
Assert.True(AzureMonitorAlertWebhookFunction.SecretMatches("s3cret-value", "s3cret-value"));
}
[Theory]
[InlineData("wrong", "right")]
[InlineData("right-but-longer", "right")]
[InlineData("", "configured")]
[InlineData(null, "configured")]
public void SecretMatches_rejects_mismatch_or_missing_provided(string? provided, string configured)
{
Assert.False(AzureMonitorAlertWebhookFunction.SecretMatches(provided, configured));
}
[Theory]
[InlineData(null)]
[InlineData("")]
public void SecretMatches_fails_closed_when_not_configured(string? configured)
{
// While the app setting is unset, NOTHING matches — not even an empty provided value.
Assert.False(AzureMonitorAlertWebhookFunction.SecretMatches("anything", configured));
Assert.False(AzureMonitorAlertWebhookFunction.SecretMatches("", configured));
Assert.False(AzureMonitorAlertWebhookFunction.SecretMatches(null, configured));
}
// ── MapSeverity ────────────────────────────────────────────────────────
[Theory]
[InlineData("Sev0", OpsEventSeverity.Critical)]
[InlineData("Sev1", OpsEventSeverity.Error)]
[InlineData("Sev2", OpsEventSeverity.Warning)]
[InlineData("Sev3", OpsEventSeverity.Info)]
[InlineData("Sev4", OpsEventSeverity.Info)]
public void MapSeverity_maps_azure_scale_to_ops_scale(string azure, string expected)
{
Assert.Equal(expected, AzureMonitorAlertWebhookFunction.MapSeverity(azure));
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData("Sev5")]
[InlineData("Critical")] // someone posting the ops scale directly must not pass through
public void MapSeverity_falls_back_to_info_on_unknown_values(string? azure)
{
Assert.Equal(OpsEventSeverity.Info, AzureMonitorAlertWebhookFunction.MapSeverity(azure));
}
// ── TryParse ───────────────────────────────────────────────────────────
private const string FullPayload = """
{
"schemaId": "azureMonitorCommonAlertSchema",
"data": {
"essentials": {
"alertId": "/subscriptions/xxx/providers/Microsoft.AlertsManagement/alerts/abc",
"alertRule": "Portal RSC prefetch tail",
"severity": "Sev2",
"signalType": "Log",
"monitorCondition": "Fired",
"monitoringService": "Log Alerts V2",
"alertTargetIDs": [
"/subscriptions/xxx/resourcegroups/rg-autopilotmonitor-prd-gwc/providers/microsoft.insights/components/autopilotmonitorweb"
],
"description": "RSC prefetches exceeding 10s",
"firedDateTime": "2026-07-27T15:30:00.000Z"
},
"alertContext": {
"condition": {
"windowSize": "PT1H",
"allOf": [
{
"searchQuery": "dependencies | where name contains '_rsc='",
"metricValue": 18.0,
"operator": "GreaterThan",
"threshold": "0"
}
]
}
}
}
}
""";
[Fact]
public void TryParse_extracts_essentials_and_metric_value()
{
var alert = AzureMonitorAlertWebhookFunction.TryParse(FullPayload);
Assert.NotNull(alert);
Assert.Equal("Portal RSC prefetch tail", alert!.AlertRule);
Assert.Equal("Sev2", alert.Severity);
Assert.Equal("Fired", alert.MonitorCondition);
Assert.Equal("RSC prefetches exceeding 10s", alert.Description);
Assert.Equal("Log Alerts V2", alert.MonitoringService);
// No targetResourceName in the payload → resource name segment of the first target ID.
Assert.Equal("autopilotmonitorweb", alert.TargetResource);
Assert.Equal(18.0, alert.MetricValue);
}
[Fact]
public void TryParse_prefers_explicit_target_resource_name()
{
const string payload = """
{"data":{"essentials":{"alertRule":"r","targetResourceName":"explicit-name",
"alertTargetIDs":["/sub/x/other-name"]}}}
""";
var alert = AzureMonitorAlertWebhookFunction.TryParse(payload);
Assert.Equal("explicit-name", alert!.TargetResource);
}
[Fact]
public void TryParse_minimal_payload_needs_only_the_alert_rule()
{
var alert = AzureMonitorAlertWebhookFunction.TryParse(
"""{"data":{"essentials":{"alertRule":"minimal"}}}""");
Assert.NotNull(alert);
Assert.Equal("minimal", alert!.AlertRule);
Assert.Null(alert.Severity);
Assert.Equal("Fired", alert.MonitorCondition); // default when absent
Assert.Null(alert.Description);
Assert.Null(alert.MetricValue);
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("not json at all")]
[InlineData("{}")] // no data
[InlineData("""{"data":{}}""")] // no essentials
[InlineData("""{"data":{"essentials":{}}}""")] // no alertRule
[InlineData("""{"data":{"essentials":{"alertRule":""}}}""")] // empty alertRule
[InlineData("""{"data":{"essentials":{"alertRule":123}}}""")] // wrong type
public void TryParse_rejects_unusable_payloads(string? body)
{
Assert.Null(AzureMonitorAlertWebhookFunction.TryParse(body!));
}
[Fact]
public void TryParse_tolerates_malformed_alert_context()
{
// alertContext with unexpected shapes must not break essentials extraction.
const string payload = """
{"data":{"essentials":{"alertRule":"r","severity":"Sev0"},
"alertContext":{"condition":{"allOf":"not-an-array"}}}}
""";
var alert = AzureMonitorAlertWebhookFunction.TryParse(payload);
Assert.NotNull(alert);
Assert.Equal("Sev0", alert!.Severity);
Assert.Null(alert.MetricValue);
}
}
@@ -75,8 +75,9 @@ public class DeletionManifestBuilderTests
Assert.Equal(2, c["classifierVerdictsByIdLevel"]);
Assert.Equal(11, c["signalsByKind"]);
// 15 cascade tables + tombstone = 16 steps; no inventory steps when no contributions row.
Assert.Equal(16, manifest.Steps.Count);
// 16 cascade tables (incl. SessionTimeBreakdowns, F1 PR2) + tombstone = 17 steps;
// no inventory steps when no contributions row.
Assert.Equal(17, manifest.Steps.Count);
Assert.Equal(DeletionStepClass.Final, manifest.Steps.Last().Class);
}
@@ -102,7 +103,8 @@ public class DeletionManifestBuilderTests
.WithSessionsByStage(1, propValue: "stage_marker")
.WithDeadEndsByReason(1, propValue: "deadend_marker")
.WithClassifierVerdictsByIdLevel(1, propValue: "classifier_marker")
.WithSignalsByKind(1, propValue: "signalkind_marker"));
.WithSignalsByKind(1, propValue: "signalkind_marker")
.WithSessionTimeBreakdown(propValue: "breakdown_marker"));
var manifest = await NewBuilder(reader).BuildAsync(
TenantId, SessionId, "admin_delete",
@@ -145,8 +147,8 @@ public class DeletionManifestBuilderTests
new DeletionRetentionContext { TenantRetentionDays = 90 });
var nonTombstoneSteps = manifest.Steps.Where(s => s.Class != DeletionStepClass.Final).ToList();
// Without a contributions row, neither inventory step is emitted → exactly 15 cascade-table steps.
Assert.Equal(15, nonTombstoneSteps.Count);
// Without a contributions row, neither inventory step is emitted → exactly 16 cascade-table steps.
Assert.Equal(16, nonTombstoneSteps.Count);
foreach (var step in nonTombstoneSteps)
{
Assert.Equal(0, step.RowCount);
@@ -175,8 +177,8 @@ public class DeletionManifestBuilderTests
new DeletionActor { Type = "admin", Actor = "alice@example.com" },
new DeletionRetentionContext { TenantRetentionDays = 90 });
// Expect: 15 cascade + 2 inventory + 1 tombstone = 18 steps.
Assert.Equal(18, manifest.Steps.Count);
// Expect: 16 cascade + 2 inventory + 1 tombstone = 19 steps.
Assert.Equal(19, manifest.Steps.Count);
var aggregate = manifest.Steps.Single(s => s.Class == DeletionStepClass.Aggregate);
Assert.Equal(DeletionStepNames.SoftwareInventoryDecrement, aggregate.Step);
@@ -640,6 +642,7 @@ public class DeletionManifestBuilderTests
private TableEntity? _vulnerabilityReport;
private TableEntity? _deviceSnapshot;
private TableEntity? _eventSessionIndex;
private TableEntity? _sessionTimeBreakdown;
private TableEntity? _sessionInventoryContributions;
private string _sessionsIndexRowKey = "INDEX_" + SessionId;
@@ -719,6 +722,16 @@ public class DeletionManifestBuilderTests
return this;
}
public ReaderSeed WithSessionTimeBreakdown(string propValue = "tab")
{
_sessionTimeBreakdown = new TableEntity(TenantId, SessionId)
{
["Marker"] = propValue,
};
_sessionTimeBreakdown.ETag = new ETag("0xTAB");
return this;
}
public ReaderSeed WithEventTypeIndex(int n, string propValue = "eti")
{
// PK = {tenantId}_{eventType}, RK ends with _{sessionId}.
@@ -796,6 +809,8 @@ public class DeletionManifestBuilderTests
.ReturnsAsync(_deviceSnapshot);
reader.Setup(r => r.GetEntityOrNullAsync(Constants.TableNames.EventSessionIndex, TenantId, SessionId, It.IsAny<CancellationToken>()))
.ReturnsAsync(_eventSessionIndex);
reader.Setup(r => r.GetEntityOrNullAsync(Constants.TableNames.SessionTimeBreakdowns, TenantId, SessionId, It.IsAny<CancellationToken>()))
.ReturnsAsync(_sessionTimeBreakdown);
reader.Setup(r => r.GetEntityOrNullAsync(Constants.TableNames.SessionInventoryContributions, TenantId, SessionId, It.IsAny<CancellationToken>()))
.ReturnsAsync(_sessionInventoryContributions);
@@ -0,0 +1,809 @@
using AutopilotMonitor.Functions.Functions.Metrics;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// F2 PR4 (insights spec §F2): device-key normalization (junk serials), terminal-only chain
/// maintenance (upsert, cap, deletion-ref pruning), journey grouping (first-success end,
/// 30-day gap, Incomplete attempts, WG-Pending stays open), the pure daily FTR aggregation
/// core (<see cref="MaintenanceService.BuildDeviceJourneyAggregates"/>) and the persistence
/// round-trips (table-storage-serialization rule — every property must survive Store→Map).
/// </summary>
public class DeviceJourneyAndFtrTests
{
private static readonly DateTime T0 = new(2026, 7, 20, 10, 0, 0, DateTimeKind.Utc);
private const string TenantA = "aaaaaaaa-0000-0000-0000-000000000001";
private const string TenantB = "bbbbbbbb-0000-0000-0000-000000000002";
private static SessionSummary Session(
string sessionId,
SessionStatus status = SessionStatus.Succeeded,
DateTime? startedAt = null,
DateTime? completedAt = null,
int? durationSeconds = 1800,
string? serial = "PF4X1ABC",
string tenantId = TenantA,
bool wg = false,
string enrollmentType = "v1",
string? adminMarkedAction = null)
=> new()
{
TenantId = tenantId,
SessionId = sessionId,
SerialNumber = serial!,
Status = status,
StartedAt = startedAt ?? T0,
CompletedAt = completedAt ?? (startedAt ?? T0).AddSeconds(durationSeconds ?? 0),
DurationSeconds = durationSeconds,
IsPreProvisioned = wg,
EnrollmentType = enrollmentType,
AdminMarkedAction = adminMarkedAction,
Manufacturer = "Contoso",
Model = "Laptop 5",
};
private static DeviceSessionRef Ref(
string sessionId, SessionStatus status, DateTime startedAt, DateTime? completedAt = null)
=> new()
{
SessionId = sessionId,
Status = status.ToString(),
StartedAt = startedAt,
CompletedAt = completedAt ?? startedAt.AddMinutes(30),
};
// ── serial normalization / junk exclusion ───────────────────────────────
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("abc")] // shorter than 4 chars
[InlineData("0")]
[InlineData("None")]
[InlineData("INVALID")]
[InlineData("Unknown")] // the agent's WMI-failure sentinel (audit Q5)
[InlineData("unknown")]
[InlineData("System Serial Number")]
[InlineData("TO BE FILLED BY O.E.M.")]
[InlineData("Default String")]
[InlineData(" Default string ")] // junk match happens after trim + case-fold
public void NormalizeSerial_JunkAndPlaceholders_YieldNull(string? raw)
{
Assert.Null(DeviceJourneyCalculator.NormalizeSerial(raw));
}
[Theory]
[InlineData("PF4X1ABC", "pf4x1abc")]
[InlineData(" PF4X1ABC ", "pf4x1abc")] // trim
[InlineData("pf4x1abc", "pf4x1abc")]
[InlineData("1234", "1234")] // exactly 4 chars is a valid identity
public void NormalizeSerial_RealSerials_TrimAndCaseFold(string raw, string expected)
{
Assert.Equal(expected, DeviceJourneyCalculator.NormalizeSerial(raw));
}
// ── terminal set / ref building ─────────────────────────────────────────
[Theory]
[InlineData(SessionStatus.Succeeded, true)]
[InlineData(SessionStatus.Failed, true)]
[InlineData(SessionStatus.Incomplete, true)]
[InlineData(SessionStatus.InProgress, false)]
[InlineData(SessionStatus.Pending, false)] // WG part 1 sealed — an OPEN session
[InlineData(SessionStatus.Stalled, false)]
[InlineData(SessionStatus.AwaitingUser, false)]
[InlineData(SessionStatus.Unknown, false)]
public void IsTerminal_MatchesTheF2TerminalSet(SessionStatus status, bool expected)
{
Assert.Equal(expected, DeviceJourneyCalculator.IsTerminal(status));
}
[Fact]
public void BuildSessionRef_MapsAllFields_DurationIsAuthoritativeDurationSeconds()
{
// CompletedAt deliberately diverges from StartedAt + DurationSeconds (25 % of terminal
// sessions in production) — the ref must carry DurationSeconds verbatim.
var session = Session("s-1", startedAt: T0, completedAt: T0.AddHours(5), durationSeconds: 1800,
wg: true, enrollmentType: "v1", adminMarkedAction: "Succeeded");
var reference = DeviceJourneyCalculator.BuildSessionRef(session)!;
Assert.Equal("s-1", reference.SessionId);
Assert.Equal(T0, reference.StartedAt);
Assert.Equal(T0.AddHours(5), reference.CompletedAt);
Assert.Equal("Succeeded", reference.Status);
Assert.Equal("v1", reference.EnrollmentType);
Assert.True(reference.IsPreProvisioned);
Assert.Equal(1800, reference.DurationSeconds);
Assert.True(reference.AdminMarked);
}
[Fact]
public void BuildSessionRef_WhiteGlovePending_YieldsNull_NeverAnAttempt()
{
var pending = Session("s-1", status: SessionStatus.Pending, wg: true, durationSeconds: null);
Assert.Null(DeviceJourneyCalculator.BuildSessionRef(pending));
}
[Fact]
public void BuildSessionRef_Incomplete_KeepsNullDuration()
{
var incomplete = Session("s-1", status: SessionStatus.Incomplete, durationSeconds: null);
var reference = DeviceJourneyCalculator.BuildSessionRef(incomplete)!;
Assert.Equal("Incomplete", reference.Status);
Assert.Null(reference.DurationSeconds);
Assert.False(reference.AdminMarked);
}
// ── chain maintenance ───────────────────────────────────────────────────
[Fact]
public void MergeChain_UpsertsBySessionId_ReclassificationReplacesTheEntry()
{
var existing = new List<DeviceSessionRef> { Ref("s-1", SessionStatus.Incomplete, T0) };
var update = Ref("s-1", SessionStatus.Succeeded, T0);
var merged = DeviceJourneyCalculator.MergeChain(existing, new[] { update });
var entry = Assert.Single(merged);
Assert.Equal("Succeeded", entry.Status);
}
[Fact]
public void MergeChain_OrdersByStartedAt_TiebreakSessionId_AndCapsToMostRecent()
{
var refs = new List<DeviceSessionRef>();
for (var i = 0; i < 25; i++)
refs.Add(Ref($"s-{i:D2}", SessionStatus.Failed, T0.AddDays(i)));
// Same StartedAt as s-24 — the ordinal sessionId tiebreak must keep the order stable.
refs.Add(Ref("s-00-twin", SessionStatus.Failed, T0.AddDays(24)));
var merged = DeviceJourneyCalculator.MergeChain(null, refs);
Assert.Equal(20, merged.Count); // cap 20 most recent
Assert.Equal("s-06", merged[0].SessionId); // s-00..s-05 dropped (oldest)
Assert.Equal(T0.AddDays(24), merged[^1].StartedAt);
Assert.Equal("s-24", merged[^1].SessionId); // "s-00-twin" < "s-24" ordinal → sorts first
Assert.Equal("s-00-twin", merged[^2].SessionId);
}
[Fact]
public void MergeChain_DropsNonTerminalRefs_Defensively()
{
var corrupt = new List<DeviceSessionRef>
{
Ref("s-1", SessionStatus.Succeeded, T0),
new() { SessionId = "s-2", Status = "Pending", StartedAt = T0.AddDays(1) },
new() { SessionId = "s-3", Status = "garbage", StartedAt = T0.AddDays(2) },
};
var merged = DeviceJourneyCalculator.MergeChain(corrupt, Array.Empty<DeviceSessionRef>());
Assert.Equal("s-1", Assert.Single(merged).SessionId);
}
[Fact]
public void RemoveSessionRefs_DropsDeletedSessions_EmptyChainSignalsRowDeletion()
{
var chain = new List<DeviceSessionRef>
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Succeeded, T0.AddDays(1)),
};
var partial = DeviceJourneyCalculator.RemoveSessionRefs(
chain, new HashSet<string>(StringComparer.Ordinal) { "s-1" });
Assert.Equal("s-2", Assert.Single(partial).SessionId);
var empty = DeviceJourneyCalculator.RemoveSessionRefs(
chain, new HashSet<string>(StringComparer.Ordinal) { "s-1", "s-2" });
Assert.Empty(empty);
}
// ── journey grouping ────────────────────────────────────────────────────
[Fact]
public void GroupJourneys_SingleSuccess_IsOneCompletedFirstTimeRightJourney()
{
var journeys = DeviceJourneyCalculator.GroupJourneys(
new[] { Ref("s-1", SessionStatus.Succeeded, T0) });
var journey = Assert.Single(journeys);
Assert.True(journey.Completed);
Assert.Single(journey.Attempts);
Assert.Equal("s-1", journey.CompletingRef!.SessionId);
}
[Fact]
public void GroupJourneys_FailAndIncompleteAreAttempts_JourneyEndsWithFirstSuccess()
{
var journeys = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Incomplete, T0.AddHours(2)),
Ref("s-3", SessionStatus.Succeeded, T0.AddHours(4)),
});
var journey = Assert.Single(journeys);
Assert.True(journey.Completed);
Assert.Equal(3, journey.Attempts.Count); // Incomplete is a non-successful attempt
}
[Fact]
public void GroupJourneys_SessionAfterSuccess_StartsANewJourney_Redeployment()
{
var journeys = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Succeeded, T0),
Ref("s-2", SessionStatus.Failed, T0.AddDays(2)),
});
Assert.Equal(2, journeys.Count);
Assert.True(journeys[0].Completed);
Assert.False(journeys[1].Completed); // open — no failed verdict, just not done yet
Assert.Null(journeys[1].CompletingRef);
}
[Fact]
public void GroupJourneys_FailuresOnly_JourneyStaysOpen_NeverCompleted()
{
var journeys = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Failed, T0.AddDays(1)),
});
var journey = Assert.Single(journeys);
Assert.False(journey.Completed);
Assert.Equal(2, journey.Attempts.Count);
}
[Fact]
public void GroupJourneys_GapOver30Days_StartsANewJourney_AbandonedStaysUncompleted()
{
var journeys = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0, completedAt: T0.AddHours(1)),
Ref("s-2", SessionStatus.Failed, T0.AddHours(1).AddDays(31)),
Ref("s-3", SessionStatus.Succeeded, T0.AddHours(1).AddDays(31).AddHours(3)),
});
Assert.Equal(2, journeys.Count);
Assert.False(journeys[0].Completed); // shelved/abandoned — never counts for FTR
Assert.Single(journeys[0].Attempts);
Assert.True(journeys[1].Completed);
Assert.Equal(2, journeys[1].Attempts.Count);
}
[Fact]
public void GroupJourneys_GapBoundary_Exactly30DaysStaysOneJourney_StrictlyMoreSplits()
{
var prevEnd = T0.AddHours(1);
var exactly30 = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0, completedAt: prevEnd),
Ref("s-2", SessionStatus.Failed, prevEnd.AddDays(30)),
});
Assert.Single(exactly30); // spec: "> 30 days" — the boundary itself does not split
var over30 = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0, completedAt: prevEnd),
Ref("s-2", SessionStatus.Failed, prevEnd.AddDays(30).AddHours(1)),
});
Assert.Equal(2, over30.Count);
}
[Fact]
public void GroupJourneys_GapMeasuredFromCompletedAt_NotStartedAt()
{
// Previous attempt STARTED 40 days before the next one but went terminal only 10 days
// before it — the gap is 10 days, one journey. StartedAt-based measuring would split.
var journeys = DeviceJourneyCalculator.GroupJourneys(new[]
{
Ref("s-1", SessionStatus.Failed, T0.AddDays(-40), completedAt: T0.AddDays(-10)),
Ref("s-2", SessionStatus.Succeeded, T0),
});
var journey = Assert.Single(journeys);
Assert.Equal(2, journey.Attempts.Count);
}
[Fact]
public void GroupJourneys_WhiteGloveIsOneSessionRow_OneAttempt_NoStitching()
{
// WG part 1 + part 2 share one session row (second-sweep finding): a completed WG
// enrollment is exactly one ref and one first-time-right attempt.
var wg = Session("s-1", wg: true, durationSeconds: 5400, completedAt: T0.AddDays(6));
var reference = DeviceJourneyCalculator.BuildSessionRef(wg)!;
var journeys = DeviceJourneyCalculator.GroupJourneys(new[] { reference });
var journey = Assert.Single(journeys);
Assert.True(journey.Completed);
Assert.Single(journey.Attempts);
Assert.Equal(5400, journey.Attempts[0].DurationSeconds); // pause-free combined duration, verbatim
}
[Fact]
public void Derive_CurrentJourneyAttempts_IsTheLastJourneys_OpenOrCompleted()
{
var (journeyCount, currentAttempts) = DeviceJourneyCalculator.Derive(new[]
{
Ref("s-1", SessionStatus.Succeeded, T0),
Ref("s-2", SessionStatus.Failed, T0.AddDays(1)),
Ref("s-3", SessionStatus.Failed, T0.AddDays(2)),
});
Assert.Equal(2, journeyCount);
Assert.Equal(2, currentAttempts); // the open redeployment journey has 2 attempts so far
Assert.Equal((0, 0), DeviceJourneyCalculator.Derive(Array.Empty<DeviceSessionRef>()));
}
// ── history row assembly (display precedence) ───────────────────────────
[Fact]
public void BuildDeviceHistoryRow_DisplayFields_FollowTheNewestChainEntry()
{
var newest = Session("s-2", serial: " PF4X1ABC ", startedAt: T0.AddDays(1));
var chain = new List<DeviceSessionRef>
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Succeeded, T0.AddDays(1)),
};
var row = TableStorageService.BuildDeviceHistoryRow(TenantA, "pf4x1abc", chain, existing: null, newest);
Assert.Equal("PF4X1ABC", row.SerialNumber); // trimmed original casing
Assert.Equal("Contoso", row.Manufacturer);
Assert.Equal("Laptop 5", row.Model);
Assert.Equal(2, row.CurrentJourneyAttempts);
Assert.Equal(1, row.JourneyCount);
Assert.Equal(DeviceJourneyCalculator.CurrentVersion, row.JourneyVersion);
}
[Fact]
public void BuildDeviceHistoryRow_BackfillingAnOlderSession_KeepsExistingDisplayFields()
{
var older = Session("s-1", serial: "pf4x1abc", startedAt: T0);
older.Model = "Laptop 4"; // stale model from the older enrollment
var chain = new List<DeviceSessionRef>
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Succeeded, T0.AddDays(1)),
};
var existing = new DeviceHistory
{
TenantId = TenantA, SerialKey = "pf4x1abc",
SerialNumber = "PF4X1ABC", Manufacturer = "Contoso", Model = "Laptop 5",
};
var row = TableStorageService.BuildDeviceHistoryRow(TenantA, "pf4x1abc", chain, existing, older);
Assert.Equal("PF4X1ABC", row.SerialNumber); // not regressed to the older session's data
Assert.Equal("Laptop 5", row.Model);
}
// ── persistence round-trips ─────────────────────────────────────────────
[Fact]
public void DeviceHistoryEntity_RoundTripsAllFields_IncludingChainJson()
{
var history = new DeviceHistory
{
TenantId = TenantA,
SerialKey = "pf4x1abc",
SerialNumber = "PF4X1ABC",
Manufacturer = "Contoso",
Model = "Laptop 5",
CurrentJourneyAttempts = 2,
JourneyCount = 3,
JourneyVersion = 1,
LastUpdated = T0,
Chain = new List<DeviceSessionRef>
{
new()
{
SessionId = "s-1", StartedAt = T0, CompletedAt = T0.AddHours(5),
Status = "Succeeded", EnrollmentType = "v2", IsPreProvisioned = true,
DurationSeconds = 1800, AdminMarked = true,
},
new()
{
SessionId = "s-2", StartedAt = T0.AddDays(1), CompletedAt = null,
Status = "Incomplete", EnrollmentType = "v1", DurationSeconds = null,
},
},
};
var mapped = TableStorageService.MapToDeviceHistory(
TableStorageService.BuildDeviceHistoryEntity(history));
Assert.Equal(TenantA, mapped.TenantId);
Assert.Equal("pf4x1abc", mapped.SerialKey);
Assert.Equal("PF4X1ABC", mapped.SerialNumber);
Assert.Equal("Contoso", mapped.Manufacturer);
Assert.Equal("Laptop 5", mapped.Model);
Assert.Equal(2, mapped.CurrentJourneyAttempts);
Assert.Equal(3, mapped.JourneyCount);
Assert.Equal(1, mapped.JourneyVersion);
Assert.Equal(T0, mapped.LastUpdated);
Assert.Equal(2, mapped.Chain.Count);
var first = mapped.Chain[0];
Assert.Equal("s-1", first.SessionId);
Assert.Equal(T0, first.StartedAt);
Assert.Equal(T0.AddHours(5), first.CompletedAt);
Assert.Equal("Succeeded", first.Status);
Assert.Equal("v2", first.EnrollmentType);
Assert.True(first.IsPreProvisioned);
Assert.Equal(1800, first.DurationSeconds);
Assert.True(first.AdminMarked);
var second = mapped.Chain[1];
Assert.Null(second.CompletedAt); // Incomplete: no terminal timestamp requirement
Assert.Null(second.DurationSeconds); // Incomplete: deliberately no duration — stays null
Assert.False(second.AdminMarked);
}
[Fact]
public void DeviceHistoryEntity_RowKeyEncoding_MakesForbiddenKeyCharsSafe_AndRoundTrips()
{
// '/', '\', '#', '?' are forbidden in Table Storage keys — a defensive encoding keeps a
// pathological serial writable, and the mapper restores the exact normalized serial.
var weird = "ab/cd\\ef#1?x";
var history = new DeviceHistory { TenantId = TenantA, SerialKey = weird };
var entity = TableStorageService.BuildDeviceHistoryEntity(history);
Assert.All(new[] { "/", "\\", "#", "?" }, c => Assert.DoesNotContain(c, entity.RowKey));
Assert.Equal(weird, TableStorageService.MapToDeviceHistory(entity).SerialKey);
// Ordinary serials stay human-readable.
Assert.Equal("pf4x1abc", TableStorageService.SerialRowKey("pf4x1abc"));
}
[Fact]
public void DeviceJourneyAggregateEntity_RoundTripsAllFields_RowKeyIsTheDate()
{
var aggregate = new DeviceJourneyDailyAggregate
{
TenantId = TenantA,
Date = "2026-07-20",
JourneyVersion = 1,
CompletedJourneyCount = 23,
FirstTimeRightCount = 19,
ExcludedSessionCount = 2,
ComputedAt = T0,
AttemptHistogram = new List<DeviceJourneyAttemptBucket>
{
new() { Attempts = 1, JourneyCount = 19 },
new() { Attempts = 2, JourneyCount = 3 },
new() { Attempts = 4, JourneyCount = 1 },
},
};
var entity = TableStorageService.BuildDeviceJourneyAggregateEntity(aggregate);
Assert.Equal("2026-07-20", entity.RowKey);
var mapped = TableStorageService.MapToDeviceJourneyAggregate(entity);
Assert.Equal(TenantA, mapped.TenantId);
Assert.Equal("2026-07-20", mapped.Date);
Assert.Equal(1, mapped.JourneyVersion);
Assert.Equal(23, mapped.CompletedJourneyCount);
Assert.Equal(19, mapped.FirstTimeRightCount);
Assert.Equal(2, mapped.ExcludedSessionCount);
Assert.Equal(T0, mapped.ComputedAt);
Assert.Equal(3, mapped.AttemptHistogram.Count);
Assert.Equal(1, mapped.AttemptHistogram[0].Attempts);
Assert.Equal(19, mapped.AttemptHistogram[0].JourneyCount);
Assert.Equal(4, mapped.AttemptHistogram[2].Attempts);
}
// ── daily FTR aggregation core ──────────────────────────────────────────
private static readonly DateTime WindowStart = T0.Date.AddDays(-30);
private static readonly DateTime WindowEnd = T0.Date.AddDays(1);
private static DeviceHistory History(string tenantId, params DeviceSessionRef[] chain)
=> new() { TenantId = tenantId, SerialKey = Guid.NewGuid().ToString("N"), Chain = chain.ToList() };
[Fact]
public void FtrAggregates_BucketByCompletingSessionsStartDate_AndMirrorGlobal()
{
var histories = new List<DeviceHistory>
{
// Tenant A device 1: fail → success (2 attempts, completes on T0 date).
History(TenantA,
Ref("a1-1", SessionStatus.Failed, T0.AddDays(-1)),
Ref("a1-2", SessionStatus.Succeeded, T0)),
// Tenant A device 2: first-time right on the same date.
History(TenantA, Ref("a2-1", SessionStatus.Succeeded, T0.AddHours(2))),
// Tenant B device: first-time right on a different date.
History(TenantB, Ref("b1-1", SessionStatus.Succeeded, T0.AddDays(-3))),
};
var rows = MaintenanceService.BuildDeviceJourneyAggregates(
histories, Array.Empty<SessionSummary>(), WindowStart, WindowEnd, T0);
// Tenant A (1 date) + Tenant B (1 date) + global (2 dates) = 4 rows.
Assert.Equal(4, rows.Count);
var tenantARow = rows.Single(r => r.TenantId == TenantA);
Assert.Equal(T0.ToString("yyyy-MM-dd"), tenantARow.Date);
Assert.Equal(2, tenantARow.CompletedJourneyCount);
Assert.Equal(1, tenantARow.FirstTimeRightCount);
Assert.Equal(DeviceJourneyCalculator.CurrentVersion, tenantARow.JourneyVersion);
Assert.Equal(2, tenantARow.AttemptHistogram.Count);
Assert.Equal(1, tenantARow.AttemptHistogram[0].Attempts);
Assert.Equal(1, tenantARow.AttemptHistogram[0].JourneyCount);
Assert.Equal(2, tenantARow.AttemptHistogram[1].Attempts);
var globalOnT0 = rows.Single(r => r.TenantId == "global" && r.Date == T0.ToString("yyyy-MM-dd"));
Assert.Equal(2, globalOnT0.CompletedJourneyCount); // tenant B's journey sits on its own date
}
[Fact]
public void FtrAggregates_OpenAndAbandonedJourneys_NeverCount()
{
var histories = new List<DeviceHistory>
{
// Open journey: failures only (e.g. a WG device still waiting for its user session
// has, at most, prior failed attempts in the chain — the Pending row itself never
// becomes a ref).
History(TenantA, Ref("a1-1", SessionStatus.Failed, T0)),
// Abandoned journey (31d gap, no success) followed by a completed journey.
History(TenantA,
Ref("a2-1", SessionStatus.Failed, T0.AddDays(-20), completedAt: T0.AddDays(-20).AddHours(1)),
Ref("a2-2", SessionStatus.Succeeded, T0.AddDays(11).AddHours(2))),
};
var rows = MaintenanceService.BuildDeviceJourneyAggregates(
histories, Array.Empty<SessionSummary>(), WindowStart, T0.Date.AddDays(12), T0);
// Only the completed journey produced rows: tenant + global on one date.
Assert.Equal(2, rows.Count);
var tenantRow = rows.Single(r => r.TenantId == TenantA);
Assert.Equal(1, tenantRow.CompletedJourneyCount);
Assert.Equal(1, tenantRow.FirstTimeRightCount); // the abandoned attempt belongs to a DIFFERENT journey
Assert.Equal(T0.AddDays(11).ToString("yyyy-MM-dd"), tenantRow.Date);
}
[Fact]
public void FtrAggregates_JourneysCompletedOutsideTheWindow_AreSkipped()
{
var histories = new List<DeviceHistory>
{
History(TenantA, Ref("a1-1", SessionStatus.Succeeded, WindowStart.AddDays(-1))),
};
var rows = MaintenanceService.BuildDeviceJourneyAggregates(
histories, Array.Empty<SessionSummary>(), WindowStart, WindowEnd, T0);
Assert.Empty(rows);
}
[Fact]
public void FtrAggregates_JunkSerialExclusions_ProduceDisclosureRows()
{
var excluded = new List<SessionSummary>
{
Session("x-1", serial: "Unknown", startedAt: T0),
Session("x-2", serial: "Unknown", startedAt: T0.AddHours(1)),
Session("x-3", serial: "Unknown", startedAt: WindowStart.AddDays(-2)), // outside window — ignored
};
var rows = MaintenanceService.BuildDeviceJourneyAggregates(
Array.Empty<DeviceHistory>(), excluded, WindowStart, WindowEnd, T0);
Assert.Equal(2, rows.Count); // tenant + global disclosure rows, no journey claim
var tenantRow = rows.Single(r => r.TenantId == TenantA);
Assert.Equal(0, tenantRow.CompletedJourneyCount);
Assert.Equal(0, tenantRow.FirstTimeRightCount);
Assert.Empty(tenantRow.AttemptHistogram);
Assert.Equal(2, tenantRow.ExcludedSessionCount);
Assert.Equal(2, rows.Single(r => r.TenantId == "global").ExcludedSessionCount);
}
[Fact]
public void FtrAggregates_TwoCompletedJourneysOfOneDevice_SameDay_BothCount()
{
// Redeploy on the same day: success closes journey 1, the next success closes journey 2.
var histories = new List<DeviceHistory>
{
History(TenantA,
Ref("a1-1", SessionStatus.Succeeded, T0),
Ref("a1-2", SessionStatus.Succeeded, T0.AddHours(3))),
};
var rows = MaintenanceService.BuildDeviceJourneyAggregates(
histories, Array.Empty<SessionSummary>(), WindowStart, WindowEnd, T0);
var tenantRow = rows.Single(r => r.TenantId == TenantA);
Assert.Equal(2, tenantRow.CompletedJourneyCount);
Assert.Equal(2, tenantRow.FirstTimeRightCount);
}
// ── attempt number (session banner, PR5) ────────────────────────────────
[Fact]
public void ComputeAttemptNumber_TerminalSessionInChain_ReturnsPositionWithinItsJourney()
{
var chain = new[]
{
Ref("s-1", SessionStatus.Failed, T0),
Ref("s-2", SessionStatus.Succeeded, T0.AddHours(2)),
Ref("s-3", SessionStatus.Failed, T0.AddDays(5)), // redeployment → new journey
};
Assert.Equal(1, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-1", T0));
Assert.Equal(2, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-2", T0.AddHours(2)));
Assert.Equal(1, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-3", T0.AddDays(5)));
}
[Fact]
public void ComputeAttemptNumber_LiveSession_ContinuesTheOpenJourney()
{
var chain = new[] { Ref("s-1", SessionStatus.Failed, T0) };
// s-2 is still InProgress — not a chain ref; the virtual-attempt rule places it.
Assert.Equal(2, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-2", T0.AddDays(1)));
}
[Fact]
public void ComputeAttemptNumber_LiveSession_AfterCompletedJourney_IsAttemptOne()
{
var chain = new[] { Ref("s-1", SessionStatus.Succeeded, T0) };
Assert.Equal(1, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-2", T0.AddDays(2)));
}
[Fact]
public void ComputeAttemptNumber_LiveSession_AfterThirtyDayGap_IsAttemptOne()
{
var prevEnd = T0.AddHours(1);
var chain = new[] { Ref("s-1", SessionStatus.Failed, T0, completedAt: prevEnd) };
Assert.Equal(1, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-2", prevEnd.AddDays(31)));
// Inside the gap boundary the open journey continues instead.
Assert.Equal(2, DeviceJourneyCalculator.ComputeAttemptNumber(chain, "s-3", prevEnd.AddDays(29)));
}
[Fact]
public void ComputeAttemptNumber_EmptyChain_YieldsNull_NoGuessedPosition()
{
Assert.Null(DeviceJourneyCalculator.ComputeAttemptNumber(
Array.Empty<DeviceSessionRef>(), "s-1", T0));
}
// ── fleet response helpers (PR5) ────────────────────────────────────────
[Fact]
public void SumAggregates_SumsAdditiveCounts_AndMergesHistograms()
{
var daily = new List<DeviceJourneyDailyAggregate>
{
new()
{
CompletedJourneyCount = 2, FirstTimeRightCount = 1, ExcludedSessionCount = 1,
AttemptHistogram = new List<DeviceJourneyAttemptBucket>
{
new() { Attempts = 1, JourneyCount = 1 },
new() { Attempts = 2, JourneyCount = 1 },
},
},
new()
{
CompletedJourneyCount = 1, FirstTimeRightCount = 1, ExcludedSessionCount = 0,
AttemptHistogram = new List<DeviceJourneyAttemptBucket> { new() { Attempts = 1, JourneyCount = 1 } },
},
};
var totals = DeviceJourneyMetricsResponse.SumAggregates(daily);
Assert.Equal(3, totals.CompletedJourneys);
Assert.Equal(2, totals.FirstTimeRight);
Assert.Equal(66.7, totals.FtrRatePct);
Assert.Equal(1, totals.ExcludedSessions);
Assert.Equal(2, totals.AttemptHistogram.Count);
Assert.Equal(2, totals.AttemptHistogram.Single(b => b.Attempts == 1).JourneyCount);
Assert.Equal(1, totals.AttemptHistogram.Single(b => b.Attempts == 2).JourneyCount);
}
[Fact]
public void SumAggregates_NoCompletedJourneys_RateIsNull_NeverZero()
{
var totals = DeviceJourneyMetricsResponse.SumAggregates(new List<DeviceJourneyDailyAggregate>
{
new() { CompletedJourneyCount = 0, FirstTimeRightCount = 0, ExcludedSessionCount = 3 },
});
Assert.Equal(0, totals.CompletedJourneys);
Assert.Null(totals.FtrRatePct);
Assert.Equal(3, totals.ExcludedSessions);
}
[Theory]
[InlineData(null, 30)] // default window
[InlineData("garbage", 30)]
[InlineData("7", 7)]
[InlineData("0", 1)] // clamped low
[InlineData("999", 180)] // clamped to aggregate retention
public void ClampDays_DefaultsAndClamps(string? raw, int expected)
{
Assert.Equal(expected, DeviceJourneyMetricsResponse.ClampDays(raw));
}
[Fact]
public void InclusiveWindowStart_YieldsExactlyNCalendarDays()
{
// Codex review: both range ends are inclusive, so "today - days" returned N+1 keys —
// days=1 summed yesterday AND today.
var today = new DateTime(2026, 7, 27);
Assert.Equal(today, DeviceJourneyMetricsResponse.InclusiveWindowStart(today, 1));
Assert.Equal(today.AddDays(-29), DeviceJourneyMetricsResponse.InclusiveWindowStart(today, 30));
}
private static DeviceHistory RepeaterHistory(
string serial, int attempts, params DeviceSessionRef[] chain)
=> new()
{
TenantId = TenantA,
SerialKey = serial,
SerialNumber = serial.ToUpperInvariant(),
Chain = chain.ToList(),
CurrentJourneyAttempts = attempts,
JourneyCount = 1,
};
[Fact]
public void SelectRepeatDevices_FiltersGate_WindowsOnNewestRef_AndPicksNewestFailure()
{
var histories = new List<DeviceHistory>
{
// Single attempt — not a repeat device.
RepeaterHistory("clean-1", attempts: 1, Ref("c1", SessionStatus.Succeeded, T0)),
// Repeat device, active in window: newest failed ref carries the reason lookup.
RepeaterHistory("hot-1", attempts: 3,
Ref("h1", SessionStatus.Failed, T0.AddDays(-2)),
Ref("h2", SessionStatus.Failed, T0.AddDays(-1)),
Ref("h3", SessionStatus.Incomplete, T0)),
// Repeat device, but its pain is OLDER than the window — current lists only.
RepeaterHistory("old-1", attempts: 4,
Ref("o1", SessionStatus.Failed, T0.AddDays(-40))),
// Lower attempt count → sorts after hot-1.
RepeaterHistory("warm-1", attempts: 2,
Ref("w1", SessionStatus.Failed, T0.AddDays(-3)),
Ref("w2", SessionStatus.Succeeded, T0.AddDays(-2))),
};
var selected = DeviceJourneyMetricsResponse.SelectRepeatDevices(histories, T0.AddDays(-30));
Assert.Equal(2, selected.Count);
Assert.Equal("hot-1", selected[0].History.SerialKey); // attempts desc
Assert.Equal("h3", selected[0].Newest.SessionId);
Assert.Equal("h2", selected[0].NewestFailed!.SessionId); // last FAILED, not last ref
Assert.Equal("warm-1", selected[1].History.SerialKey);
}
[Fact]
public void SelectRepeatDevices_CapsAtTen_ByAttemptsThenRecency()
{
var histories = new List<DeviceHistory>();
for (var i = 0; i < 12; i++)
{
histories.Add(RepeaterHistory($"dev-{i:D2}", attempts: 2 + i,
Ref($"r-{i}", SessionStatus.Failed, T0.AddDays(-i))));
}
var selected = DeviceJourneyMetricsResponse.SelectRepeatDevices(histories, T0.AddDays(-30));
Assert.Equal(10, selected.Count);
Assert.Equal("dev-11", selected[0].History.SerialKey); // highest attempts first
Assert.DoesNotContain(selected, c => c.History.SerialKey == "dev-00");
Assert.DoesNotContain(selected, c => c.History.SerialKey == "dev-01");
}
}
@@ -50,6 +50,8 @@ public class GeoMetricsProjectionEquivalenceTests
["GeoCity"] = "Frankfurt",
["GeoLoc"] = "50.11,8.68",
["IsPreProvisioned"] = false,
["AvgApiLatencyMs"] = 217.4,
["ApiRequestCount"] = 270,
// Noise the geo aggregation never reads.
["SerialNumber"] = "SN-FULL",
["Manufacturer"] = "Contoso",
@@ -76,6 +78,8 @@ public class GeoMetricsProjectionEquivalenceTests
Assert.Equal(fromFull.GeoRegion, fromProjected.GeoRegion);
Assert.Equal(fromFull.GeoCity, fromProjected.GeoCity);
Assert.Equal(fromFull.GeoLoc, fromProjected.GeoLoc);
Assert.Equal(fromFull.AvgApiLatencyMs, fromProjected.AvgApiLatencyMs);
Assert.Equal(fromFull.ApiRequestCount, fromProjected.ApiRequestCount);
// The grouping key must be identical too — it is composed from the Geo* fields.
Assert.Equal(
@@ -0,0 +1,124 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Functions.Functions.Metrics;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the API-latency aggregation of
/// <see cref="GetGeographicMetricsFunction.ComputeGeographicMetrics"/> (2026-07-28): both the
/// per-location and the global figure are REQUEST-WEIGHTED averages —
/// Σ(sessionAvg·requests)/Σ(requests) — not averages of session averages, so a long session
/// with many uploads counts proportionally. Sessions without latency data (agents predating
/// <c>AvgApiLatencyMs</c>) drop out of the weighting entirely.
/// </summary>
public class GeographicMetricsApiLatencyTests
{
private static SessionSummary S(string country, string city, double? avgLatencyMs, int? requestCount)
=> new()
{
TenantId = "00000000-0000-0000-0000-000000000fe0",
SessionId = Guid.NewGuid().ToString(),
Status = SessionStatus.Succeeded,
StartedAt = DateTime.UtcNow.AddHours(-2),
GeoCountry = country,
GeoRegion = country,
GeoCity = city,
GeoLoc = "0,0",
AvgApiLatencyMs = avgLatencyMs,
ApiRequestCount = requestCount,
};
[Fact]
public void LocationLatency_IsRequestWeighted_NotAverageOfAverages()
{
var sessions = new List<SessionSummary>
{
S("ID", "Jakarta", 1000, 300),
S("ID", "Jakarta", 500, 100),
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "country");
var loc = Assert.Single(result.Locations);
// (1000·300 + 500·100) / 400 = 875 — an unweighted mean would say 750.
Assert.Equal(875d, loc.AvgApiLatencyMs);
Assert.Equal(2, loc.ApiLatencySessionCount);
}
[Fact]
public void SessionsWithoutLatencyData_DropOutOfWeighting_ButStayInSessionCount()
{
var sessions = new List<SessionSummary>
{
S("DE", "Frankfurt", 200, 100),
S("DE", "Frankfurt", avgLatencyMs: null, requestCount: null), // pre-feature agent
S("DE", "Frankfurt", avgLatencyMs: 0, requestCount: 50), // zero = no signal
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "country");
var loc = Assert.Single(result.Locations);
Assert.Equal(3, loc.SessionCount);
Assert.Equal(1, loc.ApiLatencySessionCount);
Assert.Equal(200d, loc.AvgApiLatencyMs);
}
[Fact]
public void NoLatencyDataAnywhere_YieldsZeroes_NotDivisionByZero()
{
var sessions = new List<SessionSummary> { S("DE", "Frankfurt", null, null) };
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "country");
var loc = Assert.Single(result.Locations);
Assert.Equal(0d, loc.AvgApiLatencyMs);
Assert.Equal(0, loc.ApiLatencySessionCount);
Assert.Equal(0d, loc.ApiLatencyVsGlobalPct);
Assert.Equal(0d, result.GlobalAverages.AvgApiLatencyMs);
}
[Fact]
public void GlobalLatency_WeightsAcrossLocations_AndVsGlobalPctCompares()
{
var sessions = new List<SessionSummary>
{
S("DE", "Frankfurt", 200, 100),
S("ID", "Jakarta", 800, 100),
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "country");
// Equal request weights → global = (200 + 800) / 2 = 500.
Assert.Equal(500d, result.GlobalAverages.AvgApiLatencyMs);
var de = result.Locations.Single(l => l.Country == "DE");
var id = result.Locations.Single(l => l.Country == "ID");
Assert.Equal(-60d, de.ApiLatencyVsGlobalPct); // 200 vs 500
Assert.Equal(60d, id.ApiLatencyVsGlobalPct); // 800 vs 500
}
[Fact]
public void LocationWithoutData_KeepsZeroPct_EvenWhenGlobalHasData()
{
var sessions = new List<SessionSummary>
{
S("DE", "Frankfurt", 200, 100),
S("FR", "Paris", null, null),
};
var result = GetGeographicMetricsFunction.ComputeGeographicMetrics(
sessions, new List<AppInstallSummary>(), "country");
var fr = result.Locations.Single(l => l.Country == "FR");
Assert.Equal(0d, fr.AvgApiLatencyMs);
Assert.Equal(0d, fr.ApiLatencyVsGlobalPct); // no data ≠ "0 ms fast"
}
}
@@ -0,0 +1,189 @@
using System.Text.Json;
using AutopilotMonitor.Functions.Functions.Metrics;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Guards the cross-tenant redaction on GET metrics/ime-versions.
///
/// The route is <c>MemberRead</c> (EndpointAccessPolicyCatalog) and the archive itself is a
/// single global partition — every tenant member reads the SAME platform-wide rows. The only
/// thing standing between an ordinary tenant user and another tenant's identifiers is the
/// projection in <see cref="GetImeVersionHistoryFunction.BuildResponsePayload"/>. That made it
/// worth pinning: the redaction had no test at all, and a boundary nobody asserts is one a
/// later refactor can open without a single failure.
///
/// The JSON tests are the ones that matter — a property dropped from an anonymous type is only
/// a real redaction once it is also absent from the wire, and the wire is what the customer's
/// MCP client receives. camelCase mirrors Program.cs's PropertyNamingPolicy.
/// </summary>
public class GetImeVersionHistoryRedactionTests
{
private static readonly DateTime First = new(2026, 7, 8, 3, 18, 13, DateTimeKind.Utc);
private static readonly DateTime Last = new(2026, 7, 27, 16, 59, 34, DateTimeKind.Utc);
private static readonly JsonSerializerOptions CamelCase =
new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase };
// =========================================================================
// Non-global caller (ordinary tenant member) — the redaction
// =========================================================================
[Fact]
public void NonGlobalCaller_DropsForeignTenantAndSessionIds()
{
var json = SerializeFor(hasGlobalScope: false, MakeEntry());
Assert.DoesNotContain("firstSeenTenantId", json, StringComparison.OrdinalIgnoreCase);
Assert.DoesNotContain("firstSeenSessionId", json, StringComparison.OrdinalIgnoreCase);
}
/// <summary>
/// The values, not just the property names: a rename of the model property must not let the
/// identifier survive under a different key. Asserting on the GUIDs catches that.
/// </summary>
[Fact]
public void NonGlobalCaller_ForeignIdentifierValuesAppearNowhereInThePayload()
{
var json = SerializeFor(hasGlobalScope: false, MakeEntry());
Assert.DoesNotContain("57f34dd1-6a42-47f9-9bc0-b921fa6caa30", json, StringComparison.OrdinalIgnoreCase);
Assert.DoesNotContain("baae0453-6d19-47b5-aa58-8bed5a572fc8", json, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public void NonGlobalCaller_ExposesExactlyTheFourAllowedProperties()
{
using var doc = JsonDocument.Parse(SerializeFor(hasGlobalScope: false, MakeEntry()));
var keys = doc.RootElement[0].EnumerateObject().Select(p => p.Name).OrderBy(n => n).ToArray();
Assert.Equal(new[] { "firstSeenAt", "lastSeenAt", "sessionCount", "version" }, keys);
}
/// <summary>
/// SessionCount stays on purpose — it is a platform-wide total, and the route exists so any
/// tenant member can track Microsoft's IME rollout. Pinned so that "the counts are global"
/// stays a decision someone made rather than something that quietly drifts either way.
/// </summary>
[Fact]
public void NonGlobalCaller_KeepsVersionDatesAndGlobalSessionCount()
{
using var doc = JsonDocument.Parse(SerializeFor(hasGlobalScope: false, MakeEntry()));
var row = doc.RootElement[0];
Assert.Equal("1.103.101.0", row.GetProperty("version").GetString());
Assert.Equal(First, row.GetProperty("firstSeenAt").GetDateTime());
Assert.Equal(Last, row.GetProperty("lastSeenAt").GetDateTime());
Assert.Equal(3109, row.GetProperty("sessionCount").GetInt32());
}
// =========================================================================
// Global caller (Global Admin or Global Reader) — full archive
// =========================================================================
[Fact]
public void GlobalCaller_KeepsTenantAndSessionIds()
{
using var doc = JsonDocument.Parse(SerializeFor(hasGlobalScope: true, MakeEntry()));
var row = doc.RootElement[0];
Assert.Equal("57f34dd1-6a42-47f9-9bc0-b921fa6caa30", row.GetProperty("firstSeenTenantId").GetString());
Assert.Equal("baae0453-6d19-47b5-aa58-8bed5a572fc8", row.GetProperty("firstSeenSessionId").GetString());
}
[Fact]
public void GlobalCaller_ExposesTheFullEntry()
{
using var doc = JsonDocument.Parse(SerializeFor(hasGlobalScope: true, MakeEntry()));
var keys = doc.RootElement[0].EnumerateObject().Select(p => p.Name).OrderBy(n => n).ToArray();
Assert.Equal(
new[] { "firstSeenAt", "firstSeenSessionId", "firstSeenTenantId", "lastSeenAt", "sessionCount", "version" },
keys);
}
// =========================================================================
// Shape-preserving properties of the projection
// =========================================================================
[Fact]
public void BothScopes_EmitAJsonArray_EvenForASingleRow()
{
foreach (var scope in new[] { true, false })
{
using var doc = JsonDocument.Parse(SerializeFor(scope, MakeEntry()));
Assert.Equal(JsonValueKind.Array, doc.RootElement.ValueKind);
Assert.Equal(1, doc.RootElement.GetArrayLength());
}
}
[Fact]
public void BothScopes_EmptyArchive_SerializesAsEmptyArray()
{
foreach (var scope in new[] { true, false })
{
using var doc = JsonDocument.Parse(SerializeFor(scope));
Assert.Equal(JsonValueKind.Array, doc.RootElement.ValueKind);
Assert.Equal(0, doc.RootElement.GetArrayLength());
}
}
/// <summary>
/// The repository orders by FirstSeenAt descending; the projection must not reshuffle, or a
/// redacted caller would see a different sequence than a Global Admin for the same archive.
/// </summary>
[Fact]
public void NonGlobalCaller_PreservesRowOrderAndCount()
{
var newest = MakeEntry("1.103.101.0", First);
var older = MakeEntry("1.101.111.0", First.AddDays(-48));
var oldest = MakeEntry("1.99.101.0", First.AddDays(-131));
using var doc = JsonDocument.Parse(SerializeFor(false, newest, older, oldest));
Assert.Equal(3, doc.RootElement.GetArrayLength());
Assert.Equal("1.103.101.0", doc.RootElement[0].GetProperty("version").GetString());
Assert.Equal("1.101.111.0", doc.RootElement[1].GetProperty("version").GetString());
Assert.Equal("1.99.101.0", doc.RootElement[2].GetProperty("version").GetString());
}
/// <summary>
/// BuildResponsePayload returns `object`; the branch that keeps everything hands back the
/// entries themselves. System.Text.Json resolves an `object` declared type via the runtime
/// type, which is what keeps the wire format identical to the pre-extraction inline code —
/// assert it rather than trust it, since a regression here would silently emit "{}" rows.
/// </summary>
[Fact]
public void GlobalPayload_SerializedAsObject_StillEmitsPopulatedRows()
{
var payload = GetImeVersionHistoryFunction.BuildResponsePayload([MakeEntry()], hasGlobalScope: true);
var json = JsonSerializer.Serialize(payload, CamelCase);
Assert.Contains("1.103.101.0", json, StringComparison.Ordinal);
Assert.NotEqual("[{}]", json);
}
// =========================================================================
// Helpers
// =========================================================================
private static string SerializeFor(bool hasGlobalScope, params ImeVersionHistoryEntry[] versions) =>
JsonSerializer.Serialize(
GetImeVersionHistoryFunction.BuildResponsePayload(versions, hasGlobalScope),
CamelCase);
private static ImeVersionHistoryEntry MakeEntry(string version = "1.103.101.0", DateTime? firstSeenAt = null) =>
new()
{
Version = version,
FirstSeenAt = firstSeenAt ?? First,
FirstSeenSessionId = "baae0453-6d19-47b5-aa58-8bed5a572fc8",
FirstSeenTenantId = "57f34dd1-6a42-47f9-9bc0-b921fa6caa30",
LastSeenAt = Last,
SessionCount = 3109,
};
}
@@ -135,8 +135,64 @@ public class HardwareRejectionNotificationTrackerDedupTests
Assert.Equal(2, harness.Store.Count);
}
// ── F3 rule regression: one bell per episode, keyspaces stay independent ──────
[Fact]
public async Task RuleRegression_SecondRegister_SameRule_ReturnsFalse_OneBellPerEpisode()
{
var harness = new Harness();
var alert = new AutopilotMonitor.Shared.Models.RuleRegressionAlert
{
TenantId = TenantId, RuleId = "ANALYZE-NET-001",
FirstNotifiedAt = DateTime.UtcNow, LastEvaluatedAt = DateTime.UtcNow,
};
Assert.True(await harness.Sut.TryRegisterRuleRegressionAsync(TenantId, alert));
Assert.False(await harness.Sut.TryRegisterRuleRegressionAsync(TenantId, alert));
var entity = Assert.Single(harness.Store.Values);
Assert.Equal("ruleregression|analyze-net-001", entity.RowKey);
Assert.NotNull(entity.GetDateTime("FirstNotifiedAt")); // the retention sweep depends on it
}
[Fact]
public async Task RuleRegression_DeleteReArms_AndDoesNotTouchOtherKeyspaces()
{
var harness = new Harness();
var alert = new AutopilotMonitor.Shared.Models.RuleRegressionAlert
{
TenantId = TenantId, RuleId = "ANALYZE-NET-001",
FirstNotifiedAt = DateTime.UtcNow, LastEvaluatedAt = DateTime.UtcNow,
};
Assert.True(await harness.Sut.TryRegisterRuleRegressionAsync(TenantId, alert));
Assert.True(await harness.Sut.TryRegisterFirstTpmPssNotificationAsync(TenantId, Serial));
await harness.Sut.DeleteRuleRegressionAsync(TenantId, "ANALYZE-NET-001");
// The episode re-arms; the unrelated tpmpss row survives.
Assert.Equal("tpmpss|s4sq8685", Assert.Single(harness.Store.Values).RowKey);
Assert.True(await harness.Sut.TryRegisterRuleRegressionAsync(TenantId, alert));
}
// ── Retention sweep ───────────────────────────────────────────────────────────
[Fact]
public async Task DeleteOlderThanAsync_PrunesRuleRegressionRows_ReArmingLongBurningEpisodes()
{
var harness = new Harness();
Assert.True(await harness.Sut.TryRegisterRuleRegressionAsync(TenantId,
new AutopilotMonitor.Shared.Models.RuleRegressionAlert
{
TenantId = TenantId, RuleId = "ANALYZE-NET-001",
FirstNotifiedAt = DateTime.UtcNow, LastEvaluatedAt = DateTime.UtcNow,
}));
harness.AgeRows(TimeSpan.FromDays(31));
// Spec §F3: retention cleanup re-arms — a month-old still-burning regression rings again.
Assert.Equal(1, await harness.Sut.DeleteOlderThanAsync(DateTime.UtcNow.AddDays(-30)));
Assert.Empty(harness.Store);
}
[Fact]
public async Task DeleteOlderThanAsync_PrunesTpmPssRows_AndReArmsThatDevicesBell()
{
@@ -0,0 +1,165 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the per-session API-latency projection (2026-07-28): the V2 agent emits cumulative
/// counters <c>net_total_latency_ms</c> / <c>net_total_requests</c> in every
/// <c>agent_metrics_snapshot</c>; ingest derives the session-wide average from the LAST
/// snapshot of the batch and persists it via <c>UpdateSessionNetworkLatencyAsync</c>
/// (last-write-wins on cumulative counters → replay-idempotent).
/// Wire-format note: the ingest body is parsed with Newtonsoft into
/// <c>Dictionary&lt;string, object&gt;</c>, so integer JSON numbers arrive as boxed
/// <c>long</c> — fixtures mirror that, not the agent's in-memory types.
/// </summary>
public class IngestApiLatencyProjectionTests
{
private static EnrollmentEvent Snapshot(object? totalLatencyMs, object? totalRequests)
{
var evt = new EnrollmentEvent
{
EventType = "agent_metrics_snapshot",
Data = new Dictionary<string, object>
{
["agent_cpu_percent"] = 3.62,
["net_requests"] = 50L,
["net_avg_latency_ms"] = 513.0,
},
};
if (totalLatencyMs != null) evt.Data["net_total_latency_ms"] = totalLatencyMs;
if (totalRequests != null) evt.Data["net_total_requests"] = totalRequests;
return evt;
}
[Fact]
public void CumulativeCounters_YieldSessionAverage()
{
var events = new List<EnrollmentEvent> { Snapshot(51_300L, 100L) };
Assert.True(EventIngestProcessor.TryComputeSessionApiLatency(events, out var avg, out var count));
Assert.Equal(513.0, avg);
Assert.Equal(100, count);
}
[Fact]
public void AverageIsRounded_ToOneDecimal()
{
var events = new List<EnrollmentEvent> { Snapshot(1_000L, 3L) };
Assert.True(EventIngestProcessor.TryComputeSessionApiLatency(events, out var avg, out _));
Assert.Equal(333.3, avg);
}
[Fact]
public void LastSnapshotOfBatch_Wins()
{
// Cumulative counters grow monotonically — the last snapshot carries the most
// complete session average.
var events = new List<EnrollmentEvent>
{
Snapshot(51_300L, 100L),
new EnrollmentEvent { EventType = "download_progress" },
Snapshot(120_000L, 200L),
};
Assert.True(EventIngestProcessor.TryComputeSessionApiLatency(events, out var avg, out var count));
Assert.Equal(600.0, avg);
Assert.Equal(200, count);
}
[Fact]
public void OldAgentSnapshot_WithoutTotalLatencyField_DoesNotMatch()
{
// Agents predating net_total_latency_ms emit net_total_requests only.
var events = new List<EnrollmentEvent> { Snapshot(totalLatencyMs: null, totalRequests: 100L) };
Assert.False(EventIngestProcessor.TryComputeSessionApiLatency(events, out _, out _));
}
[Fact]
public void SnapshotWithLatencyButMissingRequests_DoesNotMatch()
{
var events = new List<EnrollmentEvent> { Snapshot(51_300L, totalRequests: null) };
Assert.False(EventIngestProcessor.TryComputeSessionApiLatency(events, out _, out _));
}
[Fact]
public void ZeroRequests_DoesNotMatch()
{
// No requests yet (first snapshot before the first upload completes) — no division.
var events = new List<EnrollmentEvent> { Snapshot(0L, 0L) };
Assert.False(EventIngestProcessor.TryComputeSessionApiLatency(events, out _, out _));
}
[Fact]
public void MatchingLastSnapshot_SkipsOlderAgentSnapshotAfterIt()
{
// A field-less (old-schema) snapshot later in the batch must not shadow an earlier
// one that carries the counters — the predicate selects the last MATCHING snapshot.
var events = new List<EnrollmentEvent>
{
Snapshot(51_300L, 100L),
Snapshot(totalLatencyMs: null, totalRequests: 150L),
};
Assert.True(EventIngestProcessor.TryComputeSessionApiLatency(events, out var avg, out var count));
Assert.Equal(513.0, avg);
Assert.Equal(100, count);
}
[Fact]
public void BatchWithoutSnapshots_DoesNotMatch()
{
var events = new List<EnrollmentEvent>
{
new EnrollmentEvent { EventType = "esp_phase_changed" },
new EnrollmentEvent { EventType = "app_install_started" },
};
Assert.False(EventIngestProcessor.TryComputeSessionApiLatency(events, out _, out _));
}
// -------- TryGetDouble coercion ladder (mirrors PlatformMetricsService.GetDouble) --------
[Theory]
[InlineData(513.5)] // double — decimal JSON number
[InlineData(513)] // int — in-memory constructed events (tests, V1 paths)
[InlineData(513L)] // long — Newtonsoft wire form for integer JSON numbers
[InlineData(513f)] // float
public void TryGetDouble_CoercesBoxedNumericPrimitives(object boxed)
{
var data = new Dictionary<string, object> { ["k"] = boxed };
Assert.True(EventIngestProcessor.TryGetDouble(data, "k", out var value));
Assert.Equal(Convert.ToDouble(boxed), value);
}
[Fact]
public void TryGetDouble_ParsesStrings_InvariantCulture()
{
var data = new Dictionary<string, object> { ["k"] = "513.5" };
Assert.True(EventIngestProcessor.TryGetDouble(data, "k", out var value));
Assert.Equal(513.5, value);
}
[Theory]
[InlineData("not-a-number")]
[InlineData("")]
public void TryGetDouble_RejectsUnparseableStrings(string raw)
{
var data = new Dictionary<string, object> { ["k"] = raw };
Assert.False(EventIngestProcessor.TryGetDouble(data, "k", out _));
}
[Fact]
public void TryGetDouble_MissingKey_ReturnsFalse()
{
Assert.False(EventIngestProcessor.TryGetDouble(new Dictionary<string, object>(), "k", out _));
}
}
@@ -137,6 +137,120 @@ public class MetricsMathAppPayloadTests
Assert.Equal(50d, app.GetProperty("failureRate").GetDouble());
}
// ── PR0 (2026-07-26): skip/unmeasured classification ────────────────────
[Fact]
public void Skips_LeaveTheFailureRate_AndAreReportedSeparately()
{
// 1 Failed + 1 real install + 3 skips ("Update for X" not applicable).
// Rate must be 1/(1+1) = 50%, NOT 1/(1+4) = 20% — skips are not attempts.
var summaries = new List<AppInstallSummary>
{
new() { AppName = "App A", Status = "Failed", TerminalState = "Error" },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 30 },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Skipped" },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Skipped" },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Postponed" },
};
var root = Build(summaries);
var app = root.GetProperty("topFailingApps")[0];
Assert.Equal(50d, app.GetProperty("failureRate").GetDouble());
Assert.Equal(1, app.GetProperty("succeeded").GetInt32());
Assert.Equal(3, app.GetProperty("skipped").GetInt32());
Assert.Equal(0, app.GetProperty("unmeasured").GetInt32());
Assert.Equal(3, root.GetProperty("totalSkipped").GetInt32());
}
[Fact]
public void ZeroDurations_AreExcludedFromAverages_AndCountedAsUnmeasured()
{
// Two measured installs (100s, 200s) + one whose start was never observed (audit §0.5:
// agent attach window — duration UNKNOWN). avg must be 150, not (100+200+0)/3 = 100.
var summaries = new List<AppInstallSummary>
{
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 100 },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 200 },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 0 },
};
var root = Build(summaries);
Assert.Equal(1, root.GetProperty("totalUnmeasured").GetInt32());
// Rebuild with a third measured row to clear the sample floor and read the average.
summaries.Add(new AppInstallSummary { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 150 });
var slowest = Build(summaries).GetProperty("slowestApps");
Assert.Equal(1, slowest.GetArrayLength());
Assert.Equal(150d, slowest[0].GetProperty("avgDurationSeconds").GetDouble());
Assert.Equal(1, slowest[0].GetProperty("unmeasured").GetInt32());
}
[Fact]
public void LegacyZeroDurationRows_StayInRate_ButNeverInDurations()
{
// Rows written before TerminalState existed: cannot be classified as skip vs unmeasured.
// They keep the legacy rate behavior (counted as succeeded) and are excluded from
// durations (counted as unmeasured — duration genuinely not measured).
var summaries = new List<AppInstallSummary>
{
new() { AppName = "App A", Status = "Failed" },
new() { AppName = "App A", Status = "Succeeded", DurationSeconds = 0 }, // legacy, no TerminalState
new() { AppName = "App A", Status = "Succeeded", DurationSeconds = 60, TerminalState = "Installed" },
};
var app = Build(summaries).GetProperty("topFailingApps")[0];
Assert.Equal(2, app.GetProperty("succeeded").GetInt32()); // legacy row stays in the rate
Assert.Equal(33.3d, app.GetProperty("failureRate").GetDouble());
Assert.Equal(1, app.GetProperty("unmeasured").GetInt32());
Assert.Equal(0, app.GetProperty("skipped").GetInt32());
}
[Fact]
public void ImplausibleDurations_AreExcludedFromAverages_AndCountedAsUnmeasured()
{
// A duration beyond the agent's max observation lifetime is a back-stamped emission
// artifact (CompletedAt pinned at session end — production: 60 175 s app rows inside a
// 2 697 s session), not a slow install. It must fall out of the averages exactly like
// an unobserved start. Boundary: exactly 6 h is still plausible/measured.
var summaries = new List<AppInstallSummary>
{
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 100 },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 200 },
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed",
DurationSeconds = MetricsMath.MaxPlausibleInstallDurationSeconds }, // boundary: measured
new() { AppName = "App A", Status = "Succeeded", TerminalState = "Installed",
DurationSeconds = MetricsMath.MaxPlausibleInstallDurationSeconds + 1 }, // artifact: unmeasured
};
var root = Build(summaries);
Assert.Equal(1, root.GetProperty("totalUnmeasured").GetInt32());
var slowest = root.GetProperty("slowestApps");
Assert.Equal(1, slowest.GetArrayLength());
Assert.Equal(3, slowest[0].GetProperty("measuredInstalls").GetInt32());
// avg over {100, 200, 21600} — the 21601 artifact contributes nothing.
Assert.Equal(7300d, slowest[0].GetProperty("avgDurationSeconds").GetDouble());
Assert.Equal(MetricsMath.MaxPlausibleInstallDurationSeconds, slowest[0].GetProperty("maxDurationSeconds").GetInt32());
}
[Fact]
public void SlowestApps_GateOnMeasuredInstalls_NotOnSucceededCount()
{
// 3 succeeded but only 2 measured → below the floor; an app must not rank "fast"
// (or at all) on a handful of unobserved zeros.
var summaries = new List<AppInstallSummary>
{
new() { AppName = "Mostly Unmeasured", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 5 },
new() { AppName = "Mostly Unmeasured", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 5 },
new() { AppName = "Mostly Unmeasured", Status = "Succeeded", TerminalState = "Installed", DurationSeconds = 0 },
};
var slowest = Build(summaries).GetProperty("slowestApps");
Assert.Equal(0, slowest.GetArrayLength());
}
[Fact]
public void SlowestApps_DropsAppsBelowSampleFloor()
{
@@ -125,5 +125,6 @@ public class OffboardingSessionEnumeratorTests
public Task<bool> IsSessionIndexEmptyAsync() => throw new NotImplementedException();
public Task<List<OrphanedEventSession>> GetOrphanedEventSessionsAsync(TimeSpan gracePeriod) => throw new NotImplementedException();
public Task DeleteEventSessionIndexEntryAsync(string tenantId, string sessionId) => throw new NotImplementedException();
public Task<List<(string TenantId, string SessionId)>> GetAllSessionTombstoneKeysAsync() => throw new NotImplementedException();
}
}
@@ -0,0 +1,405 @@
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// F3 PR6 (insights spec §F3): the Wilson primitive (hand-computed vectors), the radar's
/// detection gates and EVERY suppression branch (table-driven), the re-arm rule, the
/// dimension-concentration gates, the tracker keyspace round-trip (tri-states included)
/// and the alert wording contracts ("correlated, not necessarily causal").
/// </summary>
public class RuleRegressionRadarTests
{
private static readonly DateTime Target = new(2026, 7, 26, 0, 0, 0, DateTimeKind.Utc);
private const string TenantA = "aaaaaaaa-0000-0000-0000-000000000001";
private const string RuleX = "ANALYZE-NET-001";
// ── Wilson interval (MetricsMath) — hand-computed 95 % vectors ──────────
[Theory]
[InlineData(10, 20, 0.2993, 0.7007)] // symmetric mid-range
[InlineData(0, 20, 0.0000, 0.1611)] // zero successes: lower pinned at 0
[InlineData(5, 25, 0.0886, 0.3913)]
public void WilsonInterval_MatchesHandComputedVectors(int successes, int trials, double lower, double upper)
{
var (l, u) = MetricsMath.WilsonInterval(successes, trials);
Assert.Equal(lower, l, 3);
Assert.Equal(upper, u, 3);
}
[Fact]
public void WilsonInterval_ZeroTrials_IsUninformative_NeverClaimsSeparation()
{
Assert.Equal((0.0, 1.0), MetricsMath.WilsonInterval(0, 0));
// A zero-denominator window can never be "separated" from anything.
Assert.False(MetricsMath.RateIncreaseSeparated(0, 0, 10, 500));
}
[Fact]
public void RateIncreaseSeparated_DisjointIntervals_True_OverlappingFalse()
{
// 15 % on n=100 (lower ≈ 0.093) vs 2 % baseline on n=500 (upper ≈ 0.036) → separated.
Assert.True(MetricsMath.RateIncreaseSeparated(15, 100, 10, 500));
// 5 % on n=100 vs 4 % baseline on n=500 → intervals overlap → noise, not a regression.
Assert.False(MetricsMath.RateIncreaseSeparated(5, 100, 20, 500));
}
// ── radar evaluation — gates + suppression branches ─────────────────────
private static RuleStatsEntry Entry(
string date, int fire, int sessions, string ruleId = RuleX, string ruleType = "analyze")
=> new()
{
TenantId = TenantA,
Date = date,
RuleId = ruleId,
RuleType = ruleType,
RuleTitle = "Network check",
Category = "network",
Severity = "warning",
FireCount = fire,
EvaluationCount = sessions,
SessionsEvaluated = sessions,
};
private static string Day(int daysBeforeTarget) => Target.AddDays(-daysBeforeTarget).ToString("yyyy-MM-dd");
/// <summary>Window fire=15/100 across two days; baseline fire=10/500 across three days — fires all gates.</summary>
private static List<RuleStatsEntry> RegressedEntries() => new()
{
Entry(Day(0), 10, 60),
Entry(Day(6), 5, 40), // windowStart boundary — still window
Entry(Day(7), 4, 200), // first baseline day
Entry(Day(20), 4, 200),
Entry(Day(34), 2, 100), // baselineStart boundary — still baseline
};
private static Dictionary<string, RuleRegressionRadar.RuleTimestamps> OldRule(
DateTime? createdAt = null, DateTime? updatedAt = null, string ruleId = RuleX)
=> new(StringComparer.OrdinalIgnoreCase)
{
[ruleId] = new RuleRegressionRadar.RuleTimestamps(
createdAt ?? Target.AddDays(-100), updatedAt ?? Target.AddDays(-50)),
};
[Fact]
public void Evaluate_AllGatesPass_FiresWithFullNumbers()
{
var findings = RuleRegressionRadar.Evaluate(RegressedEntries(), Target, OldRule());
var finding = Assert.Single(findings);
Assert.Equal(RuleX, finding.RuleId);
Assert.Equal(15, finding.WindowFireCount);
Assert.Equal(100, finding.WindowSessionCount);
Assert.Equal(10, finding.BaselineFireCount);
Assert.Equal(500, finding.BaselineSessionCount);
Assert.Equal(15.0, finding.WindowRatePct);
Assert.Equal(2.0, finding.BaselineRatePct);
Assert.Equal(7.5, finding.Lift);
Assert.Equal(Day(6), finding.WindowStartDate);
Assert.Equal(Day(0), finding.WindowEndDate);
}
[Fact]
public void Evaluate_ZeroBaselineFires_FiresAsNewSignal_WithNullLift()
{
var entries = new List<RuleStatsEntry>
{
Entry(Day(0), 15, 100),
Entry(Day(10), 0, 500),
};
var finding = Assert.Single(RuleRegressionRadar.Evaluate(entries, Target, OldRule()));
Assert.Null(finding.Lift); // no finite lift is ever invented (truthfulness rule 1)
Assert.Equal(0.0, finding.BaselineRatePct);
}
public static TheoryData<string, List<RuleStatsEntry>> SuppressedByCounts() => new()
{
// Below the ≥5 hit-session gate.
{ "below hit gate", new List<RuleStatsEntry> { Entry(Day(0), 4, 100), Entry(Day(10), 1, 500) } },
// Below the ≥20 evaluated-session gate.
{ "below session gate", new List<RuleStatsEntry> { Entry(Day(0), 5, 19), Entry(Day(10), 1, 500) } },
// No baseline denominator at all (rule disabled through the baseline, re-enabled recently).
{ "no baseline sessions", new List<RuleStatsEntry> { Entry(Day(0), 15, 100) } },
// Lift below 2× (10 % vs 6 %).
{ "lift below 2x", new List<RuleStatsEntry> { Entry(Day(0), 10, 100), Entry(Day(10), 30, 500) } },
// Lift 2.1× but Wilson intervals overlap (5/20 vs 60/500) — small-n noise never alerts.
{ "not separated", new List<RuleStatsEntry> { Entry(Day(0), 5, 20), Entry(Day(10), 60, 500) } },
// Gather rows are ignored entirely (audit Q6: per-batch dedup — bad math until fixed).
{ "gather ignored", new List<RuleStatsEntry>
{ Entry(Day(0), 50, 60, ruleType: "gather"), Entry(Day(10), 1, 500, ruleType: "gather") } },
};
[Theory]
[MemberData(nameof(SuppressedByCounts))]
public void Evaluate_SuppressionBranches_ProduceNoFinding(string branch, List<RuleStatsEntry> entries)
{
Assert.True(RuleRegressionRadar.Evaluate(entries, Target, OldRule()).Count == 0,
$"branch '{branch}' must not fire");
}
[Fact]
public void Evaluate_RuleEntityGates_Suppress()
{
var entries = RegressedEntries();
// Deleted rule (no entity) never alerts.
Assert.Empty(RuleRegressionRadar.Evaluate(
entries, Target, new Dictionary<string, RuleRegressionRadar.RuleTimestamps>()));
// Grace period: created inside baseline+window (< 34 days before target).
Assert.Empty(RuleRegressionRadar.Evaluate(
entries, Target, OldRule(createdAt: Target.AddDays(-20))));
// Edited inside the 7-day window — edits legitimately change hit rates.
Assert.Empty(RuleRegressionRadar.Evaluate(
entries, Target, OldRule(updatedAt: Target.AddDays(-2))));
// Edited just BEFORE the window → not suppressed.
Assert.Single(RuleRegressionRadar.Evaluate(
entries, Target, OldRule(updatedAt: Target.AddDays(-8))));
}
[Fact]
public void SumWindows_BoundaryDates_LandInTheRightBucket()
{
var entries = new List<RuleStatsEntry>
{
Entry(Target.AddDays(1).ToString("yyyy-MM-dd"), 100, 100), // future — ignored
Entry(Day(0), 1, 10),
Entry(Day(6), 2, 20), // last window day
Entry(Day(7), 4, 40), // first baseline day
Entry(Day(34), 8, 80), // last baseline day
Entry(Day(35), 100, 100), // beyond horizon — ignored
};
var sums = RuleRegressionRadar.SumWindows(entries, Target);
Assert.Equal(3, sums.WindowFire);
Assert.Equal(30, sums.WindowSessions);
Assert.Equal(12, sums.BaselineFire);
Assert.Equal(120, sums.BaselineSessions);
}
// ── re-arm rule ─────────────────────────────────────────────────────────
[Fact]
public void ShouldReArm_FiresStopped_OrRateBackUnderOnePointFive()
{
// Nothing fired in the window anymore → re-arm (also covers "no stats rows at all").
Assert.True(RuleRegressionRadar.ShouldReArm(new List<RuleStatsEntry>(), Target));
Assert.True(RuleRegressionRadar.ShouldReArm(
new List<RuleStatsEntry> { Entry(Day(0), 0, 100), Entry(Day(10), 10, 500) }, Target));
// 2 % window vs 2 % baseline → 1.0× < 1.5 → re-arm.
Assert.True(RuleRegressionRadar.ShouldReArm(
new List<RuleStatsEntry> { Entry(Day(0), 2, 100), Entry(Day(10), 10, 500) }, Target));
// 4 % window vs 2 % baseline → 2.0× ≥ 1.5 → still elevated, keep the episode.
Assert.False(RuleRegressionRadar.ShouldReArm(
new List<RuleStatsEntry> { Entry(Day(0), 4, 100), Entry(Day(10), 10, 500) }, Target));
// Zero-baseline episode ("new signal"): no rate threshold to fall under — only the
// fires-stopped branch re-arms it.
Assert.False(RuleRegressionRadar.ShouldReArm(
new List<RuleStatsEntry> { Entry(Day(0), 3, 100), Entry(Day(10), 0, 500) }, Target));
}
// ── dimension concentration ─────────────────────────────────────────────
private static SessionSummary DimSession(string osBuild = "", string model = "", string agentVersion = "")
=> new()
{
TenantId = TenantA,
SessionId = Guid.NewGuid().ToString(),
SerialNumber = "PF4X1ABC",
OsBuild = osBuild,
Manufacturer = "Contoso",
Model = model,
AgentVersion = agentVersion,
ImeAgentVersion = "",
StartedAt = Target,
};
[Fact]
public void DimensionConcentration_DominantOsBuild_ReportsBothShares()
{
var all = new List<SessionSummary>();
for (var i = 0; i < 20; i++) all.Add(DimSession(osBuild: "26100.4652"));
for (var i = 0; i < 80; i++) all.Add(DimSession(osBuild: "22631.3880"));
var hits = all.Take(16).Concat(all.Skip(20).Take(4)).ToList(); // 16 on the new build, 4 on the old
var result = RuleRegressionRadar.ComputeDimensionConcentration(hits, all)!;
Assert.Equal("osBuild", result.Dimension);
Assert.Equal("26100.4652", result.Value);
Assert.Equal(16, result.HitCount);
Assert.Equal(80.0, result.HitSharePct);
Assert.Equal(20.0, result.AllSharePct);
Assert.Equal(4.0, result.Lift);
}
[Fact]
public void DimensionConcentration_BelowGates_YieldsNull_NeverStretched()
{
var all = Enumerable.Range(0, 100).Select(_ => DimSession(osBuild: "26100.1")).ToList();
// Every session shares the build → lift 1.0 < 2 → no claim.
Assert.Null(RuleRegressionRadar.ComputeDimensionConcentration(all.Take(10).ToList(), all));
// Concentrated but only 4 hit sessions (< 5) → no claim.
var mixed = Enumerable.Range(0, 96).Select(_ => DimSession(osBuild: "22631.1"))
.Concat(Enumerable.Range(0, 4).Select(_ => DimSession(osBuild: "26100.9")))
.ToList();
var hits4 = mixed.Skip(96).ToList();
Assert.Null(RuleRegressionRadar.ComputeDimensionConcentration(hits4, mixed));
// No hit sessions at all → no claim.
Assert.Null(RuleRegressionRadar.ComputeDimensionConcentration(new List<SessionSummary>(), all));
}
[Fact]
public void DimensionConcentration_PicksTheHighestLiftAcrossDimensions()
{
var all = new List<SessionSummary>();
for (var i = 0; i < 25; i++) all.Add(DimSession(osBuild: "26100.1", model: "Laptop 5"));
for (var i = 0; i < 75; i++) all.Add(DimSession(osBuild: "22631.1", model: i < 25 ? "Laptop 5" : "Laptop 4"));
// Hits: 8 of 10 on the new build (lift 80/25 = 3.2); model "Contoso Laptop 5" is 50 % of
// fleet and 80 % of hits (lift 1.6 — below gate).
var hits = all.Take(8).Concat(all.Skip(25).Take(2)).ToList();
var result = RuleRegressionRadar.ComputeDimensionConcentration(hits, all)!;
Assert.Equal("osBuild", result.Dimension);
Assert.Equal(3.2, result.Lift);
}
// ── tracker keyspace (round-trip + key shape) ───────────────────────────
[Fact]
public void RuleRegressionRowKey_PrefixedAndCaseFolded()
{
Assert.Equal("ruleregression|analyze-net-001",
TableHardwareRejectionNotificationTracker.BuildRuleRegressionRowKey(" ANALYZE-NET-001 "));
}
[Fact]
public void RuleRegressionEntity_RoundTripsAllFields_IncludingDimension()
{
var alert = new RuleRegressionAlert
{
TenantId = TenantA,
RuleId = RuleX,
RuleTitle = "Network check",
WindowFireCount = 15,
WindowSessionCount = 100,
BaselineFireCount = 10,
BaselineSessionCount = 500,
WindowRatePct = 15.0,
BaselineRatePct = 2.0,
Lift = 7.5,
WindowStartDate = "2026-07-20",
WindowEndDate = "2026-07-26",
Dimension = new RuleRegressionDimension
{
Dimension = "osBuild", Value = "26100.4652",
HitCount = 12, HitSharePct = 80.0, AllSharePct = 20.0, Lift = 4.0,
},
FirstNotifiedAt = Target,
LastEvaluatedAt = Target.AddHours(2),
};
var entity = TableHardwareRejectionNotificationTracker.BuildRuleRegressionEntity(TenantA, alert);
Assert.Equal(TenantA.ToLowerInvariant(), entity.PartitionKey);
Assert.Equal("ruleregression|analyze-net-001", entity.RowKey);
var mapped = TableHardwareRejectionNotificationTracker.MapToRuleRegressionAlert(entity);
Assert.Equal(RuleX, mapped.RuleId);
Assert.Equal("Network check", mapped.RuleTitle);
Assert.Equal(15, mapped.WindowFireCount);
Assert.Equal(100, mapped.WindowSessionCount);
Assert.Equal(10, mapped.BaselineFireCount);
Assert.Equal(500, mapped.BaselineSessionCount);
Assert.Equal(15.0, mapped.WindowRatePct);
Assert.Equal(2.0, mapped.BaselineRatePct);
Assert.Equal(7.5, mapped.Lift);
Assert.Equal("2026-07-20", mapped.WindowStartDate);
Assert.Equal("2026-07-26", mapped.WindowEndDate);
Assert.Equal(Target, mapped.FirstNotifiedAt);
Assert.Equal(Target.AddHours(2), mapped.LastEvaluatedAt);
Assert.NotNull(mapped.Dimension);
Assert.Equal("osBuild", mapped.Dimension!.Dimension);
Assert.Equal(4.0, mapped.Dimension.Lift);
}
[Fact]
public void RuleRegressionEntity_TriStates_AbsentColumnsMapToNull_NeverInvented()
{
var alert = new RuleRegressionAlert
{
TenantId = TenantA, RuleId = RuleX,
Lift = null, // new signal — no finite lift
Dimension = null, // no clear concentration
FirstNotifiedAt = Target, LastEvaluatedAt = Target,
};
var entity = TableHardwareRejectionNotificationTracker.BuildRuleRegressionEntity(TenantA, alert);
Assert.False(entity.ContainsKey("Lift"));
Assert.False(entity.ContainsKey("DimensionJson"));
var mapped = TableHardwareRejectionNotificationTracker.MapToRuleRegressionAlert(entity);
Assert.Null(mapped.Lift);
Assert.Null(mapped.Dimension);
}
// ── wording contracts ───────────────────────────────────────────────────
[Fact]
public void DescribeDimension_UsesCorrelationWording_NeverCausal()
{
var text = MaintenanceService.DescribeDimension(new RuleRegressionDimension
{
Dimension = "osBuild", Value = "26100.4652",
HitCount = 12, HitSharePct = 87.0, AllSharePct = 22.0, Lift = 4.0,
})!;
Assert.Contains("87% of affected sessions are on osBuild 26100.4652", text);
Assert.Contains("vs 22% of all sessions (lift 4x)", text);
Assert.Contains("correlated, not necessarily causal", text);
Assert.Null(MaintenanceService.DescribeDimension(null));
}
[Fact]
public void BuildRegressionMessage_CarriesFullNumbers_AndHonestNoDimensionFallback()
{
var finding = new RuleRegressionFinding
{
TenantId = TenantA, RuleId = RuleX, RuleTitle = "Network check",
WindowFireCount = 15, WindowSessionCount = 100, WindowRatePct = 15.0,
BaselineFireCount = 10, BaselineSessionCount = 500, BaselineRatePct = 2.0,
Lift = 7.5, WindowStartDate = "2026-07-20", WindowEndDate = "2026-07-26",
};
var message = MaintenanceService.BuildRegressionMessage(finding, dimensionSummary: null);
Assert.Contains("15 of 100 evaluated sessions", message);
Assert.Contains("(15%)", message);
Assert.Contains("baseline 2% (10/500 over the prior 28 days)", message);
Assert.Contains("lift 7.5x", message);
Assert.Contains("No clear dimension concentration.", message);
var newSignal = MaintenanceService.BuildRegressionMessage(
new RuleRegressionFinding
{
RuleId = RuleX, RuleTitle = "Network check",
WindowFireCount = 15, WindowSessionCount = 100, WindowRatePct = 15.0,
BaselineFireCount = 0, BaselineSessionCount = 500, BaselineRatePct = 0.0,
Lift = null, WindowStartDate = "2026-07-20", WindowEndDate = "2026-07-26",
},
dimensionSummary: "dim summary");
Assert.Contains("new signal", newSignal);
Assert.Contains("dim summary.", newSignal);
}
}
@@ -0,0 +1,562 @@
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Golden fixtures for <see cref="TimeAttributionCalculator"/> (F1 PR1, insights spec §F1) —
/// synthetic sessions only (no customer data), each pinning one semantic decision:
/// linear phase partition, clock skew, late start, WhiteGlove two-window split, missing /
/// truncated blocking lists, IME retry occupancy, and reboot gaps. EVERY fixture asserts the
/// exact-partition invariant: sum(span seconds) + unattributed == wall clock ==
/// the session's DurationSeconds (truthfulness rule 2 — never normalized, never interpolated).
/// </summary>
public class TimeAttributionCalculatorTests
{
private static readonly DateTime T0 = new(2026, 7, 26, 10, 0, 0, DateTimeKind.Utc);
private const string AppA = "aaaaaaaa-1111-2222-3333-444444444444"; // in blocking list
private const string AppB = "bbbbbbbb-1111-2222-3333-444444444444"; // in blocking list, never observed
private const string AppC = "cccccccc-1111-2222-3333-444444444444"; // observed, NOT listed
private static long _seq;
private static EnrollmentEvent Evt(
DateTime ts, string eventType,
EnrollmentPhase phase = EnrollmentPhase.Unknown,
Dictionary<string, object>? data = null)
=> new()
{
EventType = eventType,
Timestamp = ts,
Sequence = ++_seq,
Phase = phase,
Data = data ?? new Dictionary<string, object>(),
};
private static Dictionary<string, object> AppData(string appId, string appName, string? state = null)
{
var d = new Dictionary<string, object> { ["appId"] = appId, ["appName"] = appName };
if (state != null) d["state"] = state;
return d;
}
/// <summary>Storage-shape blocking lists (EventDataNormalizer yields List&lt;object&gt;).</summary>
private static Dictionary<string, object> EspLists(long? win32Count = null, params string[] win32Ids)
=> new()
{
["espTrackedWin32AppIds"] = new List<object>(win32Ids),
["espTrackedUserWin32AppIds"] = new List<object>(),
["espTrackedMsiProductCodes"] = new List<object>(),
["espTrackedModernAppPfns"] = new List<object>(),
["espTrackedWin32Count"] = win32Count ?? win32Ids.Length,
["espTrackedMsiCount"] = 0L,
["espTrackedModernCount"] = 0L,
};
private static TimeAttributionInput Input(
List<EnrollmentEvent> events, DateTime completedAt, int durationSeconds,
string status = "Succeeded", bool wg = false, DateTime? resumedAt = null)
=> new()
{
TenantId = "00000000-0000-0000-0000-0000000000t1",
SessionId = "00000000-0000-0000-0000-0000000000s1",
Status = status,
StartedAt = T0,
CompletedAt = completedAt,
DurationSeconds = durationSeconds,
IsPreProvisioned = wg,
ResumedAt = resumedAt,
Events = events,
};
/// <summary>The invariant every fixture must satisfy exactly (spec §0 rule 2).</summary>
private static void AssertExactPartition(SessionTimeBreakdown b)
{
Assert.Equal(b.WallClockSeconds, b.Segments.Sum(s => s.Seconds) + b.UnattributedSeconds);
Assert.True(b.UnattributedSeconds >= 0, "unattributed must never be negative");
}
// ── golden: linear UDE session ──────────────────────────────────────────
private static List<EnrollmentEvent> LinearSession()
=> new()
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddMinutes(2), "esp_config_detected", data: EspLists(win32Ids: new[] { AppA, AppB })),
Evt(T0.AddMinutes(5), "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(T0.AddMinutes(6), "app_install_started", data: AppData(AppA, "App A")),
Evt(T0.AddMinutes(10), "app_install_completed", data: AppData(AppA, "App A", "Installed")),
Evt(T0.AddMinutes(11), "app_install_started", data: AppData(AppC, "Untracked App")),
Evt(T0.AddMinutes(13), "app_install_completed", data: AppData(AppC, "Untracked App", "Installed")),
Evt(T0.AddMinutes(15), "esp_phase_changed", EnrollmentPhase.AccountSetup),
Evt(T0.AddMinutes(20), "esp_phase_changed", EnrollmentPhase.AppsUser),
Evt(T0.AddMinutes(25), "esp_phase_changed", EnrollmentPhase.FinalizingSetup),
Evt(T0.AddMinutes(28), "desktop_arrived"),
Evt(T0.AddMinutes(30), "enrollment_complete"),
};
[Fact]
public void Linear_Session_PartitionsIntoAllFiveSegments_Exactly()
{
var b = TimeAttributionCalculator.Compute(
Input(LinearSession(), completedAt: T0.AddMinutes(30), durationSeconds: 1800))!;
AssertExactPartition(b);
Assert.Equal(1800, b.WallClockSeconds);
Assert.Equal(0, b.UnattributedSeconds);
Assert.Equal(TimeAttributionFlags.None, b.QualityFlags);
Assert.Equal(TimeAttributionCalculator.CurrentVersion, b.AttributionVersion);
var totals = b.GetSegmentTotals();
Assert.Equal(300, totals[TimeAttributionSegments.DevicePrep]); // T0 → apps entry (+5m)
Assert.Equal(600, totals[TimeAttributionSegments.EspApps]); // +5m → +15m
Assert.Equal(480, totals[TimeAttributionSegments.IdentityHello]); // AccountSetup 300 + Finalizing 180
Assert.Equal(300, totals[TimeAttributionSegments.UserEsp]); // +20m → +25m
Assert.Equal(120, totals[TimeAttributionSegments.DesktopHandoff]); // desktop_arrived → terminal
}
[Fact]
public void Linear_Session_BlockingJoin_IsPositiveEvidenceOnly()
{
var b = TimeAttributionCalculator.Compute(
Input(LinearSession(), completedAt: T0.AddMinutes(30), durationSeconds: 1800))!;
// App A is listed → blocking, interval from EVENT timestamps (started → completed).
var interval = Assert.Single(b.BlockingApps);
Assert.Equal(AppA, interval.AppId);
Assert.Equal(240, interval.Seconds); // +6m → +10m
// App C installed fine but is NOT in the lists → unknown, NEVER "not blocking":
// it appears nowhere (no negative claim), and only App A counts as matched.
Assert.Equal(1, b.BlockingAppCount);
// Occupancy = App A's interval clipped to the esp_apps span.
Assert.Equal(240, b.EspAppsOccupancySeconds);
}
// ── skew: backward anchor is dropped, dust stays honest ─────────────────
[Fact]
public void BackwardPhaseAnchor_IsDropped_WithClockSkewFlag()
{
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddMinutes(10), "esp_phase_changed", EnrollmentPhase.AppsDevice),
// Replayed declaration whose timestamp runs BACKWARD in canonical order.
Evt(T0.AddMinutes(3), "esp_phase_changed", EnrollmentPhase.AccountSetup),
Evt(T0.AddMinutes(30), "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt: T0.AddMinutes(30), durationSeconds: 1800))!;
AssertExactPartition(b);
Assert.True(b.QualityFlags.HasFlag(TimeAttributionFlags.ClockSkewDropped));
var totals = b.GetSegmentTotals();
Assert.Equal(600, totals[TimeAttributionSegments.DevicePrep]);
Assert.Equal(1200, totals[TimeAttributionSegments.EspApps]); // runs to window end — no phantom AccountSetup
Assert.False(totals.ContainsKey(TimeAttributionSegments.IdentityHello));
}
[Fact]
public void SubSecondBoundaries_FloorIntoUnattributedDust_NeverInvented()
{
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddSeconds(90.5), "esp_phase_changed", EnrollmentPhase.AppsDevice),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt: T0.AddSeconds(300), durationSeconds: 300))!;
AssertExactPartition(b);
var totals = b.GetSegmentTotals();
Assert.Equal(90, totals[TimeAttributionSegments.DevicePrep]); // floor(90.5)
Assert.Equal(209, totals[TimeAttributionSegments.EspApps]); // floor(209.5)
Assert.Equal(1, b.UnattributedSeconds); // the dust, disclosed
}
// ── late start ──────────────────────────────────────────────────────────
[Fact]
public void AgentLateStart_SetsPartialObservation()
{
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddSeconds(1), "agent_late_start"),
Evt(T0.AddMinutes(10), "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt: T0.AddMinutes(10), durationSeconds: 600))!;
AssertExactPartition(b);
Assert.True(b.QualityFlags.HasFlag(TimeAttributionFlags.PartialObservation));
}
// ── WhiteGlove: two windows in ONE session, pause never attributed ──────
[Fact]
public void WhiteGlove_SplitsIntoTwoWindows_PauseExcluded_WallClockIsDurationSeconds()
{
var part1End = T0.AddMinutes(20);
var resumedAt = T0.AddDays(2); // 2-day pause
var completedAt = resumedAt.AddMinutes(10); // part 2 = 600 s
// Stored combined duration: part1 (1200 s) + part2 (600 s) — pause excluded by design.
const int duration = 1800;
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddMinutes(1), "esp_config_detected", data: EspLists(win32Ids: new[] { AppA })),
Evt(T0.AddMinutes(2), "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(part1End, "whiteglove_part1_complete"),
// — pause —
Evt(resumedAt.AddSeconds(30), "agent_started", EnrollmentPhase.AccountSetup),
Evt(resumedAt.AddMinutes(5), "esp_phase_changed", EnrollmentPhase.FinalizingSetup),
Evt(resumedAt.AddMinutes(8), "desktop_arrived"),
Evt(completedAt, "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt, duration, wg: true, resumedAt: resumedAt))!;
AssertExactPartition(b);
// Wall clock is the stored DurationSeconds — NOT CompletedAt − StartedAt (≈ 2 days).
Assert.Equal(1800, b.WallClockSeconds);
Assert.Equal(TimeAttributionFlags.None, b.QualityFlags); // part1 anchored on the event
var totals = b.GetSegmentTotals();
Assert.Equal(120, totals[TimeAttributionSegments.DevicePrep]); // W1: T0 → +2m
Assert.Equal(1080, totals[TimeAttributionSegments.EspApps]); // W1: +2m → part1End
Assert.Equal(450, totals[TimeAttributionSegments.IdentityHello]); // W2: +30s → +8m
Assert.Equal(120, totals[TimeAttributionSegments.DesktopHandoff]); // W2: +8m → +10m
// W2's 30 s resume gap has no phase context — unattributed, not guessed.
Assert.Equal(30, b.UnattributedSeconds);
// No span may live inside the pause.
Assert.DoesNotContain(b.Segments, s => s.StartUtc > part1End && s.EndUtc < resumedAt);
}
[Fact]
public void WhiteGlove_WithoutPart1CompleteEvent_FallsBackAndFlags()
{
var resumedAt = T0.AddDays(1);
var completedAt = resumedAt.AddMinutes(10);
var lastPart1Event = T0.AddMinutes(20);
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(lastPart1Event, "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(resumedAt.AddSeconds(10), "agent_started", EnrollmentPhase.AccountSetup),
Evt(completedAt, "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt, durationSeconds: 1800, wg: true, resumedAt: resumedAt))!;
AssertExactPartition(b);
Assert.True(b.QualityFlags.HasFlag(TimeAttributionFlags.WhiteGloveAnchorsIncomplete));
Assert.Equal(1800, b.WallClockSeconds);
}
[Fact]
public void WhiteGlove_Part2LongerThanStoredDuration_DegradesToSingleWindow_Flagged()
{
// Stored duration (600 s) contradicts the resume stamp (part 2 alone spans 900 s):
// no honest two-window split exists — degrade to the single derived window instead
// of inventing a negative part 1.
var resumedAt = T0.AddDays(1);
var completedAt = resumedAt.AddMinutes(15);
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(resumedAt.AddSeconds(5), "agent_started", EnrollmentPhase.AccountSetup),
Evt(completedAt, "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt, durationSeconds: 600, wg: true, resumedAt: resumedAt))!;
AssertExactPartition(b);
Assert.Equal(600, b.WallClockSeconds);
Assert.True(b.QualityFlags.HasFlag(TimeAttributionFlags.WhiteGloveAnchorsIncomplete));
}
// ── blocking list missing / truncated ───────────────────────────────────
[Fact]
public void NoBlockingLists_MeansUnknown_NotZero()
{
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddMinutes(2), "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(T0.AddMinutes(3), "app_install_started", data: AppData(AppA, "App A")),
Evt(T0.AddMinutes(8), "app_install_completed", data: AppData(AppA, "App A", "Installed")),
Evt(T0.AddMinutes(10), "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt: T0.AddMinutes(10), durationSeconds: 600))!;
AssertExactPartition(b);
Assert.True(b.QualityFlags.HasFlag(TimeAttributionFlags.BlockingSetUnknown));
Assert.Empty(b.BlockingApps);
Assert.Equal(0, b.BlockingAppCount);
Assert.Null(b.EspAppsOccupancySeconds); // unknown — NOT 0 (rule 1)
}
[Fact]
public void EspConfigWithoutListKeys_IsNotEvidence_LaterEmissionNeverErasesEarlierLists()
{
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddMinutes(1), "esp_config_detected", data: EspLists(win32Ids: new[] { AppA })),
Evt(T0.AddMinutes(2), "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(T0.AddMinutes(3), "app_install_started", data: AppData(AppA, "App A")),
Evt(T0.AddMinutes(8), "app_install_completed", data: AppData(AppA, "App A", "Installed")),
// Later emission whose probe found no Diagnostics key — no espTracked* keys at all.
Evt(T0.AddMinutes(9), "esp_config_detected",
data: new Dictionary<string, object> { ["source"] = "registry_firstsync" }),
Evt(T0.AddMinutes(10), "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt: T0.AddMinutes(10), durationSeconds: 600))!;
Assert.False(b.QualityFlags.HasFlag(TimeAttributionFlags.BlockingSetUnknown));
Assert.Equal(AppA, Assert.Single(b.BlockingApps).AppId);
}
[Fact]
public void TruncatedBlockingList_SetsFlag_AbsentAppsStayUnknown()
{
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
// Uncapped total (60) exceeds the emitted list (1 id) → cap dropped entries.
Evt(T0.AddMinutes(1), "esp_config_detected", data: EspLists(win32Count: 60, AppA)),
Evt(T0.AddMinutes(2), "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(T0.AddMinutes(3), "app_install_started", data: AppData(AppC, "Maybe Capped App")),
Evt(T0.AddMinutes(7), "app_install_completed", data: AppData(AppC, "Maybe Capped App", "Installed")),
Evt(T0.AddMinutes(10), "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt: T0.AddMinutes(10), durationSeconds: 600))!;
Assert.True(b.QualityFlags.HasFlag(TimeAttributionFlags.BlockingSetTruncated));
// App C is absent from the (capped) list → it stays UNKNOWN: no interval, no claim.
Assert.Empty(b.BlockingApps);
Assert.Equal(0, b.BlockingAppCount);
}
// ── retry: occupancy spans to the LAST terminal event (audit Q1) ────────
[Fact]
public void Retry_IntervalRunsFromFirstStart_ToLastTerminalEvent()
{
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddMinutes(1), "esp_config_detected", data: EspLists(win32Ids: new[] { AppA })),
Evt(T0.AddMinutes(2), "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(T0.AddMinutes(6), "app_install_started", data: AppData(AppA, "App A")),
Evt(T0.AddMinutes(8), "app_install_failed", data: AppData(AppA, "App A")),
Evt(T0.AddMinutes(9), "app_install_started", data: AppData(AppA, "App A")), // IME retry
Evt(T0.AddMinutes(12), "app_install_completed", data: AppData(AppA, "App A", "Installed")),
Evt(T0.AddMinutes(15), "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt: T0.AddMinutes(15), durationSeconds: 900))!;
AssertExactPartition(b);
var interval = Assert.Single(b.BlockingApps);
// First started (+6m) → LAST terminal (+12m) = 360 s. The payload timing would have
// frozen at the first failure (+8m) — event timestamps are the only valid source.
Assert.Equal(360, interval.Seconds);
Assert.Equal(360, b.EspAppsOccupancySeconds);
}
// ── reboots: spans from event gaps, never the detection event's stamp ───
[Fact]
public void RebootSpan_ComesFromEventGapAroundLastBootUtc_NotFromDetectionTimestamp()
{
var boot = T0.AddMinutes(12);
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddMinutes(5), "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(T0.AddMinutes(10), "app_install_started", data: AppData(AppA, "App A")), // last pre-reboot
Evt(T0.AddMinutes(14), "system_reboot_detected", // next run, detection-time
data: new Dictionary<string, object> { ["lastBootUtc"] = boot.ToString("o") }),
// Duplicate detection of the SAME boot (batch replay) — must not double-count.
Evt(T0.AddMinutes(14.5), "system_reboot_detected",
data: new Dictionary<string, object> { ["lastBootUtc"] = boot.ToString("o") }),
Evt(T0.AddMinutes(20), "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt: T0.AddMinutes(20), durationSeconds: 1200))!;
AssertExactPartition(b);
var span = Assert.Single(b.RebootSpans);
// Gap = last event before boot (+10m) → first observation at/after boot (+14m).
Assert.Equal(T0.AddMinutes(10), span.StartUtc);
Assert.Equal(T0.AddMinutes(14), span.EndUtc);
Assert.Equal(240, span.Seconds);
Assert.Equal(240, b.RebootSeconds);
// The reboot began inside the apps phase — cross-cutting annotation, segment unchanged.
Assert.Equal(TimeAttributionSegments.EspApps, span.SegmentKey);
Assert.Equal(1200 - 300, b.GetSegmentTotals()[TimeAttributionSegments.EspApps]);
}
// ── WhiteGlove pause vs. intervals/reboots (Codex review P1): the pause lies INSIDE
// the windows' hull, so hull-clamping alone would count it — a reboot bracketing the
// pause must contribute only its in-window flanks, never the pause itself ─────────────
[Fact]
public void WhiteGlove_RebootGapAcrossPause_CountsOnlyInWindowFlanks()
{
var part1End = T0.AddMinutes(20);
var resumedAt = T0.AddDays(2); // 2-day pause
var completedAt = resumedAt.AddMinutes(10); // part 2 = 600 s → part 1 = 1200 s
var bootInsidePause = T0.AddDays(1);
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddMinutes(2), "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(part1End, "whiteglove_part1_complete"),
// — pause: device sealed, user boots it a day later —
Evt(resumedAt.AddSeconds(30), "agent_started", EnrollmentPhase.AccountSetup),
Evt(resumedAt.AddSeconds(40), "system_reboot_detected",
data: new Dictionary<string, object> { ["lastBootUtc"] = bootInsidePause.ToString("o") }),
Evt(completedAt, "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt, durationSeconds: 1800, wg: true, resumedAt: resumedAt))!;
AssertExactPartition(b);
var span = Assert.Single(b.RebootSpans);
// Gap = part1End → first post-boot event (+30 s into part 2). In-window seconds are
// ONLY the part-2 flank (30 s) — the ~2-day pause never counts (pre-fix: ~172 830 s
// of "reboot" in a 30-minute enrollment).
Assert.Equal(part1End, span.StartUtc);
Assert.Equal(resumedAt.AddSeconds(30), span.EndUtc);
Assert.Equal(30, span.Seconds);
Assert.Equal(30, b.RebootSeconds);
}
[Fact]
public void WhiteGlove_BlockingIntervalAcrossPause_CountsOnlyInWindowSeconds()
{
var part1End = T0.AddMinutes(20);
var resumedAt = T0.AddDays(2);
var completedAt = resumedAt.AddMinutes(10);
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddMinutes(1), "esp_config_detected", data: EspLists(win32Ids: new[] { AppA })),
Evt(T0.AddMinutes(2), "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(T0.AddMinutes(10), "app_install_started", data: AppData(AppA, "App A")),
Evt(part1End, "whiteglove_part1_complete"),
// — pause —
Evt(resumedAt.AddSeconds(30), "agent_started", EnrollmentPhase.AccountSetup),
// The app's final terminal lands in part 2 (user-scope finish after resume).
Evt(resumedAt.AddMinutes(2), "app_install_completed", data: AppData(AppA, "App A", "Installed")),
Evt(completedAt, "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt, durationSeconds: 1800, wg: true, resumedAt: resumedAt))!;
AssertExactPartition(b);
var interval = Assert.Single(b.BlockingApps);
// Chronological endpoints stay the real event times…
Assert.Equal(T0.AddMinutes(10), interval.StartUtc);
Assert.Equal(resumedAt.AddMinutes(2), interval.EndUtc);
// …but Seconds is the in-window sum: part 1 tail (10 min) + part 2 head (2 min).
Assert.Equal(720, interval.Seconds);
// Occupancy intersects with the esp_apps span (part 1 only) — pause never leaks in.
Assert.Equal(600, b.EspAppsOccupancySeconds);
}
[Fact]
public void WhiteGlove_IntervalWhollyInsidePause_MakesNoClaim_AndNoSkewFlag()
{
var part1End = T0.AddMinutes(20);
var resumedAt = T0.AddDays(2);
var completedAt = resumedAt.AddMinutes(10);
var events = new List<EnrollmentEvent>
{
Evt(T0, "agent_started", EnrollmentPhase.DeviceSetup),
Evt(T0.AddMinutes(1), "esp_config_detected", data: EspLists(win32Ids: new[] { AppA })),
Evt(T0.AddMinutes(2), "esp_phase_changed", EnrollmentPhase.AppsDevice),
Evt(part1End, "whiteglove_part1_complete"),
// Both endpoints inside the pause (post-seal tail activity) — observed, but in no window.
Evt(part1End.AddHours(1), "app_install_started", data: AppData(AppA, "App A")),
Evt(part1End.AddHours(2), "app_install_completed", data: AppData(AppA, "App A", "Installed")),
Evt(resumedAt.AddSeconds(30), "agent_started", EnrollmentPhase.AccountSetup),
Evt(completedAt, "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt, durationSeconds: 1800, wg: true, resumedAt: resumedAt))!;
AssertExactPartition(b);
// No in-window observation → no interval claim; and nothing is wrong with the clocks,
// so the skew flag must NOT fire (it would exclude the session from fleet medians).
Assert.Empty(b.BlockingApps);
Assert.False(b.QualityFlags.HasFlag(TimeAttributionFlags.ClockSkewDropped));
}
// ── non-computable sessions ─────────────────────────────────────────────
[Theory]
[InlineData("Incomplete")] // terminal but deliberately NO DurationSeconds
[InlineData("InProgress")]
[InlineData("Pending")]
public void NonComputableStatuses_ReturnNull(string status)
{
var input = Input(LinearSession(), completedAt: T0.AddMinutes(30), durationSeconds: 1800, status: status);
Assert.Null(TimeAttributionCalculator.Compute(input));
}
[Fact]
public void MissingDuration_ReturnsNull_NeverGuessesFromCompletedMinusStarted()
{
var input = Input(LinearSession(), completedAt: T0.AddMinutes(30), durationSeconds: 0);
Assert.Null(TimeAttributionCalculator.Compute(input));
}
// ── no phase evidence at all ────────────────────────────────────────────
[Fact]
public void NoPhaseDeclarations_LeaveEverythingUnattributed()
{
var events = new List<EnrollmentEvent>
{
Evt(T0.AddMinutes(1), "network_state_change"),
Evt(T0.AddMinutes(9), "enrollment_complete"),
};
var b = TimeAttributionCalculator.Compute(
Input(events, completedAt: T0.AddMinutes(10), durationSeconds: 600))!;
AssertExactPartition(b);
Assert.Empty(b.Segments);
Assert.Equal(600, b.UnattributedSeconds); // no evidence → no claim (rule 1)
}
}
@@ -0,0 +1,458 @@
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// F1 PR2 (insights spec §F1 "Data &amp; compute changes"): persistence round-trips for the
/// SessionTimeBreakdowns / TimeAttributionAggregates rows (table-storage-serialization rule —
/// every property must survive Store→Map, tri-states included), the pure daily-aggregation
/// core (<see cref="MaintenanceService.BuildTimeAttributionAggregates"/>: clean/flagged/missing
/// disclosure, per-class bucketing, fixed segment stack, per-app gates) and the what-if bound.
/// </summary>
public class TimeAttributionPersistenceAndAggregationTests
{
private static readonly DateTime T0 = new(2026, 7, 26, 10, 0, 0, DateTimeKind.Utc);
private const string TenantA = "aaaaaaaa-0000-0000-0000-000000000001";
private const string TenantB = "bbbbbbbb-0000-0000-0000-000000000002";
private const string AppX = "aaaaaaaa-1111-2222-3333-444444444444";
private const string AppY = "bbbbbbbb-1111-2222-3333-444444444444";
// ── breakdown row round-trip ────────────────────────────────────────────
[Fact]
public void BreakdownEntity_RoundTripsAllFields_IncludingJsonLists()
{
var breakdown = new SessionTimeBreakdown
{
TenantId = TenantA,
SessionId = "s-1",
AttributionVersion = 1,
EventCountAtCompute = 261,
WallClockSeconds = 1800,
UnattributedSeconds = 30,
RebootSeconds = 240,
BlockingAppCount = 3,
EspAppsOccupancySeconds = 600,
QualityFlags = TimeAttributionFlags.PartialObservation | TimeAttributionFlags.BlockingSetTruncated,
Segments = new List<TimeAttributionSpan>
{
new() { SegmentKey = TimeAttributionSegments.DevicePrep, StartUtc = T0, EndUtc = T0.AddMinutes(5), Seconds = 300 },
new() { SegmentKey = TimeAttributionSegments.EspApps, StartUtc = T0.AddMinutes(5), EndUtc = T0.AddMinutes(30), Seconds = 1500 },
},
RebootSpans = new List<RebootSpan>
{
new() { StartUtc = T0.AddMinutes(10), EndUtc = T0.AddMinutes(14), Seconds = 240, SegmentKey = TimeAttributionSegments.EspApps },
},
BlockingApps = new List<BlockingAppInterval>
{
new() { AppId = AppX, AppName = "App X", StartUtc = T0.AddMinutes(6), EndUtc = T0.AddMinutes(16), Seconds = 600 },
},
};
var mapped = TableStorageService.MapToSessionTimeBreakdown(
TableStorageService.BuildSessionTimeBreakdownEntity(breakdown));
Assert.Equal(TenantA, mapped.TenantId);
Assert.Equal("s-1", mapped.SessionId);
Assert.Equal(1, mapped.AttributionVersion);
// Sweep change signal (Codex review: late batches must trigger a recompute) — a lost
// column would read as 0 and recompute forever OR mask real drift.
Assert.Equal(261, mapped.EventCountAtCompute);
Assert.Equal(1800, mapped.WallClockSeconds);
Assert.Equal(30, mapped.UnattributedSeconds);
Assert.Equal(240, mapped.RebootSeconds);
Assert.Equal(3, mapped.BlockingAppCount);
Assert.Equal(600, mapped.EspAppsOccupancySeconds);
Assert.Equal(TimeAttributionFlags.PartialObservation | TimeAttributionFlags.BlockingSetTruncated, mapped.QualityFlags);
Assert.Equal(2, mapped.Segments.Count);
Assert.Equal(TimeAttributionSegments.DevicePrep, mapped.Segments[0].SegmentKey);
Assert.Equal(T0, mapped.Segments[0].StartUtc);
Assert.Equal(300, mapped.Segments[0].Seconds);
var reboot = Assert.Single(mapped.RebootSpans);
Assert.Equal(TimeAttributionSegments.EspApps, reboot.SegmentKey);
Assert.Equal(240, reboot.Seconds);
var app = Assert.Single(mapped.BlockingApps);
Assert.Equal(AppX, app.AppId);
Assert.Equal("App X", app.AppName);
Assert.Equal(600, app.Seconds);
}
[Fact]
public void BreakdownEntity_OccupancyTriState_AbsentColumnMapsToNull_NeverZero()
{
var unknown = new SessionTimeBreakdown
{
TenantId = TenantA, SessionId = "s-1", WallClockSeconds = 600,
EspAppsOccupancySeconds = null, QualityFlags = TimeAttributionFlags.BlockingSetUnknown,
};
var entity = TableStorageService.BuildSessionTimeBreakdownEntity(unknown);
Assert.False(entity.ContainsKey("EspAppsOccupancySeconds"));
Assert.Null(TableStorageService.MapToSessionTimeBreakdown(entity).EspAppsOccupancySeconds);
// Zero is a real measured value (blocking set known, nothing occupied) — must survive.
var zero = new SessionTimeBreakdown
{
TenantId = TenantA, SessionId = "s-2", WallClockSeconds = 600, EspAppsOccupancySeconds = 0,
};
Assert.Equal(0, TableStorageService.MapToSessionTimeBreakdown(
TableStorageService.BuildSessionTimeBreakdownEntity(zero)).EspAppsOccupancySeconds);
}
// ── aggregate row round-trip ────────────────────────────────────────────
[Fact]
public void AggregateEntity_RoundTripsAllFields()
{
var aggregate = new TimeAttributionDailyAggregate
{
TenantId = TenantA,
Date = "2026-07-26",
EnrollmentClass = "user_driven",
AttributionVersion = 1,
CleanSessionCount = 23,
FlaggedExcludedCount = 4,
MissingBreakdownCount = 1,
ComputedAt = T0,
SegmentStats = new List<TimeAttributionSegmentStat>
{
new() { SegmentKey = TimeAttributionSegments.EspApps, MedianSeconds = 600, P75Seconds = 700, P90Seconds = 900 },
},
TopBlockingApps = new List<TimeAttributionBlockingAppStat>
{
new()
{
AppId = AppX, AppName = "App X", SessionCount = 12,
MedianSeconds = 300, P75Seconds = 400, MedianSavingSeconds = 120, P75SavingSeconds = 200,
},
},
};
var entity = TableStorageService.BuildTimeAttributionAggregateEntity(aggregate);
Assert.Equal("2026-07-26|user_driven", entity.RowKey);
var mapped = TableStorageService.MapToTimeAttributionAggregate(entity);
Assert.Equal(TenantA, mapped.TenantId);
Assert.Equal("2026-07-26", mapped.Date);
Assert.Equal("user_driven", mapped.EnrollmentClass);
Assert.Equal(1, mapped.AttributionVersion);
Assert.Equal(23, mapped.CleanSessionCount);
Assert.Equal(4, mapped.FlaggedExcludedCount);
Assert.Equal(1, mapped.MissingBreakdownCount);
var segment = Assert.Single(mapped.SegmentStats);
Assert.Equal(TimeAttributionSegments.EspApps, segment.SegmentKey);
Assert.Equal(600, segment.MedianSeconds);
Assert.Equal(900, segment.P90Seconds);
var app = Assert.Single(mapped.TopBlockingApps);
Assert.Equal(AppX, app.AppId);
Assert.Equal(12, app.SessionCount);
Assert.Equal(120, app.MedianSavingSeconds);
}
// ── enrollment class ────────────────────────────────────────────────────
[Fact]
public void EnrollmentClass_Precedence_WhiteGlove_Wdp_SelfDeploying_UserDriven()
{
Assert.Equal("whiteglove", TimeAttributionCalculator.GetEnrollmentClass(
new SessionSummary { IsPreProvisioned = true, EnrollmentType = "v2", IsSelfDeployingProfile = true }));
Assert.Equal("device_preparation", TimeAttributionCalculator.GetEnrollmentClass(
new SessionSummary { EnrollmentType = "v2", IsSelfDeployingProfile = true }));
Assert.Equal("self_deploying", TimeAttributionCalculator.GetEnrollmentClass(
new SessionSummary { IsSelfDeployingProfile = true }));
Assert.Equal("user_driven", TimeAttributionCalculator.GetEnrollmentClass(new SessionSummary()));
}
// ── what-if bound ───────────────────────────────────────────────────────
private static BlockingAppInterval Interval(string appId, DateTime start, DateTime end)
=> new() { AppId = appId, AppName = appId, StartUtc = start, EndUtc = end, Seconds = (int)(end - start).TotalSeconds };
[Fact]
public void WhatIf_LastEndingApp_SavesTailToPreviousEnd_OthersSaveNothing()
{
var intervals = new List<BlockingAppInterval>
{
Interval(AppX, T0, T0.AddMinutes(10)), // ends 10:10
Interval(AppY, T0.AddMinutes(5), T0.AddMinutes(20)), // ends 10:20 — critical-path end
};
// Removing Y moves the path end 10:20 → 10:10.
Assert.Equal(600, TimeAttributionCalculator.WhatIfSavingSeconds(intervals, AppY));
// Removing X changes nothing — Y still ends at 10:20.
Assert.Equal(0, TimeAttributionCalculator.WhatIfSavingSeconds(intervals, AppX));
// An app without a measured interval yields no claim.
Assert.Equal(0, TimeAttributionCalculator.WhatIfSavingSeconds(intervals, "cccccccc-0000-0000-0000-000000000000"));
}
[Fact]
public void WhatIf_OnlyBlockingApp_FallsBackToOwnStart()
{
var intervals = new List<BlockingAppInterval> { Interval(AppX, T0.AddMinutes(5), T0.AddMinutes(20)) };
Assert.Equal(900, TimeAttributionCalculator.WhatIfSavingSeconds(intervals, AppX));
}
[Fact]
public void WhatIf_IdleGapBeforeLastApp_CountsIntoTheBound()
{
// A ends 10:00; X runs 10:05 → 10:20. Removing X exposes 10:00 as the path end —
// the ESP waited for X to START too, so the bound is 20 min, not 15 (spec formula).
var intervals = new List<BlockingAppInterval>
{
Interval(AppY, T0.AddMinutes(-30), T0),
Interval(AppX, T0.AddMinutes(5), T0.AddMinutes(20)),
};
Assert.Equal(1200, TimeAttributionCalculator.WhatIfSavingSeconds(intervals, AppX));
}
// ── daily aggregation core ──────────────────────────────────────────────
private static SessionSummary Session(string tenantId, DateTime startedAt, bool wg = false)
=> new()
{
TenantId = tenantId,
SessionId = Guid.NewGuid().ToString(),
StartedAt = startedAt,
Status = SessionStatus.Succeeded,
IsPreProvisioned = wg,
};
private static SessionTimeBreakdown Breakdown(
TimeAttributionFlags flags = TimeAttributionFlags.None,
int espAppsSeconds = 600,
int unattributed = 0,
List<BlockingAppInterval>? apps = null)
=> new()
{
AttributionVersion = TimeAttributionCalculator.CurrentVersion,
WallClockSeconds = espAppsSeconds + 300 + unattributed,
UnattributedSeconds = unattributed,
QualityFlags = flags,
Segments = new List<TimeAttributionSpan>
{
new() { SegmentKey = TimeAttributionSegments.DevicePrep, StartUtc = T0, EndUtc = T0.AddSeconds(300), Seconds = 300 },
new() { SegmentKey = TimeAttributionSegments.EspApps, StartUtc = T0.AddSeconds(300), EndUtc = T0.AddSeconds(300 + espAppsSeconds), Seconds = espAppsSeconds },
},
BlockingApps = apps ?? new List<BlockingAppInterval>(),
};
[Fact]
public void Aggregates_BucketPerTenantClassDate_AndMirrorGlobal()
{
var pairs = new List<(SessionSummary, SessionTimeBreakdown)>
{
(Session(TenantA, T0), Breakdown(espAppsSeconds: 100)),
(Session(TenantA, T0), Breakdown(espAppsSeconds: 300)),
(Session(TenantA, T0, wg: true), Breakdown(espAppsSeconds: 900)), // separate class
(Session(TenantB, T0), Breakdown(espAppsSeconds: 500)), // separate tenant
};
var aggregates = MaintenanceService.BuildTimeAttributionAggregates(
pairs, Array.Empty<SessionSummary>(), T0);
// Tenant A: user_driven + whiteglove; Tenant B: user_driven; global: user_driven + whiteglove.
Assert.Equal(5, aggregates.Count);
var tenantAUde = aggregates.Single(a => a.TenantId == TenantA && a.EnrollmentClass == "user_driven");
Assert.Equal("2026-07-26", tenantAUde.Date);
Assert.Equal(2, tenantAUde.CleanSessionCount);
Assert.Equal(TimeAttributionCalculator.CurrentVersion, tenantAUde.AttributionVersion);
// Median of {100, 300} = 100 (nearest-rank: ceil(0.5·2)−1 = index 0 — repo convention).
Assert.Equal(100, tenantAUde.SegmentStats.Single(s => s.SegmentKey == TimeAttributionSegments.EspApps).MedianSeconds);
var wg = aggregates.Single(a => a.TenantId == TenantA && a.EnrollmentClass == "whiteglove");
Assert.Equal(1, wg.CleanSessionCount); // classes never mixed
var globalUde = aggregates.Single(a => a.TenantId == "global" && a.EnrollmentClass == "user_driven");
Assert.Equal(3, globalUde.CleanSessionCount); // A(2) + B(1)
}
[Fact]
public void Aggregates_FlaggedSessions_AreExcludedFromStats_ButDisclosed()
{
var pairs = new List<(SessionSummary, SessionTimeBreakdown)>
{
(Session(TenantA, T0), Breakdown(espAppsSeconds: 100)),
(Session(TenantA, T0), Breakdown(flags: TimeAttributionFlags.PartialObservation, espAppsSeconds: 99999)),
};
var aggregate = MaintenanceService.BuildTimeAttributionAggregates(
pairs, Array.Empty<SessionSummary>(), T0)
.Single(a => a.TenantId == TenantA);
Assert.Equal(1, aggregate.CleanSessionCount);
Assert.Equal(1, aggregate.FlaggedExcludedCount);
// The flagged 99999s outlier must not shape the median.
Assert.Equal(100, aggregate.SegmentStats.Single(s => s.SegmentKey == TimeAttributionSegments.EspApps).MedianSeconds);
}
[Fact]
public void Aggregates_AllFlaggedDay_YieldsDisclosureRow_WithoutStats()
{
var pairs = new List<(SessionSummary, SessionTimeBreakdown)>
{
(Session(TenantA, T0), Breakdown(flags: TimeAttributionFlags.ClockSkewDropped)),
};
var aggregate = MaintenanceService.BuildTimeAttributionAggregates(
pairs, Array.Empty<SessionSummary>(), T0)
.Single(a => a.TenantId == TenantA);
Assert.Equal(0, aggregate.CleanSessionCount);
Assert.Equal(1, aggregate.FlaggedExcludedCount);
Assert.Empty(aggregate.SegmentStats); // no clean sessions → no statistical claim
Assert.Empty(aggregate.TopBlockingApps);
}
[Fact]
public void Aggregates_BlockingOnlyFlags_StayInSegmentStats()
{
// BlockingSetUnknown/Truncated limit per-app blocking EVIDENCE, not the measured spans.
// Gating on any flag starved the fleet aggregates in production (903/1000 breakdowns
// carried BlockingSetUnknown, 2026-07-27) — blocking-only flagged sessions must count
// as clean; only duration-critical flags exclude.
var pairs = new List<(SessionSummary, SessionTimeBreakdown)>
{
(Session(TenantA, T0), Breakdown(flags: TimeAttributionFlags.BlockingSetUnknown, espAppsSeconds: 100)),
(Session(TenantA, T0), Breakdown(flags: TimeAttributionFlags.BlockingSetTruncated, espAppsSeconds: 300)),
(Session(TenantA, T0), Breakdown(
flags: TimeAttributionFlags.BlockingSetUnknown | TimeAttributionFlags.PartialObservation,
espAppsSeconds: 99999)), // duration-critical bit set → still excluded
};
var aggregate = MaintenanceService.BuildTimeAttributionAggregates(
pairs, Array.Empty<SessionSummary>(), T0)
.Single(a => a.TenantId == TenantA);
Assert.Equal(2, aggregate.CleanSessionCount);
Assert.Equal(1, aggregate.FlaggedExcludedCount);
// Median of {100, 300} = 100 (nearest-rank, lower value — repo convention).
Assert.Equal(100, aggregate.SegmentStats.Single(s => s.SegmentKey == TimeAttributionSegments.EspApps).MedianSeconds);
}
[Fact]
public void Aggregates_MissingBreakdowns_AreCounted_NeverGuessed()
{
var missing = new List<SessionSummary> { Session(TenantA, T0) };
var aggregate = MaintenanceService.BuildTimeAttributionAggregates(
Array.Empty<(SessionSummary, SessionTimeBreakdown)>(), missing, T0)
.Single(a => a.TenantId == TenantA);
Assert.Equal(0, aggregate.CleanSessionCount);
Assert.Equal(1, aggregate.MissingBreakdownCount);
Assert.Empty(aggregate.SegmentStats);
}
[Fact]
public void Aggregates_SegmentStack_AlwaysCarriesAllSegmentsPlusUnattributed_AbsentAsZero()
{
var pairs = new List<(SessionSummary, SessionTimeBreakdown)>
{
(Session(TenantA, T0), Breakdown(unattributed: 42)),
};
var aggregate = MaintenanceService.BuildTimeAttributionAggregates(
pairs, Array.Empty<SessionSummary>(), T0)
.Single(a => a.TenantId == TenantA);
Assert.Equal(6, aggregate.SegmentStats.Count); // 5 canonical + unattributed — the full stack
// The fixture has no user_esp span → honest 0, not absent.
Assert.Equal(0, aggregate.SegmentStats.Single(s => s.SegmentKey == TimeAttributionSegments.UserEsp).MedianSeconds);
Assert.Equal(42, aggregate.SegmentStats.Single(s => s.SegmentKey == TimeAttributionSegments.Unattributed).MedianSeconds);
}
[Fact]
public void Aggregates_RollingDateKey_MergesAllDatesIntoRangeStatistics()
{
// Two sessions on DIFFERENT days — the rolling row must merge them (a median of
// per-day medians would not be the range median; the rolling rows are the honest
// range statistics the fleet panel reads).
var pairs = new List<(SessionSummary, SessionTimeBreakdown)>
{
(Session(TenantA, T0), Breakdown(espAppsSeconds: 100)),
(Session(TenantA, T0.AddDays(-3)), Breakdown(espAppsSeconds: 300)),
};
var rolling = MaintenanceService.BuildTimeAttributionAggregates(
pairs, Array.Empty<SessionSummary>(), T0, MaintenanceService.RollingAggregateDateKey);
var row = rolling.Single(a => a.TenantId == TenantA);
Assert.Equal(MaintenanceService.RollingAggregateDateKey, row.Date);
Assert.Equal(2, row.CleanSessionCount);
// Rolling rows sort AFTER every "yyyy-MM-dd|…" key so date-range reads and the
// age-based retention filter never touch them.
Assert.True(string.CompareOrdinal($"{row.Date}|user_driven", "2026-12-31|zzz") > 0);
}
[Fact]
public void Aggregates_PerAppRows_GateAtFiveSessions_AndCarryWhatIfBounds()
{
var pairs = new List<(SessionSummary, SessionTimeBreakdown)>();
// App X in 5 sessions (ends last each time → saving = its full tail over device-prep-only baseline).
for (var i = 0; i < 5; i++)
{
pairs.Add((Session(TenantA, T0), Breakdown(apps: new List<BlockingAppInterval>
{
Interval(AppX, T0.AddMinutes(5), T0.AddMinutes(15)), // only measured blocking app → saving 600
})));
}
// App Y in only 4 sessions → below the per-app row gate.
for (var i = 0; i < 4; i++)
{
pairs.Add((Session(TenantA, T0), Breakdown(apps: new List<BlockingAppInterval>
{
Interval(AppY, T0.AddMinutes(5), T0.AddMinutes(6)),
})));
}
var aggregate = MaintenanceService.BuildTimeAttributionAggregates(
pairs, Array.Empty<SessionSummary>(), T0)
.Single(a => a.TenantId == TenantA);
var row = Assert.Single(aggregate.TopBlockingApps);
Assert.Equal(AppX, row.AppId);
Assert.Equal(5, row.SessionCount);
Assert.Equal(600, row.MedianSeconds);
Assert.Equal(600, row.MedianSavingSeconds); // sole blocking app → bound = full interval
}
// ── stale-bucket reconcile key consistency (Codex review) ───────────────
[Fact]
public void RollingAggregateEntity_RowKey_MatchesTargetedDeleteKey()
{
// The sweep's reconcile deletes by (Date, EnrollmentClass) — the entity builder and the
// delete path must agree on the RK shape for BOTH daily and rolling rows, or stale
// rolling rows would never be removed.
var rolling = new TimeAttributionDailyAggregate
{
TenantId = TenantA, Date = "rolling30", EnrollmentClass = "whiteglove",
};
Assert.Equal("rolling30|whiteglove", TableStorageService.BuildTimeAttributionAggregateEntity(rolling).RowKey);
var daily = new TimeAttributionDailyAggregate
{
TenantId = TenantA, Date = "2026-07-27", EnrollmentClass = "user_driven",
};
Assert.Equal("2026-07-27|user_driven", TableStorageService.BuildTimeAttributionAggregateEntity(daily).RowKey);
}
[Fact]
public void TimeAttribution_InclusiveWindowStart_YieldsExactly30CalendarDays()
{
// Codex review: both range ends are inclusive — subtracting the full WindowDays
// returned 31 day keys under a "windowDays: 30" label.
var today = new System.DateTime(2026, 7, 27);
Assert.Equal(today.AddDays(-29),
Functions.Metrics.TimeAttributionResponse.InclusiveWindowStart(today, Functions.Metrics.TimeAttributionResponse.WindowDays));
}
}
@@ -1,6 +1,8 @@
using System.Text.Json;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Azure;
using Azure.Data.Tables;
using Microsoft.Extensions.Logging;
@@ -13,6 +15,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
/// - hardware rejection: "{manufacturer-lower}|{model-lower}" (trimmed)
/// - TPM PSS unsupported: "tpmpss|{serial-lower}" (trimmed; the literal "tpmpss" prefix
/// cannot collide with a hardware key because no real manufacturer is named "tpmpss")
/// - F3 rule regression: "ruleregression|{ruleId-lower}" (trimmed; payload-carrying rows
/// that are refreshed while the episode is active and deleted on re-arm)
/// Race-safe via AddEntityAsync: Azure Table Storage returns 409 Conflict if the entity already exists.
/// </summary>
public class TableHardwareRejectionNotificationTracker : IHardwareRejectionNotificationTracker
@@ -109,16 +113,102 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
}
}
public async Task<bool> TryRegisterRuleRegressionAsync(string tenantId, RuleRegressionAlert alert)
{
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(alert?.RuleId))
return false;
try
{
await _table.AddEntityAsync(BuildRuleRegressionEntity(tenantId, alert!));
_logger.LogInformation(
"RuleRegression tracker registered: tenant={TenantId} rule={RuleId}", tenantId, alert!.RuleId);
return true;
}
catch (RequestFailedException ex) when (ex.Status == 409)
{
// Episode already active — one bell per episode.
return false;
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"RuleRegression tracker failed: tenant={TenantId} rule={RuleId}", tenantId, alert!.RuleId);
// Fail closed so we do not double-fire if the row was actually written.
return false;
}
}
public async Task RefreshRuleRegressionAsync(string tenantId, RuleRegressionAlert alert)
{
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(alert?.RuleId))
return;
try
{
// Replace-mode upsert with the FULL payload incl. the caller-carried
// FirstNotifiedAt — the retention sweep re-arms on the ORIGINAL age, so a
// refresh must never rejuvenate the row.
await _table.UpsertEntityAsync(BuildRuleRegressionEntity(tenantId, alert!), TableUpdateMode.Replace);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"RuleRegression tracker refresh failed: tenant={TenantId} rule={RuleId} (stale numbers remain)",
tenantId, alert!.RuleId);
}
}
public async Task DeleteRuleRegressionAsync(string tenantId, string ruleId)
{
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(ruleId))
return;
try
{
await _table.DeleteEntityAsync(tenantId.ToLowerInvariant(), BuildRuleRegressionRowKey(ruleId));
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
// Already gone (retention sweep or concurrent pass) — idempotent.
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"RuleRegression tracker delete failed: tenant={TenantId} rule={RuleId} (episode stays until retention)",
tenantId, ruleId);
}
}
public async Task<List<RuleRegressionAlert>> GetRuleRegressionsAsync(string tenantId)
{
var results = new List<RuleRegressionAlert>();
if (string.IsNullOrWhiteSpace(tenantId)) return results;
try
{
var partitionKey = tenantId.ToLowerInvariant();
// '}' (0x7D) sorts directly after '|' (0x7C) — prefix range over "ruleregression|…".
var filter = $"PartitionKey eq '{partitionKey}' and RowKey ge '{RuleRegressionRowKeyPrefix}' and RowKey lt 'ruleregression}}'";
await foreach (var entity in _table.QueryAsync<TableEntity>(filter: filter))
{
results.Add(MapToRuleRegressionAlert(entity));
}
}
catch (Exception ex)
{
_logger.LogWarning(ex, "RuleRegression tracker list failed for tenant {TenantId}", tenantId);
}
return results;
}
public async Task<int> DeleteOlderThanAsync(DateTime cutoffUtc)
{
try
{
// Rows are insert-once (AddEntityAsync, never updated), so FirstNotifiedAt == creation time.
// Pruning here resets the lifetime dedup for BOTH key spaces in this table: a hardware
// model rejected again after the cutoff rings once more, and so does a TPM-PSS device
// that reports again ("tpmpss|{serial}" rows are pruned by the same sweep). Acceptable
// in both cases — the portal only surfaces recent rejections anyway, and a device whose
// TPM was never fixed is worth surfacing again after a month of silence.
// FirstNotifiedAt == the row's first registration (rule-regression refreshes carry
// it forward unchanged). Pruning here resets the dedup for EVERY key space in this
// table: a hardware model rejected again after the cutoff rings once more, so does
// a TPM-PSS device that reports again ("tpmpss|{serial}" rows), and a rule
// regression still active after the window fires a fresh bell — a month-old
// still-burning regression is worth a reminder (spec §F3 retention re-arm).
var filter = $"FirstNotifiedAt lt datetime'{cutoffUtc:yyyy-MM-ddTHH:mm:ss}Z'";
var query = _table.QueryAsync<TableEntity>(filter: filter, select: new[] { "PartitionKey", "RowKey" });
@@ -160,5 +250,72 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
var sn = (serialNumber ?? string.Empty).Trim().ToLowerInvariant();
return $"tpmpss|{sn}";
}
internal const string RuleRegressionRowKeyPrefix = "ruleregression|";
internal static string BuildRuleRegressionRowKey(string ruleId)
{
var id = (ruleId ?? string.Empty).Trim().ToLowerInvariant();
return $"{RuleRegressionRowKeyPrefix}{id}";
}
// internal static: entity builder + mapper are pinned by round-trip unit tests
// (table-storage-serialization rule — every property must survive Store→Map).
internal static TableEntity BuildRuleRegressionEntity(string tenantId, RuleRegressionAlert alert)
{
var entity = new TableEntity(tenantId.ToLowerInvariant(), BuildRuleRegressionRowKey(alert.RuleId))
{
["TenantId"] = tenantId,
["RuleId"] = alert.RuleId,
["RuleTitle"] = alert.RuleTitle ?? string.Empty,
["WindowFireCount"] = alert.WindowFireCount,
["WindowSessionCount"] = alert.WindowSessionCount,
["BaselineFireCount"] = alert.BaselineFireCount,
["BaselineSessionCount"] = alert.BaselineSessionCount,
["WindowRatePct"] = alert.WindowRatePct,
["BaselineRatePct"] = alert.BaselineRatePct,
["WindowStartDate"] = alert.WindowStartDate ?? string.Empty,
["WindowEndDate"] = alert.WindowEndDate ?? string.Empty,
["FirstNotifiedAt"] = alert.FirstNotifiedAt,
["LastEvaluatedAt"] = alert.LastEvaluatedAt,
};
// Tri-states: absent column = "no claim" — a zero-baseline alert has no finite lift,
// and a missing dimension means "no clear concentration", never a guessed one.
if (alert.Lift.HasValue)
entity["Lift"] = alert.Lift.Value;
if (alert.Dimension != null)
entity["DimensionJson"] = JsonSerializer.Serialize(alert.Dimension);
return entity;
}
internal static RuleRegressionAlert MapToRuleRegressionAlert(TableEntity entity)
{
RuleRegressionDimension? dimension = null;
var dimensionJson = entity.GetString("DimensionJson");
if (!string.IsNullOrEmpty(dimensionJson))
{
try { dimension = JsonSerializer.Deserialize<RuleRegressionDimension>(dimensionJson!); }
catch (JsonException) { dimension = null; } // corrupt column — degrade to "no claim"
}
return new RuleRegressionAlert
{
TenantId = entity.GetString("TenantId") ?? entity.PartitionKey,
RuleId = entity.GetString("RuleId") ?? entity.RowKey.Substring(RuleRegressionRowKeyPrefix.Length),
RuleTitle = entity.GetString("RuleTitle") ?? string.Empty,
WindowFireCount = entity.GetInt32("WindowFireCount") ?? 0,
WindowSessionCount = entity.GetInt32("WindowSessionCount") ?? 0,
BaselineFireCount = entity.GetInt32("BaselineFireCount") ?? 0,
BaselineSessionCount = entity.GetInt32("BaselineSessionCount") ?? 0,
WindowRatePct = entity.GetDouble("WindowRatePct") ?? 0,
BaselineRatePct = entity.GetDouble("BaselineRatePct") ?? 0,
Lift = entity.GetDouble("Lift"),
WindowStartDate = entity.GetString("WindowStartDate") ?? string.Empty,
WindowEndDate = entity.GetString("WindowEndDate") ?? string.Empty,
Dimension = dimension,
FirstNotifiedAt = entity.GetDateTimeOffset("FirstNotifiedAt")?.UtcDateTime ?? DateTime.MinValue,
LastEvaluatedAt = entity.GetDateTimeOffset("LastEvaluatedAt")?.UtcDateTime ?? DateTime.MinValue,
};
}
}
}
@@ -99,6 +99,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task DeleteEventSessionIndexEntryAsync(string tenantId, string sessionId)
=> _storage.DeleteEventSessionIndexEntryAsync(tenantId, sessionId);
public Task<List<(string TenantId, string SessionId)>> GetAllSessionTombstoneKeysAsync()
=> _storage.GetAllSessionTombstoneKeysAsync();
/// <summary>
/// Tenant offboarding cascade-worker session enumerator. Yields one row per session
/// belonging to <paramref name="tenantId"/>. Storage exceptions are intentionally
@@ -111,5 +111,53 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<int> DeleteRuleStatsOlderThanAsync(DateTime cutoffDate)
=> _storage.DeleteRuleStatsOlderThanAsync(cutoffDate);
public Task<SessionTimeBreakdown?> ComputeAndStoreSessionTimeBreakdownAsync(string tenantId, string sessionId)
=> _storage.ComputeAndStoreSessionTimeBreakdownAsync(tenantId, sessionId);
public Task<SessionTimeBreakdown?> GetSessionTimeBreakdownAsync(string tenantId, string sessionId)
=> _storage.GetSessionTimeBreakdownAsync(tenantId, sessionId);
public Task ResolveEspBlockingForSessionAsync(string tenantId, string sessionId)
=> _storage.ResolveEspBlockingForSessionAsync(tenantId, sessionId);
public Task<bool> SaveTimeAttributionAggregateAsync(TimeAttributionDailyAggregate aggregate)
=> _storage.SaveTimeAttributionAggregateAsync(aggregate);
public Task<List<TimeAttributionDailyAggregate>> GetTimeAttributionAggregatesAsync(string tenantId, DateTime startDate, DateTime endDate)
=> _storage.GetTimeAttributionAggregatesAsync(tenantId, startDate, endDate);
public Task<List<TimeAttributionDailyAggregate>> GetRollingTimeAttributionAggregatesAsync(string tenantId)
=> _storage.GetRollingTimeAttributionAggregatesAsync(tenantId);
public Task DeleteTimeAttributionAggregateAsync(string tenantId, string dateKey, string enrollmentClass)
=> _storage.DeleteTimeAttributionAggregateAsync(tenantId, dateKey, enrollmentClass);
public Task<int> DeleteTimeAttributionAggregatesOlderThanAsync(DateTime cutoffDate)
=> _storage.DeleteTimeAttributionAggregatesOlderThanAsync(cutoffDate);
public Task<DeviceHistory?> GetDeviceHistoryAsync(string tenantId, string serialKey)
=> _storage.GetDeviceHistoryAsync(tenantId, serialKey);
public Task<List<DeviceHistory>> GetDeviceHistoriesByTenantAsync(string tenantId)
=> _storage.GetDeviceHistoriesByTenantAsync(tenantId);
public Task<bool> UpsertDeviceHistoryAsync(DeviceHistory history)
=> _storage.UpsertDeviceHistoryAsync(history);
public Task DeleteDeviceHistoryAsync(string tenantId, string serialKey)
=> _storage.DeleteDeviceHistoryAsync(tenantId, serialKey);
public Task<bool> SaveDeviceJourneyAggregateAsync(DeviceJourneyDailyAggregate aggregate)
=> _storage.SaveDeviceJourneyAggregateAsync(aggregate);
public Task<List<DeviceJourneyDailyAggregate>> GetDeviceJourneyAggregatesAsync(string tenantId, DateTime startDate, DateTime endDate)
=> _storage.GetDeviceJourneyAggregatesAsync(tenantId, startDate, endDate);
public Task DeleteDeviceJourneyAggregateAsync(string tenantId, string dateKey)
=> _storage.DeleteDeviceJourneyAggregateAsync(tenantId, dateKey);
public Task<int> DeleteDeviceJourneyAggregatesOlderThanAsync(DateTime cutoffDate)
=> _storage.DeleteDeviceJourneyAggregatesOlderThanAsync(cutoffDate);
}
}
@@ -30,6 +30,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<List<RuleResult>> GetRuleResultsAsync(string tenantId, string sessionId)
=> _storage.GetRuleResultsAsync(tenantId, sessionId);
public Task<List<string>> GetRuleHitSessionIdsAsync(string tenantId, string ruleId, DateTime sinceUtc, int maxResults = 2000)
=> _storage.GetRuleHitSessionIdsAsync(tenantId, ruleId, sinceUtc, maxResults);
public Task<bool> StoreGatherRuleAsync(GatherRule rule, string tenantId = "global")
=> _storage.StoreGatherRuleAsync(rule, tenantId);
@@ -84,7 +84,32 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
// otherwise a swept-Incomplete session keeps whatever stale live count it had.
// Idempotent + fail-soft, so the redundancy is cheap.
if (transitioned && (status == SessionStatus.Succeeded || status == SessionStatus.Failed || status == SessionStatus.Incomplete))
{
await _storage.ReconcileSessionCountersAsync(tenantId, sessionId);
// F1 PR1: same single-writer seam — join the session's app rows against its
// latest esp_config_detected lists exactly once, when the event stream is
// complete. Positive evidence only (listed ⇒ EspBlocking=true, absent ⇒ stays
// unknown). Idempotent + fail-soft like the counter reconcile above.
await _storage.ResolveEspBlockingForSessionAsync(tenantId, sessionId);
// F1 PR2: compute + persist the time-attribution breakdown once, now that the
// terminal write stamped CompletedAt/DurationSeconds and the event stream is
// complete. Succeeded/Failed only — Incomplete deliberately stores no duration,
// so there is no wall clock to partition (the compute would no-op anyway; the
// gate just saves the point-reads). Fail-soft; the 30d maintenance sweep
// self-heals any miss.
if (status == SessionStatus.Succeeded || status == SessionStatus.Failed)
await _storage.ComputeAndStoreSessionTimeBreakdownAsync(tenantId, sessionId);
// F2 PR4: upsert this terminal session into its device's history chain at the
// same seam, so the session-detail "Attempt N" banner is fresh rather than a
// sweep-cycle old. ALL three terminal statuses — Incomplete is a non-successful
// journey attempt, unlike the duration-gated breakdown above. Re-terminal
// reclassifications (admin mark, late completion) flow through here too and
// update the chain entry's status. Fail-soft + idempotent; the maintenance
// sweep self-heals any miss.
await _storage.UpdateDeviceHistoryForSessionAsync(tenantId, sessionId);
}
return transitioned;
}
@@ -123,6 +148,10 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task UpdateSessionImeAgentVersionAsync(string tenantId, string sessionId, string version)
=> _storage.UpdateSessionImeAgentVersionAsync(tenantId, sessionId, version);
public Task UpdateSessionNetworkLatencyAsync(string tenantId, string sessionId,
double avgLatencyMs, int requestCount)
=> _storage.UpdateSessionNetworkLatencyAsync(tenantId, sessionId, avgLatencyMs, requestCount);
public Task<List<SessionSummary>> GetSessionsWithEventCountAboveAsync(string tenantId, int threshold)
=> _storage.GetSessionsWithEventCountAboveAsync(tenantId, threshold);
@@ -141,13 +141,21 @@ namespace AutopilotMonitor.Functions.Functions.Apps
var summaries = allSummaries.Where(s => s.StartedAt >= cutoff).ToList();
var apps = summaries.GroupBy(s => s.AppName).Select(g =>
// F1 PR1 (audit Q3): appId-collision rows merge two distinct apps under one name —
// they are excluded from every per-app group and disclosed via collisionExcluded.
var collisionExcluded = summaries.Count(s => s.AppIdCollision);
var apps = summaries.Where(s => !s.AppIdCollision).GroupBy(s => s.AppName).Select(g =>
{
// PR0 classification (see MetricsMath.IsSkipTerminalState / HasMeasuredDuration):
// skips leave the rate + durations; duration stats read measured rows only.
var total = g.Count();
var succeeded = g.Count(s => s.Status == "Succeeded");
var succeededAll = g.Where(s => s.Status == "Succeeded").ToList();
var skipped = succeededAll.Count(MetricsMath.IsSkipTerminalState);
var installed = succeededAll.Where(s => !MetricsMath.IsSkipTerminalState(s)).ToList();
var measured = installed.Where(MetricsMath.HasMeasuredDuration).ToList();
var failed = g.Count(s => s.Status == "Failed");
var completed = g.Where(s => s.Status == "Succeeded").ToList();
var failureRate = MetricsMath.TerminalFailureRatePct(failed, succeeded);
var failureRate = MetricsMath.TerminalFailureRatePct(failed, installed.Count);
var firstHalf = g.Where(s => s.StartedAt < midpoint).ToList();
var secondHalf = g.Where(s => s.StartedAt >= midpoint).ToList();
@@ -158,12 +166,14 @@ namespace AutopilotMonitor.Functions.Functions.Apps
appName = g.Key,
appType = g.Select(s => s.AppType).FirstOrDefault(t => !string.IsNullOrEmpty(t)) ?? string.Empty,
totalInstalls = total,
succeeded,
succeeded = installed.Count,
skipped,
unmeasured = installed.Count - measured.Count,
failed,
failureRate,
avgDurationSeconds = completed.Count > 0 ? Math.Round(completed.Average(s => s.DurationSeconds), 0) : 0,
maxDurationSeconds = completed.Count > 0 ? completed.Max(s => s.DurationSeconds) : 0,
avgDownloadBytes = completed.Count > 0 ? (long)completed.Average(s => s.DownloadBytes) : 0,
avgDurationSeconds = measured.Count > 0 ? Math.Round(measured.Average(s => s.DurationSeconds), 0) : 0,
maxDurationSeconds = measured.Count > 0 ? measured.Max(s => s.DurationSeconds) : 0,
avgDownloadBytes = measured.Count > 0 ? (long)measured.Average(s => s.DownloadBytes) : 0,
trend,
trendDelta,
lastSeenAt = g.Max(s => s.CompletedAt ?? s.StartedAt)
@@ -182,6 +192,7 @@ namespace AutopilotMonitor.Functions.Functions.Apps
success = true,
totalApps = apps.Count,
totalInstalls = summaries.Count,
collisionExcluded,
windowDays = days,
apps
};
@@ -198,6 +209,7 @@ namespace AutopilotMonitor.Functions.Functions.Apps
success = true,
totalApps = apps.Count,
totalInstalls = summaries.Count,
collisionExcluded,
windowDays = days,
count = page.Count,
offset,
@@ -223,11 +235,16 @@ namespace AutopilotMonitor.Functions.Functions.Apps
var cutoff = now.AddDays(-days);
var midpoint = now.AddDays(-days / 2.0);
var summaries = allSummaries
var inWindow = allSummaries
.Where(s => string.Equals(s.AppName, appName, StringComparison.OrdinalIgnoreCase)
&& s.StartedAt >= cutoff)
.ToList();
// F1 PR1 (audit Q3): collision rows under this name mix in a second app's outcomes —
// excluded from every stat below, disclosed via collisionExcluded.
var collisionExcluded = inWindow.Count(s => s.AppIdCollision);
var summaries = inWindow.Where(s => !s.AppIdCollision).ToList();
if (summaries.Count == 0)
{
return new
@@ -236,11 +253,14 @@ namespace AutopilotMonitor.Functions.Functions.Apps
appName,
appType = string.Empty,
windowDays = days,
collisionExcluded,
bucket = "day",
summary = new
{
totalInstalls = 0,
succeeded = 0,
skipped = 0,
unmeasured = 0,
failed = 0,
failureRate = 0,
avgDurationSeconds = 0,
@@ -260,13 +280,17 @@ namespace AutopilotMonitor.Functions.Functions.Apps
}
var total = summaries.Count;
var succeeded = summaries.Count(s => s.Status == "Succeeded");
// PR0 classification — skips leave rate + durations; durations read measured rows only.
var succeededAll = summaries.Where(s => s.Status == "Succeeded").ToList();
var skipped = succeededAll.Count(MetricsMath.IsSkipTerminalState);
var installed = succeededAll.Where(s => !MetricsMath.IsSkipTerminalState(s)).ToList();
var measured = installed.Where(MetricsMath.HasMeasuredDuration).ToList();
var succeeded = installed.Count;
var failed = summaries.Count(s => s.Status == "Failed");
var completed = summaries.Where(s => s.Status == "Succeeded").ToList();
var failureRate = MetricsMath.TerminalFailureRatePct(failed, succeeded);
var avgDurationSeconds = completed.Count > 0 ? Math.Round(completed.Average(s => s.DurationSeconds), 0) : 0;
var p95DurationSeconds = Percentile(completed.Select(s => s.DurationSeconds).ToList(), 0.95);
var avgDownloadBytes = completed.Count > 0 ? (long)completed.Average(s => s.DownloadBytes) : 0;
var avgDurationSeconds = measured.Count > 0 ? Math.Round(measured.Average(s => s.DurationSeconds), 0) : 0;
var p95DurationSeconds = Percentile(measured.Select(s => s.DurationSeconds).ToList(), 0.95);
var avgDownloadBytes = measured.Count > 0 ? (long)measured.Average(s => s.DownloadBytes) : 0;
// Trend (same rule as list endpoint).
var firstHalf = summaries.Where(s => s.StartedAt < midpoint).ToList();
@@ -287,7 +311,8 @@ namespace AutopilotMonitor.Functions.Functions.Apps
{
var vTotal = g.Count();
var vFailed = g.Count(s => s.Status == "Failed");
var vSucceeded = g.Count(s => s.Status == "Succeeded");
// Same PR0 convention as the top-level rate: skips are not attempts.
var vSucceeded = g.Count(s => s.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(s));
return new
{
appVersion = g.Key,
@@ -347,7 +372,8 @@ namespace AutopilotMonitor.Functions.Functions.Apps
{
var modelTotal = g.Count();
var modelFailed = g.Count(x => x.Summary.Status == "Failed");
var modelSucceeded = g.Count(x => x.Summary.Status == "Succeeded");
// PR0 convention: skips are not attempts (same population as the headline rate).
var modelSucceeded = g.Count(x => x.Summary.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(x.Summary));
var modelFailureRate = MetricsMath.TerminalFailureRatePct(modelFailed, modelSucceeded);
var lift = failureRate > 0
? Math.Round(modelFailureRate / failureRate, 2)
@@ -373,11 +399,14 @@ namespace AutopilotMonitor.Functions.Functions.Apps
appName,
appType,
windowDays = days,
collisionExcluded,
bucket,
summary = new
{
totalInstalls = total,
succeeded,
skipped,
unmeasured = installed.Count - measured.Count,
failed,
failureRate,
avgDurationSeconds,
@@ -490,10 +519,13 @@ namespace AutopilotMonitor.Functions.Functions.Apps
private static (string Trend, double? TrendDelta) ComputeFailureTrend(
List<AppInstallSummary> firstHalf, List<AppInstallSummary> secondHalf)
{
// PR0 convention: skips are not attempts — keep the trend on the same population
// as the headline failure rate, or a wave of not-applicable update policies would
// fake an "improving" trend.
var fhFailed = firstHalf.Count(s => s.Status == "Failed");
var fhSucceeded = firstHalf.Count(s => s.Status == "Succeeded");
var fhSucceeded = firstHalf.Count(s => s.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(s));
var shFailed = secondHalf.Count(s => s.Status == "Failed");
var shSucceeded = secondHalf.Count(s => s.Status == "Succeeded");
var shSucceeded = secondHalf.Count(s => s.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(s));
if (fhFailed + fhSucceeded < 5 || shFailed + shSucceeded < 5)
return ("stable", null);
@@ -531,16 +563,17 @@ namespace AutopilotMonitor.Functions.Functions.Apps
var items = kv.Value;
var bTotal = items.Count;
var bFailed = items.Count(s => s.Status == "Failed");
var bSucceeded = items.Count(s => s.Status == "Succeeded");
var bCompleted = items.Where(s => s.Status == "Succeeded").ToList();
// PR0 convention: skips are not attempts; durations read measured rows only.
var bInstalled = items.Where(s => s.Status == "Succeeded" && !MetricsMath.IsSkipTerminalState(s)).ToList();
var bMeasured = bInstalled.Where(MetricsMath.HasMeasuredDuration).ToList();
return (object)new
{
bucketStart = DateTime.SpecifyKind(kv.Key, DateTimeKind.Utc),
installs = bTotal,
succeeded = bSucceeded,
succeeded = bInstalled.Count,
failed = bFailed,
failureRate = MetricsMath.TerminalFailureRatePct(bFailed, bSucceeded),
avgDurationSeconds = bCompleted.Count > 0 ? Math.Round(bCompleted.Average(s => s.DurationSeconds), 0) : 0
failureRate = MetricsMath.TerminalFailureRatePct(bFailed, bInstalled.Count),
avgDurationSeconds = bMeasured.Count > 0 ? Math.Round(bMeasured.Average(s => s.DurationSeconds), 0) : 0
};
})
.ToList();
@@ -0,0 +1,216 @@
using System;
using System.IO;
using System.Net;
using System.Security.Cryptography;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Infrastructure
{
/// <summary>
/// Receives Azure Monitor alert notifications (Common Alert Schema) from an Action Group
/// webhook receiver and re-emits them as ops events, so platform alerts flow through the
/// same OpsEventService → OpsAlertDispatchService fan-out (Telegram/Teams/Slack providers +
/// the portal ops-events list) as every internally raised operational event.
///
/// The delivery path is deliberately independent of the portal: Azure Monitor → this
/// function → providers. A portal (SWA) outage cannot suppress these notifications — the
/// action group's e-mail receiver additionally stays as the backend-independent fallback,
/// because a self-hosted channel cannot alert on its own outage.
///
/// Security: anonymous route (Action Group webhook receivers can only call a bare URI — no
/// custom headers), gated by a shared secret in the query string compared in constant time
/// against the OpsAlertWebhookSecret app setting. Fail-closed: while the setting is unset,
/// every request is rejected. Dispatch to providers only happens if an enabled ops alert
/// rule for EventType "AzureMonitorAlert" matches the mapped severity — the event itself is
/// always stored.
/// </summary>
public class AzureMonitorAlertWebhookFunction
{
internal const string SecretSettingName = "OpsAlertWebhookSecret";
// Common Alert Schema payloads for log-search alerts carry the search query, link URLs
// and dimension sets — a few KB in practice. 64 KB is a generous shape gate, not a throttle.
internal const int MaxContentLength = 64 * 1024;
private readonly OpsEventService _opsEventService;
private readonly ILogger<AzureMonitorAlertWebhookFunction> _logger;
public AzureMonitorAlertWebhookFunction(
OpsEventService opsEventService,
ILogger<AzureMonitorAlertWebhookFunction> logger)
{
_opsEventService = opsEventService;
_logger = logger;
}
[Function("AzureMonitorAlertWebhook")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "ops/alert-webhook")] HttpRequestData req)
{
var configuredSecret = Environment.GetEnvironmentVariable(SecretSettingName);
var providedSecret = System.Web.HttpUtility.ParseQueryString(req.Url.Query ?? string.Empty)["secret"];
if (!SecretMatches(providedSecret, configuredSecret))
{
// Deliberately no detail in the response; the warning makes a misconfigured
// (unset) secret operator-visible in the logs.
_logger.LogWarning(
"Azure Monitor alert webhook rejected: {Reason}",
string.IsNullOrEmpty(configuredSecret) ? "secret not configured (fail-closed)" : "secret mismatch");
return req.CreateResponse(HttpStatusCode.Unauthorized);
}
// Bounded read instead of a Body.Length check: with ASP.NET Core integration the
// request body is a forward-only stream whose Length property throws
// NotSupportedException. Reading at most cap+1 chars detects oversize without ever
// needing to know the total size. (Char-based cap ≈ byte cap for a shape gate.)
string body;
using (var reader = new StreamReader(req.Body, Encoding.UTF8))
{
var buffer = new char[MaxContentLength + 1];
var read = await reader.ReadBlockAsync(buffer, 0, buffer.Length);
if (read > MaxContentLength)
{
_logger.LogWarning("Azure Monitor alert webhook rejected: payload exceeds {Cap} char cap", MaxContentLength);
return req.CreateResponse(HttpStatusCode.BadRequest);
}
body = new string(buffer, 0, read);
}
var alert = TryParse(body);
if (alert == null)
{
_logger.LogWarning("Azure Monitor alert webhook rejected: body is not a Common Alert Schema payload");
return req.CreateResponse(HttpStatusCode.BadRequest);
}
await _opsEventService.RecordAzureMonitorAlertAsync(
alert.AlertRule,
MapSeverity(alert.Severity),
alert.MonitorCondition,
alert.Description,
alert.Severity,
alert.MonitoringService,
alert.TargetResource,
alert.MetricValue);
_logger.LogInformation(
"Azure Monitor alert '{AlertRule}' {Condition} ({Severity}) recorded as ops event",
alert.AlertRule, alert.MonitorCondition, alert.Severity);
return req.CreateResponse(HttpStatusCode.OK);
}
/// <summary>
/// Constant-time secret comparison. Fail-closed: an unset/empty configured secret never
/// matches anything, so the endpoint stays dark until the app setting is provisioned.
/// </summary>
internal static bool SecretMatches(string? provided, string? configured)
{
if (string.IsNullOrEmpty(configured) || string.IsNullOrEmpty(provided))
return false;
return CryptographicOperations.FixedTimeEquals(
Encoding.UTF8.GetBytes(provided),
Encoding.UTF8.GetBytes(configured));
}
/// <summary>
/// Maps Azure Monitor severities (Sev0 = most severe … Sev4 = verbose) onto the ops event
/// severity scale. Unknown values fall back to Info so a schema drift can never escalate.
/// </summary>
internal static string MapSeverity(string? azureSeverity) => azureSeverity switch
{
"Sev0" => Shared.DataAccess.OpsEventSeverity.Critical,
"Sev1" => Shared.DataAccess.OpsEventSeverity.Error,
"Sev2" => Shared.DataAccess.OpsEventSeverity.Warning,
_ => Shared.DataAccess.OpsEventSeverity.Info,
};
/// <summary>
/// Parsed subset of the Common Alert Schema. Only <see cref="AlertRule"/> is mandatory;
/// everything else is best-effort so a partial payload still produces a usable ops event.
/// </summary>
internal sealed record ParsedAlert(
string AlertRule,
string? Severity,
string MonitorCondition,
string? Description,
string? MonitoringService,
string? TargetResource,
double? MetricValue);
/// <summary>
/// Defensively extracts data.essentials (+ the first condition's metricValue) from a
/// Common Alert Schema payload. Returns null when the body is not JSON or carries no
/// alert rule name — the two states in which no meaningful ops event can be formed.
/// </summary>
internal static ParsedAlert? TryParse(string body)
{
if (string.IsNullOrWhiteSpace(body))
return null;
try
{
using var doc = JsonDocument.Parse(body);
if (!doc.RootElement.TryGetProperty("data", out var data) ||
!data.TryGetProperty("essentials", out var essentials))
return null;
var alertRule = GetString(essentials, "alertRule");
if (string.IsNullOrWhiteSpace(alertRule))
return null;
var targetResource = GetString(essentials, "targetResourceName");
if (string.IsNullOrEmpty(targetResource) &&
essentials.TryGetProperty("alertTargetIDs", out var targets) &&
targets.ValueKind == JsonValueKind.Array && targets.GetArrayLength() > 0 &&
targets[0].ValueKind == JsonValueKind.String)
{
// Fall back to the resource name segment of the first target resource ID.
var id = targets[0].GetString() ?? string.Empty;
targetResource = id[(id.LastIndexOf('/') + 1)..];
}
// Log-search alerts carry the observed value under alertContext.condition.allOf[0].
double? metricValue = null;
if (data.TryGetProperty("alertContext", out var ctx) &&
ctx.ValueKind == JsonValueKind.Object &&
ctx.TryGetProperty("condition", out var condition) &&
condition.ValueKind == JsonValueKind.Object &&
condition.TryGetProperty("allOf", out var allOf) &&
allOf.ValueKind == JsonValueKind.Array && allOf.GetArrayLength() > 0 &&
allOf[0].ValueKind == JsonValueKind.Object &&
allOf[0].TryGetProperty("metricValue", out var mv) &&
mv.ValueKind == JsonValueKind.Number)
{
metricValue = mv.GetDouble();
}
return new ParsedAlert(
alertRule!,
GetString(essentials, "severity"),
GetString(essentials, "monitorCondition") ?? "Fired",
GetString(essentials, "description"),
GetString(essentials, "monitoringService"),
targetResource,
metricValue);
}
catch (JsonException)
{
return null;
}
}
private static string? GetString(JsonElement element, string property)
=> element.TryGetProperty(property, out var value) && value.ValueKind == JsonValueKind.String
? value.GetString()
: null;
}
}
@@ -0,0 +1,358 @@
using System.Net;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Metrics
{
/// <summary>
/// F2 device-history / First-Time-Right read surfaces (insights spec §F2 "Surfaces", PR5).
/// All endpoints serve rows the terminal seam + maintenance sweep maintain — the journey
/// semantics live server-side (<see cref="DeviceJourneyCalculator"/>); no consumer re-derives
/// them. Durations in chain refs are the sessions' authoritative DurationSeconds verbatim.
/// </summary>
public class GetDeviceHistoryFunction
{
private readonly ILogger<GetDeviceHistoryFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
private readonly ISessionRepository _sessionRepo;
public GetDeviceHistoryFunction(
ILogger<GetDeviceHistoryFunction> logger,
IMetricsRepository metricsRepo,
ISessionRepository sessionRepo)
{
_logger = logger;
_metricsRepo = metricsRepo;
_sessionRepo = sessionRepo;
}
/// <summary>
/// One device's enrollment history by serial number. history=null is a NORMAL outcome
/// (unknown device, junk/placeholder serial, or every ref pruned) — the banner simply
/// stays hidden. With ?sessionId= the response additionally carries that session's
/// attempt number within its journey (server-computed; works for live sessions too via
/// the virtual-attempt rule). Serves the session-detail banner and the MCP tool; a
/// Global Admin passes ?tenantId= for cross-tenant reads (TenantScoping.QueryParam),
/// with the sessionId-based owner fallback mirroring GetSessionTimeAttribution.
/// </summary>
[Function("GetDeviceHistory")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "metrics/device-history")] HttpRequestData req)
{
try
{
// Authentication + MemberRead authorization enforced by PolicyEnforcementMiddleware;
// cross-tenant access via TargetTenantId (TenantScoping.QueryParam).
var requestCtx = req.GetRequestContext();
var query = System.Web.HttpUtility.ParseQueryString(req.Url.Query);
var rawSerial = query["serialNumber"];
var sessionId = query["sessionId"];
if (string.IsNullOrWhiteSpace(rawSerial))
{
var badRequest = req.CreateResponse(HttpStatusCode.BadRequest);
await badRequest.WriteAsJsonAsync(new { success = false, message = "serialNumber is required" });
return badRequest;
}
// Junk/placeholder serials have no device identity — by design they never have a
// chain (disclosed in the fleet aggregate instead), so null here is honest.
var serialKey = DeviceJourneyCalculator.NormalizeSerial(rawSerial);
var effectiveTenantId = requestCtx.TargetTenantId;
var history = serialKey == null
? null
: await _metricsRepo.GetDeviceHistoryAsync(effectiveTenantId, serialKey);
// Global-scope fallback (mirrors GetSessionTimeAttribution): a GA/operator opening
// a session without an explicit tenantId query param reads their home partition —
// resolve the owning tenant via the session and retry once.
if (history == null && serialKey != null && requestCtx.HasGlobalScope && !string.IsNullOrEmpty(sessionId))
{
var resolvedTenantId = await _sessionRepo.FindSessionTenantIdAsync(sessionId!);
if (resolvedTenantId != null &&
!string.Equals(resolvedTenantId, effectiveTenantId, StringComparison.OrdinalIgnoreCase))
{
effectiveTenantId = resolvedTenantId;
history = await _metricsRepo.GetDeviceHistoryAsync(effectiveTenantId, serialKey);
}
}
// Attempt number for the requesting session — fail-soft: a missing session or an
// empty basis yields null, never a guessed position.
int? attemptNumber = null;
if (history != null && !string.IsNullOrEmpty(sessionId))
{
var session = await _sessionRepo.GetSessionAsync(effectiveTenantId, sessionId!);
if (session != null)
attemptNumber = DeviceJourneyCalculator.ComputeAttemptNumber(history.Chain, sessionId!, session.StartedAt);
}
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { success = true, history, attemptNumber });
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error fetching device history");
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
await errorResponse.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
return errorResponse;
}
}
}
/// <summary>
/// Tenant fleet view: daily First-Time-Right rows of the requested window plus their sums,
/// the merged attempt histogram, and the "repeat devices" violator list. FTR counts are
/// additive, so unlike the median-based time-attribution panel this endpoint honors the
/// page's days selector honestly (window rate = sum over daily rows).
/// </summary>
public class GetDeviceJourneyMetricsFunction
{
private readonly ILogger<GetDeviceJourneyMetricsFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
private readonly ISessionRepository _sessionRepo;
public GetDeviceJourneyMetricsFunction(
ILogger<GetDeviceJourneyMetricsFunction> logger,
IMetricsRepository metricsRepo,
ISessionRepository sessionRepo)
{
_logger = logger;
_metricsRepo = metricsRepo;
_sessionRepo = sessionRepo;
}
[Function("GetDeviceJourneyMetrics")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "metrics/device-journeys")] HttpRequestData req)
{
try
{
// Authentication + MemberRead authorization enforced by PolicyEnforcementMiddleware
var tenantId = TenantHelper.GetTenantId(req);
var query = System.Web.HttpUtility.ParseQueryString(req.Url.Query);
var days = DeviceJourneyMetricsResponse.ClampDays(query["days"]);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(await DeviceJourneyMetricsResponse.BuildAsync(
_metricsRepo, _sessionRepo, tenantId, days, includeRepeatDevices: true));
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error fetching device journey metrics");
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
await errorResponse.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
return errorResponse;
}
}
}
/// <summary>
/// Global-admin variant: tenantId filters to one tenant (full response incl. repeat
/// devices); absent = the cross-tenant "global" aggregate rows, WITHOUT a repeat-devices
/// list — that would require scanning every tenant's DeviceHistories partition, and the
/// aggregated view has no per-device drill anyway. repeatDevices=null discloses the gap.
/// </summary>
public class GetGlobalDeviceJourneyMetricsFunction
{
private readonly ILogger<GetGlobalDeviceJourneyMetricsFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
private readonly ISessionRepository _sessionRepo;
public GetGlobalDeviceJourneyMetricsFunction(
ILogger<GetGlobalDeviceJourneyMetricsFunction> logger,
IMetricsRepository metricsRepo,
ISessionRepository sessionRepo)
{
_logger = logger;
_metricsRepo = metricsRepo;
_sessionRepo = sessionRepo;
}
[Function("GetGlobalDeviceJourneyMetrics")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "global/metrics/device-journeys")] HttpRequestData req)
{
try
{
// Authentication + GlobalReadOrAdmin authorization enforced by PolicyEnforcementMiddleware
var query = System.Web.HttpUtility.ParseQueryString(req.Url.Query);
var tenantIdFilter = query["tenantId"];
var days = DeviceJourneyMetricsResponse.ClampDays(query["days"]);
var partition = string.IsNullOrWhiteSpace(tenantIdFilter) ? "global" : tenantIdFilter!;
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(await DeviceJourneyMetricsResponse.BuildAsync(
_metricsRepo, _sessionRepo, partition, days, includeRepeatDevices: partition != "global"));
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error fetching global device journey metrics");
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
await errorResponse.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
return errorResponse;
}
}
}
/// <summary>Shared response shape for the tenant and global variants (mirrors TimeAttributionResponse).</summary>
internal static class DeviceJourneyMetricsResponse
{
internal const int DefaultWindowDays = 30;
internal const int MaxWindowDays = 180; // aggregate retention — older rows no longer exist
internal const int MaxRepeatDevices = 10;
internal static int ClampDays(string? raw)
{
if (!int.TryParse(raw, out var days)) return DefaultWindowDays;
return Math.Clamp(days, 1, MaxWindowDays);
}
/// <summary>
/// "Last N days" = exactly N calendar day keys including today — both range ends are
/// inclusive, so subtracting the full N would return N+1 days (Codex review: days=1
/// summed yesterday AND today).
/// </summary>
internal static DateTime InclusiveWindowStart(DateTime today, int days) => today.AddDays(-(days - 1));
internal static async Task<object> BuildAsync(
IMetricsRepository metricsRepo, ISessionRepository sessionRepo,
string partition, int days, bool includeRepeatDevices)
{
var today = DateTime.UtcNow.Date;
var daily = await metricsRepo.GetDeviceJourneyAggregatesAsync(partition, InclusiveWindowStart(today, days), today);
var totals = SumAggregates(daily);
List<object>? repeatDevices = null;
if (includeRepeatDevices)
{
var histories = await metricsRepo.GetDeviceHistoriesByTenantAsync(partition);
var candidates = SelectRepeatDevices(histories, today.AddDays(-days));
repeatDevices = new List<object>(candidates.Count);
foreach (var candidate in candidates)
{
// Failure reason of the newest failed attempt — a bounded point-read per top
// row (≤10), fail-soft: an unreadable/deleted session just leaves it empty.
var lastFailureReason = string.Empty;
if (candidate.NewestFailed != null)
{
try
{
var failedSession = await sessionRepo.GetSessionAsync(partition, candidate.NewestFailed.SessionId);
lastFailureReason = failedSession?.FailureReason ?? string.Empty;
}
catch
{
// fail-soft — the row renders without a reason
}
}
repeatDevices.Add(new
{
serialNumber = candidate.History.SerialNumber,
manufacturer = candidate.History.Manufacturer,
model = candidate.History.Model,
attempts = candidate.History.CurrentJourneyAttempts,
journeyCount = candidate.History.JourneyCount,
lastStatus = candidate.Newest.Status,
lastSessionId = candidate.Newest.SessionId,
lastStartedAt = candidate.Newest.StartedAt,
lastFailureReason,
});
}
}
return new
{
success = true,
windowDays = days,
totals,
daily = daily.OrderBy(d => d.Date, StringComparer.Ordinal).ToList(),
repeatDevices,
};
}
/// <summary>Window totals (serialized camelCase by the worker's global serializer, like the model rows).</summary>
internal sealed class DeviceJourneyWindowTotals
{
public int CompletedJourneys { get; init; }
public int FirstTimeRight { get; init; }
/// <summary>Null with zero completed journeys — no rate claim, never 0 (truthfulness rule 1).</summary>
public double? FtrRatePct { get; init; }
public int ExcludedSessions { get; init; }
public List<DeviceJourneyAttemptBucket> AttemptHistogram { get; init; } = new();
}
/// <summary>
/// Window sums over the daily rows — honest because every field is an additive count
/// (rule: the FTR window rate IS the ratio of summed counts). internal static: pinned
/// by unit tests.
/// </summary>
internal static DeviceJourneyWindowTotals SumAggregates(IReadOnlyList<DeviceJourneyDailyAggregate> daily)
{
var completed = daily.Sum(d => d.CompletedJourneyCount);
var firstTimeRight = daily.Sum(d => d.FirstTimeRightCount);
return new DeviceJourneyWindowTotals
{
CompletedJourneys = completed,
FirstTimeRight = firstTimeRight,
FtrRatePct = completed > 0 ? Math.Round(firstTimeRight * 100.0 / completed, 1) : null,
ExcludedSessions = daily.Sum(d => d.ExcludedSessionCount),
AttemptHistogram = daily
.SelectMany(d => d.AttemptHistogram)
.GroupBy(b => b.Attempts)
.Select(g => new DeviceJourneyAttemptBucket { Attempts = g.Key, JourneyCount = g.Sum(b => b.JourneyCount) })
.OrderBy(b => b.Attempts)
.ToList(),
};
}
internal sealed class RepeatDeviceCandidate
{
public DeviceHistory History { get; init; } = null!;
public DeviceSessionRef Newest { get; init; } = null!;
public DeviceSessionRef? NewestFailed { get; init; }
}
/// <summary>
/// Violator selection (spec: "repeat devices" table): devices whose LAST journey took
/// ≥2 attempts and whose newest terminal session falls inside the window — current pain,
/// not ancient history (a device that retried long ago but enrolls cleanly today does
/// not belong on a violator list). Ordered by attempts desc, then recency; capped at
/// <see cref="MaxRepeatDevices"/>. internal static: pinned by unit tests.
/// </summary>
internal static List<RepeatDeviceCandidate> SelectRepeatDevices(
IReadOnlyList<DeviceHistory> histories, DateTime windowStartUtc)
{
var candidates = new List<RepeatDeviceCandidate>();
foreach (var history in histories)
{
if (history.CurrentJourneyAttempts < 2 || history.Chain.Count == 0)
continue;
var newest = history.Chain[history.Chain.Count - 1];
if (newest.StartedAt < windowStartUtc)
continue;
candidates.Add(new RepeatDeviceCandidate
{
History = history,
Newest = newest,
NewestFailed = history.Chain.LastOrDefault(r =>
string.Equals(r.Status, nameof(SessionStatus.Failed), StringComparison.OrdinalIgnoreCase)),
});
}
return candidates
.OrderByDescending(c => c.History.CurrentJourneyAttempts)
.ThenByDescending(c => c.Newest.StartedAt)
.Take(MaxRepeatDevices)
.ToList();
}
}
}
@@ -153,6 +153,18 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
.Where(s => appsBySession.ContainsKey(s.SessionId))
.Count(s => appsBySession[s.SessionId].Any(a => a.DoDownloadMode >= 0));
// Agent→backend API latency: weight each session's stored average by its request
// count so the location figure equals Σ(latency·requests)/Σ(requests) — the exact
// whole-population average, not an average of session averages. Sessions from
// agents predating the feature carry no data and drop out.
var latencySessions = group
.Where(s => s.AvgApiLatencyMs is > 0 && s.ApiRequestCount is > 0)
.ToList();
var latencyRequestSum = latencySessions.Sum(s => (double)s.ApiRequestCount!.Value);
var avgApiLatency = latencyRequestSum > 0
? latencySessions.Sum(s => s.AvgApiLatencyMs!.Value * s.ApiRequestCount!.Value) / latencyRequestSum
: 0;
locations.Add(new LocationMetrics
{
LocationKey = group.Key,
@@ -171,6 +183,8 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
MinutesPerApp = Math.Round(minutesPerApp, 2),
AvgThroughputBytesPerSec = Math.Round(avgThroughput, 0),
TotalDownloadBytes = totalBytes,
AvgApiLatencyMs = Math.Round(avgApiLatency, 0),
ApiLatencySessionCount = latencySessions.Count,
// Delivery Optimization
DoSessionCount = doSessionCount,
AvgDoPercentPeerCaching = Math.Round(doAgg.PercentPeerCaching, 1),
@@ -202,6 +216,17 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
.Select(l => l.AvgThroughputBytesPerSec).ToList();
var globalAvgThroughput = allThroughputs.Count > 0 ? allThroughputs.Average() : 0;
// Global API latency: request-weighted over ALL geo sessions (not location averages),
// so big sites weigh in proportionally — same Σ(latency·requests)/Σ(requests) math as
// the per-location figure.
var globalLatencySessions = geoSessions
.Where(s => s.AvgApiLatencyMs is > 0 && s.ApiRequestCount is > 0)
.ToList();
var globalLatencyRequestSum = globalLatencySessions.Sum(s => (double)s.ApiRequestCount!.Value);
var globalAvgApiLatency = globalLatencyRequestSum > 0
? globalLatencySessions.Sum(s => s.AvgApiLatencyMs!.Value * s.ApiRequestCount!.Value) / globalLatencyRequestSum
: 0;
// Global DO metrics (weighted by bytes, not simple average)
var globalDoPeers = locations.Sum(l => l.TotalDoBytesFromPeers);
var globalDoHttp = locations.Sum(l => l.TotalDoBytesFromHttp);
@@ -219,6 +244,10 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
loc.ThroughputVsGlobalPct = globalAvgThroughput > 0
? Math.Round((loc.AvgThroughputBytesPerSec - globalAvgThroughput) / globalAvgThroughput * 100, 1) : 0;
// API latency vs global (positive = slower/farther from the backend region)
loc.ApiLatencyVsGlobalPct = globalAvgApiLatency > 0 && loc.AvgApiLatencyMs > 0
? Math.Round((loc.AvgApiLatencyMs - globalAvgApiLatency) / globalAvgApiLatency * 100, 1) : 0;
// AppLoadScore: normalize minutesPerApp to global median = 100
loc.AppLoadScore = globalMedianMinutesPerApp > 0
? Math.Round(loc.MinutesPerApp / globalMedianMinutesPerApp * 100, 0) : 0;
@@ -243,6 +272,7 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
AvgMinutesPerApp = Math.Round(globalAvgMinutesPerApp, 2),
AvgThroughputBytesPerSec = Math.Round(globalAvgThroughput, 0),
StdDevDurationMinutes = Math.Round(globalStdDev, 1),
AvgApiLatencyMs = Math.Round(globalAvgApiLatency, 0),
AvgDoPercentPeerCaching = Math.Round(globalDoPct, 1),
TotalDoBytesFromPeers = globalDoPeers,
TotalDoBytesFromHttp = globalDoHttp
@@ -1,6 +1,7 @@
using System.Net;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
@@ -34,24 +35,7 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
var versions = await _sessionRepo.GetImeVersionHistoryAsync();
var response = req.CreateResponse(HttpStatusCode.OK);
if (requestCtx.HasGlobalScope)
{
await response.WriteAsJsonAsync(versions);
}
else
{
// Strip sensitive cross-tenant data for non-Global Admins
var redacted = versions.Select(v => new
{
v.Version,
v.FirstSeenAt,
v.LastSeenAt,
v.SessionCount
});
await response.WriteAsJsonAsync(redacted);
}
await response.WriteAsJsonAsync(BuildResponsePayload(versions, requestCtx.HasGlobalScope));
return response;
}
catch (Exception ex)
@@ -62,5 +46,35 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
return errorResponse;
}
}
/// <summary>
/// Projects the archive for the caller's scope.
///
/// Extracted from <see cref="Run"/> so the cross-tenant redaction boundary is
/// directly testable: this suite has no HTTP-level harness, so an inline projection
/// is unreachable from a test — and an unguarded redaction is exactly what a later
/// refactor opens silently.
///
/// FirstSeenTenantId and FirstSeenSessionId name ANOTHER tenant's device and are
/// dropped for non-global callers. Version, dates and SessionCount deliberately stay:
/// the archive is a platform-wide view of Microsoft's IME rollout, and the route is
/// MemberRead precisely so any tenant member can read that view.
///
/// Returned as <see cref="object"/> on purpose — System.Text.Json serializes an
/// `object` declared type using the RUNTIME type, so both branches keep the exact
/// wire format they had while the projection was inline.
/// </summary>
public static object BuildResponsePayload(
IEnumerable<ImeVersionHistoryEntry> versions, bool hasGlobalScope)
{
if (hasGlobalScope)
{
return versions.ToList();
}
return versions
.Select(v => new { v.Version, v.FirstSeenAt, v.LastSeenAt, v.SessionCount })
.ToList();
}
}
}
@@ -0,0 +1,196 @@
using System.Net;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.DataAccess;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Metrics
{
/// <summary>
/// F1 time-attribution read surfaces (insights spec §F1 "Surfaces", PR3). All three
/// endpoints serve PRE-COMPUTED rows — the per-session breakdown written at the terminal
/// transition and the aggregate rows the maintenance sweep maintains. No request-time
/// derivation: what the calculator computed is what every consumer (web, MCP) sees.
/// </summary>
public class GetSessionTimeAttributionFunction
{
private readonly ILogger<GetSessionTimeAttributionFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
private readonly ISessionRepository _sessionRepo;
public GetSessionTimeAttributionFunction(
ILogger<GetSessionTimeAttributionFunction> logger,
IMetricsRepository metricsRepo,
ISessionRepository sessionRepo)
{
_logger = logger;
_metricsRepo = metricsRepo;
_sessionRepo = sessionRepo;
}
/// <summary>
/// Per-session breakdown for the session-detail attribution lane. A missing row is a
/// NORMAL outcome (pre-feature session, non-terminal, Incomplete — no wall clock) and
/// returns success with breakdown=null so the UI simply omits the lane.
/// </summary>
[Function("GetSessionTimeAttribution")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "sessions/{sessionId}/time-attribution")] HttpRequestData req,
string sessionId)
{
try
{
// Authentication + MemberRead authorization enforced by PolicyEnforcementMiddleware;
// cross-tenant access via TargetTenantId (TenantScoping.QueryParam).
var requestCtx = req.GetRequestContext();
var breakdown = await _metricsRepo.GetSessionTimeBreakdownAsync(requestCtx.TargetTenantId, sessionId);
// Global-scope fallback (mirrors GetSessionEventsFunction): a GA/operator opening
// a session without an explicit tenantId query param reads their home partition —
// resolve the owning tenant and retry once.
if (breakdown == null && requestCtx.HasGlobalScope)
{
var resolvedTenantId = await _sessionRepo.FindSessionTenantIdAsync(sessionId);
if (resolvedTenantId != null &&
!string.Equals(resolvedTenantId, requestCtx.TargetTenantId, StringComparison.OrdinalIgnoreCase))
{
breakdown = await _metricsRepo.GetSessionTimeBreakdownAsync(resolvedTenantId, sessionId);
}
}
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { success = true, breakdown });
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error fetching time attribution for session {SessionId}", sessionId);
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
await errorResponse.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
return errorResponse;
}
}
}
/// <summary>
/// Tenant fleet view: the rolling 30-day range statistics per enrollment class (the panel's
/// stacked medians + top blocking apps) plus the daily rows for the per-day trend. The range
/// window is FIXED at the sweep's 30 days — daily medians cannot be merged into an arbitrary
/// range honestly, so no days parameter is offered.
/// </summary>
public class GetTimeAttributionMetricsFunction
{
private readonly ILogger<GetTimeAttributionMetricsFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
public GetTimeAttributionMetricsFunction(
ILogger<GetTimeAttributionMetricsFunction> logger,
IMetricsRepository metricsRepo)
{
_logger = logger;
_metricsRepo = metricsRepo;
}
[Function("GetTimeAttributionMetrics")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "metrics/time-attribution")] HttpRequestData req)
{
try
{
// Authentication + MemberRead authorization enforced by PolicyEnforcementMiddleware
var tenantId = TenantHelper.GetTenantId(req);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(await TimeAttributionResponse.BuildAsync(_metricsRepo, tenantId));
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error fetching time attribution metrics");
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
await errorResponse.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
return errorResponse;
}
}
}
/// <summary>Global-admin variant: tenantId filters to one tenant; absent = cross-tenant "global" rows.</summary>
public class GetGlobalTimeAttributionMetricsFunction
{
private readonly ILogger<GetGlobalTimeAttributionMetricsFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
public GetGlobalTimeAttributionMetricsFunction(
ILogger<GetGlobalTimeAttributionMetricsFunction> logger,
IMetricsRepository metricsRepo)
{
_logger = logger;
_metricsRepo = metricsRepo;
}
[Function("GetGlobalTimeAttributionMetrics")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "global/metrics/time-attribution")] HttpRequestData req)
{
try
{
// Authentication + GlobalReadOrAdmin authorization enforced by PolicyEnforcementMiddleware
var query = System.Web.HttpUtility.ParseQueryString(req.Url.Query);
var tenantIdFilter = query["tenantId"];
var partition = string.IsNullOrWhiteSpace(tenantIdFilter) ? "global" : tenantIdFilter!;
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(await TimeAttributionResponse.BuildAsync(_metricsRepo, partition));
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error fetching global time attribution metrics");
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
await errorResponse.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
return errorResponse;
}
}
}
/// <summary>Shared response shape for the tenant and global variants (mirrors the MetricsMath single-source pattern).</summary>
internal static class TimeAttributionResponse
{
internal const int WindowDays = 30;
/// <summary>
/// Rolling rows are rewritten by every 2h sweep as long as the partition has window
/// sessions; a partition that went fully quiet stops being rewritten, so its rows would
/// otherwise serve arbitrarily old numbers as "last 30 days" (Codex review). 48h
/// tolerates a full day of sweep outage before the panel honestly goes empty.
/// </summary>
internal static readonly TimeSpan RollingMaxAge = TimeSpan.FromHours(48);
/// <summary>
/// "Last N days" = exactly N calendar day keys including today — both range ends are
/// inclusive, so subtracting the full N would return N+1 days (Codex review).
/// </summary>
internal static DateTime InclusiveWindowStart(DateTime today, int days) => today.AddDays(-(days - 1));
internal static async Task<object> BuildAsync(IMetricsRepository metricsRepo, string partition)
{
var freshCutoff = DateTime.UtcNow - RollingMaxAge;
var rolling = (await metricsRepo.GetRollingTimeAttributionAggregatesAsync(partition))
.Where(r => r.ComputedAt >= freshCutoff)
.ToList();
var today = DateTime.UtcNow.Date;
var daily = await metricsRepo.GetTimeAttributionAggregatesAsync(partition, InclusiveWindowStart(today, WindowDays), today);
return new
{
success = true,
windowDays = WindowDays,
// Range statistics per enrollment class (never mixed) — the UI gates rendering
// at cleanSessionCount >= 20 and shows "insufficient data (n=…)" below it.
classes = rolling.OrderBy(r => r.EnrollmentClass, StringComparer.Ordinal).ToList(),
daily = daily.OrderBy(d => d.Date, StringComparer.Ordinal).ToList(),
};
}
}
}
@@ -16,13 +16,16 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
{
private readonly ILogger<RuleStatsFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
private readonly IHardwareRejectionNotificationTracker _notificationTracker;
public RuleStatsFunction(
ILogger<RuleStatsFunction> logger,
IMetricsRepository metricsRepo)
IMetricsRepository metricsRepo,
IHardwareRejectionNotificationTracker notificationTracker)
{
_logger = logger;
_metricsRepo = metricsRepo;
_notificationTracker = notificationTracker;
}
/// <summary>
@@ -81,6 +84,9 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
var result = new
{
rules = aggregated,
// F3 PR6: active regression episodes (tracker rows) — the rules-page badge
// and MCP get_rule_stats read them from here. Empty list = nothing regressed.
regressions = await _notificationTracker.GetRuleRegressionsAsync(tenantId),
summary = new
{
totalEvaluations,
@@ -165,6 +171,11 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
var result = new
{
rules = aggregated,
// Regression episodes are tenant-scoped: present when drilling into one
// tenant, empty for the cross-tenant "global" aggregate (no episode there).
regressions = queryTenantId != "global"
? await _notificationTracker.GetRuleRegressionsAsync(queryTenantId)
: new List<AutopilotMonitor.Shared.Models.RuleRegressionAlert>(),
summary = new
{
totalEvaluations,
@@ -0,0 +1,244 @@
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Helpers;
/// <summary>
/// One journey (a single deployment intent for one device): consecutive terminal attempts of a
/// device key, ended by the first terminal success. Transient grouping result — never persisted.
/// </summary>
public sealed class DeviceJourney
{
/// <summary>Terminal session refs of this journey, ordered by StartedAt ascending.</summary>
public List<DeviceSessionRef> Attempts { get; } = new();
/// <summary>True when the journey ended with a terminal success (its last attempt Succeeded).</summary>
public bool Completed { get; set; }
/// <summary>The success ref that completed the journey; null while open/abandoned.</summary>
public DeviceSessionRef? CompletingRef => Completed ? Attempts[Attempts.Count - 1] : null;
}
/// <summary>
/// Deterministic, I/O-free core of the F2 device-history / First-Time-Right feature
/// (insights spec §F2 — JourneyVersion 1). Owns the device-key normalization (trim +
/// case-fold + junk-serial exclusion), the terminal-only chain maintenance (upsert by
/// sessionId, cap 20 most recent) and the journey grouping:
/// <list type="bullet">
/// <item>Terminal set = Succeeded / Failed / Incomplete (production-verified status catalogue).
/// A WhiteGlove <c>Pending</c> row is an OPEN session — WG part 1 + part 2 share ONE session
/// row (second-sweep finding), so journeys never stitch WG parts and a WG device still waiting
/// for its user session leaves the journey open, never failed.</item>
/// <item>A journey ends with the first terminal SUCCESS; the next session starts a new journey
/// (redeployment). A gap of &gt; 30 days since the previous terminal session also starts a new
/// journey (device shelved/repurposed) — production gaps: median 4.9 h, 92 % &lt; 7 d.</item>
/// <item>Attempt count = terminal sessions in the journey; Incomplete and Failed are
/// non-successful attempts.</item>
/// </list>
/// </summary>
public static class DeviceJourneyCalculator
{
/// <summary>Bump on any grouping-semantics change so aggregates never mix definitions (truthfulness rule 8).</summary>
public const int CurrentVersion = 1;
/// <summary>Chain cap: the 20 most recent terminal sessions (spec §F2 — journey counts are chain-scoped).</summary>
internal const int MaxChainLength = 20;
/// <summary>New-journey boundary without a prior success. Constant in v1, not a setting (spec §F2).</summary>
internal const int JourneyGapDays = 30;
/// <summary>
/// Placeholder serials that are not device identities (spec §F2 + audit Q5: "Unknown" is the
/// agent's WMI-failure sentinel). Compared against the NORMALIZED (trimmed, lower-cased)
/// serial; anything shorter than 4 characters is junk regardless.
/// </summary>
private static readonly HashSet<string> JunkSerials = new(StringComparer.Ordinal)
{
"system serial number",
"to be filled by o.e.m.",
"default string",
"0",
"none",
"invalid",
"unknown",
};
/// <summary>
/// Normalizes a raw serial to the device-key form (trim + lower-case invariant) or returns
/// null for junk/placeholder serials — excluded devices never enter FTR and never get a
/// history chain (the exclusion is disclosed in the daily aggregate instead).
/// </summary>
public static string? NormalizeSerial(string? rawSerial)
{
if (string.IsNullOrWhiteSpace(rawSerial)) return null;
var normalized = rawSerial.Trim().ToLowerInvariant();
if (normalized.Length < 4) return null;
if (JunkSerials.Contains(normalized)) return null;
return normalized;
}
/// <summary>F2 terminal set — everything else (InProgress, Pending, Stalled, AwaitingUser, Unknown) is an open session.</summary>
public static bool IsTerminal(SessionStatus status)
=> status == SessionStatus.Succeeded || status == SessionStatus.Failed || status == SessionStatus.Incomplete;
/// <summary>
/// Builds the chain ref for a terminal session; null for non-terminal statuses (defense in
/// depth — callers gate on <see cref="IsTerminal"/> already). Duration is the session's
/// authoritative <c>DurationSeconds</c> verbatim, never CompletedAt − StartedAt.
/// </summary>
public static DeviceSessionRef? BuildSessionRef(SessionSummary session)
{
if (!IsTerminal(session.Status)) return null;
return new DeviceSessionRef
{
SessionId = session.SessionId,
StartedAt = session.StartedAt,
CompletedAt = session.CompletedAt,
Status = session.Status.ToString(),
EnrollmentType = string.IsNullOrEmpty(session.EnrollmentType) ? "v1" : session.EnrollmentType,
IsPreProvisioned = session.IsPreProvisioned,
DurationSeconds = session.DurationSeconds,
AdminMarked = !string.IsNullOrEmpty(session.AdminMarkedAction),
};
}
/// <summary>
/// Merges refs into an existing chain: upsert by sessionId (a re-terminal reclassification,
/// e.g. Incomplete → Succeeded, replaces the entry), non-terminal refs dropped defensively,
/// ordered by StartedAt (tiebreak sessionId, ordinal) and capped to the
/// <see cref="MaxChainLength"/> MOST RECENT entries. Pure — inputs are not mutated.
/// </summary>
public static List<DeviceSessionRef> MergeChain(
IReadOnlyList<DeviceSessionRef>? existing, IEnumerable<DeviceSessionRef> updates)
{
var bySessionId = new Dictionary<string, DeviceSessionRef>(StringComparer.Ordinal);
if (existing != null)
{
foreach (var reference in existing)
{
if (!string.IsNullOrEmpty(reference.SessionId))
bySessionId[reference.SessionId] = reference;
}
}
foreach (var update in updates)
{
if (update != null && !string.IsNullOrEmpty(update.SessionId))
bySessionId[update.SessionId] = update;
}
return SortAndCap(bySessionId.Values.Where(IsTerminalRef));
}
/// <summary>
/// Drops refs whose sessions were deleted (tombstoned or cascaded away) and re-normalizes
/// the chain. An empty result means the device has no observable history left — the caller
/// deletes the row rather than keeping an empty claim.
/// </summary>
public static List<DeviceSessionRef> RemoveSessionRefs(
IReadOnlyList<DeviceSessionRef> chain, ISet<string> deletedSessionIds)
=> SortAndCap(chain.Where(r => IsTerminalRef(r) && !deletedSessionIds.Contains(r.SessionId)));
private static List<DeviceSessionRef> SortAndCap(IEnumerable<DeviceSessionRef> refs)
{
var sorted = refs
.OrderBy(r => r.StartedAt)
.ThenBy(r => r.SessionId, StringComparer.Ordinal)
.ToList();
if (sorted.Count > MaxChainLength)
sorted.RemoveRange(0, sorted.Count - MaxChainLength);
return sorted;
}
/// <summary>
/// A chain entry parses as terminal. Chains only ever receive terminal refs, but the JSON
/// column is data, not a proof — parse defensively (wire-type lesson: mirror the canonical
/// reader, never assume).
/// </summary>
private static bool IsTerminalRef(DeviceSessionRef reference)
=> Enum.TryParse<SessionStatus>(reference.Status, ignoreCase: true, out var status) && IsTerminal(status);
/// <summary>
/// Groups an ordered terminal chain into journeys. The chain is re-sorted defensively;
/// non-terminal refs are ignored. See the class doc for the grouping rules.
/// </summary>
public static List<DeviceJourney> GroupJourneys(IReadOnlyList<DeviceSessionRef> chain)
{
var ordered = SortAndCap(chain.Where(IsTerminalRef));
var journeys = new List<DeviceJourney>();
DeviceJourney? current = null;
DeviceSessionRef? previous = null;
foreach (var reference in ordered)
{
var startNew =
current == null
|| current.Completed
// Gap rule: > 30 days between the previous attempt's terminal moment and this
// start. CompletedAt is the honest "last seen terminal" timestamp; StartedAt is
// the defensive fallback (production: 0 terminal sessions without CompletedAt).
|| (previous != null && reference.StartedAt - (previous.CompletedAt ?? previous.StartedAt) > TimeSpan.FromDays(JourneyGapDays));
if (startNew)
{
current = new DeviceJourney();
journeys.Add(current);
}
current!.Attempts.Add(reference);
if (string.Equals(reference.Status, nameof(SessionStatus.Succeeded), StringComparison.OrdinalIgnoreCase))
current.Completed = true;
previous = reference;
}
return journeys;
}
/// <summary>
/// Derived counts for the history row: journeys in the retained chain and the attempt count
/// of the LAST journey (open or completed) — the "Attempt N" the session banner renders.
/// </summary>
public static (int JourneyCount, int CurrentJourneyAttempts) Derive(IReadOnlyList<DeviceSessionRef> chain)
{
var journeys = GroupJourneys(chain);
if (journeys.Count == 0) return (0, 0);
return (journeys.Count, journeys[journeys.Count - 1].Attempts.Count);
}
/// <summary>
/// Attempt number of one session within its journey — the "Attempt N for this device" the
/// session-detail banner renders, computed server-side so no consumer re-derives journey
/// semantics. Terminal sessions sit in the chain and take their real position. A LIVE
/// (non-terminal) session is not a chain ref yet, so its position is computed by inserting
/// a virtual non-successful attempt at the session's StartedAt — the redeploy rule (a prior
/// completed journey starts a new one) and the 30-day gap rule then place it exactly like
/// the real ref would land once terminal. Null when the chain gives no basis (empty).
/// </summary>
public static int? ComputeAttemptNumber(
IReadOnlyList<DeviceSessionRef> chain, string sessionId, DateTime sessionStartedAt)
{
var effectiveChain = chain;
if (!chain.Any(r => string.Equals(r.SessionId, sessionId, StringComparison.Ordinal)))
{
if (chain.Count == 0) return null;
// Virtual ref: status is a NON-successful terminal placeholder purely for position
// math — it never persists and never marks the journey completed.
var virtualRef = new DeviceSessionRef
{
SessionId = sessionId,
StartedAt = sessionStartedAt,
Status = nameof(SessionStatus.Failed),
};
effectiveChain = MergeChain(chain, new[] { virtualRef });
}
foreach (var journey in GroupJourneys(effectiveChain))
{
for (var i = 0; i < journey.Attempts.Count; i++)
{
if (string.Equals(journey.Attempts[i].SessionId, sessionId, StringComparison.Ordinal))
return i + 1;
}
}
return null; // session fell off the 20-cap when merged — no honest position claim
}
}
@@ -0,0 +1,115 @@
using System.Collections;
namespace AutopilotMonitor.Functions.Helpers;
/// <summary>
/// Parsed ESP blocking/tracking lists from an <c>esp_config_detected</c> event payload
/// (agent emitter: <c>EspTrackingInfoProbe</c> → <c>DeviceInfoCollector.CollectEspConfiguration</c>).
///
/// Semantics (source-data audit Q2, insights spec §0.5): the lists are a POSITIVE-EVIDENCE set.
/// An app id found in any list is proven ESP-blocking; an id that is absent is UNKNOWN — never
/// "not blocking" — because the registry lists grow progressively (one timestamped subkey per
/// CSP status write), the user-scope <c>S-&lt;SID&gt;</c> lists usually appear only after
/// sign-in, each list is capped at 50 entries (uncapped totals ride in the <c>espTracked*Count</c>
/// fields), and MSI ProductCodes / PFNs live in namespaces the IME <c>appId</c> may not match.
/// Consumers must therefore always use the LATEST payload of a session that carries lists, and
/// must map "absent" to null/unknown, never to false.
/// </summary>
public sealed class EspBlockingSets
{
private readonly HashSet<string> _all;
private EspBlockingSets(
HashSet<string> all,
HashSet<string> userWin32AppIds,
int listedTotal,
bool isTruncated)
{
_all = all;
UserWin32AppIds = userWin32AppIds;
ListedCount = listedTotal;
IsTruncated = isTruncated;
}
/// <summary>Distinct ids across all four lists (Win32 device+user, MSI, PFN).</summary>
public int ListedCount { get; }
/// <summary>
/// True when any <c>espTracked*Count</c> exceeds its emitted list length — the 50-per-category
/// cap dropped entries, so even the positive evidence is incomplete for that category.
/// </summary>
public bool IsTruncated { get; }
/// <summary>
/// The user-scope Win32 subset (<c>espTrackedUserWin32AppIds</c>). Usually empty until the
/// agent-side re-emit fix ships (audit Q2) — exposed so callers can label user-ESP coverage
/// as unknown rather than inferring it from the merged list.
/// </summary>
public IReadOnlyCollection<string> UserWin32AppIds { get; }
/// <summary>Positive-evidence membership test (case-insensitive, all four lists).</summary>
public bool Contains(string? appId)
=> !string.IsNullOrEmpty(appId) && _all.Contains(appId!);
/// <summary>
/// Parses the payload of one <c>esp_config_detected</c> event. Returns null when the payload
/// carries none of the <c>espTracked*</c> list keys (probe found no Diagnostics key — e.g.
/// non-Autopilot device or pre-list agent build): "no lists observed" must stay
/// distinguishable from "empty lists observed".
/// Handles both value shapes: live-ingest / storage-rehydrated data (<c>List&lt;object&gt;</c>
/// via EventDataNormalizer) and test fixtures (<c>string[]</c> / <c>List&lt;string&gt;</c>).
/// </summary>
public static EspBlockingSets? FromEventData(IReadOnlyDictionary<string, object>? data)
{
if (data == null) return null;
var win32 = ReadStringList(data, "espTrackedWin32AppIds");
var userWin32 = ReadStringList(data, "espTrackedUserWin32AppIds");
var msi = ReadStringList(data, "espTrackedMsiProductCodes");
var pfn = ReadStringList(data, "espTrackedModernAppPfns");
if (win32 == null && userWin32 == null && msi == null && pfn == null)
return null;
var all = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
foreach (var list in new[] { win32, userWin32, msi, pfn })
if (list != null)
foreach (var id in list)
all.Add(id);
var userSet = new HashSet<string>(userWin32 ?? new List<string>(), StringComparer.OrdinalIgnoreCase);
// The agent emits uncapped distinct totals alongside the capped lists. The Win32 count
// covers the merged device+user list; user-Win32 has no own count field.
var truncated =
CountExceeds(data, "espTrackedWin32Count", win32) ||
CountExceeds(data, "espTrackedMsiCount", msi) ||
CountExceeds(data, "espTrackedModernCount", pfn);
return new EspBlockingSets(all, userSet, all.Count, truncated);
}
private static bool CountExceeds(IReadOnlyDictionary<string, object> data, string countKey, List<string>? list)
{
if (list == null) return false;
if (!data.TryGetValue(countKey, out var raw)) return false;
return long.TryParse(raw?.ToString(), out var count) && count > list.Count;
}
private static List<string>? ReadStringList(IReadOnlyDictionary<string, object> data, string key)
{
if (!data.TryGetValue(key, out var raw) || raw == null) return null;
// Strings enumerate as chars — exclude explicitly before the IEnumerable fallback.
if (raw is string) return null;
if (raw is not IEnumerable enumerable) return null;
var result = new List<string>();
foreach (var item in enumerable)
{
var s = item?.ToString()?.Trim();
if (!string.IsNullOrEmpty(s))
result.Add(s!);
}
return result;
}
}
@@ -20,6 +20,42 @@ public static class MetricsMath
return finished > 0 ? Math.Round((double)failed / finished * 100, 1) : 0;
}
/// <summary>
/// True when the row's terminal state says no real install attempt happened —
/// Skipped (not applicable, e.g. WinGet "Update for X" with the target app absent)
/// or Postponed. PR0 decision (2026-07-26): such rows are excluded from duration
/// statistics AND from the failure/success rate; they are reported as their own
/// category instead. Rows written before the TerminalState column existed return
/// false here (empty state) — they keep the legacy rate behavior because they
/// cannot be reclassified honestly.
/// </summary>
public static bool IsSkipTerminalState(AppInstallSummary s)
=> s.TerminalState == "Skipped" || s.TerminalState == "Postponed";
/// <summary>
/// Upper plausibility bound for an observed install duration: 6 h, the agent's default
/// max observation lifetime (AgentMaxLifetimeMinutes = 360) — nothing longer can have
/// been continuously watched. Rows above it are emission artifacts, not slow installs:
/// a batch emission at session end pins CompletedAt far past the real finish (verified
/// in production 2026-07-27: app rows carrying 60 175 s in a session whose own wall
/// clock was 2 697 s; 490 rows &gt; 6 h overall, worst 117 days — dominating the
/// slowest-apps ranking with six-figure averages).
/// </summary>
public const int MaxPlausibleInstallDurationSeconds = 6 * 3600;
/// <summary>
/// True for a succeeded row whose install duration was actually observed. Zero /
/// absent duration on a non-skip succeeded row means the START was never observed
/// (agent attach window — audit finding, §0.5 of the insights spec); a duration above
/// <see cref="MaxPlausibleInstallDurationSeconds"/> means the END was never observed
/// (completion back-stamped at session end). Either way the duration is UNKNOWN —
/// averaging zeros understates (measured −18 % in production), averaging back-stamps
/// inflates. Duration statistics must only read rows where this is true; everything
/// else is reported as "unmeasured".
/// </summary>
public static bool HasMeasuredDuration(AppInstallSummary s)
=> s.DurationSeconds > 0 && s.DurationSeconds <= MaxPlausibleInstallDurationSeconds;
/// <summary>
/// Builds the complete app-metrics response object from a (pre-time-filtered) set of app
/// install summaries. Single source of truth for both the tenant (<c>metrics/app</c>) and
@@ -36,9 +72,24 @@ public static class MetricsMath
{
var summaryList = summaries as IList<AppInstallSummary> ?? summaries.ToList();
var appGroups = summaryList.GroupBy(s => s.AppName).Select(g =>
// F1 PR1 (audit Q3): a name-keyed row that merged two distinct appIds carries an
// unattributable status/duration mix — it must not shape any per-app group. Excluded
// here with a disclosed count (truthfulness rule 7); the fleet-wide DO rollup below
// keeps every row, since transferred bytes are real regardless of identity mixing.
var totalCollisionExcluded = summaryList.Count(s => s.AppIdCollision);
var appGroups = summaryList.Where(s => !s.AppIdCollision).GroupBy(s => s.AppName).Select(g =>
{
var completed = g.Where(s => s.Status == "Succeeded").ToList();
// PR0 (2026-07-26) classification — see IsSkipTerminalState / HasMeasuredDuration:
// skipped = no real install attempt (TerminalState Skipped/Postponed)
// installed = succeeded rows that are not known skips (includes legacy rows
// without TerminalState — they cannot be reclassified)
// measured = installed rows with an observed duration — the ONLY duration input
// unmeasured = installed rows whose start was never observed (duration unknown)
var succeededAll = g.Where(s => s.Status == "Succeeded").ToList();
var skipped = succeededAll.Where(IsSkipTerminalState).ToList();
var installed = succeededAll.Where(s => !IsSkipTerminalState(s)).ToList();
var measured = installed.Where(HasMeasuredDuration).ToList();
var failed = g.Where(s => s.Status == "Failed").ToList();
var total = g.Count();
@@ -51,12 +102,16 @@ public static class MetricsMath
{
appName = g.Key,
totalInstalls = total,
succeeded = completed.Count,
succeeded = installed.Count,
skipped = skipped.Count,
unmeasured = installed.Count - measured.Count,
failed = failed.Count,
failureRate = TerminalFailureRatePct(failed.Count, completed.Count),
avgDurationSeconds = completed.Count > 0 ? Math.Round(completed.Average(s => s.DurationSeconds), 0) : 0,
maxDurationSeconds = completed.Count > 0 ? completed.Max(s => s.DurationSeconds) : 0,
avgDownloadBytes = completed.Count > 0 ? (long)completed.Average(s => s.DownloadBytes) : 0,
// Skips leave the rate (they are not attempts); legacy rows stay in it.
failureRate = TerminalFailureRatePct(failed.Count, installed.Count),
avgDurationSeconds = measured.Count > 0 ? Math.Round(measured.Average(s => s.DurationSeconds), 0) : 0,
maxDurationSeconds = measured.Count > 0 ? measured.Max(s => s.DurationSeconds) : 0,
measuredInstalls = measured.Count,
avgDownloadBytes = measured.Count > 0 ? (long)measured.Average(s => s.DownloadBytes) : 0,
doTotalBytesDownloaded = doG.TotalBytesDownloaded,
doBytesFromPeers = doG.BytesFromPeers,
doBytesFromCacheServer = doG.BytesFromCacheServer,
@@ -71,8 +126,10 @@ public static class MetricsMath
};
}).ToList();
// Slowest ranking gates on MEASURED installs — an app whose durations were mostly
// unobserved must not rank as "fast" on a handful of zeros (audit §0.5).
var slowestApps = SelectSlowestApps(
appGroups, a => a.succeeded, a => (double)a.avgDurationSeconds, minSamples: 3, take: 10);
appGroups, a => a.measuredInstalls, a => (double)a.avgDurationSeconds, minSamples: 3, take: 10);
var topFailingApps = appGroups
.Where(a => a.failed > 0)
@@ -88,6 +145,9 @@ public static class MetricsMath
success = true,
totalApps = appGroups.Count,
totalInstalls = summaryList.Count,
totalSkipped = appGroups.Sum(a => a.skipped),
totalUnmeasured = appGroups.Sum(a => a.unmeasured),
totalCollisionExcluded,
slowestApps,
topFailingApps,
deliveryOptimization = new
@@ -338,6 +398,38 @@ public static class MetricsMath
.Take(take)
.ToList();
}
/// <summary>
/// Wilson score interval (95 % default) for a binomial proportion — the F3 regression
/// radar's statistical primitive (insights spec §F3: deterministic, unit-pinned). Bounds
/// are clamped to [0,1]; n = 0 yields the uninformative (0,1) so a zero-denominator
/// side can never claim separation.
/// </summary>
public static (double Lower, double Upper) WilsonInterval(int successes, int trials, double z = 1.96)
{
if (trials <= 0) return (0.0, 1.0);
var n = (double)trials;
var p = Math.Clamp((double)successes / n, 0.0, 1.0);
var z2 = z * z;
var denominator = 1.0 + z2 / n;
var center = (p + z2 / (2.0 * n)) / denominator;
var half = (z / denominator) * Math.Sqrt(p * (1.0 - p) / n + z2 / (4.0 * n * n));
return (Math.Max(0.0, center - half), Math.Min(1.0, center + half));
}
/// <summary>
/// One-sided two-proportion separation for a rate INCREASE: true when the current
/// window's Wilson lower bound lies strictly above the baseline's Wilson upper bound —
/// the intervals are disjoint in the regression direction, so the lift is statistically
/// real rather than small-n noise (insights spec §F3 detection gate 3).
/// </summary>
public static bool RateIncreaseSeparated(
int windowHits, int windowTrials, int baselineHits, int baselineTrials, double z = 1.96)
{
var window = WilsonInterval(windowHits, windowTrials, z);
var baseline = WilsonInterval(baselineHits, baselineTrials, z);
return window.Lower > baseline.Upper;
}
}
/// <summary>
@@ -0,0 +1,255 @@
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Helpers;
/// <summary>One fired regression: a rule whose hit rate rose beyond every gate (see <see cref="RuleRegressionRadar"/>).</summary>
public sealed class RuleRegressionFinding
{
public string TenantId { get; init; } = string.Empty;
public string RuleId { get; init; } = string.Empty;
public string RuleTitle { get; init; } = string.Empty;
public string Severity { get; init; } = string.Empty;
public string Category { get; init; } = string.Empty;
public int WindowFireCount { get; init; }
public int WindowSessionCount { get; init; }
public int BaselineFireCount { get; init; }
public int BaselineSessionCount { get; init; }
public double WindowRatePct { get; init; }
public double BaselineRatePct { get; init; }
/// <summary>Window rate ÷ baseline rate, one decimal. Null when the baseline rate is 0 — a NEW signal has no finite lift, and we never invent one.</summary>
public double? Lift { get; init; }
public string WindowStartDate { get; init; } = string.Empty;
public string WindowEndDate { get; init; } = string.Empty;
}
/// <summary>
/// Deterministic, I/O-free core of the F3 regression radar (insights spec §F3). Detects
/// "this analyze rule suddenly fires much more often" from the daily <see cref="RuleStatsEntry"/>
/// rows — RATES, not counts (absolute counts spike with every rollout wave): per (tenant, rule),
/// hit rate = FireCount ÷ SessionsEvaluated, audit-verified to be session-deduplicated for
/// analyze rules (gather rules are excluded until their per-batch dedup is fixed — Q6 follow-up,
/// never silently included with bad math). A regression fires only when ALL hold:
/// <list type="number">
/// <item>trailing 7-day window has ≥ 5 hit sessions AND ≥ 20 evaluated sessions;</item>
/// <item>window rate ≥ 2× the prior 28-day baseline rate;</item>
/// <item>the Wilson 95 % intervals are disjoint in the increase direction
/// (<see cref="MetricsMath.RateIncreaseSeparated"/>) — small-n noise never alerts.</item>
/// </list>
/// Suppressed (false positives are trust damage on a monitoring product): rules younger than
/// baseline + window (grace period), rules edited inside the window (edits legitimately change
/// hit rates), rules without an entity (deleted), and rules with an empty baseline denominator.
/// </summary>
public static class RuleRegressionRadar
{
public const int WindowDays = 7;
public const int BaselineDays = 28;
public const int MinWindowHitSessions = 5;
public const int MinWindowSessions = 20;
public const double MinLift = 2.0;
/// <summary>An active alert re-arms once the window rate falls back under 1.5× baseline (spec §F3 suppression).</summary>
public const double ReArmLiftFactor = 1.5;
/// <summary>Dimension-concentration gates: a value must cover ≥5 hit sessions at ≥2× lift, or the alert says "no clear dimension concentration".</summary>
public const int MinDimensionSessions = 5;
public const double MinDimensionLift = 2.0;
/// <summary>Rule entity timestamps the suppression gates read (CreatedAt = grace period; UpdatedAt = edit-in-window).</summary>
public readonly record struct RuleTimestamps(DateTime CreatedAt, DateTime UpdatedAt);
/// <summary>
/// Evaluates one tenant's analyze-rule stats over the [target − 34d, target] horizon.
/// <paramref name="tenantEntries"/> must be the tenant's rows only (no "global" mirror);
/// gather rows are ignored defensively. <paramref name="ruleTimestamps"/> maps ruleId →
/// entity timestamps — a rule without an entry is suppressed (deleted rules never alert).
/// </summary>
public static List<RuleRegressionFinding> Evaluate(
IReadOnlyList<RuleStatsEntry> tenantEntries,
DateTime targetDateUtc,
IReadOnlyDictionary<string, RuleTimestamps> ruleTimestamps)
{
var findings = new List<RuleRegressionFinding>();
foreach (var group in AnalyzeRuleGroups(tenantEntries))
{
var sums = SumWindows(group.Value, targetDateUtc);
if (sums.WindowFire < MinWindowHitSessions) continue;
if (sums.WindowSessions < MinWindowSessions) continue;
// Entity gates: deleted rules never alert; new rules get the full baseline+window
// grace period; an edit inside the window legitimately changes the rate.
if (!ruleTimestamps.TryGetValue(group.Key, out var timestamps)) continue;
var windowStart = targetDateUtc.Date.AddDays(-(WindowDays - 1));
if (timestamps.CreatedAt > targetDateUtc.Date.AddDays(-(WindowDays - 1 + BaselineDays))) continue;
if (timestamps.UpdatedAt >= windowStart) continue;
// No observed baseline denominator (e.g. rule disabled throughout the baseline,
// re-enabled recently) — no honest comparison basis.
if (sums.BaselineSessions <= 0) continue;
var windowRate = (double)sums.WindowFire / sums.WindowSessions;
var baselineRate = (double)sums.BaselineFire / sums.BaselineSessions;
if (windowRate < MinLift * baselineRate) continue;
if (!MetricsMath.RateIncreaseSeparated(
sums.WindowFire, sums.WindowSessions, sums.BaselineFire, sums.BaselineSessions))
continue;
var first = group.Value[0];
findings.Add(new RuleRegressionFinding
{
TenantId = first.TenantId,
RuleId = group.Key,
RuleTitle = first.RuleTitle,
Severity = first.Severity,
Category = first.Category,
WindowFireCount = sums.WindowFire,
WindowSessionCount = sums.WindowSessions,
BaselineFireCount = sums.BaselineFire,
BaselineSessionCount = sums.BaselineSessions,
WindowRatePct = Math.Round(windowRate * 100, 1),
BaselineRatePct = Math.Round(baselineRate * 100, 1),
Lift = baselineRate > 0 ? Math.Round(windowRate / baselineRate, 1) : null,
WindowStartDate = windowStart.ToString("yyyy-MM-dd"),
WindowEndDate = targetDateUtc.Date.ToString("yyyy-MM-dd"),
});
}
return findings
.OrderByDescending(f => f.WindowRatePct)
.ThenBy(f => f.RuleId, StringComparer.Ordinal)
.ToList();
}
/// <summary>
/// True when an ACTIVE alert for this rule may re-arm: nothing fired in the window anymore,
/// or the window rate fell back under <see cref="ReArmLiftFactor"/>× the baseline rate.
/// A zero-baseline alert ("new signal") re-arms only via the fires-stopped branch — there
/// is no rate threshold to fall under.
/// </summary>
public static bool ShouldReArm(IReadOnlyList<RuleStatsEntry> ruleEntries, DateTime targetDateUtc)
{
var sums = SumWindows(ruleEntries, targetDateUtc);
if (sums.WindowFire == 0) return true;
if (sums.BaselineFire <= 0 || sums.BaselineSessions <= 0 || sums.WindowSessions <= 0) return false;
var windowRate = (double)sums.WindowFire / sums.WindowSessions;
var baselineRate = (double)sums.BaselineFire / sums.BaselineSessions;
return windowRate < ReArmLiftFactor * baselineRate;
}
/// <summary>
/// The dominant dimension concentration among a regression's hit sessions, or null when no
/// dimension value reaches ≥<see cref="MinDimensionLift"/>× lift with
/// ≥<see cref="MinDimensionSessions"/> hit sessions — the alert then says
/// "no clear dimension concentration" instead of stretching for one (truthfulness rules 1/6).
/// Wording contract for consumers: "correlated — not necessarily causal".
/// </summary>
public static RuleRegressionDimension? ComputeDimensionConcentration(
IReadOnlyList<SessionSummary> hitSessions, IReadOnlyList<SessionSummary> allSessions)
{
if (hitSessions.Count == 0 || allSessions.Count == 0) return null;
var dimensions = new (string Name, Func<SessionSummary, string?> Selector)[]
{
("osBuild", s => s.OsBuild),
("model", s => $"{s.Manufacturer} {s.Model}".Trim()),
("agentVersion", s => s.AgentVersion),
("imeVersion", s => s.ImeAgentVersion),
};
RuleRegressionDimension? best = null;
foreach (var (name, selector) in dimensions)
{
var allCounts = CountValues(allSessions, selector);
foreach (var hit in CountValues(hitSessions, selector))
{
if (hit.Value < MinDimensionSessions) continue;
if (!allCounts.TryGetValue(hit.Key, out var allCount) || allCount <= 0) continue;
var hitShare = (double)hit.Value / hitSessions.Count;
var allShare = (double)allCount / allSessions.Count;
var lift = hitShare / allShare;
if (lift < MinDimensionLift) continue;
if (best == null || lift > best.Lift || (lift == best.Lift && hit.Value > best.HitCount))
{
best = new RuleRegressionDimension
{
Dimension = name,
Value = hit.Key,
HitCount = hit.Value,
HitSharePct = Math.Round(hitShare * 100, 1),
AllSharePct = Math.Round(allShare * 100, 1),
Lift = Math.Round(lift, 1),
};
}
}
}
return best;
}
private static Dictionary<string, int> CountValues(
IReadOnlyList<SessionSummary> sessions, Func<SessionSummary, string?> selector)
{
var counts = new Dictionary<string, int>(StringComparer.OrdinalIgnoreCase);
foreach (var session in sessions)
{
var value = selector(session)?.Trim();
if (string.IsNullOrEmpty(value)) continue;
counts.TryGetValue(value!, out var current);
counts[value!] = current + 1;
}
return counts;
}
/// <summary>Groups a tenant's entries by ruleId, analyze rows only (defensive re-filter).</summary>
internal static Dictionary<string, List<RuleStatsEntry>> AnalyzeRuleGroups(IReadOnlyList<RuleStatsEntry> entries)
{
var groups = new Dictionary<string, List<RuleStatsEntry>>(StringComparer.OrdinalIgnoreCase);
foreach (var entry in entries)
{
if (!string.Equals(entry.RuleType, "analyze", StringComparison.OrdinalIgnoreCase)) continue;
if (string.IsNullOrEmpty(entry.RuleId)) continue;
if (!groups.TryGetValue(entry.RuleId, out var list))
{
list = new List<RuleStatsEntry>();
groups[entry.RuleId] = list;
}
list.Add(entry);
}
return groups;
}
internal readonly record struct WindowSums(int WindowFire, int WindowSessions, int BaselineFire, int BaselineSessions);
/// <summary>
/// Sums one rule's daily rows into the trailing window [target−6d, target] and the prior
/// baseline [target−34d, target−7d]. Dates are compared as ISO strings (the rows' own key
/// format); rows outside the horizon are ignored.
/// </summary>
internal static WindowSums SumWindows(IReadOnlyList<RuleStatsEntry> ruleEntries, DateTime targetDateUtc)
{
var windowStart = targetDateUtc.Date.AddDays(-(WindowDays - 1)).ToString("yyyy-MM-dd");
var baselineStart = targetDateUtc.Date.AddDays(-(WindowDays - 1 + BaselineDays)).ToString("yyyy-MM-dd");
var target = targetDateUtc.Date.ToString("yyyy-MM-dd");
int windowFire = 0, windowSessions = 0, baselineFire = 0, baselineSessions = 0;
foreach (var entry in ruleEntries)
{
if (string.CompareOrdinal(entry.Date, target) > 0) continue;
if (string.CompareOrdinal(entry.Date, windowStart) >= 0)
{
windowFire += entry.FireCount;
windowSessions += entry.SessionsEvaluated;
}
else if (string.CompareOrdinal(entry.Date, baselineStart) >= 0)
{
baselineFire += entry.FireCount;
baselineSessions += entry.SessionsEvaluated;
}
}
return new WindowSums(windowFire, windowSessions, baselineFire, baselineSessions);
}
}
@@ -0,0 +1,666 @@
using System.Globalization;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Helpers;
/// <summary>
/// Inputs for <see cref="TimeAttributionCalculator.Compute"/> — the session facts the terminal
/// writer persisted plus the session's full event stream in canonical order.
/// </summary>
public sealed class TimeAttributionInput
{
public string TenantId { get; set; } = string.Empty;
public string SessionId { get; set; } = string.Empty;
/// <summary>Session status string. Only "Succeeded" and "Failed" are computable (Incomplete deliberately carries no DurationSeconds).</summary>
public string Status { get; set; } = string.Empty;
public DateTime StartedAt { get; set; }
public DateTime? CompletedAt { get; set; }
/// <summary>
/// The session's authoritative duration (second-sweep finding, insights spec §0.5): for
/// WhiteGlove it is part 1 + part 2 WITHOUT the pause, and re-terminal stamps move
/// CompletedAt — so this is the wall clock, never CompletedAt − StartedAt.
/// </summary>
public int? DurationSeconds { get; set; }
public bool IsPreProvisioned { get; set; }
/// <summary>WhiteGlove part-2 resume timestamp (session row), when present.</summary>
public DateTime? ResumedAt { get; set; }
/// <summary>Session events. Canonical order is Sequence ascending; the calculator re-sorts defensively.</summary>
public IReadOnlyList<EnrollmentEvent> Events { get; set; } = Array.Empty<EnrollmentEvent>();
}
/// <summary>
/// Deterministic, I/O-free computation of a session's <see cref="SessionTimeBreakdown"/>
/// (F1, insights spec — AttributionVersion 1). Every number is traceable to observed events;
/// missing signals produce "unknown"/unattributed, never guesses (truthfulness rules 1/2).
///
/// Semantics pinned by the golden-fixture tests:
/// <list type="bullet">
/// <item><b>Wall clock = session DurationSeconds.</b> Observation windows are constructed to sum
/// to it exactly: non-WhiteGlove = [CompletedAt − Duration, CompletedAt]; WhiteGlove = two windows
/// in ONE session — [part1End − part1, part1End] and [ResumedAt, CompletedAt], where
/// part1 = Duration − part2 and part1End is anchored on <c>whiteglove_part1_complete</c>.
/// The pause between the windows is never attributed.</item>
/// <item><b>Segments from phase-declaration events only</b> (phase strategy: only declaration
/// events carry Phase != Unknown), bucketed Start/DevicePreparation/DeviceSetup → device_prep,
/// AppsDevice → esp_apps, AccountSetup + FinalizingSetup → identity_hello, AppsUser → user_esp,
/// Complete / <c>desktop_arrived</c> → desktop_handoff. A Failed-phase declaration ends
/// attribution (the tail is unattributed, not relabeled). Anchors that go backward in canonical
/// order are dropped (<see cref="TimeAttributionFlags.ClockSkewDropped"/>).</item>
/// <item><b>Exact-partition invariant:</b> span seconds are floored, the remainder —
/// unattributed holes plus sub-second dust — is reported as UnattributedSeconds, so
/// sum(spans) + unattributed == wall clock holds exactly and is never normalized.</item>
/// <item><b>Per-app intervals use event timestamps</b> (first started/download event → LAST
/// terminal event, covering IME retries) — never the agent payload timing fields, which freeze
/// at the first terminal transition (audit Q1).</item>
/// <item><b>Blocking membership is positive evidence</b> from the LATEST
/// <c>esp_config_detected</c> lists: listed ⇒ blocking; absent ⇒ unknown, never false (audit Q2).
/// No lists ⇒ <see cref="TimeAttributionFlags.BlockingSetUnknown"/> and occupancy null.</item>
/// <item><b>Reboot spans from event gaps</b> around the <c>lastBootUtc</c> payload — the
/// <c>system_reboot_detected</c> event is detection-time-stamped by the next run and never used
/// as the reboot moment (audit Q7). Reboots are a cross-cutting annotation, not a segment.</item>
/// </list>
/// </summary>
public static class TimeAttributionCalculator
{
/// <summary>Bump on any semantic change so aggregates never mix definitions (truthfulness rule 8).</summary>
public const int CurrentVersion = 1;
/// <summary>
/// Enrollment class for fleet aggregation — classes are NEVER mixed in one aggregate (a
/// WhiteGlove or WDP flow has a structurally different time profile than user-driven ESP).
/// Precedence: WhiteGlove (both parts live in one session row) → Windows Device Preparation
/// (EnrollmentType "v2" — different phase machinery) → self-deploying profile → user-driven.
/// </summary>
public static string GetEnrollmentClass(SessionSummary session)
{
if (session.IsPreProvisioned) return "whiteglove";
if (session.EnrollmentType == "v2") return "device_preparation";
if (session.IsSelfDeployingProfile) return "self_deploying";
return "user_driven";
}
/// <summary>Cap for the persisted per-app interval list (spec: top 20 by duration + uncapped count).</summary>
internal const int MaxBlockingAppIntervals = 20;
private readonly struct Window
{
public Window(DateTime start, DateTime end) { Start = start; End = end; }
public DateTime Start { get; }
public DateTime End { get; }
}
private sealed class Anchor
{
public DateTime Ts;
/// <summary>Segment key, or null = attribution ends here (Failed-phase declaration).</summary>
public string? Bucket;
}
private sealed class AppActivity
{
public string AppId = string.Empty;
public string AppName = string.Empty;
public DateTime? FirstStart;
public DateTime? LastTerminal;
}
/// <summary>
/// Returns the breakdown, or null when the session has no computable wall clock
/// (non-terminal, Incomplete — which deliberately stores no DurationSeconds — or a
/// missing/non-positive duration).
/// </summary>
public static SessionTimeBreakdown? Compute(TimeAttributionInput input)
{
if (input.Status != "Succeeded" && input.Status != "Failed") return null;
if (!input.CompletedAt.HasValue) return null;
if (!input.DurationSeconds.HasValue || input.DurationSeconds.Value <= 0) return null;
var completedAt = input.CompletedAt.Value;
var wallClock = input.DurationSeconds.Value;
var flags = TimeAttributionFlags.None;
// Canonical order (Sequence asc) — the authoritative event order per repo convention.
var events = input.Events.OrderBy(e => e.Sequence).ToList();
var windows = BuildWindows(input, completedAt, wallClock, events, ref flags);
var anchors = BuildAnchors(events, ref flags);
var spans = BuildSegmentSpans(windows, anchors);
var attributedSeconds = spans.Sum(s => s.Seconds);
var unattributed = wallClock - attributedSeconds;
// Blocking set: latest esp_config_detected emission that carries lists (positive
// evidence is never erased by a later probe failure; lists grow progressively).
EspBlockingSets? blockingSets = null;
for (var i = events.Count - 1; i >= 0 && blockingSets == null; i--)
{
if (events[i].EventType == Constants.EventTypes.EspConfigDetected)
blockingSets = EspBlockingSets.FromEventData(events[i].Data);
}
if (blockingSets == null || blockingSets.ListedCount == 0)
flags |= TimeAttributionFlags.BlockingSetUnknown;
else if (blockingSets.IsTruncated)
flags |= TimeAttributionFlags.BlockingSetTruncated;
var (blockingApps, blockingAppCount) = BuildBlockingAppIntervals(events, blockingSets, windows, ref flags);
int? occupancy = null;
if (blockingSets != null && blockingSets.ListedCount > 0)
{
var espAppsSpans = spans.Where(s => s.SegmentKey == TimeAttributionSegments.EspApps).ToList();
occupancy = ComputeOccupancySeconds(blockingApps, espAppsSpans);
}
var rebootSpans = BuildRebootSpans(events, windows, spans);
if (events.Any(e => e.EventType == Constants.EventTypes.AgentLateStart))
flags |= TimeAttributionFlags.PartialObservation;
return new SessionTimeBreakdown
{
TenantId = input.TenantId,
SessionId = input.SessionId,
AttributionVersion = CurrentVersion,
WallClockSeconds = wallClock,
Segments = spans,
UnattributedSeconds = unattributed,
RebootSeconds = rebootSpans.Sum(r => r.Seconds),
RebootSpans = rebootSpans,
BlockingApps = blockingApps
.OrderByDescending(a => a.Seconds)
.ThenBy(a => a.AppName, StringComparer.OrdinalIgnoreCase)
.Take(MaxBlockingAppIntervals)
.ToList(),
BlockingAppCount = blockingAppCount,
EspAppsOccupancySeconds = occupancy,
QualityFlags = flags,
};
}
// ── observation windows ─────────────────────────────────────────────────
/// <summary>
/// Windows are END-anchored on the authoritative timestamps (CompletedAt; part1End for WG
/// part 1) with the start derived so window lengths sum EXACTLY to DurationSeconds — the
/// invariant is against the stored duration, not against CompletedAt − StartedAt.
/// </summary>
private static List<Window> BuildWindows(
TimeAttributionInput input, DateTime completedAt, int wallClock,
List<EnrollmentEvent> events, ref TimeAttributionFlags flags)
{
if (input.IsPreProvisioned)
{
if (input.ResumedAt.HasValue && input.ResumedAt.Value < completedAt)
{
var resumedAt = input.ResumedAt.Value;
// Same truncation the terminal writer used for part2, so part1 comes out as the
// exact stored part-1 duration.
var part2 = (int)(completedAt - resumedAt).TotalSeconds;
var part1 = wallClock - part2;
if (part1 > 0)
{
var part1End = ResolvePart1End(events, resumedAt, ref flags);
if (part1End.HasValue)
{
var w1Start = part1End.Value.AddSeconds(-part1);
return new List<Window>
{
new Window(w1Start, part1End.Value),
new Window(resumedAt, completedAt),
};
}
}
// part1 <= 0 (stored duration contradicts the resume stamp) or no usable part-1
// anchor: fall through to a single derived window rather than inventing a split.
}
// WhiteGlove without a resolvable two-window split — the partition is still exact,
// but the WG semantics (pause exclusion boundaries) rest on no anchor.
flags |= TimeAttributionFlags.WhiteGloveAnchorsIncomplete;
}
return new List<Window> { new Window(completedAt.AddSeconds(-wallClock), completedAt) };
}
/// <summary>
/// Part-1 end anchor: the <c>whiteglove_part1_complete</c> event; fallback: the last event
/// at/before ResumedAt (flagged — the boundary is then observational, not declarative).
/// An anchor after ResumedAt is clamped to ResumedAt (windows must not overlap).
/// </summary>
private static DateTime? ResolvePart1End(
List<EnrollmentEvent> events, DateTime resumedAt, ref TimeAttributionFlags flags)
{
DateTime? part1End = null;
foreach (var evt in events)
{
if (evt.EventType == Constants.EventTypes.WhiteGlovePart1Complete)
part1End = evt.Timestamp;
}
if (!part1End.HasValue)
{
flags |= TimeAttributionFlags.WhiteGloveAnchorsIncomplete;
foreach (var evt in events)
{
if (evt.Timestamp <= resumedAt &&
(!part1End.HasValue || evt.Timestamp > part1End.Value))
{
part1End = evt.Timestamp;
}
}
}
else if (part1End.Value > resumedAt)
{
flags |= TimeAttributionFlags.WhiteGloveAnchorsIncomplete;
part1End = resumedAt;
}
return part1End;
}
// ── segment spans ───────────────────────────────────────────────────────
private static string? MapPhaseToSegment(EnrollmentPhase phase) => phase switch
{
EnrollmentPhase.Start => TimeAttributionSegments.DevicePrep,
EnrollmentPhase.DevicePreparation => TimeAttributionSegments.DevicePrep,
EnrollmentPhase.DeviceSetup => TimeAttributionSegments.DevicePrep,
EnrollmentPhase.AppsDevice => TimeAttributionSegments.EspApps,
EnrollmentPhase.AccountSetup => TimeAttributionSegments.IdentityHello,
EnrollmentPhase.AppsUser => TimeAttributionSegments.UserEsp,
EnrollmentPhase.FinalizingSetup => TimeAttributionSegments.IdentityHello,
EnrollmentPhase.Complete => TimeAttributionSegments.DesktopHandoff,
// Failed: attribution ends — the post-failure tail stays unattributed rather than
// being relabeled into a segment no phase ever declared.
EnrollmentPhase.Failed => null,
_ => null,
};
/// <summary>
/// Anchor list in canonical order: phase-declaration events (Phase != Unknown) plus
/// <c>desktop_arrived</c> (Phase=Unknown by convention, but the spec-designated handoff
/// anchor). Consecutive same-bucket anchors collapse; anchors whose timestamp goes
/// backward vs. the previously accepted anchor are dropped with ClockSkewDropped.
/// </summary>
private static List<Anchor> BuildAnchors(List<EnrollmentEvent> events, ref TimeAttributionFlags flags)
{
var anchors = new List<Anchor>();
foreach (var evt in events)
{
string? bucket;
var isDeclaration = false;
if (evt.EventType == Constants.EventTypes.DesktopArrived)
{
bucket = TimeAttributionSegments.DesktopHandoff;
isDeclaration = true;
}
else if (evt.Phase != EnrollmentPhase.Unknown)
{
bucket = MapPhaseToSegment(evt.Phase);
isDeclaration = true;
}
else
{
bucket = null;
}
if (!isDeclaration) continue;
if (anchors.Count > 0)
{
var prev = anchors[anchors.Count - 1];
if (evt.Timestamp < prev.Ts)
{
flags |= TimeAttributionFlags.ClockSkewDropped;
continue;
}
if (prev.Bucket == bucket)
continue; // same-phase re-declaration (re-registration, replay) — no boundary
}
anchors.Add(new Anchor { Ts = evt.Timestamp, Bucket = bucket });
}
return anchors;
}
/// <summary>
/// Slices each window at the in-window anchors and buckets every slice by the phase context
/// active at its start. Context carries INTO window 0 from anchors at/before its start, but
/// never across the WhiteGlove pause into window 1+ (the resume gap makes no phase claim).
/// The leading gap of window 0 with no context is device_prep only when the first observed
/// anchor is still pre/at apps entry (the spec defines device_prep as session start → first
/// apps-phase entry); otherwise it stays unattributed.
/// </summary>
private static List<TimeAttributionSpan> BuildSegmentSpans(List<Window> windows, List<Anchor> anchors)
{
var spans = new List<TimeAttributionSpan>();
for (var w = 0; w < windows.Count; w++)
{
var window = windows[w];
string? context = null;
if (w == 0)
{
foreach (var a in anchors)
{
if (a.Ts <= window.Start) context = a.Bucket;
else break;
}
}
var inWindow = anchors.Where(a => a.Ts > window.Start && a.Ts < window.End).ToList();
if (w == 0 && context == null)
{
var first = inWindow.FirstOrDefault();
if (first != null &&
(first.Bucket == TimeAttributionSegments.DevicePrep ||
first.Bucket == TimeAttributionSegments.EspApps))
{
context = TimeAttributionSegments.DevicePrep;
}
}
var cursor = window.Start;
foreach (var anchor in inWindow)
{
AddSpan(spans, context, cursor, anchor.Ts);
cursor = anchor.Ts;
context = anchor.Bucket;
}
AddSpan(spans, context, cursor, window.End);
}
return spans;
}
private static void AddSpan(List<TimeAttributionSpan> spans, string? bucket, DateTime start, DateTime end)
{
if (bucket == null || end <= start) return;
var seconds = (int)Math.Floor((end - start).TotalSeconds);
if (seconds <= 0) return; // sub-second slivers land in the unattributed remainder
spans.Add(new TimeAttributionSpan { SegmentKey = bucket, StartUtc = start, EndUtc = end, Seconds = seconds });
}
// ── blocking-app intervals + occupancy ──────────────────────────────────
private static (List<BlockingAppInterval> Intervals, int MatchedCount) BuildBlockingAppIntervals(
List<EnrollmentEvent> events, EspBlockingSets? sets, List<Window> windows, ref TimeAttributionFlags flags)
{
var intervals = new List<BlockingAppInterval>();
if (sets == null || sets.ListedCount == 0)
return (intervals, 0);
// Fold per-app activity from EVENT timestamps (never payload timing — audit Q1).
var apps = new Dictionary<string, AppActivity>(StringComparer.OrdinalIgnoreCase);
foreach (var evt in events)
{
var isStart = evt.EventType == Constants.EventTypes.AppInstallStart ||
evt.EventType == Constants.EventTypes.AppDownloadStarted;
var isTerminal = evt.EventType == Constants.EventTypes.AppInstallComplete ||
evt.EventType == Constants.EventTypes.AppInstallFailed ||
evt.EventType == Constants.EventTypes.AppInstallSkipped;
if (!isStart && !isTerminal) continue;
if (evt.Data == null || !evt.Data.TryGetValue("appId", out var appIdObj)) continue;
var appId = appIdObj?.ToString()?.Trim();
if (string.IsNullOrEmpty(appId)) continue;
if (!apps.TryGetValue(appId!, out var activity))
{
activity = new AppActivity { AppId = appId! };
apps[appId!] = activity;
}
if (string.IsNullOrEmpty(activity.AppName) &&
evt.Data.TryGetValue("appName", out var nameObj))
{
activity.AppName = nameObj?.ToString()?.Trim() ?? string.Empty;
}
if (isStart && (!activity.FirstStart.HasValue || evt.Timestamp < activity.FirstStart.Value))
activity.FirstStart = evt.Timestamp;
// LAST terminal wins — an IME retry (Error → retry → Installed) extends the
// occupancy to the final outcome instead of freezing at the first failure.
if (isTerminal && (!activity.LastTerminal.HasValue || evt.Timestamp > activity.LastTerminal.Value))
activity.LastTerminal = evt.Timestamp;
}
var matched = 0;
foreach (var activity in apps.Values)
{
if (!sets.Contains(activity.AppId)) continue; // absent ⇒ unknown, never "not blocking"
matched++;
if (!activity.FirstStart.HasValue || !activity.LastTerminal.HasValue)
continue; // unobserved endpoint → no interval claim (attach window, audit §0.5)
if (activity.LastTerminal.Value < activity.FirstStart.Value)
{
flags |= TimeAttributionFlags.ClockSkewDropped;
continue;
}
var clamped = ClampToWindows(activity.FirstStart.Value, activity.LastTerminal.Value, windows);
if (!clamped.HasValue)
{
// Outside the hull entirely → pathological timestamps (flagged, as before).
// Inside the hull but touching no window (wholly inside the WhiteGlove pause):
// no in-window observation → no interval claim, and nothing wrong with clocks.
if (activity.LastTerminal.Value < windows[0].Start ||
activity.FirstStart.Value > windows[windows.Count - 1].End)
{
flags |= TimeAttributionFlags.ClockSkewDropped;
}
continue;
}
intervals.Add(new BlockingAppInterval
{
AppId = activity.AppId,
AppName = activity.AppName,
StartUtc = clamped.Value.Start,
EndUtc = clamped.Value.End,
// In-window seconds — a WG-pause-straddling interval never counts the pause.
Seconds = clamped.Value.Seconds,
});
}
return (intervals, matched);
}
/// <summary>
/// Clips [start, end] to the observation windows. Start/End are the clamped chronological
/// endpoints; <c>Seconds</c> counts ONLY time inside the windows — the WhiteGlove pause
/// between part 1 and part 2 lies inside the hull but is excluded, matching the session's
/// own <c>DurationSeconds</c> semantics (pause excluded by design). Null when the interval
/// overlaps no window at all: either it lies outside the hull entirely (pathological
/// timestamps) or wholly inside the pause (no in-window observation).
/// </summary>
private static (DateTime Start, DateTime End, int Seconds)? ClampToWindows(DateTime start, DateTime end, List<Window> windows)
{
DateTime? s = null, e = null;
double total = 0;
foreach (var window in windows)
{
var ws = start < window.Start ? window.Start : start;
var we = end > window.End ? window.End : end;
if (we < ws) continue;
if (s == null || ws < s.Value) s = ws;
if (e == null || we > e.Value) e = we;
total += (we - ws).TotalSeconds;
}
if (s == null || e == null) return null;
return (s.Value, e.Value, (int)Math.Floor(total));
}
/// <summary>
/// Critical-path occupancy: overlap-merged union of the blocking intervals, intersected
/// with the esp_apps spans. Whole-second floor on the total (consistent with span math).
/// </summary>
internal static int ComputeOccupancySeconds(
List<BlockingAppInterval> intervals, List<TimeAttributionSpan> espAppsSpans)
{
if (intervals.Count == 0 || espAppsSpans.Count == 0) return 0;
var merged = new List<(DateTime Start, DateTime End)>();
foreach (var iv in intervals.OrderBy(i => i.StartUtc))
{
if (merged.Count > 0 && iv.StartUtc <= merged[merged.Count - 1].End)
{
var last = merged[merged.Count - 1];
if (iv.EndUtc > last.End)
merged[merged.Count - 1] = (last.Start, iv.EndUtc);
}
else
{
merged.Add((iv.StartUtc, iv.EndUtc));
}
}
double total = 0;
foreach (var span in espAppsSpans)
{
foreach (var (ms, me) in merged)
{
var s = ms > span.StartUtc ? ms : span.StartUtc;
var e = me < span.EndUtc ? me : span.EndUtc;
if (e > s) total += (e - s).TotalSeconds;
}
}
return (int)Math.Floor(total);
}
// ── what-if bound (fleet aggregation, F1 PR2) ───────────────────────────
/// <summary>
/// Upper-bound saving from removing app <paramref name="appId"/> from the ESP blocking set
/// of ONE session: the critical-path end (latest interval end) recomputed without the app's
/// interval — <c>max(0, cpEnd − cpEndWithoutX)</c> per the spec. When the app is the only
/// measured blocking interval, the fallback baseline is its own start (nothing else was
/// observed holding the path). An upper bound BY CONSTRUCTION — removing an app cannot slow
/// anything down, but hidden serialization may reduce real savings — so consumers must
/// phrase it as "up to", never as a promise (truthfulness rule 3). Returns 0 when the app
/// has no measured interval or does not end last.
/// </summary>
public static int WhatIfSavingSeconds(IReadOnlyList<BlockingAppInterval> intervals, string appId)
{
BlockingAppInterval? target = null;
DateTime? cpEnd = null, cpEndWithout = null;
foreach (var interval in intervals)
{
if (!cpEnd.HasValue || interval.EndUtc > cpEnd.Value)
cpEnd = interval.EndUtc;
if (string.Equals(interval.AppId, appId, StringComparison.OrdinalIgnoreCase))
{
target = interval; // one interval per app per session by construction
}
else if (!cpEndWithout.HasValue || interval.EndUtc > cpEndWithout.Value)
{
cpEndWithout = interval.EndUtc;
}
}
if (target == null || !cpEnd.HasValue) return 0;
// Spec formula verbatim: cpEnd − cpEndWithoutX. When X's removal exposes an earlier
// path end, the idle gap before X's start counts too — the ESP was waiting for X to
// START as much as to finish. Negative/zero → another app ends at or after X: no claim.
var baseline = cpEndWithout ?? target.StartUtc;
var saving = (cpEnd.Value - baseline).TotalSeconds;
return saving > 0 ? (int)Math.Floor(saving) : 0;
}
// ── reboot spans ────────────────────────────────────────────────────────
/// <summary>
/// One span per distinct observed boot: the event-stream gap bracketing <c>lastBootUtc</c>
/// (last event before the boot → first event at/after it). Fallback without a usable
/// payload: previous event → the detection event itself (which is genuinely post-boot —
/// usable as an observation bound, never as the boot moment). Spans are clipped to the
/// observation hull, so a WhiteGlove-pause reboot contributes nothing.
/// </summary>
private static List<RebootSpan> BuildRebootSpans(
List<EnrollmentEvent> events, List<Window> windows, List<TimeAttributionSpan> spans)
{
var result = new List<RebootSpan>();
var seenBoots = new HashSet<DateTime>();
for (var i = 0; i < events.Count; i++)
{
var evt = events[i];
if (evt.EventType != Constants.EventTypes.SystemRebootDetected) continue;
DateTime? lastBoot = null;
if (evt.Data != null && evt.Data.TryGetValue("lastBootUtc", out var lastBootObj))
{
var raw = lastBootObj?.ToString();
if (!string.IsNullOrEmpty(raw) &&
DateTime.TryParse(raw, CultureInfo.InvariantCulture,
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal, out var parsed))
{
lastBoot = parsed;
}
}
DateTime? gapStart = null, gapEnd = null;
if (lastBoot.HasValue)
{
if (!seenBoots.Add(lastBoot.Value)) continue; // duplicate detection of the same boot
foreach (var e in events)
{
if (e.Timestamp < lastBoot.Value)
{
if (!gapStart.HasValue || e.Timestamp > gapStart.Value) gapStart = e.Timestamp;
}
else if (!gapEnd.HasValue || e.Timestamp < gapEnd.Value)
{
gapEnd = e.Timestamp;
}
}
}
else if (i > 0)
{
gapStart = events[i - 1].Timestamp;
gapEnd = evt.Timestamp;
}
if (!gapStart.HasValue || !gapEnd.HasValue || gapEnd.Value <= gapStart.Value) continue;
var clamped = ClampToWindows(gapStart.Value, gapEnd.Value, windows);
if (!clamped.HasValue) continue;
// In-window seconds — a reboot gap bracketing the WhiteGlove pause contributes
// only its in-window flanks, never the pause itself.
var seconds = clamped.Value.Seconds;
if (seconds <= 0) continue;
result.Add(new RebootSpan
{
StartUtc = clamped.Value.Start,
EndUtc = clamped.Value.End,
Seconds = seconds,
SegmentKey = SegmentAt(spans, clamped.Value.Start),
});
}
return result;
}
private static string SegmentAt(List<TimeAttributionSpan> spans, DateTime ts)
{
foreach (var span in spans)
{
if (ts >= span.StartUtc && ts < span.EndUtc)
return span.SegmentKey;
}
return TimeAttributionSegments.Unattributed;
}
}
@@ -170,6 +170,11 @@ public static class EndpointAccessPolicyCatalog
// ticket is the sole authority — authz happened at mint time (diagnostics/download-ticket,
// MemberRead). See DiagnosticsTicketDownloadFunction.
new("GET", "diagnostics/download", EndpointPolicy.PublicAnonymous),
// Azure Monitor action-group webhook (Common Alert Schema → ops event). Action groups
// cannot send custom headers, so the shared secret travels as ?secret= and is compared
// in constant time against the OpsAlertWebhookSecret app setting (fail-closed while
// unset). See AzureMonitorAlertWebhookFunction.
new("POST", "ops/alert-webhook", EndpointPolicy.PublicAnonymous),
// ── DeviceOrBootstrapAuth ───────────────────────────────────────
new("POST", "agent/register-session", EndpointPolicy.DeviceOrBootstrapAuth),
@@ -227,8 +232,14 @@ public static class EndpointAccessPolicyCatalog
// (Inspector v1 — Plan §M6). Lift back to MemberRead+QueryParam at the v2 adminMode lift.
new("GET", "sessions/{sessionId}/analysis", EndpointPolicy.MemberRead, TenantScoping.QueryParam),
new("GET", "sessions/{sessionId}/vulnerability-report", EndpointPolicy.MemberRead, TenantScoping.QueryParam),
new("GET", "sessions/{sessionId}/time-attribution", EndpointPolicy.MemberRead, TenantScoping.QueryParam),
new("GET", "metrics/app", EndpointPolicy.MemberRead),
new("GET", "metrics/fleet-health", EndpointPolicy.MemberRead),
new("GET", "metrics/time-attribution", EndpointPolicy.MemberRead),
// F2 device history: QueryParam scoping so the session-detail banner works cross-tenant
// (GA/delegated drill-in) over the SAME route MCP uses — no global duplicate needed.
new("GET", "metrics/device-history", EndpointPolicy.MemberRead, TenantScoping.QueryParam),
new("GET", "metrics/device-journeys", EndpointPolicy.MemberRead),
new("GET", "apps/list", EndpointPolicy.MemberRead),
new("GET", "apps/{appName}/analytics", EndpointPolicy.MemberRead),
new("GET", "apps/{appName}/sessions", EndpointPolicy.MemberRead),
@@ -394,6 +405,8 @@ public static class EndpointAccessPolicyCatalog
new("GET", "global/metrics/platform", EndpointPolicy.GlobalReadOrAdmin),
new("GET", "global/metrics/app", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/fleet-health", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/time-attribution", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/device-journeys", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/apps/list", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/apps/{appName}/analytics", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/apps/{appName}/sessions", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
@@ -114,17 +114,20 @@ namespace AutopilotMonitor.Functions.Services.Deletion
await AddDiscriminatorPkPropStepAsync(manifest, order: 14, table: Constants.TableNames.ClassifierVerdictsByIdLevel, tenantId, sessionId, safeTenantId, safeSessionId, cancellationToken);
await AddDiscriminatorPkPropStepAsync(manifest, order: 15, table: Constants.TableNames.SignalsByKind, tenantId, sessionId, safeTenantId, safeSessionId, cancellationToken);
// ---- Steps 16 + 17: SoftwareInventory side-row (omit both for pre-side-row sessions). ----
// ---- Step 16: F1 time-attribution breakdown (PK=tenant, RK=sessionId; PR2). ----
await AddPkRkExactStepAsync(manifest, order: 16, table: Constants.TableNames.SessionTimeBreakdowns, partitionKey: tenantId, rowKey: sessionId, cancellationToken);
// ---- Steps 17 + 18: SoftwareInventory side-row (omit both for pre-side-row sessions). ----
var contributionsRow = await _reader.GetEntityOrNullAsync(
Constants.TableNames.SessionInventoryContributions, tenantId, sessionId, cancellationToken);
if (contributionsRow != null)
{
AddSoftwareInventoryDecrementStep(manifest, order: 16, contributionsRow);
AddContributionsRowStep(manifest, order: 17, contributionsRow);
AddSoftwareInventoryDecrementStep(manifest, order: 17, contributionsRow);
AddContributionsRowStep(manifest, order: 18, contributionsRow);
}
// ---- Step 18: Tombstone (SessionsIndex first, then Sessions). ----
AddTombstoneStep(manifest, order: 18, sessionsIndexRow, sessionRow);
// ---- Step 19: Tombstone (SessionsIndex first, then Sessions). ----
AddTombstoneStep(manifest, order: 19, sessionsIndexRow, sessionRow);
// PreflightCounts derive from each step's RowCount, plus the AGGREGATE decrements length.
manifest.PreflightCounts = ComputePreflightCounts(manifest);
@@ -9,7 +9,9 @@ namespace AutopilotMonitor.Functions.Services
/// </summary>
public sealed partial class EventIngestProcessor
{
private void AggregateAppInstallEvent(EnrollmentEvent evt, string tenantId, string sessionId, Dictionary<string, AppInstallAggregationState> summaries)
// internal static (was private instance): pure function over its parameters — exposed as a
// test seam so the TerminalState / status-fold contract is pinned by unit tests (PR0).
internal static void AggregateAppInstallEvent(EnrollmentEvent evt, string tenantId, string sessionId, Dictionary<string, AppInstallAggregationState> summaries)
{
bool isRelevant =
evt.EventType == "app_install_started" || evt.EventType == "app_install_start" ||
@@ -51,6 +53,23 @@ namespace AutopilotMonitor.Functions.Services
if (evt.Data != null)
{
// F1 PR1 (audit Q3): the summaries dict — like the RowKey — is keyed by app NAME,
// but events carry the Intune app identity in `appId`. Adopt the first observed
// appId; if the same name later shows a DIFFERENT appId (device- + user-scope
// assignment, duplicate display names across rings), flag the row as a collision —
// its folded status/duration mixes two real apps and per-app fleet aggregates
// must exclude it. First-seen appId wins so the row stays deterministic.
if (evt.Data.TryGetValue("appId", out var appIdObj))
{
var appId = appIdObj?.ToString()?.Trim();
if (!string.IsNullOrEmpty(appId))
{
if (string.IsNullOrEmpty(summary.AppId))
summary.AppId = appId!;
else if (!string.Equals(summary.AppId, appId, StringComparison.OrdinalIgnoreCase))
summary.AppIdCollision = true;
}
}
if (evt.Data.TryGetValue("appVersion", out var appVersionObj))
{
var appVersion = appVersionObj?.ToString();
@@ -110,10 +129,21 @@ namespace AutopilotMonitor.Functions.Services
summary.CompletedAt = evt.Timestamp;
if (summary.StartedAt != DateTime.MinValue)
summary.DurationSeconds = Math.Max(1, EventTimestampValidator.SafeDurationSeconds(summary.StartedAt, evt.Timestamp));
// PR0 (2026-07-26): the agent emits app_install_completed for EVERY terminal
// transition — including Skipped/Postponed (V1 wire parity). The payload's
// `state` field is the only way to tell a real install apart from a no-op
// (e.g. WinGet "Update for X" policies that were not applicable). Persist it
// so metrics can exclude skips from durations and rates. Unknown/absent state
// leaves the sentinel empty — never guessed.
var terminalStateRaw = evt.Data?.ContainsKey("state") == true
? evt.Data["state"]?.ToString() : null;
if (terminalStateRaw == "Installed" || terminalStateRaw == "Skipped" || terminalStateRaw == "Postponed")
summary.TerminalState = terminalStateRaw;
break;
case "app_install_failed":
summary.Status = "Failed";
summary.TerminalState = "Error";
summary.CompletedAt = evt.Timestamp;
if (summary.StartedAt != DateTime.MinValue)
summary.DurationSeconds = Math.Max(1, EventTimestampValidator.SafeDurationSeconds(summary.StartedAt, evt.Timestamp));
@@ -143,6 +173,10 @@ namespace AutopilotMonitor.Functions.Services
// Empty (sentinel: no observation yet) and "InProgress" both flip to Succeeded.
if (summary.Status == "InProgress" || summary.Status == string.Empty)
summary.Status = "Succeeded";
// Same guard for the terminal state: a dedicated skipped event never overrides
// a stronger terminal (Installed/Error) already seen in this batch.
if (string.IsNullOrEmpty(summary.TerminalState))
summary.TerminalState = "Skipped";
break;
case "download_progress":
@@ -143,6 +143,14 @@ namespace AutopilotMonitor.Functions.Services
}
}
if (TryComputeSessionApiLatency(request.Events, out var avgLatencyMs, out var apiRequestCount))
{
_ = _sessionRepo.UpdateSessionNetworkLatencyAsync(
request.TenantId, request.SessionId, avgLatencyMs, apiRequestCount)
.ContinueWith(t => _logger.LogWarning(t.Exception?.InnerException,
"Fire-and-forget UpdateSessionNetworkLatencyAsync failed"), TaskContinuationOptions.OnlyOnFaulted);
}
var classification = ClassifyEvents(storedEvents);
foreach (var summary in classification.AppInstallUpdates.Values)
@@ -395,6 +403,58 @@ namespace AutopilotMonitor.Functions.Services
};
}
/// <summary>
/// Session-wide API latency projection: the agent emits cumulative counters
/// (net_total_latency_ms / net_total_requests) in every agent_metrics_snapshot, so
/// the LAST snapshot of the batch carries the whole-session average up to now and a
/// plain overwrite is idempotent against replays. Agents predating the field simply
/// never match (net_total_latency_ms absent) — no fallback, fleet turns over
/// per-enrollment. Exposed as internal for unit testing.
/// </summary>
internal static bool TryComputeSessionApiLatency(
List<EnrollmentEvent> events, out double avgLatencyMs, out int requestCount)
{
avgLatencyMs = 0;
requestCount = 0;
var snapshot = events.LastOrDefault(e =>
e.EventType == Shared.Constants.EventTypes.AgentMetricsSnapshot &&
e.Data?.ContainsKey("net_total_latency_ms") == true);
if (snapshot?.Data == null ||
!TryGetDouble(snapshot.Data, "net_total_latency_ms", out var totalLatencyMs) ||
!TryGetDouble(snapshot.Data, "net_total_requests", out var totalRequests) ||
totalRequests <= 0)
{
return false;
}
avgLatencyMs = Math.Round(totalLatencyMs / totalRequests, 1);
requestCount = (int)totalRequests;
return true;
}
/// <summary>
/// Numeric reader for agent event Data values, which arrive as boxed Newtonsoft
/// primitives (integer → long, decimal → double) — same coercion ladder as
/// PlatformMetricsService.GetDouble (the proven parser for these snapshot fields),
/// plus invariant-culture string parsing. Exposed as internal for unit testing.
/// </summary>
internal static bool TryGetDouble(Dictionary<string, object> data, string key, out double value)
{
value = 0;
if (!data.TryGetValue(key, out var raw))
return false;
switch (raw)
{
case double d: value = d; return true;
case int i: value = i; return true;
case long l: value = l; return true;
case float f: value = f; return true;
}
return double.TryParse(raw?.ToString(),
System.Globalization.NumberStyles.Float,
System.Globalization.CultureInfo.InvariantCulture, out value);
}
/// <summary>
/// Stamps authoritative server-side fields onto all events before storage.
/// TenantId and SessionId always come from the validated request metadata,
@@ -260,6 +260,298 @@ namespace AutopilotMonitor.Functions.Services
}
}
// ===== F1 TIME ATTRIBUTION (insights spec §F1, PR2) =====
/// <summary>Rolling sweep window: sessions started in the last 30 days (spec backfill horizon).</summary>
private const int TimeAttributionSweepDays = 30;
/// <summary>
/// RowKey date component of the rolling-window aggregate rows ("rolling30|{class}").
/// Sorts after every "yyyy-MM-dd|…" key, so date-range reads and the age-based retention
/// sweep never touch these perpetually-refreshed rows.
/// </summary>
internal const string RollingAggregateDateKey = "rolling30";
/// <summary>
/// Self-healing sweep for the F1 time attribution — deliberately NOT hooked into the
/// usage-snapshot catch-up (that pass skips dates whose snapshot already exists, and a
/// session that terminates days after it STARTED would then never reach its date's
/// aggregate). Owns both halves for the rolling window:
/// 1. Breakdown backfill: every terminal Succeeded/Failed session gets its breakdown
/// computed if missing or written by an older AttributionVersion (rule 8: aggregates
/// never mix algorithm versions — a stale row that cannot be recomputed, because its
/// events aged out, counts as missing rather than polluting the stats). The inline
/// terminal-transition compute is the primary writer; this converges to point-read
/// existence checks.
/// 2. Daily aggregates: recomputed idempotently per (tenant × enrollment class × date)
/// plus the "global" rows — late-terminating sessions and backfilled breakdowns are
/// folded in on the next run by construction.
/// Sessions with a deletion cascade in flight are skipped so the sweep cannot write a
/// breakdown row after the session's deletion manifest was snapshotted.
/// </summary>
private async Task SweepTimeAttributionAsync()
{
try
{
var sw = Stopwatch.StartNew();
var today = DateTime.UtcNow.Date;
var sessions = await _maintenanceRepo.GetSessionsByDateRangeAsync(
today.AddDays(-TimeAttributionSweepDays), today.AddDays(1));
var pairs = new List<(SessionSummary Session, SessionTimeBreakdown Breakdown)>();
var missing = new List<SessionSummary>();
var computed = 0;
foreach (var session in sessions)
{
if (session.Status != SessionStatus.Succeeded && session.Status != SessionStatus.Failed)
continue;
if (!session.DurationSeconds.HasValue || session.DurationSeconds.Value <= 0)
continue;
if (!string.IsNullOrEmpty(session.DeletionState) && session.DeletionState != "None")
continue;
var breakdown = await _metricsRepo.GetSessionTimeBreakdownAsync(session.TenantId, session.SessionId);
// Recompute when the row is missing, algorithm-stale, OR computed from an
// incomplete event stream: the inline terminal compute is one-shot, but
// batches can arrive AFTER the terminal transition (upload lag, replays,
// the agent's async esp_config re-emits) — the session's EventCount is the
// change signal (Codex review finding).
var eventCountMoved = breakdown != null && breakdown.EventCountAtCompute != session.EventCount;
if (breakdown == null || breakdown.AttributionVersion != TimeAttributionCalculator.CurrentVersion || eventCountMoved)
{
var hadNoRow = breakdown == null;
breakdown = await _metricsRepo.ComputeAndStoreSessionTimeBreakdownAsync(session.TenantId, session.SessionId);
if (breakdown != null) computed++;
// Late batches can also carry esp_config_detected re-emits and app
// events — re-join the blocking evidence from the fuller stream.
// Idempotent, positive-evidence-only; also heals sessions that went
// terminal before the resolution seam existed (hadNoRow backfill).
if (eventCountMoved || hadNoRow)
await _metricsRepo.ResolveEspBlockingForSessionAsync(session.TenantId, session.SessionId);
}
if (breakdown == null)
missing.Add(session); // events aged out before backfill — disclosed, never guessed
else
pairs.Add((session, breakdown));
}
var now = DateTime.UtcNow;
var aggregates = BuildTimeAttributionAggregates(pairs, missing, now);
// Rolling window rows (RK "rolling30|{class}"): range statistics computed over the
// window's actual sessions. The daily rows can NOT be merged into a range claim
// (a median of per-day medians is not the range median), so the fleet panel reads
// these; the daily rows serve the per-day trend. Refreshed whole every sweep.
// "Last 30 days" means exactly 30 calendar days including today — the sweep's own
// query window is one day wider (backfill/late-heal coverage), so filter here.
var rollingWindowStart = today.AddDays(-(TimeAttributionSweepDays - 1));
aggregates.AddRange(BuildTimeAttributionAggregates(
pairs.Where(p => p.Session.StartedAt >= rollingWindowStart).ToList(),
missing.Where(s => s.StartedAt >= rollingWindowStart).ToList(),
now, RollingAggregateDateKey));
var saved = 0;
foreach (var aggregate in aggregates)
{
if (await _metricsRepo.SaveTimeAttributionAggregateAsync(aggregate))
saved++;
}
// Stale-bucket reconcile (Codex review): a bucket that existed on a previous run
// but was NOT regenerated now (its sessions were deleted, or its class left the
// window) must not keep serving old numbers — a rolling row would otherwise stay
// stale forever. Scope: every partition this run wrote rows for; a partition that
// went FULLY quiet is not enumerable here — its rolling rows are neutralized
// read-side by the ComputedAt age filter in TimeAttributionResponse, and its
// daily rows age out of every queried window by construction.
var generatedKeys = new HashSet<(string TenantId, string Date, string Class)>(
aggregates.Select(a => (a.TenantId, a.Date, a.EnrollmentClass)));
var removedStale = 0;
foreach (var partition in aggregates.Select(a => a.TenantId).Distinct().ToList())
{
var existingRows = await _metricsRepo.GetRollingTimeAttributionAggregatesAsync(partition);
existingRows.AddRange(await _metricsRepo.GetTimeAttributionAggregatesAsync(
partition, today.AddDays(-TimeAttributionSweepDays), today));
foreach (var row in existingRows)
{
if (generatedKeys.Contains((partition, row.Date, row.EnrollmentClass))) continue;
await _metricsRepo.DeleteTimeAttributionAggregateAsync(partition, row.Date, row.EnrollmentClass);
removedStale++;
}
}
sw.Stop();
_logger.LogInformation(
"Time-attribution sweep: {Sessions} terminal sessions in {Days}d window, {Computed} breakdowns computed, {Missing} missing, {Aggregates} aggregate rows written, {Removed} stale buckets removed in {Ms}ms",
pairs.Count + missing.Count, TimeAttributionSweepDays, computed, missing.Count, saved, removedStale, sw.ElapsedMilliseconds);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Time-attribution sweep failed (non-fatal)");
}
}
/// <summary>
/// Pure aggregation core behind <see cref="SweepTimeAttributionAsync"/> — internal static
/// so the bucketing/gating contract is pinned by unit tests. Per (tenant × class × date)
/// bucket plus mirrored "global" rows: only breakdowns without DURATION-critical flags
/// (<see cref="TimeAttributionFlagQuality.DurationCriticalFlags"/>) form the statistics —
/// blocking-set-only flags stay in (their spans are sound; unknown blocking simply
/// contributes no per-app intervals); excluded and missing ones are counted, never
/// silently dropped (rule 7).
/// Rows are written even below the ≥20 UI gate — the UI needs the n (rule 4). Segment
/// stats always carry the five canonical segments + unattributed (a session without a
/// span of a segment contributes 0 — the honest "per enrollment of this class" answer).
/// Per-app rows gate at ≥5 sessions, order by median interval, cap 20.
/// </summary>
internal static List<TimeAttributionDailyAggregate> BuildTimeAttributionAggregates(
IReadOnlyList<(SessionSummary Session, SessionTimeBreakdown Breakdown)> pairs,
IReadOnlyList<SessionSummary> missing,
DateTime computedAtUtc,
string? fixedDateKey = null)
{
const int minSessionsPerAppRow = 5;
const int maxAppRows = 20;
var buckets = new Dictionary<(string TenantId, string Date, string Class), List<SessionTimeBreakdown>>();
var flaggedCounts = new Dictionary<(string, string, string), int>();
var missingCounts = new Dictionary<(string, string, string), int>();
void Bump(Dictionary<(string, string, string), int> map, (string, string, string) key)
{
map.TryGetValue(key, out var current);
map[key] = current + 1;
}
foreach (var (session, breakdown) in pairs)
{
var date = fixedDateKey ?? session.StartedAt.ToString("yyyy-MM-dd");
var cls = TimeAttributionCalculator.GetEnrollmentClass(session);
foreach (var tenantKey in new[] { session.TenantId, "global" })
{
var key = (tenantKey, date, cls);
if (TimeAttributionFlagQuality.ExcludesFromFleetStats(breakdown.QualityFlags))
{
Bump(flaggedCounts, key);
// Flagged sessions still materialize the bucket so a day whose sessions
// were ALL flagged yields a row disclosing exactly that.
if (!buckets.ContainsKey(key)) buckets[key] = new List<SessionTimeBreakdown>();
continue;
}
if (!buckets.TryGetValue(key, out var list))
{
list = new List<SessionTimeBreakdown>();
buckets[key] = list;
}
list.Add(breakdown);
}
}
foreach (var session in missing)
{
var date = fixedDateKey ?? session.StartedAt.ToString("yyyy-MM-dd");
var cls = TimeAttributionCalculator.GetEnrollmentClass(session);
foreach (var tenantKey in new[] { session.TenantId, "global" })
{
var key = (tenantKey, date, cls);
Bump(missingCounts, key);
if (!buckets.ContainsKey(key)) buckets[key] = new List<SessionTimeBreakdown>();
}
}
var segmentKeys = new[]
{
TimeAttributionSegments.DevicePrep,
TimeAttributionSegments.EspApps,
TimeAttributionSegments.IdentityHello,
TimeAttributionSegments.UserEsp,
TimeAttributionSegments.DesktopHandoff,
TimeAttributionSegments.Unattributed,
};
var result = new List<TimeAttributionDailyAggregate>();
foreach (var entry in buckets)
{
var (tenantId, date, cls) = entry.Key;
var clean = entry.Value;
var aggregate = new TimeAttributionDailyAggregate
{
TenantId = tenantId,
Date = date,
EnrollmentClass = cls,
AttributionVersion = TimeAttributionCalculator.CurrentVersion,
CleanSessionCount = clean.Count,
FlaggedExcludedCount = flaggedCounts.TryGetValue(entry.Key, out var f) ? f : 0,
MissingBreakdownCount = missingCounts.TryGetValue(entry.Key, out var m) ? m : 0,
ComputedAt = computedAtUtc,
};
if (clean.Count > 0)
{
var totalsPerSession = clean
.Select(b =>
{
var totals = b.GetSegmentTotals();
totals[TimeAttributionSegments.Unattributed] = b.UnattributedSeconds;
return totals;
})
.ToList();
foreach (var segmentKey in segmentKeys)
{
var values = totalsPerSession
.Select(t => t.TryGetValue(segmentKey, out var v) ? (double)v : 0d)
.OrderBy(v => v)
.ToList();
aggregate.SegmentStats.Add(new TimeAttributionSegmentStat
{
SegmentKey = segmentKey,
MedianSeconds = (int)Math.Round(MetricsMath.Percentile(values, 50)),
P75Seconds = (int)Math.Round(MetricsMath.Percentile(values, 75)),
P90Seconds = (int)Math.Round(MetricsMath.Percentile(values, 90)),
});
}
aggregate.TopBlockingApps = clean
.SelectMany(b => b.BlockingApps.Select(interval => (Breakdown: b, Interval: interval)))
.GroupBy(x => x.Interval.AppId, StringComparer.OrdinalIgnoreCase)
.Where(g => g.Count() >= minSessionsPerAppRow)
.Select(g =>
{
var seconds = g.Select(x => (double)x.Interval.Seconds).OrderBy(v => v).ToList();
var savings = g
// Cap at the session's wall clock: the what-if delta is computed
// from raw interval endpoints, which can straddle the WhiteGlove
// pause — no removal can save more than the enrollment took.
.Select(x => (double)Math.Min(
TimeAttributionCalculator.WhatIfSavingSeconds(x.Breakdown.BlockingApps, g.Key),
x.Breakdown.WallClockSeconds))
.OrderBy(v => v)
.ToList();
return new TimeAttributionBlockingAppStat
{
AppId = g.Key,
AppName = g.Select(x => x.Interval.AppName).FirstOrDefault(n => !string.IsNullOrEmpty(n)) ?? string.Empty,
SessionCount = g.Count(),
MedianSeconds = (int)Math.Round(MetricsMath.Percentile(seconds, 50)),
P75Seconds = (int)Math.Round(MetricsMath.Percentile(seconds, 75)),
MedianSavingSeconds = (int)Math.Round(MetricsMath.Percentile(savings, 50)),
P75SavingSeconds = (int)Math.Round(MetricsMath.Percentile(savings, 75)),
};
})
.OrderByDescending(a => a.MedianSeconds)
.ThenBy(a => a.AppName, StringComparer.OrdinalIgnoreCase)
.Take(maxAppRows)
.ToList();
}
result.Add(aggregate);
}
return result;
}
/// <summary>
/// Deletes usage tracking records older than 90 days from UserUsageLog.
/// </summary>
@@ -782,6 +1074,34 @@ namespace AutopilotMonitor.Functions.Services
{
_logger.LogWarning(ex, "Failed to cleanup old backup job records");
}
// Time-attribution daily aggregates: same 180d window as the usage-metrics snapshots
// they sit beside. (SessionTimeBreakdowns needs no age sweep — breakdown rows are
// deleted with their session via the deletion-manifest cascade / offboarding wipe.)
try
{
var deleted = await _metricsRepo.DeleteTimeAttributionAggregatesOlderThanAsync(now.AddDays(-usageMetricsRetentionDays));
if (deleted > 0)
_logger.LogInformation("Time-attribution aggregate cleanup: deleted {Count} rows older than {Days} days", deleted, usageMetricsRetentionDays);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to cleanup old time-attribution aggregates");
}
// Device-journey (FTR) daily aggregates: same 180d window. (DeviceHistories needs no
// age sweep — chain refs of deleted sessions are pruned tombstone-driven by
// SweepDeviceJourneysAsync and the rows die with tenant offboarding.)
try
{
var deleted = await _metricsRepo.DeleteDeviceJourneyAggregatesOlderThanAsync(now.AddDays(-usageMetricsRetentionDays));
if (deleted > 0)
_logger.LogInformation("Device-journey aggregate cleanup: deleted {Count} rows older than {Days} days", deleted, usageMetricsRetentionDays);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to cleanup old device-journey aggregates");
}
}
/// <summary>
/// Detects and cleans up orphaned events — events stored in the Events table
@@ -0,0 +1,278 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Linq;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// Partial: F2 device-history / First-Time-Right sweep (insights spec §F2, PR4).
/// </summary>
public partial class MaintenanceService
{
/// <summary>Rolling sweep window: sessions started in the last 30 days (mirrors the F1 attribution sweep).</summary>
private const int DeviceJourneySweepDays = 30;
/// <summary>
/// Self-healing sweep for the F2 device histories + daily FTR aggregates. The inline
/// terminal-transition update is the primary chain writer; this pass owns everything the
/// inline path cannot:
/// 1. Deleted-session cleanup: every session deletion (cascade + retention) leaves a
/// SessionTombstones marker for ~7d — far longer than the sweep cadence — so dropping
/// chain refs whose sessionId is tombstoned keeps chains truthful without per-ref
/// existence point-reads. A chain left empty is deleted (a device with no observable
/// sessions has no history claim).
/// 2. Chain backfill/heal: terminal sessions of the rolling window are merged into their
/// device chains (pre-deploy sessions, inline misses, re-terminal reclassifications).
/// 3. Daily FTR aggregates: recomputed idempotently per (tenant × completion date) plus
/// "global" mirror rows from the freshly merged chains — a journey buckets on the
/// StartedAt date of its completing success session, so late-terminating sessions are
/// folded into their date on the next pass by construction (F1 bucketing convention).
/// Cost: one tombstone-table scan, one window session scan, and ~1 point-read + 1 upsert
/// per window device per tick — same order as the F1 sweep; negligible at current fleet
/// size (the scaling lever is gating the per-device RMW on new terminal sessions).
/// </summary>
private async Task SweepDeviceJourneysAsync()
{
try
{
var sw = Stopwatch.StartNew();
var today = DateTime.UtcNow.Date;
var windowStart = today.AddDays(-DeviceJourneySweepDays);
var windowEnd = today.AddDays(1);
// ---- Phase 1: drop chain refs of deleted sessions (tombstone-driven). ----
var tombstones = await _maintenanceRepo.GetAllSessionTombstoneKeysAsync();
var deletedByTenant = tombstones
.GroupBy(t => t.TenantId, StringComparer.OrdinalIgnoreCase)
.ToDictionary(
g => g.Key,
g => (ISet<string>)new HashSet<string>(g.Select(x => x.SessionId), StringComparer.Ordinal),
StringComparer.OrdinalIgnoreCase);
var refsCleaned = 0;
var rowsDeleted = 0;
foreach (var pair in deletedByTenant)
{
var histories = await _metricsRepo.GetDeviceHistoriesByTenantAsync(pair.Key);
foreach (var history in histories)
{
if (!history.Chain.Any(r => pair.Value.Contains(r.SessionId)))
continue;
var cleaned = DeviceJourneyCalculator.RemoveSessionRefs(history.Chain, pair.Value);
if (cleaned.Count == 0)
{
await _metricsRepo.DeleteDeviceHistoryAsync(pair.Key, history.SerialKey);
rowsDeleted++;
continue;
}
var (journeyCount, currentAttempts) = DeviceJourneyCalculator.Derive(cleaned);
history.Chain = cleaned;
history.JourneyCount = journeyCount;
history.CurrentJourneyAttempts = currentAttempts;
history.JourneyVersion = DeviceJourneyCalculator.CurrentVersion;
history.LastUpdated = DateTime.UtcNow;
await _metricsRepo.UpsertDeviceHistoryAsync(history);
refsCleaned++;
}
}
// ---- Phase 2: merge the window's terminal sessions into their device chains. ----
var sessions = await _maintenanceRepo.GetSessionsByDateRangeAsync(windowStart, windowEnd);
var excludedSessions = new List<SessionSummary>();
var byDevice = new Dictionary<(string TenantId, string SerialKey), List<SessionSummary>>();
foreach (var session in sessions)
{
if (!DeviceJourneyCalculator.IsTerminal(session.Status))
continue;
// Never (re-)add a ref for a session inside a deletion cascade — its
// tombstone may already have been consumed by an earlier pass.
if (!string.IsNullOrEmpty(session.DeletionState) && session.DeletionState != "None")
continue;
var serialKey = DeviceJourneyCalculator.NormalizeSerial(session.SerialNumber);
if (serialKey == null)
{
excludedSessions.Add(session); // junk serial — disclosed, never silently dropped
continue;
}
var key = (session.TenantId, serialKey);
if (!byDevice.TryGetValue(key, out var list))
{
list = new List<SessionSummary>();
byDevice[key] = list;
}
list.Add(session);
}
var mergedHistories = new List<DeviceHistory>(byDevice.Count);
foreach (var pair in byDevice)
{
var (tenantId, serialKey) = pair.Key;
var existing = await _metricsRepo.GetDeviceHistoryAsync(tenantId, serialKey);
var refs = pair.Value
.Select(DeviceJourneyCalculator.BuildSessionRef)
.Where(r => r != null)
.Select(r => r!);
var chain = DeviceJourneyCalculator.MergeChain(existing?.Chain, refs);
if (deletedByTenant.TryGetValue(tenantId, out var deletedIds))
chain = DeviceJourneyCalculator.RemoveSessionRefs(chain, deletedIds);
if (chain.Count == 0)
continue;
// Display fields follow the chain's newest entry (BuildDeviceHistoryRow rule);
// pick the window session that IS that entry, falling back to any of the
// device's window sessions (then the existing row's values win inside).
var newestId = chain[chain.Count - 1].SessionId;
var displaySource = pair.Value.FirstOrDefault(s => s.SessionId == newestId) ?? pair.Value[0];
var history = TableStorageService.BuildDeviceHistoryRow(tenantId, serialKey, chain, existing, displaySource);
await _metricsRepo.UpsertDeviceHistoryAsync(history);
mergedHistories.Add(history);
}
// ---- Phase 3: daily FTR aggregates from the merged chains. ----
var aggregates = BuildDeviceJourneyAggregates(
mergedHistories, excludedSessions, windowStart, windowEnd, DateTime.UtcNow);
var saved = 0;
foreach (var aggregate in aggregates)
{
if (await _metricsRepo.SaveDeviceJourneyAggregateAsync(aggregate))
saved++;
}
// Stale-bucket reconcile (Codex review): a date row that was not regenerated this
// run (its last completing journey was deleted or reclassified) must not keep
// serving old counts — FTR window rates are SUMS over these rows. Scope: every
// partition this run wrote rows for; fully-quiet partitions' rows age out of any
// queried window by construction (the FTR query is always today-anchored).
var generatedKeys = new HashSet<(string TenantId, string Date)>(
aggregates.Select(a => (a.TenantId, a.Date)));
var removedStale = 0;
foreach (var partition in aggregates.Select(a => a.TenantId).Distinct().ToList())
{
var existingRows = await _metricsRepo.GetDeviceJourneyAggregatesAsync(
partition, windowStart, windowEnd.AddDays(-1));
foreach (var row in existingRows)
{
if (generatedKeys.Contains((partition, row.Date))) continue;
await _metricsRepo.DeleteDeviceJourneyAggregateAsync(partition, row.Date);
removedStale++;
}
}
sw.Stop();
_logger.LogInformation(
"Device-journey sweep: {Devices} devices from {Sessions} terminal window sessions, {RefsCleaned} chains cleaned / {RowsDeleted} rows deleted (tombstones), {Excluded} junk-serial sessions excluded, {Aggregates} FTR rows written, {Removed} stale buckets removed in {Ms}ms",
mergedHistories.Count, byDevice.Sum(d => d.Value.Count), refsCleaned, rowsDeleted, excludedSessions.Count, saved, removedStale, sw.ElapsedMilliseconds);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Device-journey sweep failed (non-fatal)");
}
}
/// <summary>
/// Pure aggregation core behind <see cref="SweepDeviceJourneysAsync"/> — internal static
/// so the FTR contract is pinned by unit tests. A journey counts on the StartedAt date
/// (UTC) of its COMPLETING success session; only completed journeys enter numerator and
/// denominator (open and gap-abandoned journeys never count — spec §F2). Junk-serial
/// exclusions are disclosed per day (rule 7), producing disclosure-only rows when a day
/// has no completed journey. Rows are written even below the ≥20 UI gate (rule 4) and
/// mirrored into the "global" partition. Counts are additive across days — window rates
/// are sums over daily rows, so no rolling-window row exists (unlike the median-based F1
/// aggregates).
/// </summary>
internal static List<DeviceJourneyDailyAggregate> BuildDeviceJourneyAggregates(
IReadOnlyList<DeviceHistory> histories,
IReadOnlyList<SessionSummary> excludedSessions,
DateTime windowStartDate,
DateTime windowEndDateExclusive,
DateTime computedAtUtc)
{
var buckets = new Dictionary<(string TenantId, string Date), JourneyBucket>();
JourneyBucket Bucket((string, string) key)
{
if (!buckets.TryGetValue(key, out var bucket))
{
bucket = new JourneyBucket();
buckets[key] = bucket;
}
return bucket;
}
foreach (var history in histories)
{
foreach (var journey in DeviceJourneyCalculator.GroupJourneys(history.Chain))
{
if (!journey.Completed)
continue;
var completedOn = journey.CompletingRef!.StartedAt.Date;
if (completedOn < windowStartDate || completedOn >= windowEndDateExclusive)
continue;
var date = completedOn.ToString("yyyy-MM-dd");
var attempts = journey.Attempts.Count;
foreach (var tenantKey in new[] { history.TenantId, "global" })
{
var bucket = Bucket((tenantKey, date));
bucket.CompletedJourneys++;
if (attempts == 1) bucket.FirstTimeRight++;
bucket.Histogram.TryGetValue(attempts, out var count);
bucket.Histogram[attempts] = count + 1;
}
}
}
foreach (var session in excludedSessions)
{
var date = session.StartedAt.Date;
if (date < windowStartDate || date >= windowEndDateExclusive)
continue;
var dateStr = date.ToString("yyyy-MM-dd");
foreach (var tenantKey in new[] { session.TenantId, "global" })
{
Bucket((tenantKey, dateStr)).ExcludedSessions++;
}
}
var result = new List<DeviceJourneyDailyAggregate>(buckets.Count);
foreach (var entry in buckets)
{
var (tenantId, date) = entry.Key;
result.Add(new DeviceJourneyDailyAggregate
{
TenantId = tenantId,
Date = date,
JourneyVersion = DeviceJourneyCalculator.CurrentVersion,
CompletedJourneyCount = entry.Value.CompletedJourneys,
FirstTimeRightCount = entry.Value.FirstTimeRight,
AttemptHistogram = entry.Value.Histogram
.OrderBy(h => h.Key)
.Select(h => new DeviceJourneyAttemptBucket { Attempts = h.Key, JourneyCount = h.Value })
.ToList(),
ExcludedSessionCount = entry.Value.ExcludedSessions,
ComputedAt = computedAtUtc,
});
}
return result;
}
private sealed class JourneyBucket
{
public int CompletedJourneys;
public int FirstTimeRight;
public int ExcludedSessions;
public readonly Dictionary<int, int> Histogram = new Dictionary<int, int>();
}
}
}
@@ -0,0 +1,266 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Globalization;
using System.Linq;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// Partial: F3 regression radar (insights spec §F3, PR6) — analyze rules only.
/// </summary>
public partial class MaintenanceService
{
/// <summary>App setting kill switch (spec §F3): set to "true" to skip the radar entirely. Fail-open — the radar only notifies, it never mutates data.</summary>
internal const string RadarKillSwitchSetting = "RuleRegressionRadarDisabled";
/// <summary>
/// Detects per-tenant analyze rules whose hit rate regressed (7d window ≥2× the prior
/// 28d baseline, Wilson-separated — <see cref="RuleRegressionRadar"/>) and reconciles the
/// alert episodes in the notification tracker:
/// new finding → dimension correlation (on-fire only) + tracker row + tenant bell +
/// RuleFrequencyRegression ops event, exactly once per episode; still-regressed → numbers
/// refreshed (FirstNotifiedAt untouched); no longer regressed → re-arm check
/// (fires stopped, or rate back under 1.5× baseline) deletes the row so the badge clears
/// and a future spike rings again. The tracker's 30d retention sweep re-arms
/// long-burning episodes by design (spec: retention cleanup re-arms).
/// <para>
/// Anchored on <paramref name="targetDate"/> (timer path: yesterday — whole days only,
/// a partial "today" would understate the window rate). Idempotent per anchor: re-runs
/// hit the tracker dedup. Fail-soft per tenant and overall.
/// </para>
/// </summary>
private async Task RunRuleRegressionRadarAsync(DateTime targetDate)
{
try
{
if (string.Equals(_configuration[RadarKillSwitchSetting], "true", StringComparison.OrdinalIgnoreCase))
{
_logger.LogInformation("Rule-regression radar skipped ({Setting}=true)", RadarKillSwitchSetting);
return;
}
var sw = Stopwatch.StartNew();
var horizonStart = targetDate.Date.AddDays(-(RuleRegressionRadar.WindowDays - 1 + RuleRegressionRadar.BaselineDays));
var entries = await _metricsRepo.GetRuleStatsAsync(
tenantId: null,
startDate: horizonStart.ToString("yyyy-MM-dd"),
endDate: targetDate.Date.ToString("yyyy-MM-dd"),
ruleType: "analyze",
maxResults: int.MaxValue);
var byTenant = entries
.Where(e => !string.Equals(e.TenantId, "global", StringComparison.OrdinalIgnoreCase))
.GroupBy(e => e.TenantId, StringComparer.OrdinalIgnoreCase);
int fired = 0, refreshed = 0, rearmed = 0, tenants = 0;
foreach (var tenantGroup in byTenant)
{
tenants++;
try
{
var (f, r, a) = await EvaluateTenantRegressionsAsync(tenantGroup.Key, tenantGroup.ToList(), targetDate.Date);
fired += f;
refreshed += r;
rearmed += a;
}
catch (Exception tenantEx)
{
_logger.LogWarning(tenantEx, "Rule-regression radar failed for tenant {TenantId} (non-fatal)", tenantGroup.Key);
}
}
sw.Stop();
_logger.LogInformation(
"Rule-regression radar: {Tenants} tenants over {Rows} stat rows — {Fired} fired, {Refreshed} refreshed, {Rearmed} re-armed in {Ms}ms (anchor {Anchor})",
tenants, entries.Count, fired, refreshed, rearmed, sw.ElapsedMilliseconds, targetDate.Date.ToString("yyyy-MM-dd"));
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Rule-regression radar failed (non-fatal)");
}
}
private async Task<(int Fired, int Refreshed, int Rearmed)> EvaluateTenantRegressionsAsync(
string tenantId, List<RuleStatsEntry> tenantEntries, DateTime targetDate)
{
// Entity timestamps drive the suppression gates (grace period via CreatedAt,
// edit-in-window via UpdatedAt); a rule without an entity (deleted) never alerts.
var rules = await _analyzeRuleService.GetAllRulesForTenantAsync(tenantId);
var timestamps = new Dictionary<string, RuleRegressionRadar.RuleTimestamps>(StringComparer.OrdinalIgnoreCase);
foreach (var rule in rules)
{
if (!string.IsNullOrEmpty(rule.RuleId))
timestamps[rule.RuleId] = new RuleRegressionRadar.RuleTimestamps(rule.CreatedAt, rule.UpdatedAt);
}
var findings = RuleRegressionRadar.Evaluate(tenantEntries, targetDate, timestamps);
var active = await _hardwareRejectionTracker.GetRuleRegressionsAsync(tenantId);
var activeByRule = active.ToDictionary(a => a.RuleId, StringComparer.OrdinalIgnoreCase);
var groups = RuleRegressionRadar.AnalyzeRuleGroups(tenantEntries);
int fired = 0, refreshed = 0, rearmed = 0;
var firedRuleIds = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
foreach (var finding in findings)
{
firedRuleIds.Add(finding.RuleId);
if (activeByRule.TryGetValue(finding.RuleId, out var existing))
{
// Episode still burning: refresh numbers for the badge/regressions[] block;
// the dimension stays the first-fire capture (stable story) and
// FirstNotifiedAt never moves (retention re-arm counts from the bell).
await _hardwareRejectionTracker.RefreshRuleRegressionAsync(
tenantId, BuildAlert(finding, existing.Dimension, existing.FirstNotifiedAt));
refreshed++;
continue;
}
var dimension = await TryCorrelateDimensionsAsync(tenantId, finding);
var alert = BuildAlert(finding, dimension, DateTime.UtcNow);
if (!await _hardwareRejectionTracker.TryRegisterRuleRegressionAsync(tenantId, alert))
continue; // a concurrent pass won the episode — its bell suffices
fired++;
var dimensionSummary = DescribeDimension(dimension);
await _tenantNotificationService.CreateNotificationAsync(
tenantId,
type: "rule_frequency_regression",
title: $"Rule firing more often: {finding.RuleTitle}",
message: BuildRegressionMessage(finding, dimensionSummary),
href: $"/analyze-rules#rule-card-{Uri.EscapeDataString(finding.RuleId)}");
await _opsEventService.RecordRuleFrequencyRegressionAsync(
tenantId, finding.RuleId, finding.RuleTitle,
finding.WindowFireCount, finding.WindowSessionCount, finding.WindowRatePct,
finding.BaselineFireCount, finding.BaselineSessionCount, finding.BaselineRatePct,
finding.Lift, dimensionSummary);
}
foreach (var alert in active)
{
if (firedRuleIds.Contains(alert.RuleId))
continue;
groups.TryGetValue(alert.RuleId, out var ruleEntries);
if (RuleRegressionRadar.ShouldReArm(ruleEntries ?? new List<RuleStatsEntry>(), targetDate))
{
await _hardwareRejectionTracker.DeleteRuleRegressionAsync(tenantId, alert.RuleId);
rearmed++;
}
else if (ruleEntries != null)
{
// Elevated but no longer fully separated (between 1.5× and the fire gates):
// keep the episode, refresh the numbers so badge and regressions[] stay honest.
var sums = RuleRegressionRadar.SumWindows(ruleEntries, targetDate);
await _hardwareRejectionTracker.RefreshRuleRegressionAsync(
tenantId, BuildOngoingAlert(alert, sums, targetDate));
refreshed++;
}
}
return (fired, refreshed, rearmed);
}
/// <summary>
/// On-fire dimension correlation: the window's sessions vs. the rule's hit sessions
/// (RuleResults scan intersected with the loaded window — a hit whose session started
/// before the window is dropped rather than skewing the shares). Fail-soft: any error
/// yields null, and the alert simply carries no dimension claim (never a guessed one).
/// </summary>
private async Task<RuleRegressionDimension?> TryCorrelateDimensionsAsync(string tenantId, RuleRegressionFinding finding)
{
try
{
var windowStart = DateTime.ParseExact(
finding.WindowStartDate, "yyyy-MM-dd", CultureInfo.InvariantCulture,
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal);
var windowEndExclusive = DateTime.ParseExact(
finding.WindowEndDate, "yyyy-MM-dd", CultureInfo.InvariantCulture,
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal).AddDays(1);
var allSessions = await _maintenanceRepo.GetSessionsByDateRangeAsync(windowStart, windowEndExclusive, tenantId);
if (allSessions.Count == 0) return null;
var hitIds = await _ruleRepo.GetRuleHitSessionIdsAsync(tenantId, finding.RuleId, windowStart);
var hitSet = new HashSet<string>(hitIds, StringComparer.OrdinalIgnoreCase);
var hitSessions = allSessions.Where(s => hitSet.Contains(s.SessionId)).ToList();
return RuleRegressionRadar.ComputeDimensionConcentration(hitSessions, allSessions);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Rule-regression dimension correlation failed for rule {RuleId} in tenant {TenantId}", finding.RuleId, tenantId);
return null;
}
}
private static RuleRegressionAlert BuildAlert(
RuleRegressionFinding finding, RuleRegressionDimension? dimension, DateTime firstNotifiedAt)
=> new()
{
TenantId = finding.TenantId,
RuleId = finding.RuleId,
RuleTitle = finding.RuleTitle,
WindowFireCount = finding.WindowFireCount,
WindowSessionCount = finding.WindowSessionCount,
BaselineFireCount = finding.BaselineFireCount,
BaselineSessionCount = finding.BaselineSessionCount,
WindowRatePct = finding.WindowRatePct,
BaselineRatePct = finding.BaselineRatePct,
Lift = finding.Lift,
WindowStartDate = finding.WindowStartDate,
WindowEndDate = finding.WindowEndDate,
Dimension = dimension,
FirstNotifiedAt = firstNotifiedAt,
LastEvaluatedAt = DateTime.UtcNow,
};
/// <summary>Refresh payload for a still-open episode below the fire gates: current window sums, everything identifying carried over.</summary>
private static RuleRegressionAlert BuildOngoingAlert(
RuleRegressionAlert existing, RuleRegressionRadar.WindowSums sums, DateTime targetDate)
=> new()
{
TenantId = existing.TenantId,
RuleId = existing.RuleId,
RuleTitle = existing.RuleTitle,
WindowFireCount = sums.WindowFire,
WindowSessionCount = sums.WindowSessions,
BaselineFireCount = sums.BaselineFire,
BaselineSessionCount = sums.BaselineSessions,
WindowRatePct = sums.WindowSessions > 0 ? Math.Round(sums.WindowFire * 100.0 / sums.WindowSessions, 1) : 0,
BaselineRatePct = sums.BaselineSessions > 0 ? Math.Round(sums.BaselineFire * 100.0 / sums.BaselineSessions, 1) : 0,
Lift = sums.BaselineFire > 0 && sums.BaselineSessions > 0 && sums.WindowSessions > 0
? Math.Round((sums.WindowFire / (double)sums.WindowSessions) / (sums.BaselineFire / (double)sums.BaselineSessions), 1)
: null,
WindowStartDate = targetDate.AddDays(-(RuleRegressionRadar.WindowDays - 1)).ToString("yyyy-MM-dd"),
WindowEndDate = targetDate.ToString("yyyy-MM-dd"),
Dimension = existing.Dimension,
FirstNotifiedAt = existing.FirstNotifiedAt,
LastEvaluatedAt = DateTime.UtcNow,
};
/// <summary>Correlation sentence for bell + ops event — wording contract: "correlated — not necessarily causal" (rule 6).</summary>
internal static string? DescribeDimension(RuleRegressionDimension? dimension)
=> dimension == null
? null
: $"{dimension.HitSharePct}% of affected sessions are on {dimension.Dimension} {dimension.Value} " +
$"vs {dimension.AllSharePct}% of all sessions (lift {dimension.Lift}x) — correlated, not necessarily causal";
/// <summary>Bell message with the full numbers (spec: the admin can verify without a portal round-trip).</summary>
internal static string BuildRegressionMessage(RuleRegressionFinding finding, string? dimensionSummary)
{
var baseline = finding.Lift.HasValue
? $"baseline {finding.BaselineRatePct}% ({finding.BaselineFireCount}/{finding.BaselineSessionCount} over the prior 28 days) — lift {finding.Lift.Value}x"
: $"baseline {finding.BaselineRatePct}% ({finding.BaselineFireCount}/{finding.BaselineSessionCount} over the prior 28 days) — new signal";
var dimension = dimensionSummary != null ? $" {dimensionSummary}." : " No clear dimension concentration.";
return $"Fired in {finding.WindowFireCount} of {finding.WindowSessionCount} evaluated sessions " +
$"({finding.WindowRatePct}%) in the last 7 days ({finding.WindowStartDate} to {finding.WindowEndDate}); " +
$"{baseline}.{dimension}";
}
}
}
@@ -60,6 +60,9 @@ namespace AutopilotMonitor.Functions.Services
private readonly IHardwareRejectionNotificationTracker _hardwareRejectionTracker;
private readonly DataAccess.TableStorage.BackupJobsRepository _backupJobsRepo;
private readonly OpsEventService _opsEventService;
private readonly IRuleRepository _ruleRepo;
private readonly AnalyzeRuleService _analyzeRuleService;
private readonly TenantNotificationService _tenantNotificationService;
private readonly Analyze.IAnalyzeOnEnrollmentEndProducer _analyzeProducer;
private readonly IAzureMonitorMetricsReader _metricsReader;
private readonly IPoisonQueueProbe _poisonQueueProbe;
@@ -89,6 +92,9 @@ namespace AutopilotMonitor.Functions.Services
IHardwareRejectionNotificationTracker hardwareRejectionTracker,
DataAccess.TableStorage.BackupJobsRepository backupJobsRepo,
OpsEventService opsEventService,
IRuleRepository ruleRepo,
AnalyzeRuleService analyzeRuleService,
TenantNotificationService tenantNotificationService,
Analyze.IAnalyzeOnEnrollmentEndProducer analyzeProducer,
IAzureMonitorMetricsReader metricsReader,
IPoisonQueueProbe poisonQueueProbe,
@@ -113,6 +119,9 @@ namespace AutopilotMonitor.Functions.Services
_hardwareRejectionTracker = hardwareRejectionTracker;
_backupJobsRepo = backupJobsRepo;
_opsEventService = opsEventService;
_ruleRepo = ruleRepo;
_analyzeRuleService = analyzeRuleService;
_tenantNotificationService = tenantNotificationService;
_analyzeProducer = analyzeProducer;
_metricsReader = metricsReader;
_poisonQueueProbe = poisonQueueProbe;
@@ -135,6 +144,17 @@ namespace AutopilotMonitor.Functions.Services
await MarkStalledSessionsAsTimedOutAsync();
await DetectExcessiveEventSessionsAsync();
await AggregateMetricsWithCatchUpAsync();
// F3 PR6: rule-frequency regression radar, right after the rule-stats
// aggregation so the window rows are fresh. Anchored on YESTERDAY — whole
// days only, a partial today would understate the window rate. Fail-soft.
await RunRuleRegressionRadarAsync(DateTime.UtcNow.Date.AddDays(-1));
// F1 PR2: rolling 30d breakdown backfill + daily attribution aggregates. Owns
// its own window (NOT the snapshot-gated catch-up above) so late-terminating
// sessions still reach their StartedAt-date's aggregate. Fail-soft internally.
await SweepTimeAttributionAsync();
// F2 PR4: device-history chain heal (incl. deleted-session ref cleanup) + daily
// FTR aggregates over the same rolling window. Fail-soft internally.
await SweepDeviceJourneysAsync();
// Plan §5 PR6 / §16 R14: session retention fanout extracted out of the 2h timer
// into the dedicated 12h SessionDeletionMaintenanceFunction so cascade-lifecycle
// work has independent cadence + kill-switch + OpsEvent watchdogs. The non-session
@@ -199,6 +219,16 @@ namespace AutopilotMonitor.Functions.Services
result.AggregatedDate = dateToAggregate.ToString("yyyy-MM-dd");
result.MetricsAggregated = true;
// Timer-path parity: radar re-runs are idempotent (tracker dedup), so a manual
// aggregation also re-evaluates regressions anchored on the aggregated date.
await RunRuleRegressionRadarAsync(dateToAggregate);
// Timer-path parity: manual maintenance also refreshes the attribution
// breakdowns + daily aggregates and the device-history/FTR rollups
// (rolling 30d windows, cheap once converged).
await SweepTimeAttributionAsync();
await SweepDeviceJourneysAsync();
if (!aggregateOnly)
{
// Plan §5 PR6: session retention is now SessionDeletionMaintenanceFunction's
@@ -73,6 +73,16 @@ namespace AutopilotMonitor.Functions.Services.Offboarding
Constants.TableNames.SessionInventoryContributions,
Constants.TableNames.SessionTombstones,
Constants.TableNames.PreviewWhitelist,
// F1 time attribution (PR2): both PK=TenantId. Breakdowns also die per-session via
// the deletion manifest; the partition wipe covers offboarding + any stragglers.
Constants.TableNames.SessionTimeBreakdowns,
Constants.TableNames.TimeAttributionAggregates,
// F2 device history / FTR (PR4): both PK=TenantId. Histories are NOT in the
// per-session deletion manifest (a row aggregates many sessions; the maintenance
// sweep prunes deleted-session refs) — the partition wipe here is their only
// whole-row deletion besides chain-emptying.
Constants.TableNames.DeviceHistories,
Constants.TableNames.DeviceJourneyAggregates,
};
// Plan §6.4 — composite-PK "{tenantId}_..." wipes (Variant A range).
@@ -486,6 +486,37 @@ namespace AutopilotMonitor.Functions.Services
tenantId, "System.TrialSweep", new { domainName, trialExpiredUtc });
}
/// <summary>
/// F3 regression radar (insights spec §F3): an analyze rule's 7-day hit rate rose ≥2×
/// over its 28-day baseline with disjoint Wilson intervals. Fired ONCE per episode (the
/// notification-tracker row is the dedup); the message carries the full numbers so the
/// event is verifiable without a portal round-trip. Dimension wording is correlation
/// only — never causal. Dual-registered in the web OPS_EVENT_TYPES catalog
/// (memory: feedback_ops_event_types_dual_register).
/// </summary>
public Task RecordRuleFrequencyRegressionAsync(
string tenantId, string ruleId, string ruleTitle,
int windowFireCount, int windowSessionCount, double windowRatePct,
int baselineFireCount, int baselineSessionCount, double baselineRatePct,
double? lift, string? dimensionSummary)
=> WriteAsync(OpsEventCategory.Tenant, "RuleFrequencyRegression", OpsEventSeverity.Warning,
$"Rule '{ruleTitle}' ({ruleId}) fired in {windowFireCount}/{windowSessionCount} sessions ({windowRatePct}%) over 7d " +
$"vs {baselineRatePct}% baseline ({baselineFireCount}/{baselineSessionCount} over 28d)" +
(lift.HasValue ? $" — lift {lift.Value}x" : " — new signal (no baseline fires)") +
(string.IsNullOrEmpty(dimensionSummary) ? string.Empty : $". {dimensionSummary}"),
tenantId, "System.Maintenance",
new
{
ruleId,
windowFireCount,
windowSessionCount,
windowRatePct,
baselineFireCount,
baselineSessionCount,
baselineRatePct,
lift,
});
// ── Agent ──────────────────────────────────────────────────────────────
public Task RecordSessionTimeoutsAsync(string tenantId, int sessionCount, int timeoutHours)
@@ -563,6 +594,24 @@ namespace AutopilotMonitor.Functions.Services
null, "System.SlaEvaluation",
new { tenantsEvaluated, breachesDetected, notificationsSent, durationMs });
// ── Platform (Azure Monitor) ───────────────────────────────────────────
/// <summary>
/// Records an Azure Monitor alert notification delivered via the ops alert webhook
/// (AzureMonitorAlertWebhookFunction). severity is already mapped onto the ops scale;
/// azureSeverity keeps the original Sev0–Sev4 value for the details payload.
/// </summary>
public Task RecordAzureMonitorAlertAsync(string alertRule, string severity, string monitorCondition,
string? description, string? azureSeverity, string? monitoringService, string? targetResource,
double? metricValue)
{
var suffix = string.IsNullOrWhiteSpace(description) ? string.Empty : $": {description}";
return WriteAsync(OpsEventCategory.Platform, "AzureMonitorAlert", severity,
$"Azure Monitor alert '{alertRule}' {monitorCondition}{suffix}",
null, "System.AzureMonitor",
new { alertRule, monitorCondition, azureSeverity, monitoringService, targetResource, metricValue });
}
// ── Core write method ──────────────────────────────────────────────────
private async Task WriteAsync(string category, string eventType, string severity,
@@ -213,6 +213,15 @@ namespace AutopilotMonitor.Functions.Services
var lastSnapshot = snapshots.Last();
// Prefer the exact session average from the cumulative counters (agents ≥ 2026-07-28
// emit net_total_latency_ms); fall back to the unweighted mean of the per-interval
// delta averages for older sessions.
var totalLatencyMs = GetDouble(lastSnapshot, "net_total_latency_ms");
var totalLatencyRequests = GetDouble(lastSnapshot, "net_total_requests");
var sessionAvgLatency = totalLatencyMs > 0 && totalLatencyRequests > 0
? totalLatencyMs / totalLatencyRequests
: (latValues.Count > 0 ? latValues.Average() : 0);
// Resolve agent version: prefer from snapshot, fallback to session
var agentVersion = snapshots
.Select(s => GetString(s, "agent_version"))
@@ -239,7 +248,7 @@ namespace AutopilotMonitor.Functions.Services
AvgWorkingSet = wsValues.Count > 0 ? wsValues.Average() : 0,
MaxWorkingSet = wsValues.Count > 0 ? wsValues.Max() : 0,
AvgPrivateBytes = pbValues.Count > 0 ? pbValues.Average() : 0,
AvgLatency = latValues.Count > 0 ? latValues.Average() : 0,
AvgLatency = sessionAvgLatency,
AvgSpoolDepth = spoolValues.Count > 0 ? spoolValues.Average() : 0,
MaxSpoolDepth = spoolValues.Count > 0 ? spoolValues.Max() : 0,
PeakSpoolDepth = peakValues.Count > 0 ? peakValues.Last() : 0,
@@ -193,6 +193,9 @@ namespace AutopilotMonitor.Functions.Services
var raw = await _metricsRepo.GetAppInstallSummariesByTenantAsync(config.TenantId, now.AddDays(-8));
appInstalls = raw
.Where(a => (a.Status == "Succeeded" || a.Status == "Failed")
// PR0 (2026-07-26): skips are not install attempts — keep the breach
// evaluation on the same population as the SLA dashboard rate.
&& !Helpers.MetricsMath.IsSkipTerminalState(a)
&& SlaMetricsService.GetIsoWeekKey(a.StartedAt) == currentWeekKey)
.ToList();
}
@@ -146,7 +146,10 @@ namespace AutopilotMonitor.Functions.Services
{
var weekApps = appInstalls
.Where(a => GetIsoWeekKey(a.StartedAt) == group.Key &&
(a.Status == "Succeeded" || a.Status == "Failed"))
(a.Status == "Succeeded" || a.Status == "Failed") &&
// PR0 (2026-07-26): skips are not install attempts — they must
// not pad the SLA app-install success rate.
!Helpers.MetricsMath.IsSkipTerminalState(a))
.ToList();
if (weekApps.Count >= 5)
{
@@ -175,7 +178,9 @@ namespace AutopilotMonitor.Functions.Services
{
var currentWeekApps = appInstalls
.Where(a => GetIsoWeekKey(a.StartedAt) == currentWeekKey &&
(a.Status == "Succeeded" || a.Status == "Failed"))
(a.Status == "Succeeded" || a.Status == "Failed") &&
// PR0 (2026-07-26): same convention as the weekly trend above.
!Helpers.MetricsMath.IsSkipTerminalState(a))
.ToList();
if (currentWeekApps.Count >= 5)
@@ -8,6 +8,7 @@ using Azure;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Services.Deletion;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
@@ -166,6 +167,33 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// All tombstone keys, INCLUDING expired-but-unpruned rows: a tombstone proves the
/// session was deleted regardless of the marker's 410-gate window, and that proof is
/// exactly what the F2 device-journey sweep needs to drop chain refs of deleted
/// sessions. Cross-partition scan over the small table (one row per recently-deleted
/// session, physically pruned after ~7d) with a key-only projection.
/// </summary>
public virtual async Task<List<(string TenantId, string SessionId)>> GetAllSessionTombstoneKeysAsync()
{
var keys = new List<(string TenantId, string SessionId)>();
try
{
var tableClient = _tableServiceClient.GetTableClient(Shared.Constants.TableNames.SessionTombstones);
await foreach (var entity in tableClient.QueryAsync<TableEntity>(
select: new[] { "PartitionKey", "RowKey" }))
{
keys.Add((entity.PartitionKey, entity.RowKey));
}
}
catch (Exception ex)
{
// Fail-soft: the sweep then simply prunes nothing this pass and retries next tick.
_logger.LogWarning(ex, "Failed to enumerate session tombstone keys");
}
return keys;
}
/// <summary>
/// Enumerates tombstone rows whose <c>ExpiresAt</c> column is at or before <paramref name="now"/>.
/// Maintenance pruning consumer (<c>SessionDeletionMaintenanceFunction</c>) iterates this
@@ -0,0 +1,336 @@
using System.Text.Json;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// Partial: F2 device-history / First-Time-Right persistence (insights spec §F2, PR4).
/// DeviceHistories: PK=TenantId, RK=encoded normalized serial — one row per device key,
/// holding the terminal-session chain (cap 20) + derived journey counts. Written inline at
/// the session-terminal seam and healed by the rolling maintenance sweep (which also drops
/// refs of deleted sessions and deletes emptied rows). Deliberately NOT part of the
/// per-session deletion manifest — the row aggregates many sessions; tenant offboarding
/// wipes the partition. DeviceJourneyAggregates: PK=TenantId ("global" for the cross-tenant
/// row), RK="{yyyy-MM-dd}" — daily FTR rollups, recomputed idempotently, 180d retention.
/// </summary>
public partial class TableStorageService
{
// ===== DEVICE HISTORIES =====
/// <summary>
/// Inline seam entry (F2 counterpart of <see cref="ComputeAndStoreSessionTimeBreakdownAsync"/>):
/// upserts the terminal session into its device's history chain so the session-detail
/// banner is fresh, not a day old. No-ops for non-terminal sessions, junk/placeholder
/// serials (excluded by design, disclosed in the daily aggregate) and sessions inside a
/// deletion cascade — unlike the F1 breakdown row, a chain ref does NOT die with its
/// session automatically; only the sweep's tombstone pass prunes it, so it must never be
/// added for a session already being deleted. Fail-soft + idempotent; the maintenance
/// sweep self-heals any miss.
/// </summary>
public async Task UpdateDeviceHistoryForSessionAsync(string tenantId, string sessionId)
{
try
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
SecurityValidator.EnsureValidGuid(sessionId, nameof(sessionId));
var session = await GetSessionAsync(tenantId, sessionId);
if (session == null) return;
if (!DeviceJourneyCalculator.IsTerminal(session.Status)) return;
if (!string.IsNullOrEmpty(session.DeletionState) && session.DeletionState != "None") return;
var serialKey = DeviceJourneyCalculator.NormalizeSerial(session.SerialNumber);
if (serialKey == null) return;
var reference = DeviceJourneyCalculator.BuildSessionRef(session);
if (reference == null) return;
var existing = await GetDeviceHistoryAsync(tenantId, serialKey);
var chain = DeviceJourneyCalculator.MergeChain(existing?.Chain, new[] { reference });
var history = BuildDeviceHistoryRow(tenantId, serialKey, chain, existing, session);
await UpsertDeviceHistoryAsync(history);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Device-history update failed for session {SessionId} (fail-soft)", sessionId);
}
}
/// <summary>
/// Assembles the row from a merged chain: derived journey counts via the calculator, and
/// display fields (serial casing, manufacturer, model) taken from the given session only
/// when it IS the chain's newest entry (or the row has none yet) — a sweep backfilling an
/// older session must not regress the display data to stale values. internal static so
/// the display-precedence rule is pinned by unit tests.
/// </summary>
internal static DeviceHistory BuildDeviceHistoryRow(
string tenantId, string serialKey, List<DeviceSessionRef> chain,
DeviceHistory? existing, SessionSummary session)
{
var (journeyCount, currentAttempts) = DeviceJourneyCalculator.Derive(chain);
var sessionIsNewest = chain.Count > 0 && chain[chain.Count - 1].SessionId == session.SessionId;
return new DeviceHistory
{
TenantId = tenantId,
SerialKey = serialKey,
SerialNumber = sessionIsNewest || string.IsNullOrEmpty(existing?.SerialNumber)
? (session.SerialNumber ?? string.Empty).Trim()
: existing!.SerialNumber,
Manufacturer = sessionIsNewest || string.IsNullOrEmpty(existing?.Manufacturer)
? (session.Manufacturer ?? string.Empty).Trim()
: existing!.Manufacturer,
Model = sessionIsNewest || string.IsNullOrEmpty(existing?.Model)
? (session.Model ?? string.Empty).Trim()
: existing!.Model,
Chain = chain,
CurrentJourneyAttempts = currentAttempts,
JourneyCount = journeyCount,
JourneyVersion = DeviceJourneyCalculator.CurrentVersion,
LastUpdated = DateTime.UtcNow,
};
}
/// <summary>Point-reads one device's history row; null when the device has none. Takes the NORMALIZED serial (RK encoding is internal).</summary>
public async Task<DeviceHistory?> GetDeviceHistoryAsync(string tenantId, string serialKey)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
if (string.IsNullOrEmpty(serialKey)) return null;
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.DeviceHistories);
var result = await tableClient.GetEntityIfExistsAsync<TableEntity>(tenantId, SerialRowKey(serialKey));
return result.HasValue ? MapToDeviceHistory(result.Value!) : null;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to get device history for tenant {TenantId}", tenantId);
return null;
}
}
/// <summary>All device-history rows of one tenant (the sweep's tombstone-cleanup scan; bounded by the tenant's device count).</summary>
public async Task<List<DeviceHistory>> GetDeviceHistoriesByTenantAsync(string tenantId)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.DeviceHistories);
var results = new List<DeviceHistory>();
await foreach (var entity in tableClient.QueryAsync<TableEntity>(filter: $"PartitionKey eq '{tenantId}'"))
{
results.Add(MapToDeviceHistory(entity));
}
return results;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to query device histories for tenant {TenantId}", tenantId);
return new List<DeviceHistory>();
}
}
/// <summary>Upserts the history row (Replace — chain maintenance is read-modify-write, the caller owns the whole row).</summary>
public async Task<bool> UpsertDeviceHistoryAsync(DeviceHistory history)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.DeviceHistories);
await tableClient.UpsertEntityAsync(BuildDeviceHistoryEntity(history), TableUpdateMode.Replace);
return true;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to upsert device history for tenant {TenantId}", history.TenantId);
return false;
}
}
/// <summary>Deletes one device's history row (the sweep calls this when every chain ref belonged to deleted sessions).</summary>
public async Task DeleteDeviceHistoryAsync(string tenantId, string serialKey)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
if (string.IsNullOrEmpty(serialKey)) return;
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.DeviceHistories);
await tableClient.DeleteEntityAsync(tenantId, SerialRowKey(serialKey));
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to delete device history for tenant {TenantId}", tenantId);
}
}
/// <summary>
/// RowKey encoding for the normalized serial: percent-encoding keeps ordinary serials
/// human-readable while making the characters Table Storage forbids in keys
/// ('/', '\', '#', '?', controls) safe. Deterministic and reversible — the mapper
/// restores the exact normalized serial.
/// </summary>
internal static string SerialRowKey(string serialKey) => Uri.EscapeDataString(serialKey);
// internal static: entity builder + mapper are pinned by round-trip unit tests
// (table-storage-serialization rule — every property must survive Store→Map).
internal static TableEntity BuildDeviceHistoryEntity(DeviceHistory history)
{
return new TableEntity(history.TenantId, SerialRowKey(history.SerialKey))
{
["SerialNumber"] = history.SerialNumber,
["Manufacturer"] = history.Manufacturer,
["Model"] = history.Model,
["ChainJson"] = JsonSerializer.Serialize(history.Chain),
["CurrentJourneyAttempts"] = history.CurrentJourneyAttempts,
["JourneyCount"] = history.JourneyCount,
["JourneyVersion"] = history.JourneyVersion,
["LastUpdated"] = EnsureUtc(history.LastUpdated),
};
}
internal static DeviceHistory MapToDeviceHistory(TableEntity entity)
{
return new DeviceHistory
{
TenantId = entity.PartitionKey,
SerialKey = Uri.UnescapeDataString(entity.RowKey),
SerialNumber = entity.GetString("SerialNumber") ?? string.Empty,
Manufacturer = entity.GetString("Manufacturer") ?? string.Empty,
Model = entity.GetString("Model") ?? string.Empty,
Chain = DeserializeJsonColumn<DeviceSessionRef>(entity.GetString("ChainJson")),
CurrentJourneyAttempts = entity.GetInt32("CurrentJourneyAttempts") ?? 0,
JourneyCount = entity.GetInt32("JourneyCount") ?? 0,
JourneyVersion = entity.GetInt32("JourneyVersion") ?? 0,
LastUpdated = entity.GetDateTimeOffset("LastUpdated")?.UtcDateTime ?? DateTime.MinValue,
};
}
// ===== DEVICE JOURNEY (FTR) DAILY AGGREGATES =====
/// <summary>Upserts one daily FTR aggregate row (Replace — the sweep recomputes rows whole).</summary>
public async Task<bool> SaveDeviceJourneyAggregateAsync(DeviceJourneyDailyAggregate aggregate)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.DeviceJourneyAggregates);
await tableClient.UpsertEntityAsync(BuildDeviceJourneyAggregateEntity(aggregate), TableUpdateMode.Replace);
return true;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to save device-journey aggregate {Date} for tenant {TenantId}",
aggregate.Date, aggregate.TenantId);
return false;
}
}
/// <summary>
/// Reads the FTR aggregate rows of one tenant partition ("global" allowed) for an
/// inclusive date range. RowKeys are exactly "{yyyy-MM-dd}", so a string range filter
/// covers the window; dates are server-formatted (injection-safe). Counts are additive —
/// a window rate is the sum over these rows.
/// </summary>
public async Task<List<DeviceJourneyDailyAggregate>> GetDeviceJourneyAggregatesAsync(
string tenantId, DateTime startDate, DateTime endDate)
{
if (tenantId != "global")
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.DeviceJourneyAggregates);
var filter = $"PartitionKey eq '{tenantId}' and RowKey ge '{startDate:yyyy-MM-dd}' and RowKey le '{endDate:yyyy-MM-dd}'";
var results = new List<DeviceJourneyDailyAggregate>();
await foreach (var entity in tableClient.QueryAsync<TableEntity>(filter: filter))
{
results.Add(MapToDeviceJourneyAggregate(entity));
}
return results;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to query device-journey aggregates for tenant {TenantId}", tenantId);
return new List<DeviceJourneyDailyAggregate>();
}
}
// internal static: pinned by round-trip tests together with MapToDeviceJourneyAggregate.
internal static TableEntity BuildDeviceJourneyAggregateEntity(DeviceJourneyDailyAggregate aggregate)
{
return new TableEntity(aggregate.TenantId, aggregate.Date)
{
["Date"] = aggregate.Date,
["JourneyVersion"] = aggregate.JourneyVersion,
["CompletedJourneyCount"] = aggregate.CompletedJourneyCount,
["FirstTimeRightCount"] = aggregate.FirstTimeRightCount,
["AttemptHistogramJson"] = JsonSerializer.Serialize(aggregate.AttemptHistogram),
["ExcludedSessionCount"] = aggregate.ExcludedSessionCount,
["ComputedAt"] = EnsureUtc(aggregate.ComputedAt),
};
}
internal static DeviceJourneyDailyAggregate MapToDeviceJourneyAggregate(TableEntity entity)
{
return new DeviceJourneyDailyAggregate
{
TenantId = entity.PartitionKey,
Date = entity.GetString("Date") ?? entity.RowKey,
JourneyVersion = entity.GetInt32("JourneyVersion") ?? 0,
CompletedJourneyCount = entity.GetInt32("CompletedJourneyCount") ?? 0,
FirstTimeRightCount = entity.GetInt32("FirstTimeRightCount") ?? 0,
AttemptHistogram = DeserializeJsonColumn<DeviceJourneyAttemptBucket>(entity.GetString("AttemptHistogramJson")),
ExcludedSessionCount = entity.GetInt32("ExcludedSessionCount") ?? 0,
ComputedAt = entity.GetDateTimeOffset("ComputedAt")?.UtcDateTime ?? DateTime.MinValue,
};
}
/// <summary>
/// Deletes one daily FTR row. Used by the sweep's stale-bucket reconcile: a date whose
/// last completing journey was deleted must not keep serving old counts. Missing rows
/// are a no-op.
/// </summary>
public async Task DeleteDeviceJourneyAggregateAsync(string tenantId, string dateKey)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.DeviceJourneyAggregates);
await tableClient.DeleteEntityAsync(tenantId, dateKey);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to delete stale device-journey aggregate {Date} for tenant {TenantId}",
dateKey, tenantId);
}
}
/// <summary>
/// Retention: deletes FTR aggregate rows older than the cutoff (RowKey IS the date, so a
/// string compare works across partitions). Mirrors the UsageMetrics 180d policy.
/// DeviceHistories needs no age sweep — refs are pruned tombstone-driven and the rows
/// die with tenant offboarding.
/// </summary>
public async Task<int> DeleteDeviceJourneyAggregatesOlderThanAsync(DateTime cutoffDate)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.DeviceJourneyAggregates);
var filter = $"RowKey lt '{cutoffDate:yyyy-MM-dd}'";
var deleted = 0;
await foreach (var entity in tableClient.QueryAsync<TableEntity>(
filter: filter, select: new[] { "PartitionKey", "RowKey" }))
{
await tableClient.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
deleted++;
}
return deleted;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to delete old device-journey aggregates");
return 0;
}
}
}
}
@@ -586,7 +586,8 @@ namespace AutopilotMonitor.Functions.Services
internal static readonly string[] GeoMetricsSessionProjection =
{
"PartitionKey", "RowKey", "StartedAt", "CompletedAt", "Status", "DurationSeconds",
"IsPreProvisioned", "ResumedAt", "GeoCountry", "GeoRegion", "GeoCity", "GeoLoc"
"IsPreProvisioned", "ResumedAt", "GeoCountry", "GeoRegion", "GeoCity", "GeoLoc",
"AvgApiLatencyMs", "ApiRequestCount"
};
/// <summary>
@@ -1,5 +1,6 @@
using Azure;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
@@ -216,7 +217,46 @@ namespace AutopilotMonitor.Functions.Services
var existingResult = await tableClient.GetEntityIfExistsAsync<TableEntity>(summary.TenantId, rowKey);
if (existingResult.HasValue)
{
var existing = existingResult.Value!;
ReconcileAppInstallSummaryWithExisting(summary, existingResult.Value!);
}
var entity = BuildAppInstallSummaryEntity(summary, rowKey);
// Merge-mode (default) preserves columns absent from the entity. Combined with the
// dynamic property-add inside BuildAppInstallSummaryEntity this means a batch that
// observed only progress / telemetry events for an app cannot clobber a prior
// terminal Status / CompletedAt / DurationSeconds / FailureCode / FailureMessage.
await tableClient.UpsertEntityAsync(entity, TableUpdateMode.Merge);
return true;
}
catch (Exception ex)
{
_logger.LogError(ex, $"Failed to store app install summary for {summary.AppName}");
return false;
}
}
/// <summary>
/// Reconciles an incoming per-batch <see cref="AppInstallSummary"/> against the stored row
/// before the Merge-mode upsert. Extracted as an internal static seam so the cross-batch
/// contract (earliest StartedAt wins, terminal status is sticky, out-of-order duration
/// recompute, prior-batch metadata preservation) is pinned by unit tests.
/// </summary>
internal static void ReconcileAppInstallSummaryWithExisting(AppInstallSummary summary, TableEntity existing)
{
// Terminal status is sticky (PR0 hardening, same out-of-order family as Q4):
// a late or replayed batch that only observed pre-terminal events carries
// Status "InProgress" — a real (non-sentinel) value that Merge-mode would
// happily write over a row the terminal batch already closed. Only another
// terminal observation may change a terminal status (Failed→Succeeded on a
// successful retry stays legal).
var existingStatus = existing.GetString("Status");
if ((existingStatus == "Succeeded" || existingStatus == "Failed") &&
summary.Status != "Succeeded" && summary.Status != "Failed")
{
summary.Status = existingStatus;
}
var existingStartedAt = existing.GetDateTimeOffset("StartedAt")?.UtcDateTime;
if (existingStartedAt.HasValue && existingStartedAt.Value != DateTime.MinValue)
{
@@ -231,6 +271,21 @@ namespace AutopilotMonitor.Functions.Services
}
}
// Q4 (source-data audit 2026-07-26): out-of-order arrival — the terminal batch
// landed FIRST (row already carries CompletedAt), the started batch arrives now
// with the earlier StartedAt. Neither batch alone could compute the duration;
// recompute it here against the stored CompletedAt so the row doesn't keep a
// permanently unset duration despite both endpoints being known.
if (!summary.CompletedAt.HasValue && summary.StartedAt != DateTime.MinValue)
{
var existingCompletedAt = existing.GetDateTimeOffset("CompletedAt")?.UtcDateTime;
if (existingCompletedAt.HasValue && existingCompletedAt.Value >= summary.StartedAt)
{
summary.CompletedAt = existingCompletedAt.Value;
summary.DurationSeconds = (int)(existingCompletedAt.Value - summary.StartedAt).TotalSeconds;
}
}
// Preserve DownloadDurationSeconds and DownloadBytes from prior batch if current batch has no value
if (summary.DownloadDurationSeconds == 0)
{
@@ -311,24 +366,115 @@ namespace AutopilotMonitor.Functions.Services
summary.DetectionResult = existingDetection;
}
// F1 PR1 — AppId identity across batches (audit Q3). First-seen appId wins
// (mirrors the in-batch fold rule); a later batch carrying a DIFFERENT appId
// under the same name proves the name-keyed row merges two apps → collision.
var existingAppId = existing.GetString("AppId");
if (!string.IsNullOrEmpty(existingAppId))
{
if (string.IsNullOrEmpty(summary.AppId))
{
summary.AppId = existingAppId!;
}
else if (!string.Equals(summary.AppId, existingAppId, StringComparison.OrdinalIgnoreCase))
{
summary.AppIdCollision = true;
summary.AppId = existingAppId!;
}
}
// Collision is one-way sticky: once any batch proved the identity mix, no
// later single-identity batch can unprove it.
if (!summary.AppIdCollision && (existing.GetBoolean("AppIdCollision") ?? false))
summary.AppIdCollision = true;
// EspBlocking is written only by the session-terminal resolution (positive
// evidence). A late app-event batch carries null → adopt the stored verdict
// so the in-memory summary matches what Merge-mode preserves on disk.
if (!summary.EspBlocking.HasValue)
{
var existingEspBlocking = existing.GetBoolean("EspBlocking");
if (existingEspBlocking.HasValue)
summary.EspBlocking = existingEspBlocking;
}
}
/// <summary>
/// Resolves <see cref="AppInstallSummary.EspBlocking"/> for every app row of a session by
/// joining the rows' <c>AppId</c> against the session's LATEST <c>esp_config_detected</c>
/// tracking lists (F1 PR1; source-data audit Q2). Positive evidence only: listed rows are
/// merge-stamped <c>EspBlocking=true</c>; absent rows keep null (unknown) — never false.
/// Runs once per session at the terminal transition (TableSessionRepository seam — the
/// same funnel as the counter reconcile, covering ingest, admin marks, maintenance sweep
/// and rule-engine fails alike). Idempotent and fail-soft: any error is logged and
/// swallowed; app rows created by late batches after this ran simply stay unknown.
/// </summary>
/// <returns>Number of rows stamped (0 on no lists / nothing to stamp / error).</returns>
public async Task<int> ResolveEspBlockingForSessionAsync(string tenantId, string sessionId)
{
try
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
SecurityValidator.EnsureValidGuid(sessionId, nameof(sessionId));
// Latest emission that actually carries lists wins: the registry lists grow
// progressively, so later reads are supersets — but a later emission whose probe
// found no Diagnostics key (no espTracked* keys at all) must not erase earlier
// positive evidence.
var espEvents = await GetSessionEventsByTypeAsync(
tenantId, sessionId, Constants.EventTypes.EspConfigDetected, maxResults: 50);
EspBlockingSets? sets = null;
for (var i = espEvents.Count - 1; i >= 0 && sets == null; i--)
sets = EspBlockingSets.FromEventData(espEvents[i].Data);
if (sets == null || sets.ListedCount == 0)
return 0;
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.AppInstallSummaries);
var filter = $"PartitionKey eq '{tenantId}' and RowKey ge '{sessionId}_' and RowKey lt '{sessionId}`'";
var query = tableClient.QueryAsync<TableEntity>(
filter: filter,
select: new[] { "PartitionKey", "RowKey", "AppId", "EspBlocking", "AppIdCollision" });
var stamped = 0;
await foreach (var row in query)
{
if (!ShouldStampEspBlocking(row, sets)) continue;
var update = new TableEntity(row.PartitionKey, row.RowKey)
{
["EspBlocking"] = true
};
await tableClient.UpsertEntityAsync(update, TableUpdateMode.Merge);
stamped++;
}
var entity = BuildAppInstallSummaryEntity(summary, rowKey);
// Merge-mode (default) preserves columns absent from the entity. Combined with the
// dynamic property-add inside BuildAppInstallSummaryEntity this means a batch that
// observed only progress / telemetry events for an app cannot clobber a prior
// terminal Status / CompletedAt / DurationSeconds / FailureCode / FailureMessage.
await tableClient.UpsertEntityAsync(entity, TableUpdateMode.Merge);
return true;
if (stamped > 0)
_logger.LogInformation(
"Session {SessionId}: stamped EspBlocking=true on {Count} app rows ({Listed} ids in blocking set)",
sessionId, stamped, sets.ListedCount);
return stamped;
}
catch (Exception ex)
{
_logger.LogError(ex, $"Failed to store app install summary for {summary.AppName}");
return false;
_logger.LogWarning(ex, "EspBlocking resolution failed for session {SessionId} (fail-soft)", sessionId);
return 0;
}
}
/// <summary>
/// Pure per-row stamping predicate behind <see cref="ResolveEspBlockingForSessionAsync"/> —
/// internal static so the join contract is pinned by unit tests: rows without an AppId and
/// collision rows (identity ambiguous) stay unknown; only positive membership stamps; an
/// already-stamped row is skipped (idempotency).
/// </summary>
internal static bool ShouldStampEspBlocking(TableEntity row, EspBlockingSets sets)
{
var appId = row.GetString("AppId");
if (string.IsNullOrEmpty(appId)) return false;
if (row.GetBoolean("AppIdCollision") ?? false) return false;
if (row.GetBoolean("EspBlocking") == true) return false;
return sets.Contains(appId);
}
/// <summary>
/// Gets app install summaries for a tenant (fleet-level metrics). When <paramref name="sinceUtc"/>
/// is supplied, a server-side <c>StartedAt ge</c> filter is applied so a windowed view (e.g. the
@@ -392,16 +538,18 @@ namespace AutopilotMonitor.Functions.Services
/// <summary>
/// Columns MetricsMath.BuildAppMetricsPayload consumes (slowest/failing ranking + DO rollup):
/// grouping key, status/duration/bytes, failure code and every counter DoAggregator sums.
/// grouping key, status + terminal state (PR0 skip classification — dropping TerminalState
/// silently zeroed totalSkipped in production, found 2026-07-27), duration/bytes, failure
/// code and every counter DoAggregator sums.
/// internal so AppsProjectionEquivalenceTests derives its keep-set from this array.
/// </summary>
internal static readonly string[] AppMetricsProjection =
{
"PartitionKey", "RowKey", "AppName", "Status", "StartedAt",
"PartitionKey", "RowKey", "AppName", "Status", "TerminalState", "StartedAt",
"DurationSeconds", "DownloadBytes", "FailureCode",
"DoDownloadMode", "DoTotalBytesDownloaded", "DoBytesFromPeers", "DoBytesFromHttp",
"DoBytesFromLanPeers", "DoBytesFromGroupPeers", "DoBytesFromInternetPeers",
"DoBytesFromLinkLocalPeers", "DoBytesFromCacheServer"
"DoBytesFromLinkLocalPeers", "DoBytesFromCacheServer", "AppIdCollision"
};
/// <summary>
@@ -421,8 +569,9 @@ namespace AutopilotMonitor.Functions.Services
internal static readonly string[] AppsDashboardProjection =
{
"PartitionKey", "RowKey", "TenantId", "SessionId", "AppName", "AppType", "AppVersion",
"Status", "StartedAt", "CompletedAt", "DurationSeconds", "DownloadBytes",
"AttemptNumber", "InstallerPhase", "FailureCode", "FailureMessage", "ExitCode", "DetectionResult"
"Status", "TerminalState", "StartedAt", "CompletedAt", "DurationSeconds", "DownloadBytes",
"AttemptNumber", "InstallerPhase", "FailureCode", "FailureMessage", "ExitCode", "DetectionResult",
"AppId", "EspBlocking", "AppIdCollision"
};
/// <summary>
@@ -514,6 +663,7 @@ namespace AutopilotMonitor.Functions.Services
SessionId = entity.GetString("SessionId") ?? string.Empty,
TenantId = entity.GetString("TenantId") ?? entity.PartitionKey,
Status = entity.GetString("Status") ?? "InProgress",
TerminalState = entity.GetString("TerminalState") ?? string.Empty,
DurationSeconds = entity.GetInt32("DurationSeconds") ?? 0,
DownloadBytes = entity.GetInt64("DownloadBytes") ?? 0,
DownloadDurationSeconds = entity.GetInt32("DownloadDurationSeconds") ?? 0,
@@ -541,7 +691,11 @@ namespace AutopilotMonitor.Functions.Services
AttemptNumber = entity.GetInt32("AttemptNumber") ?? 0,
InstallerPhase = entity.GetString("InstallerPhase") ?? string.Empty,
ExitCode = entity.GetInt32("ExitCode"),
DetectionResult = entity.GetString("DetectionResult") ?? string.Empty
DetectionResult = entity.GetString("DetectionResult") ?? string.Empty,
// F1 PR1 identity/blocking columns (absent on legacy rows → sentinels)
AppId = entity.GetString("AppId") ?? string.Empty,
EspBlocking = entity.GetBoolean("EspBlocking"),
AppIdCollision = entity.GetBoolean("AppIdCollision") ?? false
};
}
@@ -1381,6 +1535,21 @@ namespace AutopilotMonitor.Functions.Services
// Sentinel-gated lifecycle columns: only write when the current batch observed them.
if (!string.IsNullOrEmpty(summary.Status))
entity["Status"] = summary.Status;
if (!string.IsNullOrEmpty(summary.TerminalState))
entity["TerminalState"] = summary.TerminalState;
// F1 PR1 identity/blocking columns — all sentinel-gated so a batch that observed
// nothing about them cannot clobber a prior value under Merge-mode:
// AppId empty = not observed in this batch
// EspBlocking null = unknown (only the session-terminal resolution ever sets it,
// and only to true — positive evidence, never false)
// AppIdCollision false = never written; the flag is one-way sticky, so only true
// is persisted (Merge cannot un-set an absent column).
if (!string.IsNullOrEmpty(summary.AppId))
entity["AppId"] = summary.AppId;
if (summary.EspBlocking.HasValue)
entity["EspBlocking"] = summary.EspBlocking.Value;
if (summary.AppIdCollision)
entity["AppIdCollision"] = true;
if (summary.DurationSeconds > 0)
entity["DurationSeconds"] = summary.DurationSeconds;
if (summary.CompletedAt.HasValue)
@@ -128,6 +128,45 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// F3 radar hit set: sessionIds where <paramref name="ruleId"/> produced a result at/after
/// <paramref name="sinceUtc"/>. RuleResults PK = "{tenantId}_{sessionId}", RK = ruleId, so a
/// tenant partition-range scan with server-side RowKey + DetectedAt filters returns exactly
/// the hit rows; only SessionId is transferred. Runs on-fire only (rare) and is fail-soft —
/// an empty result just means the alert carries no dimension claim.
/// </summary>
public async Task<List<string>> GetRuleHitSessionIdsAsync(
string tenantId, string ruleId, DateTime sinceUtc, int maxResults = 2000)
{
var sessionIds = new List<string>();
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(ruleId))
return sessionIds;
try
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.RuleResults);
var safeTenantId = ODataSanitizer.EscapeValue(tenantId);
var safeRuleId = ODataSanitizer.EscapeValue(ruleId);
var filter = $"PartitionKey ge '{safeTenantId}_' and PartitionKey lt '{safeTenantId}_~' " +
$"and RowKey eq '{safeRuleId}' and DetectedAt ge datetime'{sinceUtc:yyyy-MM-ddTHH:mm:ss}Z'";
await foreach (var entity in tableClient.QueryAsync<TableEntity>(
filter: filter, select: new[] { "SessionId" }))
{
var sessionId = entity.GetString("SessionId");
if (!string.IsNullOrEmpty(sessionId))
sessionIds.Add(sessionId!);
if (sessionIds.Count >= maxResults) break;
}
return sessionIds;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to get rule hit sessions for rule {RuleId} in tenant {TenantId}", ruleId, tenantId);
return sessionIds;
}
}
// ===== GATHER RULES METHODS =====
/// <summary>
@@ -215,6 +215,16 @@ namespace AutopilotMonitor.Functions.Services
if (stalledAt.HasValue)
indexEntity["StalledAt"] = EnsureUtc(stalledAt.Value);
// Agent→backend HTTP latency projection (UpdateSessionNetworkLatencyAsync merges
// these) — must be in this superset or a StartedAt-shift full upsert drops them.
var avgApiLatencyMs = sessionEntity.GetDouble("AvgApiLatencyMs");
if (avgApiLatencyMs.HasValue)
indexEntity["AvgApiLatencyMs"] = avgApiLatencyMs.Value;
var apiRequestCount = sessionEntity.GetInt32("ApiRequestCount");
if (apiRequestCount.HasValue)
indexEntity["ApiRequestCount"] = apiRequestCount.Value;
return indexEntity;
}
@@ -2655,6 +2665,55 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// Stores the session-wide average agent→backend HTTP latency (Merge-mode, two fields).
/// Derived from cumulative counters, so a plain last-write-wins overwrite is idempotent
/// against batch replays and monotonically converges to the whole-session average.
/// Same UpdateEntity-not-Upsert rationale as <see cref="UpdateSessionImeAgentVersionAsync"/>:
/// a tombstoned Sessions row must stay tombstoned (404 → no-op).
/// Non-fatal: failures are logged as warnings and do not block ingest.
/// </summary>
public async Task UpdateSessionNetworkLatencyAsync(string tenantId, string sessionId,
double avgLatencyMs, int requestCount)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.Sessions);
var update = new TableEntity(tenantId, sessionId)
{
["AvgApiLatencyMs"] = avgLatencyMs,
["ApiRequestCount"] = requestCount
};
await tableClient.UpdateEntityAsync(update, ETag.All, TableUpdateMode.Merge);
// Mirror to SessionsIndex — the index is a full denormalized mirror and serves the
// session list / GA cross-tenant views where per-geo latency aggregation happens.
var idxRef = await tableClient.GetEntityAsync<TableEntity>(
tenantId, sessionId, select: new[] { "IndexRowKey" });
var indexRowKey = idxRef.Value.GetString("IndexRowKey");
if (!string.IsNullOrEmpty(indexRowKey))
{
await MergeSessionIndexAsync(tenantId, indexRowKey,
new TableEntity(tenantId, indexRowKey)
{
["AvgApiLatencyMs"] = avgLatencyMs,
["ApiRequestCount"] = requestCount
});
}
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
_logger.LogDebug(
"UpdateSessionNetworkLatency no-op: Sessions row {Tenant}/{Session} is absent (tombstoned or never registered)",
tenantId, sessionId);
}
catch (Exception ex)
{
// Non-fatal — don't block ingest
_logger.LogWarning(ex, "Failed to update network latency for session {SessionId}", sessionId);
}
}
// ===== SESSION/EVENT MAPPING HELPERS =====
/// <summary>
@@ -2748,6 +2807,8 @@ namespace AutopilotMonitor.Functions.Services
PendingActionsQueuedAt = SafeGetDateTime(entity, "PendingActionsQueuedAt"),
EventCount = SafeGetInt32(entity, "EventCount") ?? 0,
DurationSeconds = ComputeEffectiveDuration(entity, status, startedAt, completedAt),
AvgApiLatencyMs = SafeGetDouble(entity, "AvgApiLatencyMs"),
ApiRequestCount = SafeGetInt32(entity, "ApiRequestCount"),
EnrollmentType = entity.GetString("EnrollmentType") ?? "v1",
DiagnosticsBlobName = entity.GetString("DiagnosticsBlobName"),
// Legacy rows that predate this field surface as null → download path treats
@@ -0,0 +1,325 @@
using System.Text.Json;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// Partial: F1 time-attribution persistence (insights spec §F1, PR2).
/// SessionTimeBreakdowns: PK=TenantId, RK=SessionId — one row per terminal session,
/// written once at the terminal transition and self-healed by the 30-day maintenance
/// sweep; deleted with its session (deletion manifest step + offboarding partition wipe).
/// TimeAttributionAggregates: PK=TenantId ("global" for the cross-tenant row),
/// RK="{yyyy-MM-dd}|{enrollmentClass}" — daily fleet rollups, recomputed idempotently.
/// </summary>
public partial class TableStorageService
{
// ===== SESSION TIME BREAKDOWNS =====
/// <summary>
/// Loads the session facts + events and computes/stores the breakdown via
/// <see cref="TimeAttributionCalculator"/>. Returns the stored breakdown, or null when
/// the session is not computable (non-terminal, Incomplete — which deliberately has no
/// DurationSeconds — or missing) or on any storage failure. Fail-soft by design: this
/// runs on the terminal-transition path and the maintenance sweep, and must never fail
/// either. Idempotent — recomputing from the same events yields the same row (the
/// calculator is deterministic and Replace-mode overwrites).
/// </summary>
public async Task<SessionTimeBreakdown?> ComputeAndStoreSessionTimeBreakdownAsync(string tenantId, string sessionId)
{
try
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
SecurityValidator.EnsureValidGuid(sessionId, nameof(sessionId));
var session = await GetSessionAsync(tenantId, sessionId);
if (session == null) return null;
if (session.Status != SessionStatus.Succeeded && session.Status != SessionStatus.Failed) return null;
if (!session.DurationSeconds.HasValue || session.DurationSeconds.Value <= 0) return null;
var events = await GetSessionEventsAsync(tenantId, sessionId);
if (events.Count == 0) return null; // events aged out (retention) — no evidence, no row
var breakdown = TimeAttributionCalculator.Compute(new TimeAttributionInput
{
TenantId = tenantId,
SessionId = sessionId,
Status = session.Status.ToString(),
StartedAt = session.StartedAt,
CompletedAt = session.CompletedAt,
DurationSeconds = session.DurationSeconds,
IsPreProvisioned = session.IsPreProvisioned,
ResumedAt = session.ResumedAt,
Events = events,
});
if (breakdown == null) return null;
// Change signal for the sweep: a session whose EventCount moved after this
// write (late/replayed batches) gets recomputed from the fuller stream.
breakdown.EventCountAtCompute = session.EventCount;
return await StoreSessionTimeBreakdownAsync(breakdown) ? breakdown : null;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Time-attribution compute failed for session {SessionId} (fail-soft)", sessionId);
return null;
}
}
/// <summary>Upserts the breakdown row (Replace — a recompute owns the whole row).</summary>
public async Task<bool> StoreSessionTimeBreakdownAsync(SessionTimeBreakdown breakdown)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.SessionTimeBreakdowns);
await tableClient.UpsertEntityAsync(
BuildSessionTimeBreakdownEntity(breakdown), TableUpdateMode.Replace);
return true;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to store time breakdown for session {SessionId}", breakdown.SessionId);
return false;
}
}
/// <summary>Point-reads one breakdown row; null when the session has none.</summary>
public async Task<SessionTimeBreakdown?> GetSessionTimeBreakdownAsync(string tenantId, string sessionId)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
SecurityValidator.EnsureValidGuid(sessionId, nameof(sessionId));
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.SessionTimeBreakdowns);
var result = await tableClient.GetEntityIfExistsAsync<TableEntity>(tenantId, sessionId);
return result.HasValue ? MapToSessionTimeBreakdown(result.Value!) : null;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to get time breakdown for session {SessionId}", sessionId);
return null;
}
}
// internal static: entity builder + mapper are pinned by round-trip unit tests
// (table-storage-serialization rule — every property must survive Store→Map).
internal static TableEntity BuildSessionTimeBreakdownEntity(SessionTimeBreakdown b)
{
var entity = new TableEntity(b.TenantId, b.SessionId)
{
["AttributionVersion"] = b.AttributionVersion,
["EventCountAtCompute"] = b.EventCountAtCompute,
["WallClockSeconds"] = b.WallClockSeconds,
["UnattributedSeconds"] = b.UnattributedSeconds,
["RebootSeconds"] = b.RebootSeconds,
["BlockingAppCount"] = b.BlockingAppCount,
["QualityFlags"] = (int)b.QualityFlags,
["SegmentsJson"] = JsonSerializer.Serialize(b.Segments),
["RebootSpansJson"] = JsonSerializer.Serialize(b.RebootSpans),
["BlockingAppsJson"] = JsonSerializer.Serialize(b.BlockingApps),
["ComputedAt"] = DateTime.UtcNow,
};
// Tri-state: absent column = occupancy unknown (blocking set never observed) — the
// mapper must yield null, never 0 (truthfulness rule 1).
if (b.EspAppsOccupancySeconds.HasValue)
entity["EspAppsOccupancySeconds"] = b.EspAppsOccupancySeconds.Value;
return entity;
}
internal static SessionTimeBreakdown MapToSessionTimeBreakdown(TableEntity entity)
{
return new SessionTimeBreakdown
{
TenantId = entity.PartitionKey,
SessionId = entity.RowKey,
AttributionVersion = entity.GetInt32("AttributionVersion") ?? 0,
EventCountAtCompute = entity.GetInt32("EventCountAtCompute") ?? 0,
WallClockSeconds = entity.GetInt32("WallClockSeconds") ?? 0,
UnattributedSeconds = entity.GetInt32("UnattributedSeconds") ?? 0,
RebootSeconds = entity.GetInt32("RebootSeconds") ?? 0,
BlockingAppCount = entity.GetInt32("BlockingAppCount") ?? 0,
EspAppsOccupancySeconds = entity.GetInt32("EspAppsOccupancySeconds"),
QualityFlags = (TimeAttributionFlags)(entity.GetInt32("QualityFlags") ?? 0),
Segments = DeserializeJsonColumn<TimeAttributionSpan>(entity.GetString("SegmentsJson")),
RebootSpans = DeserializeJsonColumn<RebootSpan>(entity.GetString("RebootSpansJson")),
BlockingApps = DeserializeJsonColumn<BlockingAppInterval>(entity.GetString("BlockingAppsJson")),
};
}
private static List<T> DeserializeJsonColumn<T>(string? json)
{
if (string.IsNullOrEmpty(json)) return new List<T>();
try
{
return JsonSerializer.Deserialize<List<T>>(json!) ?? new List<T>();
}
catch (JsonException)
{
return new List<T>(); // corrupt/truncated column — degrade to empty, never throw on read
}
}
// ===== TIME ATTRIBUTION DAILY AGGREGATES =====
/// <summary>Upserts one daily aggregate row (Replace — the sweep recomputes rows whole).</summary>
public async Task<bool> SaveTimeAttributionAggregateAsync(TimeAttributionDailyAggregate aggregate)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.TimeAttributionAggregates);
await tableClient.UpsertEntityAsync(
BuildTimeAttributionAggregateEntity(aggregate), TableUpdateMode.Replace);
return true;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to save time-attribution aggregate {Date}/{Class} for tenant {TenantId}",
aggregate.Date, aggregate.EnrollmentClass, aggregate.TenantId);
return false;
}
}
// internal static: pinned by round-trip tests together with MapToTimeAttributionAggregate.
internal static TableEntity BuildTimeAttributionAggregateEntity(TimeAttributionDailyAggregate aggregate)
{
return new TableEntity(aggregate.TenantId, $"{aggregate.Date}|{aggregate.EnrollmentClass}")
{
["Date"] = aggregate.Date,
["EnrollmentClass"] = aggregate.EnrollmentClass,
["AttributionVersion"] = aggregate.AttributionVersion,
["CleanSessionCount"] = aggregate.CleanSessionCount,
["FlaggedExcludedCount"] = aggregate.FlaggedExcludedCount,
["MissingBreakdownCount"] = aggregate.MissingBreakdownCount,
["SegmentStatsJson"] = JsonSerializer.Serialize(aggregate.SegmentStats),
["TopBlockingAppsJson"] = JsonSerializer.Serialize(aggregate.TopBlockingApps),
["ComputedAt"] = EnsureUtc(aggregate.ComputedAt),
};
}
/// <summary>
/// Reads the aggregate rows of one tenant partition ("global" allowed) for an inclusive
/// date range. RowKeys are "{yyyy-MM-dd}|{class}", so a string range filter covers the
/// window; dates are server-formatted (injection-safe).
/// </summary>
public async Task<List<TimeAttributionDailyAggregate>> GetTimeAttributionAggregatesAsync(
string tenantId, DateTime startDate, DateTime endDate)
{
if (tenantId != "global")
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.TimeAttributionAggregates);
// '~' (0x7E) sorts after '|' (0x7C): "{end}~" upper-bounds every "{end}|{class}" key.
var filter = $"PartitionKey eq '{tenantId}' and RowKey ge '{startDate:yyyy-MM-dd}' and RowKey lt '{endDate:yyyy-MM-dd}~'";
var results = new List<TimeAttributionDailyAggregate>();
await foreach (var entity in tableClient.QueryAsync<TableEntity>(filter: filter))
{
results.Add(MapToTimeAttributionAggregate(entity));
}
return results;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to query time-attribution aggregates for tenant {TenantId}", tenantId);
return new List<TimeAttributionDailyAggregate>();
}
}
/// <summary>
/// Reads the rolling-window aggregate rows (RK "rolling30|{class}") of one tenant
/// partition ("global" allowed) — the range statistics the fleet panel renders. The
/// date-range getter never returns these (its digit-prefixed upper bound sorts below
/// "rolling…"), and vice versa.
/// </summary>
public async Task<List<TimeAttributionDailyAggregate>> GetRollingTimeAttributionAggregatesAsync(string tenantId)
{
if (tenantId != "global")
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.TimeAttributionAggregates);
// '}' (0x7D) sorts after '|' (0x7C) — prefix range over "rolling30|…".
var filter = $"PartitionKey eq '{tenantId}' and RowKey ge 'rolling30|' and RowKey lt 'rolling30}}'";
var results = new List<TimeAttributionDailyAggregate>();
await foreach (var entity in tableClient.QueryAsync<TableEntity>(filter: filter))
{
results.Add(MapToTimeAttributionAggregate(entity));
}
return results;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to query rolling time-attribution aggregates for tenant {TenantId}", tenantId);
return new List<TimeAttributionDailyAggregate>();
}
}
internal static TimeAttributionDailyAggregate MapToTimeAttributionAggregate(TableEntity entity)
{
return new TimeAttributionDailyAggregate
{
TenantId = entity.PartitionKey,
Date = entity.GetString("Date") ?? string.Empty,
EnrollmentClass = entity.GetString("EnrollmentClass") ?? string.Empty,
AttributionVersion = entity.GetInt32("AttributionVersion") ?? 0,
CleanSessionCount = entity.GetInt32("CleanSessionCount") ?? 0,
FlaggedExcludedCount = entity.GetInt32("FlaggedExcludedCount") ?? 0,
MissingBreakdownCount = entity.GetInt32("MissingBreakdownCount") ?? 0,
SegmentStats = DeserializeJsonColumn<TimeAttributionSegmentStat>(entity.GetString("SegmentStatsJson")),
TopBlockingApps = DeserializeJsonColumn<TimeAttributionBlockingAppStat>(entity.GetString("TopBlockingAppsJson")),
ComputedAt = entity.GetDateTimeOffset("ComputedAt")?.UtcDateTime ?? DateTime.MinValue,
};
}
/// <summary>
/// Deletes one aggregate row (daily "{date}|{class}" or rolling "rolling30|{class}").
/// Used by the sweep's stale-bucket reconcile: a bucket that was not regenerated this
/// run (its sessions were deleted or its class left the window) must not keep serving
/// old numbers. Missing rows are a no-op.
/// </summary>
public async Task DeleteTimeAttributionAggregateAsync(string tenantId, string dateKey, string enrollmentClass)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.TimeAttributionAggregates);
await tableClient.DeleteEntityAsync(tenantId, $"{dateKey}|{enrollmentClass}");
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to delete stale time-attribution aggregate {Date}/{Class} for tenant {TenantId}",
dateKey, enrollmentClass, tenantId);
}
}
/// <summary>
/// Retention: deletes aggregate rows older than the cutoff (RowKey starts with the date,
/// so a string compare works across partitions). Mirrors the UsageMetrics 180d policy.
/// </summary>
public async Task<int> DeleteTimeAttributionAggregatesOlderThanAsync(DateTime cutoffDate)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.TimeAttributionAggregates);
var filter = $"RowKey lt '{cutoffDate:yyyy-MM-dd}'";
var deleted = 0;
await foreach (var entity in tableClient.QueryAsync<TableEntity>(
filter: filter, select: new[] { "PartitionKey", "RowKey" }))
{
await tableClient.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
deleted++;
}
return deleted;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to delete old time-attribution aggregates");
return 0;
}
}
}
}
@@ -159,6 +159,37 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// Safely reads a Double property from a TableEntity.
/// Returns null instead of throwing when the property has a different type (legacy data).
/// Also accepts Int32/Int64-typed cells — Table Storage stores a whole-number double
/// as Int when written via JSON paths that drop the decimal point.
/// </summary>
private double? SafeGetDouble(TableEntity entity, string key)
{
try
{
return entity.GetDouble(key);
}
catch (InvalidOperationException)
{
if (entity.TryGetValue(key, out var raw))
{
switch (raw)
{
case int i: return i;
case long l: return l;
case string s when double.TryParse(s,
System.Globalization.NumberStyles.Float,
System.Globalization.CultureInfo.InvariantCulture, out var parsed):
return parsed;
}
}
_logger.LogWarning("Property '{Key}' on entity {PK}/{RK} is not Double and could not be coerced", key, entity.PartitionKey, entity.RowKey);
return null;
}
}
/// <summary>
/// Safely reads a DateTime property from a TableEntity.
/// Returns null instead of throwing when the property has a different type (legacy data).
@@ -30,6 +30,9 @@ public static class TenantNotificationAudienceCatalog
["sla_breach"] = NotificationAudience.Member,
["sla_consecutive_failures"] = NotificationAudience.Member,
["sla_resolved"] = NotificationAudience.Member,
// F3 regression radar: rule-tuning is an admin concern (the alert deep-links into the
// analyze-rules page, which members cannot act on).
["rule_frequency_regression"] = NotificationAudience.Admin,
};
public static NotificationAudience Resolve(string? type)
+14 -1
View File
@@ -70,7 +70,20 @@ ENV NODE_ENV=production
ENV HF_CACHE_DIR=/app/.hf-cache
COPY --from=build /app/package.json /app/package-lock.json ./
RUN npm ci --omit=dev --ignore-scripts && npm cache clean --force
# npm is a BUILD-time tool only: the runtime is `node dist/index.js`, the
# HEALTHCHECK is `node -e`, and the server never shells out (no child_process
# anywhere in src/). But the npm that node:22-slim bundles carries its own
# vendored dependency tree under /usr/local/lib/node_modules/npm, and a
# container image scanner reports every CVE in it against this image —
# @sigstore/core, brace-expansion, tar, sigstore, picomatch, ip-address, plus
# npm itself, none of which appear in our package-lock and none of which
# Dependabot can see.
#
# Removing npm clears all of them; upgrading it does not (npm 11.x still
# vendors @sigstore/core 3.1.0, inside the vulnerable <= 3.2.0 range). The rm
# must share this RUN so the deletion lands in the same layer as the install.
RUN npm ci --omit=dev --ignore-scripts && npm cache clean --force && \
rm -rf /usr/local/lib/node_modules/npm /usr/local/bin/npm /usr/local/bin/npx
COPY --from=build /app/dist/ ./dist/
COPY --from=build /app/.hf-cache/ ./.hf-cache/
+11 -8
View File
@@ -9,7 +9,7 @@
"version": "1.5.1",
"dependencies": {
"@huggingface/transformers": "^3.8.1",
"@modelcontextprotocol/sdk": "^1.29.0",
"@modelcontextprotocol/sdk": "^1.30.0",
"compression": "^1.8.1",
"express": "^5.2.1",
"fuse.js": "^7.5.0",
@@ -25,6 +25,9 @@
"tsx": "^4.23.1",
"typescript": "^5.0.0",
"vitest": "^4.1.10"
},
"engines": {
"node": ">=22"
}
},
"node_modules/@emnapi/core": {
@@ -1066,12 +1069,12 @@
"license": "MIT"
},
"node_modules/@modelcontextprotocol/sdk": {
"version": "1.29.0",
"resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.29.0.tgz",
"integrity": "sha512-zo37mZA9hJWpULgkRpowewez1y6ML5GsXJPY8FI0tBBCd77HEvza4jDqRKOXgHNn867PVGCyTdzqpz0izu5ZjQ==",
"version": "1.30.0",
"resolved": "https://registry.npmjs.org/@modelcontextprotocol/sdk/-/sdk-1.30.0.tgz",
"integrity": "sha512-xKd8OIzlqNzcqcNumGAa6g+PW2kjD5vrpcKOnfldAUPP3j7lnqMPwlTXQm8gF+UwH72z0lqaRbjr9hqGz0eITA==",
"license": "MIT",
"dependencies": {
"@hono/node-server": "^1.19.9",
"@hono/node-server": "^1.19.9 || ^2.0.5",
"ajv": "^8.17.1",
"ajv-formats": "^3.0.1",
"content-type": "^1.0.5",
@@ -3772,9 +3775,9 @@
"license": "MIT"
},
"node_modules/tar": {
"version": "7.5.20",
"resolved": "https://registry.npmjs.org/tar/-/tar-7.5.20.tgz",
"integrity": "sha512-9FcyK4PA6+WbzlTM9WhQm6vB5W7cP7dUiPsv1g7YDwEQnQ1CGpK3MGlKk/ITVWMk05kHZuBhmVhiv8LZoy/PFQ==",
"version": "7.5.22",
"resolved": "https://registry.npmjs.org/tar/-/tar-7.5.22.tgz",
"integrity": "sha512-MFO/QzvtAOmJbkhOaCTvbGcFN9L9b+JunIsDwaKljSOdcLMea3NJ1k9Usz/rjdfSXTq4dfzfeS7W4p4YOAAHeA==",
"license": "BlueOak-1.0.0",
"dependencies": {
"@isaacs/fs-minipass": "^4.0.0",
@@ -20,7 +20,7 @@
},
"dependencies": {
"@huggingface/transformers": "^3.8.1",
"@modelcontextprotocol/sdk": "^1.29.0",
"@modelcontextprotocol/sdk": "^1.30.0",
"compression": "^1.8.1",
"express": "^5.2.1",
"fuse.js": "^7.5.0",
@@ -111,6 +111,10 @@ describe('role catalog snapshot — privilege-leak guard', () => {
'get_api_usage',
'get_app_install_metrics',
'get_audit_logs',
// Available to every role: backed by MemberRead endpoints with TenantScoping.QueryParam /
// role-aware routing to the GlobalReadOrAdmin fleet variant — same placement as
// get_time_attribution (F2 counterpart).
'get_device_history',
'get_geographic_metrics',
'get_geographic_sessions',
'get_ime_version_history',
@@ -124,6 +128,9 @@ describe('role catalog snapshot — privilege-leak guard', () => {
'get_session_events',
'get_session_summary',
'get_software_inventory',
// Available to every role: backed by MemberRead endpoints (tenant) with role-aware
// routing to the GlobalReadOrAdmin variant — same placement as get_app_install_metrics.
'get_time_attribution',
'get_usage_metrics',
'get_vulnerability_summary',
'list_blocked_devices',
@@ -1047,7 +1047,11 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
'Get rule firing statistics for analyze and gather rules. Shows which rules fire most often, ' +
'their hit rates (fires/evaluations), and daily trends. Use to identify commonly triggered rules, ' +
'optimize rule definitions, or understand tenant-specific failure patterns. ' +
(ga ? 'Without tenantId returns global stats (cross-tenant). With tenantId returns tenant-specific stats. ' : '') +
'The tenant-scoped response also carries "regressions": active rule-frequency regression episodes ' +
'(analyze rules whose 7-day hit rate rose >=2x over their 28-day baseline with disjoint Wilson ' +
'intervals) incl. window/baseline counts and an optional dimension concentration — correlation ' +
'only, never causal. ' +
(ga ? 'Without tenantId returns global stats (cross-tenant; no regressions there). With tenantId returns tenant-specific stats. ' : '') +
'NOTE: default window is 30 days × every rule × per-day trend rows — for 20+ rules the response ' +
'easily exceeds 70 KB and can trip the response cap. Pass a tighter `startDate`/`endDate` window ' +
'(7 days is usually plenty) and/or `ruleType` filter to keep responses lean.',
@@ -1155,6 +1159,108 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
})
);
// Tool: get_time_attribution — F1 "where did the enrollment time go?" (insights spec §F1).
// Two modes over PRE-COMPUTED rows (nothing derived at request time): a per-session
// breakdown, or the fleet's rolling 30-day rollup per enrollment class. Registered for ALL
// callers with the usual role-aware routing.
server.registerTool(
'get_time_attribution',
{
title: 'Enrollment Time Attribution',
description:
'Answer "where did the enrollment time go?". Two modes: pass sessionId for one terminal session\'s ' +
'time breakdown — wall-clock partition into device_prep / esp_apps / identity_hello / user_esp / ' +
'desktop_handoff spans plus an EXPLICIT unattributed remainder (sums are exact against the session\'s ' +
'authoritative duration; the WhiteGlove pause is excluded by design), the ESP-blocking app install ' +
'intervals with critical-path occupancy, reboot outage spans, and data-quality flags. Omit sessionId ' +
'for the fleet rollup: rolling 30-day median/p75/p90 per segment per enrollment class (classes are ' +
'never mixed) plus the top time-consuming ESP-blocking apps with "up to" what-if savings bounds. ' +
'breakdown=null means the session has no computable attribution (pre-feature, non-terminal, or ' +
'Incomplete) — that is "unknown", never zero. ' +
(ga
? 'Fleet mode: omit tenantId for the cross-tenant aggregate, or pass tenantId to scope to one tenant. '
: ''),
inputSchema: {
sessionId: SessionIdSchema.optional()
.describe('Session (GUID) whose breakdown to fetch. Omit for the fleet rollup.'),
tenantId: z.string().optional().describe(tenantIdDescription(ga, delegated,
'Session mode: the session\'s tenant (needed cross-tenant). Fleet mode: filter to one tenant; omit for the cross-tenant aggregate.',
'Optional; ignored — data is scoped to your tenant.')),
},
annotations: READ_ONLY,
},
async (args) => withToolTelemetry('get_time_attribution', async () => {
try {
const { sessionId, tenantId: rawTenantId } = args;
const tenantId = enforceDelegatedTenant(rawTenantId);
if (sessionId) {
const data = await apiFetch(`/api/sessions/${sessionId}/time-attribution${buildQuery({ tenantId })}`);
return toolResultText(data, MAX_RESULT_SIZE_CHARS.small);
}
const path = pickGlobalOrTenantPath('/api/global/metrics/time-attribution', '/api/metrics/time-attribution', tenantId);
const data = await apiFetch(`${path}${buildQuery({ tenantId })}`);
return toolResultText(data, MAX_RESULT_SIZE_CHARS.adminStream);
} catch (error: unknown) {
return toolError('get_time_attribution', args, error);
}
})
);
// Tool: get_device_history — F2 device history / First-Time-Right (insights spec §F2).
// Two modes over rows the terminal seam + maintenance sweep maintain (journey semantics are
// server-side; nothing is re-derived here): a per-device terminal-enrollment chain by serial
// number, or the fleet FTR rollup. Registered for ALL callers with role-aware routing —
// same placement as get_time_attribution.
server.registerTool(
'get_device_history',
{
title: 'Device Enrollment History & First-Time-Right',
description:
'Answer "how often did this device retry?" and "how many devices enrolled right on the first try?". ' +
'Two modes: pass serialNumber for one device\'s enrollment history — the chain of its terminal ' +
'sessions (Succeeded/Failed/Incomplete; capped at the 20 most recent) with per-attempt status, ' +
'timestamps, the session\'s authoritative durationSeconds (WhiteGlove pause excluded; Incomplete has ' +
'none), plus derived journey counts (a journey = attempts until the first success; a >30-day gap ' +
'starts a new journey). history=null means no recorded history (unknown device, placeholder serial). ' +
'Omit serialNumber for the fleet First-Time-Right rollup: daily completed-journey and ' +
'first-time-right counts (additive — window rate = ratio of sums), attempt histogram, junk-serial ' +
'exclusion disclosure, and the repeat-devices list (devices whose current journey took ≥2 attempts). ' +
'Open journeys (device not successfully enrolled yet, incl. WhiteGlove awaiting its user session) ' +
'never count toward FTR. ' +
(ga
? 'Fleet mode: omit tenantId for the cross-tenant aggregate (no repeat-devices list there), or pass ' +
'tenantId to scope to one tenant. Device mode: pass tenantId for cross-tenant serial lookups. '
: ''),
inputSchema: {
serialNumber: z.string().min(1).max(128).optional()
.describe('Device serial number (trimmed, case-insensitive). Omit for the fleet FTR rollup.'),
days: z.coerce.number().int().min(1).max(180).optional()
.describe('Fleet mode window in days (default 30, max 180 — aggregate retention).'),
tenantId: z.string().optional().describe(tenantIdDescription(ga, delegated,
'Device mode: the device\'s tenant (needed cross-tenant). Fleet mode: filter to one tenant; omit for the cross-tenant aggregate.',
'Optional; ignored — data is scoped to your tenant.')),
},
annotations: READ_ONLY,
},
async (args) => withToolTelemetry('get_device_history', async () => {
try {
const { serialNumber, days, tenantId: rawTenantId } = args;
const tenantId = enforceDelegatedTenant(rawTenantId);
if (serialNumber) {
// Member route with TenantScoping.QueryParam — the same route serves every role;
// a GA/delegated caller passes tenantId for cross-tenant reads.
const data = await apiFetch(`/api/metrics/device-history${buildQuery({ serialNumber, tenantId })}`);
return toolResultText(data, MAX_RESULT_SIZE_CHARS.small);
}
const path = pickGlobalOrTenantPath('/api/global/metrics/device-journeys', '/api/metrics/device-journeys', tenantId);
const data = await apiFetch(`${path}${buildQuery({ tenantId, days })}`);
return toolResultText(data, MAX_RESULT_SIZE_CHARS.adminStream);
} catch (error: unknown) {
return toolError('get_device_history', args, error);
}
})
);
// Tool: get_software_inventory — installed-software catalog from the SoftwareInventory table.
// Registered for ALL callers, but role-aware: a non-GA caller only ever sees their own tenant's
// inventory (no tenantId / no cross-tenant scope in the schema — nothing to leak or tempt with).
@@ -877,6 +877,29 @@ namespace AutopilotMonitor.Shared
// Rule telemetry (daily per-rule fire/evaluation counters)
public const string RuleStats = "RuleStats";
// F1 time attribution (insights spec §F1). Breakdowns: PK=TenantId, RK=SessionId —
// one row per terminal session, computed once at the terminal transition (plus the
// 30d self-healing maintenance backfill); deleted with the session via the deletion
// manifest cascade + tenant offboarding partition wipe. Aggregates: PK=TenantId
// (or "global"), RK="{yyyy-MM-dd}|{enrollmentClass}" — daily fleet rollups
// (median/p75/p90 per segment, top blocking apps, what-if bounds), recomputed by the
// rolling maintenance sweep; 180d retention like UsageMetrics.
public const string SessionTimeBreakdowns = "SessionTimeBreakdowns";
public const string TimeAttributionAggregates = "TimeAttributionAggregates";
// F2 device history / First-Time-Right (insights spec §F2). DeviceHistories:
// PK=TenantId, RK=encoded normalized serial — one row per device key holding the
// terminal-session chain (cap 20) + derived journey counts; written inline at the
// session-terminal seam, healed by the rolling maintenance sweep (which also drops
// refs of deleted sessions tombstone-driven and deletes emptied rows). Deliberately
// NOT in the per-session deletion manifest (the row spans many sessions); wiped on
// tenant offboarding. DeviceJourneyAggregates: PK=TenantId (or "global"),
// RK="{yyyy-MM-dd}" — daily FTR rollups (completed journeys, first-time-right count,
// attempt histogram, junk-serial disclosure), additive across days; 180d retention
// like UsageMetrics.
public const string DeviceHistories = "DeviceHistories";
public const string DeviceJourneyAggregates = "DeviceJourneyAggregates";
// Configuration tables
public const string TenantConfiguration = "TenantConfiguration";
public const string AdminConfiguration = "AdminConfiguration";
@@ -1051,6 +1074,10 @@ namespace AutopilotMonitor.Shared
RuleStates,
AppInstallSummaries,
PlatformStats,
SessionTimeBreakdowns,
TimeAttributionAggregates,
DeviceHistories,
DeviceJourneyAggregates,
TenantConfiguration,
AdminConfiguration,
GlobalAdmins,
@@ -1,13 +1,16 @@
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Shared.DataAccess
{
/// <summary>
/// First-notification dedup store for distress-driven bell notifications. Used by
/// ReportDistressFunction to fire each tenant-admin notification at most once per subject
/// (lifetime dedup): hardware rejections per (tenant, manufacturer, model) and TPM PSS
/// incompatibilities per (tenant, serial number).
/// Notification dedup store (one table, RowKey-prefixed key spaces). Subjects:
/// hardware rejections per (tenant, manufacturer, model), TPM PSS incompatibilities per
/// (tenant, serial number) — both lifetime insert-once — and F3 rule-frequency
/// regressions per (tenant, ruleId), whose rows additionally carry the alert payload,
/// are refreshed while the episode stays active, and are deleted when the rate re-arms.
/// </summary>
public interface IHardwareRejectionNotificationTracker
{
@@ -28,13 +31,36 @@ namespace AutopilotMonitor.Shared.DataAccess
/// </summary>
Task<bool> TryRegisterFirstTpmPssNotificationAsync(string tenantId, string serialNumber);
// --- F3 rule-frequency regressions (RowKey "ruleregression|{ruleId}") ---
/// <summary>
/// Atomically opens a regression episode for (tenantId, alert.RuleId). Returns true when
/// this pass owns the episode (caller fires bell + ops event exactly once), false when an
/// episode is already active or on failure (fail-closed — never double-fire).
/// </summary>
Task<bool> TryRegisterRuleRegressionAsync(string tenantId, RuleRegressionAlert alert);
/// <summary>
/// Refreshes an active episode's numbers (window/baseline counts, rates, LastEvaluatedAt)
/// so badges and the regressions[] block stay current. FirstNotifiedAt is carried from the
/// alert unchanged — the retention sweep re-arms on the ORIGINAL notification age.
/// Fail-soft; never fires notifications.
/// </summary>
Task RefreshRuleRegressionAsync(string tenantId, RuleRegressionAlert alert);
/// <summary>Closes an episode (rate re-armed: fell under 1.5× baseline or stopped firing). 404-tolerant.</summary>
Task DeleteRuleRegressionAsync(string tenantId, string ruleId);
/// <summary>Active regression episodes of one tenant (RowKey-prefix scan; empty on failure — fail-soft reads).</summary>
Task<List<RuleRegressionAlert>> GetRuleRegressionsAsync(string tenantId);
/// <summary>
/// Retention cleanup: deletes tracker rows whose FirstNotifiedAt is older than
/// <paramref name="cutoffUtc"/>. Both key spaces live in one table, so pruning re-arms the
/// dedup for BOTH subjects: a hardware model rejected again after the cutoff rings once more,
/// and so does a TPM-PSS-incompatible device that reports again. That is intended — a bell
/// that can never ring a second time would go silent on a device the tenant never fixed.
/// Returns rows deleted.
/// <paramref name="cutoffUtc"/>. Every key space lives in one table, so pruning re-arms
/// them all: a hardware model rejected again after the cutoff rings once more, so does a
/// TPM-PSS-incompatible device that reports again, and a rule regression still active
/// after 30 days fires a fresh bell (spec §F3: tracker retention cleanup re-arms — a
/// month-old still-burning regression is worth a reminder). Returns rows deleted.
/// </summary>
Task<int> DeleteOlderThanAsync(DateTime cutoffUtc);
}
@@ -119,6 +119,14 @@ namespace AutopilotMonitor.Shared.DataAccess
Task<List<OrphanedEventSession>> GetOrphanedEventSessionsAsync(TimeSpan gracePeriod);
Task DeleteEventSessionIndexEntryAsync(string tenantId, string sessionId);
// --- Session Tombstones (F2 device-journey sweep input) ---
/// <summary>
/// All session-tombstone keys, including expired-but-unpruned markers: every session
/// deletion (cascade + retention) writes one, so these are the proof set the F2 sweep
/// uses to drop device-history chain refs of deleted sessions.
/// </summary>
Task<List<(string TenantId, string SessionId)>> GetAllSessionTombstoneKeysAsync();
// --- Tenant Offboarding ---
/// <summary>
@@ -95,6 +95,36 @@ namespace AutopilotMonitor.Shared.DataAccess
Task<List<RuleStatsEntry>> GetRuleStatsAsync(string? tenantId = null, string? startDate = null,
string? endDate = null, string? ruleType = null, int maxResults = 500);
Task<int> DeleteRuleStatsOlderThanAsync(DateTime cutoffDate);
// --- F1 Time Attribution (insights spec §F1, PR2) ---
/// <summary>Computes + stores the session's time breakdown; null when not computable (non-terminal, no duration, events aged out) — fail-soft.</summary>
Task<SessionTimeBreakdown?> ComputeAndStoreSessionTimeBreakdownAsync(string tenantId, string sessionId);
Task<SessionTimeBreakdown?> GetSessionTimeBreakdownAsync(string tenantId, string sessionId);
/// <summary>Re-joins the session's app rows against its latest esp_config_detected lists (positive evidence only). Idempotent + fail-soft — the terminal seam runs it once; the sweep re-runs it when late events changed the stream.</summary>
Task ResolveEspBlockingForSessionAsync(string tenantId, string sessionId);
Task<bool> SaveTimeAttributionAggregateAsync(TimeAttributionDailyAggregate aggregate);
/// <summary>Aggregate rows of one tenant partition ("global" allowed) for an inclusive date range.</summary>
Task<List<TimeAttributionDailyAggregate>> GetTimeAttributionAggregatesAsync(string tenantId, DateTime startDate, DateTime endDate);
/// <summary>Rolling-window rows (Date "rolling30") of one tenant partition ("global" allowed) — the fleet panel's range statistics.</summary>
Task<List<TimeAttributionDailyAggregate>> GetRollingTimeAttributionAggregatesAsync(string tenantId);
/// <summary>Deletes one aggregate row (daily date key or "rolling30") — the sweep's stale-bucket reconcile.</summary>
Task DeleteTimeAttributionAggregateAsync(string tenantId, string dateKey, string enrollmentClass);
Task<int> DeleteTimeAttributionAggregatesOlderThanAsync(DateTime cutoffDate);
// --- F2 Device History / First-Time-Right (insights spec §F2, PR4) ---
/// <summary>Point-reads one device's history row by NORMALIZED serial; null when absent.</summary>
Task<DeviceHistory?> GetDeviceHistoryAsync(string tenantId, string serialKey);
/// <summary>All device-history rows of one tenant (sweep's tombstone-cleanup scan).</summary>
Task<List<DeviceHistory>> GetDeviceHistoriesByTenantAsync(string tenantId);
Task<bool> UpsertDeviceHistoryAsync(DeviceHistory history);
/// <summary>Deletes a device's history row (used when every chain ref belonged to deleted sessions).</summary>
Task DeleteDeviceHistoryAsync(string tenantId, string serialKey);
Task<bool> SaveDeviceJourneyAggregateAsync(DeviceJourneyDailyAggregate aggregate);
/// <summary>Daily FTR rows of one tenant partition ("global" allowed), inclusive date range; counts are additive across days.</summary>
Task<List<DeviceJourneyDailyAggregate>> GetDeviceJourneyAggregatesAsync(string tenantId, DateTime startDate, DateTime endDate);
/// <summary>Deletes one daily FTR row — the sweep's stale-bucket reconcile.</summary>
Task DeleteDeviceJourneyAggregateAsync(string tenantId, string dateKey);
Task<int> DeleteDeviceJourneyAggregatesOlderThanAsync(DateTime cutoffDate);
}
/// <summary>
@@ -45,6 +45,8 @@ namespace AutopilotMonitor.Shared.DataAccess
public const string Tenant = "Tenant";
public const string Agent = "Agent";
public const string Sla = "SLA";
/// <summary>Platform infrastructure alerts relayed from Azure Monitor (ops alert webhook).</summary>
public const string Platform = "Platform";
}
/// <summary>
@@ -1,3 +1,4 @@
using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using AutopilotMonitor.Shared.Models;
@@ -14,6 +15,16 @@ namespace AutopilotMonitor.Shared.DataAccess
Task<bool> StoreRuleResultAsync(RuleResult result);
Task<List<RuleResult>> GetRuleResultsAsync(string tenantId, string sessionId);
/// <summary>
/// SessionIds where <paramref name="ruleId"/> produced a result detected at/after
/// <paramref name="sinceUtc"/> — the F3 radar's hit set for on-fire dimension
/// correlation. Tenant-scoped partition-range scan with server-side RowKey (= ruleId)
/// and DetectedAt filters; bounded by <paramref name="maxResults"/> and only run for
/// the rare fired alert. Fail-soft (empty on error — the alert then simply carries no
/// dimension claim).
/// </summary>
Task<List<string>> GetRuleHitSessionIdsAsync(string tenantId, string ruleId, DateTime sinceUtc, int maxResults = 2000);
// --- Gather Rules ---
Task<bool> StoreGatherRuleAsync(GatherRule rule, string tenantId = "global");
Task<List<GatherRule>> GetGatherRulesAsync(string partitionKey);
@@ -107,6 +107,14 @@ namespace AutopilotMonitor.Shared.DataAccess
Task UpdateSessionGeoAsync(string tenantId, string sessionId,
string? country, string? region, string? city, string? loc);
Task UpdateSessionImeAgentVersionAsync(string tenantId, string sessionId, string version);
/// <summary>
/// Persists the session-wide average agent→backend HTTP latency derived from the
/// cumulative counters of the latest <c>agent_metrics_snapshot</c>. Last-write-wins
/// (cumulative counters make replays idempotent); fail-soft like the other
/// supplementary-field updaters.
/// </summary>
Task UpdateSessionNetworkLatencyAsync(string tenantId, string sessionId,
double avgLatencyMs, int requestCount);
// --- Server→Agent Actions ---
/// <summary>
@@ -54,6 +54,16 @@ namespace AutopilotMonitor.Shared.Models
/// <summary>Percentage difference from global avg throughput (positive = faster)</summary>
public double ThroughputVsGlobalPct { get; set; }
/// <summary>
/// Average agent→backend HTTP round-trip (ms) at this location, weighted per session by
/// its request count. 0 when no session here carries latency data (pre-feature agents).
/// </summary>
public double AvgApiLatencyMs { get; set; }
/// <summary>Sessions at this location that carry API-latency data</summary>
public int ApiLatencySessionCount { get; set; }
/// <summary>Percentage difference from global avg API latency (positive = slower/farther)</summary>
public double ApiLatencyVsGlobalPct { get; set; }
public bool IsOutlier { get; set; }
/// <summary>"fast", "slow", or null</summary>
public string? OutlierDirection { get; set; }
@@ -142,6 +152,8 @@ namespace AutopilotMonitor.Shared.Models
GeoRegion = s.GeoRegion,
GeoCity = s.GeoCity,
GeoLoc = s.GeoLoc,
AvgApiLatencyMs = s.AvgApiLatencyMs,
ApiRequestCount = s.ApiRequestCount,
PlatformScriptCount = s.PlatformScriptCount,
RemediationScriptCount = s.RemediationScriptCount,
PendingActionsJson = s.PendingActionsJson,
@@ -173,6 +185,8 @@ namespace AutopilotMonitor.Shared.Models
public double AvgMinutesPerApp { get; set; }
public double AvgThroughputBytesPerSec { get; set; }
public double StdDevDurationMinutes { get; set; }
/// <summary>Global request-weighted average agent→backend API latency (ms); 0 = no data yet</summary>
public double AvgApiLatencyMs { get; set; }
/// <summary>Global weighted average peer caching percentage</summary>
public double AvgDoPercentPeerCaching { get; set; }
/// <summary>Total peer bytes across all locations with DO data</summary>
@@ -12,6 +12,36 @@ namespace AutopilotMonitor.Shared.Models
public string SessionId { get; set; } = string.Empty;
public string TenantId { get; set; } = string.Empty;
/// <summary>
/// Intune app identity from the agent's app events (<c>appId</c> payload field) —
/// Win32/IME apps carry the Intune app GUID in lowercase dashed form
/// (<c>AppPackageState.Id</c>). Empty = sentinel: written before this column existed
/// (2026-07 F1 PR1) or the events carried no appId. The row stays name-keyed
/// (RowKey = {SessionId}_{AppName}); this column adds identity without a key migration.
/// </summary>
public string AppId { get; set; } = string.Empty;
/// <summary>
/// Whether this app is in the ESP's own blocking/tracking set
/// (<c>esp_config_detected</c> lists, joined by <see cref="AppId"/> at session-terminal
/// processing). Tri-state by design (source-data audit Q2): <c>true</c> = listed
/// (positive evidence), <c>null</c> = unknown (absent from the lists, lists never
/// observed, or row predates the column) — NEVER <c>false</c>: the lists are read
/// early/partially and MSI/PFN namespaces may not match, so absence is not evidence
/// of non-blocking.
/// </summary>
public bool? EspBlocking { get; set; }
/// <summary>
/// True when the same app display name was observed with a DIFFERENT <c>appId</c> in
/// this session (source-data audit Q3): the name-keyed row then merges two distinct
/// apps (e.g. device- + user-scope assignment of "Company Portal") and its
/// status/duration mix is unattributable. Flagged rows are excluded from per-app
/// fleet aggregates (with a disclosed exclusion count) instead of attempting a RowKey
/// migration. Sticky once set.
/// </summary>
public bool AppIdCollision { get; set; }
/// <summary>
/// Lifecycle status: Succeeded, Failed, InProgress, or empty.
/// Empty (default) is a sentinel meaning "no status-relevant event observed in the current
@@ -23,6 +53,19 @@ namespace AutopilotMonitor.Shared.Models
/// </summary>
public string Status { get; set; } = string.Empty;
/// <summary>
/// Terminal lifecycle state carried by the agent payload on the closing event:
/// "Installed", "Skipped", "Postponed" (from <c>app_install_completed</c>'s
/// <c>state</c> field / <c>app_install_skipped</c>) or "Error" (from
/// <c>app_install_failed</c>). Empty = sentinel: no terminal event observed in the
/// current batch, or a row written before this column existed (2026-07 PR0) —
/// storage omits the column on empty so Merge-mode preserves a prior value.
/// Skipped/Postponed rows are no real install attempt: they are excluded from
/// duration statistics and from the failure/success rate (PR0 decision 2026-07-26);
/// Status stays "Succeeded" for backward compatibility.
/// </summary>
public string TerminalState { get; set; } = string.Empty;
/// <summary>Total installation duration in seconds (from start to complete/failed)</summary>
public int DurationSeconds { get; set; }
@@ -0,0 +1,77 @@
using System;
using System.Collections.Generic;
namespace AutopilotMonitor.Shared.Models
{
/// <summary>
/// One terminal session of a device inside its <see cref="DeviceHistory.Chain"/> — the
/// compact ref the F2 journey grouping runs on (insights spec §F2). Only terminal sessions
/// (Succeeded / Failed / Incomplete) become refs: a WhiteGlove <c>Pending</c> or an
/// AwaitingUser/Stalled session is an OPEN session and must never appear as an attempt.
/// </summary>
public class DeviceSessionRef
{
public string SessionId { get; set; } = string.Empty;
public DateTime StartedAt { get; set; }
/// <summary>Terminal timestamp; feeds the 30-day journey-gap rule (fallback: StartedAt).</summary>
public DateTime? CompletedAt { get; set; }
/// <summary><see cref="SessionStatus"/> name ("Succeeded" / "Failed" / "Incomplete").</summary>
public string Status { get; set; } = string.Empty;
/// <summary>"v1" (Autopilot Classic/ESP) or "v2" (Windows Device Preparation).</summary>
public string EnrollmentType { get; set; } = "v1";
public bool IsPreProvisioned { get; set; }
/// <summary>
/// The session's authoritative <c>DurationSeconds</c> verbatim (WhiteGlove pause excluded
/// by design) — F2 surfaces must never recompute CompletedAt − StartedAt, which is later
/// in 25 % of terminal sessions. Null for Incomplete (deliberately stores no duration).
/// </summary>
public int? DurationSeconds { get; set; }
/// <summary>An administrator flipped this session's terminal status via the portal — the chain flags it (truthfulness guard §F2).</summary>
public bool AdminMarked { get; set; }
}
/// <summary>
/// Per-device enrollment history for F2 (insights spec §F2) — one row per device key
/// (TenantId, normalized serial), holding the compact chain of the device's terminal
/// sessions (capped at the 20 most recent) plus derived journey counts. Written inline at
/// every session-terminal transition (so the session-detail banner is fresh) and healed by
/// the rolling maintenance sweep, which also drops refs of deleted sessions (tombstone-driven)
/// and deletes the row when the chain empties. Junk serials (placeholder identities) never
/// get a row. Persisted in <c>DeviceHistories</c>; wiped on tenant offboarding — deliberately
/// NOT part of the per-session deletion manifest, because the row aggregates many sessions.
/// </summary>
public class DeviceHistory
{
public string TenantId { get; set; } = string.Empty;
/// <summary>Normalized device key component: trimmed + lower-cased serial (spec: trim + case-fold).</summary>
public string SerialKey { get; set; } = string.Empty;
/// <summary>Display serial as last reported (trimmed, original casing).</summary>
public string SerialNumber { get; set; } = string.Empty;
public string Manufacturer { get; set; } = string.Empty;
public string Model { get; set; } = string.Empty;
/// <summary>Terminal session refs ordered by StartedAt ascending, capped at the 20 most recent.</summary>
public List<DeviceSessionRef> Chain { get; set; } = new List<DeviceSessionRef>();
/// <summary>Attempt count of the LAST journey in the chain (open or completed) — the "Attempt N" the session banner shows.</summary>
public int CurrentJourneyAttempts { get; set; }
/// <summary>Journeys represented in the retained chain (the 20-cap can hide older journeys — this is a chain-scoped count, not a lifetime claim).</summary>
public int JourneyCount { get; set; }
/// <summary>Journey-grouping algorithm version (truthfulness rule 8: a definition change never silently mixes semantics).</summary>
public int JourneyVersion { get; set; }
public DateTime LastUpdated { get; set; }
}
}
@@ -0,0 +1,51 @@
using System;
using System.Collections.Generic;
namespace AutopilotMonitor.Shared.Models
{
/// <summary>One attempt-histogram bucket: how many completed journeys took exactly <see cref="Attempts"/> attempts.</summary>
public class DeviceJourneyAttemptBucket
{
public int Attempts { get; set; }
public int JourneyCount { get; set; }
}
/// <summary>
/// Daily First-Time-Right rollup for one (tenant, date) — F2 PR4 (insights spec §F2).
/// A journey counts on the StartedAt date of its COMPLETING success session (same
/// StartedAt-date bucketing as every other daily aggregate; the rolling sweep re-buckets
/// late-terminating sessions idempotently). Only journeys that ended with a terminal
/// success are counted — open journeys (no success yet, incl. WhiteGlove waiting for its
/// user session) and gap-abandoned journeys never enter numerator or denominator.
/// Counts are additive across days, so a window rate is the sum of daily rows — no
/// rolling-window row is needed (unlike the median-based time-attribution aggregates).
/// Rows are written even below the ≥20-completed-journeys UI gate: the UI needs the n to
/// say "insufficient data (n=3)" (truthfulness rule 4). Junk-serial exclusions are
/// disclosed per day (rule 7). "global" mirror rows sum all tenants.
/// </summary>
public class DeviceJourneyDailyAggregate
{
/// <summary>Calendar day (UTC, "yyyy-MM-dd") the completing success sessions STARTED.</summary>
public string Date { get; set; } = string.Empty;
/// <summary>Tenant GUID, or "global" for the cross-tenant row.</summary>
public string TenantId { get; set; } = string.Empty;
/// <summary>Journey-grouping algorithm version the counts were computed with (rule 8).</summary>
public int JourneyVersion { get; set; }
/// <summary>Journeys completed (first terminal success) on this date — the FTR denominator.</summary>
public int CompletedJourneyCount { get; set; }
/// <summary>Completed journeys with attempt count == 1 — the FTR numerator.</summary>
public int FirstTimeRightCount { get; set; }
/// <summary>Attempt distribution across the completed journeys, ordered by attempts ascending.</summary>
public List<DeviceJourneyAttemptBucket> AttemptHistogram { get; set; } = new List<DeviceJourneyAttemptBucket>();
/// <summary>Terminal sessions on this date excluded for junk/placeholder serials — disclosed, never silent (rule 7).</summary>
public int ExcludedSessionCount { get; set; }
public DateTime ComputedAt { get; set; }
}
}
@@ -0,0 +1,68 @@
using System;
namespace AutopilotMonitor.Shared.Models
{
/// <summary>
/// The dominant dimension value among a regression's hit sessions — CORRELATION only, and
/// every consumer's wording must say so ("correlated — not necessarily causal", insights
/// spec §F3 / truthfulness rule 6). Null on an alert means "no clear dimension
/// concentration" — the radar never stretches for one.
/// </summary>
public class RuleRegressionDimension
{
/// <summary>"osBuild", "model", "agentVersion" or "imeVersion".</summary>
public string Dimension { get; set; } = string.Empty;
public string Value { get; set; } = string.Empty;
/// <summary>Hit sessions carrying this value (gate: ≥5).</summary>
public int HitCount { get; set; }
public double HitSharePct { get; set; }
public double AllSharePct { get; set; }
/// <summary>HitShare ÷ AllShare (gate: ≥2.0).</summary>
public double Lift { get; set; }
}
/// <summary>
/// One ACTIVE rule-frequency regression (F3, insights spec §F3): an analyze rule whose
/// 7-day hit rate rose ≥2× over its 28-day baseline with disjoint Wilson intervals.
/// Persisted as the <c>ruleregression|{ruleId}</c> keyspace of the notification tracker
/// table — the row IS the dedup (one bell per episode), the badge state (rules page) and
/// the <c>regressions[]</c> payload (rule-stats response). Deleted when the rate re-arms
/// (falls under 1.5× baseline or stops firing) or by the tracker's 30-day retention sweep
/// (spec: retention cleanup re-arms). Numbers are refreshed on every radar pass while the
/// episode stays active; <see cref="FirstNotifiedAt"/> never moves.
/// </summary>
public class RuleRegressionAlert
{
public string TenantId { get; set; } = string.Empty;
public string RuleId { get; set; } = string.Empty;
public string RuleTitle { get; set; } = string.Empty;
public int WindowFireCount { get; set; }
public int WindowSessionCount { get; set; }
public int BaselineFireCount { get; set; }
public int BaselineSessionCount { get; set; }
public double WindowRatePct { get; set; }
public double BaselineRatePct { get; set; }
/// <summary>Window rate ÷ baseline rate. Null when the baseline rate is 0 (a NEW signal has no finite lift — never invented).</summary>
public double? Lift { get; set; }
/// <summary>Trailing 7-day window ("yyyy-MM-dd", inclusive) the current numbers describe.</summary>
public string WindowStartDate { get; set; } = string.Empty;
public string WindowEndDate { get; set; } = string.Empty;
/// <summary>Dimension concentration captured when the alert FIRST fired (stable story); null = no clear concentration.</summary>
public RuleRegressionDimension? Dimension { get; set; }
/// <summary>When the episode first fired (bell + ops event moment). Never moves on refresh; drives the 30d retention re-arm.</summary>
public DateTime FirstNotifiedAt { get; set; }
/// <summary>Last radar pass that re-confirmed/refreshed this episode.</summary>
public DateTime LastEvaluatedAt { get; set; }
}
}
@@ -0,0 +1,211 @@
using System;
using System.Collections.Generic;
namespace AutopilotMonitor.Shared.Models
{
/// <summary>
/// Canonical segment keys for the F1 time-attribution partition (insights spec §F1).
/// A segment may comprise multiple spans (e.g. identity_hello covers the AccountSetup
/// span AND the FinalizingSetup/Hello-wait span; WhiteGlove sessions split every segment
/// across two observation windows).
/// </summary>
public static class TimeAttributionSegments
{
/// <summary>Session start → first ESP apps-phase entry (Start/DevicePreparation/DeviceSetup phases).</summary>
public const string DevicePrep = "device_prep";
/// <summary>Device-scope ESP apps phase (EnrollmentPhase.AppsDevice spans).</summary>
public const string EspApps = "esp_apps";
/// <summary>Identity / sign-in / Windows Hello (AccountSetup + FinalizingSetup spans).</summary>
public const string IdentityHello = "identity_hello";
/// <summary>User-scope ESP apps phase (EnrollmentPhase.AppsUser spans), when present.</summary>
public const string UserEsp = "user_esp";
/// <summary>Last phase exit (desktop_arrived / Complete declaration) → terminal timestamp.</summary>
public const string DesktopHandoff = "desktop_handoff";
/// <summary>
/// Explicit remainder — time inside the observation window(s) no observed signal
/// accounts for. Reported, never redistributed (truthfulness rule 2: the partition
/// is exact and never normalized to 100 %).
/// </summary>
public const string Unattributed = "unattributed";
}
/// <summary>
/// Data-quality flags for a <see cref="SessionTimeBreakdown"/> (truthfulness rule 7:
/// problems flag the record and exclude it from fleet aggregates with a disclosed count,
/// rather than silently skewing them).
/// </summary>
[Flags]
public enum TimeAttributionFlags
{
None = 0,
/// <summary>At least one anchor/interval was dropped for going backward in time or falling outside the observation window.</summary>
ClockSkewDropped = 1,
/// <summary>The agent started late (<c>agent_late_start</c>): early phases are underobserved; excluded from fleet segment medians.</summary>
PartialObservation = 2,
/// <summary>No <c>esp_config_detected</c> tracking lists observed — per-app blocking membership is entirely unknown.</summary>
BlockingSetUnknown = 4,
/// <summary>The tracking lists hit the 50-per-category emission cap — positive evidence is incomplete.</summary>
BlockingSetTruncated = 8,
/// <summary>
/// WhiteGlove window boundaries rest on fallback anchors (no <c>whiteglove_part1_complete</c>
/// event, missing ResumedAt, or a derived part-1 length that contradicts the stored
/// combined duration) — the two-window split is best-effort rather than event-anchored.
/// </summary>
WhiteGloveAnchorsIncomplete = 16
}
/// <summary>
/// Interpretation helpers for <see cref="TimeAttributionFlags"/>. The duration mask lives
/// here as a const rather than as a composite enum member on purpose: the wire format
/// serializes flag combinations as comma-joined member names (the web splits that string),
/// and a composite member would hijack ToString for combinations that happen to equal it.
/// </summary>
public static class TimeAttributionFlagQuality
{
/// <summary>
/// Flags that make the SEGMENT DURATIONS themselves untrustworthy — dropped anchors,
/// underobserved early phases, best-effort WhiteGlove window boundaries. Only these
/// exclude a breakdown from fleet segment statistics. The blocking-set flags
/// (<see cref="TimeAttributionFlags.BlockingSetUnknown"/> / BlockingSetTruncated) say
/// nothing about the measured spans — they only limit per-app blocking evidence, which
/// is positive-evidence-based anyway (unknown contributes no intervals). Gating fleet
/// stats on ANY flag starved the aggregates in production (2026-07-27: 903/1000
/// breakdowns carried BlockingSetUnknown → CleanSessionCount 12 vs 331 excluded).
/// </summary>
public const TimeAttributionFlags DurationCriticalFlags =
TimeAttributionFlags.ClockSkewDropped |
TimeAttributionFlags.PartialObservation |
TimeAttributionFlags.WhiteGloveAnchorsIncomplete;
/// <summary>True when the breakdown's segment durations are unfit for fleet statistics.</summary>
public static bool ExcludesFromFleetStats(TimeAttributionFlags flags)
=> (flags & DurationCriticalFlags) != TimeAttributionFlags.None;
}
/// <summary>One contiguous attributed span inside an observation window.</summary>
public class TimeAttributionSpan
{
/// <summary>Segment key — one of the <see cref="TimeAttributionSegments"/> constants (never "unattributed": the remainder is a scalar, not a span claim).</summary>
public string SegmentKey { get; set; } = string.Empty;
public DateTime StartUtc { get; set; }
public DateTime EndUtc { get; set; }
/// <summary>Whole seconds (floored). The sub-second dust lands in the unattributed remainder — never invented into a segment.</summary>
public int Seconds { get; set; }
}
/// <summary>
/// Install interval of one ESP-blocking app, measured from EVENT timestamps
/// (first started/download event → last terminal event; source-data audit Q1 forbids the
/// agent payload timing, which freezes at the first terminal state across IME retries).
/// </summary>
public class BlockingAppInterval
{
public string AppId { get; set; } = string.Empty;
public string AppName { get; set; } = string.Empty;
public DateTime StartUtc { get; set; }
public DateTime EndUtc { get; set; }
public int Seconds { get; set; }
}
/// <summary>
/// One observed reboot outage: the gap between the last pre-reboot event and the first
/// post-reboot event, located via the <c>lastBootUtc</c> payload (audit Q7: the
/// <c>system_reboot_detected</c> event itself is detection-time-stamped by the NEXT agent
/// run and is never used as the reboot moment).
/// </summary>
public class RebootSpan
{
public DateTime StartUtc { get; set; }
public DateTime EndUtc { get; set; }
public int Seconds { get; set; }
/// <summary>Segment the reboot started in (<see cref="TimeAttributionSegments"/> key, or "unattributed" when it began in an unattributed hole).</summary>
public string SegmentKey { get; set; } = string.Empty;
}
/// <summary>
/// Per-session enrollment time attribution (F1, insights spec) — an exact partition of the
/// session's authoritative wall clock into named segments plus an explicit unattributed
/// remainder. Invariant (unit-enforced): sum of all span seconds + <see cref="UnattributedSeconds"/>
/// == <see cref="WallClockSeconds"/> == the session's <c>DurationSeconds</c> (which excludes
/// the WhiteGlove pause by design — never CompletedAt − StartedAt, which diverges in 25 % of
/// terminal sessions). Computed once at session-terminal processing by
/// <c>TimeAttributionCalculator</c>; persisted in <c>SessionTimeBreakdowns</c> (F1 PR2).
/// </summary>
public class SessionTimeBreakdown
{
public string TenantId { get; set; } = string.Empty;
public string SessionId { get; set; } = string.Empty;
/// <summary>Attribution algorithm version — a definition change bumps this so aggregates never silently mix semantics (truthfulness rule 8).</summary>
public int AttributionVersion { get; set; }
/// <summary>
/// The session's EventCount when this breakdown was computed — the maintenance sweep's
/// change signal: late or replayed event batches move the session's count, so a mismatch
/// means the row was computed from an incomplete stream and gets recomputed (Codex
/// review: the inline terminal compute is one-shot). 0 = row written before this column
/// existed → recomputed once by the next sweep.
/// </summary>
public int EventCountAtCompute { get; set; }
/// <summary>The session's authoritative DurationSeconds at compute time.</summary>
public int WallClockSeconds { get; set; }
/// <summary>Attributed spans, ordered by StartUtc. Multiple spans may share a segment key.</summary>
public List<TimeAttributionSpan> Segments { get; set; } = new List<TimeAttributionSpan>();
/// <summary>Exact remainder: <see cref="WallClockSeconds"/> − sum of span seconds.</summary>
public int UnattributedSeconds { get; set; }
/// <summary>Total observed reboot outage seconds. Cross-cutting annotation — reboots overlap the segments they occur in and are NOT part of the wall-clock partition.</summary>
public int RebootSeconds { get; set; }
public List<RebootSpan> RebootSpans { get; set; } = new List<RebootSpan>();
/// <summary>
/// Install intervals of ESP-blocking apps (positive-evidence join against the latest
/// <c>esp_config_detected</c> lists), top 20 by duration. <see cref="BlockingAppCount"/>
/// carries the uncapped count of matched blocking apps (including those without a
/// measurable interval, e.g. unobserved start).
/// </summary>
public List<BlockingAppInterval> BlockingApps { get; set; } = new List<BlockingAppInterval>();
/// <summary>Uncapped count of apps matched against the blocking set (see <see cref="BlockingApps"/>).</summary>
public int BlockingAppCount { get; set; }
/// <summary>
/// Overlap-merged union of the blocking-app intervals clipped to the esp_apps spans —
/// the critical-path occupancy. esp_apps total − occupancy = in-phase wait (provider
/// stalls, settle, IME idle). Null when the blocking set is unknown (no lists observed):
/// unknown, not zero.
/// </summary>
public int? EspAppsOccupancySeconds { get; set; }
public TimeAttributionFlags QualityFlags { get; set; }
/// <summary>Convenience rollup: total seconds per segment key across all spans (unattributed NOT included — read <see cref="UnattributedSeconds"/>).</summary>
public Dictionary<string, int> GetSegmentTotals()
{
var totals = new Dictionary<string, int>(StringComparer.Ordinal);
foreach (var span in Segments)
{
totals.TryGetValue(span.SegmentKey, out var current);
totals[span.SegmentKey] = current + span.Seconds;
}
return totals;
}
}
}
@@ -0,0 +1,83 @@
using System;
using System.Collections.Generic;
namespace AutopilotMonitor.Shared.Models
{
/// <summary>
/// Distribution of one attribution segment across the clean sessions of an aggregate row.
/// Values are whole seconds; a session that has no span of this segment contributes 0 —
/// the distribution answers "time spent in this segment per enrollment of this class",
/// not "per enrollment that happened to enter it".
/// </summary>
public class TimeAttributionSegmentStat
{
/// <summary>Segment key (<see cref="TimeAttributionSegments"/>), including "unattributed" — the stack must sum to the wall clock (truthfulness rule 2).</summary>
public string SegmentKey { get; set; } = string.Empty;
public int MedianSeconds { get; set; }
public int P75Seconds { get; set; }
public int P90Seconds { get; set; }
}
/// <summary>
/// Per-app rollup of ESP-blocking install intervals across the clean sessions of an
/// aggregate row. The what-if numbers are the per-session critical-path savings from
/// removing this app (recomputed union end without it) — an upper BOUND by construction:
/// UI copy must say "up to", never "you will save" (truthfulness rule 3).
/// </summary>
public class TimeAttributionBlockingAppStat
{
public string AppId { get; set; } = string.Empty;
public string AppName { get; set; } = string.Empty;
/// <summary>Clean sessions with a measured interval for this app (row gate: ≥5).</summary>
public int SessionCount { get; set; }
public int MedianSeconds { get; set; }
public int P75Seconds { get; set; }
public int MedianSavingSeconds { get; set; }
public int P75SavingSeconds { get; set; }
}
/// <summary>
/// Daily fleet rollup of session time attribution for one (tenant, enrollment class, date) —
/// F1 PR2 (insights spec §F1 "Data &amp; compute changes"). Enrollment classes are never
/// mixed (a WhiteGlove flow has a structurally different time profile than user-driven).
/// Only breakdowns with <see cref="TimeAttributionFlags.None"/> enter the statistics;
/// flagged sessions are excluded WITH a disclosed count (truthfulness rule 7). Rows are
/// written even below the ≥20-session UI gate — the UI needs the n to say
/// "insufficient data (n=3)" instead of silently rendering a small-n median (rule 4).
/// Recomputed idempotently by the rolling 30-day maintenance sweep.
/// </summary>
public class TimeAttributionDailyAggregate
{
/// <summary>Calendar day (UTC) the sessions STARTED, "yyyy-MM-dd" — same bucketing as the usage snapshots.</summary>
public string Date { get; set; } = string.Empty;
/// <summary>Tenant GUID, or "global" for the cross-tenant row.</summary>
public string TenantId { get; set; } = string.Empty;
/// <summary>Enrollment class: "user_driven", "whiteglove", "self_deploying" or "device_preparation" (WDP v2).</summary>
public string EnrollmentClass { get; set; } = string.Empty;
/// <summary>Attribution algorithm version the underlying breakdowns were computed with (rule 8: never mix definitions).</summary>
public int AttributionVersion { get; set; }
/// <summary>Breakdowns with QualityFlags == None that formed the statistics.</summary>
public int CleanSessionCount { get; set; }
/// <summary>Flagged breakdowns excluded from the statistics (disclosed, never silent).</summary>
public int FlaggedExcludedCount { get; set; }
/// <summary>Terminal sessions of this bucket without a computable breakdown (e.g. events aged out before backfill).</summary>
public int MissingBreakdownCount { get; set; }
public List<TimeAttributionSegmentStat> SegmentStats { get; set; } = new List<TimeAttributionSegmentStat>();
/// <summary>Top blocking apps by median interval, gated at ≥5 sessions per app, capped at 20.</summary>
public List<TimeAttributionBlockingAppStat> TopBlockingApps { get; set; } = new List<TimeAttributionBlockingAppStat>();
public DateTime ComputedAt { get; set; }
}
}
@@ -257,6 +257,24 @@ namespace AutopilotMonitor.Shared.Models
public int EventCount { get; set; }
public int? DurationSeconds { get; set; }
/// <summary>
/// Session-wide average HTTP round-trip latency (ms) of the agent's calls to the
/// backend API, as measured on the device (includes network RTT + TLS + server
/// processing). Projected at ingest from the cumulative counters of the latest
/// <c>agent_metrics_snapshot</c> (<c>net_total_latency_ms / net_total_requests</c>).
/// Null for V1 sessions and sessions from agents that predate the field.
/// Aggregate across sessions weighted by <see cref="ApiRequestCount"/> — e.g. per
/// GeoCountry to compare regional backend reachability.
/// </summary>
public double? AvgApiLatencyMs { get; set; }
/// <summary>
/// Number of HTTP requests behind <see cref="AvgApiLatencyMs"/> (the cumulative
/// <c>net_total_requests</c> at the latest snapshot). Needed as the weight when
/// averaging latency across sessions. Null whenever <see cref="AvgApiLatencyMs"/> is.
/// </summary>
public int? ApiRequestCount { get; set; }
/// <summary>
/// Enrollment type: "v1" (Autopilot Classic/ESP) or "v2" (Windows Device Preparation).
/// Defaults to "v1" for sessions that predate this field.
@@ -94,6 +94,10 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
"OffboardingFeedbackReceived",
"TenantTrialExpiring",
"TenantTrialExpired",
// F3 regression radar: an analyze rule's 7d hit rate rose ≥2× over its 28d baseline
// (Wilson-separated). Warning-tier, once per episode (tracker-deduped). Dual-register
// per memory feedback_ops_event_types_dual_register.
"RuleFrequencyRegression",
],
Agent: [
"BlobStorageMissing",
@@ -106,6 +110,15 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
// feedback_ops_event_types_dual_register.
"AgentEmergencyBreak",
],
Platform: [
// Azure Monitor alerts relayed through POST ops/alert-webhook (Common Alert Schema →
// AzureMonitorAlertWebhookFunction → RecordAzureMonitorAlertAsync). One event type for ALL
// Azure Monitor alert rules — the rule name travels in the message/details. Severity is
// mapped Sev0→Critical, Sev1→Error, Sev2→Warning, Sev3/4→Info; both Fired and Resolved
// notifications carry the same mapped severity, so a rule here delivers recovery pings too.
// Dual-register per memory feedback_ops_event_types_dual_register.
"AzureMonitorAlert",
],
};
const SEVERITIES = ["Info", "Warning", "Error", "Critical"];
@@ -116,6 +129,7 @@ const CATEGORY_COLORS: Record<string, string> = {
Security: "text-orange-700 dark:text-orange-300",
Tenant: "text-teal-700 dark:text-teal-300",
Agent: "text-cyan-700 dark:text-cyan-300",
Platform: "text-sky-700 dark:text-sky-300",
};
interface OpsAlertRulesSectionProps {
@@ -42,9 +42,10 @@ const CATEGORY_STYLES: Record<string, string> = {
Security: "bg-orange-100 text-orange-800 dark:bg-orange-900/40 dark:text-orange-300",
Tenant: "bg-teal-100 text-teal-800 dark:bg-teal-900/40 dark:text-teal-300",
Agent: "bg-cyan-100 text-cyan-800 dark:bg-cyan-900/40 dark:text-cyan-300",
Platform: "bg-sky-100 text-sky-800 dark:bg-sky-900/40 dark:text-sky-300",
};
const ALL_CATEGORIES = ["Consent", "Maintenance", "Security", "Tenant", "Agent"];
const ALL_CATEGORIES = ["Consent", "Maintenance", "Security", "Tenant", "Agent", "Platform"];
const PAGE_SIZE = 20;
@@ -7,6 +7,67 @@ import AnalyzeRuleFormFields from "./AnalyzeRuleFormFields";
import { stripInternalFields } from "@/lib/rulePageHelpers";
import { formatInlineMarkdown } from "@/lib/formatInlineMarkdown";
/** One day of rule telemetry (rule-stats `trend`; sparse — days without activity have no entry). */
export interface RuleTrendPoint {
date: string;
fireCount: number;
evaluationCount: number;
}
/** Active F3 regression episode for a rule (rule-stats `regressions[]`, tracker-backed). */
export interface RuleRegressionInfo {
ruleId: string;
windowFireCount: number;
windowSessionCount: number;
windowRatePct: number;
baselineFireCount: number;
baselineSessionCount: number;
baselineRatePct: number;
lift: number | null;
dimension?: {
dimension: string;
value: string;
hitSharePct: number;
allSharePct: number;
lift: number;
} | null;
}
/**
* 30-day fire-count sparkline (F3): densifies the sparse trend rows into one bar per day so
* gaps read as honest zeros. Hidden below lg — the collapsed header is already dense on mobile.
*/
function RuleSparkline({ trend }: { trend: RuleTrendPoint[] }) {
if (trend.length === 0) return null;
const byDate = new Map(trend.map((t) => [t.date, t.fireCount]));
const end = new Date(trend[trend.length - 1].date + "T00:00:00Z");
const values: number[] = [];
for (let i = 29; i >= 0; i--) {
const d = new Date(end);
d.setUTCDate(d.getUTCDate() - i);
values.push(byDate.get(d.toISOString().slice(0, 10)) ?? 0);
}
const max = Math.max(...values);
if (max === 0) return null;
return (
<span
className="hidden lg:inline-flex items-end flex-shrink-0 text-gray-300"
title={`Daily fires, last 30 days (peak ${max}/day)`}
aria-hidden
>
<svg width={90} height={16}>
{values.map((v, i) => {
const h = v > 0 ? Math.max(2, Math.round((v / max) * 16)) : 1;
return (
<rect key={i} x={i * 3} y={16 - h} width={2} height={h}
className={v > 0 ? "fill-indigo-300" : "fill-gray-200"} />
);
})}
</svg>
</span>
);
}
interface AnalyzeRuleCardProps {
rule: AnalyzeRule;
isExpanded: boolean;
@@ -44,6 +105,10 @@ interface AnalyzeRuleCardProps {
onScrollToCopy?: (ruleId: string) => void;
hitRate?: number | null;
fireCount?: number | null;
/** Sparse daily telemetry for the 30d sparkline (rule-stats `trend`). */
trend?: RuleTrendPoint[] | null;
/** Active regression episode → renders the F3 badge while the alert is live. */
regression?: RuleRegressionInfo | null;
}
export default function AnalyzeRuleCard({
@@ -57,7 +122,7 @@ export default function AnalyzeRuleCard({
readOnly = false,
variant = "default",
onConfigureTemplate, templateCopyExists, templateCopyRuleId, onScrollToCopy,
hitRate, fireCount,
hitRate, fireCount, trend, regression,
}: AnalyzeRuleCardProps) {
const isTemplateVariant = variant === "template";
const [showJson, setShowJson] = useState(false);
@@ -149,6 +214,7 @@ export default function AnalyzeRuleCard({
</span>
)}
<span className="text-xs text-gray-500 flex-shrink-0 hidden md:inline" title="Confidence Threshold">Threshold: {rule.confidenceThreshold}%</span>
{trend && trend.length > 0 && <RuleSparkline trend={trend} />}
{hitRate != null && hitRate > 0 && (
<span
className={`inline-flex items-center px-2 py-0.5 rounded text-xs font-medium flex-shrink-0 ${
@@ -161,6 +227,25 @@ export default function AnalyzeRuleCard({
{hitRate}% hit rate
</span>
)}
{/* F3 regression badge — visible only while an alert episode is active (tracker row).
Numbers in the tooltip mirror the bell notification; correlation wording only. */}
{regression && (
<span
className="inline-flex items-center px-2 py-0.5 rounded text-xs font-bold bg-red-100 text-red-800 border border-red-300 flex-shrink-0"
title={
`Firing more often than usual: ${regression.windowRatePct}% of evaluated sessions in the last 7 days ` +
`(${regression.windowFireCount}/${regression.windowSessionCount}) vs ${regression.baselineRatePct}% baseline ` +
`(${regression.baselineFireCount}/${regression.baselineSessionCount} over 28 days)` +
(regression.lift != null ? ` — lift ${regression.lift}x.` : " — new signal.") +
(regression.dimension
? ` ${regression.dimension.hitSharePct}% of affected sessions are on ${regression.dimension.dimension} ` +
`${regression.dimension.value} vs ${regression.dimension.allSharePct}% of all sessions — correlated, not necessarily causal.`
: "")
}
>
↑ Regression
</span>
)}
<svg className={`w-5 h-5 text-gray-400 transition-transform flex-shrink-0 ml-auto sm:ml-0 ${isExpanded ? "rotate-180" : ""}`} fill="none" stroke="currentColor" viewBox="0 0 24 24"><path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M19 9l-7 7-7-7" /></svg>
</div>
</div>
@@ -20,7 +20,7 @@ import { TenantScopeSelector } from "@/components/TenantScopeSelector";
import { isTemplateRule } from "@/lib/analyzeRuleTabs";
import { AnalyzeRule, RuleForm, EMPTY_FORM, EMPTY_CONDITION, ruleToForm } from "./types";
import AnalyzeRuleFormFields from "./components/AnalyzeRuleFormFields";
import AnalyzeRuleCard from "./components/AnalyzeRuleCard";
import AnalyzeRuleCard, { RuleTrendPoint, RuleRegressionInfo } from "./components/AnalyzeRuleCard";
import TemplateConfigModal from "./components/TemplateConfigModal";
import { DOCS_URL } from "@/utils/config";
@@ -75,8 +75,13 @@ export default function AnalyzeRulesPage() {
const [configureTemplateRule, setConfigureTemplateRule] = useState<AnalyzeRule | null>(null);
const [creatingFromTemplate, setCreatingFromTemplate] = useState(false);
// Rule telemetry stats (hit rates)
const [ruleStatsMap, setRuleStatsMap] = useState<Record<string, { hitRate: number; fireCount: number }>>({});
// Rule telemetry stats (hit rates + 30d trend for the sparkline + active F3 regressions)
const [ruleStatsMap, setRuleStatsMap] = useState<Record<string, {
hitRate: number;
fireCount: number;
trend: RuleTrendPoint[];
regression?: RuleRegressionInfo;
}>>({});
// Tenant notification channels (for the per-rule notify target selector)
const [tenantChannels, setTenantChannels] = useState<{ id: string; name: string }[]>([]);
@@ -124,10 +129,26 @@ export default function AnalyzeRulesPage() {
const response = await authenticatedFetch(statsUrl, getAccessToken);
if (response.ok) {
const data = await response.json();
const map: Record<string, { hitRate: number; fireCount: number }> = {};
const map: Record<string, {
hitRate: number; fireCount: number; trend: RuleTrendPoint[]; regression?: RuleRegressionInfo;
}> = {};
if (data.rules && Array.isArray(data.rules)) {
for (const r of data.rules) {
map[r.ruleId] = { hitRate: r.hitRate ?? 0, fireCount: r.fireCount ?? 0 };
map[r.ruleId] = {
hitRate: r.hitRate ?? 0,
fireCount: r.fireCount ?? 0,
trend: Array.isArray(r.trend) ? r.trend : [],
};
}
}
// F3: attach active regression episodes (tracker-backed) to their rules — a
// regression for a rule without stats rows still gets a synthetic entry so the
// badge renders regardless.
if (data.regressions && Array.isArray(data.regressions)) {
for (const regression of data.regressions as RuleRegressionInfo[]) {
const entry = map[regression.ruleId] ?? { hitRate: 0, fireCount: 0, trend: [] };
entry.regression = regression;
map[regression.ruleId] = entry;
}
}
setRuleStatsMap(map);
@@ -845,6 +866,8 @@ export default function AnalyzeRulesPage() {
onScrollToCopy={(copyId) => setExpandedRuleId(copyId)}
hitRate={ruleStatsMap[rule.ruleId]?.hitRate ?? null}
fireCount={ruleStatsMap[rule.ruleId]?.fireCount ?? null}
trend={ruleStatsMap[rule.ruleId]?.trend ?? null}
regression={ruleStatsMap[rule.ruleId]?.regression ?? null}
/>
</div>
))}
@@ -82,6 +82,10 @@ interface AnalyticsResponse {
summary: {
totalInstalls: number;
succeeded: number;
/** Not-applicable no-ops (TerminalState Skipped/Postponed) — not attempts, excluded from rate + durations. */
skipped: number;
/** Real installs whose start was never observed — duration unknown, excluded from duration stats. */
unmeasured: number;
failed: number;
failureRate: number;
avgDurationSeconds: number;
@@ -440,6 +444,7 @@ export default function AppDetailPage() {
label="Succeeded"
value={analytics.summary.succeeded.toString()}
color="text-emerald-700"
hint={analytics.summary.skipped > 0 ? `+${analytics.summary.skipped} skipped (not applicable)` : undefined}
/>
<SummaryCard
label="Failed"
@@ -460,6 +465,11 @@ export default function AppDetailPage() {
<SummaryCard
label="Avg Duration"
value={formatDuration(analytics.summary.avgDurationSeconds)}
hint={
analytics.summary.unmeasured > 0
? `measured installs only — ${analytics.summary.unmeasured} without observed start`
: "measured installs only"
}
/>
<SummaryCard label="Trend" value={trendText(analytics.summary.trend, analytics.summary.trendDelta)} />
<SummaryCard
@@ -23,6 +23,10 @@ interface AppRow {
appType: string;
totalInstalls: number;
succeeded: number;
/** Not-applicable no-ops (TerminalState Skipped/Postponed) — excluded from rate + durations. */
skipped: number;
/** Succeeded installs without an observed start — duration unknown, excluded from duration stats. */
unmeasured: number;
failed: number;
failureRate: number;
avgDurationSeconds: number;
@@ -288,7 +292,17 @@ export default function InstallsTab({ scope, timeRange }: InstallsTabProps) {
{appTypeBadge(row.appType)}
</td>
<td className="px-4 py-3 text-right text-gray-700">{row.totalInstalls}</td>
<td className="px-4 py-3 text-right text-emerald-700">{row.succeeded}</td>
<td className="px-4 py-3 text-right text-emerald-700">
{row.succeeded}
{row.skipped > 0 && (
<span
className="ml-1 text-xs text-gray-400 dark:text-gray-500"
title={`${row.skipped} skipped (not applicable) — excluded from failure rate and durations`}
>
+{row.skipped}s
</span>
)}
</td>
<td className="px-4 py-3 text-right text-red-700">{row.failed}</td>
<td className="px-4 py-3 text-right">
<span className={row.failureRate >= 20 ? "text-red-700 font-semibold" : row.failureRate >= 5 ? "text-amber-700" : "text-gray-700"}>

Some files were not shown because too many files have changed in this diff Show More