Compare commits

..

18 Commits

Author SHA1 Message Date
Oliver Kieselbach 249e39bc68 feat(security): cert tenant binding, stage 1 shadow mode
A chain-valid Intune MDM certificate only proves "issued to some tenant" —
the pinned roots are shared by every Intune tenant. Tenant scoping rested
entirely on the device serial in the downstream validators, which is not a
secret. Bind the tenant stamped into the auth cert to the requested tenant.

The Entra tenant id sits in OID 1.2.840.113556.5.14 of the cert the agent
already presents for mTLS, so this needs no Graph call, no consent and no
agent release. Not 5.6 (Intune Account id, a different GUID) and not 5.4
(device id, stored as 16 bare bytes) — the decoder handles both encodings
and now lives once in Shared, used by agent and backend.

Stage 1 observes only: ObserveCertTenantBinding returns void and swallows
its own exceptions, so it cannot fail an otherwise valid request. Match is
logged only on cert-cache miss, keeping the denominator without per-request
volume. Grep marker for stage 2: CERT-TENANT-BINDING-SHADOW.

Backend 4604/4604, Agent V2.Core 2519/2519, 0 warnings.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 16:03:05 +02:00
Oliver Kieselbach 6bd9139869 fix(agent): cert GUID parser accepts bare 16-byte extension values
Intune's MDM device cert stores the DeviceId in OID 1.2.840.113556.5.4 as a
bare 16-byte GUID with no ASN.1 wrapper, so the strict OCTET-STRING parser
bailed on the tag check and returned null. Verified against a live Intune
device cert: 5.4 is 16 raw bytes, while 5.6 / 5.11 / 5.14 carry the usual
04 10 wrapper.

- ParseGuidFromAsn1OctetString -> ParseGuidFromExtension (the old name no
  longer describes what it parses); both encodings now handled, with the
  shared LE-GUID construction pulled into ToGuid(). The forms cannot
  collide: a wrapped 16-byte GUID is always >= 18 bytes.
- Document the Intune MDM cert OID arc on the class: 5.4 DeviceId,
  5.6 Intune AccountId (NOT the Entra tenant), 5.14 Entra TenantId,
  5.11 undocumented GUID, 5.15/.16/.17/.18/.19/.23 INTEGER enums.
- 13 tests pinning both encodings plus the malformed-input rejections,
  including a bare GUID whose first byte is the 0x04 OCTET STRING tag.

No behaviour change for the MS-Organization-Access probe the agent uses;
TryGetTenantIdFromCert is untouched.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-24 14:41:00 +02:00
Oliver Kieselbach 73373a2c9e feat(agent): compiled-in fallback rule catalogs for the no-config default
The built-in default config shipped empty ImeLogPatterns/GatherRules lists, so a
session that got neither a live fetch nor a cached config ran with the entire IME
tracking pipeline dead — no app installs, no script results, no health-script
compliance, no IME ESP-phase signals — although the agent binary carries all 37
action handlers. The scalar knobs never drift (shared AgentConfigResponse property
initializers are also the backend's fallbacks), but the catalogs cannot come from
initializers.

BuiltInConfigCatalog embeds the SAME CI-combined rules/dist artifacts the backend
embeds and serves (ime-log-patterns.json, gather-rules.json) and feeds the enabled
entries into CreateDefaultConfig. Single source by construction: each agent release
snapshots the catalog state of its own build, so the offline fallback can never
drift further than the running agent version itself. A live fetch (startup,
post-registration recovery, rotate_config) always wins. Fail-soft: a load error
degrades to the previous empty-list behaviour.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 12:14:30 +02:00
Oliver Kieselbach 6d607c9045 fix(agent): network-link gate before initial config fetch + post-registration recovery
Root cause (tenant aebdce78 audit 2026-08-24, session ba828038): on boot-time
relaunches the agent starts before the Wi-Fi link is up. The Phase 4 config fetch
had no network gate (98f8114d only guarded Phase 6 registration), burned its
10/30/60s retry budget into a dead link and stranded the whole session on built-in
defaults (ConfigVersion=0) — up to ~50% of that tenant's sessions, with no
self-heal short of a manual rotate_config action. Not a version regression: rates
varied wildly per day within the same agent build; the failed requests never
reached the backend.

Two complementary layers:
- RemoteConfigService gates the initial fetch on the shared NetworkLinkWait poll
  (60s cap — longer than registration's 15s because the fetch runs first), so the
  retry budget starts on a live link. rotate_config stays ungated (latency-bound).
- Phase 6.5: after registration proves the backend reachable, TryRecoverConfigAsync
  refetches once and AgentRuntimeConfig.TryRecoverAfterRegistration re-applies the
  live config (same merge/integrity pipeline as Phase 4). LastFetch* keeps the
  original failure record; remote_config_fetch_failed reports it with a recovered
  flag at Info severity, agent_started flags remoteConfigRecovered.

Deliberately not re-run in recovery: kill-signal check (telemetry-channel control
signals cover post-registration kills) and endpoint migration (next start).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 11:53:04 +02:00
Oliver Kieselbach 3af94d9dcd infra(telemetry): DCR also drops host-emitted successful SignalR dependencies
Live-verified after 0d5e2572 deploy: the worker filter removes every 404/412/409
and worker SignalR row, but the SignalR output-binding calls are emitted by the
host process (SDK rdddsc) and only reachable at ingestion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 19:15:41 +02:00
Oliver Kieselbach 0d5e2572aa perf(telemetry): one canonical request record; drop expected storage outcomes and SignalR success deps
- RequestTelemetryMiddleware: SamplingPercentage=100 per item (worker copy was
  adaptively sampled, ItemCount 2-5 on weekdays, despite host.json excludedTypes)
- StorageDependencyFilterProcessor: also drop 404/412/409 storage outcomes
  (InProc + HTTP shape) and successful *.service.signalr.net REST calls
- infra/appinsights-workspace-transforms.bicep: DCR dropping the host HTTP
  request duplicate at ingestion (timer/queue rows kept)
- docs/backend/telemetry-ingest-shaping.md; skills updated

Measured baseline 7d: ~1.3 GB/week removed, requests now 100% with tenant context.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 18:31:32 +02:00
Oliver Kieselbach 793e907e60 fix(cveindex): merge findings per CVE before upsert; bounded parallelism
One CVE commonly appears under many findings of the same report (e.g. one
RD CVE across 14 installed RD components). Each occurrence produced its own
concurrent upsert on the same PK/RK, so the stored SoftwareName/OverallRisk
was race-dependent. Collapse to one entity per CVE (highest CVSS wins,
IsKev OR-ed, OverallRisk max) and write with Parallel.ForEachAsync(8) and
per-entity fail-soft logging instead of Task.WhenAll + one outer catch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 17:52:02 +02:00
Oliver Kieselbach 8e8bb66371 test(agent): thread-safe FakeSignalIngressSink (parked-tripwire CI flake)
Timer-thread Post() appended to a bare List while the test polled it in a
SpinWait foreach -> 'Collection was modified' (run 32644425162, no assert
text). Posted now returns a locked snapshot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 16:28:58 +02:00
Oliver Kieselbach 7e1389cf32 fix(ratelimit): keep history and lock in one cache entry; drop leaking per-key lock map
Both sliding-window limiters re-Set the cache entry without its eviction callback.
The replaced entry fired the callback asynchronously, removing the per-key lock while
a new one was already handed out, so two threads could mutate the same List under
different locks; the replacement entry never cleaned its lock, leaking one object per
key for the process lifetime.

The history now lives in a RequestBucket together with its own lock: same list ⇒ same
lock, no callback, no dictionary. Bucket creation is serialized (GetOrCreate is not
atomic) behind a lock-free TryGetValue fast path. The redundant re-Set is gone — the
lookup already refreshes the sliding expiration.

Concurrency tests (16 threads × 500 requests, one key) fail 3/3 on the old code and
pass on the new one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 16:26:44 +02:00
Oliver Kieselbach 3a89319472 feat(telemetry): startup duration metrics for backend and MCP
StartupTelemetryService emits BackendStartupMs (process start ->
ApplicationStarted) and BackendTableInitMs as App Insights custom
metrics with version/commit/tableInitFullPass/instance dimensions;
TableStorageService exposes LastInitialization. MCP server writes one
{"type":"boot","readyMs"} stderr line per start next to the
tool_call lines so cold-start time is queryable as a trend.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 16:04:31 +02:00
Oliver Kieselbach b37e6f975c perf(startup): table schema sentinel gates CreateTableIfNotExists pass
Cold start point-reads a SHA-256 over TableNames.All stored in
AdminConfiguration/SchemaSentinel/tables instead of issuing 66
CreateTableIfNotExists calls. Missing/stale sentinel runs the full
idempotent pass once and rewrites it. SessionsIndex startup backfill
only after a full pass, behind a conditional-insert claim (one instance
under scale-out); IsSessionIndexEmptyAsync no longer treats errors as
empty. Daily maintenance runs IStorageInitializer.EnsureAllAsync to
repair out-of-band table deletions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 15:33:45 +02:00
Oliver Kieselbach cacbd24667 feat(mcp): get_verdict_calibration response shaping (explicit nulls, minSharePct/top, omitted report)
Drop per-row window7/baseline28.sessions (duplicated trend denominators), make
reEnrollRatePct/lift always present with null = withheld, add minSharePct/top
tail trimming that never drops override-carrying rows and reports what was
omitted. Backend wire shape and admin page untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 12:51:17 +02:00
Oliver Kieselbach a157d98a40 fix(radar): verdict share regression only for backend-decided paths
First production pass alerted on agent:whiteglove_pending (two tenants in a
WhiteGlove rollout week) and agent:complete_soft (a Continue-Anyway tenant) —
customer workflow mix, not classifier drift. agent:* and register:* are now
ineligible for share_regression (still in the matrix and the silence group);
episodes opened by the broader gate re-arm on the next pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 12:32:00 +02:00
Oliver Kieselbach 7fa50d09c1 perf(maintenance): one projected 35d Sessions scan shared by all rolling sweeps
The cross-tenant StartedAt-only date-range query is a full-table drain in
Table Storage; time attribution, device journeys, verdict calibration and the
calibration radar's tenant discovery each ran their own — four identical reads
per 2h tick. RunAllAsync / RunManualAsync now load the window once through
GetMaintenanceWindowSessionsAsync (MaintenanceSweepSessionProjection: the union
of the sweeps' columns, drops FailureSnapshotJson/geo/OS/diagnostics) and the
sweeps slice it by StartedAt. MaintenanceSweepProjectionEquivalenceTests pins
that every consumed field survives the projection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 12:19:41 +02:00
Oliver Kieselbach 94afc02e62 docs(okf): verdict calibration concept (VerdictPath, prior verdicts, aggregate, radar)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:40:34 +02:00
Oliver Kieselbach 5e4c5e80be feat: verdict calibration drift radar (share / silence / evidence-gap) with ops events
Applies the rule-regression statistics (7d vs 28d, lift >=2, Wilson-separated,
small-n gates) to the verdict-calibration rows: per-path share regression,
sweep+maxlife silence-share regression (agent liveness signal) and an
absolute r6 evidence-gap gate. Episodes live in the notification tracker
(verdictcalibration|kind|path|status), fire one VerdictCalibrationDrift ops
event (operator-only, no tenant bell), re-arm under 1.5x / 15%, and surface as
alerts[] + a Drift pill in the calibration matrix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:39:06 +02:00
Oliver Kieselbach b535973531 feat: verdict calibration aggregate, GA endpoint, MCP tool and operator matrix
Daily (tenant x StartedAt date) buckets per verdict path: count, derived
attribution, 7-day re-enrollment proxy (eligible/re-enrolled), and the
correction stream attributed to the PRIOR path (admin / late completion /
other). Written by the 2h maintenance sweep after the device-journey pass,
180d retention, regenerable. Read via GET global/metrics/verdict-calibration
(GlobalReadOrAdmin, 7d-vs-28d share trend, rates withheld below n=20),
the get_verdict_calibration MCP tool, and the admin metrics section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:22:17 +02:00
Oliver Kieselbach 7e0c578a59 feat(backend): VerdictPath + PriorStatus/PriorVerdictPath on every session status write
Every status writer now declares the code path that produced the status
(VerdictPaths vocabulary, stamped for all statuses — unlike FailureSource
which is Failed-only), the timeout classifier returns its rule id, and a
write that overrides a prior verdict preserves it as PriorStatus/
PriorVerdictPath (admin mark, late completion, retro). Pre-instrumentation
rows derive a path read-side (VerdictPathDerivation, legacy:* when
ambiguous). Foundation for the verdict calibration aggregate + radar.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:06:37 +02:00
103 changed files with 6684 additions and 254 deletions
+127
View File
@@ -0,0 +1,127 @@
---
type: Concept
title: Client-Certificate Tenant Binding — Closing the Cross-Tenant Gap
description: Why a chain-valid Intune MDM certificate does not prove the caller belongs to the tenant it claims, which certificate extension carries the Entra tenant id (and which two look like it but do not), and the shadow-then-enforce rollout that measures field coverage before it can lock devices out.
resource: src/Backend/AutopilotMonitor.Functions/Security/CertTenantBinding.cs
tags:
- backend
- security
- authentication
- certificates
- intune
- multi-tenant
timestamp: 2026-08-24T00:00:00+02:00
---
# Problem
Agent requests authenticate with mTLS. `CertificateValidator` pins the chain to the embedded
Intune roots (`X509ChainTrustMode.CustomRootTrust`), checks the validity window and requires the
Client-Authentication EKU. All of that together proves exactly one thing: **the certificate was
issued by the Microsoft Intune MDM Device CA to some tenant**.
Those roots are shared by every Intune tenant on the planet. Nothing in the certificate stage ties
the caller to the tenant in the request. The tenant scoping came entirely from the downstream
device validators — the serial number must appear in that tenant's Autopilot / imported-device /
Cloud PC inventory. Device serial numbers are not secrets; they are printed on the chassis.
So the pre-existing attack was: enroll a device in your own Intune tenant, obtain a legitimate
device certificate, learn one serial number of the victim tenant, and submit telemetry as that
tenant. Certificate validation passes, device validation passes.
# Mechanism
The Intune MDM Device CA stamps the customer's identifiers into the leaf certificate. Since the
agent presents this certificate in the TLS handshake, its contents are proof of possession and the
backend can compare them against the requested tenant — no Graph call, no additional consent, and
nothing the agent has to send.
## Which extension carries what
Verified by decoding a real field certificate (issued 2026-05-08, `Functions.Tests/device-cert-sample.pem`):
| OID | Encoding | Content |
| --- | --- | --- |
| `1.2.840.113556.5.14` | nested OCTET STRING (`04 10` + 16 bytes) | **Entra tenant id** — the one to compare |
| `1.2.840.113556.5.6` | nested OCTET STRING (`04 10` + 16 bytes) | Intune **Account** id — a *different* GUID, never equal to the tenant id |
| `1.2.840.113556.5.4` | **16 raw bytes, no wrapper** | Intune device id, identical to the certificate's Subject CN |
Two traps are worth stating explicitly, because both cost real debugging time:
1. `…5.6` is widely described as "the tenant id (AccountID)". It is not the Entra tenant id.
Comparing it against the requested tenant would reject every request in production.
2. The encodings are not uniform. `…5.4` holds the GUID bare while the other two nest it in a
second OCTET STRING, so a decoder written against one shape silently fails on the other.
`MsDeviceCertificateOids.TryParseGuid` accepts both, and all GUIDs are Microsoft
little-endian (what `new Guid(byte[])` expects).
A 16-byte input is inherently ambiguous — it is indistinguishable from a truncated wrapper and is
always read as a bare GUID. That is harmless here: the value is only ever compared for equality
against a known tenant id, never trusted as a standalone claim.
## Where it is NOT
`EntraDeviceCertHelper` reads tenant and device ids from the **MS-Organization-Access** certificate
(`1.2.840.113556.1.5.284.*`). That certificate is not part of the authentication path at all — the
agent selects the Intune MDM certificate by exact issuer match (`CN=Microsoft Intune MDM Device CA`,
see `CertificateHelper.SelectMdmCertificate`) and the backend pins only the MDM chain. The Entra
certificate is used purely as a local fallback for `TenantIdResolver`; it never leaves the device.
Sending it in a header instead would prove nothing: only the certificate used in the TLS handshake
demonstrates possession of a private key. A certificate copied into a custom header can be replayed
by anyone who has seen it once.
# Rollout
Enforcing immediately would be reckless: any device population whose certificates predate the
extension, or whose CA behaves differently (sovereign clouds), would be locked out at once. The
rollout therefore has staged consequences.
**Stage 1 — shadow (current).** `SecurityValidator.ObserveCertTenantBinding` evaluates the binding
after certificate validation and logs the outcome. It returns `void`, changes no state, and swallows
its own exceptions, so it cannot fail a request that would otherwise be accepted. The pure comparison
lives in `CertTenantBinding.Evaluate`; the outcome codes (`Match`, `Mismatch`, `ExtensionMissing`,
`Unparseable`, `RequestTenantNotAGuid`) are stable and queried by exact match in KQL.
`WouldRejectUnderEnforcement` records what stage 2 *would* have done, so the telemetry can be read
without re-deriving the rule at query time.
Volume control: the outcomes that inform the decision (mismatch, missing, undecodable) are always
logged, because they are expected to be rare and each is individually interesting. The bulk `Match`
case is logged only when the certificate was freshly validated rather than served from the 5-minute
thumbprint cache (`CertificateValidationResult.FromCache`). Both numerator and denominator are
therefore counted per certificate-window rather than per request, which keeps the ratio meaningful
while cutting the volume by roughly the request-per-device factor.
**Stage 2 — enforce.** Once telemetry shows the `ExtensionMissing` rate is negligible, `Mismatch`
becomes a rejection. The decision that still has to be made is the policy for certificates without
the extension: skip with a warning, or reject. Grep marker for every stage-2 site:
`CERT-TENANT-BINDING-SHADOW`.
The bootstrap-token path (`X-Bootstrap-Token`) bypasses certificate validation by design and is
unaffected by either stage.
# Examples
Read the shadow telemetry:
```kusto
traces
| where message has "AgentCertTenantBinding"
| extend Outcome = tostring(customDimensions.Outcome),
Tenant = tostring(customDimensions.TenantId)
| summarize count() by Outcome, Tenant
```
Anything in `Mismatch` is a cross-tenant certificate; anything in `ExtensionMissing` is a device
that stage-2 enforcement would have to make a policy decision about.
# Citations
* `src/Shared/AutopilotMonitor.Shared/Security/MsDeviceCertificateOids.cs` — OID catalog and the
shared ASN.1 GUID decoder (one implementation for agent and backend).
* `src/Backend/AutopilotMonitor.Functions/Security/CertificateValidator.cs` — extension extraction
alongside chain validation, cached per thumbprint.
* `src/Backend/AutopilotMonitor.Functions/Security/CertTenantBinding.cs` — the pure comparison.
* `src/Backend/AutopilotMonitor.Functions/Security/SecurityValidator.cs` — shadow observation.
* [W365 Cloud PC Device Validation](cloudpc-device-validation.md) — the other consumer of this
certificate's identity fields (Subject CN = Intune device id).
+2 -1
View File
@@ -78,7 +78,8 @@ Durations shown for attempts are the sessions' authoritative `DurationSeconds` v
completed journeys, first-time-right count, attempt histogram, junk-serial exclusion
count, `JourneyVersion`. A journey buckets on the StartedAt date of its completing
success session (the platform's StartedAt-date convention; late terminals converge via
the sweep). All counts are **additive**, so a window rate is the ratio of summed daily
the sweep, which slices the tick's shared projected window scan — see
`time-attribution.md`). All counts are **additive**, so a window rate is the ratio of summed daily
rows — there is no rolling-window row (unlike the median-based time-attribution
aggregates, which cannot merge). 180-day retention. Rows exist below the ≥20 UI gate
on purpose (the UI needs the n for "insufficient data (n=…)").
+67
View File
@@ -0,0 +1,67 @@
---
type: Concept
title: Table Schema Sentinel (startup table initialization)
description: Why a cold start no longer issues one CreateTableIfNotExists per registered table - a hash over TableNames.All stored in AdminConfiguration gates the full pass, the SessionsIndex backfill only runs after a full pass and behind a conditional-insert claim, and daily maintenance repairs out-of-band deletions.
resource: /src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.cs
tags:
- backend
- table-storage
- startup
- maintenance
timestamp: 2026-08-23
---
# Schema
## Problem
`TableInitializerService` (hosted service) called `InitializeTablesAsync` on every process start, which issued `CreateTableIfNotExistsAsync` for every entry of `Constants.TableNames.All` (66 at the time of writing, DoP 8). Each call is a Create POST whose 409 is swallowed - on Flex Consumption with frequent cold starts that was ~9 serial round-trip waves before the first request. It was followed by a SessionsIndex emptiness query whose *error* path returned "empty", so a transient storage error triggered a full Sessions scan with serial upserts - on every scaled-out instance at once.
## Sentinel
* Row `AdminConfiguration` / PK `SchemaSentinel` / RK `tables`, property `TableSchemaHash` (+ `TableCount`, `UpdatedUtc`).
* Value = `ComputeTableSchemaHash(TableNames.All)`: SHA-256 over the ordinal-sorted, newline-joined table names. Derived, never hand-maintained: adding a table to `TableNames.All` changes the hash by construction (`TableLifecycleBucketTests` already forces every constant into `All`).
* `InitializeTablesAsync` (returns `bool fullPassRan`):
1. Point-read the sentinel. Match -> mark initialized, **zero** table calls, return `false`.
2. Missing / stale / unreadable (fresh storage has no AdminConfiguration table yet) -> `EnsureAllTablesAsync` (the old full pass), then upsert the sentinel **only if no table failed**, return `true`. A failed pass leaves `_tablesInitialized = false`, so the next call retries.
* Scale-out: two instances that both see a stale sentinel both run the idempotent pass - harmless.
## SessionsIndex backfill
* Only evaluated when the full pass ran (`fullPassRan == true`): an empty index can only exist on fresh storage, which is exactly the case without a sentinel.
* `IsSessionIndexEmptyAsync` now returns `false` on error (skip), not `true`.
* `TryClaimSessionIndexBackfillAsync`: conditional `AddEntityAsync` on `SchemaSentinel` / `sessionIndexBackfill` - only the winner scans; 409 or any error -> skip. The manual maintenance backfill (`RunManualAsync`) remains the safety net.
## Known gap and repair
A table deleted out-of-band (cleanup skill, portal, tests) is not noticed at startup anymore. `IStorageInitializer.EnsureAllAsync` runs as the first step of the daily timer maintenance (`MaintenanceService.RunAllAsync`), recreating it within 24h. Storage helpers are fail-soft in the meantime.
## Measuring it
`StartupTelemetryService` (hosted service) emits two App Insights metrics on `IHostApplicationLifetime.ApplicationStarted` - i.e. after every hosted service finished `StartAsync`:
* `BackendStartupMs` - process start (`Process.StartTime`) to host ready; includes runtime, DI and all hosted services.
* `BackendTableInitMs` - the `InitializeTablesAsync` slice alone (`TableStorageService.LastInitialization`).
Dimensions: `version`, `commit`, `tableInitFullPass`, `instance`. Metrics bypass adaptive sampling and the worker log level, so they arrive even though Information traces do not. The MCP server logs the same idea as one `{"type":"boot","readyMs":…}` stderr line per start (ContainerAppConsoleLogs_CL, see `.claude/commands/mcp-telemetry.md`).
```kusto
customMetrics
| where name in ("BackendStartupMs", "BackendTableInitMs")
| summarize p50 = percentile(value, 50), p95 = percentile(value, 95), n = count()
by name, version = tostring(customDimensions.version), fullPass = tostring(customDimensions.tableInitFullPass)
```
# Examples
* Deploy that adds a table: first cold start after the deploy runs the full pass once (log: "Table schema sentinel missing or stale"), every later start logs "Table schema sentinel matches (...)".
* Fresh storage account: sentinel read throws TableNotFound -> full pass -> index empty -> one instance claims and backfills.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.cs` - sentinel, `InitializeTablesAsync`, `EnsureAllTablesAsync`, `TryClaimSessionIndexBackfillAsync`
* `src/Backend/AutopilotMonitor.Functions/Services/TableInitializerService.cs` - startup gating
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs` - daily `EnsureAllTablesAsync`
* `src/Backend/AutopilotMonitor.Functions/Services/StartupTelemetryService.cs` - `BackendStartupMs` / `BackendTableInitMs`
* `src/McpServer/autopilot-monitor-mcp/src/index.ts` - `{"type":"boot"}` line
* `src/Backend/AutopilotMonitor.Functions.Tests/TableSchemaSentinelTests.cs`, `StartupTelemetryServiceTests.cs`
+89
View File
@@ -0,0 +1,89 @@
---
type: Concept
title: Telemetry Ingest Shaping (App Insights / Log Analytics)
description: What the backend deliberately keeps out of Application Insights and why - one canonical request record (worker copy, per-item sampling bypass; host HTTP duplicate dropped by a workspace transformation), expected storage outcomes (404/412/409) and successful SignalR REST calls filtered in the worker, and the query rules that follow.
resource: /src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs
tags:
- backend
- telemetry
- application-insights
- cost
timestamp: 2026-08-23
---
# Telemetry Ingest Shaping
The backend is storage-I/O heavy and runs on the Functions isolated worker, which emits telemetry from TWO processes (host and worker). Left alone, Log Analytics ingestion is dominated by rows nobody reads. This document records what is removed, where, and what that means for queries.
# Schema
## Request telemetry — one canonical record
| Source | Emitted by | Content | Fate |
|---|---|---|---|
| Host HTTP request (`Properties.Source` empty, `Url` set) | Functions host process | Name, duration, status — no tenant/user/correlation | **Dropped at ingestion** by the workspace transformation on `AppRequests` (`infra/appinsights-workspace-transforms.bicep`) |
| Host non-HTTP invocation (`Url` empty) | Functions host process | Timer / queue executions | Kept (the worker middleware never sees them) |
| Worker request (`Properties.Source == 'WorkerMiddleware'`) | `RequestTelemetryMiddleware` | TenantId, UserId, UserRole, CorrelationId, ClientSource, McpToolName, HttpPath | **Kept, unsampled** — the canonical record |
Why two mechanisms:
* The host request item cannot be suppressed from worker code or from `host.json` (`Host.Results` controls log traces, not the request item; verified 2026-06-09). The only lever is a Log Analytics workspace transformation: `source | where tostring(Properties.Source) == 'WorkerMiddleware' or isempty(Url)`.
* `host.json` `samplingSettings.excludedTypes: "Request;Event"` only applies to the host process. The worker's `AddApplicationInsightsTelemetryWorkerService` runs its own adaptive sampling; before 2026-08-23 worker request rows carried `ItemCount` 2–5 on weekdays (≈30 % of individual requests missing, `count()` undercounting by the same share — `sum(ItemCount)` matched the host copy to 1.00 per day). Re-configuring the worker sampling chain did not take effect in the isolated worker (see the L4 note in `Program.cs`). The per-item opt-out does: the middleware sets `SamplingPercentage = 100` on its own `RequestTelemetry`, and the SDK sampling processor passes through every item whose sampling percentage is already set.
## Dependency telemetry — keep only signal
`StorageDependencyFilterProcessor` (worker pipeline) drops:
* successful Azure Storage calls (Table/Queue/Blob) in both shapes — the HTTP shape (`Azure table`, target `*.table.core.windows.net`) and the Azure SDK ActivitySource shape (`InProc | Microsoft.Tables`, `InProc | Microsoft.Storage`);
* storage calls with an **expected outcome**: `404` (point-read miss — EventTypeIndex, BlockedDevices lookups), `412` (ETag precondition), `409` (idempotent insert conflict). HTTP-shape rows carry the code in `ResultCode`; InProc rows only in `Properties["Error"]` (`Status: 404 (`). Before this filter these were 100 % of the remaining "failed" storage rows (~540 MB/week, two rows per call);
* successful Azure SignalR Service REST calls the worker makes itself (`*.service.signalr.net`).
The bulk of SignalR REST traffic (group add/remove per connection, ~470 MB/week) is emitted by the **host** process for the SignalR output binding (`SDKVersion rdddsc:*`) and never passes the worker pipeline — live-verified after the 2026-08-23 deploy (worker rows: 0, host rows: ~560/h). Those are removed by the second data flow of the same workspace transformation: `AppDependencies | where not(Success == true and Target endswith '.service.signalr.net')`.
Everything else passes: storage 429/5xx/auth/timeouts, failed SignalR calls, all HTTP/Graph/NVD/MSRC dependencies, the worker's own `InProc Invoke` span, and every non-dependency item.
## Measured baseline (7 days before the change, `_BilledSize`, `autopilotmonitor-api-eu`)
| Table | Rows | MB | After |
|---|---|---|---|
| AppDependencies | 976k | 1248 | ≈ 240 (SignalR + expected-outcome rows gone) |
| AppRequests host copy | 274k | 323 | ≈ 0 (only timer/queue rows) |
| AppRequests worker copy | 189k | 208 | ≈ 300 (now 100 % of requests) |
| AppTraces | 142k | 137 | unchanged |
| AppEvents | 6k | 4 | unchanged — not worth touching |
Net ≈ −1.3 GB/week with MORE usable data (every request now has tenant context).
# Examples
Counting requests (exact since 2026-08-23; for older data replace `count()` with `sum(itemCount)`):
```kql
requests
| where customDimensions.Source == 'WorkerMiddleware'
| summarize total=count(), failed=countif(success == false) by name
```
Real storage trouble only:
```kql
dependencies
| where type startswith 'InProc | Microsoft.Tables' or target endswith '.table.core.windows.net'
| summarize count() by resultCode, tostring(customDimensions.Error)
```
Post-deploy verification of the per-item sampling bypass (must be `ItemCount == 1` only):
```kql
AppRequests
| where TimeGenerated > ago(1h) and tostring(Properties.Source) == 'WorkerMiddleware'
| summarize rows=count() by ItemCount
```
# Citations
* `src/Backend/AutopilotMonitor.Functions/Middleware/RequestTelemetryMiddleware.cs` — `SamplingPercentage = 100` before `TrackRequest`.
* `src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs` + `StorageDependencyFilterProcessorTests.cs` — the filter contract.
* `src/Backend/AutopilotMonitor.Functions/Program.cs` — the failed L4 worker-sampling attempt and why it must not be retried.
* `infra/appinsights-workspace-transforms.bicep` — the AppRequests transformation DCR, apply/verify/rollback commands.
* `.claude/commands/backend-logs.md`, `session-debug.md` — query conventions.
+5
View File
@@ -105,6 +105,11 @@ the partition).
* Self-healing 30-day maintenance sweep (2h tick): backfills missing breakdowns, folds
late-terminating sessions into their StartedAt-date aggregates, and enforces version
purity — a stale-version row whose events aged out counts as missing, never mixed.
Reads the tick's **shared window scan** (`MaintenanceService.LoadSweepWindowSessionsAsync`,
one projected 35-day cross-tenant drain via `MaintenanceSweepSessionProjection` that
also feeds the device-journey and verdict-calibration sweeps) — the StartedAt-only
filter is a full-table read in Table Storage, so it is done once per tick, not once per
sweep; `MaintenanceSweepProjectionEquivalenceTests` pins the column set.
* `TimeAttributionAggregates` (PK = TenantId or `global`, RK = `{yyyy-MM-dd}|{class}` and
`rolling30|{class}`): per enrollment class (`user_driven`, `whiteglove`, `self_deploying`,
`device_preparation` — never mixed) median/p75/p90 per segment over the fixed six-segment
+164
View File
@@ -0,0 +1,164 @@
---
type: Concept
title: Verdict Calibration (classifier thermometer)
description: How every session status write declares its VerdictPath, how overriding a verdict preserves it as PriorStatus/PriorVerdictPath, how the daily calibration aggregate counts paths with the re-enrollment proxy and the correction stream, and how the drift radar (share / silence / evidence-gap) turns that into operator-only ops events.
resource: src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.VerdictCalibration.cs
tags:
- backend
- sessions
- classifier
- calibration
- regression-radar
timestamp: 2026-08-23T18:00:00+02:00
---
# Verdict Calibration (classifier thermometer)
A session's terminal status is written by ~20 code paths: the agent's own terminal events,
the silence classifier (`EnrollmentTimeoutClassifier`, rules 1–6) reached via the
maintenance sweep / the agent's max-lifetime shutdown / late-telemetry heal / retro
reclassification, the analyze-rule engine, admin marks and session superseding. Verdict
calibration makes every one of those paths **countable**, tracks how often a path's verdict
was later **overridden**, attaches the delayed **re-enrollment proxy**, and raises an
operator-only alarm when a path's share **drifts**. It is deterministic, fact-based tuning
input for the rules — explicitly not a learned model, and never a customer-facing feature.
# Schema
## VerdictPath on the session row
`SessionSummary.VerdictPath` (`VerdictPaths` vocabulary, `origin:detail`, append-only) is
stamped by **every** status write — `UpdateSessionStatusAsync` takes it as a required
parameter, `StoreSessionAsync` stamps `register:new` / `register:whiteglove_resume` and
preserves an existing path through the re-registration Replace, `SetSessionPreProvisionedAsync`
requires it whenever it writes a status. Unlike `FailureSource` (persisted only on `Failed`),
the path survives for Incomplete, AwaitingUser, Stalled and Pending.
| Origin | Paths |
| --- | --- |
| `agent` | `complete`, `complete_soft` (Continue-Anyway), `failed`, `esp_failure_fallback`, `gather_complete`, `whiteglove_pending`, `whiteglove_resumed`, `stall_probe`, `stall_heal` |
| `ingest` | `wg_awaiting` (stall probe mapped to WhiteGlove AwaitingUser) |
| `sweep` | `stalled`, `wg_awaiting`, `sd_reconcile`, `r{rule}` (stage-2 classifier) |
| `maxlife` / `late` / `retro` | `r{rule}` (classifier via max-lifetime shutdown / late-telemetry heal / retro reclassification); `retro:sd_reconcile`, `retro:superseded` |
| `register` | `new`, `whiteglove_resume`, `superseded` |
| `rule` | `rule:{ruleId}` (analyze rule with MarkSessionAsFailed) |
| `manual` | `failed`, `succeeded` |
| `legacy` | read-side derivation only (`legacy:{rule}` when the rule is recoverable from the reason literal, `legacy:unknown` otherwise) — never written |
Classifier rule ids (`ClassifierRules`): `r1`, `r1b_awaiting`, `r1b_succeeded`, `r1c`, `r2`,
`r3`, `r4`, `r5_awaiting`, `r5_incomplete`, `r6`. `ClassifyTimedOutSession` returns
`(Status, Reason, Rule)` so each `return` names its rule.
## PriorStatus / PriorVerdictPath — the correction stream
When a write replaces a prior **verdict** (existing status ∈ Succeeded/Failed/Incomplete/
AwaitingUser with a stamped path, incoming status different), the storage seam preserves the
pair (`TableStorageService.ComputePriorVerdict`, both the ETag path and the force path). A
stall marker healing to InProgress, a same-status refresh, or an unstamped legacy row never
sets it. Admin marks, late agent completions and retro reclassifications all flow through it.
## Read-side derivation for pre-instrumentation rows
`VerdictPathDerivation.Derive(session)` returns the stamped path verbatim, otherwise derives
from `AdminMarkedAction`, `FailureSource`, `ReconcileReason` and `FailureReason` using the
writers' fixed literals (classifier rule prefixes, the `max-lifetime watchdog` / `Retro-
reclassified` suffixes, `Rule: `, `Superseded by session`, `Late completion report received`).
Unambiguous rows get the real path (`agent:complete`, `manual:failed`, `rule:…`,
`maxlife:r6`), ambiguous-but-rule-known rows `legacy:{rule}`, the rest `legacy:unknown`. No
mass backfill: the aggregate counts derived rows with `DerivedCount` so the matrix shows
history while flagging its weaker evidence. Tests pin the literals against the classifier
output so a wording change fails loudly.
## Daily aggregate — `VerdictCalibrationAggregates`
PK = `TenantId` | `"global"`, RK = `yyyy-MM-dd` (session **StartedAt** date, like every daily
aggregate). `VerdictCalibrationDailyAggregate { Version, SessionCount, TerminalSessionCount,
Buckets[], ComputedAt }` with one bucket per (VerdictPath, Status):
| Field | Meaning |
| --- | --- |
| `Count` / `DerivedCount` | sessions currently on the path; of those, derived read-side |
| `Eligible7d` | terminal sessions whose end (`CompletedAt ?? LastEventAt ?? StartedAt`) lies ≥ 7 d before compute time |
| `ReEnrolled7d` | of the eligible: the device's `DeviceHistories` chain shows the next terminal session starting < 7 d after this one's end |
| `OverriddenByAdmin` / `OverriddenByLateCompletion` / `OverriddenOther` | sessions whose **PriorVerdictPath** is this bucket — attributed to the path that was overridden, so a bucket can carry overrides with `Count = 0` |
`MaintenanceService.SweepVerdictCalibrationAsync` (2-hourly, after the device-journey sweep
so the chains are fresh; manual maintenance parity; kill switch
`VerdictCalibrationSweepDisabled`) slices the tick's shared projected window scan
(`LoadSweepWindowSessionsAsync` — one 35-day cross-tenant drain feeds the time-attribution,
device-journey and calibration sweeps plus the radar's tenant discovery; the StartedAt-only
filter is a full-table read, so never per sweep), recomputes the rolling 30-day window whole (Replace),
deletes stale date rows it did not regenerate, and the unbounded-tables cleanup applies the
180-day retention. Sessions inside a deletion cascade are skipped; the table is regenerable
and is wiped with the tenant on offboarding.
## Read surfaces
* `GET global/metrics/verdict-calibration?days=&tenantId=` (GlobalReadOrAdmin,
TenantScoping.QueryParam; `VerdictCalibrationResponse.Build`): per path — count, share of
all window sessions, derived count, eligible/re-enrolled with `reEnrollRatePct` **null below
20 eligible**, the three override counters, a today-anchored 7d-vs-28d share trend
(`window7`, `baseline28`, `lift` null without baseline) — plus totals, the trend basis and
the active `alerts[]`. Empty but well-formed before the first sweep.
* MCP `get_verdict_calibration` (platform scope: GA + Global Reader). Same endpoint, LLM-shaped in
`verdict-calibration-shape.ts`: per-row `window7.sessions` / `baseline28.sessions` are dropped (the
denominators live once in `trend`), `reEnrollRatePct` / `lift` are always present (`null` = withheld),
and `minSharePct` / `top` trim the one-session tail — rows carrying overrides are never trimmed and
everything dropped is reported in `omitted`. The wire shape the admin page consumes is untouched.
* Portal: Global Admin → Metrics → Verdict Calibration (`/admin/metrics/verdict-calibration`):
matrix grouped by origin, status pill, `derived n` gray pill, trend glyph (arrow only from
5 window hits and lift ≥ 2 / ≤ 0.5), re-enrollment rate or `— (n=…)`, overrides
`admin / late / other`, `↑ Drift` pill on rows with an active episode, and the active
episodes list.
## Drift radar — `VerdictCalibrationRadar`
Same statistics as the [rule regression radar](rule-regression-radar.md) (7-day window vs the
prior 28 days, ≥ 5 window hits, ≥ 20 window sessions, lift ≥ 2, `MetricsMath.
RateIncreaseSeparated`), anchored on yesterday, over every partition with rows (each tenant
and `global`):
| Kind | Signal | Re-arm |
| --- | --- | --- |
| `share_regression` | one **backend-decided** path's share of all sessions doubled (`sweep`/`maxlife`/`late`/`retro`/`rule`/`manual`/`ingest`; `agent:*` and `register:*` mirror customer workflow mix — first prod pass fired on `agent:whiteglove_pending` rollout weeks — and `legacy:*` is derived, all excluded) | share < 1.5× baseline, the path stops occurring, or the path is no longer eligible |
| `silence_share_regression` | the `sweep:*` + `maxlife:*` share doubled — the backend had to decide more often because the agent went silent (a liveness signal, not a classifier one) | same |
| `evidence_gap` | absolute: `r6` (pure fallthrough) decides ≥ 20 % of the window's classifier verdicts, ≥ 20 verdicts | < 15 % or < 20 verdicts |
Episodes are tracker rows `verdictcalibration|{kind}|{path}|{status}` (register = `AddEntity`,
409 = already burning, fail-closed; refresh carries `FirstNotifiedAt`; 30-day tracker
retention re-arms long burners). A new episode fires **one** `VerdictCalibrationDrift` ops
event (category Maintenance, Warning; dual-registered in the web `OPS_EVENT_TYPES`) — there
is deliberately **no tenant bell**. Per-path findings on a tenant partition get the on-fire
dimension correlation (`RuleRegressionRadar.ComputeDimensionConcentration` over the window's
sessions whose derived path matches), reported with the shared wording contract "correlated,
not necessarily causal"; group kinds and the global partition carry no dimension claim.
Kill switch `VerdictCalibrationRadarDisabled`.
# Examples
* **Reading the thermometer (prod, 2026-08-23 hand analysis that motivated this):**
re-enrollment within 7 days after Succeeded 8.9 % (lab background), after Failed 24.3 %,
after Incomplete 10.6 % — Incomplete behaves like Succeeded, so rules 5/6 are probably too
cautious; the matrix now shows this per rule instead of per status.
* **Correction stream:** a session swept to `sweep:r5_incomplete`, later completed by the
agent → row `agent:complete` with `PriorVerdictPath = sweep:r5_incomplete`; the aggregate
increments `OverriddenByLateCompletion` on the `sweep:r5_incomplete / Incomplete` bucket.
Four admin marks since launch were invisible before — now each one counts against the path
it corrected.
* **Evidence gap:** a tenant whose agent build stopped emitting the Device Setup rollup line
drives its sweep verdicts into `r6`; the absolute gate fires at 20 % and names the
tenant — the fix is agent-side, the classifier is only the messenger.
# Citations
* `src/Shared/AutopilotMonitor.Shared/Models/VerdictPaths.cs` — vocabulary + `ClassifierRules`
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Sessions.cs` — `ComputePriorVerdict`, `ApplyVerdictPath`, re-registration preserve
* `src/Backend/AutopilotMonitor.Functions/Helpers/VerdictPathDerivation.cs`
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.VerdictCalibration.cs` — sweep + `BuildVerdictCalibrationAggregates`
* `src/Backend/AutopilotMonitor.Functions/Functions/Metrics/GetVerdictCalibrationFunction.cs`
* `src/Backend/AutopilotMonitor.Functions/Helpers/VerdictCalibrationRadar.cs`, `Services/MaintenanceService.VerdictCalibrationRadar.cs`
* `src/McpServer/autopilot-monitor-mcp/src/tools/admin.ts` — `get_verdict_calibration`
* `src/McpServer/autopilot-monitor-mcp/src/verdict-calibration-shape.ts` — LLM-facing response shaping (`minSharePct` / `top`, explicit nulls)
* `src/Web/autopilot-monitor-web/app/admin/metrics/sections/SectionVerdictCalibration.tsx`
* Tests: `VerdictPathTests`, `VerdictCalibrationTests`, `VerdictCalibrationRadarTests`
* Related: [Silent-Session Classification](silent-session-classification.md), [Device Journeys](device-journeys.md), [Rule Regression Radar](rule-regression-radar.md)
+4
View File
@@ -45,6 +45,8 @@ https://docs.autopilotmonitor.com (separate repository).
# Backend
* [Telemetry Ingest Shaping](backend/telemetry-ingest-shaping.md) - One canonical App Insights request record (worker copy with tenant context, per-item sampling bypass; host HTTP duplicate dropped by a workspace transformation DCR), expected storage outcomes 404/412/409 and successful SignalR REST calls filtered in the worker, and the count()/sum(itemCount) query rules that follow.
* [Table Schema Sentinel](backend/table-schema-sentinel.md) - Startup no longer issues one CreateTableIfNotExists per table: a SHA-256 over TableNames.All in AdminConfiguration gates the full pass (derived, never hand-maintained), the SessionsIndex backfill runs only after a full pass behind a conditional-insert claim, and daily maintenance repairs out-of-band deletions.
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
* [Send-Time Frame Separation](backend/send-time-frame-separation.md) - One device-clock send timestamp per upload batch (X-Send-Time-Utc → SentAt on Events) splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset; the clock_skew condition measures SentAt batches directly (spool-immune), the CMTrace tripwire deliberately stays on its differential math.
* [OOBE Bootstrap go-Code Flow](backend/oobe-bootstrap-go-flow.md) - The bootstrap short-code lifecycle (portal-created session → `irm go.autopilotmonitor.com/CODE | iex` at Shift+F10 → GetBootstrapScriptFunction), the always-HTTP-200 text/plain contract, the two-line injection defense (GUID/URL/expiry validator + single-quoted PS literals, golden-pinned against the legacy TS template), and the per-IP rate-limit story behind Front Door.
@@ -53,6 +55,7 @@ https://docs.autopilotmonitor.com (separate repository).
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
* [Attempt-Scoped App Install Durations](backend/app-install-duration-attempt.md) - AppInstallSummary.DurationSeconds = status-defining (last) install attempt via the LastAttemptStartedAt max-fold anchor instead of the first-observation→last-terminal span that billed multi-pass IME gaps to trivial apps; InstallPassCount, the weaker-terminal guard, the radar's cutover gate, and time attribution's union-of-attempt-segments occupancy.
* [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract.
* [Verdict Calibration](backend/verdict-calibration.md) - How every session status write declares its VerdictPath (origin:detail vocabulary incl. classifier rule ids), how overriding a verdict preserves PriorStatus/PriorVerdictPath as the correction stream, the daily per-path calibration aggregate with the 7-day re-enrollment proxy and read-side legacy derivation, the GA endpoint / MCP tool / admin matrix, and the drift radar (share, silence-share, r6 evidence gap) with its operator-only ops events.
* [App Version Duration Regression Radar](backend/app-version-regression-radar.md) - How app versions whose median install duration regressed vs. the previous version are detected (first-seen-ordered versions, median lift ≥2× plus a 5-minute absolute floor over measured installs only), how alert episodes live in the appversionregression tracker keyspace with drain/median re-arm semantics, and the bell/ops-event/analytics surfaces.
* [Plan Downgrade Policy & the Retention Grace Period](backend/plan-downgrade-retention-grace.md) - What happens when a tenant loses Pro: every entitlement gates read-time and reversibly ("gate, don't destroy"), except retention — a 30-day grace keeps the Pro cap before the Community cap starts hard-deleting; the ProDowngradedUtc/trial-expiry anchor model, the TenantPlanDowngraded/RetentionGrace* ops events, and the deliberate non-grace of everything else.
* [Tenant Activation & the Once-Only Welcome Mail](backend/tenant-activation-welcome-mail.md) - How a signup becomes an activated tenant (manual approve or the auto-approve queue), and the write-then-read + conditional-marker arbitration that sends the welcome mail exactly once even though activation and the user-typed notification address race each other.
@@ -73,6 +76,7 @@ https://docs.autopilotmonitor.com (separate repository).
# Security & Identity
* [MCP OAuth Flow](mcp-oauth-flow.md) - Who authenticates where when connecting an AI client to the MCP server; two identities, three parties plus Entra ID.
* [Client-Certificate Tenant Binding](backend/client-cert-tenant-binding.md) - Why a chain-valid Intune MDM certificate only proves "issued to some tenant", which extension actually carries the Entra tenant id (`…5.14`) versus the two that look like it (`…5.6` Intune Account id, `…5.4` device id, different encodings), why the Entra join certificate is not part of the auth path, and the shadow-then-enforce rollout with its volume-controlled telemetry.
# Conventions for this bundle
+11
View File
@@ -1,7 +1,18 @@
# Log
## 2026-08-24
* **New**: `backend/client-cert-tenant-binding.md` — the agent's mTLS certificate chains to Microsoft's shared Intune roots, so certificate validation alone never proved tenant membership; tenant scoping rested entirely on the (non-secret) device serial in the downstream validators. The Intune MDM leaf carries the Entra tenant id in OID `1.2.840.113556.5.14` (nested OCTET STRING), which is NOT `…5.6` (Intune Account id, a different GUID) and NOT `…5.4` (device id, stored as 16 bare bytes) — all decoded by one shared `MsDeviceCertificateOids.TryParseGuid` now used by agent and backend. Stage 1 ships as shadow only: `CertTenantBinding.Evaluate` (pure) plus `SecurityValidator.ObserveCertTenantBinding` (returns void, swallows its own exceptions, cannot change an authorization outcome), with `Match` sampled via `CertificateValidationResult.FromCache` so the denominator survives without per-request volume. Grep marker for the stage-2 enforcement change: `CERT-TENANT-BINDING-SHADOW`.
## 2026-08-23
* **New**: `backend/telemetry-ingest-shaping.md` — `RequestTelemetryMiddleware` sets `SamplingPercentage=100` on its own item (worker request rows were adaptively sampled, ItemCount 2–5 on weekdays, despite host.json `excludedTypes`), the redundant host HTTP request copy is dropped by a workspace transformation DCR (`infra/appinsights-workspace-transforms.bicep`, timer/queue rows kept), and `StorageDependencyFilterProcessor` now also drops expected storage outcomes (404/412/409, both InProc and HTTP shapes) and successful SignalR REST calls; the host-emitted SignalR binding calls (bulk of the volume, SDK `rdddsc`) are dropped by a second `AppDependencies` flow in the same DCR. Measured ≈ −1.3 GB/week.
* **Update**: `backend/table-schema-sentinel.md` — "Measuring it": `StartupTelemetryService` emits `BackendStartupMs` (process start → ApplicationStarted) and `BackendTableInitMs` as App Insights custom metrics with version/commit/fullPass dimensions; the MCP server writes one `{"type":"boot","readyMs"}` stderr line per start (query in `.claude/commands/mcp-telemetry.md` §A2).
* **New**: `backend/table-schema-sentinel.md` — startup table initialization gated by a schema sentinel (SHA-256 over `TableNames.All` in `AdminConfiguration`/`SchemaSentinel`/`tables`): matching sentinel ⇒ zero `CreateTableIfNotExists` calls on a cold start, missing/stale ⇒ one full pass + sentinel rewrite. SessionsIndex startup backfill only after a full pass, `IsSessionIndexEmptyAsync` no longer treats errors as "empty", conditional-insert claim so one scaled-out instance backfills. Daily maintenance runs `IStorageInitializer.EnsureAllAsync` to repair out-of-band table deletions.
* **Update**: `backend/verdict-calibration.md` — MCP `get_verdict_calibration` response shaping: per-row trend denominators dropped, explicit nulls, `minSharePct` / `top` with an `omitted` report.
* **Update**: `backend/verdict-calibration.md` — share-regression radar restricted to backend-decided paths; agent-declared/registration paths (customer workflow mix) and derived legacy paths no longer alert, stale episodes re-arm themselves.
* **Update**: `backend/time-attribution.md`, `backend/device-journeys.md`, `backend/verdict-calibration.md` — the rolling maintenance sweeps (time attribution, device journeys, verdict calibration, calibration-radar discovery) now share ONE projected 35-day cross-tenant Sessions scan per tick (`LoadSweepWindowSessionsAsync` + `MaintenanceSweepSessionProjection`, pinned by `MaintenanceSweepProjectionEquivalenceTests`) instead of four full-table drains; each sweep slices its own window by StartedAt.
* **New**: `backend/verdict-calibration.md` — the classifier thermometer: every status write now stamps `VerdictPath` (required parameter on `UpdateSessionStatusAsync`; `EnrollmentTimeoutClassifier` returns its rule id), overriding a verdict preserves `PriorStatus`/`PriorVerdictPath`, `VerdictPathDerivation` attributes pre-instrumentation rows read-side (`legacy:*` when ambiguous). New daily aggregate `VerdictCalibrationAggregates` (per-path count/derived/eligible7d/reEnrolled7d/overridden-by-admin|late|other, sweep after the device-journey pass, 180d retention), `GET global/metrics/verdict-calibration`, MCP `get_verdict_calibration`, admin metrics section, and the `VerdictCalibrationRadar` (share / silence-share / r6 evidence-gap episodes in the tracker keyspace `verdictcalibration|`, `VerdictCalibrationDrift` ops event, no tenant bell).
* **New**: `backend/silent-session-classification.md` — one place for the silent-session verdict: `EnrollmentTimeoutClassifier` rule order incl. the new rule 1c `IsSelfDeployingProvisioned` (self-deploying profile + Device Setup all-succeeded + no explicit failure ⇒ Succeeded reconcile, no grace/AwaitingUser — kiosk tenant aebdce78 audit: 836 finished devices parked AwaitingUser/Incomplete), the sweep stage-1 gates (`TryMarkWhiteGloveAwaitingUserAsync`, new `TryReconcileSelfDeployingAsync`), the ingest callers (max-lifetime, late-telemetry heal, stall probe) and the new admin retro mode `self_deploying_silent` (`GetSelfDeployingSilentSessionsAsync`). Agent companion: `SessionRegistrationHelper` waits up to 15s for a network link and retries 6× (~62s) instead of 5× (~30s) so the reboot relaunch on Wi-Fi no longer exits 7 before the link is back.
* **New**: `agent/diagnostics-package.md` — the diagnostics ZIP's built-in collection is now ONE catalog in Shared (`DiagnosticsBuiltInSections.All`): the agent iterates it in order and the backend serves the same list to the portal (`GET diagnostics/paths`, MemberRead), so what administrators see is exactly what the agent collects. New conditional sections: RealmJoin client/package/Chocolatey/per-user logs behind the tenant's RealmJoin Watcher toggle (mirrored onto `AgentConfiguration.EnableRealmJoinWatcher` by the merger) and the Device Preparation bootstrapper event log behind the deterministic WDP marker (probe once per build, fail toward Classic); every skip is a manifest line. Built-ins bypass `DiagnosticsPathGuards` by design (reviewed code, `%ProgramData%\RealmJoin` is not allowlisted); caps/reparse/evtx export still apply. `MergeDiagnosticsLogPaths` dedups global ∪ tenant paths. Portal: collapsed read-only catalog + one-line rows in both cards, tenant context dropped the GA-only global fetch.
* **Update**: `rules/gather-rule-guardrails.md` — `DiagnosticsPathGuards` scope clarified: configured diagnostics paths only, never the built-in catalog.
@@ -0,0 +1,78 @@
// Autopilot-Monitor — Log Analytics workspace transformation (ingestion-time filter)
//
// Two ingestion-time filters. (2) Successful Azure SignalR REST dependencies emitted by the
// HOST process for the SignalR output binding (SDKVersion rdddsc:*) — the worker-side
// StorageDependencyFilterProcessor never sees those. Failed SignalR calls are kept.
// (1) Drops the redundant HOST-side HTTP request record from AppRequests so that exactly one
// canonical request row remains: the worker-side copy emitted by RequestTelemetryMiddleware
// (Properties.Source == 'WorkerMiddleware', carries TenantId/UserId/CorrelationId and is
// sampling-exempt per item). Non-HTTP host invocations (timer/queue triggers) have an empty
// Url and are kept — the worker middleware never sees them.
//
// Why ingestion-time: the host request item is emitted by the Functions host process and
// cannot be suppressed from worker code or host.json (verified 2026-06-09, see Program.cs).
// Live numbers 2026-08-23 (7d): host copy 274k rows / 323 MB, worker copy 189k rows / 208 MB.
//
// Cost note: transformation filtering is free up to 50% of the table's incoming volume; the
// portion beyond that (here ~11% of AppRequests) is billed at the (cheap) data-processing rate.
//
// Apply (two steps; the link is a CLI update = GET+PUT, which keeps retention/SKU intact):
// az deployment group create -g rg-autopilotmonitor-prd-gwc \
// --template-file infra/appinsights-workspace-transforms.bicep \
// --query properties.outputs.dcrId.value -o tsv
// az monitor log-analytics workspace update -g rg-autopilotmonitor-prd-gwc -n AutopilotMonitor \
// --data-collection-rule <dcrId-from-above>
//
// Verify after ~20 min (ingestion latency):
// AppRequests | where TimeGenerated > ago(15m) | where AppRoleName == 'autopilotmonitor-api-eu'
// | summarize rows=count() by source=tostring(Properties.Source), hasUrl=isnotempty(Url)
// → only (WorkerMiddleware, true) and ('', false) buckets remain.
//
// Rollback: az monitor log-analytics workspace update -g <rg> -n AutopilotMonitor
// --set properties.defaultDataCollectionRuleResourceId=null (then delete the DCR)
@description('Log Analytics workspace that backs the workspace-based Application Insights resources.')
param workspaceName string = 'AutopilotMonitor'
param location string = resourceGroup().location
resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' existing = {
name: workspaceName
}
resource transforms 'Microsoft.Insights/dataCollectionRules@2023-03-11' = {
name: 'dcr-autopilotmonitor-ws-transforms'
location: location
kind: 'WorkspaceTransforms'
properties: {
description: 'AppRequests: keep the enriched worker request record (Properties.Source == WorkerMiddleware) and non-HTTP host invocations (empty Url); drop the host HTTP duplicate.'
destinations: {
logAnalytics: [
{
workspaceResourceId: workspace.id
name: 'ws'
}
]
}
dataFlows: [
{
streams: [ 'Microsoft-Table-AppRequests' ]
destinations: [ 'ws' ]
transformKql: 'source | where tostring(Properties.Source) == \'WorkerMiddleware\' or isempty(Url)'
}
{
// Successful Azure SignalR REST calls (group add/remove per connection) are emitted by
// the HOST process for the SignalR output binding (SDKVersion rdddsc:*), so the worker's
// StorageDependencyFilterProcessor never sees them. Failed SignalR calls are kept.
streams: [ 'Microsoft-Table-AppDependencies' ]
destinations: [ 'ws' ]
transformKql: 'source | where not(Success == true and Target endswith \'.service.signalr.net\')'
}
]
}
}
// A workspace has exactly one transformation DCR, activated by linking it as the workspace's
// defaultDataCollectionRuleResourceId — deliberately NOT modelled here (see Apply above).
output dcrId string = transforms.id
@@ -0,0 +1,62 @@
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
{
/// <summary>
/// Pins the compiled-in fallback rule catalogs (<see cref="BuiltInConfigCatalog"/>):
/// the agent embeds the SAME CI-combined rules/dist artifacts the backend serves, so a
/// no-config/no-cache session still runs full IME tracking. These tests guard the
/// embedding + deserialization end-to-end — a broken LogicalName, a moved resource or a
/// shape change in the combined JSON must fail HERE, not silently strip the fallback
/// back down to zero patterns in the field.
/// </summary>
public class BuiltInConfigCatalogTests
{
[Fact]
public void ImeLogPatterns_LoadFromEmbeddedCatalog_EnabledOnly()
{
var patterns = BuiltInConfigCatalog.GetEnabledImeLogPatterns(logger: null);
Assert.NotEmpty(patterns);
Assert.All(patterns, p =>
{
Assert.True(p.Enabled);
Assert.False(string.IsNullOrWhiteSpace(p.PatternId));
Assert.False(string.IsNullOrWhiteSpace(p.Pattern));
Assert.False(string.IsNullOrWhiteSpace(p.Action));
Assert.False(string.IsNullOrWhiteSpace(p.Category));
});
Assert.Equal(patterns.Count, patterns.Select(p => p.PatternId).Distinct().Count());
}
[Fact]
public void ImeLogPatterns_CarryCoreTrackingActions()
{
// The app-tracking backbone: without these two actions the ImeLogTracker cannot
// attribute installs to apps or follow ESP phases. Their presence proves the
// deserialization actually populated the action field (a shape drift in the
// combined JSON would otherwise still pass the structural checks above).
var actions = BuiltInConfigCatalog.GetEnabledImeLogPatterns(logger: null)
.Select(p => p.Action).ToHashSet();
Assert.Contains("setCurrentApp", actions);
Assert.Contains("espPhaseDetected", actions);
}
[Fact]
public void GatherRules_LoadFromEmbeddedCatalog_EnabledOnly()
{
// The built-in gather-rule catalog ships mostly disabled rules — only the
// default-enabled ones may reach the fallback config.
var rules = BuiltInConfigCatalog.GetEnabledGatherRules(logger: null);
Assert.All(rules, r =>
{
Assert.True(r.Enabled);
Assert.False(string.IsNullOrWhiteSpace(r.RuleId));
});
}
}
}
@@ -114,6 +114,22 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.Equal(1, svc.LastFetchAttempts);
}
[Fact]
public async Task FetchConfig_DefaultsFallback_CarriesBuiltInCatalogs()
{
// A no-config/no-cache session must still get the compiled-in rule catalogs —
// an empty ImeLogPatterns list kills the entire IME tracking pipeline (the
// pre-catalog behaviour this pins against regressing to).
var svc = NewService(BackendBehaviour.AlwaysTransientFailure);
var result = await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(0, result.ConfigVersion);
Assert.NotEmpty(result.ImeLogPatterns);
Assert.All(result.ImeLogPatterns, p => Assert.True(p.Enabled));
Assert.All(result.GatherRules, r => Assert.True(r.Enabled));
}
[Fact]
public async Task FetchConfig_EmptyResponse_TreatedAsTransient()
{
@@ -128,6 +144,94 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.Equal(4, svc.LastFetchAttempts);
}
// ── Network-link gate (tenant aebdce78 audit 2026-08-24) ────────────
[Fact]
public async Task FetchConfig_InitialFetch_WaitsForNetworkLink()
{
// The initial startup fetch must gate on the link-level signal so the retry
// budget isn't burned into a NIC that is provably still down (agent relaunched
// at boot, Wi-Fi still associating).
var svc = NewService(BackendBehaviour.SuccessImmediately);
await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(1, svc.NetworkLinkWaits);
}
[Fact]
public async Task FetchConfig_RotatePath_SkipsNetworkLinkGate()
{
// rotate_config runs mid-session on a proven link and must stay latency-bounded.
var svc = NewService(BackendBehaviour.SuccessImmediately);
await svc.FetchConfigAsync(retryOnTransientErrors: false);
Assert.Equal(0, svc.NetworkLinkWaits);
}
// ── Post-registration recovery ──────────────────────────────────────
[Fact]
public async Task TryRecover_AfterFailedFetch_AppliesLiveConfig_KeepsFailureRecord()
{
// The core self-heal contract: registration proved the backend reachable, the
// recovery swaps in the live config — but the LastFetch* observability keeps
// describing the failed startup fetch so the wire event can report both.
var svc = NewService(BackendBehaviour.FailAllThenSucceed);
await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
var recovered = await svc.TryRecoverConfigAsync();
Assert.NotNull(recovered);
Assert.Equal(42, recovered.ConfigVersion);
Assert.Equal(42, svc.CurrentConfig.ConfigVersion);
Assert.True(svc.RecoveredAfterFetchFailure);
// Startup-failure record intact:
Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
Assert.Equal(1 + RemoteConfigService.InitialFetchRetryBackoff.Length, svc.LastFetchAttempts);
Assert.Equal(nameof(HttpRequestException), svc.LastFetchFailureType);
}
[Fact]
public async Task TryRecover_AfterSuccessfulFetch_IsNoOp()
{
var svc = NewService(BackendBehaviour.SuccessImmediately);
await svc.FetchConfigAsync(retryOnTransientErrors: true);
var recovered = await svc.TryRecoverConfigAsync();
Assert.Null(recovered);
Assert.False(svc.RecoveredAfterFetchFailure);
}
[Fact]
public async Task TryRecover_BeforeAnyFetch_IsNoOp()
{
var svc = NewService(BackendBehaviour.SuccessImmediately);
var recovered = await svc.TryRecoverConfigAsync();
Assert.Null(recovered);
Assert.False(svc.RecoveredAfterFetchFailure);
}
[Fact]
public async Task TryRecover_WhenBackendStillFailing_KeepsFallback_NeverThrows()
{
var svc = NewService(BackendBehaviour.AlwaysTransientFailure);
var fallback = await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(0, fallback.ConfigVersion);
var recovered = await svc.TryRecoverConfigAsync();
Assert.Null(recovered);
Assert.False(svc.RecoveredAfterFetchFailure);
Assert.Equal(0, svc.CurrentConfig.ConfigVersion); // fallback config untouched
Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
}
// ── Outcome state defaults ──────────────────────────────────────────
[Fact]
@@ -145,6 +249,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
SuccessImmediately,
AlwaysTransientFailure,
FailTwiceThenSucceed,
/// <summary>All 4 initial attempts fail; the NEXT call (the recovery fetch) succeeds.</summary>
FailAllThenSucceed,
AlwaysAuthFailure401,
AlwaysReturnNull,
}
@@ -177,6 +283,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
_behaviour = behaviour;
}
/// <summary>Number of times the network-link gate seam was awaited.</summary>
public int NetworkLinkWaits { get; private set; }
protected override Task<AgentConfigResponse> CallBackendAsync()
{
_calls++;
@@ -188,6 +297,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
if (_calls < 3)
throw new HttpRequestException("simulated cold-start timeout");
return Task.FromResult(new AgentConfigResponse { ConfigVersion = 42 });
case BackendBehaviour.FailAllThenSucceed:
if (_calls <= 1 + RemoteConfigService.InitialFetchRetryBackoff.Length)
throw new HttpRequestException("simulated dead network link");
return Task.FromResult(new AgentConfigResponse { ConfigVersion = 42 });
case BackendBehaviour.AlwaysTransientFailure:
throw new HttpRequestException("simulated network failure");
case BackendBehaviour.AlwaysAuthFailure401:
@@ -202,6 +315,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// Zero-delay so the suite finishes in ms instead of the live 10/30/60 s schedule.
protected override Task DelayBetweenAttemptsAsync(TimeSpan delay) => Task.CompletedTask;
// Record + skip the real link poll so the suite is deterministic on offline CI.
protected override Task WaitForNetworkLinkAsync()
{
NetworkLinkWaits++;
return Task.CompletedTask;
}
// In-memory + per-instance cache so a prior test's successful fetch can't
// bleed into a later test that expects UsedDefaults.
private AgentConfigResponse? _cached;
@@ -1,5 +1,6 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.DecisionCore.Signals;
@@ -11,8 +12,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
internal sealed class FakeSignalIngressSink : ISignalIngressSink
{
private readonly List<PostedSignal> _posted = new List<PostedSignal>();
private readonly object _gate = new object();
public IReadOnlyList<PostedSignal> Posted => _posted;
/// <summary>
/// Snapshot, not the live list: producers post from timer / worker threads (e.g. the
/// parked-dwell tripwire) while tests poll this in a SpinWait loop. Enumerating the live
/// list across an Add throws "Collection was modified" — a rare CI-only failure
/// (run 32644425162) with no assertion text.
/// </summary>
public IReadOnlyList<PostedSignal> Posted
{
get { lock (_gate) return _posted.ToArray(); }
}
/// <summary>When non-null, <see cref="Post"/> throws instead of capturing.</summary>
public Exception? ThrowOnPost { get; set; }
@@ -27,7 +38,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
object? typedPayload = null)
{
if (ThrowOnPost != null) throw ThrowOnPost;
_posted.Add(new PostedSignal(kind, occurredAtUtc, sourceOrigin, evidence, payload, kindSchemaVersion, typedPayload));
lock (_gate)
_posted.Add(new PostedSignal(kind, occurredAtUtc, sourceOrigin, evidence, payload, kindSchemaVersion, typedPayload));
}
internal sealed class PostedSignal
@@ -182,6 +182,73 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
Assert.Equal("FromCache", data["outcome"]);
}
// ── Post-registration recovery (Phase 6.5, tenant aebdce78 audit 2026-08-24) ─
[Fact]
public void EmitFetchFailedIfAny_AfterRecovery_MarksRecoveredAndDemotesToInfo()
{
using var rig = new EmitterRig();
// Startup fetch failed (network link not up yet), then the post-registration
// recovery fetched ConfigVersion 39. The event must keep the original failure
// details AND mark the heal.
rig.RemoteConfig.SetFakeOutcome(
RemoteConfigFetchOutcome.UsedDefaults,
configVersion: 39, // CurrentConfig now holds the recovered config
attempts: 4,
failureType: "HttpRequestException",
failureMessage: "simulated dead network link",
recovered: true);
LifecycleEmitters.EmitRemoteConfigFetchFailedIfAny(
rig.Post, rig.AgentConfig, rig.RemoteConfig, rig.Logger);
var data = AssertSingleEmit(rig.Sink, expectedEventType: "remote_config_fetch_failed");
Assert.Equal("UsedDefaults", data["outcome"]); // startup failure record intact
Assert.Equal(4, data["attempts"]);
Assert.Equal(true, data["recovered"]);
Assert.Equal(39, data["recoveredConfigVersion"]);
Assert.Equal(nameof(EventSeverity.Info), rig.Sink.Posted[0].Payload!["severity"]);
}
[Fact]
public void EmitFetchFailedIfAny_WithoutRecovery_StaysWarning()
{
using var rig = new EmitterRig();
rig.RemoteConfig.SetFakeOutcome(
RemoteConfigFetchOutcome.UsedDefaults,
configVersion: 0,
attempts: 4,
failureType: "HttpRequestException",
failureMessage: "still down");
LifecycleEmitters.EmitRemoteConfigFetchFailedIfAny(
rig.Post, rig.AgentConfig, rig.RemoteConfig, rig.Logger);
var data = AssertSingleEmit(rig.Sink, expectedEventType: "remote_config_fetch_failed");
Assert.False(data.ContainsKey("recovered"));
Assert.False(data.ContainsKey("recoveredConfigVersion"));
Assert.Equal(nameof(EventSeverity.Warning), rig.Sink.Posted[0].Payload!["severity"]);
}
[Fact]
public void EmitAgentStarted_AfterRecovery_FlagsLiveConfig()
{
using var rig = new EmitterRig();
rig.RemoteConfig.SetFakeOutcome(
RemoteConfigFetchOutcome.UsedDefaults,
configVersion: 39,
recovered: true);
LifecycleEmitters.EmitAgentStarted(
rig.Post, rig.AgentConfig, rig.PreviousExit, "2.0.806", rig.RemoteConfig, rig.Logger);
var data = AssertSingleEmit(rig.Sink, expectedEventType: "agent_started");
Assert.Equal(39, data["configVersion"]);
Assert.Equal(true, data["remoteConfigFetched"]); // running on a live-fetched config
Assert.Equal("UsedDefaults", data["remoteConfigOutcome"]); // startup outcome stays honest
Assert.Equal(true, data["remoteConfigRecovered"]);
}
// ── Helpers ─────────────────────────────────────────────────────────
private static Dictionary<string, object> AssertSingleEmit(
@@ -258,8 +325,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
int attempts = 1,
string? failureType = null,
string? failureMessage = null,
int? authStatusCode = null)
int? authStatusCode = null,
bool recovered = false)
{
typeof(RemoteConfigService).GetProperty(nameof(RecoveredAfterFetchFailure))?.SetValue(this, recovered);
// CurrentConfig is set via the protected SetConfig path — easiest:
// a real FetchConfigAsync replacement that just stashes a canned config.
typeof(RemoteConfigService).GetMethod("SetConfig",
@@ -0,0 +1,146 @@
using System;
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
{
/// <summary>
/// Pins <see cref="EntraDeviceCertHelper.ParseGuidFromExtension"/>, the pure byte-level part of the
/// cert probe. Both encodings Microsoft actually ships are covered: the ASN.1 OCTET STRING wrapper
/// used by the MS-Organization-Access TenantId/DeviceId extensions (and Intune's
/// <c>1.2.840.113556.5.6</c> / <c>5.14</c>), and the BARE 16-byte payload used by Intune's DeviceId
/// extension <c>1.2.840.113556.5.4</c> — the latter was silently returning <c>null</c> before.
/// The store-iteration shell around it needs a real machine cert store and is not unit-tested.
/// </summary>
public sealed class EntraDeviceCertHelperParseTests
{
// Synthetic — never a real tenant/device id. ToByteArray() gives the MS little-endian layout
// that both cert encodings carry, so every expectation below is derived, not hand-transcribed.
private static readonly Guid Sample = new Guid("11223344-5566-7788-99aa-bbccddeeff00");
private static byte[] Wrap(byte[] payload, params byte[] header)
{
var buf = new byte[header.Length + payload.Length];
Buffer.BlockCopy(header, 0, buf, 0, header.Length);
Buffer.BlockCopy(payload, 0, buf, header.Length, payload.Length);
return buf;
}
[Fact]
public void OctetString_ShortForm_Parses()
{
var raw = Wrap(Sample.ToByteArray(), 0x04, 0x10);
Assert.Equal(Sample, EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void OctetString_LongForm_Parses()
{
var raw = Wrap(Sample.ToByteArray(), 0x04, 0x81, 0x10);
Assert.Equal(Sample, EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
/// <summary>
/// The regression this class exists for: Intune's <c>1.2.840.113556.5.4</c> extension value is
/// exactly 16 bytes with no <c>04 10</c> prefix, so the strict OCTET-STRING parser bailed on the
/// tag check and the DeviceId was unreadable.
/// </summary>
[Fact]
public void BareSixteenBytes_Parses()
{
var raw = Sample.ToByteArray();
Assert.Equal(Sample, EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
/// <summary>
/// A bare GUID may legitimately start with 0x04 (the OCTET STRING tag). That must not be
/// mistaken for a wrapper: a wrapped 16-byte GUID is always >= 18 bytes, so the two forms are
/// disjoint by length and the bare path has to win here.
/// </summary>
[Fact]
public void BareSixteenBytes_StartingWithOctetStringTag_StillParses()
{
var guid = new Guid("aabbcc04-0110-4020-8030-405060708090");
var raw = guid.ToByteArray();
Assert.Equal(0x04, raw[0]); // guard: the byte layout really does start with the ASN.1 tag
Assert.Equal(16, raw.Length);
Assert.Equal(guid, EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void OctetString_IsLittleEndianMsLayout()
{
// 04 10 + the canonical MS binary layout of {11223344-...} => first group byte-reversed.
var raw = new byte[]
{
0x04, 0x10,
0x44, 0x33, 0x22, 0x11,
0x66, 0x55,
0x88, 0x77,
0x99, 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF, 0x00
};
Assert.Equal(Sample, EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void Null_ReturnsNull()
{
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(null));
}
[Fact]
public void Empty_ReturnsNull()
{
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(new byte[0]));
}
[Theory]
[InlineData(15)]
[InlineData(17)]
public void BarePayload_OfWrongLength_ReturnsNull(int length)
{
var raw = new byte[length];
for (var i = 0; i < length; i++) raw[i] = (byte)(0x40 + i); // no 0x04 tag, no valid wrapper
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void NonOctetStringTag_ReturnsNull()
{
// e.g. Intune's 1.2.840.113556.5.15 => ASN.1 INTEGER 2.
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(new byte[] { 0x02, 0x01, 0x02 }));
}
[Fact]
public void OctetString_WithNonGuidContentLength_ReturnsNull()
{
var raw = Wrap(new byte[8], 0x04, 0x08);
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void OctetString_TruncatedContent_ReturnsNull()
{
// Declares 16 content bytes but only carries 10.
var raw = Wrap(new byte[10], 0x04, 0x10);
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
[Fact]
public void OctetString_LongFormWithOversizedLengthHeader_ReturnsNull()
{
// 0x85 => 5 length bytes, above the 4 the parser accepts.
var raw = Wrap(Sample.ToByteArray(), 0x04, 0x85, 0x00, 0x00, 0x00, 0x00, 0x10);
Assert.Null(EntraDeviceCertHelper.ParseGuidFromExtension(raw));
}
}
}
@@ -52,6 +52,11 @@
carries the V2 assembly namespace so GatherRuleGuards locates it under
"AutopilotMonitor.Agent.V2.Core.Resources.guardrails.json". -->
<EmbeddedResource Include="..\..\..\rules\guardrails.json" LogicalName="AutopilotMonitor.Agent.V2.Core.Resources.guardrails.json" />
<!-- Built-in fallback rule catalogs (BuiltInConfigCatalog): the SAME CI-combined
rules/dist artifacts the backend embeds and serves. Snapshotted per agent build so
the no-config/no-cache fallback still runs full IME tracking instead of nothing. -->
<EmbeddedResource Include="..\..\..\rules\dist\ime-log-patterns.json" LogicalName="AutopilotMonitor.Agent.V2.Core.Resources.ime-log-patterns.json" />
<EmbeddedResource Include="..\..\..\rules\dist\gather-rules.json" LogicalName="AutopilotMonitor.Agent.V2.Core.Resources.gather-rules.json" />
</ItemGroup>
<ItemGroup>
@@ -0,0 +1,74 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Models;
using Newtonsoft.Json;
namespace AutopilotMonitor.Agent.V2.Core.Configuration
{
/// <summary>
/// Compiled-in fallback copies of the global rule catalogs, embedded from the same
/// CI-generated <c>rules/dist</c> artifacts the backend seeds and serves — single
/// source by construction: each agent release snapshots the catalog state of its own
/// build, so the offline fallback can never drift further than the running agent
/// version itself (no manually maintained default to forget).
/// <para>
/// Consumed only by <see cref="RemoteConfigService"/>'s built-in default config,
/// i.e. when a session has NO live config and NO cache. Before this existed, that
/// path shipped an empty <c>ImeLogPatterns</c> list, which left the entire IME
/// tracking pipeline (app installs, platform/remediation scripts, health-script
/// compliance, IME ESP-phase signals) dead for the whole session — the core of
/// enrollment monitoring. A live fetch (startup, post-registration recovery or
/// rotate_config) always wins over this catalog.
/// </para>
/// Fail-soft: any load/parse error returns an empty list (the pre-catalog behaviour).
/// </summary>
internal static class BuiltInConfigCatalog
{
/// <summary>Enabled built-in IME log patterns from the embedded catalog.</summary>
public static List<ImeLogPattern> GetEnabledImeLogPatterns(AgentLogger logger)
=> LoadEnabled<ImeLogPattern>("ime-log-patterns.json", p => p.Enabled, logger);
/// <summary>Enabled built-in gather rules from the embedded catalog.</summary>
public static List<GatherRule> GetEnabledGatherRules(AgentLogger logger)
=> LoadEnabled<GatherRule>("gather-rules.json", r => r.Enabled, logger);
private static List<T> LoadEnabled<T>(string name, Func<T, bool> isEnabled, AgentLogger logger)
{
try
{
var assembly = typeof(BuiltInConfigCatalog).Assembly;
using (var stream = assembly.GetManifestResourceStream(
"AutopilotMonitor.Agent.V2.Core.Resources." + name))
{
if (stream == null)
{
logger?.Warning($"Built-in config catalog resource '{name}' not found — fallback runs without it.");
return new List<T>();
}
using (var reader = new StreamReader(stream))
{
var wrapper = JsonConvert.DeserializeObject<RulesFile<T>>(reader.ReadToEnd());
var all = wrapper?.Rules ?? new List<T>();
return all.Where(isEnabled).ToList();
}
}
}
catch (Exception ex)
{
logger?.Warning($"Built-in config catalog '{name}' failed to load ({ex.Message}) — fallback runs without it.");
return new List<T>();
}
}
/// <summary>Shape of the CI-combined catalog files: <c>{ "rules": [...] }</c>.</summary>
private sealed class RulesFile<T>
{
[JsonProperty("rules")]
public List<T> Rules { get; set; }
}
}
}
@@ -42,6 +42,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
TimeSpan.FromSeconds(60),
};
/// <summary>
/// Upper bound for the pre-fetch network-link wait on the initial startup fetch.
/// Deliberately longer than the registration gate's 15s: the fetch runs FIRST after a
/// boot-time relaunch, when Wi-Fi may still be associating (802.1X, OOBE), and a fetch
/// burned into a dead link strands the whole session on built-in defaults — tenant aebdce78
/// audit 2026-08-24: ~50% of sessions lost their tenant config exactly this way.
/// </summary>
internal static readonly TimeSpan InitialFetchNetworkLinkWaitMax = TimeSpan.FromSeconds(60);
/// <summary>
/// Gets the current configuration (thread-safe)
/// </summary>
@@ -76,6 +85,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
/// <summary>HTTP status for auth failures (401/403), null otherwise.</summary>
public int? LastFetchAuthStatusCode { get; private set; }
/// <summary>
/// True once <see cref="TryRecoverConfigAsync"/> replaced a failed startup fetch with a
/// live config. The <c>LastFetch*</c> properties intentionally keep describing the failed
/// startup fetch so the wire event can report both the original failure and the recovery.
/// </summary>
public bool RecoveredAfterFetchFailure { get; private set; }
public RemoteConfigService(BackendApiClient apiClient, string tenantId, AgentLogger logger, EmergencyReporter emergencyReporter = null, DistressReporter distressReporter = null, AuthFailureTracker authFailureTracker = null)
{
_apiClient = apiClient ?? throw new ArgumentNullException(nameof(apiClient));
@@ -116,6 +132,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
LastFetchFailureMessage = null;
LastFetchAuthStatusCode = null;
// Initial startup fetch only: give a still-associating NIC a chance to come up
// before the first attempt. Without this the retry budget (10/30/60s) is burned
// into a provably dead link — the agent starts at boot, Wi-Fi associates minutes
// later, and the session runs on defaults although the backend was never the
// problem. The rotate_config path (retryOnTransientErrors:false) runs mid-session
// on a proven link and must stay latency-bounded, so it skips the gate.
if (retryOnTransientErrors)
{
try { await WaitForNetworkLinkAsync().ConfigureAwait(false); }
catch (Exception ex) { _logger.Debug($"Pre-fetch network wait failed ({ex.Message}) — fetching anyway."); }
}
var maxAttempts = retryOnTransientErrors ? InitialFetchRetryBackoff.Length + 1 : 1;
Exception lastException = null;
@@ -203,14 +231,63 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
return cached;
}
// Fall back to defaults
_logger.Info("No cached config available, using defaults (all optional collectors disabled)");
// Fall back to built-in defaults (collector/analyzer defaults from the shared
// model initializers + the compiled-in rule catalogs).
var defaults = CreateDefaultConfig();
_logger.Info($"No cached config available, using built-in defaults (ConfigVersion=0, " +
$"{defaults.ImeLogPatterns?.Count ?? 0} built-in IME patterns, " +
$"{defaults.GatherRules?.Count ?? 0} built-in gather rules)");
SetConfig(defaults);
LastFetchOutcome = RemoteConfigFetchOutcome.UsedDefaults;
return defaults;
}
/// <summary>
/// Single-attempt recovery fetch for a session whose startup fetch failed. Called right
/// after session registration succeeded — registration proves the backend is reachable,
/// so the config that Phase 4 could not get is one cheap call away instead of the whole
/// session being stranded on cache/defaults (only a manual <c>rotate_config</c> action
/// could rescue it before this existed).
/// <para>
/// Unlike <see cref="FetchConfigAsync"/> this does NOT reset the <c>LastFetch*</c>
/// observability state — those keep describing the failed startup fetch so
/// <c>remote_config_fetch_failed</c> can report the original failure AND the recovery.
/// Returns the recovered config, or null when there is nothing to recover (startup fetch
/// succeeded / never ran) or the recovery attempt itself failed. Never throws.
/// </para>
/// </summary>
public async Task<AgentConfigResponse> TryRecoverConfigAsync()
{
if (LastFetchOutcome != RemoteConfigFetchOutcome.FromCache &&
LastFetchOutcome != RemoteConfigFetchOutcome.UsedDefaults)
return null;
try
{
_logger.Info("Retrying remote config fetch after successful session registration...");
var config = await CallBackendAsync().ConfigureAwait(false);
if (config == null)
{
_logger.Warning("Config recovery fetch returned empty response — keeping fallback config.");
return null;
}
_logger.Info($"Remote config recovered: ConfigVersion={config.ConfigVersion}, " +
$"GatherRules={config.GatherRules?.Count ?? 0}");
LogCollectorSettings(config.Collectors);
SetConfig(config);
CacheConfig(config);
_authFailureTracker?.RecordSuccess();
RecoveredAfterFetchFailure = true;
return config;
}
catch (Exception ex)
{
_logger.Warning($"Config recovery fetch failed ({ex.Message}) — keeping fallback config.");
return null;
}
}
private void SetConfig(AgentConfigResponse config)
{
lock (_configLock)
@@ -319,6 +396,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
private AgentConfigResponse CreateDefaultConfig()
{
// Everything not set here falls to the shared AgentConfigResponse property
// initializers — the same values the backend uses as its `??` fallbacks, so the
// scalar knobs cannot drift. The rule catalogs are the one thing initializers
// cannot supply: without them the whole IME tracking pipeline is dead, so they
// come from the compiled-in snapshot of the served catalogs (BuiltInConfigCatalog).
return new AgentConfigResponse
{
ConfigVersion = 0,
@@ -330,8 +412,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
EnableGatherRuleDebugLog = false,
EnableEspContinueAnywayObservation = false,
Collectors = CollectorConfiguration.CreateDefault(),
GatherRules = new System.Collections.Generic.List<GatherRule>(),
ImeLogPatterns = new System.Collections.Generic.List<ImeLogPattern>()
GatherRules = BuiltInConfigCatalog.GetEnabledGatherRules(_logger),
ImeLogPatterns = BuiltInConfigCatalog.GetEnabledImeLogPatterns(_logger)
};
}
@@ -358,6 +440,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
protected virtual Task DelayBetweenAttemptsAsync(TimeSpan delay)
=> Task.Delay(delay);
/// <summary>
/// Awaits the pre-fetch network-link gate (initial startup fetch only). Production
/// polls the link-level signal via <see cref="NetworkLinkWait"/> bounded by
/// <see cref="InitialFetchNetworkLinkWaitMax"/>; tests subclass to record the call
/// and return immediately.
/// </summary>
protected virtual Task WaitForNetworkLinkAsync()
=> NetworkLinkWait.WaitAsync(_logger, InitialFetchNetworkLinkWaitMax, "Remote config fetch");
public void Dispose()
{
// Nothing to dispose
@@ -0,0 +1,38 @@
using System;
using System.Net.NetworkInformation;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
{
/// <summary>
/// Boot-time NIC grace shared by every first-contact backend call (remote-config fetch,
/// session registration, emergency-break reporter pattern): polls the cheap link-level
/// signal once per second for at most <paramref name="maxWait"/>. It cannot prove backend
/// reachability — the callers' retry loops handle that — it only keeps retry budgets from
/// being burned into a link that is provably still down (BootTrigger relaunch after a
/// mid-enrollment reboot, Wi-Fi still associating; tenant aebdce78 audits 2026-08-23/24).
/// Free on the normal path — a live link returns immediately. Probe errors end the wait,
/// never the caller's operation.
/// </summary>
internal static class NetworkLinkWait
{
internal static async Task WaitAsync(AgentLogger logger, TimeSpan maxWait, string context)
{
if (NetworkInterface.GetIsNetworkAvailable()) return;
logger?.Info($"{context}: no network link yet — waiting up to {maxWait.TotalSeconds:F0}s.");
var deadline = DateTime.UtcNow + maxWait;
while (DateTime.UtcNow < deadline)
{
await Task.Delay(1000).ConfigureAwait(false);
if (NetworkInterface.GetIsNetworkAvailable())
{
logger?.Info($"{context}: network link is up.");
return;
}
}
logger?.Warning($"{context}: still no network link after {maxWait.TotalSeconds:F0}s — attempting anyway.");
}
}
}
@@ -2,6 +2,7 @@ using System;
using System.Linq;
using System.Security.Cryptography.X509Certificates;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Security;
namespace AutopilotMonitor.Agent.V2.Core.Security
{
@@ -23,6 +24,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <item><description><c>1.2.840.113556.1.5.284.2</c> → DeviceId (same encoding)</description></item>
/// </list>
/// <para>
/// The sibling <c>Microsoft Intune MDM Device CA</c> client cert (the one the agent authenticates
/// with, see <see cref="CertificateHelper"/>) carries a DIFFERENT OID arc that identifies the same
/// device from Intune's side. Observed on a live Intune-enrolled Windows 11 device:
/// </para>
/// <list type="table">
/// <listheader><term>OID</term><description>Meaning / encoding</description></listheader>
/// <item><term><c>1.2.840.113556.5.4</c></term><description>Intune DeviceId — BARE 16-byte GUID, no ASN.1 wrapper. Equals the cert's Subject CN.</description></item>
/// <item><term><c>1.2.840.113556.5.6</c></term><description>Intune AccountId (the Intune tenant account, NOT the Entra TenantId) — OCTET STRING GUID.</description></item>
/// <item><term><c>1.2.840.113556.5.14</c></term><description>Entra TenantId — OCTET STRING GUID. This is the one to compare against an Entra tenant id.</description></item>
/// <item><term><c>1.2.840.113556.5.11</c></term><description>Further GUID, OCTET STRING; semantics not documented by Microsoft and not verified here.</description></item>
/// <item><term><c>1.2.840.113556.5.15/.16/.17/.18/.19/.23</c></term><description>ASN.1 INTEGER enums (platform / enrollment flavour); semantics undocumented.</description></item>
/// </list>
/// <para>
/// The bare-GUID encoding of <c>1.2.840.113556.5.4</c> is why
/// <see cref="ParseGuidFromExtension"/> accepts an unwrapped 16-byte payload as well —
/// a strict OCTET-STRING parser silently returns <c>null</c> for that extension.
/// </para>
/// <para>
/// All probes are best-effort: never throw, return <c>null</c> on miss, log a Warning
/// with the specific failure reason (no matching cert / extension missing / ASN.1
/// parse failed) so the agent log explains why the fallback didn't help.
@@ -31,8 +50,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
public static class EntraDeviceCertHelper
{
public const string MsOrganizationAccessIssuer = "MS-Organization-Access";
public const string TenantIdOid = "1.2.840.113556.1.5.284.5";
public const string DeviceIdOid = "1.2.840.113556.1.5.284.2";
public const string TenantIdOid = MsDeviceCertificateOids.EntraJoinCertTenantIdOid;
public const string DeviceIdOid = MsDeviceCertificateOids.EntraJoinCertDeviceIdOid;
/// <summary>
/// Reads the Entra TenantId from the MS-Organization-Access device cert.
@@ -88,7 +107,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
continue;
}
var guid = ParseGuidFromAsn1OctetString(ext.RawData);
var guid = ParseGuidFromExtension(ext.RawData);
if (guid != null)
{
logger?.Info(
@@ -98,7 +117,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
}
logger?.Warning(
$"EntraDeviceCertHelper: failed to parse {label} ASN.1 OCTET STRING in cert thumbprint={cert.Thumbprint} " +
$"EntraDeviceCertHelper: failed to parse {label} GUID extension in cert thumbprint={cert.Thumbprint} " +
$"(raw {ext.RawData.Length} bytes); trying next candidate.");
}
@@ -116,46 +135,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <summary>
/// Parses a 16-byte MS-format binary GUID (little-endian for the first three groups,
/// per <see cref="Guid(byte[])"/>) wrapped in an ASN.1 OCTET STRING. Handles both
/// short-form (<c>04 10 ...</c>) and long-form (<c>04 81 10 ...</c>) length encoding.
/// per <see cref="Guid(byte[])"/>) out of a certificate extension's raw value.
/// Two encodings are accepted, because Microsoft uses both:
/// <list type="number">
/// <item><description>Wrapped in an ASN.1 OCTET STRING — short-form (<c>04 10 ...</c>) or
/// long-form (<c>04 81 10 ...</c>) length. Used by the MS-Organization-Access TenantId/DeviceId
/// extensions and by the Intune <c>…5.6</c> / <c>…5.14</c> extensions.</description></item>
/// <item><description>A bare, unwrapped 16-byte payload. Used by the Intune DeviceId extension
/// <c>1.2.840.113556.5.4</c> — verified against a live Intune MDM device cert, whose raw value is
/// exactly 16 bytes with no <c>04 10</c> prefix.</description></item>
/// </list>
/// A valid OCTET-STRING wrapper around 16 bytes is at least 18 bytes long, so the two forms
/// never collide for well-formed input. A 16-byte value is always read as the bare form —
/// a truncated wrapper of exactly that length is indistinguishable from it.
/// Returns <c>null</c> on any structural mismatch — never throws.
/// <para>
/// The decoding itself lives in <see cref="MsDeviceCertificateOids.TryParseGuid"/> so the agent
/// and the backend's client-cert tenant binding share one implementation; this stays as the
/// agent-side nullable-returning façade.
/// </para>
/// </summary>
public static Guid? ParseGuidFromAsn1OctetString(byte[] raw)
{
if (raw == null || raw.Length < 2) return null;
// ASN.1 OCTET STRING tag.
if (raw[0] != 0x04) return null;
int contentStart;
int contentLen;
if (raw[1] < 0x80)
{
// short-form length
contentLen = raw[1];
contentStart = 2;
}
else
{
// long-form length: lower 7 bits = number of following length bytes.
int numLenBytes = raw[1] & 0x7F;
if (numLenBytes < 1 || numLenBytes > 4 || raw.Length < 2 + numLenBytes) return null;
contentLen = 0;
for (int i = 0; i < numLenBytes; i++)
contentLen = (contentLen << 8) | raw[2 + i];
contentStart = 2 + numLenBytes;
}
if (contentLen != 16 || contentStart + 16 > raw.Length) return null;
var guidBytes = new byte[16];
Buffer.BlockCopy(raw, contentStart, guidBytes, 0, 16);
try { return new Guid(guidBytes); }
catch { return null; }
}
public static Guid? ParseGuidFromExtension(byte[] raw) =>
MsDeviceCertificateOids.TryParseGuid(raw, out var value) ? value : (Guid?)null;
}
}
@@ -1,5 +1,4 @@
using System;
using System.Net.NetworkInformation;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
@@ -165,29 +164,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
=> Task.Delay(((int)Math.Pow(2, attempt)) * 1000);
/// <summary>
/// Boot-time NIC grace (same pattern as the emergency-break reporter): polls the cheap
/// link-level signal once per second for at most <see cref="NetworkLinkWaitMax"/>. It
/// cannot prove backend reachability — the retry loop handles that — it only keeps the
/// first attempts from being burned into a link that is provably still down. Probe
/// Boot-time NIC grace before the first registration attempt — shared polling loop in
/// <see cref="NetworkLinkWait"/>, bounded by <see cref="NetworkLinkWaitMax"/>. Probe
/// errors end the wait, never the registration.
/// </summary>
internal static async Task WaitForNetworkLinkAsync(AgentLogger logger)
{
if (NetworkInterface.GetIsNetworkAvailable()) return;
logger?.Info($"Session registration: no network link yet — waiting up to {NetworkLinkWaitMax.TotalSeconds:F0}s.");
var deadline = DateTime.UtcNow + NetworkLinkWaitMax;
while (DateTime.UtcNow < deadline)
{
await Task.Delay(1000).ConfigureAwait(false);
if (NetworkInterface.GetIsNetworkAvailable())
{
logger?.Info("Session registration: network link is up.");
return;
}
}
logger?.Warning($"Session registration: still no network link after {NetworkLinkWaitMax.TotalSeconds:F0}s — attempting anyway.");
}
internal static Task WaitForNetworkLinkAsync(AgentLogger logger)
=> NetworkLinkWait.WaitAsync(logger, NetworkLinkWaitMax, "Session registration");
private static SessionRegistration BuildRegistration(
AgentConfiguration agentConfig,
@@ -274,6 +274,15 @@ namespace AutopilotMonitor.Agent.V2
{
return registration.ExitCode;
}
// Phase 6.5 (config self-heal): when the Phase 4 fetch ran before the network link
// was up, registration has just proven the backend reachable — retry the fetch once
// so the session runs on the live tenant config instead of cache/defaults.
// See AgentRuntimeConfig.TryRecoverAfterRegistration for what is deliberately
// NOT re-run here (kill-signal check, endpoint migration).
Runtime.AgentRuntimeConfig.TryRecoverAfterRegistration(
bootstrap.AgentConfig, auth, runtimeConfig, GetAgentVersion(), consoleMode, logger);
// Phase 7+8 (orchestrator + shutdown lifecycle + onIngressReady wiring +
// initial signal posts + analyzer/probes startup + shutdown.Wait + finally)
// is encapsulated in AgentRuntimeHost. See Runtime/AgentRuntimeHost.cs.
@@ -69,6 +69,63 @@ namespace AutopilotMonitor.Agent.V2.Runtime
}
}
var configMergeResult = ApplyFetchedConfig(agentConfig, remoteConfig, auth, agentVersion, consoleMode, logger);
return new RuntimeConfigBundle(remoteConfigService, remoteConfig, configMergeResult, pendingMigrationTarget: null);
}
/// <summary>
/// Phase 6.5 — config self-heal. When the Phase 4 fetch ran before the network link was
/// up (agent relaunched at boot, Wi-Fi still associating — tenant aebdce78 audit 2026-08-24:
/// ~50% of sessions ran on ConfigVersion=0 exactly this way), a successful session
/// registration has just proven the backend reachable. Retry the fetch once and apply
/// the live tenant config instead of stranding the whole session on cache/defaults.
/// <para>
/// Deliberately NOT re-run here: the kill-signal check (post-registration kills arrive
/// via the telemetry-channel control signals on every upload response) and endpoint
/// migration (registration already succeeded against the current base URL; a migration
/// signal is honoured at the next agent start). Never throws — recovery is strictly
/// best-effort on top of an already-degraded-but-working fallback.
/// </para>
/// </summary>
public static void TryRecoverAfterRegistration(
AgentConfiguration agentConfig,
BackendAuthBundle auth,
RuntimeConfigBundle bundle,
string agentVersion,
bool consoleMode,
AgentLogger logger)
{
try
{
var service = bundle?.RemoteConfigService;
if (service == null) return;
var recovered = service.TryRecoverConfigAsync().GetAwaiter().GetResult();
if (recovered == null) return;
var mergeResult = ApplyFetchedConfig(agentConfig, recovered, auth, agentVersion, consoleMode, logger);
bundle.ApplyRecovered(recovered, mergeResult);
}
catch (Exception ex)
{
logger.Warning($"Post-registration config recovery failed: {ex.Message}");
}
}
/// <summary>
/// Applies a live-fetched config to the runtime: merge, tracker ceilings, log level,
/// integrity SHA + check, bootstrap-config cleanup. Shared by the Phase 4 startup path
/// and the post-registration recovery path so both apply identical semantics.
/// </summary>
private static RemoteConfigMergeResult ApplyFetchedConfig(
AgentConfiguration agentConfig,
AgentConfigResponse remoteConfig,
BackendAuthBundle auth,
string agentVersion,
bool consoleMode,
AgentLogger logger)
{
// Project remote tenant-controlled knobs onto the runtime AgentConfiguration so that
// downstream consumers (CleanupService, SummaryDialogLauncher, StartupEnvironmentProbes,
// DiagnosticsPackageService, EnrollmentTerminationHandler, the logger, the watchdog)
@@ -91,7 +148,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
TryCleanupBootstrapConfig(agentConfig, agentVersion, logger);
return new RuntimeConfigBundle(remoteConfigService, remoteConfig, configMergeResult, pendingMigrationTarget: null);
return configMergeResult;
}
/// <summary>
@@ -175,10 +232,21 @@ namespace AutopilotMonitor.Agent.V2.Runtime
internal sealed class RuntimeConfigBundle
{
public RemoteConfigService RemoteConfigService { get; }
public AgentConfigResponse RemoteConfig { get; }
public RemoteConfigMergeResult MergeResult { get; }
public AgentConfigResponse RemoteConfig { get; private set; }
public RemoteConfigMergeResult MergeResult { get; private set; }
public string PendingMigrationTarget { get; }
/// <summary>
/// Swaps in the config recovered by <see cref="AgentRuntimeConfig.TryRecoverAfterRegistration"/>
/// so Phase 7+8 (orchestrator, collectors, gather rules, IME patterns, the
/// unrestricted-mode audit) run on the live tenant config instead of the fallback.
/// </summary>
public void ApplyRecovered(AgentConfigResponse recoveredConfig, RemoteConfigMergeResult mergeResult)
{
RemoteConfig = recoveredConfig;
MergeResult = mergeResult;
}
public RuntimeConfigBundle(
RemoteConfigService remoteConfigService,
AgentConfigResponse remoteConfig,
@@ -52,6 +52,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
// prior to this, ConfigVersion=0 was only deducible by inferring from
// the absence of tenant-controlled knobs in the DataJson.
var outcome = remoteConfigService?.LastFetchOutcome ?? RemoteConfigFetchOutcome.NotAttempted;
var recovered = remoteConfigService?.RecoveredAfterFetchFailure == true;
var configVersion = remoteConfigService?.CurrentConfig?.ConfigVersion ?? 0;
var data = new Dictionary<string, object>
@@ -66,8 +67,12 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ "diagnosticsUploadMode", agentConfig.DiagnosticsUploadMode ?? "Off" },
{ "previousExitType", previousExit?.ExitType ?? "unknown" },
{ "unrestrictedMode", agentConfig.UnrestrictedMode },
// remoteConfigFetched answers "is the agent running on a live-fetched
// config" — true also when the post-registration recovery replaced a
// failed startup fetch (remoteConfigOutcome then still names the startup
// failure; remoteConfigRecovered marks the heal).
{ "configVersion", configVersion },
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded },
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded || recovered },
{ "remoteConfigOutcome", outcome.ToString() },
// WinRT OOBE state sampled at this start. Together with previousExitType
// this makes late starts provable: "completed" on a first_run means OOBE
@@ -75,6 +80,9 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ "oobeStateAtAgentStart", OobeStateReader.Read() },
};
if (recovered)
data["remoteConfigRecovered"] = true;
if (!string.IsNullOrEmpty(previousExit?.CrashExceptionType))
data["previousCrashException"] = previousExit.CrashExceptionType;
@@ -135,12 +143,25 @@ namespace AutopilotMonitor.Agent.V2.Runtime
? $"Remote config fetch failed after {remoteConfigService.LastFetchAttempts} attempt(s); using cached config."
: $"Remote config fetch failed after {remoteConfigService.LastFetchAttempts} attempt(s); running with built-in defaults (ConfigVersion=0).";
// Post-registration recovery (Phase 6.5) replaced the fallback with a live
// fetch before this emitter ran: keep the failure details (the network-late
// condition is worth surfacing) but mark the heal and demote to Info — the
// session is NOT degraded.
var recovered = remoteConfigService.RecoveredAfterFetchFailure;
if (recovered)
{
var recoveredVersion = remoteConfigService.CurrentConfig?.ConfigVersion ?? 0;
data["recovered"] = true;
data["recoveredConfigVersion"] = recoveredVersion;
msg += $" Recovered after session registration (ConfigVersion={recoveredVersion}).";
}
post.Emit(new EnrollmentEvent
{
SessionId = agentConfig.SessionId,
TenantId = agentConfig.TenantId,
EventType = SharedConstants.EventTypes.RemoteConfigFetchFailed,
Severity = EventSeverity.Warning,
Severity = recovered ? EventSeverity.Info : EventSeverity.Warning,
Source = "Agent",
Phase = EnrollmentPhase.Unknown,
Message = msg,
@@ -0,0 +1,330 @@
using System.Net;
using System.Security.Cryptography.X509Certificates;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using AutopilotMonitor.Shared.Security;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// CERT-TENANT-BINDING-SHADOW — stage 1 coverage.
///
/// The agent authenticates with the Intune MDM device certificate, whose chain is pinned to
/// Microsoft's Intune roots. Those roots are shared by every Intune tenant, so a valid chain alone
/// does not prove the caller belongs to the tenant it claims. The CA stamps the Entra tenant id
/// into OID 1.2.840.113556.5.14, which lets the backend bind the two.
///
/// These tests pin three things:
/// - the tenant id is actually extracted from a REAL field certificate (not a synthetic fixture),
/// - the pure comparison covers every outcome the telemetry can report,
/// - shadow mode observes without deciding: a cross-tenant certificate must still pass the
/// certificate stage in stage 1, or we would be enforcing before we measured.
/// </summary>
public class CertTenantBindingTests
{
private const string DeviceSamplePem = "device-cert-sample.pem";
/// <summary>
/// Tenant stamped into <c>device-cert-sample.pem</c> by the Intune MDM Device CA. The sample is
/// a public-key-only cert from an internal test device; the GUID is the test tenant's own id.
/// </summary>
private static readonly Guid SampleCertTenantId = Guid.Parse("b54dc1af-5320-4f60-b5d4-821e0cf2a359");
private static string SampleCertBase64()
{
var assemblyDir = Path.GetDirectoryName(typeof(CertTenantBindingTests).Assembly.Location)!;
var pem = File.ReadAllText(Path.Combine(assemblyDir, DeviceSamplePem));
using var cert = X509Certificate2.CreateFromPem(pem);
return Convert.ToBase64String(cert.Export(X509ContentType.Cert));
}
// ---------------------------------------------------------------- extraction
[Fact]
public void ValidateCertificate_RealIntuneCert_ExtractsEntraTenantId()
{
// The load-bearing test for the whole feature: if Microsoft ever stops stamping OID
// 1.2.840.113556.5.14, or changes its encoding, enforcement would lock every device out.
var result = CertificateValidator.ValidateCertificate(SampleCertBase64());
Assert.True(result.IsValid, $"Real device cert rejected: {result.ErrorMessage}");
Assert.Equal(CertTenantIdStatus.Present, result.CertTenantIdStatus);
Assert.Equal(SampleCertTenantId, result.CertTenantId);
}
[Fact]
public void ValidateCertificate_CachedResult_KeepsTenantIdAndFlagsCacheHit()
{
// FromCache is what samples the high-volume "Match" telemetry down to ~one line per cert
// per cache window, so it has to survive the cache and not corrupt the cached entry.
var b64 = SampleCertBase64();
_ = CertificateValidator.ValidateCertificate(b64);
var second = CertificateValidator.ValidateCertificate(b64);
var third = CertificateValidator.ValidateCertificate(b64);
Assert.True(second.FromCache);
Assert.Equal(SampleCertTenantId, second.CertTenantId);
// The cache-hit copy must not have mutated the shared cached instance.
Assert.True(third.FromCache);
Assert.Equal(SampleCertTenantId, third.CertTenantId);
Assert.True(third.IsValid);
}
// ---------------------------------------------------------------- encoding
[Fact]
public void TryParseGuid_PinsRealFieldEncodings()
{
// Exact bytes lifted from the sample cert. OID 5.14 nests the GUID in an OCTET STRING,
// OID 5.4 stores the 16 bytes bare — both shapes must decode, and both are little-endian.
var wrapped = Convert.FromHexString("0410afc14db52053604fb5d4821e0cf2a359");
var bare = Convert.FromHexString("6701bc076190b743b2430bb6aeda736e");
Assert.True(MsDeviceCertificateOids.TryParseGuid(wrapped, out var tenantId));
Assert.Equal(SampleCertTenantId, tenantId);
Assert.True(MsDeviceCertificateOids.TryParseGuid(bare, out var deviceId));
// The Intune device id is also the certificate's Subject CN.
Assert.Equal(Guid.Parse("07bc0167-9061-43b7-b243-0bb6aeda736e"), deviceId);
}
[Fact]
public void TryParseGuid_AccountIdOidIsNotTheTenantId()
{
// Guards the mistake this feature was nearly built on: OID 5.6 carries the Intune ACCOUNT
// id, a different GUID. Comparing it against the tenant id would reject every request.
var accountIdExt = Convert.FromHexString("04103d85dede74e26b409d0d46b6fe65b440");
Assert.True(MsDeviceCertificateOids.TryParseGuid(accountIdExt, out var accountId));
Assert.NotEqual(SampleCertTenantId, accountId);
}
[Theory]
[InlineData("")] // empty
[InlineData("0400")] // OCTET STRING, zero length
[InlineData("020102")] // not an OCTET STRING
[InlineData("0411afc14db52053604fb5d4821e0cf2a35900")] // wrapper claims 17 bytes
[InlineData("0410afc14db52053604fb5d4821e0cf2a3")] // wrapper claims 16, only 15 present
public void TryParseGuid_RejectsMalformed(string hex)
{
Assert.False(MsDeviceCertificateOids.TryParseGuid(Convert.FromHexString(hex), out _));
}
[Fact]
public void TryParseGuid_AnySixteenBytesDecodeAsBareGuid()
{
// Documents a deliberate limit rather than a gap: the bare form (OID 5.4) is exactly 16
// bytes with no tag, so a 16-byte input is indistinguishable from a truncated wrapper and
// is always read as a GUID. Harmless here — the value is only ever compared for equality
// against a known tenant id, never trusted as a claim in its own right.
Assert.True(MsDeviceCertificateOids.TryParseGuid(
Convert.FromHexString("0410afc14db52053604fb5d4821e0cf2"), out _));
}
[Fact]
public void TryParseGuid_RejectsNull()
{
Assert.False(MsDeviceCertificateOids.TryParseGuid(null!, out _));
}
// ---------------------------------------------------------------- comparison
[Fact]
public void Evaluate_MatchingTenant_ReturnsMatch()
{
var outcome = CertTenantBinding.Evaluate(
SampleCertTenantId, CertTenantIdStatus.Present, SampleCertTenantId.ToString());
Assert.Equal(CertTenantBinding.Outcome.Match, outcome);
Assert.False(CertTenantBinding.WouldRejectUnderEnforcement(outcome));
}
[Fact]
public void Evaluate_MatchingTenant_IsCaseInsensitive()
{
// Tenant ids travel as strings in headers/routes and casing must not decide access.
var outcome = CertTenantBinding.Evaluate(
SampleCertTenantId, CertTenantIdStatus.Present, SampleCertTenantId.ToString().ToUpperInvariant());
Assert.Equal(CertTenantBinding.Outcome.Match, outcome);
}
[Fact]
public void Evaluate_ForeignTenantCert_ReturnsMismatch()
{
// The attack this feature exists for: a valid Intune cert from the attacker's own tenant,
// replayed against a victim tenant whose device serial the attacker knows.
var outcome = CertTenantBinding.Evaluate(
SampleCertTenantId, CertTenantIdStatus.Present, "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee");
Assert.Equal(CertTenantBinding.Outcome.Mismatch, outcome);
Assert.True(CertTenantBinding.WouldRejectUnderEnforcement(outcome));
}
[Theory]
[InlineData(CertTenantIdStatus.ExtensionMissing, CertTenantBinding.Outcome.ExtensionMissing)]
[InlineData(CertTenantIdStatus.Unparseable, CertTenantBinding.Outcome.Unparseable)]
public void Evaluate_UndecodableCert_ReportsWhyWithoutClaimingMismatch(
CertTenantIdStatus status, string expected)
{
// These must stay distinguishable from Mismatch: they measure rollout readiness
// (how many field certs can be enforced against), not an attack.
var outcome = CertTenantBinding.Evaluate(null, status, SampleCertTenantId.ToString());
Assert.Equal(expected, outcome);
Assert.False(CertTenantBinding.WouldRejectUnderEnforcement(outcome));
}
[Fact]
public void Evaluate_PresentButNullGuid_FallsBackToExtensionMissing()
{
var outcome = CertTenantBinding.Evaluate(null, CertTenantIdStatus.Present, SampleCertTenantId.ToString());
Assert.Equal(CertTenantBinding.Outcome.ExtensionMissing, outcome);
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData("not-a-guid")]
public void Evaluate_NonGuidRequestTenant_IsReportedSeparately(string? requestedTenantId)
{
var outcome = CertTenantBinding.Evaluate(
SampleCertTenantId, CertTenantIdStatus.Present, requestedTenantId);
Assert.Equal(CertTenantBinding.Outcome.RequestTenantNotAGuid, outcome);
Assert.False(CertTenantBinding.WouldRejectUnderEnforcement(outcome));
}
// ---------------------------------------------------------------- shadow mode
[Fact]
public async Task ValidateRequestAsync_ForeignTenantCert_IsObservedButNotRejected()
{
// Stage 1 contract: a cross-tenant certificate is logged and still passes the certificate
// stage. If this test ever fails, enforcement was switched on without the telemetry review.
var logger = new CapturingLogger();
var validator = BuildValidator(logger);
var result = await validator.ValidateRequestAsync(
BuildRequestWithCert(SampleCertBase64()),
tenantId: "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee");
// Deliberately accepted: stage 1 measures, it does not enforce. This assertion is the one
// that must be inverted (to a 403) when stage 2 turns enforcement on.
Assert.True(result.IsValid, $"Shadow mode changed the outcome: {result.ErrorMessage} / {result.Details}");
Assert.NotEqual(HttpStatusCode.Unauthorized, result.StatusCode);
var observed = Assert.Single(logger.Entries, e => e.Message.Contains("AgentCertTenantBinding"));
Assert.Equal(LogLevel.Warning, observed.Level);
Assert.Contains(CertTenantBinding.Outcome.Mismatch, observed.Message);
Assert.Contains("enforced=False", observed.Message);
Assert.Contains("wouldReject=True", observed.Message);
}
[Fact]
public async Task ValidateRequestAsync_MatchingTenantCert_IsAccepted()
{
var logger = new CapturingLogger();
var validator = BuildValidator(logger);
var result = await validator.ValidateRequestAsync(
BuildRequestWithCert(SampleCertBase64()),
tenantId: SampleCertTenantId.ToString());
Assert.True(result.IsValid, $"Matching-tenant request rejected: {result.ErrorMessage}");
// Match is only logged on a fresh validation; the cache is process-wide and shared with the
// other tests here, so assert the outcome only when a line was actually emitted.
var observed = logger.Entries.FirstOrDefault(e => e.Message.Contains("AgentCertTenantBinding"));
if (observed != null)
{
Assert.Equal(LogLevel.Information, observed.Level);
Assert.Contains(CertTenantBinding.Outcome.Match, observed.Message);
}
}
// ---------------------------------------------------------------- harness
private static SecurityValidator BuildValidator(ILogger logger)
{
var configRepo = Mock.Of<IConfigRepository>();
var cache = new MemoryCache(new MemoryCacheOptions());
// AllowInsecureAgentRequests clears the "at least one device validator" gate so the request
// reaches — and passes — the certificate stage without needing Graph mocks.
var config = new TenantConfiguration
{
AllowInsecureAgentRequests = true,
ManufacturerWhitelist = "*",
ModelWhitelist = "*",
};
var configServiceMock = new Mock<TenantConfigurationService>(
configRepo, Mock.Of<ILogger<TenantConfigurationService>>(), cache)
{ CallBase = false };
configServiceMock
.Setup(x => x.TryGetConfigurationAsync(It.IsAny<string>()))
.ReturnsAsync((config, true));
var adminConfigServiceMock = new Mock<AdminConfigurationService>(
configRepo, Mock.Of<ILogger<AdminConfigurationService>>(), cache)
{ CallBase = false };
adminConfigServiceMock
.Setup(x => x.GetConfigurationAsync())
.ReturnsAsync(new AdminConfiguration());
return new SecurityValidator(
configService: configServiceMock.Object,
adminConfigService: adminConfigServiceMock.Object,
rateLimitService: new RateLimitService(cache, Mock.Of<ILogger<RateLimitService>>()),
autopilotDeviceValidator: null!,
corporateIdentifierValidator: null!,
logger: logger,
bootstrapSessionService: null,
deviceAssociationValidator: null);
}
private static HttpRequestData BuildRequestWithCert(string certBase64)
{
var contextMock = new Mock<Microsoft.Azure.Functions.Worker.FunctionContext>();
var reqMock = new Mock<HttpRequestData>(contextMock.Object);
var headers = new HttpHeadersCollection
{
{ "X-ARR-ClientCert", certBase64 },
// The hardware whitelist stage runs after the certificate stage and rejects requests
// with no hardware headers, which would mask what these tests are actually asserting.
{ "X-Device-Manufacturer", "Contoso" },
{ "X-Device-Model", "TestBook" },
{ "X-Device-SerialNumber", "TESTSERIAL01" },
};
reqMock.SetupGet(r => r.Headers).Returns(headers);
reqMock.SetupGet(r => r.Url).Returns(new Uri("https://example.invalid/api/telemetry"));
return reqMock.Object;
}
private sealed record LogEntry(LogLevel Level, string Message);
private sealed class CapturingLogger : ILogger
{
public List<LogEntry> Entries { get; } = new();
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(LogLevel logLevel, EventId eventId, TState state, Exception? exception,
Func<TState, Exception?, string> formatter)
=> Entries.Add(new LogEntry(logLevel, formatter(state, exception)));
}
}
@@ -0,0 +1,99 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Functions.Services;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// <see cref="TableStorageService.BuildCveIndexEntities"/> collapses report findings to one
/// CveIndex entity per CVE. Production reports routinely list the same CVE under many
/// findings (one RD CVE across 14 installed RD components), which previously produced
/// 14 concurrent upserts racing on the same PK/RK.
/// </summary>
public class CveIndexEntityMergeTests
{
private const string Tenant = "11111111-1111-1111-1111-111111111111";
private const string Session = "22222222-2222-2222-2222-222222222222";
private static Dictionary<string, object> Finding(string software, string risk, params Dictionary<string, object>[] vulns)
=> new()
{
["softwareName"] = software,
["riskLevel"] = risk,
["vulnerabilities"] = vulns.ToList(),
};
private static Dictionary<string, object> Vuln(string cve, double score, string severity, bool kev = false)
=> new()
{
["cveId"] = cve,
["cvssScore"] = score,
["cvssSeverity"] = severity,
["isKev"] = kev,
};
[Fact]
public void SameCveAcrossManyFindings_CollapsesToOneEntity()
{
var findings = Enumerable.Range(0, 14)
.Select(i => Finding($"RD Component {i}", "high", Vuln("CVE-2025-29966", 8.8, "HIGH")))
.ToList();
var entities = TableStorageService.BuildCveIndexEntities(Tenant, Session, findings);
var e = Assert.Single(entities);
Assert.Equal($"{Tenant}_CVE-2025-29966", e.PartitionKey);
Assert.Equal(Session, e.RowKey);
Assert.Equal("RD Component 0", e.GetString("SoftwareName")); // tie keeps report order
}
[Fact]
public void Merge_HighestScoreWins_KevOrs_RiskMaxes()
{
var findings = new List<Dictionary<string, object>>
{
Finding("App A", "medium", Vuln("CVE-1", 5.0, "MEDIUM", kev: true)),
Finding("App B", "critical", Vuln("CVE-1", 9.1, "CRITICAL")),
Finding("App C", "low", Vuln("CVE-1", 3.0, "LOW")),
};
var e = Assert.Single(TableStorageService.BuildCveIndexEntities(Tenant, Session, findings));
Assert.Equal(9.1, e.GetDouble("CvssScore"));
Assert.Equal("CRITICAL", e.GetString("CvssSeverity"));
Assert.Equal("App B", e.GetString("SoftwareName"));
Assert.True(e.GetBoolean("IsKev"));
Assert.Equal("critical", e.GetString("OverallRisk"));
}
[Fact]
public void DistinctCves_StayDistinct_AndInvalidEntriesAreSkipped()
{
var findings = new List<Dictionary<string, object>>
{
Finding("App A", "high", Vuln("CVE-1", 7.0, "HIGH"), Vuln("CVE-2", 6.0, "MEDIUM")),
Finding("App B", "high", new Dictionary<string, object> { ["cvssScore"] = 9.0 }), // no cveId
new() { ["softwareName"] = "No vulns" },
};
var entities = TableStorageService.BuildCveIndexEntities(Tenant, Session, findings);
Assert.Equal(2, entities.Count);
Assert.Contains(entities, x => x.GetString("CveId") == "CVE-1");
Assert.Contains(entities, x => x.GetString("CveId") == "CVE-2");
}
[Fact]
public void CveIdComparison_IsCaseInsensitive()
{
var findings = new List<Dictionary<string, object>>
{
Finding("A", "high", Vuln("CVE-2024-1", 7.0, "HIGH")),
Finding("B", "high", Vuln("cve-2024-1", 7.0, "HIGH")),
};
Assert.Single(TableStorageService.BuildCveIndexEntities(Tenant, Session, findings));
}
}
@@ -163,6 +163,12 @@ public class EnrollmentTimeoutClassifierTests
// -------- ClassifyTimedOutSession --------
private static (SessionStatus, string) Classify(IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 6, int grace = 72)
{
var (status, reason, _) = ClassifyWithRule(events, hoursSinceStart, grace);
return (status, reason);
}
private static (SessionStatus, string, string) ClassifyWithRule(IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 6, int grace = 72)
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var now = Start.AddHours(hoursSinceStart);
@@ -220,7 +226,7 @@ public class EnrollmentTimeoutClassifierTests
Esp(DeviceSetup44), Esp(AccountSetup15),
Evt("desktop_arrived"), Evt("hello_provisioning_completed"), Evt("realmjoin_detected"),
});
var (status, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
var (status, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, now, graceHours: 72, lastEventAtUtc: lastEvent);
Assert.Equal(SessionStatus.Succeeded, status);
@@ -481,8 +487,9 @@ public class EnrollmentTimeoutClassifierTests
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var now = Start.AddHours(hoursSinceStart);
return EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
var (status, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, now, grace, isPreProvisioned: true, resumedAt: Resumed);
return (status, reason);
}
/// <summary>The fairstone event shape: Part 1 sealed, Part 2 resumed, nobody signed in.</summary>
@@ -611,7 +618,7 @@ public class EnrollmentTimeoutClassifierTests
{
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup05),
});
var (status, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
var (status, _, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, Start.AddHours(6), 51, isPreProvisioned: true, resumedAt: null);
Assert.Equal(SessionStatus.Succeeded, status);
}
@@ -622,9 +629,10 @@ public class EnrollmentTimeoutClassifierTests
IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 3, int grace = 51)
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
return EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
var (status, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, Start.AddHours(hoursSinceStart), grace,
lastEventAtUtc: Start.AddMinutes(4), isSelfDeployingProfile: true);
return (status, reason);
}
// Session 195593e2 replay: DeviceSetup 4/4 (fallback-confirmed), AccountSetup registry
@@ -675,8 +683,8 @@ public class EnrollmentTimeoutClassifierTests
// Guard against the gate leaking into user-driven sessions: the identical event shape
// on a user-driven profile is still "awaiting user" within grace / Incomplete past it.
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(KioskSilentStream);
var (within, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(3), 51);
var (past, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(60), 51);
var (within, _, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(3), 51);
var (past, _, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(60), 51);
Assert.Equal(SessionStatus.AwaitingUser, within);
Assert.Equal(SessionStatus.Incomplete, past);
}
@@ -0,0 +1,166 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins that the ONE projected window scan the maintenance tick shares across its rolling
/// sweeps (<c>GetMaintenanceWindowSessionsAsync</c> / <c>MaintenanceSweepSessionProjection</c>)
/// carries every field the time-attribution, device-journey and verdict-calibration sweeps
/// read. In Azure Table Storage a <c>$select</c> returns ONLY the listed properties, so the
/// projected row here is built from the production array — dropping a column there fails the
/// sweep that consumes it. Mirrors <see cref="UsageMetricsProjectionEquivalenceTests"/>.
/// </summary>
public class MaintenanceSweepProjectionEquivalenceTests
{
private const string TenantId = "00000000-0000-0000-0000-000000000abc";
private static readonly DateTime Started = new(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
private static readonly TableStorageService Sut =
new(new Mock<TableServiceClient>().Object, NullLogger<TableStorageService>.Instance);
/// <summary>Full wide Sessions row with every column the sweeps read set to a distinct non-default value, plus noise.</summary>
private static TableEntity FullRow() => new(TenantId, "11111111-1111-1111-1111-111111111111")
{
["Status"] = "Succeeded",
["StartedAt"] = new DateTimeOffset(Started),
["CompletedAt"] = new DateTimeOffset(Started.AddHours(1)),
["LastEventAt"] = new DateTimeOffset(Started.AddHours(2)),
["DurationSeconds"] = 3600,
["EventCount"] = 321,
["DeletionState"] = "None",
["SerialNumber"] = "SN-1234",
["Manufacturer"] = "Contoso",
["Model"] = "Model-X",
["EnrollmentType"] = "v2",
["IsPreProvisioned"] = true,
["IsSelfDeployingProfile"] = true,
["IsUserDriven"] = false,
["ResumedAt"] = new DateTimeOffset(Started.AddMinutes(30)),
["AdminMarkedAction"] = "Succeeded",
["VerdictPath"] = "manual:succeeded",
["PriorStatus"] = "Incomplete",
["PriorVerdictPath"] = "sweep:r6",
["FailureReason"] = "No Device Setup completion or explicit failure signal observed before timeout",
["ReconcileReason"] = "Reconciled at timeout: user completed setup (desktop + Windows Hello)",
["FailureSource"] = "manual",
["EspSoftFailure"] = true,
// Noise the sweeps never read — present on the wide row, dropped by the projection.
["DeviceName"] = "PC-FULL",
["OsName"] = "Windows 11",
["OsBuild"] = "26200.1234",
["GeoCountry"] = "DE",
["AgentVersion"] = "2.9.0",
["DiagnosticsBlobName"] = "diag.zip",
["FailureSnapshotJson"] = "{\"big\":\"" + new string('x', 2000) + "\"}",
};
private static TableEntity Project(TableEntity full)
{
var keep = new HashSet<string>(TableStorageService.MaintenanceSweepSessionProjection, StringComparer.Ordinal);
var projected = new TableEntity(full.PartitionKey, full.RowKey);
foreach (var kv in full)
{
if (keep.Contains(kv.Key)) projected[kv.Key] = kv.Value;
}
return projected;
}
[Fact]
public void Projection_drops_the_wide_columns()
{
var projected = Project(FullRow());
Assert.False(projected.ContainsKey("FailureSnapshotJson"));
Assert.False(projected.ContainsKey("GeoCountry"));
Assert.False(projected.ContainsKey("OsBuild"));
Assert.True(projected.Count < FullRow().Count);
}
[Fact]
public void Every_field_the_sweeps_read_survives_the_projection()
{
var full = Sut.MapToSessionSummary(FullRow());
var proj = Sut.MapToSessionSummary(Project(FullRow()));
// Shared / structural
Assert.Equal(full.TenantId, proj.TenantId);
Assert.Equal(full.SessionId, proj.SessionId);
Assert.Equal(full.Status, proj.Status);
Assert.Equal(full.StartedAt, proj.StartedAt);
Assert.Equal(full.CompletedAt, proj.CompletedAt);
Assert.Equal(full.DeletionState, proj.DeletionState);
// Time attribution: Status/DurationSeconds/DeletionState/EventCount + enrollment class
Assert.Equal(full.DurationSeconds, proj.DurationSeconds);
Assert.Equal(full.EventCount, proj.EventCount);
Assert.Equal(TimeAttributionCalculator.GetEnrollmentClass(full), TimeAttributionCalculator.GetEnrollmentClass(proj));
// Device journeys: BuildSessionRef + BuildDeviceHistoryRow display fields
Assert.Equal(full.SerialNumber, proj.SerialNumber);
Assert.Equal(full.Manufacturer, proj.Manufacturer);
Assert.Equal(full.Model, proj.Model);
Assert.Equal(full.EnrollmentType, proj.EnrollmentType);
Assert.Equal(full.IsPreProvisioned, proj.IsPreProvisioned);
Assert.Equal(full.AdminMarkedAction, proj.AdminMarkedAction);
var refFull = DeviceJourneyCalculator.BuildSessionRef(full)!;
var refProj = DeviceJourneyCalculator.BuildSessionRef(proj)!;
Assert.Equal(System.Text.Json.JsonSerializer.Serialize(refFull), System.Text.Json.JsonSerializer.Serialize(refProj));
// Verdict calibration: attribution + legacy derivation inputs + proxy timestamps
Assert.Equal(full.LastEventAt, proj.LastEventAt);
Assert.Equal(full.VerdictPath, proj.VerdictPath);
Assert.Equal(full.PriorStatus, proj.PriorStatus);
Assert.Equal(full.PriorVerdictPath, proj.PriorVerdictPath);
Assert.Equal(full.FailureReason, proj.FailureReason);
Assert.Equal(full.ReconcileReason, proj.ReconcileReason);
Assert.Equal(full.FailureSource, proj.FailureSource);
Assert.Equal(full.EspSoftFailure, proj.EspSoftFailure);
Assert.Equal(VerdictPathDerivation.Derive(full), VerdictPathDerivation.Derive(proj));
// Unstamped variant exercises the derivation on the projected inputs.
var unstampedFull = FullRow(); unstampedFull.Remove("VerdictPath"); unstampedFull.Remove("AdminMarkedAction"); unstampedFull.Remove("FailureSource");
Assert.Equal(VerdictPathDerivation.Derive(Sut.MapToSessionSummary(unstampedFull)),
VerdictPathDerivation.Derive(Sut.MapToSessionSummary(Project(unstampedFull))));
}
[Fact]
public void Aggregation_cores_produce_identical_rows_from_the_projected_scan()
{
var now = new DateTime(2026, 8, 23, 12, 0, 0, DateTimeKind.Utc);
var fullList = new List<SessionSummary> { Sut.MapToSessionSummary(FullRow()) };
var projList = new List<SessionSummary> { Sut.MapToSessionSummary(Project(FullRow())) };
var histories = new Dictionary<string, List<DeviceHistory>>();
var a = MaintenanceService.BuildVerdictCalibrationAggregates(fullList, histories, now);
var b = MaintenanceService.BuildVerdictCalibrationAggregates(projList, histories, now);
Assert.Equal(System.Text.Json.JsonSerializer.Serialize(a), System.Text.Json.JsonSerializer.Serialize(b));
}
[Fact]
public void SliceSweepWindow_keeps_sessions_from_the_window_start()
{
var window = new List<SessionSummary>
{
new() { SessionId = "old", StartedAt = Started.AddDays(-40) },
new() { SessionId = "edge", StartedAt = Started },
new() { SessionId = "new", StartedAt = Started.AddDays(1) },
};
var sliced = MaintenanceService.SliceSweepWindow(window, Started);
Assert.Equal(new[] { "edge", "new" }, sliced.Select(s => s.SessionId));
}
[Fact]
public void Shared_window_covers_every_sweeps_horizon()
{
// 7d window + 28d baseline anchored on yesterday = 35 days back ≥ the 30-day rolling sweeps.
Assert.Equal(35, MaintenanceService.SweepWindowDays);
Assert.True(MaintenanceService.SweepWindowDays >= 30);
}
}
@@ -112,6 +112,7 @@ public class OffboardingSessionEnumeratorTests
public Task<List<SessionSummary>> GetSessionsOlderThanAsync(string tenantId, DateTime cutoffDate, int maxResults = int.MaxValue, bool excludeInFlightDeletions = false) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetSessionsByDateRangeAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetUsageWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetMaintenanceWindowSessionsAsync(DateTime startDate, DateTime endDate) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetStalledSessionsAsync(string tenantId, DateTime cutoffTime) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
@@ -0,0 +1,76 @@
using System.Collections.Concurrent;
using AutopilotMonitor.Functions.Services;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Both sliding-window limiters keep the request history and its lock in one cache entry.
/// Previously the entry was re-Set without its eviction callback, which (a) leaked per-key lock
/// objects and (b) let two threads mutate the same List under different locks. These tests hammer
/// one key from many threads: the count of allowed requests must be exactly the limit and nothing
/// may throw (a List mutated concurrently throws InvalidOperationException or corrupts Count).
/// </summary>
public class RateLimitServiceConcurrencyTests
{
private const int Threads = 16;
private const int RequestsPerThread = 500;
[Fact]
public void CheckRateLimit_ManyThreadsSameKey_AllowsExactlyTheLimit()
{
var sut = new RateLimitService(new MemoryCache(new MemoryCacheOptions()), NullLogger<RateLimitService>.Instance);
const int limit = 1000;
var allowed = 0;
var errors = new ConcurrentBag<Exception>();
RunInParallel(() =>
{
try
{
if (sut.CheckRateLimit("thumb-shared", limit).IsAllowed)
Interlocked.Increment(ref allowed);
}
catch (Exception ex) { errors.Add(ex); }
});
Assert.Empty(errors);
Assert.Equal(limit, allowed);
Assert.False(sut.CheckRateLimit("thumb-shared", limit).IsAllowed);
}
[Fact]
public void DistressCheck_ManyThreadsSameIp_AllowsExactlyTheIpLimit()
{
var sut = new DistressRateLimitService(new MemoryCache(new MemoryCacheOptions()), NullLogger<DistressRateLimitService>.Instance);
var allowed = 0;
var errors = new ConcurrentBag<Exception>();
RunInParallel(() =>
{
try
{
if (sut.Check("10.0.0.9", "a1b2c3d4-e5f6-7890-abcd-ef1234567890").IsAllowed)
Interlocked.Increment(ref allowed);
}
catch (Exception ex) { errors.Add(ex); }
});
Assert.Empty(errors);
Assert.Equal(5, allowed); // MaxPerIp
}
private static void RunInParallel(Action body)
{
using var gate = new ManualResetEventSlim(false);
var tasks = Enumerable.Range(0, Threads).Select(_ => Task.Run(() =>
{
gate.Wait();
for (var i = 0; i < RequestsPerThread; i++) body();
})).ToArray();
gate.Set();
Task.WaitAll(tasks);
}
}
@@ -86,6 +86,9 @@ public class SessionIndexFieldManifestTests
["ValidatedBy"] = "AutopilotV1",
["FailureSnapshotJson"] = "{\"x\":1}",
["AdminMarkedAction"] = "failed",
["VerdictPath"] = "sweep:r5_incomplete",
["PriorStatus"] = "Incomplete",
["PriorVerdictPath"] = "maxlife:r6",
["DurationSeconds"] = 360,
["DiagnosticsBlobName"] = "diag-0001.zip",
["DiagnosticsBlobDestination"] = "Hosted",
@@ -74,7 +74,7 @@ public class SessionStartedAtBackwardShiftGuardTests
var harness = new Harness(SessionRow(), earliestStoredEvent: poisoned);
var ok = await harness.Sut.UpdateSessionStatusAsync(
TenantId, SessionId, SessionStatus.Pending,
TenantId, SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
earliestEventTimestamp: poisoned, latestEventTimestamp: lastEvent);
Assert.True(ok);
@@ -93,7 +93,7 @@ public class SessionStartedAtBackwardShiftGuardTests
var harness = new Harness(SessionRow(), earliestStoredEvent: earliest);
var ok = await harness.Sut.UpdateSessionStatusAsync(
TenantId, SessionId, SessionStatus.Pending,
TenantId, SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
latestEventTimestamp: lastEvent);
Assert.True(ok);
@@ -132,7 +132,7 @@ public class SessionStartedAtBackwardShiftGuardTests
});
var ok = await harness.Sut.UpdateSessionStatusAsync(
TenantId, SessionId, SessionStatus.Pending,
TenantId, SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
latestEventTimestamp: lastEvent);
Assert.True(ok);
@@ -0,0 +1,80 @@
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using Azure;
using Azure.Data.Tables;
using Azure.Data.Tables.Models;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests
{
public class StartupTelemetryServiceTests
{
private sealed class CapturingChannel : ITelemetryChannel
{
public List<ITelemetry> Items { get; } = new();
public bool? DeveloperMode { get; set; }
public string EndpointAddress { get; set; } = "";
public void Send(ITelemetry item) => Items.Add(item);
public void Flush() { }
public void Dispose() { }
}
private sealed class Lifetime : IHostApplicationLifetime
{
private readonly CancellationTokenSource _started = new();
public CancellationToken ApplicationStarted => _started.Token;
public CancellationToken ApplicationStopping => CancellationToken.None;
public CancellationToken ApplicationStopped => CancellationToken.None;
public void StopApplication() { }
public void FireStarted() => _started.Cancel();
}
[Fact]
public async Task ApplicationStarted_EmitsStartupAndTableInitMetrics()
{
var channel = new CapturingChannel();
var telemetry = new TelemetryClient(new TelemetryConfiguration { TelemetryChannel = channel, ConnectionString = "InstrumentationKey=00000000-0000-0000-0000-000000000000" });
// Table storage: unreadable sentinel → full pass → FullPassRan = true.
var service = new Mock<TableServiceClient>();
var admin = new Mock<TableClient>();
service.Setup(s => s.GetTableClient(It.IsAny<string>())).Returns(admin.Object);
admin.Setup(c => c.GetEntityIfExistsAsync<TableEntity>(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<string>>(), It.IsAny<CancellationToken>()))
.ThrowsAsync(new RequestFailedException(404, "TableNotFound"));
admin.Setup(c => c.UpsertEntityAsync(It.IsAny<TableEntity>(), It.IsAny<TableUpdateMode>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(Mock.Of<Response>());
service.Setup(s => s.CreateTableIfNotExistsAsync(It.IsAny<string>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(Response.FromValue(new TableItem("x"), Mock.Of<Response>()));
var storage = new TableStorageService(service.Object, NullLogger<TableStorageService>.Instance);
await storage.InitializeTablesAsync();
var lifetime = new Lifetime();
var sut = new StartupTelemetryService(lifetime, telemetry, storage, new BackendBuildInfo(), NullLogger<StartupTelemetryService>.Instance);
await sut.StartAsync(CancellationToken.None);
Assert.Empty(channel.Items);
lifetime.FireStarted();
var metrics = channel.Items.ConvertAll(i => (MetricTelemetry)i);
Assert.Equal(2, metrics.Count);
var startup = metrics.Find(m => m.Name == StartupTelemetryService.StartupMetricName)!;
var tableInit = metrics.Find(m => m.Name == StartupTelemetryService.TableInitMetricName)!;
Assert.True(startup.Sum > 0);
Assert.True(tableInit.Sum >= 0);
Assert.Equal("True", startup.Properties["tableInitFullPass"]);
Assert.False(string.IsNullOrEmpty(startup.Properties["version"]));
Assert.True(startup.Sum >= tableInit.Sum);
}
}
}
@@ -48,11 +48,74 @@ public class StorageDependencyFilterProcessorTests
Assert.Empty(next.Received);
}
[Fact]
public void FailedStorageDependency_IsKept()
[Theory]
[InlineData("429")]
[InlineData("500")]
[InlineData("503")]
[InlineData("403")]
[InlineData("")] // unknown outcome (timeout/network) — keep
public void FailedStorageDependency_IsKept(string resultCode)
{
var (processor, next) = Build();
var dep = Dep("myacct.queue.core.windows.net", success: false);
dep.ResultCode = resultCode;
processor.Process(dep);
Assert.Same(dep, Assert.Single(next.Received));
}
// Expected outcomes (point-read miss, ETag precondition, insert conflict) are normal control
// flow — live-verified as 100% of the remaining "failed" storage rows. Dropped.
[Theory]
[InlineData("404")]
[InlineData("412")]
[InlineData("409")]
public void ExpectedStorageOutcome_HttpShape_IsDropped(string resultCode)
{
var (processor, next) = Build();
var dep = Dep("myacct.table.core.windows.net", success: false);
dep.ResultCode = resultCode;
processor.Process(dep);
Assert.Empty(next.Received);
}
[Theory]
[InlineData("Azure.RequestFailedException: The specified resource does not exist.\nRequestId:x\nStatus: 404 (Not Found)\nErrorCode: ResourceNotFound")]
[InlineData("Azure.RequestFailedException: The update condition specified in the request was not satisfied.\nStatus: 412 (Precondition Failed)\nErrorCode: UpdateConditionNotSatisfied")]
[InlineData("Azure.RequestFailedException: The specified entity already exists.\nStatus: 409 (Conflict)\nErrorCode: EntityAlreadyExists")]
public void ExpectedStorageOutcome_InProcShape_IsDropped(string error)
{
// InProc spans have no ResultCode — only the exception text in Properties["Error"].
var (processor, next) = Build();
var dep = Dep(target: "TableClient.GetEntity", success: false, type: "InProc | Microsoft.Tables");
dep.Properties["Error"] = error;
processor.Process(dep);
Assert.Empty(next.Received);
}
[Fact]
public void ThrottledInProcStorageDependency_IsKept()
{
var (processor, next) = Build();
var dep = Dep(target: "TableClient.SubmitTransaction", success: false, type: "InProc | Microsoft.Tables");
dep.Properties["Error"] = "Azure.RequestFailedException: Too many requests.\nStatus: 429 (Too Many Requests)\nErrorCode: ServerBusy";
processor.Process(dep);
Assert.Same(dep, Assert.Single(next.Received));
}
[Fact]
public void SuccessfulSignalRRestCall_IsDropped()
{
var (processor, next) = Build();
processor.Process(Dep("autopilotmonitor-eu.service.signalr.net", success: true, data: "POST /api/hubs/autopilotmonitor/groups/t-x/connections/y", type: "HTTP"));
Assert.Empty(next.Received);
}
[Fact]
public void FailedSignalRRestCall_IsKept()
{
var (processor, next) = Build();
var dep = Dep("autopilotmonitor-eu.service.signalr.net", success: false, type: "HTTP");
dep.ResultCode = "500";
processor.Process(dep);
Assert.Same(dep, Assert.Single(next.Received));
}
@@ -0,0 +1,180 @@
using System;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using Azure;
using Azure.Data.Tables;
using Azure.Data.Tables.Models;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests
{
/// <summary>
/// Startup table initialization is gated by a schema sentinel (hash over TableNames.All)
/// stored in AdminConfiguration: a matching sentinel means zero CreateTableIfNotExists
/// calls on a cold start; anything else runs the full idempotent pass and rewrites it.
/// </summary>
public class TableSchemaSentinelTests
{
private sealed class Harness
{
public Mock<TableServiceClient> Service { get; } = new();
public Mock<TableClient> AdminConfig { get; } = new();
public TableEntity? UpsertedSentinel { get; private set; }
public int CreateCalls;
public Harness()
{
Service.Setup(s => s.GetTableClient(Constants.TableNames.AdminConfiguration)).Returns(AdminConfig.Object);
Service.Setup(s => s.CreateTableIfNotExistsAsync(It.IsAny<string>(), It.IsAny<CancellationToken>()))
.Returns<string, CancellationToken>((_, _) =>
{
Interlocked.Increment(ref CreateCalls);
return Task.FromResult(Response.FromValue(new TableItem("x"), Mock.Of<Response>()));
});
AdminConfig.Setup(c => c.UpsertEntityAsync(It.IsAny<TableEntity>(), It.IsAny<TableUpdateMode>(), It.IsAny<CancellationToken>()))
.Callback<ITableEntity, TableUpdateMode, CancellationToken>((e, _, _) => UpsertedSentinel = (TableEntity)e)
.ReturnsAsync(Mock.Of<Response>());
}
public void SentinelReturns(string? hash)
{
var response = new Mock<NullableResponse<TableEntity>>();
response.SetupGet(r => r.HasValue).Returns(hash != null);
if (hash != null)
{
response.SetupGet(r => r.Value).Returns(new TableEntity(
TableStorageService.SchemaSentinelPartitionKey, TableStorageService.SchemaSentinelRowKey)
{
[TableStorageService.SchemaSentinelHashProperty] = hash
});
}
AdminConfig.Setup(c => c.GetEntityIfExistsAsync<TableEntity>(
TableStorageService.SchemaSentinelPartitionKey, TableStorageService.SchemaSentinelRowKey,
It.IsAny<System.Collections.Generic.IEnumerable<string>>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(response.Object);
}
public void SentinelThrows()
{
AdminConfig.Setup(c => c.GetEntityIfExistsAsync<TableEntity>(
It.IsAny<string>(), It.IsAny<string>(),
It.IsAny<System.Collections.Generic.IEnumerable<string>>(), It.IsAny<CancellationToken>()))
.ThrowsAsync(new RequestFailedException(404, "TableNotFound"));
}
public TableStorageService Build() => new(Service.Object, NullLogger<TableStorageService>.Instance);
}
private static string CurrentHash => TableStorageService.ComputeTableSchemaHash(Constants.TableNames.All);
[Fact]
public void Hash_IsOrderIndependent_AndChangesWhenTableAdded()
{
var a = TableStorageService.ComputeTableSchemaHash(new[] { "B", "A" });
var b = TableStorageService.ComputeTableSchemaHash(new[] { "A", "B" });
var c = TableStorageService.ComputeTableSchemaHash(new[] { "A", "B", "C" });
Assert.Equal(a, b);
Assert.NotEqual(a, c);
Assert.Equal(64, a.Length);
}
[Fact]
public async Task MatchingSentinel_SkipsTableCreation()
{
var h = new Harness();
h.SentinelReturns(CurrentHash);
var fullPass = await h.Build().InitializeTablesAsync();
Assert.False(fullPass);
Assert.Equal(0, h.CreateCalls);
Assert.Null(h.UpsertedSentinel);
}
[Fact]
public async Task StaleSentinel_RunsFullPass_AndRewritesSentinel()
{
var h = new Harness();
h.SentinelReturns("STALE");
var fullPass = await h.Build().InitializeTablesAsync();
Assert.True(fullPass);
Assert.Equal(Constants.TableNames.All.Length, h.CreateCalls);
Assert.NotNull(h.UpsertedSentinel);
Assert.Equal(CurrentHash, h.UpsertedSentinel!.GetString(TableStorageService.SchemaSentinelHashProperty));
}
[Fact]
public async Task MissingSentinel_RunsFullPass()
{
var h = new Harness();
h.SentinelReturns(null);
Assert.True(await h.Build().InitializeTablesAsync());
Assert.Equal(Constants.TableNames.All.Length, h.CreateCalls);
}
[Fact]
public async Task UnreadableSentinel_FreshStorage_RunsFullPass()
{
var h = new Harness();
h.SentinelThrows();
Assert.True(await h.Build().InitializeTablesAsync());
Assert.Equal(Constants.TableNames.All.Length, h.CreateCalls);
Assert.NotNull(h.UpsertedSentinel);
}
[Fact]
public async Task FailedTableCreate_DoesNotWriteSentinel_AndRetriesNextCall()
{
var h = new Harness();
h.SentinelReturns(null);
var failing = Constants.TableNames.All.First();
h.Service.Setup(s => s.CreateTableIfNotExistsAsync(failing, It.IsAny<CancellationToken>()))
.ThrowsAsync(new RequestFailedException(500, "boom"));
var svc = h.Build();
Assert.True(await svc.InitializeTablesAsync());
Assert.Null(h.UpsertedSentinel);
// Not marked initialized → a second call runs the pass again instead of short-circuiting.
var before = h.CreateCalls;
Assert.True(await svc.InitializeTablesAsync());
Assert.True(h.CreateCalls > before);
}
[Fact]
public async Task SecondCallAfterSuccess_IsNoOp()
{
var h = new Harness();
h.SentinelReturns(null);
var svc = h.Build();
await svc.InitializeTablesAsync();
var calls = h.CreateCalls;
Assert.False(await svc.InitializeTablesAsync());
Assert.Equal(calls, h.CreateCalls);
}
[Fact]
public async Task BackfillClaim_WinnerTrue_LoserFalse()
{
var h = new Harness();
h.AdminConfig.SetupSequence(c => c.AddEntityAsync(It.IsAny<TableEntity>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(Mock.Of<Response>())
.ThrowsAsync(new RequestFailedException(409, "EntityAlreadyExists"));
var svc = h.Build();
Assert.True(await svc.TryClaimSessionIndexBackfillAsync());
Assert.False(await svc.TryClaimSessionIndexBackfillAsync());
}
}
}
@@ -0,0 +1,206 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Verdict-calibration drift radar (docs/backend/verdict-calibration.md): the three finding
/// kinds, the small-n / lift / Wilson gates, re-arm, and the tracker round-trip.
/// </summary>
public class VerdictCalibrationRadarTests
{
private static readonly DateTime Target = new(2026, 8, 22, 0, 0, 0, DateTimeKind.Utc);
/// <summary>35 daily rows ending on Target; per-day buckets chosen by window/baseline membership.</summary>
private static List<VerdictCalibrationDailyAggregate> Horizon(
Func<bool, (string Path, string Status, int Count)[]> bucketsFor, int sessionsPerDay = 20)
{
var rows = new List<VerdictCalibrationDailyAggregate>();
for (var i = 34; i >= 0; i--)
{
var inWindow = i < 7;
var date = Target.AddDays(-i);
rows.Add(new VerdictCalibrationDailyAggregate
{
TenantId = "t", Date = date.ToString("yyyy-MM-dd"), SessionCount = sessionsPerDay, TerminalSessionCount = sessionsPerDay,
Buckets = bucketsFor(inWindow).Select(b => new VerdictCalibrationBucket { VerdictPath = b.Path, Status = b.Status, Count = b.Count }).ToList(),
});
}
return rows;
}
[Fact]
public void Share_regression_fires_on_a_doubled_separated_share()
{
// baseline: 1/20 per day (5%), window: 4/20 per day (20%) → lift 4, clearly separated
var rows = Horizon(w => new[] { ("sweep:r5_incomplete", "Incomplete", w ? 4 : 1), ("agent:complete", "Succeeded", w ? 16 : 19) });
var findings = VerdictCalibrationRadar.Evaluate(rows, Target);
var f = Assert.Single(findings, x => x.Kind == VerdictCalibrationAlertKinds.ShareRegression);
Assert.Equal("sweep:r5_incomplete", f.VerdictPath);
Assert.Equal("Incomplete", f.Status);
Assert.Equal(28, f.WindowHitCount);
Assert.Equal(140, f.WindowSessionCount);
Assert.Equal(28, f.BaselineHitCount);
Assert.Equal(560, f.BaselineSessionCount);
Assert.Equal(20.0, f.WindowRatePct);
Assert.Equal(5.0, f.BaselineRatePct);
Assert.Equal(4.0, f.Lift);
Assert.Equal("2026-08-16", f.WindowStartDate);
Assert.Equal("2026-08-22", f.WindowEndDate);
// sweep:* is also the silence group → silence share doubled as well
Assert.Contains(findings, x => x.Kind == VerdictCalibrationAlertKinds.SilenceShareRegression && x.VerdictPath == VerdictCalibrationRadar.SilenceGroupPath);
}
[Fact]
public void Share_regression_is_suppressed_below_lift_two_or_without_separation()
{
// 5% → 8%: lift 1.6 — under the gate
var mild = Horizon(w => new[] { ("agent:failed", "Failed", w ? 2 : 1), ("agent:complete", "Succeeded", w ? 18 : 19) }, sessionsPerDay: 20);
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(mild, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
// tiny n: 1/2 per day in window vs 0/2 baseline — lift infinite but Wilson intervals overlap / hits < 5
var tiny = Horizon(w => new[] { ("agent:failed", "Failed", w ? 1 : 0), ("agent:complete", "Succeeded", w ? 1 : 2) }, sessionsPerDay: 2);
Assert.Empty(VerdictCalibrationRadar.Evaluate(tiny, Target));
}
[Fact]
public void Agent_and_registration_paths_never_share_alert_but_still_feed_the_silence_group()
{
// Workflow mix, not classifier drift: a WhiteGlove rollout week doubles the pending share.
var wg = Horizon(w => new[] { ("agent:whiteglove_pending", "Pending", w ? 6 : 1), ("agent:complete", "Succeeded", w ? 14 : 19) });
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(wg, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
var reg = Horizon(w => new[] { ("register:superseded", "Incomplete", w ? 6 : 1), ("agent:complete", "Succeeded", w ? 14 : 19) });
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(reg, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
// Backend-decided paths stay eligible.
Assert.True(VerdictCalibrationRadar.IsShareRegressionEligible("sweep:r6"));
Assert.True(VerdictCalibrationRadar.IsShareRegressionEligible("rule:ANALYZE-ESP-001"));
Assert.True(VerdictCalibrationRadar.IsShareRegressionEligible("manual:failed"));
Assert.False(VerdictCalibrationRadar.IsShareRegressionEligible("agent:complete_soft"));
// An episode opened by the earlier, broader gate re-arms on the next pass regardless of numbers.
var stale = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "agent:whiteglove_pending", Status = "Pending" };
Assert.True(VerdictCalibrationRadar.ShouldReArm(stale, wg, Target));
}
[Fact]
public void Derived_legacy_paths_never_alert()
{
var rows = Horizon(w => new[] { ("legacy:r6", "Incomplete", w ? 10 : 1), ("agent:complete", "Succeeded", w ? 10 : 19) });
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(rows, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
}
[Fact]
public void Evidence_gap_fires_on_r6_share_and_ignores_non_rule_sweep_paths()
{
// classifier verdicts in window: r6 5/day + r5 15/day = 140; r6 share 25% → fires. sweep:stalled is not a rule verdict.
var rows = Horizon(w => new[]
{
("sweep:r6", "Incomplete", 5), ("sweep:r5_incomplete", "Incomplete", 15), ("sweep:stalled", "Stalled", 30),
}, sessionsPerDay: 50);
var f = Assert.Single(VerdictCalibrationRadar.Evaluate(rows, Target), x => x.Kind == VerdictCalibrationAlertKinds.EvidenceGap);
Assert.Equal(VerdictCalibrationRadar.EvidenceGapPath, f.VerdictPath);
Assert.Equal(35, f.WindowHitCount);
Assert.Equal(140, f.WindowSessionCount);
Assert.Equal(25.0, f.WindowRatePct);
Assert.Null(f.Lift);
}
[Fact]
public void Evidence_gap_needs_twenty_classifier_verdicts()
{
var rows = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 2 : 0), ("agent:complete", "Succeeded", 18) });
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(rows, Target), f => f.Kind == VerdictCalibrationAlertKinds.EvidenceGap);
}
[Fact]
public void ReArm_when_share_falls_under_one_point_five_or_path_stops()
{
var alert = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "sweep:r6", Status = "Incomplete" };
var stillHigh = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 4 : 1), ("agent:complete", "Succeeded", w ? 16 : 19) });
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, stillHigh, Target));
var elevatedButUnderGate = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 2 : 1), ("agent:complete", "Succeeded", w ? 18 : 19) }, sessionsPerDay: 20);
// 10% vs 5% = 2.0× — still at/above 1.5× → keep
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, elevatedButUnderGate, Target));
var back = Horizon(w => new[] { ("sweep:r6", "Incomplete", 1), ("agent:complete", "Succeeded", 19) });
Assert.True(VerdictCalibrationRadar.ShouldReArm(alert, back, Target));
var gone = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 0 : 1), ("agent:complete", "Succeeded", w ? 20 : 19) });
Assert.True(VerdictCalibrationRadar.ShouldReArm(alert, gone, Target));
}
[Fact]
public void Evidence_gap_rearms_under_fifteen_percent()
{
var alert = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.EvidenceGap, VerdictPath = "r6", Status = "*" };
var still = Horizon(w => new[] { ("sweep:r6", "Incomplete", 4), ("sweep:r5_incomplete", "Incomplete", 16) });
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, still, Target)); // 20%
var between = Horizon(w => new[] { ("sweep:r6", "Incomplete", 17), ("sweep:r5_incomplete", "Incomplete", 83) }, sessionsPerDay: 100);
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, between, Target)); // 17%
var low = Horizon(w => new[] { ("sweep:r6", "Incomplete", 1), ("sweep:r5_incomplete", "Incomplete", 19) });
Assert.True(VerdictCalibrationRadar.ShouldReArm(alert, low, Target)); // 5%
}
[Fact]
public void CurrentSums_follow_the_alert_kind()
{
var rows = Horizon(w => new[] { ("sweep:r6", "Incomplete", 2), ("maxlife:r3", "Incomplete", 1), ("agent:complete", "Succeeded", 17) });
var share = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "sweep:r6", Status = "Incomplete" }, rows, Target);
Assert.Equal((14, 140, 56, 560), share);
var silence = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.SilenceShareRegression }, rows, Target);
Assert.Equal((21, 140, 84, 560), silence);
var gap = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.EvidenceGap }, rows, Target);
Assert.Equal((14, 21, 56, 84), gap);
}
[Fact]
public void Rows_outside_the_horizon_are_ignored()
{
var rows = Horizon(w => new[] { ("agent:complete", "Succeeded", 20) });
rows.Add(new VerdictCalibrationDailyAggregate { TenantId = "t", Date = "2026-08-23", SessionCount = 1000, Buckets = { new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = 1000 } } });
rows.Add(new VerdictCalibrationDailyAggregate { TenantId = "t", Date = "2026-07-01", SessionCount = 1000, Buckets = { new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = 1000 } } });
Assert.Empty(VerdictCalibrationRadar.Evaluate(rows, Target));
}
// ---- tracker round-trip ----
[Fact]
public void Alert_round_trips_through_the_tracker_entity()
{
var alert = new VerdictCalibrationAlert
{
TenantId = "11111111-1111-1111-1111-111111111111", Kind = VerdictCalibrationAlertKinds.ShareRegression,
VerdictPath = "rule:ANALYZE-ESP-001", Status = "Failed",
WindowHitCount = 7, WindowSessionCount = 40, BaselineHitCount = 3, BaselineSessionCount = 160,
WindowRatePct = 17.5, BaselineRatePct = 1.9, Lift = 9.3, WindowStartDate = "2026-08-16", WindowEndDate = "2026-08-22",
Dimension = new RuleRegressionDimension { Dimension = "osBuild", Value = "26200.1234", HitCount = 6, HitSharePct = 85.7, AllSharePct = 20.0, Lift = 4.3 },
FirstNotifiedAt = new DateTime(2026, 8, 20, 6, 0, 0, DateTimeKind.Utc), LastEvaluatedAt = new DateTime(2026, 8, 23, 6, 0, 0, DateTimeKind.Utc),
};
var entity = TableHardwareRejectionNotificationTracker.BuildVerdictCalibrationEntity(alert.TenantId, alert);
Assert.Equal("verdictcalibration|share_regression|rule:analyze-esp-001|failed", entity.RowKey);
Assert.Equal(alert.TenantId, entity.PartitionKey);
var back = TableHardwareRejectionNotificationTracker.MapToVerdictCalibrationAlert(entity);
Assert.Equal(JsonSerializer.Serialize(alert), JsonSerializer.Serialize(back));
// Tri-states: no lift / no dimension → absent columns → null on read, never 0 / a guessed dimension.
var bare = new VerdictCalibrationAlert
{
TenantId = alert.TenantId, Kind = VerdictCalibrationAlertKinds.EvidenceGap, VerdictPath = "r6", Status = "*",
FirstNotifiedAt = alert.FirstNotifiedAt, LastEvaluatedAt = alert.LastEvaluatedAt,
};
var bareBack = TableHardwareRejectionNotificationTracker.MapToVerdictCalibrationAlert(
TableHardwareRejectionNotificationTracker.BuildVerdictCalibrationEntity(bare.TenantId, bare));
Assert.Null(bareBack.Lift);
Assert.Null(bareBack.Dimension);
Assert.Equal("verdictcalibration|evidence_gap|r6|*", TableHardwareRejectionNotificationTracker.BuildVerdictCalibrationRowKey(bare.Kind, bare.VerdictPath, bare.Status));
}
}
@@ -0,0 +1,301 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using AutopilotMonitor.Functions.Functions.Metrics;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Verdict calibration aggregate (docs/backend/verdict-calibration.md): the pure bucketing core,
/// the persistence round-trip and the matrix arithmetic of the read endpoint.
/// </summary>
public class VerdictCalibrationTests
{
private const string TenantA = "11111111-1111-1111-1111-111111111111";
private const string TenantB = "22222222-2222-2222-2222-222222222222";
private static readonly DateTime Now = new(2026, 8, 23, 12, 0, 0, DateTimeKind.Utc);
private static SessionSummary Session(string tenant, string id, SessionStatus status, DateTime started,
string? path = null, string? priorPath = null, string? priorStatus = null, string? adminMarked = null,
DateTime? completed = null, string serial = "SN-1", string? failureReason = null, string? reconcileReason = null) => new()
{
TenantId = tenant, SessionId = id, SerialNumber = serial, Status = status, StartedAt = started,
CompletedAt = completed, VerdictPath = path, PriorVerdictPath = priorPath, PriorStatus = priorStatus,
AdminMarkedAction = adminMarked, FailureReason = failureReason ?? string.Empty,
ReconcileReason = reconcileReason ?? string.Empty, FailureSource = string.Empty,
};
private static DeviceHistory History(string tenant, string serial, params (string Id, DateTime Started, DateTime? Completed, string Status)[] refs) => new()
{
TenantId = tenant, SerialKey = serial.ToLowerInvariant(), SerialNumber = serial,
Chain = refs.Select(r => new DeviceSessionRef { SessionId = r.Id, StartedAt = r.Started, CompletedAt = r.Completed, Status = r.Status }).ToList(),
};
private static VerdictCalibrationBucket Bucket(VerdictCalibrationDailyAggregate row, string path, string status)
=> Assert.Single(row.Buckets, b => b.VerdictPath == path && b.Status == status);
// ---- BuildVerdictCalibrationAggregates ----
[Fact]
public void Buckets_by_tenant_date_path_and_status_with_global_mirror()
{
var d1 = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
var sessions = new List<SessionSummary>
{
Session(TenantA, "a1", SessionStatus.Succeeded, d1, path: "agent:complete", completed: d1.AddHours(1)),
Session(TenantA, "a2", SessionStatus.Succeeded, d1.AddHours(2), path: "agent:complete", completed: d1.AddHours(3)),
Session(TenantA, "a3", SessionStatus.Incomplete, d1, path: "sweep:r6", completed: d1.AddHours(6)),
Session(TenantA, "a4", SessionStatus.Stalled, d1.AddDays(1), path: "sweep:stalled"),
Session(TenantB, "b1", SessionStatus.Failed, d1, path: "agent:failed", completed: d1.AddHours(1)),
};
var rows = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, new Dictionary<string, List<DeviceHistory>>(), Now);
var a10 = Assert.Single(rows, r => r.TenantId == TenantA && r.Date == "2026-08-10");
Assert.Equal(3, a10.SessionCount);
Assert.Equal(3, a10.TerminalSessionCount);
Assert.Equal(2, Bucket(a10, "agent:complete", "Succeeded").Count);
Assert.Equal(1, Bucket(a10, "sweep:r6", "Incomplete").Count);
var a11 = Assert.Single(rows, r => r.TenantId == TenantA && r.Date == "2026-08-11");
Assert.Equal(1, a11.SessionCount);
Assert.Equal(0, a11.TerminalSessionCount); // Stalled is not terminal
Assert.Equal(1, Bucket(a11, "sweep:stalled", "Stalled").Count);
var g10 = Assert.Single(rows, r => r.TenantId == "global" && r.Date == "2026-08-10");
Assert.Equal(4, g10.SessionCount);
Assert.Equal(1, Bucket(g10, "agent:failed", "Failed").Count);
Assert.Equal(VerdictCalibrationDailyAggregate.CurrentVersion, g10.Version);
Assert.Equal(Now, g10.ComputedAt);
// Deterministic ordering: rows by tenant then date, buckets by path then status.
Assert.Equal(rows.Select(r => (r.TenantId, r.Date)).OrderBy(k => k.TenantId, StringComparer.Ordinal).ThenBy(k => k.Date, StringComparer.Ordinal), rows.Select(r => (r.TenantId, r.Date)));
Assert.Equal(a10.Buckets.Select(b => b.VerdictPath).OrderBy(p => p, StringComparer.Ordinal), a10.Buckets.Select(b => b.VerdictPath));
}
[Fact]
public void Unstamped_rows_are_derived_and_counted_as_derived()
{
var d = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
var sessions = new List<SessionSummary>
{
Session(TenantA, "a1", SessionStatus.Succeeded, d, path: null, completed: d.AddHours(1)), // legacy agent completion
Session(TenantA, "a2", SessionStatus.Incomplete, d, path: null, completed: d.AddHours(5),
failureReason: "No Device Setup completion or explicit failure signal observed before timeout"),
Session(TenantA, "a3", SessionStatus.Succeeded, d, path: "agent:complete", completed: d.AddHours(1)),
};
var row = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, new Dictionary<string, List<DeviceHistory>>(), Now)
.Single(r => r.TenantId == TenantA);
var complete = Bucket(row, "agent:complete", "Succeeded");
Assert.Equal(2, complete.Count);
Assert.Equal(1, complete.DerivedCount);
var r6 = Bucket(row, "legacy:r6", "Incomplete");
Assert.Equal(1, r6.Count);
Assert.Equal(1, r6.DerivedCount);
}
[Fact]
public void Overrides_are_attributed_to_the_prior_path()
{
var d = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
var sessions = new List<SessionSummary>
{
// sweep said Incomplete, agent later completed → late completion against sweep:r5_incomplete
Session(TenantA, "a1", SessionStatus.Succeeded, d, path: "agent:complete", priorPath: "sweep:r5_incomplete", priorStatus: "Incomplete", completed: d.AddHours(1)),
// admin flipped an agent success to Failed
Session(TenantA, "a2", SessionStatus.Failed, d, path: "manual:failed", priorPath: "agent:complete", priorStatus: "Succeeded", adminMarked: "Failed", completed: d.AddHours(1)),
// retro reclassification Failed → Incomplete
Session(TenantA, "a3", SessionStatus.Incomplete, d, path: "retro:r6", priorPath: "agent:failed", priorStatus: "Failed", completed: d.AddHours(1)),
// admin mark to Succeeded counts as admin, not late completion
Session(TenantA, "a4", SessionStatus.Succeeded, d, path: "manual:succeeded", priorPath: "sweep:r6", priorStatus: "Incomplete", adminMarked: "Succeeded", completed: d.AddHours(1)),
};
var row = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, new Dictionary<string, List<DeviceHistory>>(), Now)
.Single(r => r.TenantId == TenantA);
var priorSweep = Bucket(row, "sweep:r5_incomplete", "Incomplete");
Assert.Equal(0, priorSweep.Count); // nobody is on it any more
Assert.Equal(1, priorSweep.OverriddenByLateCompletion);
var priorAgent = Bucket(row, "agent:complete", "Succeeded");
Assert.Equal(1, priorAgent.Count); // a1 is on it now
Assert.Equal(1, priorAgent.OverriddenByAdmin); // a2 used to be
Assert.Equal(1, Bucket(row, "agent:failed", "Failed").OverriddenOther);
Assert.Equal(1, Bucket(row, "sweep:r6", "Incomplete").OverriddenByAdmin);
Assert.Equal(0, Bucket(row, "manual:succeeded", "Succeeded").OverriddenByAdmin);
}
[Fact]
public void ReEnrollment_proxy_counts_only_eligible_terminal_sessions()
{
var old = Now.AddDays(-20);
var young = Now.AddDays(-2);
var sessions = new List<SessionSummary>
{
// old, re-enrolled 3 days later → eligible + re-enrolled
Session(TenantA, "a1", SessionStatus.Incomplete, old, path: "sweep:r6", completed: old.AddHours(5), serial: "SN-A"),
// old, next attempt 10 days later → eligible, not re-enrolled
Session(TenantA, "a2", SessionStatus.Failed, old, path: "agent:failed", completed: old.AddHours(1), serial: "SN-B"),
// old, no next attempt → eligible, not re-enrolled
Session(TenantA, "a3", SessionStatus.Succeeded, old, path: "agent:complete", completed: old.AddHours(1), serial: "SN-C"),
// too young to judge → not eligible even though the chain shows a follow-up
Session(TenantA, "a4", SessionStatus.Failed, young, path: "agent:failed", completed: young.AddHours(1), serial: "SN-D"),
// non-terminal → never eligible
Session(TenantA, "a5", SessionStatus.AwaitingUser, old, path: "sweep:r5_awaiting", serial: "SN-E"),
};
var histories = new Dictionary<string, List<DeviceHistory>>
{
[TenantA] = new()
{
History(TenantA, "SN-A", ("a1", old, old.AddHours(5), "Incomplete"), ("x1", old.AddDays(3), old.AddDays(3).AddHours(1), "Succeeded")),
History(TenantA, "SN-B", ("a2", old, old.AddHours(1), "Failed"), ("x2", old.AddDays(10), null, "Succeeded")),
History(TenantA, "SN-C", ("a3", old, old.AddHours(1), "Succeeded")),
History(TenantA, "SN-D", ("a4", young, young.AddHours(1), "Failed"), ("x4", young.AddHours(2), null, "Succeeded")),
},
};
var rows = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, histories, Now);
var oldRow = rows.Single(r => r.TenantId == TenantA && r.Date == old.ToString("yyyy-MM-dd"));
var r6 = Bucket(oldRow, "sweep:r6", "Incomplete");
Assert.Equal((1, 1), (r6.Eligible7d, r6.ReEnrolled7d));
var failed = Bucket(oldRow, "agent:failed", "Failed");
Assert.Equal((1, 0), (failed.Eligible7d, failed.ReEnrolled7d));
var complete = Bucket(oldRow, "agent:complete", "Succeeded");
Assert.Equal((1, 0), (complete.Eligible7d, complete.ReEnrolled7d));
var awaiting = Bucket(oldRow, "sweep:r5_awaiting", "AwaitingUser");
Assert.Equal((0, 0), (awaiting.Eligible7d, awaiting.ReEnrolled7d));
var youngRow = rows.Single(r => r.TenantId == TenantA && r.Date == young.ToString("yyyy-MM-dd"));
var youngFailed = Bucket(youngRow, "agent:failed", "Failed");
Assert.Equal((0, 0), (youngFailed.Eligible7d, youngFailed.ReEnrolled7d));
}
[Fact]
public void Sessions_inside_a_deletion_cascade_are_skipped()
{
var d = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
var deleting = Session(TenantA, "a1", SessionStatus.Succeeded, d, path: "agent:complete");
deleting.DeletionState = "Pending";
var rows = MaintenanceService.BuildVerdictCalibrationAggregates(new[] { deleting }, new Dictionary<string, List<DeviceHistory>>(), Now);
Assert.Empty(rows);
}
// ---- Persistence round-trip ----
[Fact]
public void Aggregate_round_trips_through_the_table_entity()
{
var aggregate = new VerdictCalibrationDailyAggregate
{
TenantId = TenantA, Date = "2026-08-10", Version = 1, SessionCount = 5, TerminalSessionCount = 4, ComputedAt = Now,
Buckets =
{
new VerdictCalibrationBucket { VerdictPath = "agent:complete", Status = "Succeeded", Count = 3, DerivedCount = 1, Eligible7d = 3, ReEnrolled7d = 1, OverriddenByAdmin = 1, OverriddenByLateCompletion = 0, OverriddenOther = 2 },
new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = 1 },
},
};
var entity = TableStorageService.BuildVerdictCalibrationAggregateEntity(aggregate);
Assert.Equal(TenantA, entity.PartitionKey);
Assert.Equal("2026-08-10", entity.RowKey);
var back = TableStorageService.MapToVerdictCalibrationAggregate(entity);
Assert.Equal(JsonSerializer.Serialize(aggregate), JsonSerializer.Serialize(back));
}
// ---- Response builder ----
private static VerdictCalibrationDailyAggregate Day(string tenant, DateTime date, int sessions, params (string Path, string Status, int Count)[] buckets) => new()
{
TenantId = tenant, Date = date.ToString("yyyy-MM-dd"), SessionCount = sessions, TerminalSessionCount = sessions, Version = 1, ComputedAt = Now,
Buckets = buckets.Select(b => new VerdictCalibrationBucket { VerdictPath = b.Path, Status = b.Status, Count = b.Count, Eligible7d = b.Count, ReEnrolled7d = b.Count / 5 }).ToList(),
};
// Same naming policy as the worker host (Program.cs: camelCase) so the assertions pin the wire shape.
private static readonly JsonSerializerOptions Wire = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase };
private static JsonElement BuildJson(IReadOnlyList<VerdictCalibrationDailyAggregate> daily, DateTime today, int days)
=> JsonSerializer.SerializeToElement(VerdictCalibrationResponse.Build(daily, "global", today, days), Wire);
[Fact]
public void Response_sums_the_window_and_computes_share_rate_and_trend()
{
var today = new DateTime(2026, 8, 23, 0, 0, 0, DateTimeKind.Utc);
var daily = new List<VerdictCalibrationDailyAggregate>();
// 35 days: baseline days carry 10 sessions (1 sweep:r6), the last 7 days carry 10 sessions (3 sweep:r6) → share 10% → 30%, lift 3.0
for (var i = 34; i >= 0; i--)
{
var date = today.AddDays(-i);
var r6 = i < 7 ? 3 : 1;
daily.Add(Day("global", date, 10, ("agent:complete", "Succeeded", 10 - r6), ("sweep:r6", "Incomplete", r6)));
}
var json = BuildJson(daily, today, days: 30);
Assert.True(json.GetProperty("success").GetBoolean());
Assert.Equal(30, json.GetProperty("windowDays").GetInt32());
Assert.Equal(300, json.GetProperty("totals").GetProperty("sessions").GetInt32());
Assert.Equal(30, json.GetProperty("totals").GetProperty("days").GetInt32());
var paths = json.GetProperty("paths").EnumerateArray().ToList();
var r6Row = paths.Single(p => p.GetProperty("verdictPath").GetString() == "sweep:r6");
Assert.Equal(7 * 3 + 23 * 1, r6Row.GetProperty("count").GetInt32());
Assert.Equal(Math.Round(100.0 * 44 / 300, 1), r6Row.GetProperty("sharePct").GetDouble());
Assert.Equal(30.0, r6Row.GetProperty("window7").GetProperty("sharePct").GetDouble());
Assert.Equal(10.0, r6Row.GetProperty("baseline28").GetProperty("sharePct").GetDouble());
Assert.Equal(3.0, r6Row.GetProperty("lift").GetDouble());
// Eligible 44 ≥ 20 → rate stated; ReEnrolled7d = count/5 per day → 7*0 + 23*0 = 0 → 0%
Assert.Equal(0.0, r6Row.GetProperty("reEnrollRatePct").GetDouble());
// Sorted by count desc → agent:complete first.
Assert.Equal("agent:complete", paths[0].GetProperty("verdictPath").GetString());
}
[Fact]
public void Response_withholds_rate_below_twenty_eligible_and_lift_without_baseline()
{
var today = new DateTime(2026, 8, 23, 0, 0, 0, DateTimeKind.Utc);
var daily = new List<VerdictCalibrationDailyAggregate>
{
Day("global", today, 5, ("maxlife:r3", "Incomplete", 5)),
};
var json = BuildJson(daily, today, days: 7);
var row = json.GetProperty("paths").EnumerateArray().Single();
Assert.Equal(JsonValueKind.Null, row.GetProperty("reEnrollRatePct").ValueKind);
Assert.Equal(JsonValueKind.Null, row.GetProperty("lift").ValueKind); // no baseline rows → never invented
Assert.Equal(100.0, row.GetProperty("sharePct").GetDouble());
}
[Fact]
public void Response_keeps_a_zero_count_bucket_that_carries_overrides()
{
var today = new DateTime(2026, 8, 23, 0, 0, 0, DateTimeKind.Utc);
var day = Day("global", today, 1, ("agent:complete", "Succeeded", 1));
day.Buckets.Add(new VerdictCalibrationBucket { VerdictPath = "sweep:r5_incomplete", Status = "Incomplete", Count = 0, OverriddenByLateCompletion = 1 });
var json = BuildJson(new[] { day }, today, days: 7);
var rows = json.GetProperty("paths").EnumerateArray().ToList();
var prior = rows.Single(p => p.GetProperty("verdictPath").GetString() == "sweep:r5_incomplete");
Assert.Equal(0, prior.GetProperty("count").GetInt32());
Assert.Equal(1, prior.GetProperty("overriddenByLateCompletion").GetInt32());
}
[Fact]
public void Response_is_empty_but_well_formed_before_the_first_sweep()
{
var json = BuildJson(Array.Empty<VerdictCalibrationDailyAggregate>(), new DateTime(2026, 8, 23), 30);
Assert.True(json.GetProperty("success").GetBoolean());
Assert.Empty(json.GetProperty("paths").EnumerateArray());
Assert.Equal(0, json.GetProperty("totals").GetProperty("sessions").GetInt32());
Assert.Equal(JsonValueKind.Null, json.GetProperty("computedAt").ValueKind);
}
[Theory]
[InlineData(null, 30)]
[InlineData("abc", 30)]
[InlineData("0", 1)]
[InlineData("500", 180)]
[InlineData("14", 14)]
public void ClampDays_bounds_the_window(string? raw, int expected)
=> Assert.Equal(expected, VerdictCalibrationResponse.ClampDays(raw));
}
@@ -0,0 +1,201 @@
using System;
using System.Collections.Generic;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Verdict calibration instrumentation (docs/backend/verdict-calibration.md): every status write
/// declares its origin, overriding a prior verdict preserves it, the classifier names the rule
/// behind each verdict, and pre-instrumentation rows derive an honest path from their literals.
/// </summary>
public class VerdictPathTests
{
private static readonly DateTime Start = new(2026, 8, 20, 8, 0, 0, DateTimeKind.Utc);
// ---- ComputePriorVerdict: the correction stream ----
[Theory]
[InlineData("Succeeded", "agent:complete", "Failed", true)] // admin mark over an agent success
[InlineData("Incomplete", "sweep:r5_incomplete", "Succeeded", true)] // late agent completion upgrades a sweep verdict
[InlineData("AwaitingUser", "sweep:r5_awaiting", "Incomplete", true)] // grace expired
[InlineData("Failed", "agent:failed", "Incomplete", true)] // retro reclassification
[InlineData("Stalled", "sweep:stalled", "InProgress", false)] // stall heal is not a correction
[InlineData("Stalled", "sweep:stalled", "Succeeded", false)]
[InlineData("InProgress", "register:new", "Succeeded", false)]
[InlineData("Pending", "agent:whiteglove_pending", "InProgress", false)]
[InlineData("AwaitingUser", "sweep:wg_awaiting", "AwaitingUser", false)] // same-status refresh keeps the original prior
public void Prior_is_preserved_only_when_a_verdict_is_overridden(string existing, string existingPath, string incoming, bool expectPrior)
{
var prior = TableStorageService.ComputePriorVerdict(existing, existingPath, Enum.Parse<SessionStatus>(incoming));
if (expectPrior)
{
Assert.NotNull(prior);
Assert.Equal(existing, prior!.Value.PriorStatus);
Assert.Equal(existingPath, prior.Value.PriorVerdictPath);
}
else
{
Assert.Null(prior);
}
}
[Fact]
public void Prior_needs_a_stamped_path_on_the_existing_row()
{
// Pre-instrumentation rows carry no path — there is nothing trustworthy to preserve.
Assert.Null(TableStorageService.ComputePriorVerdict("Incomplete", null, SessionStatus.Succeeded));
Assert.Null(TableStorageService.ComputePriorVerdict("Incomplete", "", SessionStatus.Succeeded));
Assert.Null(TableStorageService.ComputePriorVerdict(null, "sweep:r6", SessionStatus.Succeeded));
}
// ---- VerdictPaths vocabulary ----
[Fact]
public void Compose_and_Origin_round_trip()
{
var path = VerdictPaths.Compose(VerdictPaths.OriginSweep, ClassifierRules.R5DeviceSetupIncomplete);
Assert.Equal("sweep:r5_incomplete", path);
Assert.Equal("sweep", VerdictPaths.Origin(path));
Assert.True(VerdictPaths.IsClassifierPath(path));
Assert.True(VerdictPaths.IsClassifierPath("maxlife:r1"));
Assert.True(VerdictPaths.IsClassifierPath("late:r4"));
Assert.True(VerdictPaths.IsClassifierPath("retro:r6"));
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.AgentComplete));
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.SweepStalled));
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.RetroSuperseded));
Assert.False(VerdictPaths.IsClassifierPath("legacy:r6"));
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.ManualFailed));
Assert.Throws<ArgumentException>(() => VerdictPaths.Compose("", "r1"));
Assert.Throws<ArgumentException>(() => VerdictPaths.Compose("sweep", " "));
}
// ---- Classifier: every return names its rule ----
private static EnrollmentEvent Evt(string type, string? message = null) => new()
{
EventType = type, Timestamp = Start, Source = "test", Message = message ?? type,
};
private static EnrollmentEvent Esp(string message) => Evt("esp_provisioning_status", message);
private const string DeviceSetup44 = "ESP provisioning status: DeviceSetup — 4 of 4 subcategories completed";
private const string AccountSetup55 = "ESP provisioning status: AccountSetup — 5 of 5 subcategories completed";
private const string AccountSetup15 = "ESP provisioning status: AccountSetup — 1 of 5 subcategories completed";
private static string RuleOf(IReadOnlyList<EnrollmentEvent> events, double hours = 6, int grace = 72,
bool preProvisioned = false, DateTime? resumedAt = null, bool selfDeploying = false)
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var (_, _, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, Start.AddHours(hours), grace, isPreProvisioned: preProvisioned,
resumedAt: resumedAt, isSelfDeployingProfile: selfDeploying);
return rule;
}
[Fact]
public void Classifier_returns_the_rule_id_behind_each_verdict()
{
Assert.Equal(ClassifierRules.R1ExplicitFailure, RuleOf(new[] { Evt("enrollment_failed") }));
Assert.Equal(ClassifierRules.R1bWhiteGloveAwaiting, RuleOf(
new[] { Esp(DeviceSetup44), Evt("whiteglove_complete") }, hours: 6, grace: 51, preProvisioned: true, resumedAt: Start.AddHours(1)));
Assert.Equal(ClassifierRules.R1bWhiteGloveSucceeded, RuleOf(
new[] { Esp(DeviceSetup44), Evt("whiteglove_complete") }, hours: 60, grace: 51, preProvisioned: true, resumedAt: Start.AddHours(1)));
Assert.Equal(ClassifierRules.R1cSelfDeploying, RuleOf(new[] { Esp(DeviceSetup44) }, selfDeploying: true));
Assert.Equal(ClassifierRules.R2AccountSetupComplete, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup55) }));
Assert.Equal(ClassifierRules.R3EmergencyBreak, RuleOf(new[] { Esp(DeviceSetup44), Evt("agent_emergency_break") }));
Assert.Equal(ClassifierRules.R4DesktopHello, RuleOf(new[] { Esp(DeviceSetup44), Evt("desktop_arrived"), Evt("hello_provisioning_completed") }));
Assert.Equal(ClassifierRules.R5DeviceSetupAwaiting, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 6, grace: 72));
Assert.Equal(ClassifierRules.R5DeviceSetupIncomplete, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 80, grace: 72));
Assert.Equal(ClassifierRules.R6Fallthrough, RuleOf(Array.Empty<EnrollmentEvent>()));
}
// ---- Derivation for pre-instrumentation rows ----
private static SessionSummary Row(SessionStatus status, string? failureReason = null, string? reconcileReason = null,
string? failureSource = null, string? adminMarked = null, bool soft = false, string? verdictPath = null) => new()
{
SessionId = "s", TenantId = "t", Status = status,
FailureReason = failureReason ?? string.Empty, ReconcileReason = reconcileReason ?? string.Empty,
FailureSource = failureSource ?? string.Empty, AdminMarkedAction = adminMarked, EspSoftFailure = soft,
VerdictPath = verdictPath,
};
private static string Classified(IReadOnlyList<EnrollmentEvent> events, double hours = 80, int grace = 72, bool selfDeploying = false)
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var (_, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, Start.AddHours(hours), grace, isSelfDeployingProfile: selfDeploying);
return reason;
}
[Fact]
public void Stamped_rows_are_returned_verbatim()
{
var (path, derived) = VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete, failureReason: "anything", verdictPath: "sweep:r6"));
Assert.Equal("sweep:r6", path);
Assert.False(derived);
}
[Fact]
public void Derivation_uses_the_real_classifier_literals()
{
// Reasons come straight from the classifier so a wording change there surfaces here.
Assert.Equal(("legacy:r6", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
failureReason: Classified(Array.Empty<EnrollmentEvent>()))));
Assert.Equal(("legacy:r5_incomplete", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
failureReason: Classified(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }))));
Assert.Equal(("legacy:r5_awaiting", true), VerdictPathDerivation.Derive(Row(SessionStatus.AwaitingUser,
failureReason: Classified(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 6))));
Assert.Equal(("legacy:r3", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
failureReason: Classified(new[] { Esp(DeviceSetup44), Evt("agent_emergency_break") }))));
Assert.Equal(("legacy:r2", true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
reconcileReason: Classified(new[] { Esp(DeviceSetup44), Esp(AccountSetup55) }))));
Assert.Equal(("legacy:r4", true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
reconcileReason: Classified(new[] { Esp(DeviceSetup44), Evt("desktop_arrived"), Evt("hello_provisioning_completed") }))));
Assert.Equal(("legacy:r1c", true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
reconcileReason: Classified(new[] { Esp(DeviceSetup44) }, selfDeploying: true))));
Assert.Equal(("legacy:r1", true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed,
failureReason: Classified(new[] { Evt("enrollment_failed") }))));
}
[Fact]
public void Derivation_recovers_the_origin_from_writer_suffixes()
{
var r6 = Classified(Array.Empty<EnrollmentEvent>());
Assert.Equal(("maxlife:r6", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
failureReason: r6 + " Verdict triggered by the agent's max-lifetime watchdog shutdown.")));
Assert.Equal(("retro:r6", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
failureReason: r6 + " Retro-reclassified from the legacy blanket timeout verdict.")));
}
[Fact]
public void Derivation_attributes_unambiguous_non_classifier_paths()
{
Assert.Equal((VerdictPaths.ManualFailed, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed,
failureReason: "Manually marked as failed by administrator", failureSource: "manual", adminMarked: "Failed")));
Assert.Equal((VerdictPaths.ManualSucceeded, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded, adminMarked: "Succeeded")));
Assert.Equal(("rule:ANALYZE-ESP-001", true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed,
failureReason: "Rule: ESP timeout", failureSource: "rule:ANALYZE-ESP-001")));
Assert.Equal((VerdictPaths.AgentComplete, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded)));
Assert.Equal((VerdictPaths.AgentComplete, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
reconcileReason: "Late completion report received — upgraded prior 'Incomplete' verdict")));
Assert.Equal((VerdictPaths.AgentCompleteSoft, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded, soft: true)));
Assert.Equal((VerdictPaths.AgentFailed, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed, failureReason: "ESP timeout (0x800705B4)")));
Assert.Equal((VerdictPaths.AgentEspFailureFallback, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed, failureReason: "ESP failure (backend fallback)")));
Assert.Equal((VerdictPaths.SweepStalled, true), VerdictPathDerivation.Derive(Row(SessionStatus.Stalled, failureReason: "Agent silent for 130min (detected by maintenance sweep)")));
Assert.Equal((VerdictPaths.AgentStallProbe, true), VerdictPathDerivation.Derive(Row(SessionStatus.Stalled, failureReason: "Agent reported stall after 60min without progress (stall_probe)")));
Assert.Equal(("legacy:superseded", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete, failureReason: "Superseded by session abc: re-registered")));
Assert.Equal((VerdictPaths.AgentWhiteGlovePending, true), VerdictPathDerivation.Derive(Row(SessionStatus.Pending)));
}
[Fact]
public void Derivation_never_invents_a_path()
{
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed, failureReason: "Session timed out after 5 hours (started at 2026-01-01 UTC)")));
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete)));
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.InProgress)));
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded, reconcileReason: "Some future wording")));
}
}
@@ -422,6 +422,152 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
};
}
// ===== VERDICT CALIBRATION DRIFT (docs/backend/verdict-calibration.md) =====
internal const string VerdictCalibrationRowKeyPrefix = "verdictcalibration|";
internal static string BuildVerdictCalibrationRowKey(string kind, string verdictPath, string status)
{
// Paths are a closed vocabulary plus rule ids (ANALYZE-...), statuses are enum names and
// the group marker "*" - none contain table-key-hostile characters, but sanitize anyway.
var k = SanitizeTableKey((kind ?? string.Empty).Trim().ToLowerInvariant());
var p = SanitizeTableKey((verdictPath ?? string.Empty).Trim().ToLowerInvariant());
var s = SanitizeTableKey((status ?? string.Empty).Trim().ToLowerInvariant());
return $"{VerdictCalibrationRowKeyPrefix}{k}|{p}|{s}";
}
public async Task<bool> TryRegisterVerdictCalibrationAlertAsync(string tenantId, VerdictCalibrationAlert alert)
{
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(alert?.Kind) || string.IsNullOrWhiteSpace(alert?.VerdictPath))
return false;
try
{
await _table.AddEntityAsync(BuildVerdictCalibrationEntity(tenantId, alert!));
_logger.LogInformation(
"VerdictCalibration tracker registered: tenant={TenantId} kind={Kind} path={Path}", tenantId, alert!.Kind, alert.VerdictPath);
return true;
}
catch (RequestFailedException ex) when (ex.Status == 409)
{
return false; // episode already active - one ops event per episode
}
catch (Exception ex)
{
_logger.LogWarning(ex, "VerdictCalibration tracker failed: tenant={TenantId} kind={Kind} path={Path}", tenantId, alert!.Kind, alert.VerdictPath);
return false; // fail closed - never double-fire
}
}
public async Task RefreshVerdictCalibrationAlertAsync(string tenantId, VerdictCalibrationAlert alert)
{
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(alert?.Kind) || string.IsNullOrWhiteSpace(alert?.VerdictPath))
return;
try
{
await _table.UpsertEntityAsync(BuildVerdictCalibrationEntity(tenantId, alert!), TableUpdateMode.Replace);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "VerdictCalibration tracker refresh failed: tenant={TenantId} kind={Kind} path={Path}", tenantId, alert!.Kind, alert.VerdictPath);
}
}
public async Task DeleteVerdictCalibrationAlertAsync(string tenantId, string kind, string verdictPath, string status)
{
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(kind) || string.IsNullOrWhiteSpace(verdictPath))
return;
try
{
await _table.DeleteEntityAsync(tenantId.ToLowerInvariant(), BuildVerdictCalibrationRowKey(kind, verdictPath, status));
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
// already gone - idempotent
}
catch (Exception ex)
{
_logger.LogWarning(ex, "VerdictCalibration tracker delete failed: tenant={TenantId} kind={Kind} path={Path}", tenantId, kind, verdictPath);
}
}
public async Task<List<VerdictCalibrationAlert>> GetVerdictCalibrationAlertsAsync(string tenantId)
{
var results = new List<VerdictCalibrationAlert>();
if (string.IsNullOrWhiteSpace(tenantId)) return results;
try
{
var partitionKey = tenantId.ToLowerInvariant();
// '}' sorts directly after '|' - prefix range over "verdictcalibration|...".
var filter = $"PartitionKey eq '{partitionKey}' and RowKey ge '{VerdictCalibrationRowKeyPrefix}' and RowKey lt 'verdictcalibration}}'";
await foreach (var entity in _table.QueryAsync<TableEntity>(filter: filter))
{
results.Add(MapToVerdictCalibrationAlert(entity));
}
}
catch (Exception ex)
{
_logger.LogWarning(ex, "VerdictCalibration tracker list failed for tenant {TenantId}", tenantId);
}
return results;
}
// internal static: pinned by round-trip unit tests (table-storage-serialization rule).
internal static TableEntity BuildVerdictCalibrationEntity(string tenantId, VerdictCalibrationAlert alert)
{
var entity = new TableEntity(tenantId.ToLowerInvariant(), BuildVerdictCalibrationRowKey(alert.Kind, alert.VerdictPath, alert.Status))
{
["TenantId"] = tenantId,
["Kind"] = alert.Kind,
["VerdictPath"] = alert.VerdictPath,
["Status"] = alert.Status ?? string.Empty,
["WindowHitCount"] = alert.WindowHitCount,
["WindowSessionCount"] = alert.WindowSessionCount,
["BaselineHitCount"] = alert.BaselineHitCount,
["BaselineSessionCount"] = alert.BaselineSessionCount,
["WindowRatePct"] = alert.WindowRatePct,
["BaselineRatePct"] = alert.BaselineRatePct,
["WindowStartDate"] = alert.WindowStartDate ?? string.Empty,
["WindowEndDate"] = alert.WindowEndDate ?? string.Empty,
["FirstNotifiedAt"] = alert.FirstNotifiedAt,
["LastEvaluatedAt"] = alert.LastEvaluatedAt,
};
if (alert.Lift.HasValue)
entity["Lift"] = alert.Lift.Value;
if (alert.Dimension != null)
entity["DimensionJson"] = JsonSerializer.Serialize(alert.Dimension);
return entity;
}
internal static VerdictCalibrationAlert MapToVerdictCalibrationAlert(TableEntity entity)
{
RuleRegressionDimension? dimension = null;
var dimensionJson = entity.GetString("DimensionJson");
if (!string.IsNullOrEmpty(dimensionJson))
{
try { dimension = JsonSerializer.Deserialize<RuleRegressionDimension>(dimensionJson!); }
catch (JsonException) { dimension = null; }
}
return new VerdictCalibrationAlert
{
TenantId = entity.GetString("TenantId") ?? entity.PartitionKey,
Kind = entity.GetString("Kind") ?? string.Empty,
VerdictPath = entity.GetString("VerdictPath") ?? string.Empty,
Status = entity.GetString("Status") ?? string.Empty,
WindowHitCount = entity.GetInt32("WindowHitCount") ?? 0,
WindowSessionCount = entity.GetInt32("WindowSessionCount") ?? 0,
BaselineHitCount = entity.GetInt32("BaselineHitCount") ?? 0,
BaselineSessionCount = entity.GetInt32("BaselineSessionCount") ?? 0,
WindowRatePct = entity.GetDouble("WindowRatePct") ?? 0,
BaselineRatePct = entity.GetDouble("BaselineRatePct") ?? 0,
Lift = entity.GetDouble("Lift"),
WindowStartDate = entity.GetString("WindowStartDate") ?? string.Empty,
WindowEndDate = entity.GetString("WindowEndDate") ?? string.Empty,
Dimension = dimension,
FirstNotifiedAt = entity.GetDateTimeOffset("FirstNotifiedAt")?.UtcDateTime ?? DateTime.MinValue,
LastEvaluatedAt = entity.GetDateTimeOffset("LastEvaluatedAt")?.UtcDateTime ?? DateTime.MinValue,
};
}
internal const string AppVersionRegressionRowKeyPrefix = "appversionregression|";
/// <summary>
@@ -60,6 +60,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<List<SessionSummary>> GetUsageWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null)
=> _storage.GetUsageWindowSessionsAsync(startDate, endDate, tenantId);
public Task<List<SessionSummary>> GetMaintenanceWindowSessionsAsync(DateTime startDate, DateTime endDate)
=> _storage.GetMaintenanceWindowSessionsAsync(startDate, endDate);
public Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null)
=> _storage.GetGeoWindowSessionsAsync(startDate, endDate, tenantId);
@@ -159,5 +159,17 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<int> DeleteDeviceJourneyAggregatesOlderThanAsync(DateTime cutoffDate)
=> _storage.DeleteDeviceJourneyAggregatesOlderThanAsync(cutoffDate);
public Task<bool> SaveVerdictCalibrationAggregateAsync(VerdictCalibrationDailyAggregate aggregate)
=> _storage.SaveVerdictCalibrationAggregateAsync(aggregate);
public Task<List<VerdictCalibrationDailyAggregate>> GetVerdictCalibrationAggregatesAsync(string tenantId, DateTime startDate, DateTime endDate)
=> _storage.GetVerdictCalibrationAggregatesAsync(tenantId, startDate, endDate);
public Task DeleteVerdictCalibrationAggregateAsync(string tenantId, string dateKey)
=> _storage.DeleteVerdictCalibrationAggregateAsync(tenantId, dateKey);
public Task<int> DeleteVerdictCalibrationAggregatesOlderThanAsync(DateTime cutoffDate)
=> _storage.DeleteVerdictCalibrationAggregatesOlderThanAsync(cutoffDate);
}
}
@@ -58,7 +58,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
=> _storage.GetAllSessionStatsAsync(tenantIdFilter, days, allowedTenantIds);
public async Task<bool> UpdateSessionStatusAsync(
string tenantId, string sessionId, SessionStatus status,
string tenantId, string sessionId, SessionStatus status, string verdictPath,
EnrollmentPhase? currentPhase = null, string? failureReason = null,
DateTime? completedAt = null, DateTime? earliestEventTimestamp = null,
DateTime? latestEventTimestamp = null, bool? isPreProvisioned = null,
@@ -68,7 +68,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
string? failureSnapshotJson = null, bool allowTerminalReclassification = false,
bool espSoftFailure = false, string? completionSource = null)
{
var transitioned = await _storage.UpdateSessionStatusAsync(tenantId, sessionId, status,
var transitioned = await _storage.UpdateSessionStatusAsync(tenantId, sessionId, status, verdictPath,
currentPhase, failureReason, completedAt, earliestEventTimestamp,
latestEventTimestamp, isPreProvisioned, isUserDriven, resumedAt,
stalledAt, clearStalledAt, clearFailureReason, failureSource, adminMarkedAction,
@@ -142,8 +142,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
=> _storage.UpdateSessionDiagnosticsBlobAsync(tenantId, sessionId, blobName, destination);
public Task SetSessionPreProvisionedAsync(string tenantId, string sessionId, bool isPreProvisioned,
SessionStatus? status = null, bool? isUserDriven = null)
=> _storage.SetSessionPreProvisionedAsync(tenantId, sessionId, isPreProvisioned, status, isUserDriven);
SessionStatus? status = null, bool? isUserDriven = null, string? verdictPath = null)
=> _storage.SetSessionPreProvisionedAsync(tenantId, sessionId, isPreProvisioned, status, isUserDriven, verdictPath);
public Task UpdateSessionGeoAsync(string tenantId, string sessionId,
string? country, string? region, string? city, string? loc)
@@ -18,5 +18,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task InitializeAsync()
=> _storage.InitializeTablesAsync();
public Task EnsureAllAsync()
=> _storage.EnsureAllTablesAsync();
}
}
@@ -0,0 +1,232 @@
using System.Globalization;
using System.Net;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Metrics
{
/// <summary>
/// Operator-only verdict calibration matrix (docs/backend/verdict-calibration.md): per verdict
/// path, how many sessions it produced in the window, its share, how often it was overridden
/// (admin / late completion / other), the 7-day re-enrollment proxy, and a 7d-vs-28d trend.
/// Platform scope ("global" partition) or one tenant via ?tenantId=. Serves the admin
/// metrics page and the <c>get_verdict_calibration</c> MCP tool. Never customer-facing —
/// these are platform-internal classifier diagnostics, not product features.
/// </summary>
public class GetVerdictCalibrationFunction
{
private readonly ILogger<GetVerdictCalibrationFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
private readonly IHardwareRejectionNotificationTracker _tracker;
public GetVerdictCalibrationFunction(
ILogger<GetVerdictCalibrationFunction> logger,
IMetricsRepository metricsRepo,
IHardwareRejectionNotificationTracker tracker)
{
_logger = logger;
_metricsRepo = metricsRepo;
_tracker = tracker;
}
[Function("GetVerdictCalibration")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "global/metrics/verdict-calibration")] HttpRequestData req)
{
try
{
// Authentication + GlobalReadOrAdmin authorization enforced by PolicyEnforcementMiddleware
var query = System.Web.HttpUtility.ParseQueryString(req.Url.Query);
var tenantIdFilter = query["tenantId"];
var days = VerdictCalibrationResponse.ClampDays(query["days"]);
var partition = "global";
if (!string.IsNullOrWhiteSpace(tenantIdFilter))
{
if (!Guid.TryParse(tenantIdFilter, out var parsed))
{
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
await bad.WriteAsJsonAsync(new { success = false, message = "tenantId must be a GUID" });
return bad;
}
partition = parsed.ToString("D");
}
var today = DateTime.UtcNow.Date;
// One read covers both the requested window and the trend horizon (7d window + 28d baseline).
var readDays = Math.Max(days, VerdictCalibrationResponse.TrendHorizonDays);
var daily = await _metricsRepo.GetVerdictCalibrationAggregatesAsync(
partition, VerdictCalibrationResponse.InclusiveWindowStart(today, readDays), today);
// Active drift episodes of this partition (tracker keyspace; fail-soft empty).
var alerts = await _tracker.GetVerdictCalibrationAlertsAsync(partition);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(VerdictCalibrationResponse.Build(daily, partition, today, days, alerts));
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error fetching verdict calibration");
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
await errorResponse.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
return errorResponse;
}
}
}
/// <summary>Pure response builder — internal static so the matrix arithmetic is pinned by tests.</summary>
internal static class VerdictCalibrationResponse
{
internal const int DefaultWindowDays = 30;
internal const int MaxWindowDays = 180; // aggregate retention
internal const int TrendWindowDays = 7;
internal const int TrendBaselineDays = 28;
internal const int TrendHorizonDays = TrendWindowDays + TrendBaselineDays;
internal static int ClampDays(string? raw)
{
if (!int.TryParse(raw, out var days)) return DefaultWindowDays;
return Math.Clamp(days, 1, MaxWindowDays);
}
/// <summary>"Last N days" = exactly N calendar day keys including today (both range ends inclusive).</summary>
internal static DateTime InclusiveWindowStart(DateTime today, int days) => today.AddDays(-(days - 1));
internal sealed class PathRow
{
public string VerdictPath { get; set; } = string.Empty;
public string Status { get; set; } = string.Empty;
public int Count { get; set; }
public double SharePct { get; set; }
public int DerivedCount { get; set; }
public int Eligible7d { get; set; }
public int ReEnrolled7d { get; set; }
/// <summary>Null below the minimum eligible count — never a rate on a handful of sessions.</summary>
public double? ReEnrollRatePct { get; set; }
public int OverriddenByAdmin { get; set; }
public int OverriddenByLateCompletion { get; set; }
public int OverriddenOther { get; set; }
public TrendWindow Window7 { get; set; } = new();
public TrendWindow Baseline28 { get; set; } = new();
/// <summary>Window share ÷ baseline share; null when the baseline share is 0 (a new path has no finite lift — never invented).</summary>
public double? Lift { get; set; }
}
internal sealed class TrendWindow
{
public int Count { get; set; }
public int Sessions { get; set; }
public double SharePct { get; set; }
}
/// <summary>Minimum eligible sessions before a re-enrollment rate is stated (truthfulness: no rate on n&lt;20).</summary>
internal const int MinEligibleForRate = 20;
internal static object Build(
IReadOnlyList<VerdictCalibrationDailyAggregate> daily, string partition, DateTime today, int days,
IReadOnlyList<VerdictCalibrationAlert>? alerts = null)
{
var windowStart = Key(InclusiveWindowStart(today, days));
var trendWindowStart = Key(InclusiveWindowStart(today, TrendWindowDays));
var baselineStart = Key(InclusiveWindowStart(today, TrendHorizonDays));
var rows = new Dictionary<(string Path, string Status), PathRow>();
int sessions = 0, terminal = 0, derived = 0;
int trendSessions = 0, baselineSessions = 0;
DateTime? computedAt = null;
var versions = new HashSet<int>();
PathRow Row(VerdictCalibrationBucket b)
{
var key = (b.VerdictPath, b.Status);
if (!rows.TryGetValue(key, out var row))
{
row = new PathRow { VerdictPath = b.VerdictPath, Status = b.Status };
rows[key] = row;
}
return row;
}
foreach (var day in daily)
{
var inWindow = string.CompareOrdinal(day.Date, windowStart) >= 0;
var inTrendWindow = string.CompareOrdinal(day.Date, trendWindowStart) >= 0;
var inBaseline = !inTrendWindow && string.CompareOrdinal(day.Date, baselineStart) >= 0;
if (inWindow)
{
sessions += day.SessionCount;
terminal += day.TerminalSessionCount;
versions.Add(day.Version);
if (computedAt == null || day.ComputedAt > computedAt) computedAt = day.ComputedAt;
}
if (inTrendWindow) trendSessions += day.SessionCount;
if (inBaseline) baselineSessions += day.SessionCount;
foreach (var b in day.Buckets)
{
// Overrides are attributed to the prior path and can exist on a bucket with Count=0.
var row = Row(b);
if (inWindow)
{
row.Count += b.Count;
row.DerivedCount += b.DerivedCount;
row.Eligible7d += b.Eligible7d;
row.ReEnrolled7d += b.ReEnrolled7d;
row.OverriddenByAdmin += b.OverriddenByAdmin;
row.OverriddenByLateCompletion += b.OverriddenByLateCompletion;
row.OverriddenOther += b.OverriddenOther;
derived += b.DerivedCount;
}
if (inTrendWindow) row.Window7.Count += b.Count;
if (inBaseline) row.Baseline28.Count += b.Count;
}
}
var list = rows.Values
.Where(r => r.Count > 0 || r.OverriddenByAdmin + r.OverriddenByLateCompletion + r.OverriddenOther > 0)
.OrderByDescending(r => r.Count)
.ThenBy(r => r.VerdictPath, StringComparer.Ordinal)
.ThenBy(r => r.Status, StringComparer.Ordinal)
.ToList();
foreach (var r in list)
{
r.SharePct = Pct(r.Count, sessions);
r.ReEnrollRatePct = r.Eligible7d >= MinEligibleForRate ? Pct(r.ReEnrolled7d, r.Eligible7d) : null;
r.Window7.Sessions = trendSessions;
r.Window7.SharePct = Pct(r.Window7.Count, trendSessions);
r.Baseline28.Sessions = baselineSessions;
r.Baseline28.SharePct = Pct(r.Baseline28.Count, baselineSessions);
r.Lift = r.Baseline28.SharePct > 0 && trendSessions > 0
? Math.Round(r.Window7.SharePct / r.Baseline28.SharePct, 2)
: null;
}
return new
{
success = true,
tenantId = partition,
windowDays = days,
windowStart,
windowEnd = Key(today),
computedAt,
versions = versions.OrderBy(v => v).ToArray(),
totals = new { sessions, terminal, derived, days = daily.Count(d => string.CompareOrdinal(d.Date, windowStart) >= 0) },
trend = new { windowDays = TrendWindowDays, baselineDays = TrendBaselineDays, windowSessions = trendSessions, baselineSessions },
paths = list,
// Wording contract shared with the rule radar: a dimension is correlated, not causal.
alerts = (alerts ?? Array.Empty<VerdictCalibrationAlert>())
.OrderByDescending(a => a.FirstNotifiedAt)
.ToList(),
};
}
private static string Key(DateTime d) => d.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture);
private static double Pct(int part, int whole) => whole <= 0 ? 0 : Math.Round(100.0 * part / whole, 1);
}
}
@@ -49,6 +49,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
tenantId,
sessionId,
SessionStatus.Failed,
VerdictPaths.ManualFailed,
currentPhase: null, // Keep current phase
failureReason: "Manually marked as failed by administrator",
failureSource: "manual",
@@ -49,6 +49,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
tenantId,
sessionId,
SessionStatus.Succeeded,
VerdictPaths.ManualSucceeded,
currentPhase: null, // Keep current phase
failureReason: "Manually marked as succeeded by administrator",
adminMarkedAction: "Succeeded"
@@ -305,7 +305,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
$"enrollment session at {registration.StartedAt:yyyy-MM-dd HH:mm} UTC before this one " +
"reached a terminal state.";
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
registration.TenantId, stale.SessionId, SessionStatus.Incomplete,
registration.TenantId, stale.SessionId, SessionStatus.Incomplete, VerdictPaths.RegisterSuperseded,
failureReason: reason, failureSource: "superseded_by_reregistration");
if (!transitioned)
continue;
@@ -0,0 +1,281 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Helpers;
/// <summary>
/// One drift finding of the verdict-calibration radar (pure output; the maintenance partial
/// turns it into a tracker episode + ops event).
/// </summary>
public class VerdictCalibrationFinding
{
public string Kind { get; set; } = string.Empty;
public string VerdictPath { get; set; } = string.Empty;
public string Status { get; set; } = string.Empty;
public int WindowHitCount { get; set; }
public int WindowSessionCount { get; set; }
public int BaselineHitCount { get; set; }
public int BaselineSessionCount { get; set; }
public double WindowRatePct { get; set; }
public double BaselineRatePct { get; set; }
public double? Lift { get; set; }
public string WindowStartDate { get; set; } = string.Empty;
public string WindowEndDate { get; set; } = string.Empty;
}
/// <summary>
/// Pure drift detection over one tenant's verdict-calibration daily rows
/// (docs/backend/verdict-calibration.md) — the same statistics as the rule-regression radar
/// (7-day window vs the prior 28 days, lift ≥2×, Wilson-separated via
/// <see cref="MetricsMath.RateIncreaseSeparated"/>, small-n gates) applied to three questions:
/// <list type="bullet">
/// <item><b>share_regression</b> — one verdict path's share of all sessions doubled. Points at a rule
/// (or an agent path) whose conditions suddenly match much more often.</item>
/// <item><b>silence_share_regression</b> — the share of sessions the backend had to decide because
/// the agent went silent (sweep:* + maxlife:*) doubled. An agent-liveness signal, not a classifier one.</item>
/// <item><b>evidence_gap</b> — absolute gate: the pure fallthrough rule (r6) decides ≥20 % of the
/// window's classifier verdicts. The evidence the other rules need is not arriving.</item>
/// </list>
/// Derived (<c>legacy:*</c>) paths are excluded from all three — pre-instrumentation attribution
/// is too weak to alert on. Agent-declared paths (<c>agent:*</c>, <c>register:*</c>) are excluded
/// from the per-path share regression: their share tracks the customer's workflow mix (a
/// WhiteGlove rollout week doubles <c>agent:whiteglove_pending</c>; a Continue-Anyway tenant
/// lives on <c>agent:complete_soft</c>) — real signals, but not about the BACKEND classifier
/// this radar calibrates. First production pass (2026-08-23) fired exactly those three. Re-arm
/// mirrors the rule radar: share back under 1.5× baseline or the path stops occurring; the
/// evidence gap re-arms under 15 %.
/// </summary>
public static class VerdictCalibrationRadar
{
public const int WindowDays = RuleRegressionRadar.WindowDays;
public const int BaselineDays = RuleRegressionRadar.BaselineDays;
public const int MinWindowHits = RuleRegressionRadar.MinWindowHitSessions;
public const int MinWindowSessions = RuleRegressionRadar.MinWindowSessions;
public const double MinLift = RuleRegressionRadar.MinLift;
public const double ReArmLiftFactor = RuleRegressionRadar.ReArmLiftFactor;
/// <summary>Evidence gap fires at ≥20 % r6 share of the window's classifier verdicts (≥20 verdicts) and re-arms under 15 %.</summary>
public const double EvidenceGapFirePct = 20.0;
public const double EvidenceGapReArmPct = 15.0;
public const int EvidenceGapMinVerdicts = 20;
public const string SilenceGroupPath = "sweep+maxlife";
public const string EvidenceGapPath = "r6";
public const string GroupStatus = "*";
private const string LegacyOrigin = "legacy";
/// <summary>
/// True when a path is eligible for the per-path share regression: backend-decided paths only
/// (sweep / maxlife / late / retro / rule / manual / ingest). Agent-declared and registration
/// paths mirror customer workflow, legacy paths are derived.
/// </summary>
public static bool IsShareRegressionEligible(string path)
{
var origin = VerdictPaths.Origin(path);
return origin != LegacyOrigin
&& origin != "agent"
&& origin != "register";
}
public static List<VerdictCalibrationFinding> Evaluate(IReadOnlyList<VerdictCalibrationDailyAggregate> tenantRows, DateTime targetDateUtc)
{
var findings = new List<VerdictCalibrationFinding>();
var horizon = Summarize(tenantRows, targetDateUtc);
var windowStart = targetDateUtc.Date.AddDays(-(WindowDays - 1)).ToString("yyyy-MM-dd");
var windowEnd = targetDateUtc.Date.ToString("yyyy-MM-dd");
// 1. Per-path share regression.
foreach (var pair in horizon.Paths)
{
if (!IsShareRegressionEligible(pair.Key.Path)) continue;
var f = TryShareRegression(VerdictCalibrationAlertKinds.ShareRegression, pair.Key.Path, pair.Key.Status,
pair.Value.WindowHits, horizon.WindowSessions, pair.Value.BaselineHits, horizon.BaselineSessions, windowStart, windowEnd);
if (f != null) findings.Add(f);
}
// 2. Silence-share regression (group).
{
var f = TryShareRegression(VerdictCalibrationAlertKinds.SilenceShareRegression, SilenceGroupPath, GroupStatus,
horizon.SilenceWindowHits, horizon.WindowSessions, horizon.SilenceBaselineHits, horizon.BaselineSessions, windowStart, windowEnd);
if (f != null) findings.Add(f);
}
// 3. Evidence gap (absolute).
if (horizon.ClassifierWindowVerdicts >= EvidenceGapMinVerdicts)
{
var pct = 100.0 * horizon.R6WindowHits / horizon.ClassifierWindowVerdicts;
if (pct >= EvidenceGapFirePct)
{
findings.Add(new VerdictCalibrationFinding
{
Kind = VerdictCalibrationAlertKinds.EvidenceGap,
VerdictPath = EvidenceGapPath,
Status = GroupStatus,
WindowHitCount = horizon.R6WindowHits,
WindowSessionCount = horizon.ClassifierWindowVerdicts,
BaselineHitCount = horizon.R6BaselineHits,
BaselineSessionCount = horizon.ClassifierBaselineVerdicts,
WindowRatePct = Math.Round(pct, 1),
BaselineRatePct = horizon.ClassifierBaselineVerdicts > 0
? Math.Round(100.0 * horizon.R6BaselineHits / horizon.ClassifierBaselineVerdicts, 1) : 0,
Lift = null,
WindowStartDate = windowStart,
WindowEndDate = windowEnd,
});
}
}
return findings
.OrderByDescending(f => f.WindowRatePct)
.ThenBy(f => f.Kind, StringComparer.Ordinal)
.ThenBy(f => f.VerdictPath, StringComparer.Ordinal)
.ToList();
}
/// <summary>True when an ACTIVE alert may re-arm given the current horizon sums (or when its path is no longer eligible — episodes from an earlier, broader gate clear themselves).</summary>
public static bool ShouldReArm(VerdictCalibrationAlert alert, IReadOnlyList<VerdictCalibrationDailyAggregate> tenantRows, DateTime targetDateUtc)
{
if (alert.Kind == VerdictCalibrationAlertKinds.ShareRegression && !IsShareRegressionEligible(alert.VerdictPath))
return true;
var horizon = Summarize(tenantRows, targetDateUtc);
switch (alert.Kind)
{
case VerdictCalibrationAlertKinds.EvidenceGap:
if (horizon.ClassifierWindowVerdicts < EvidenceGapMinVerdicts) return true;
return 100.0 * horizon.R6WindowHits / horizon.ClassifierWindowVerdicts < EvidenceGapReArmPct;
case VerdictCalibrationAlertKinds.SilenceShareRegression:
return ReArmByRate(horizon.SilenceWindowHits, horizon.WindowSessions, horizon.SilenceBaselineHits, horizon.BaselineSessions);
default:
horizon.Paths.TryGetValue((alert.VerdictPath, alert.Status), out var sums);
return ReArmByRate(sums.WindowHits, horizon.WindowSessions, sums.BaselineHits, horizon.BaselineSessions);
}
}
/// <summary>Current numbers for an ongoing episode (refresh without re-firing).</summary>
public static (int WindowHits, int WindowSessions, int BaselineHits, int BaselineSessions) CurrentSums(
VerdictCalibrationAlert alert, IReadOnlyList<VerdictCalibrationDailyAggregate> tenantRows, DateTime targetDateUtc)
{
var horizon = Summarize(tenantRows, targetDateUtc);
switch (alert.Kind)
{
case VerdictCalibrationAlertKinds.EvidenceGap:
return (horizon.R6WindowHits, horizon.ClassifierWindowVerdicts, horizon.R6BaselineHits, horizon.ClassifierBaselineVerdicts);
case VerdictCalibrationAlertKinds.SilenceShareRegression:
return (horizon.SilenceWindowHits, horizon.WindowSessions, horizon.SilenceBaselineHits, horizon.BaselineSessions);
default:
horizon.Paths.TryGetValue((alert.VerdictPath, alert.Status), out var sums);
return (sums.WindowHits, horizon.WindowSessions, sums.BaselineHits, horizon.BaselineSessions);
}
}
private static bool ReArmByRate(int windowHits, int windowSessions, int baselineHits, int baselineSessions)
{
if (windowHits == 0 || windowSessions == 0) return true;
if (baselineSessions == 0 || baselineHits == 0) return false; // zero-baseline episode re-arms only via hits-stopped
var windowRate = (double)windowHits / windowSessions;
var baselineRate = (double)baselineHits / baselineSessions;
return windowRate < ReArmLiftFactor * baselineRate;
}
private static VerdictCalibrationFinding? TryShareRegression(string kind, string path, string status,
int windowHits, int windowSessions, int baselineHits, int baselineSessions, string windowStart, string windowEnd)
{
if (windowHits < MinWindowHits) return null;
if (windowSessions < MinWindowSessions) return null;
if (baselineSessions <= 0) return null;
var windowRate = (double)windowHits / windowSessions;
var baselineRate = (double)baselineHits / baselineSessions;
if (windowRate < MinLift * baselineRate) return null;
if (!MetricsMath.RateIncreaseSeparated(windowHits, windowSessions, baselineHits, baselineSessions)) return null;
return new VerdictCalibrationFinding
{
Kind = kind,
VerdictPath = path,
Status = status,
WindowHitCount = windowHits,
WindowSessionCount = windowSessions,
BaselineHitCount = baselineHits,
BaselineSessionCount = baselineSessions,
WindowRatePct = Math.Round(windowRate * 100, 1),
BaselineRatePct = Math.Round(baselineRate * 100, 1),
Lift = baselineRate > 0 ? Math.Round(windowRate / baselineRate, 1) : null,
WindowStartDate = windowStart,
WindowEndDate = windowEnd,
};
}
internal readonly record struct PathSums(int WindowHits, int BaselineHits);
internal sealed class Horizon
{
public int WindowSessions;
public int BaselineSessions;
public int SilenceWindowHits;
public int SilenceBaselineHits;
public int R6WindowHits;
public int R6BaselineHits;
public int ClassifierWindowVerdicts;
public int ClassifierBaselineVerdicts;
public readonly Dictionary<(string Path, string Status), PathSums> Paths = new();
}
/// <summary>
/// Sums the daily rows into the trailing window [target−6d, target] and the prior baseline
/// [target−34d, target−7d] (ISO-string date compare, rows outside the horizon ignored).
/// Silence group = sweep:* + maxlife:* (incl. non-rule sweep paths like sweep:stalled — the
/// question is "how often did the backend have to decide", not which rule did). Classifier
/// verdicts = stamped rule paths (<see cref="VerdictPaths.IsClassifierPath"/>); r6 = their
/// fallthrough subset.
/// </summary>
internal static Horizon Summarize(IReadOnlyList<VerdictCalibrationDailyAggregate> rows, DateTime targetDateUtc)
{
var windowStart = targetDateUtc.Date.AddDays(-(WindowDays - 1)).ToString("yyyy-MM-dd");
var baselineStart = targetDateUtc.Date.AddDays(-(WindowDays - 1 + BaselineDays)).ToString("yyyy-MM-dd");
var target = targetDateUtc.Date.ToString("yyyy-MM-dd");
var h = new Horizon();
foreach (var row in rows)
{
if (string.CompareOrdinal(row.Date, target) > 0) continue;
bool inWindow;
if (string.CompareOrdinal(row.Date, windowStart) >= 0) inWindow = true;
else if (string.CompareOrdinal(row.Date, baselineStart) >= 0) inWindow = false;
else continue;
if (inWindow) h.WindowSessions += row.SessionCount; else h.BaselineSessions += row.SessionCount;
foreach (var b in row.Buckets)
{
if (b.Count == 0) continue;
var key = (b.VerdictPath, b.Status);
h.Paths.TryGetValue(key, out var sums);
h.Paths[key] = inWindow
? new PathSums(sums.WindowHits + b.Count, sums.BaselineHits)
: new PathSums(sums.WindowHits, sums.BaselineHits + b.Count);
var origin = VerdictPaths.Origin(b.VerdictPath);
if (origin == VerdictPaths.OriginSweep || origin == VerdictPaths.OriginMaxLifetime)
{
if (inWindow) h.SilenceWindowHits += b.Count; else h.SilenceBaselineHits += b.Count;
}
if (VerdictPaths.IsClassifierPath(b.VerdictPath))
{
if (inWindow) h.ClassifierWindowVerdicts += b.Count; else h.ClassifierBaselineVerdicts += b.Count;
if (b.VerdictPath.EndsWith(":" + ClassifierRules.R6Fallthrough, StringComparison.Ordinal))
{
if (inWindow) h.R6WindowHits += b.Count; else h.R6BaselineHits += b.Count;
}
}
}
}
return h;
}
}
@@ -0,0 +1,127 @@
using System;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Helpers
{
/// <summary>
/// Read-side attribution for sessions written BEFORE <see cref="SessionSummary.VerdictPath"/>
/// was stamped. No mass backfill: the calibration aggregate calls <see cref="Derive"/> and
/// counts the result with <c>Derived=true</c>, so pre-instrumentation history still shows up
/// in the matrix and is visibly weaker. The derivation reads the fixed reason literals the
/// writers emit (classifier rule prefixes, the max-lifetime / retro suffixes, the manual and
/// rule attributions) — when the origin is ambiguous but the rule is known the path is
/// <c>legacy:{rule}</c>; when nothing is attributable it is <see cref="VerdictPaths.LegacyUnknown"/>.
/// A stamped row is returned verbatim with <c>Derived=false</c>.
/// </summary>
public static class VerdictPathDerivation
{
private const string OriginLegacy = "legacy";
// Literals are owned by EnrollmentTimeoutClassifier / the writers; pinned by
// VerdictPathDerivationTests so a wording change there fails here, not silently.
private const string R1Prefix = "Enrollment reported an explicit failure before timeout";
private const string R1bAwaitingPrefix = "Pre-provisioning (WhiteGlove) completed; no user sign-in since Part 2 resumed";
private const string R1bSucceededPrefix = "Reconciled at timeout: pre-provisioning (WhiteGlove Part 1) completed";
private const string R1cPrefix = "Reconciled: self-deploying profile";
private const string R2Prefix = "Reconciled at timeout: Account Setup completed";
private const string R3Prefix = "Agent emergency break fired";
private const string R4Prefix = "Reconciled at timeout: user completed setup";
private const string R5AwaitingPrefix = "Device Setup completed; awaiting user / Account Setup phase";
private const string R5IncompletePrefix = "No completion signal within";
private const string R6Prefix = "No Device Setup completion or explicit failure signal observed before timeout";
private const string SupersededPrefix = "Superseded by session";
private const string SweepStalledPrefix = "Agent silent for";
private const string AgentStallProbePrefix = "Agent reported stall after";
private const string EspFallbackLiteral = "ESP failure (backend fallback)";
/// <summary>Pre-classifier blanket timeout verdict (LegacyReclassificationService) — never agent-reported.</summary>
private const string LegacyBlanketTimeoutPrefix = "Session timed out";
private const string MaxLifetimeSuffix = "max-lifetime watchdog";
private const string RetroSuffix = "Retro-reclassified";
public static (string Path, bool Derived) Derive(SessionSummary s)
{
if (s == null) throw new ArgumentNullException(nameof(s));
if (!string.IsNullOrWhiteSpace(s.VerdictPath))
return (s.VerdictPath!, false);
return (DeriveLegacy(s), true);
}
private static string DeriveLegacy(SessionSummary s)
{
var failureReason = s.FailureReason ?? string.Empty;
var reconcileReason = s.ReconcileReason ?? string.Empty;
var failureSource = s.FailureSource ?? string.Empty;
// Unambiguous attributions first — independent of status text.
if (string.Equals(s.AdminMarkedAction, "Succeeded", StringComparison.OrdinalIgnoreCase))
return VerdictPaths.ManualSucceeded;
if (string.Equals(s.AdminMarkedAction, "Failed", StringComparison.OrdinalIgnoreCase))
return VerdictPaths.ManualFailed;
if (failureSource.StartsWith("rule:", StringComparison.OrdinalIgnoreCase))
return VerdictPaths.Compose(VerdictPaths.OriginRule, failureSource.Substring(5));
switch (s.Status)
{
case SessionStatus.Succeeded:
if (reconcileReason.Length == 0
|| reconcileReason.StartsWith("Late completion report received", StringComparison.Ordinal))
return s.EspSoftFailure ? VerdictPaths.AgentCompleteSoft : VerdictPaths.AgentComplete;
if (Starts(reconcileReason, R1bSucceededPrefix)) return Classifier(reconcileReason, ClassifierRules.R1bWhiteGloveSucceeded);
if (Starts(reconcileReason, R1cPrefix)) return Classifier(reconcileReason, ClassifierRules.R1cSelfDeploying);
if (Starts(reconcileReason, R2Prefix)) return Classifier(reconcileReason, ClassifierRules.R2AccountSetupComplete);
if (Starts(reconcileReason, R4Prefix)) return Classifier(reconcileReason, ClassifierRules.R4DesktopHello);
return VerdictPaths.LegacyUnknown;
case SessionStatus.Failed:
if (Starts(failureReason, R1Prefix)) return Classifier(failureReason, ClassifierRules.R1ExplicitFailure);
if (string.Equals(failureSource, "max_lifetime_watchdog", StringComparison.OrdinalIgnoreCase))
return VerdictPaths.Compose(VerdictPaths.OriginMaxLifetime, ClassifierRules.R1ExplicitFailure);
if (failureReason.Contains(EspFallbackLiteral, StringComparison.Ordinal)) return VerdictPaths.AgentEspFailureFallback;
if (Starts(failureReason, LegacyBlanketTimeoutPrefix)) return VerdictPaths.LegacyUnknown;
if (failureReason.Length > 0 && failureSource.Length == 0) return VerdictPaths.AgentFailed;
return VerdictPaths.LegacyUnknown;
case SessionStatus.Incomplete:
if (Starts(failureReason, R3Prefix)) return Classifier(failureReason, ClassifierRules.R3EmergencyBreak);
if (Starts(failureReason, R5IncompletePrefix)) return Classifier(failureReason, ClassifierRules.R5DeviceSetupIncomplete);
if (Starts(failureReason, R6Prefix)) return Classifier(failureReason, ClassifierRules.R6Fallthrough);
if (Starts(failureReason, SupersededPrefix)) return VerdictPaths.Compose(OriginLegacy, "superseded");
return VerdictPaths.LegacyUnknown;
case SessionStatus.AwaitingUser:
if (Starts(failureReason, R1bAwaitingPrefix)) return VerdictPaths.Compose(OriginLegacy, "wg_awaiting");
if (Starts(failureReason, R5AwaitingPrefix)) return Classifier(failureReason, ClassifierRules.R5DeviceSetupAwaiting);
return VerdictPaths.LegacyUnknown;
case SessionStatus.Stalled:
if (Starts(failureReason, SweepStalledPrefix)) return VerdictPaths.SweepStalled;
if (Starts(failureReason, AgentStallProbePrefix)) return VerdictPaths.AgentStallProbe;
return VerdictPaths.LegacyUnknown;
case SessionStatus.Pending:
return VerdictPaths.AgentWhiteGlovePending;
default:
return VerdictPaths.LegacyUnknown;
}
}
/// <summary>
/// Classifier verdict with a known rule: the origin is recoverable only from the suffixes
/// the max-lifetime and retro writers append; sweep / late-reconcile share the bare text
/// and stay <c>legacy:{rule}</c>.
/// </summary>
private static string Classifier(string reason, string rule)
{
if (reason.Contains(MaxLifetimeSuffix, StringComparison.Ordinal))
return VerdictPaths.Compose(VerdictPaths.OriginMaxLifetime, rule);
if (reason.Contains(RetroSuffix, StringComparison.Ordinal))
return VerdictPaths.Compose(VerdictPaths.OriginRetro, rule);
return VerdictPaths.Compose(OriginLegacy, rule);
}
private static bool Starts(string text, string prefix)
=> text.StartsWith(prefix, StringComparison.Ordinal);
}
}
@@ -1,5 +1,6 @@
using System.Diagnostics;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Middleware;
@@ -140,6 +141,15 @@ public class RequestTelemetryMiddleware : IFunctionsWorkerMiddleware
if (caughtException != null)
requestTelemetry.Properties["ExceptionType"] = caughtException.GetType().Name;
// Bypass the worker's adaptive sampling for this item. The SDK sampling processor
// passes through any item whose SamplingPercentage is already set, so this is the
// per-item opt-out that host.json "excludedTypes" (host process only) cannot provide.
// Live-verified 2026-08-23 BEFORE this change: worker request rows carried
// ItemCount 2-5 on weekdays (~30% of individual requests missing), so count()-based
// queries undercounted. This copy is the single canonical request record — the host
// duplicate is dropped by a workspace DCR transformation on AppRequests.
((ISupportSampling)requestTelemetry).SamplingPercentage = 100;
try
{
_telemetryClient.TrackRequest(requestTelemetry);
@@ -71,6 +71,10 @@ builder.Services
// unchanged host duplicate). The host request item is emitted by the HOST process and cannot be
// dropped from worker code or by host.json "Host.Results" level. Re-attempt only with a mechanism
// verified against the isolated-worker telemetry pipeline — do not re-add the manual chain.
// 2026-08-23: solved differently — RequestTelemetryMiddleware sets SamplingPercentage=100 on its
// own item (per-item sampling bypass), and the host duplicate is removed by a workspace DCR
// transformation on AppRequests (keeps rows with Properties.Source=='WorkerMiddleware' plus
// non-HTTP host rows with empty Url, i.e. timer/queue invocations). See docs/backend/telemetry-ingest-shaping.md.
// Drop successful Azure Storage dependencies (Table/Queue/Blob) from the worker telemetry
// pipeline to curb AppDependencies ingestion cost — this backend is storage-I/O heavy and those
@@ -226,6 +230,7 @@ builder.Services.AddHostedService<AutopilotMonitor.Functions.Services.Offboardin
// no AddHostedService call needed. DI for its dependencies comes from the singletons above.
builder.Services.AddHostedService<TableInitializerService>(); // Initialize all tables at startup
builder.Services.AddHostedService<StartupTelemetryService>(); // BackendStartupMs / BackendTableInitMs metrics on ApplicationStarted
// Data Access Layer — repository interfaces backed by Table Storage.
// To switch to Cosmos DB: replace AddTableStorageDataAccess() with AddCosmosDataAccess().
@@ -0,0 +1,81 @@
using System;
namespace AutopilotMonitor.Functions.Security
{
/// <summary>
/// Compares the Entra TenantId stamped into the agent's mTLS client certificate against the
/// tenant the request claims to belong to.
/// <para>
/// Without this, a validated certificate only proves "issued by the Intune MDM Device CA to
/// some tenant" — the chain is pinned to Microsoft's roots, which every Intune tenant on the
/// planet shares. The tenant scoping comes solely from the downstream device validators, so an
/// attacker with their own Intune tenant and a known serial number of the victim tenant would
/// pass certificate validation. Binding the cert's tenant to the requested tenant closes that.
/// </para>
/// </summary>
/// <remarks>
/// SHADOW MODE (stage 1): the outcome is observed and logged only — it never changes an
/// authorization decision. Enforcement is stage 2, once telemetry shows how many field
/// certificates actually carry the extension. Grep marker for the stage-2 change:
/// <c>CERT-TENANT-BINDING-SHADOW</c>.
/// </remarks>
public static class CertTenantBinding
{
/// <summary>
/// Stable outcome codes emitted in the <c>AgentCertTenantBinding</c> structured log.
/// Keep these strings stable — they are queried by exact match in KQL
/// (<c>customDimensions.Outcome == "Mismatch"</c>).
/// </summary>
public static class Outcome
{
/// <summary>Certificate tenant equals the requested tenant — the expected case.</summary>
public const string Match = "Match";
/// <summary>Certificate belongs to a different tenant than the request claims.</summary>
public const string Mismatch = "Mismatch";
/// <summary>Certificate carries no tenant extension (older enrollment / unexpected CA).</summary>
public const string ExtensionMissing = "ExtensionMissing";
/// <summary>Tenant extension present but undecodable.</summary>
public const string Unparseable = "Unparseable";
/// <summary>Requested tenant id was not a GUID, so no meaningful comparison is possible.</summary>
public const string RequestTenantNotAGuid = "RequestTenantNotAGuid";
}
/// <summary>
/// Evaluates the binding. Pure — no logging, no I/O, no side effects.
/// </summary>
/// <param name="certTenantId">Tenant GUID decoded from the client certificate, if any.</param>
/// <param name="status">Whether the certificate carried a decodable tenant extension.</param>
/// <param name="requestedTenantId">The tenant id the request is scoped to.</param>
/// <returns>One of the <see cref="Outcome"/> codes.</returns>
public static string Evaluate(Guid? certTenantId, CertTenantIdStatus status, string? requestedTenantId)
{
switch (status)
{
case CertTenantIdStatus.ExtensionMissing:
return Outcome.ExtensionMissing;
case CertTenantIdStatus.Unparseable:
return Outcome.Unparseable;
}
if (certTenantId == null)
return Outcome.ExtensionMissing;
if (!Guid.TryParse(requestedTenantId, out var requested))
return Outcome.RequestTenantNotAGuid;
return certTenantId.Value == requested ? Outcome.Match : Outcome.Mismatch;
}
/// <summary>
/// Whether an outcome would block the request once stage-2 enforcement is switched on.
/// Nothing consumes this for authorization yet — it exists so the shadow telemetry can be
/// read as "would this have been rejected?" without re-deriving the rule at query time.
/// </summary>
public static bool WouldRejectUnderEnforcement(string outcome) =>
outcome == Outcome.Mismatch;
}
}
@@ -6,6 +6,7 @@ using System.Linq;
using System.Reflection;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using AutopilotMonitor.Shared.Security;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Security
@@ -138,7 +139,8 @@ namespace AutopilotMonitor.Functions.Security
if (cached.ExpiresAt > DateTime.UtcNow)
{
logger?.LogDebug($"Certificate validation cache HIT for thumbprint {thumbprint}");
return cached.Result;
// Copy so FromCache is per-call state and never mutates the shared cached instance.
return cached.Result.AsCacheHit();
}
else
{
@@ -253,6 +255,12 @@ namespace AutopilotMonitor.Functions.Security
};
}
// 4. Extract the Entra TenantId the CA stamped into the cert (OID 1.2.840.113556.5.14).
// Pure function of the cert, so it is cached alongside the validation result.
// Read only - the tenant binding it enables is evaluated (and currently only observed)
// by SecurityValidator; see CertTenantBinding.
var (certTenantId, certTenantIdStatus) = ExtractEntraTenantId(certificate);
// All checks passed - create and cache success result
logger?.LogInformation($"Certificate validated successfully: Thumbprint={thumbprint}");
var successResult = new CertificateValidationResult
@@ -260,7 +268,9 @@ namespace AutopilotMonitor.Functions.Security
IsValid = true,
Thumbprint = thumbprint,
Subject = certificate.Subject,
Issuer = certificate.Issuer
Issuer = certificate.Issuer,
CertTenantId = certTenantId,
CertTenantIdStatus = certTenantIdStatus
};
// Cache successful validation
@@ -288,6 +298,30 @@ namespace AutopilotMonitor.Functions.Security
}
}
/// <summary>
/// Reads the Entra TenantId the Intune MDM Device CA stamps into the client certificate
/// (OID <c>1.2.840.113556.5.14</c>). Never throws — a cert without the extension, or with an
/// undecodable one, yields the corresponding status instead of an error, because this value
/// is currently observed only and must never turn a valid cert into a rejected one.
/// </summary>
/// <remarks>
/// Deliberately NOT OID <c>1.2.840.113556.5.6</c>: that one carries the Intune <b>Account</b>
/// ID, which is a different GUID from the Entra tenant ID.
/// </remarks>
private static (Guid? TenantId, CertTenantIdStatus Status) ExtractEntraTenantId(X509Certificate2 certificate)
{
var ext = certificate.Extensions
.OfType<X509Extension>()
.FirstOrDefault(e => e.Oid?.Value == MsDeviceCertificateOids.IntuneCertEntraTenantIdOid);
if (ext?.RawData == null)
return (null, CertTenantIdStatus.ExtensionMissing);
return MsDeviceCertificateOids.TryParseGuid(ext.RawData, out var tenantId)
? (tenantId, CertTenantIdStatus.Present)
: (null, CertTenantIdStatus.Unparseable);
}
/// <summary>
/// Loads all certs whose embedded resource filename starts with the given prefix.
/// Multiple PEM blocks per file are supported (cross-signed bundles). Duplicates
@@ -378,5 +412,58 @@ namespace AutopilotMonitor.Functions.Security
/// Error message if validation failed
/// </summary>
public string? ErrorMessage { get; set; }
/// <summary>
/// Entra TenantId stamped into the certificate by the Intune MDM Device CA
/// (OID <c>1.2.840.113556.5.14</c>), or <c>null</c> when
/// <see cref="CertTenantIdStatus"/> is not <see cref="Security.CertTenantIdStatus.Present"/>.
/// </summary>
public Guid? CertTenantId { get; set; }
/// <summary>
/// Whether the certificate carried a decodable Entra TenantId extension.
/// </summary>
public CertTenantIdStatus CertTenantIdStatus { get; set; }
/// <summary>
/// True when this result was served from the 5-minute thumbprint cache rather than freshly
/// validated. Used to sample per-request observability down to roughly one line per
/// certificate per cache window.
/// </summary>
public bool FromCache { get; set; }
/// <summary>
/// Returns a copy of this result marked as a cache hit, so the shared cached instance is
/// never mutated by a caller.
/// </summary>
internal CertificateValidationResult AsCacheHit() => new()
{
IsValid = IsValid,
Thumbprint = Thumbprint,
Subject = Subject,
Issuer = Issuer,
ErrorMessage = ErrorMessage,
CertTenantId = CertTenantId,
CertTenantIdStatus = CertTenantIdStatus,
FromCache = true
};
}
/// <summary>
/// Whether a validated client certificate carried a decodable Entra TenantId extension.
/// </summary>
public enum CertTenantIdStatus
{
/// <summary>Certificate was not validated far enough to look, or validation failed.</summary>
NotEvaluated = 0,
/// <summary>Extension present and decoded into a GUID.</summary>
Present = 1,
/// <summary>Certificate carries no such extension (expected on older enrollments).</summary>
ExtensionMissing = 2,
/// <summary>Extension present but its value could not be decoded as a GUID.</summary>
Unparseable = 3
}
}
@@ -464,6 +464,7 @@ public static class EndpointAccessPolicyCatalog
new("GET", "global/metrics/fleet-health", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/time-attribution", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/device-journeys", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/verdict-calibration", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/apps/list", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/apps/{appName}/analytics", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/apps/{appName}/sessions", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
@@ -275,6 +275,12 @@ namespace AutopilotMonitor.Functions.Security
};
}
// 1b. CERT-TENANT-BINDING-SHADOW — observe only, never decide.
// The cert chain is pinned to Microsoft's Intune roots, which every Intune tenant shares,
// so a valid cert alone does not prove the caller belongs to THIS tenant. Stage 1 measures
// how reliably the certificate's own tenant stamp matches before stage 2 enforces it.
ObserveCertTenantBinding(certValidation, tenantId, req, sessionId);
// 2. Check rate limit (DoS protection)
// Effective limit = per-tenant override if set, otherwise the global default.
var rateLimitValue = await ResolveDeviceRateLimitAsync(config);
@@ -514,6 +520,71 @@ namespace AutopilotMonitor.Functions.Security
return false;
}
/// <summary>
/// CERT-TENANT-BINDING-SHADOW — stage 1 observability for the client-certificate tenant
/// binding. Evaluates whether the tenant stamped into the agent's certificate matches the
/// tenant the request claims, and logs the outcome. It deliberately returns nothing and
/// changes no state: enforcement is stage 2, after this telemetry shows the miss rate.
/// </summary>
/// <remarks>
/// Volume control: outcomes that need enforcement decisions (mismatch, missing or
/// undecodable extension) are always logged — they are expected to be rare and each one is
/// individually interesting. The bulk "Match" case is logged only when the certificate was
/// freshly validated rather than served from the 5-minute thumbprint cache, which samples it
/// down to roughly one line per certificate per cache window while keeping the ratio between
/// numerator and denominator meaningful (both are counted per cert-window, not per request).
/// </remarks>
private void ObserveCertTenantBinding(
CertificateValidationResult certValidation,
string tenantId,
HttpRequestData req,
string? sessionId)
{
try
{
var outcome = CertTenantBinding.Evaluate(
certValidation.CertTenantId,
certValidation.CertTenantIdStatus,
tenantId);
if (outcome == CertTenantBinding.Outcome.Match && certValidation.FromCache)
return;
var agentVersion = req.Headers.Contains("X-Agent-Version")
? req.Headers.GetValues("X-Agent-Version").FirstOrDefault() ?? "n/a"
: "n/a";
// Only the cert's tenant is logged, never the raw certificate. On a mismatch this is
// the attacker-controlled side of the comparison and the single most useful field.
const string template =
"AgentCertTenantBinding outcome={Outcome} enforced={Enforced} wouldReject={WouldReject} "
+ "tenant={TenantId} certTenant={CertTenantId} status={Status} thumbprint={Thumbprint} "
+ "session={SessionId} ver={AgentVersion}";
var certTenant = certValidation.CertTenantId?.ToString() ?? "n/a";
var wouldReject = CertTenantBinding.WouldRejectUnderEnforcement(outcome);
if (outcome == CertTenantBinding.Outcome.Match)
{
_logger.LogInformation(template,
outcome, false, wouldReject, tenantId, certTenant, certValidation.CertTenantIdStatus,
certValidation.Thumbprint ?? "n/a", sessionId ?? "n/a", agentVersion);
}
else
{
_logger.LogWarning(template,
outcome, false, wouldReject, tenantId, certTenant, certValidation.CertTenantIdStatus,
certValidation.Thumbprint ?? "n/a", sessionId ?? "n/a", agentVersion);
}
}
catch (Exception ex)
{
// Shadow-mode observability must never be able to fail a request that would
// otherwise have been accepted.
_logger.LogWarning(ex, "AgentCertTenantBinding observation failed for tenant {TenantId}", tenantId);
}
}
/// <summary>
/// Emits a structured warning with request-side context (path, hardware headers, agent
/// version) at every fail-fast rejection point. Pairs with the cert-side warnings emitted
@@ -1,5 +1,4 @@
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Extensions.Caching.Memory;
@@ -21,8 +20,15 @@ namespace AutopilotMonitor.Functions.Services
private readonly IMemoryCache _cache;
private readonly ILogger<DistressRateLimitService> _logger;
// Per-key locks (same proven pattern as RateLimitService)
private readonly ConcurrentDictionary<string, object> _locks = new();
// History + lock in one cache entry (same pattern as RateLimitService): same list ⇒ same lock.
private sealed class RequestBucket
{
public readonly List<DateTime> History = new();
public readonly object Sync = new();
}
// Serializes bucket creation (GetOrCreate is not atomic); hot path is a lock-free TryGetValue.
private readonly object _createLock = new();
// Layer 1: Per-IP
private const int MaxPerIp = 5;
@@ -91,17 +97,12 @@ namespace AutopilotMonitor.Functions.Services
/// </summary>
private bool CheckSlidingWindow(string cacheKey, int maxRequests, TimeSpan window, DateTime now)
{
var lockObj = _locks.GetOrAdd(cacheKey, _ => new object());
var history = _cache.GetOrCreate(cacheKey, entry =>
{
entry.SlidingExpiration = window;
entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(2);
entry.RegisterPostEvictionCallback((key, _, _, _) => _locks.TryRemove((string)key, out _));
return new List<DateTime>();
})!;
var bucket = GetOrCreateBucket(cacheKey, window);
lock (lockObj)
lock (bucket.Sync)
{
var history = bucket.History;
// Remove entries outside the window
var windowStart = now.Subtract(window);
history.RemoveAll(ts => ts < windowStart);
@@ -113,16 +114,26 @@ namespace AutopilotMonitor.Functions.Services
history.Add(now);
_cache.Set(cacheKey, history, new MemoryCacheEntryOptions
{
SlidingExpiration = window,
AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(2)
});
return true;
}
}
private RequestBucket GetOrCreateBucket(string cacheKey, TimeSpan window)
{
if (_cache.TryGetValue(cacheKey, out RequestBucket? existing) && existing != null)
return existing;
lock (_createLock)
{
return _cache.GetOrCreate(cacheKey, entry =>
{
entry.SlidingExpiration = window;
entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(2);
return new RequestBucket();
})!;
}
}
private bool IsCircuitOpen(DateTime now)
{
lock (_globalLock)
@@ -320,7 +320,7 @@ namespace AutopilotMonitor.Functions.Services
/// <param name="isPreProvisioned">Session's IsPreProvisioned flag (WhiteGlove Part-2 gate input).</param>
/// <param name="resumedAt">Session's ResumedAt — set exclusively by the whiteglove_resumed ingest.</param>
/// <param name="isSelfDeployingProfile">Session's IsSelfDeployingProfile flag (rule 1c gate input).</param>
public static (SessionStatus Status, string Reason) ClassifyTimedOutSession(
public static (SessionStatus Status, string Reason, string Rule) ClassifyTimedOutSession(
EspProvisioningRollup rollup, DateTime startedAtUtc, DateTime nowUtc, int graceHours,
DateTime? lastEventAtUtc = null, bool isPreProvisioned = false, DateTime? resumedAt = null,
bool isSelfDeployingProfile = false)
@@ -328,7 +328,7 @@ namespace AutopilotMonitor.Functions.Services
// 1. An explicit terminal failure event is a real failure (defensive — such a session
// would normally already be terminal via ingest and not reach the sweep).
if (rollup.HasExplicitFailure)
return (SessionStatus.Failed, "Enrollment reported an explicit failure before timeout");
return (SessionStatus.Failed, "Enrollment reported an explicit failure before timeout", ClassifierRules.R1ExplicitFailure);
// 1b. WhiteGlove Part 2 with zero user evidence since the resume: the device is parked
// between technician and end user (sealed box / powered off at the logon screen).
@@ -341,11 +341,11 @@ namespace AutopilotMonitor.Functions.Services
if (IsWhiteGloveAwaitingUser(rollup, isPreProvisioned, resumedAt))
{
if ((nowUtc - startedAtUtc).TotalHours < graceHours)
return (SessionStatus.AwaitingUser, WhiteGloveAwaitingUserReason(rollup));
return (SessionStatus.AwaitingUser, WhiteGloveAwaitingUserReason(rollup), ClassifierRules.R1bWhiteGloveAwaiting);
return (SessionStatus.Succeeded, AppendReconcileTiming(
$"Reconciled at timeout: pre-provisioning (WhiteGlove Part 1) completed — no user sign-in observed within {graceHours}h grace",
startedAtUtc, nowUtc, lastEventAtUtc));
startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R1bWhiteGloveSucceeded);
}
// 1c. Self-deploying profile with Device Setup fully provisioned: there is no user phase
@@ -355,7 +355,7 @@ namespace AutopilotMonitor.Functions.Services
// contradict it (a kiosk autologon is not a user) and an explicit failure already
// returned in rule 1.
if (IsSelfDeployingProvisioned(rollup, isSelfDeployingProfile))
return (SessionStatus.Succeeded, SelfDeployingReconcileReason(startedAtUtc, nowUtc, lastEventAtUtc));
return (SessionStatus.Succeeded, SelfDeployingReconcileReason(startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R1cSelfDeploying);
// 2. Account Setup fully succeeded (or a terminal completion event) → reconcile to success.
// Once Part 2 has resumed, the Part-1 whiteglove_complete in the same stream no longer
@@ -368,14 +368,15 @@ namespace AutopilotMonitor.Functions.Services
if (rollup.AccountSetupAllSucceeded || terminalCompleteProvesOutcome)
return (SessionStatus.Succeeded, AppendReconcileTiming(
"Reconciled at timeout: Account Setup completed (all subcategories succeeded / enrollment_complete observed)",
startedAtUtc, nowUtc, lastEventAtUtc));
startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R2AccountSetupComplete);
// 3. The agent's absolute-age emergency break fired — the agent has cleaned up and exited, so
// nothing more will ever arrive for this session. Terminalize NOW (skip the AwaitingUser grace):
// it did not complete (2) and did not explicitly fail (1), so the honest verdict is Incomplete.
if (rollup.HasAgentEmergencyBreak)
return (SessionStatus.Incomplete,
"Agent emergency break fired (absolute session-age cap) — agent gone without completion");
"Agent emergency break fired (absolute session-age cap) — agent gone without completion",
ClassifierRules.R3EmergencyBreak);
// 4. The user demonstrably finished setup: a real-user desktop was observed AND
// Windows Hello reached a positive terminal (provisioned or skipped). Those are
@@ -410,7 +411,7 @@ namespace AutopilotMonitor.Functions.Services
: "agent went silent before reporting completion";
return (SessionStatus.Succeeded, AppendReconcileTiming(
$"Reconciled at timeout: user completed setup ({evidence}) — {detail}",
startedAtUtc, nowUtc, lastEventAtUtc));
startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R4DesktopHello);
}
// 5. Device Setup fully provisioned (device is AADJ + MDM-enrolled), user phase pending.
@@ -423,7 +424,8 @@ namespace AutopilotMonitor.Functions.Services
? $" (Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal})"
: " (Account Setup not yet started)";
return (SessionStatus.AwaitingUser,
$"Device Setup completed; awaiting user / Account Setup phase — agent silent, within {graceHours}h grace{acct}");
$"Device Setup completed; awaiting user / Account Setup phase — agent silent, within {graceHours}h grace{acct}",
ClassifierRules.R5DeviceSetupAwaiting);
}
// "0/0" would suggest a parsed rollup that never existed — when no Account Setup
@@ -432,12 +434,14 @@ namespace AutopilotMonitor.Functions.Services
? $"last Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal}"
: "Account Setup progress never observed";
return (SessionStatus.Incomplete,
$"No completion signal within {graceHours}h grace after Device Setup completed ({acctDetail})");
$"No completion signal within {graceHours}h grace after Device Setup completed ({acctDetail})",
ClassifierRules.R5DeviceSetupIncomplete);
}
// 6. Silent before Device Setup completed, with no explicit failure → unknown, not a failure.
return (SessionStatus.Incomplete,
"No Device Setup completion or explicit failure signal observed before timeout");
"No Device Setup completion or explicit failure signal observed before timeout",
ClassifierRules.R6Fallthrough);
}
/// <summary>
@@ -286,7 +286,7 @@ namespace AutopilotMonitor.Functions.Services
var effectiveStart = session?.ResumedAt ?? session?.StartedAt ?? triggerEvent.Timestamp;
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
var (targetStatus, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session?.LastEventAt ?? c.LatestEventTimestamp,
isPreProvisioned: session?.IsPreProvisioned == true, resumedAt: session?.ResumedAt,
isSelfDeployingProfile: session?.IsSelfDeployingProfile == true);
@@ -315,7 +315,8 @@ namespace AutopilotMonitor.Functions.Services
}
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, targetStatus, triggerEvent.Phase, reason,
request.TenantId, request.SessionId, targetStatus, VerdictPaths.Compose(VerdictPaths.OriginMaxLifetime, rule),
triggerEvent.Phase, reason,
completedAt: isTerminalNonSuccess ? triggerEvent.Timestamp : (DateTime?)null,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
failureSource: isTerminalNonSuccess ? "max_lifetime_watchdog" : null,
@@ -357,7 +358,7 @@ namespace AutopilotMonitor.Functions.Services
var now = DateTime.UtcNow;
var effectiveStart = session.ResumedAt ?? session.StartedAt;
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
var (targetStatus, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt ?? c.LatestEventTimestamp,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
@@ -367,6 +368,7 @@ namespace AutopilotMonitor.Functions.Services
var healed = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Succeeded,
VerdictPaths.Compose(VerdictPaths.OriginLateReconcile, rule),
failureReason: reason,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
if (healed)
@@ -406,7 +408,7 @@ namespace AutopilotMonitor.Functions.Services
return false;
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.AwaitingUser,
request.TenantId, request.SessionId, SessionStatus.AwaitingUser, VerdictPaths.IngestWhiteGloveAwaiting,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
failureReason: EnrollmentTimeoutClassifier.WhiteGloveAwaitingUserReason(rollup));
if (transitioned)
@@ -465,7 +467,9 @@ namespace AutopilotMonitor.Functions.Services
? c.CompletionEvent.Data["completionSource"]?.ToString() : null;
statusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Succeeded, c.CompletionEvent.Phase,
request.TenantId, request.SessionId, SessionStatus.Succeeded,
espSoftFailure ? VerdictPaths.AgentCompleteSoft : VerdictPaths.AgentComplete,
c.CompletionEvent.Phase,
completedAt: c.CompletionEvent.Timestamp,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
espSoftFailure: espSoftFailure, completionSource: completionSource);
@@ -492,7 +496,8 @@ namespace AutopilotMonitor.Functions.Services
: c.FailureEvent.Message;
statusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Failed, c.FailureEvent.Phase, failureReason,
request.TenantId, request.SessionId, SessionStatus.Failed, VerdictPaths.AgentFailed,
c.FailureEvent.Phase, failureReason,
completedAt: c.FailureEvent.Timestamp,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
_logger.LogWarning("{SessionPrefix} Status: Failed - {FailureReason} (transitioned={Transitioned})", sessionPrefix, failureReason, statusTransitioned);
@@ -502,7 +507,8 @@ namespace AutopilotMonitor.Functions.Services
{
failureReason = c.EspFailureEvent.Message ?? "ESP failure (backend fallback)";
statusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Failed, c.EspFailureEvent.Phase, failureReason,
request.TenantId, request.SessionId, SessionStatus.Failed, VerdictPaths.AgentEspFailureFallback,
c.EspFailureEvent.Phase, failureReason,
completedAt: c.EspFailureEvent.Timestamp,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
_logger.LogWarning("{SessionPrefix} Status: Failed via esp_failure fallback - {FailureReason} (transitioned={Transitioned})",
@@ -511,7 +517,8 @@ namespace AutopilotMonitor.Functions.Services
else if (c.GatherCompletionEvent != null)
{
await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Succeeded, c.GatherCompletionEvent.Phase,
request.TenantId, request.SessionId, SessionStatus.Succeeded, VerdictPaths.AgentGatherComplete,
c.GatherCompletionEvent.Phase,
completedAt: c.GatherCompletionEvent.Timestamp,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
_logger.LogInformation("{SessionPrefix} Status: Succeeded (gather_rules)", sessionPrefix);
@@ -519,7 +526,8 @@ namespace AutopilotMonitor.Functions.Services
else if (c.WhiteGloveEvent != null)
{
whiteGloveStatusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Pending, EnrollmentPhase.AppsDevice,
request.TenantId, request.SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
EnrollmentPhase.AppsDevice,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
isPreProvisioned: true, isUserDriven: false);
@@ -540,7 +548,7 @@ namespace AutopilotMonitor.Functions.Services
_logger.LogWarning("{SessionPrefix} WhiteGlove UpdateSessionStatusAsync failed, attempting unconditional fallback for IsPreProvisioned + Status", sessionPrefix);
try
{
await _sessionRepo.SetSessionPreProvisionedAsync(request.TenantId, request.SessionId, true, SessionStatus.Pending, isUserDriven: false);
await _sessionRepo.SetSessionPreProvisionedAsync(request.TenantId, request.SessionId, true, SessionStatus.Pending, isUserDriven: false, verdictPath: VerdictPaths.AgentWhiteGlovePending);
whiteGloveStatusTransitioned = true;
_logger.LogInformation("{SessionPrefix} WhiteGlove fallback succeeded: IsPreProvisioned + Status=Pending set via unconditional merge", sessionPrefix);
}
@@ -580,7 +588,8 @@ namespace AutopilotMonitor.Functions.Services
if (currentSession?.Status == SessionStatus.Pending)
{
await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.InProgress, c.WhiteGloveResumedEvent.Phase,
request.TenantId, request.SessionId, SessionStatus.InProgress, VerdictPaths.AgentWhiteGloveResumed,
c.WhiteGloveResumedEvent.Phase,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
isUserDriven: true, resumedAt: c.WhiteGloveResumedEvent.Timestamp);
_logger.LogInformation("{SessionPrefix} Status: InProgress (WhiteGlove Part 2 resumed, IsUserDriven=true)", sessionPrefix);
@@ -626,7 +635,7 @@ namespace AutopilotMonitor.Functions.Services
var stalledReason = "Agent reported stall after 60min without progress (stall_probe)";
var stalledTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Stalled,
request.TenantId, request.SessionId, SessionStatus.Stalled, VerdictPaths.AgentStallProbe,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
stalledAt: c.SessionStalledEvent.Timestamp, failureReason: stalledReason);
if (stalledTransitioned)
@@ -649,7 +658,7 @@ namespace AutopilotMonitor.Functions.Services
if (currentStatus == SessionStatus.Stalled)
{
var healed = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.InProgress,
request.TenantId, request.SessionId, SessionStatus.InProgress, VerdictPaths.AgentStallHeal,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
clearStalledAt: true, clearFailureReason: true);
if (healed)
@@ -114,7 +114,7 @@ namespace AutopilotMonitor.Functions.Services
var events = await _sessionRepo.GetSessionEventsAsync(tenantId, session.SessionId, maxResults: 1000);
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var effectiveStart = session.ResumedAt ?? session.StartedAt;
var (target, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
var (target, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
@@ -145,7 +145,7 @@ namespace AutopilotMonitor.Functions.Services
}
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
tenantId, session.SessionId, target,
tenantId, session.SessionId, target, VerdictPaths.Compose(VerdictPaths.OriginRetro, rule),
failureReason: reason,
failureSource: target == SessionStatus.Incomplete ? "legacy_timeout_retro_reconcile" : null,
allowTerminalReclassification: true);
@@ -215,7 +215,7 @@ namespace AutopilotMonitor.Functions.Services
var events = await _sessionRepo.GetSessionEventsAsync(tenantId, session.SessionId, maxResults: 1000);
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var effectiveStart = session.ResumedAt ?? session.StartedAt;
var (target, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
var (target, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
@@ -240,7 +240,7 @@ namespace AutopilotMonitor.Functions.Services
// reconcile gate (no terminal-reclassification override needed); the storage
// layer additionally refuses admin-marked rows.
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
tenantId, session.SessionId, SessionStatus.Succeeded,
tenantId, session.SessionId, SessionStatus.Succeeded, VerdictPaths.RetroSelfDeployingReconcile,
failureReason: reason);
if (transitioned)
{
@@ -314,7 +314,7 @@ namespace AutopilotMonitor.Functions.Services
// Pending is non-terminal — the standard transition guard covers this write.
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
tenantId, pending.SessionId, SessionStatus.Incomplete,
tenantId, pending.SessionId, SessionStatus.Incomplete, VerdictPaths.RetroSuperseded,
failureReason: reason, failureSource: "superseded_by_reregistration");
if (transitioned) { result.Changed++; result.ToIncomplete++; }
else result.Skipped++;
@@ -289,14 +289,13 @@ namespace AutopilotMonitor.Functions.Services
/// Sessions with a deletion cascade in flight are skipped so the sweep cannot write a
/// breakdown row after the session's deletion manifest was snapshotted.
/// </summary>
private async Task SweepTimeAttributionAsync()
private async Task SweepTimeAttributionAsync(IReadOnlyList<SessionSummary> sweepWindow)
{
try
{
var sw = Stopwatch.StartNew();
var today = DateTime.UtcNow.Date;
var sessions = await _maintenanceRepo.GetSessionsByDateRangeAsync(
today.AddDays(-TimeAttributionSweepDays), today.AddDays(1));
var sessions = SliceSweepWindow(sweepWindow, today.AddDays(-TimeAttributionSweepDays));
var pairs = new List<(SessionSummary Session, SessionTimeBreakdown Breakdown)>();
var missing = new List<SessionSummary>();
@@ -1102,6 +1101,19 @@ namespace AutopilotMonitor.Functions.Services
{
_logger.LogWarning(ex, "Failed to cleanup old device-journey aggregates");
}
// Verdict-calibration daily aggregates: same 180d window (regenerable; the radar
// needs 35d, the matrix serves up to 180d).
try
{
var deleted = await _metricsRepo.DeleteVerdictCalibrationAggregatesOlderThanAsync(now.AddDays(-usageMetricsRetentionDays));
if (deleted > 0)
_logger.LogInformation("Verdict-calibration aggregate cleanup: deleted {Count} rows older than {Days} days", deleted, usageMetricsRetentionDays);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to cleanup old verdict-calibration aggregates");
}
}
/// <summary>
/// Detects and cleans up orphaned events — events stored in the Events table
@@ -36,7 +36,7 @@ namespace AutopilotMonitor.Functions.Services
/// per window device per tick — same order as the F1 sweep; negligible at current fleet
/// size (the scaling lever is gating the per-device RMW on new terminal sessions).
/// </summary>
private async Task SweepDeviceJourneysAsync()
private async Task SweepDeviceJourneysAsync(IReadOnlyList<SessionSummary> sweepWindow)
{
try
{
@@ -84,7 +84,7 @@ namespace AutopilotMonitor.Functions.Services
}
// ---- Phase 2: merge the window's terminal sessions into their device chains. ----
var sessions = await _maintenanceRepo.GetSessionsByDateRangeAsync(windowStart, windowEnd);
var sessions = SliceSweepWindow(sweepWindow, windowStart);
var excludedSessions = new List<SessionSummary>();
var byDevice = new Dictionary<(string TenantId, string SerialKey), List<SessionSummary>>();
@@ -0,0 +1,259 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Globalization;
using System.Linq;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// Partial: verdict-calibration daily aggregates (docs/backend/verdict-calibration.md) — the
/// operator thermometer for the rule classifier: which verdict path produced how many
/// sessions per day, how many of those were later overridden (admin / late completion /
/// other), and how many devices re-enrolled within 7 days.
/// </summary>
public partial class MaintenanceService
{
/// <summary>App setting kill switch: "true" skips the sweep. Fail-open — the sweep only writes its own regenerable table.</summary>
internal const string VerdictCalibrationKillSwitchSetting = "VerdictCalibrationSweepDisabled";
/// <summary>Rolling window: sessions started in the last 30 days (same as the F1/F2 sweeps).</summary>
private const int VerdictCalibrationSweepDays = 30;
/// <summary>Re-enrollment proxy horizon: another terminal session of the same device within this many days.</summary>
internal const int VerdictCalibrationReEnrollDays = 7;
/// <summary>
/// Recomputes the verdict-calibration rows for the rolling window — idempotent per
/// (tenant × StartedAt date) plus "global" mirror rows. Slices the tick's shared window
/// scan (no own drain) and reads each window tenant's device histories (one partition
/// scan per tenant) for the re-enrollment proxy. Stale date rows a run did not regenerate
/// are deleted so window sums never serve ghost counts.
/// </summary>
private async Task SweepVerdictCalibrationAsync(IReadOnlyList<SessionSummary> sweepWindow)
{
try
{
if (string.Equals(_configuration[VerdictCalibrationKillSwitchSetting], "true", StringComparison.OrdinalIgnoreCase))
{
_logger.LogInformation("Verdict-calibration sweep skipped ({Setting}=true)", VerdictCalibrationKillSwitchSetting);
return;
}
var sw = Stopwatch.StartNew();
var now = DateTime.UtcNow;
var today = now.Date;
var windowStart = today.AddDays(-VerdictCalibrationSweepDays);
var windowEnd = today.AddDays(1);
var sessions = SliceSweepWindow(sweepWindow, windowStart);
var historiesByTenant = new Dictionary<string, List<DeviceHistory>>(StringComparer.OrdinalIgnoreCase);
foreach (var tenantId in sessions.Select(s => s.TenantId).Distinct(StringComparer.OrdinalIgnoreCase))
{
historiesByTenant[tenantId] = await _metricsRepo.GetDeviceHistoriesByTenantAsync(tenantId);
}
var aggregates = BuildVerdictCalibrationAggregates(sessions, historiesByTenant, now);
var saved = 0;
foreach (var aggregate in aggregates)
{
if (await _metricsRepo.SaveVerdictCalibrationAggregateAsync(aggregate))
saved++;
}
var generatedKeys = new HashSet<(string TenantId, string Date)>(aggregates.Select(a => (a.TenantId, a.Date)));
var removedStale = 0;
foreach (var partition in aggregates.Select(a => a.TenantId).Distinct().ToList())
{
var existingRows = await _metricsRepo.GetVerdictCalibrationAggregatesAsync(partition, windowStart, windowEnd.AddDays(-1));
foreach (var row in existingRows)
{
if (generatedKeys.Contains((partition, row.Date))) continue;
await _metricsRepo.DeleteVerdictCalibrationAggregateAsync(partition, row.Date);
removedStale++;
}
}
sw.Stop();
_logger.LogInformation(
"Verdict-calibration sweep: {Sessions} window sessions across {Tenants} tenants, {Rows} rows written, {Removed} stale rows removed in {Ms}ms",
sessions.Count, historiesByTenant.Count, saved, removedStale, sw.ElapsedMilliseconds);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Verdict-calibration sweep failed (non-fatal)");
}
}
/// <summary>
/// Pure aggregation core behind <see cref="SweepVerdictCalibrationAsync"/> — internal
/// static so the contract is pinned by unit tests. Per session: the current
/// (VerdictPath, Status) bucket gets <c>Count</c> (+ <c>DerivedCount</c> when the path was
/// derived read-side), terminal sessions old enough get <c>Eligible7d</c> and, if the
/// device's history chain shows another terminal session starting within 7 days of this
/// one's end, <c>ReEnrolled7d</c>; a session carrying a prior verdict additionally
/// increments the PRIOR bucket's <c>Overridden*</c> counter. Sessions inside a deletion
/// cascade are skipped. Rows are emitted per (tenant, StartedAt date) and mirrored into
/// the "global" partition; buckets are sorted for deterministic output.
/// </summary>
internal static List<VerdictCalibrationDailyAggregate> BuildVerdictCalibrationAggregates(
IReadOnlyList<SessionSummary> sessions,
IReadOnlyDictionary<string, List<DeviceHistory>> historiesByTenant,
DateTime nowUtc)
{
var nextStartBySession = BuildNextStartLookup(historiesByTenant);
var eligibleCutoff = nowUtc.AddDays(-VerdictCalibrationReEnrollDays);
var rows = new Dictionary<(string TenantId, string Date), RowAccumulator>();
RowAccumulator Row(string tenantId, string date)
{
var key = (tenantId, date);
if (!rows.TryGetValue(key, out var acc))
{
acc = new RowAccumulator();
rows[key] = acc;
}
return acc;
}
foreach (var session in sessions)
{
if (session == null) continue;
if (!string.IsNullOrEmpty(session.DeletionState) && session.DeletionState != "None")
continue;
var date = session.StartedAt.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture);
var (path, derived) = VerdictPathDerivation.Derive(session);
var statusName = session.Status.ToString();
var isTerminal = DeviceJourneyCalculator.IsTerminal(session.Status);
var eligible = false;
var reEnrolled = false;
if (isTerminal)
{
var end = session.CompletedAt ?? session.LastEventAt ?? session.StartedAt;
eligible = end <= eligibleCutoff;
if (eligible
&& nextStartBySession.TryGetValue((session.TenantId, session.SessionId), out var next)
&& next.NextStart.HasValue
&& (next.NextStart.Value - next.End) < TimeSpan.FromDays(VerdictCalibrationReEnrollDays))
{
reEnrolled = true;
}
}
var overridden = ClassifyOverride(session);
foreach (var partition in new[] { session.TenantId, "global" })
{
var acc = Row(partition, date);
acc.SessionCount++;
if (isTerminal) acc.TerminalSessionCount++;
var bucket = acc.Bucket(path, statusName);
bucket.Count++;
if (derived) bucket.DerivedCount++;
if (eligible) bucket.Eligible7d++;
if (reEnrolled) bucket.ReEnrolled7d++;
if (overridden.HasValue)
{
var prior = acc.Bucket(session.PriorVerdictPath!, session.PriorStatus!);
switch (overridden.Value)
{
case OverrideKind.Admin: prior.OverriddenByAdmin++; break;
case OverrideKind.LateCompletion: prior.OverriddenByLateCompletion++; break;
default: prior.OverriddenOther++; break;
}
}
}
}
return rows
.OrderBy(r => r.Key.TenantId, StringComparer.Ordinal)
.ThenBy(r => r.Key.Date, StringComparer.Ordinal)
.Select(r => new VerdictCalibrationDailyAggregate
{
TenantId = r.Key.TenantId,
Date = r.Key.Date,
Version = VerdictCalibrationDailyAggregate.CurrentVersion,
SessionCount = r.Value.SessionCount,
TerminalSessionCount = r.Value.TerminalSessionCount,
Buckets = r.Value.Buckets.Values
.OrderBy(b => b.VerdictPath, StringComparer.Ordinal)
.ThenBy(b => b.Status, StringComparer.Ordinal)
.ToList(),
ComputedAt = nowUtc,
})
.ToList();
}
private enum OverrideKind { Admin, LateCompletion, Other }
/// <summary>
/// A session whose current status overrode a prior verdict names the writer that did it:
/// an admin mark (AdminMarkedAction set), a late agent completion (now Succeeded without
/// admin involvement), or anything else (retro-reclassification, grace expiry, supersede).
/// </summary>
private static OverrideKind? ClassifyOverride(SessionSummary session)
{
if (string.IsNullOrEmpty(session.PriorVerdictPath) || string.IsNullOrEmpty(session.PriorStatus))
return null;
if (!string.IsNullOrEmpty(session.AdminMarkedAction))
return OverrideKind.Admin;
if (session.Status == SessionStatus.Succeeded)
return OverrideKind.LateCompletion;
return OverrideKind.Other;
}
/// <summary>
/// (tenant, sessionId) → this session's chain end and the next chain entry's StartedAt —
/// the re-enrollment proxy source. Chains are ascending by StartedAt and capped at 20, so
/// a window session older than the cap simply has no entry (counts as not re-enrolled
/// only when eligible — the cap is a known, small bias toward under-counting).
/// </summary>
internal static Dictionary<(string TenantId, string SessionId), (DateTime End, DateTime? NextStart)> BuildNextStartLookup(
IReadOnlyDictionary<string, List<DeviceHistory>> historiesByTenant)
{
var lookup = new Dictionary<(string, string), (DateTime, DateTime?)>();
foreach (var pair in historiesByTenant)
{
foreach (var history in pair.Value)
{
var chain = history.Chain;
for (var i = 0; i < chain.Count; i++)
{
var end = chain[i].CompletedAt ?? chain[i].StartedAt;
DateTime? nextStart = i + 1 < chain.Count ? chain[i + 1].StartedAt : null;
lookup[(pair.Key, chain[i].SessionId)] = (end, nextStart);
}
}
}
return lookup;
}
private sealed class RowAccumulator
{
public int SessionCount;
public int TerminalSessionCount;
public readonly Dictionary<(string Path, string Status), VerdictCalibrationBucket> Buckets = new();
public VerdictCalibrationBucket Bucket(string path, string status)
{
var key = (path, status);
if (!Buckets.TryGetValue(key, out var bucket))
{
bucket = new VerdictCalibrationBucket { VerdictPath = path, Status = status };
Buckets[key] = bucket;
}
return bucket;
}
}
}
}
@@ -0,0 +1,206 @@
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Globalization;
using System.Linq;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// Partial: verdict-calibration drift radar (docs/backend/verdict-calibration.md). Same
/// episode/tracker pattern as the rule-regression radar, over the verdict-calibration daily
/// rows (<see cref="VerdictCalibrationRadar"/>). Operator-only: fires a VerdictCalibrationDrift
/// ops event once per episode and surfaces the episode in the calibration endpoint's
/// <c>alerts[]</c>; there is deliberately no tenant bell.
/// </summary>
public partial class MaintenanceService
{
/// <summary>App setting kill switch: "true" skips the radar. Fail-open — it only notifies.</summary>
internal const string VerdictCalibrationRadarKillSwitchSetting = "VerdictCalibrationRadarDisabled";
/// <summary>
/// Evaluates every partition that has calibration rows in the horizon — each tenant plus
/// the "global" mirror (a platform-wide drift is its own signal; per-tenant alerts catch
/// the local ones). Anchored on <paramref name="targetDate"/> (timer path: yesterday —
/// whole days only). Idempotent per anchor via the tracker dedup; fail-soft per partition.
/// </summary>
private async Task RunVerdictCalibrationRadarAsync(DateTime targetDate, IReadOnlyList<SessionSummary> sweepWindow)
{
try
{
if (string.Equals(_configuration[VerdictCalibrationRadarKillSwitchSetting], "true", StringComparison.OrdinalIgnoreCase))
{
_logger.LogInformation("Verdict-calibration radar skipped ({Setting}=true)", VerdictCalibrationRadarKillSwitchSetting);
return;
}
var sw = Stopwatch.StartNew();
var horizonStart = targetDate.Date.AddDays(-(VerdictCalibrationRadar.WindowDays - 1 + VerdictCalibrationRadar.BaselineDays));
// Tenants are discovered from the tick's shared window scan (the calibration
// sweep wrote a row per tenant it saw) — no own drain.
var tenantIds = sweepWindow
.Select(s => s.TenantId)
.Where(t => !string.IsNullOrEmpty(t))
.Distinct(StringComparer.OrdinalIgnoreCase)
.ToList();
tenantIds.Add("global");
int fired = 0, refreshed = 0, rearmed = 0;
foreach (var partition in tenantIds)
{
try
{
var rows = await _metricsRepo.GetVerdictCalibrationAggregatesAsync(partition, horizonStart, targetDate.Date);
if (rows.Count == 0) continue;
var (f, r, a) = await EvaluateVerdictCalibrationPartitionAsync(partition, rows, targetDate);
fired += f; refreshed += r; rearmed += a;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Verdict-calibration radar failed for partition {Partition} (non-fatal)", partition);
}
}
sw.Stop();
_logger.LogInformation(
"Verdict-calibration radar: {Partitions} partitions, {Fired} fired, {Refreshed} refreshed, {Rearmed} re-armed in {Ms}ms (anchor {Anchor})",
tenantIds.Count, fired, refreshed, rearmed, sw.ElapsedMilliseconds, targetDate.ToString("yyyy-MM-dd"));
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Verdict-calibration radar failed (non-fatal)");
}
}
private async Task<(int Fired, int Refreshed, int Rearmed)> EvaluateVerdictCalibrationPartitionAsync(
string partition, List<VerdictCalibrationDailyAggregate> rows, DateTime targetDate)
{
var findings = VerdictCalibrationRadar.Evaluate(rows, targetDate);
var active = await _hardwareRejectionTracker.GetVerdictCalibrationAlertsAsync(partition);
var activeByKey = active.ToDictionary(a => AlertKey(a.Kind, a.VerdictPath, a.Status), StringComparer.OrdinalIgnoreCase);
int fired = 0, refreshed = 0, rearmed = 0;
var firedKeys = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
foreach (var finding in findings)
{
var key = AlertKey(finding.Kind, finding.VerdictPath, finding.Status);
firedKeys.Add(key);
if (activeByKey.TryGetValue(key, out var existing))
{
await _hardwareRejectionTracker.RefreshVerdictCalibrationAlertAsync(
partition, BuildVerdictAlert(partition, finding, existing.Dimension, existing.FirstNotifiedAt));
refreshed++;
continue;
}
var dimension = await TryCorrelateVerdictDimensionsAsync(partition, finding);
var alert = BuildVerdictAlert(partition, finding, dimension, DateTime.UtcNow);
if (!await _hardwareRejectionTracker.TryRegisterVerdictCalibrationAlertAsync(partition, alert))
continue;
fired++;
await _opsEventService.RecordVerdictCalibrationDriftAsync(
partition, finding.Kind, finding.VerdictPath, finding.Status,
finding.WindowHitCount, finding.WindowSessionCount, finding.WindowRatePct,
finding.BaselineHitCount, finding.BaselineSessionCount, finding.BaselineRatePct,
finding.Lift, DescribeDimension(dimension));
}
foreach (var alert in active)
{
if (firedKeys.Contains(AlertKey(alert.Kind, alert.VerdictPath, alert.Status)))
continue;
if (VerdictCalibrationRadar.ShouldReArm(alert, rows, targetDate))
{
await _hardwareRejectionTracker.DeleteVerdictCalibrationAlertAsync(partition, alert.Kind, alert.VerdictPath, alert.Status);
rearmed++;
}
else
{
// Elevated but no longer fully separated: keep the episode, refresh the numbers.
var (wh, ws, bh, bs) = VerdictCalibrationRadar.CurrentSums(alert, rows, targetDate);
alert.WindowHitCount = wh;
alert.WindowSessionCount = ws;
alert.BaselineHitCount = bh;
alert.BaselineSessionCount = bs;
alert.WindowRatePct = ws > 0 ? Math.Round(100.0 * wh / ws, 1) : 0;
alert.BaselineRatePct = bs > 0 ? Math.Round(100.0 * bh / bs, 1) : 0;
alert.Lift = alert.Kind == VerdictCalibrationAlertKinds.EvidenceGap || bh == 0 || bs == 0 || ws == 0
? null
: Math.Round(((double)wh / ws) / ((double)bh / bs), 1);
alert.WindowStartDate = targetDate.Date.AddDays(-(VerdictCalibrationRadar.WindowDays - 1)).ToString("yyyy-MM-dd");
alert.WindowEndDate = targetDate.Date.ToString("yyyy-MM-dd");
alert.LastEvaluatedAt = DateTime.UtcNow;
await _hardwareRejectionTracker.RefreshVerdictCalibrationAlertAsync(partition, alert);
refreshed++;
}
}
return (fired, refreshed, rearmed);
}
private static string AlertKey(string kind, string path, string status) => $"{kind}|{path}|{status}";
internal static VerdictCalibrationAlert BuildVerdictAlert(
string partition, VerdictCalibrationFinding finding, RuleRegressionDimension? dimension, DateTime firstNotifiedAt) => new()
{
TenantId = partition,
Kind = finding.Kind,
VerdictPath = finding.VerdictPath,
Status = finding.Status,
WindowHitCount = finding.WindowHitCount,
WindowSessionCount = finding.WindowSessionCount,
BaselineHitCount = finding.BaselineHitCount,
BaselineSessionCount = finding.BaselineSessionCount,
WindowRatePct = finding.WindowRatePct,
BaselineRatePct = finding.BaselineRatePct,
Lift = finding.Lift,
WindowStartDate = finding.WindowStartDate,
WindowEndDate = finding.WindowEndDate,
Dimension = dimension,
FirstNotifiedAt = firstNotifiedAt,
LastEvaluatedAt = DateTime.UtcNow,
};
/// <summary>
/// On-fire dimension correlation for a per-path finding: the window's sessions whose
/// (derived) verdict path matches vs all window sessions. Group kinds and the "global"
/// partition (a cross-tenant scan would dwarf the signal) carry no dimension claim.
/// Fail-soft: any error yields null — never a guessed dimension.
/// </summary>
private async Task<RuleRegressionDimension?> TryCorrelateVerdictDimensionsAsync(string partition, VerdictCalibrationFinding finding)
{
if (partition == "global" || finding.Kind != VerdictCalibrationAlertKinds.ShareRegression)
return null;
try
{
var windowStart = DateTime.ParseExact(finding.WindowStartDate, "yyyy-MM-dd", CultureInfo.InvariantCulture,
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal);
var windowEndExclusive = DateTime.ParseExact(finding.WindowEndDate, "yyyy-MM-dd", CultureInfo.InvariantCulture,
DateTimeStyles.AssumeUniversal | DateTimeStyles.AdjustToUniversal).AddDays(1);
var allSessions = await _maintenanceRepo.GetSessionsByDateRangeAsync(windowStart, windowEndExclusive, partition);
if (allSessions.Count == 0) return null;
var hitSessions = allSessions
.Where(s => s.Status.ToString() == finding.Status
&& VerdictPathDerivation.Derive(s).Path == finding.VerdictPath)
.ToList();
return RuleRegressionRadar.ComputeDimensionConcentration(hitSessions, allSessions);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Verdict-calibration dimension correlation failed for {Path} in {Partition}", finding.VerdictPath, partition);
return null;
}
}
}
}
@@ -69,6 +69,7 @@ namespace AutopilotMonitor.Functions.Services
private readonly IConfiguration _configuration;
private readonly IHttpClientFactory _httpClientFactory;
private readonly PreviewWhitelistService _previewWhitelistService;
private readonly IStorageInitializer _storageInitializer;
private readonly ILogger<MaintenanceService> _logger;
private const string PlatformStatsAliasFileName = "platform-stats.json";
@@ -101,6 +102,7 @@ namespace AutopilotMonitor.Functions.Services
IConfiguration configuration,
IHttpClientFactory httpClientFactory,
PreviewWhitelistService previewWhitelistService,
IStorageInitializer storageInitializer,
ILogger<MaintenanceService> logger)
{
_maintenanceRepo = maintenanceRepo;
@@ -128,9 +130,22 @@ namespace AutopilotMonitor.Functions.Services
_configuration = configuration;
_httpClientFactory = httpClientFactory;
_previewWhitelistService = previewWhitelistService;
_storageInitializer = storageInitializer;
_logger = logger;
}
private async Task EnsureAllTablesAsync()
{
try
{
await _storageInitializer.EnsureAllAsync();
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Daily table existence pass failed");
}
}
/// <summary>
/// Runs all maintenance tasks (used by the daily timer trigger)
/// </summary>
@@ -141,6 +156,10 @@ namespace AutopilotMonitor.Functions.Services
try
{
// Startup only point-reads the table schema sentinel (see
// TableStorageService.InitializeTablesAsync); this daily full pass is what
// recreates a table that was deleted out-of-band. Cheap (idempotent creates).
await EnsureAllTablesAsync();
await MarkStalledSessionsAsTimedOutAsync();
await DetectExcessiveEventSessionsAsync();
await AggregateMetricsWithCatchUpAsync();
@@ -151,13 +170,24 @@ namespace AutopilotMonitor.Functions.Services
// App-version duration regression radar: same episode/tracker pattern over the
// install summaries (trailing 35d horizon loaded internally). Fail-soft.
await RunAppVersionRegressionRadarAsync();
// ONE projected cross-tenant window scan feeds every rolling sweep below (the
// StartedAt-only filter is a full-table drain in Table Storage — four of them per
// tick was four times the same read). Each sweep slices its own window.
var sweepWindow = await LoadSweepWindowSessionsAsync();
// F1 PR2: rolling 30d breakdown backfill + daily attribution aggregates. Owns
// its own window (NOT the snapshot-gated catch-up above) so late-terminating
// sessions still reach their StartedAt-date's aggregate. Fail-soft internally.
await SweepTimeAttributionAsync();
await SweepTimeAttributionAsync(sweepWindow);
// F2 PR4: device-history chain heal (incl. deleted-session ref cleanup) + daily
// FTR aggregates over the same rolling window. Fail-soft internally.
await SweepDeviceJourneysAsync();
await SweepDeviceJourneysAsync(sweepWindow);
// Verdict calibration: per-verdict-path daily buckets over the same rolling
// window, AFTER the device-journey sweep so the re-enrollment proxy reads
// freshly merged chains. Fail-soft internally.
await SweepVerdictCalibrationAsync(sweepWindow);
// Verdict-calibration drift radar over the rows the sweep just refreshed; anchored
// on yesterday like the rule radar (whole days only). Fail-soft.
await RunVerdictCalibrationRadarAsync(DateTime.UtcNow.Date.AddDays(-1), sweepWindow);
// Plan §5 PR6 / §16 R14: session retention fanout extracted out of the 2h timer
// into the dedicated 12h SessionDeletionMaintenanceFunction so cascade-lifecycle
// work has independent cadence + kill-switch + OpsEvent watchdogs. The non-session
@@ -199,6 +229,33 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// Shared sweep window: the longest horizon any rolling sweep needs — the calibration
/// radar's 7d window + 28d baseline anchored on YESTERDAY (= 35 days back from today) —
/// drained once per tick with the maintenance projection. Sweeps with a shorter window
/// slice it by StartedAt (<see cref="SliceSweepWindow"/>). Fail-soft: an empty list makes
/// every sweep a no-op rather than failing the tick.
/// </summary>
internal const int SweepWindowDays = Helpers.VerdictCalibrationRadar.WindowDays + Helpers.VerdictCalibrationRadar.BaselineDays;
private async Task<List<SessionSummary>> LoadSweepWindowSessionsAsync()
{
try
{
var today = DateTime.UtcNow.Date;
return await _maintenanceRepo.GetMaintenanceWindowSessionsAsync(today.AddDays(-SweepWindowDays), today.AddDays(1));
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Sweep window load failed — rolling sweeps skip this tick (non-fatal)");
return new List<SessionSummary>();
}
}
/// <summary>Sessions of the shared window that started at or after <paramref name="windowStart"/> (UTC).</summary>
internal static List<SessionSummary> SliceSweepWindow(IReadOnlyList<SessionSummary> window, DateTime windowStart)
=> window.Where(s => s.StartedAt >= windowStart).ToList();
/// <summary>
/// Manually triggered maintenance with flexible date selection
/// </summary>
@@ -230,8 +287,11 @@ namespace AutopilotMonitor.Functions.Services
// Timer-path parity: manual maintenance also refreshes the attribution
// breakdowns + daily aggregates and the device-history/FTR rollups
// (rolling 30d windows, cheap once converged).
await SweepTimeAttributionAsync();
await SweepDeviceJourneysAsync();
var sweepWindow = await LoadSweepWindowSessionsAsync();
await SweepTimeAttributionAsync(sweepWindow);
await SweepDeviceJourneysAsync(sweepWindow);
await SweepVerdictCalibrationAsync(sweepWindow);
await RunVerdictCalibrationRadarAsync(dateToAggregate, sweepWindow);
if (!aggregateOnly)
{
@@ -392,6 +452,7 @@ namespace AutopilotMonitor.Functions.Services
silent.TenantId,
silent.SessionId,
SessionStatus.Stalled,
VerdictPaths.SweepStalled,
stalledAt: now,
failureReason: $"Agent silent for {silentMinutes}min (detected by maintenance sweep)");
silentMarked++;
@@ -482,7 +543,7 @@ namespace AutopilotMonitor.Functions.Services
// Device Setup done + within grace → AwaitingUser, else Incomplete; silent before
// Device Setup → Incomplete. Never Failed without an explicit failure signal.
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
var (targetStatus, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
@@ -495,6 +556,7 @@ namespace AutopilotMonitor.Functions.Services
session.TenantId,
session.SessionId,
targetStatus,
VerdictPaths.Compose(VerdictPaths.OriginSweep, rule),
failureReason: reason,
failureSnapshotJson: snapshotJson
);
@@ -624,6 +686,7 @@ namespace AutopilotMonitor.Functions.Services
silent.TenantId,
silent.SessionId,
SessionStatus.AwaitingUser,
VerdictPaths.SweepWhiteGloveAwaiting,
failureReason: EnrollmentTimeoutClassifier.WhiteGloveAwaitingUserReason(rollup));
if (transitioned)
_logger.LogInformation(
@@ -660,6 +723,7 @@ namespace AutopilotMonitor.Functions.Services
silent.TenantId,
silent.SessionId,
SessionStatus.Succeeded,
VerdictPaths.SweepSelfDeployingReconcile,
failureReason: EnrollmentTimeoutClassifier.SelfDeployingReconcileReason(
silent.ResumedAt ?? silent.StartedAt, now, lastContactAt));
if (transitioned)
@@ -83,6 +83,8 @@ namespace AutopilotMonitor.Functions.Services.Offboarding
// whole-row deletion besides chain-emptying.
Constants.TableNames.DeviceHistories,
Constants.TableNames.DeviceJourneyAggregates,
// Verdict calibration daily buckets (PK=TenantId) — regenerable from Sessions, wiped with the tenant.
Constants.TableNames.VerdictCalibrationAggregates,
// Pre-write config snapshots (PK=tenantId). Wiped in the bulk phase; the
// TenantConfiguration row itself is deleted LAST via SafeWipe (a delete, not a
// save), so the wipe cannot re-create a snapshot afterwards. The admin-config
@@ -641,6 +641,40 @@ namespace AutopilotMonitor.Functions.Services
lift,
});
/// <summary>
/// Verdict-calibration drift radar (docs/backend/verdict-calibration.md): a verdict
/// path's share of sessions doubled over its 28d baseline, the silence share
/// (sweep+maxlife — agent went quiet, backend had to decide) doubled, or the pure
/// fallthrough rule r6 decides ≥20 % of classifier verdicts. Operator-only diagnostic —
/// there is deliberately NO tenant bell. Fired ONCE per (kind, path, status) episode
/// (tracker-deduped). Dual-registered in the web OPS_EVENT_TYPES catalog
/// (memory: feedback_ops_event_types_dual_register).
/// </summary>
public Task RecordVerdictCalibrationDriftAsync(
string tenantId, string kind, string verdictPath, string status,
int windowHitCount, int windowSessionCount, double windowRatePct,
int baselineHitCount, int baselineSessionCount, double baselineRatePct,
double? lift, string? dimensionSummary)
=> WriteAsync(OpsEventCategory.Maintenance, "VerdictCalibrationDrift", OpsEventSeverity.Warning,
$"Verdict calibration [{kind}] {verdictPath}/{status}: {windowHitCount}/{windowSessionCount} ({windowRatePct}%) over 7d " +
$"vs {baselineRatePct}% baseline ({baselineHitCount}/{baselineSessionCount} over 28d)" +
(lift.HasValue ? $" — lift {lift.Value}x" : string.Empty) +
(string.IsNullOrEmpty(dimensionSummary) ? string.Empty : $". {dimensionSummary}"),
tenantId, "System.Maintenance",
new
{
kind,
verdictPath,
status,
windowHitCount,
windowSessionCount,
windowRatePct,
baselineHitCount,
baselineSessionCount,
baselineRatePct,
lift,
});
// ── Agent ──────────────────────────────────────────────────────────────
public Task RecordSessionTimeoutsAsync(string tenantId, int sessionCount, int timeoutHours)
@@ -1,5 +1,4 @@
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Extensions.Caching.Memory;
@@ -17,10 +16,37 @@ namespace AutopilotMonitor.Functions.Services
private readonly ILogger<RateLimitService> _logger;
private readonly TimeSpan _windowDuration;
// Dedicated lock objects per device to avoid race conditions when cache evicts the request history list.
// Using the list itself as a lock target is unsafe because cache eviction can create a new list instance,
// causing different threads to lock on different objects.
private readonly ConcurrentDictionary<string, object> _locks = new();
// The request history and its lock travel together in one cache entry. Whoever observes the
// same list observes the same lock, so two threads can never mutate one list under different
// locks. A thread holding an already-evicted bucket only mutates a discarded list, which is
// harmless at the expiry boundary. A separate per-key lock dictionary is deliberately NOT
// used: its eviction-callback cleanup raced with lock re-creation and leaked on replaced entries.
private sealed class RequestBucket
{
public readonly List<DateTime> History = new();
public readonly object Sync = new();
}
// IMemoryCache.GetOrCreate is not atomic: two first-time callers can each create a bucket and
// the loser would count its request into a discarded list. Creation is serialized here
// (cheap: only cache misses take it); the hot path is a lock-free TryGetValue.
private readonly object _createLock = new();
private RequestBucket GetOrCreateBucket(string cacheKey)
{
if (_cache.TryGetValue(cacheKey, out RequestBucket? existing) && existing != null)
return existing;
lock (_createLock)
{
return _cache.GetOrCreate(cacheKey, entry =>
{
entry.SlidingExpiration = _windowDuration;
entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(1);
return new RequestBucket();
})!;
}
}
public RateLimitService(IMemoryCache cache, ILogger<RateLimitService> logger)
{
@@ -56,18 +82,14 @@ namespace AutopilotMonitor.Functions.Services
var cacheKey = $"ratelimit_{deviceThumbprint}";
var now = DateTime.UtcNow;
// Get or create request history for this device
var lockObj = _locks.GetOrAdd(cacheKey, _ => new object());
var requestHistory = _cache.GetOrCreate(cacheKey, entry =>
{
entry.SlidingExpiration = _windowDuration;
entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(1);
entry.RegisterPostEvictionCallback((key, _, _, _) => _locks.TryRemove((string)key, out _));
return new List<DateTime>();
})!;
// Get or create request history for this device. The bucket is mutated in place; the
// cache lookup itself refreshes the sliding expiration, so no re-Set is needed.
var bucket = GetOrCreateBucket(cacheKey);
lock (lockObj)
lock (bucket.Sync)
{
var requestHistory = bucket.History;
// Remove requests outside the sliding window
var windowStart = now.Subtract(_windowDuration);
requestHistory.RemoveAll(timestamp => timestamp < windowStart);
@@ -104,13 +126,6 @@ namespace AutopilotMonitor.Functions.Services
// Add current request to history
requestHistory.Add(now);
// Update cache
_cache.Set(cacheKey, requestHistory, new MemoryCacheEntryOptions
{
SlidingExpiration = _windowDuration,
AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(1)
});
return new RateLimitResult
{
IsAllowed = true,
@@ -311,7 +311,7 @@ namespace AutopilotMonitor.Functions.Services
// DurationSeconds. Letting UpdateSessionStatusAsync fall back to LastEventAt
// anchors duration on when the session actually went silent.
await _sessionRepo.UpdateSessionStatusAsync(
tenantId, sessionId, SessionStatus.Failed,
tenantId, sessionId, SessionStatus.Failed, VerdictPaths.Compose(VerdictPaths.OriginRule, rule.RuleId),
failureReason: failureReason,
failureSource: failureSource);
@@ -74,6 +74,9 @@ namespace AutopilotMonitor.Functions.Services
"ValidatedBy",
"FailureSnapshotJson",
"AdminMarkedAction",
"VerdictPath",
"PriorStatus",
"PriorVerdictPath",
"DurationSeconds",
"DiagnosticsBlobName",
"DiagnosticsBlobDestination",
@@ -0,0 +1,89 @@
using System.Diagnostics;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services;
/// <summary>
/// Emits one App Insights metric per process start so startup duration is a trend, not an
/// anecdote: <c>BackendStartupMs</c> = process start → host ApplicationStarted (includes
/// runtime + DI + every hosted service), plus <c>BackendTableInitMs</c> for the table
/// initialization slice (see TableStorageService.InitializeTablesAsync / schema sentinel).
/// Metrics are not subject to adaptive sampling and reach App Insights regardless of the
/// worker log level, unlike Information traces.
/// Query: <c>customMetrics | where name == "BackendStartupMs" | project timestamp, value, customDimensions</c>.
/// </summary>
public sealed class StartupTelemetryService : IHostedService
{
public const string StartupMetricName = "BackendStartupMs";
public const string TableInitMetricName = "BackendTableInitMs";
private readonly IHostApplicationLifetime _lifetime;
private readonly TelemetryClient _telemetry;
private readonly TableStorageService _tableStorage;
private readonly BackendBuildInfo _buildInfo;
private readonly ILogger<StartupTelemetryService> _logger;
public StartupTelemetryService(
IHostApplicationLifetime lifetime,
TelemetryClient telemetry,
TableStorageService tableStorage,
BackendBuildInfo buildInfo,
ILogger<StartupTelemetryService> logger)
{
_lifetime = lifetime;
_telemetry = telemetry;
_tableStorage = tableStorage;
_buildInfo = buildInfo;
_logger = logger;
}
public Task StartAsync(CancellationToken cancellationToken)
{
// ApplicationStarted fires after ALL hosted services completed StartAsync — that is
// the moment the host is ready to serve, independent of registration order.
_lifetime.ApplicationStarted.Register(Emit);
return Task.CompletedTask;
}
public Task StopAsync(CancellationToken cancellationToken) => Task.CompletedTask;
private void Emit()
{
try
{
var startupMs = (DateTime.UtcNow - Process.GetCurrentProcess().StartTime.ToUniversalTime()).TotalMilliseconds;
var init = _tableStorage.LastInitialization;
var props = new Dictionary<string, string>
{
["version"] = _buildInfo.Version,
["commit"] = _buildInfo.CommitHash,
["tableInitFullPass"] = init.FullPassRan.ToString(),
["instance"] = Environment.MachineName
};
_telemetry.TrackMetric(new MetricTelemetry(StartupMetricName, Math.Round(startupMs)).WithProperties(props));
_telemetry.TrackMetric(new MetricTelemetry(TableInitMetricName, Math.Round(init.DurationMs)).WithProperties(props));
_logger.LogInformation("Backend startup took {StartupMs}ms (table init {TableInitMs}ms, fullPass={FullPass})",
Math.Round(startupMs), Math.Round(init.DurationMs), init.FullPassRan);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to emit startup telemetry");
}
}
}
internal static class MetricTelemetryExtensions
{
public static MetricTelemetry WithProperties(this MetricTelemetry metric, IReadOnlyDictionary<string, string> props)
{
foreach (var (key, value) in props)
metric.Properties[key] = value;
return metric;
}
}
@@ -6,7 +6,7 @@ namespace AutopilotMonitor.Functions.Services;
/// <summary>
/// Hosted service that initializes all Azure Table Storage tables at application startup.
/// This ensures all tables exist before any requests are processed.
/// Also performs a one-time backfill of the SessionsIndex table if it is empty.
/// Also performs a one-time backfill of the SessionsIndex table on fresh storage (see StartAsync).
/// </summary>
public class TableInitializerService : IHostedService
{
@@ -23,12 +23,13 @@ public class TableInitializerService : IHostedService
public async Task StartAsync(CancellationToken cancellationToken)
{
_logger.LogInformation("TableInitializerService starting - initializing all Azure Table Storage tables");
_logger.LogInformation("TableInitializerService starting");
var fullPassRan = false;
try
{
await _tableStorageService.InitializeTablesAsync();
_logger.LogInformation("TableInitializerService completed - all tables ready");
fullPassRan = await _tableStorageService.InitializeTablesAsync();
_logger.LogInformation("TableInitializerService completed - all tables ready (fullPass={FullPass})", fullPassRan);
}
catch (Exception ex)
{
@@ -37,10 +38,15 @@ public class TableInitializerService : IHostedService
// Individual operations will fail gracefully with appropriate error messages
}
// One-time backfill: if SessionsIndex is empty, copy all sessions from Sessions table
// One-time backfill: SessionsIndex can only be empty on fresh storage, which is exactly
// when the schema sentinel was missing and the full pass ran. Skipped on the fast path,
// and guarded by a conditional-insert claim so only one scaled-out instance scans.
if (!fullPassRan) return;
try
{
if (await _tableStorageService.IsSessionIndexEmptyAsync())
if (await _tableStorageService.IsSessionIndexEmptyAsync()
&& await _tableStorageService.TryClaimSessionIndexBackfillAsync())
{
_logger.LogInformation("SessionsIndex table is empty — starting startup backfill");
var count = await _tableStorageService.BackfillSessionIndexAsync();
@@ -245,61 +245,43 @@ namespace AutopilotMonitor.Functions.Services
/// <summary>
/// Upserts CVE index entries so sessions can be searched by CVE identifier.
/// PartitionKey = {tenantId}_{cveId}, RowKey = sessionId
/// Uses individual parallel upserts (not batch transactions) because PK differs per CVE.
/// One CVE commonly appears under several findings of the same report (e.g. one
/// Remote Desktop CVE across 14 installed RD components), so occurrences are merged
/// per CVE first — otherwise concurrent upserts race on the same entity and the
/// winner is arbitrary. Writes go out with bounded parallelism (PK differs per CVE,
/// so batch transactions are not an option). Per-entity failures are counted, not
/// fatal — the index is a search aid, the report itself is stored separately.
/// </summary>
public async Task UpsertCveIndexEntriesAsync(string tenantId, string sessionId, List<Dictionary<string, object>> findings)
{
try
{
var entities = BuildCveIndexEntities(tenantId, sessionId, findings);
if (entities.Count == 0) return;
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.CveIndex);
var tasks = new List<Task>();
var failed = 0;
foreach (var finding in findings)
{
var softwareName = finding.TryGetValue("softwareName", out var sn) ? sn?.ToString() ?? "" : "";
var overallRisk = finding.TryGetValue("riskLevel", out var rl) ? rl?.ToString() ?? "" : "";
if (!finding.TryGetValue("vulnerabilities", out var vulnsObj) || vulnsObj == null) continue;
if (vulnsObj is not System.Collections.IEnumerable vulnList) continue;
foreach (var vulnObj in vulnList)
await Parallel.ForEachAsync(
entities,
new ParallelOptions { MaxDegreeOfParallelism = 8 },
async (entity, ct) =>
{
Dictionary<string, object>? vuln = null;
if (vulnObj is Dictionary<string, object> vd)
vuln = vd;
if (vuln == null) continue;
var cveId = vuln.TryGetValue("cveId", out var cid) ? cid?.ToString() : null;
if (string.IsNullOrEmpty(cveId)) continue;
var partitionKey = $"{tenantId}_{cveId}";
double cvssScore = 0;
try { if (vuln.TryGetValue("cvssScore", out var cs) && cs != null) cvssScore = Convert.ToDouble(cs); } catch { }
var cvssSeverity = vuln.TryGetValue("cvssSeverity", out var csvs) ? csvs?.ToString() ?? "" : "";
bool isKev = false;
try { if (vuln.TryGetValue("isKev", out var ik) && ik is bool ikb) isKev = ikb; } catch { }
var entity = new TableEntity(partitionKey, sessionId)
try
{
["SessionId"] = sessionId,
["TenantId"] = tenantId,
["CveId"] = cveId,
["SoftwareName"] = softwareName,
["CvssScore"] = cvssScore,
["CvssSeverity"] = cvssSeverity,
["IsKev"] = isKev,
["OverallRisk"] = overallRisk,
["DetectedAt"] = DateTime.UtcNow,
};
await tableClient.UpsertEntityAsync(entity, cancellationToken: ct);
}
catch (Exception ex)
{
Interlocked.Increment(ref failed);
_logger.LogWarning(ex, "Failed to upsert CveIndex entry {CveId} for session {SessionId}",
entity.GetString("CveId"), sessionId);
}
});
tasks.Add(tableClient.UpsertEntityAsync(entity));
}
}
if (tasks.Count > 0)
await Task.WhenAll(tasks);
if (failed > 0)
_logger.LogWarning("CveIndex upsert for session {SessionId}: {Failed}/{Total} entries failed",
sessionId, failed, entities.Count);
}
catch (Exception ex)
{
@@ -307,6 +289,78 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// Collapses report findings to one entity per CVE. Merge rule: the occurrence with the
/// highest CVSS score supplies score/severity/software name (ties keep report order),
/// IsKev is OR-ed, OverallRisk is the highest risk level seen across occurrences.
/// </summary>
internal static List<TableEntity> BuildCveIndexEntities(string tenantId, string sessionId, List<Dictionary<string, object>> findings)
{
var byCve = new Dictionary<string, TableEntity>(StringComparer.OrdinalIgnoreCase);
var now = DateTime.UtcNow;
foreach (var finding in findings)
{
var softwareName = finding.TryGetValue("softwareName", out var sn) ? sn?.ToString() ?? "" : "";
var overallRisk = finding.TryGetValue("riskLevel", out var rl) ? rl?.ToString() ?? "" : "";
if (!finding.TryGetValue("vulnerabilities", out var vulnsObj) || vulnsObj == null) continue;
if (vulnsObj is not System.Collections.IEnumerable vulnList) continue;
foreach (var vulnObj in vulnList)
{
if (vulnObj is not Dictionary<string, object> vuln) continue;
var cveId = vuln.TryGetValue("cveId", out var cid) ? cid?.ToString() : null;
if (string.IsNullOrEmpty(cveId)) continue;
double cvssScore = 0;
try { if (vuln.TryGetValue("cvssScore", out var cs) && cs != null) cvssScore = Convert.ToDouble(cs); } catch { }
var cvssSeverity = vuln.TryGetValue("cvssSeverity", out var csvs) ? csvs?.ToString() ?? "" : "";
var isKev = vuln.TryGetValue("isKev", out var ik) && ik is bool ikb && ikb;
if (byCve.TryGetValue(cveId, out var existing))
{
if (cvssScore > existing.GetDouble("CvssScore"))
{
existing["CvssScore"] = cvssScore;
existing["CvssSeverity"] = cvssSeverity;
existing["SoftwareName"] = softwareName;
}
existing["IsKev"] = existing.GetBoolean("IsKev") == true || isKev;
if (RiskRank(overallRisk) > RiskRank(existing.GetString("OverallRisk")))
existing["OverallRisk"] = overallRisk;
continue;
}
byCve[cveId] = new TableEntity($"{tenantId}_{cveId}", sessionId)
{
["SessionId"] = sessionId,
["TenantId"] = tenantId,
["CveId"] = cveId,
["SoftwareName"] = softwareName,
["CvssScore"] = cvssScore,
["CvssSeverity"] = cvssSeverity,
["IsKev"] = isKev,
["OverallRisk"] = overallRisk,
["DetectedAt"] = now,
};
}
}
return byCve.Values.ToList();
}
private static int RiskRank(string? risk) => risk?.ToLowerInvariant() switch
{
"critical" => 4,
"high" => 3,
"medium" => 2,
"low" => 1,
_ => 0,
};
// ===== SEARCH METHODS =====
/// <summary>Typeahead only inspects sessions started within this many days (recent-window scan bound).</summary>
@@ -595,6 +595,36 @@ namespace AutopilotMonitor.Functions.Services
public Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null)
=> QuerySessionsByDateRangeAsync(startDate, endDate, tenantId, GeoMetricsSessionProjection);
/// <summary>
/// Columns the rolling maintenance sweeps consume from the SHARED window scan (one
/// cross-tenant drain per tick feeds the time-attribution, device-journey and
/// verdict-calibration sweeps plus the calibration radar's tenant discovery — see
/// MaintenanceService.LoadSweepWindowSessionsAsync). Union of the three sweeps' reads:
/// identity/device (SerialNumber, Manufacturer, Model), status + timestamps, the
/// enrollment-class inputs, DurationSeconds/EventCount, the deletion-cascade marker,
/// the verdict attribution (VerdictPath/Prior*) and the legacy-derivation inputs
/// (FailureReason/ReconcileReason/FailureSource/AdminMarkedAction/EspSoftFailure).
/// Drops the wide row's FailureSnapshotJson, geo, OS and diagnostics columns. internal so
/// MaintenanceSweepProjectionEquivalenceTests derives its keep-set from this exact array.
/// </summary>
internal static readonly string[] MaintenanceSweepSessionProjection =
{
"PartitionKey", "RowKey", "StartedAt", "CompletedAt", "LastEventAt", "Status",
"DurationSeconds", "EventCount", "DeletionState",
"SerialNumber", "Manufacturer", "Model",
"EnrollmentType", "IsPreProvisioned", "IsSelfDeployingProfile", "IsUserDriven", "ResumedAt",
"AdminMarkedAction", "VerdictPath", "PriorStatus", "PriorVerdictPath",
"FailureReason", "ReconcileReason", "FailureSource", "EspSoftFailure",
};
/// <summary>
/// Column-projected cross-tenant date-range query backing the shared maintenance sweep
/// window. Same filter semantics as <see cref="GetSessionsByDateRangeAsync"/>; only
/// <see cref="MaintenanceSweepSessionProjection"/> is transferred.
/// </summary>
public Task<List<SessionSummary>> GetMaintenanceWindowSessionsAsync(DateTime startDate, DateTime endDate)
=> QuerySessionsByDateRangeAsync(startDate, endDate, tenantId: null, MaintenanceSweepSessionProjection);
private async Task<List<SessionSummary>> QuerySessionsByDateRangeAsync(DateTime startDate, DateTime endDate, string? tenantId, string[]? select)
{
if (!string.IsNullOrEmpty(tenantId))
@@ -1118,8 +1148,10 @@ namespace AutopilotMonitor.Functions.Services
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to check if session index is empty");
return true; // Assume empty on error → trigger backfill
// A transient read error must NOT trigger a full Sessions scan with serial upserts.
// The manual maintenance backfill remains the safety net.
_logger.LogWarning(ex, "Failed to check if session index is empty — skipping startup backfill");
return false;
}
}
@@ -209,6 +209,16 @@ namespace AutopilotMonitor.Functions.Services
if (!string.IsNullOrEmpty(adminMarkedAction))
indexEntity["AdminMarkedAction"] = adminMarkedAction;
// Verdict calibration attribution (docs/backend/verdict-calibration.md): the aggregate
// sweep reads these off the session row, and the index mirror must not drop them on a
// StartedAt-shift full upsert.
foreach (var col in new[] { "VerdictPath", "PriorStatus", "PriorVerdictPath" })
{
var val = sessionEntity.GetString(col);
if (!string.IsNullOrEmpty(val))
indexEntity[col] = val;
}
var durationSeconds = sessionEntity.GetInt32("DurationSeconds");
if (durationSeconds.HasValue)
indexEntity["DurationSeconds"] = durationSeconds.Value;
@@ -492,6 +502,12 @@ namespace AutopilotMonitor.Functions.Services
// race the in-flight cascade. Empty/null is fine on first registration.
string? existingDeletionState = null;
string? existingPendingDeletionManifestId = null;
// Verdict calibration: a fresh row's status comes from registration itself; a
// re-registration preserves the path that produced the (non-terminal) status it
// keeps, and the prior-verdict pair, so the Replace never blanks the attribution.
string verdictPath = VerdictPaths.RegisterNew;
string? priorStatus = null;
string? priorVerdictPath = null;
try
{
@@ -544,6 +560,9 @@ namespace AutopilotMonitor.Functions.Services
existingIndexRowKey = existingEntity.GetString("IndexRowKey");
existingDeletionState = existingEntity.GetString("DeletionState");
existingPendingDeletionManifestId = existingEntity.GetString("PendingDeletionManifestId");
verdictPath = existingEntity.GetString("VerdictPath") ?? verdictPath;
priorStatus = existingEntity.GetString("PriorStatus");
priorVerdictPath = existingEntity.GetString("PriorVerdictPath");
// Guard: never regress a terminal status (Succeeded/Failed) back to InProgress.
// StoreSessionAsync uses UpsertEntity (Replace mode) which overwrites all fields.
@@ -561,6 +580,7 @@ namespace AutopilotMonitor.Functions.Services
{
_logger.LogInformation($"Session {registration.SessionId} resuming from Pending (WhiteGlove Part 2)");
status = SessionStatus.InProgress.ToString();
verdictPath = VerdictPaths.RegisterWhiteGloveResume;
// Store ResumedAt as fallback if whiteglove_resumed event hasn't set it yet
if (!resumedAt.HasValue)
resumedAt = registration.StartedAt;
@@ -618,6 +638,12 @@ namespace AutopilotMonitor.Functions.Services
if (!string.IsNullOrWhiteSpace(reconcileReason))
entity["ReconcileReason"] = reconcileReason;
entity["VerdictPath"] = verdictPath;
if (!string.IsNullOrWhiteSpace(priorStatus))
entity["PriorStatus"] = priorStatus;
if (!string.IsNullOrWhiteSpace(priorVerdictPath))
entity["PriorVerdictPath"] = priorVerdictPath;
// Latest non-Unknown validation wins: a Bootstrap-registered session that later
// re-registers under cert auth upgrades to the cert-path validator; an Unknown
// result (device validation disabled for the tenant) preserves whatever an
@@ -691,7 +717,7 @@ namespace AutopilotMonitor.Functions.Services
// fresh-read window isn't reverted by the Replace below (see guard).
"Status", "CurrentPhase", "CompletedAt", "FailureReason", "ReconcileReason",
"FailureSnapshotJson", "FailureSource", "AdminMarkedAction", "DurationSeconds",
"EspSoftFailure", "CompletionSource",
"EspSoftFailure", "CompletionSource", "VerdictPath", "PriorStatus", "PriorVerdictPath",
});
freshEtag = freshResponse.Value.ETag;
freshEntity = freshResponse.Value;
@@ -731,7 +757,7 @@ namespace AutopilotMonitor.Functions.Services
|| freshStatus == SessionStatus.Incomplete.ToString())
{
entity["Status"] = freshStatus;
foreach (var col in new[] { "CurrentPhase", "CompletedAt", "FailureReason", "ReconcileReason", "FailureSnapshotJson", "FailureSource", "AdminMarkedAction", "DurationSeconds", "EspSoftFailure", "CompletionSource" })
foreach (var col in new[] { "CurrentPhase", "CompletedAt", "FailureReason", "ReconcileReason", "FailureSnapshotJson", "FailureSource", "AdminMarkedAction", "DurationSeconds", "EspSoftFailure", "CompletionSource", "VerdictPath", "PriorStatus", "PriorVerdictPath" })
{
if (freshEntity!.TryGetValue(col, out var val) && val is not null)
entity[col] = val;
@@ -1709,10 +1735,53 @@ namespace AutopilotMonitor.Functions.Services
: null;
}
public async Task<bool> UpdateSessionStatusAsync(string tenantId, string sessionId, SessionStatus status, EnrollmentPhase? currentPhase = null, string? failureReason = null, DateTime? completedAt = null, DateTime? earliestEventTimestamp = null, DateTime? latestEventTimestamp = null, bool? isPreProvisioned = null, bool? isUserDriven = null, DateTime? resumedAt = null, DateTime? stalledAt = null, bool clearStalledAt = false, bool clearFailureReason = false, string? failureSource = null, string? adminMarkedAction = null, string? failureSnapshotJson = null, bool allowTerminalReclassification = false, bool espSoftFailure = false, string? completionSource = null)
/// <summary>
/// Decides whether a status write overrides a prior VERDICT and must therefore preserve
/// it in <c>PriorStatus</c>/<c>PriorVerdictPath</c> — the correction stream the verdict
/// calibration aggregate counts (admin mark, late agent completion upgrading a sweep
/// Incomplete, retro-reclassification). Only a prior Succeeded/Failed/Incomplete/
/// AwaitingUser counts as a verdict; a stall marker healing back to InProgress is not a
/// correction. Same-status writes (AwaitingUser refresh) keep the original prior. Null =
/// nothing to preserve. Static + pure so the matrix is unit-testable without Table Storage.
/// </summary>
internal static (string PriorStatus, string PriorVerdictPath)? ComputePriorVerdict(
string? existingStatus, string? existingVerdictPath, SessionStatus incoming)
{
if (string.IsNullOrEmpty(existingStatus) || string.IsNullOrEmpty(existingVerdictPath))
return null;
if (existingStatus == incoming.ToString())
return null;
bool existingIsVerdict = existingStatus == SessionStatus.Succeeded.ToString()
|| existingStatus == SessionStatus.Failed.ToString()
|| existingStatus == SessionStatus.Incomplete.ToString()
|| existingStatus == SessionStatus.AwaitingUser.ToString();
return existingIsVerdict ? (existingStatus!, existingVerdictPath!) : null;
}
/// <summary>
/// Stamps <c>VerdictPath</c> (always — every status write declares its origin, unlike
/// FailureSource which is Failed-only) and the prior-verdict pair when
/// <see cref="ComputePriorVerdict"/> says the write overrides one. Shared by the normal and
/// the ETag-exhaustion force path so both stay in lock-step.
/// </summary>
private static void ApplyVerdictPath(TableEntity update, TableEntity existing, SessionStatus status, string verdictPath)
{
update["VerdictPath"] = verdictPath;
var prior = ComputePriorVerdict(existing.GetString("Status"), existing.GetString("VerdictPath"), status);
if (prior.HasValue)
{
update["PriorStatus"] = prior.Value.PriorStatus;
update["PriorVerdictPath"] = prior.Value.PriorVerdictPath;
}
}
public async Task<bool> UpdateSessionStatusAsync(string tenantId, string sessionId, SessionStatus status, string verdictPath, EnrollmentPhase? currentPhase = null, string? failureReason = null, DateTime? completedAt = null, DateTime? earliestEventTimestamp = null, DateTime? latestEventTimestamp = null, bool? isPreProvisioned = null, bool? isUserDriven = null, DateTime? resumedAt = null, DateTime? stalledAt = null, bool clearStalledAt = false, bool clearFailureReason = false, string? failureSource = null, string? adminMarkedAction = null, string? failureSnapshotJson = null, bool allowTerminalReclassification = false, bool espSoftFailure = false, string? completionSource = null)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
SecurityValidator.EnsureValidGuid(sessionId, nameof(sessionId));
if (string.IsNullOrWhiteSpace(verdictPath))
throw new ArgumentException("Every status write must declare its VerdictPath (see VerdictPaths)", nameof(verdictPath));
const int maxRetries = 5;
int retryCount = 0;
@@ -1792,6 +1861,7 @@ namespace AutopilotMonitor.Functions.Services
}
update["Status"] = status.ToString();
ApplyVerdictPath(update, session, status, verdictPath);
// Update current phase if provided.
if (currentPhase.HasValue)
@@ -2080,6 +2150,7 @@ namespace AutopilotMonitor.Functions.Services
var forceUpdate = new TableEntity(tenantId, sessionId);
forceUpdate["Status"] = status.ToString();
ApplyVerdictPath(forceUpdate, freshSession, status, verdictPath);
if (currentPhase.HasValue)
forceUpdate["CurrentPhase"] = (int)currentPhase.Value;
@@ -2833,10 +2904,12 @@ namespace AutopilotMonitor.Functions.Services
/// unconditional Merge-mode write. Uses ETag.All to bypass optimistic-concurrency conflicts,
/// making this suitable as a last-resort fallback when ETag-based updates have been exhausted.
/// </summary>
public async Task SetSessionPreProvisionedAsync(string tenantId, string sessionId, bool isPreProvisioned, SessionStatus? status = null, bool? isUserDriven = null)
public async Task SetSessionPreProvisionedAsync(string tenantId, string sessionId, bool isPreProvisioned, SessionStatus? status = null, bool? isUserDriven = null, string? verdictPath = null)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
SecurityValidator.EnsureValidGuid(sessionId, nameof(sessionId));
if (status.HasValue && string.IsNullOrWhiteSpace(verdictPath))
throw new ArgumentException("A status write must declare its VerdictPath (see VerdictPaths)", nameof(verdictPath));
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.Sessions);
@@ -2848,6 +2921,7 @@ namespace AutopilotMonitor.Functions.Services
if (status.HasValue)
{
update["Status"] = status.Value.ToString();
update["VerdictPath"] = verdictPath!;
}
if (isUserDriven.HasValue)
@@ -3162,6 +3236,9 @@ namespace AutopilotMonitor.Functions.Services
EspSoftFailure = entity.GetBoolean("EspSoftFailure") ?? false,
CompletionSource = entity.GetString("CompletionSource") ?? string.Empty,
AdminMarkedAction = entity.GetString("AdminMarkedAction"),
VerdictPath = entity.GetString("VerdictPath"),
PriorStatus = entity.GetString("PriorStatus"),
PriorVerdictPath = entity.GetString("PriorVerdictPath"),
ValidatedBy = entity.GetString("ValidatedBy") ?? string.Empty,
PendingActionsJson = entity.GetString("PendingActionsJson") ?? string.Empty,
PendingActionsQueuedAt = SafeGetDateTime(entity, "PendingActionsQueuedAt"),
@@ -0,0 +1,125 @@
using System;
using System.Collections.Generic;
using System.Text.Json;
using System.Threading.Tasks;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services
{
/// <summary>
/// Partial: verdict-calibration daily aggregate persistence (docs/backend/verdict-calibration.md).
/// VerdictCalibrationAggregates: PK=TenantId ("global" for the cross-tenant row),
/// RK="{yyyy-MM-dd}" — one row per (tenant, StartedAt date) holding the per-verdict-path
/// buckets as a JSON column. Recomputed whole (Replace) by the maintenance sweep, 180d
/// retention, regenerable from Sessions + DeviceHistories.
/// </summary>
public partial class TableStorageService
{
public async Task<bool> SaveVerdictCalibrationAggregateAsync(VerdictCalibrationDailyAggregate aggregate)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.VerdictCalibrationAggregates);
await tableClient.UpsertEntityAsync(BuildVerdictCalibrationAggregateEntity(aggregate), TableUpdateMode.Replace);
return true;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to save verdict-calibration aggregate {Date} for tenant {TenantId}",
aggregate.Date, aggregate.TenantId);
return false;
}
}
public async Task<List<VerdictCalibrationDailyAggregate>> GetVerdictCalibrationAggregatesAsync(
string tenantId, DateTime startDate, DateTime endDate)
{
if (tenantId != "global")
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.VerdictCalibrationAggregates);
var filter = $"PartitionKey eq '{tenantId}' and RowKey ge '{startDate:yyyy-MM-dd}' and RowKey le '{endDate:yyyy-MM-dd}'";
var results = new List<VerdictCalibrationDailyAggregate>();
await foreach (var entity in tableClient.QueryAsync<TableEntity>(filter: filter))
{
results.Add(MapToVerdictCalibrationAggregate(entity));
}
return results;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to query verdict-calibration aggregates for tenant {TenantId}", tenantId);
return new List<VerdictCalibrationDailyAggregate>();
}
}
// internal static: pinned by round-trip tests together with MapToVerdictCalibrationAggregate.
internal static TableEntity BuildVerdictCalibrationAggregateEntity(VerdictCalibrationDailyAggregate aggregate)
{
return new TableEntity(aggregate.TenantId, aggregate.Date)
{
["Date"] = aggregate.Date,
["Version"] = aggregate.Version,
["SessionCount"] = aggregate.SessionCount,
["TerminalSessionCount"] = aggregate.TerminalSessionCount,
["BucketsJson"] = JsonSerializer.Serialize(aggregate.Buckets),
["ComputedAt"] = EnsureUtc(aggregate.ComputedAt),
};
}
internal static VerdictCalibrationDailyAggregate MapToVerdictCalibrationAggregate(TableEntity entity)
{
return new VerdictCalibrationDailyAggregate
{
TenantId = entity.PartitionKey,
Date = entity.GetString("Date") ?? entity.RowKey,
Version = entity.GetInt32("Version") ?? 0,
SessionCount = entity.GetInt32("SessionCount") ?? 0,
TerminalSessionCount = entity.GetInt32("TerminalSessionCount") ?? 0,
Buckets = DeserializeJsonColumn<VerdictCalibrationBucket>(entity.GetString("BucketsJson")),
ComputedAt = entity.GetDateTimeOffset("ComputedAt")?.UtcDateTime ?? DateTime.MinValue,
};
}
public async Task DeleteVerdictCalibrationAggregateAsync(string tenantId, string dateKey)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.VerdictCalibrationAggregates);
await tableClient.DeleteEntityAsync(tenantId, dateKey);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to delete stale verdict-calibration aggregate {Date} for tenant {TenantId}",
dateKey, tenantId);
}
}
public async Task<int> DeleteVerdictCalibrationAggregatesOlderThanAsync(DateTime cutoffDate)
{
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.VerdictCalibrationAggregates);
var filter = $"RowKey lt '{cutoffDate:yyyy-MM-dd}'";
var deleted = 0;
await foreach (var entity in tableClient.QueryAsync<TableEntity>(
filter: filter, select: new[] { "PartitionKey", "RowKey" }))
{
await tableClient.DeleteEntityAsync(entity.PartitionKey, entity.RowKey);
deleted++;
}
return deleted;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to delete old verdict-calibration aggregates");
return 0;
}
}
}
}
@@ -78,25 +78,92 @@ namespace AutopilotMonitor.Functions.Services
return _tableServiceClient.GetTableClient(tableName);
}
// ===== TABLE SCHEMA SENTINEL =====
//
// A cold start used to issue one CreateTableIfNotExists per table (each a Create POST
// whose 409 is swallowed). The sentinel row in AdminConfiguration stores a hash derived
// from Constants.TableNames.All; when it matches, startup is a single point-read.
// Adding a table to TableNames.All changes the hash, so the next start runs the full
// pass once and rewrites the sentinel — no manually maintained version to forget.
// Gap: a table deleted by hand is not noticed until the daily maintenance full pass
// (EnsureAllTablesAsync) recreates it.
internal const string SchemaSentinelPartitionKey = "SchemaSentinel";
internal const string SchemaSentinelRowKey = "tables";
internal const string SchemaSentinelHashProperty = "TableSchemaHash";
internal const string SessionIndexBackfillClaimRowKey = "sessionIndexBackfill";
/// <summary>Outcome of the last <see cref="InitializeTablesAsync"/> call, read by StartupTelemetryService.</summary>
public readonly record struct InitializationSnapshot(double DurationMs, bool FullPassRan);
public InitializationSnapshot LastInitialization { get; private set; }
/// <summary>
/// Deterministic hash over the table registry (ordinal-sorted, so declaration order is irrelevant).
/// </summary>
internal static string ComputeTableSchemaHash(IEnumerable<string> tableNames)
{
var joined = string.Join("\n", tableNames.OrderBy(n => n, StringComparer.Ordinal));
var bytes = System.Security.Cryptography.SHA256.HashData(System.Text.Encoding.UTF8.GetBytes(joined));
return Convert.ToHexString(bytes);
}
/// <summary>
/// Initializes all Azure Table Storage tables.
/// This method is idempotent and safe to call multiple times.
/// Should be called at application startup.
/// Fast path: the schema sentinel matches the current table registry → no table calls.
/// Slow path (sentinel missing, stale, or unreadable): CreateTableIfNotExists for every
/// table, then the sentinel is (re)written. Idempotent and safe under scale-out — two
/// instances that both see a stale sentinel simply both run the idempotent full pass.
/// Returns true when the full pass ran (callers use this to gate one-time work such as
/// the SessionsIndex backfill, which can only be needed on fresh storage).
/// </summary>
public async Task InitializeTablesAsync()
public async Task<bool> InitializeTablesAsync()
{
if (_tablesInitialized)
{
_logger.LogDebug("Tables already initialized, skipping");
return;
return false;
}
lock (_initLock)
{
if (_tablesInitialized) return;
if (_tablesInitialized) return false;
}
_logger.LogInformation("Initializing Azure Table Storage tables...");
var initStopwatch = System.Diagnostics.Stopwatch.StartNew();
var expectedHash = ComputeTableSchemaHash(Constants.TableNames.All);
if (await SchemaSentinelMatchesAsync(expectedHash))
{
_logger.LogInformation("Table schema sentinel matches ({HashPrefix}) — skipping table creation pass",
expectedHash.Substring(0, 12));
lock (_initLock) { _tablesInitialized = true; }
LastInitialization = new InitializationSnapshot(initStopwatch.Elapsed.TotalMilliseconds, FullPassRan: false);
return false;
}
_logger.LogInformation("Table schema sentinel missing or stale — initializing Azure Table Storage tables...");
var failCount = await EnsureAllTablesAsync();
if (failCount == 0)
{
await WriteSchemaSentinelAsync(expectedHash);
}
lock (_initLock)
{
_tablesInitialized = failCount == 0;
}
LastInitialization = new InitializationSnapshot(initStopwatch.Elapsed.TotalMilliseconds, FullPassRan: true);
return true;
}
/// <summary>
/// Unconditional full pass: CreateTableIfNotExists for every table in the registry.
/// Used by the startup slow path and by daily maintenance (recreates manually deleted tables).
/// Returns the number of tables that failed to initialize.
/// </summary>
public async Task<int> EnsureAllTablesAsync()
{
var stopwatch = System.Diagnostics.Stopwatch.StartNew();
var successCount = 0;
var failCount = 0;
@@ -126,9 +193,74 @@ namespace AutopilotMonitor.Functions.Services
// CPE mapping seed is imported via Admin UI "Re-Seed Mappings" button
// (pulls from GitHub, not embedded resource). No auto-import at startup.
lock (_initLock)
return failCount;
}
private async Task<bool> SchemaSentinelMatchesAsync(string expectedHash)
{
try
{
_tablesInitialized = failCount == 0;
var table = _tableServiceClient.GetTableClient(Constants.TableNames.AdminConfiguration);
var response = await table.GetEntityIfExistsAsync<TableEntity>(
SchemaSentinelPartitionKey, SchemaSentinelRowKey, select: new[] { SchemaSentinelHashProperty });
if (!response.HasValue) return false;
return string.Equals(response.Value!.GetString(SchemaSentinelHashProperty), expectedHash, StringComparison.Ordinal);
}
catch (Exception ex)
{
// Table missing (fresh storage) or transient error — fall back to the full pass.
_logger.LogInformation(ex, "Table schema sentinel not readable — running full table initialization");
return false;
}
}
private async Task WriteSchemaSentinelAsync(string hash)
{
try
{
var table = _tableServiceClient.GetTableClient(Constants.TableNames.AdminConfiguration);
var entity = new TableEntity(SchemaSentinelPartitionKey, SchemaSentinelRowKey)
{
[SchemaSentinelHashProperty] = hash,
["TableCount"] = Constants.TableNames.All.Length,
["UpdatedUtc"] = DateTime.UtcNow
};
await table.UpsertEntityAsync(entity, TableUpdateMode.Replace);
}
catch (Exception ex)
{
// Non-fatal: the next start simply runs the full pass again.
_logger.LogWarning(ex, "Failed to write table schema sentinel");
}
}
/// <summary>
/// Scale-out guard for the one-time SessionsIndex backfill: a conditional insert that
/// only one instance can win. Returns true for the winner. A lost race (409) or any
/// error yields false — the manual maintenance backfill remains the safety net.
/// </summary>
public async Task<bool> TryClaimSessionIndexBackfillAsync()
{
try
{
var table = _tableServiceClient.GetTableClient(Constants.TableNames.AdminConfiguration);
var entity = new TableEntity(SchemaSentinelPartitionKey, SessionIndexBackfillClaimRowKey)
{
["ClaimedUtc"] = DateTime.UtcNow,
["Instance"] = Environment.MachineName
};
await table.AddEntityAsync(entity);
return true;
}
catch (RequestFailedException ex) when (ex.Status == 409)
{
_logger.LogInformation("SessionsIndex backfill already claimed by another instance");
return false;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to claim SessionsIndex backfill");
return false;
}
}
@@ -17,8 +17,14 @@ namespace AutopilotMonitor.Functions.Telemetry;
/// - Only <see cref="DependencyTelemetry"/> is considered; requests, traces, exceptions,
/// metrics and all NON-storage dependencies (HTTP, Microsoft Graph, SQL, SignalR, ...) pass
/// through untouched.
/// - FAILED storage calls are KEPT (Success == false): they are rare and are real signal for
/// troubleshooting throttling / transient faults, so the noise reduction does not blind us.
/// - FAILED storage calls are KEPT (Success == false) EXCEPT the expected-outcome status codes
/// 404 (point-read miss), 412 (ETag precondition) and 409 (insert conflict): those are normal
/// control flow for this backend (index lookups, optimistic concurrency, idempotent inserts).
/// Live-verified 2026-08-23: 100% of the remaining "failed" storage rows were 404/412/409
/// (~540 MB/week, two rows per call: InProc span + HTTP span). Throttling (429), 5xx, auth
/// failures and timeouts stay — they are the real signal.
/// - Successful Azure SignalR REST management calls (group add/remove per connection, ~470 MB/week)
/// are dropped for the same reason; failed SignalR calls are kept.
///
/// Registered in Program.cs via AddApplicationInsightsTelemetryProcessor so it runs inside the
/// isolated worker's telemetry pipeline, where the app's own Azure SDK dependencies are tracked.
@@ -46,6 +52,12 @@ public sealed class StorageDependencyFilterProcessor : ITelemetryProcessor
"InProc | Microsoft.Storage", // Azure.Storage.Queues + Azure.Storage.Blobs
};
// Azure SignalR Service REST endpoint (group membership management per connection).
private const string SignalRServiceSuffix = ".service.signalr.net";
// Expected storage outcomes that are normal control flow, not failures worth billing.
private static readonly string[] ExpectedStorageStatusCodes = { "404", "412", "409" };
private readonly ITelemetryProcessor _next;
public StorageDependencyFilterProcessor(ITelemetryProcessor next) => _next = next;
@@ -68,17 +80,50 @@ public sealed class StorageDependencyFilterProcessor : ITelemetryProcessor
return false;
}
// Keep failures — rare, high-value diagnostic signal. Only successful storage chatter is noise.
if (dependency.Success == false)
{
return false;
}
return IsStorageEndpoint(dependency.Target)
var isStorage = IsStorageEndpoint(dependency.Target)
|| IsStorageEndpoint(dependency.Data)
|| IsStorageInProcType(dependency.Type);
if (isStorage)
{
// Successful storage chatter is noise; so are expected 404/412/409 outcomes.
// Every other failure (429, 5xx, auth, timeout) is kept.
return dependency.Success != false || IsExpectedStorageOutcome(dependency);
}
// SignalR REST management calls: keep only failures.
return dependency.Success != false && IsSignalRService(dependency.Target);
}
private static bool IsExpectedStorageOutcome(DependencyTelemetry dependency)
{
// HTTP-shaped rows ("Azure table") carry the status in ResultCode; the Azure SDK InProc
// span has no ResultCode, only the exception text in Properties["Error"]
// ("...Status: 404 (Not Found)...").
var code = dependency.ResultCode;
if (!string.IsNullOrEmpty(code))
{
return Array.IndexOf(ExpectedStorageStatusCodes, code) >= 0;
}
if (dependency.Properties.TryGetValue("Error", out var error) && !string.IsNullOrEmpty(error))
{
foreach (var expected in ExpectedStorageStatusCodes)
{
if (error.Contains("Status: " + expected + " (", System.StringComparison.Ordinal))
{
return true;
}
}
}
return false;
}
private static bool IsSignalRService(string? target)
=> !string.IsNullOrEmpty(target)
&& target.EndsWith(SignalRServiceSuffix, System.StringComparison.OrdinalIgnoreCase);
private static bool IsStorageInProcType(string? type)
{
if (string.IsNullOrEmpty(type))
@@ -136,6 +136,9 @@ describe('role catalog snapshot — privilege-leak guard', () => {
// routing to the GlobalReadOrAdmin variant — same placement as get_app_install_metrics.
'get_time_attribution',
'get_usage_metrics',
// Platform scope (GA + Global Reader): GlobalReadOrAdmin endpoint, operator-only
// classifier diagnostics.
'get_verdict_calibration',
'get_vulnerability_summary',
'list_blocked_devices',
'list_session_annotations',
@@ -197,6 +200,7 @@ describe('role catalog snapshot — privilege-leak guard', () => {
'get_platform_metrics',
'get_tenant_config',
'get_tenant_config_schema',
'get_verdict_calibration',
'list_blocked_devices',
'list_session_annotations',
'list_session_reports',
@@ -0,0 +1,83 @@
import { describe, it, expect } from 'vitest';
import { shapeVerdictCalibration, type VerdictCalibrationPathRow } from '../verdict-calibration-shape.js';
function row(verdictPath: string, count: number, sharePct: number, extra: Partial<VerdictCalibrationPathRow> = {}): VerdictCalibrationPathRow {
return {
verdictPath,
status: 'Succeeded',
count,
sharePct,
overriddenByAdmin: 0,
overriddenByLateCompletion: 0,
overriddenOther: 0,
window7: { count, sessions: 1000, sharePct },
baseline28: { count, sessions: 2000, sharePct },
...extra,
};
}
function payload(paths: VerdictCalibrationPathRow[]) {
return {
success: true,
totals: { sessions: 1000 },
trend: { windowSessions: 1000, baselineSessions: 2000 },
paths,
alerts: [],
};
}
describe('shapeVerdictCalibration', () => {
it('drops the per-row denominators and makes withheld values explicit null', () => {
const out = shapeVerdictCalibration(payload([row('agent:complete', 500, 50)])) as any;
expect(out.paths[0].window7).toEqual({ count: 500, sharePct: 50 });
expect(out.paths[0].baseline28).toEqual({ count: 500, sharePct: 50 });
expect(out.paths[0]).toHaveProperty('reEnrollRatePct', null);
expect(out.paths[0]).toHaveProperty('lift', null);
expect(out.omitted).toBeUndefined();
expect(out.trend).toEqual({ windowSessions: 1000, baselineSessions: 2000 });
});
it('keeps real numeric values (0 is not null)', () => {
const out = shapeVerdictCalibration(payload([row('legacy:r3', 4, 0.1, { reEnrollRatePct: 0, lift: 0 })])) as any;
expect(out.paths[0].reEnrollRatePct).toBe(0);
expect(out.paths[0].lift).toBe(0);
});
it('minSharePct trims the long tail and reports what was omitted', () => {
const out = shapeVerdictCalibration(payload([
row('agent:complete', 900, 90),
row('legacy:r4', 10, 1),
row('manual:succeeded', 1, 0.1),
]), { minSharePct: 1 }) as any;
expect(out.paths.map((p: any) => p.verdictPath)).toEqual(['agent:complete', 'legacy:r4']);
expect(out.omitted).toEqual({ paths: 1, sessions: 1, reason: expect.stringContaining('share < 1%') });
});
it('top keeps the N largest rows by backend order', () => {
const out = shapeVerdictCalibration(payload([
row('a', 50, 50), row('b', 30, 30), row('c', 20, 20),
]), { top: 2 }) as any;
expect(out.paths.map((p: any) => p.verdictPath)).toEqual(['a', 'b']);
expect(out.omitted.paths).toBe(1);
expect(out.omitted.reason).toContain('rank > 2');
});
it('never trims a row that carries overrides — that is the signal', () => {
const out = shapeVerdictCalibration(payload([
row('agent:complete', 999, 99.9),
row('sweep:r5_incomplete', 1, 0.1, { overriddenByLateCompletion: 3 }),
]), { minSharePct: 5, top: 1 }) as any;
expect(out.paths.map((p: any) => p.verdictPath)).toEqual(['agent:complete', 'sweep:r5_incomplete']);
expect(out.omitted).toBeUndefined();
});
it('does not mutate the input and passes non-matrix payloads through', () => {
const input = payload([row('x', 1, 100)]);
const snapshot = JSON.stringify(input);
shapeVerdictCalibration(input, { top: 0 });
expect(JSON.stringify(input)).toBe(snapshot);
const err = { success: false, message: 'nope' };
expect(shapeVerdictCalibration(err)).toBe(err);
expect(shapeVerdictCalibration(null)).toBeNull();
});
});
@@ -476,6 +476,17 @@ const server = app.listen(PORT, '0.0.0.0', () => {
// The number a user actually waits for on a cold start, minus the platform's
// own activation (pod scheduling, container start), which is not visible here.
console.error(`[boot ${bootMark()}] Autopilot-Monitor MCP Server listening on port ${PORT} — ready to serve.`);
// Same number as a structured line so it is a trend, not prose: queryable in
// ContainerAppConsoleLogs_CL next to the tool_call lines (see telemetry.ts),
// e.g. `| where Log_s startswith '{"type":"boot"'`. Always on — one line per start.
console.error(JSON.stringify({
type: 'boot',
readyMs: Math.round(process.uptime() * 1000),
version: SERVER_VERSION,
commit: BUILD_COMMIT,
docsCommit: DOCS_COMMIT,
timestamp: new Date().toISOString(),
}));
});
// --- Graceful shutdown ---
@@ -5,6 +5,7 @@ import { withToolTelemetry } from '../telemetry.js';
import { getResourceContent, assertKnownEventType } from '../resource-catalog.js';
import { READ_ONLY, READ_ONLY_OPEN, MUTATING, MAX_RESULT_SIZE_CHARS, toolResultText, SessionIdSchema, TenantGuidSchema, tenantIdDescription } from './shared.js';
import { toolError } from './error-handler.js';
import { shapeVerdictCalibration } from '../verdict-calibration-shape.js';
/**
* Non-sensitive tenant fields safe to surface to the model. Keep-list (not
@@ -838,6 +839,53 @@ export function registerAdminTools(server: McpServer, ga: boolean, strictGa: boo
})
);
// Tool: get_verdict_calibration — platform scope (GA + Global Reader). Operator diagnostics for
// the session-verdict classifier (docs/backend/verdict-calibration.md) — never a customer surface.
if (ga) server.registerTool(
'get_verdict_calibration',
{
title: 'Verdict Calibration (classifier thermometer)',
description:
'Answer "which code path produced our session verdicts, and does any of them look wrong?". Per verdict ' +
'path (VerdictPaths vocabulary — agent:complete, agent:failed, sweep:r5_incomplete, maxlife:r6, ' +
'late:r4, retro:r6, rule:<id>, manual:failed, register:superseded, legacy:* for pre-instrumentation ' +
'rows derived read-side): session count and share in the window, derivedCount (rows attributed by ' +
'derivation — weaker evidence), the 7-day re-enrollment proxy (eligible7d = terminal sessions old ' +
'enough to judge, reEnrolled7d = the same device registered another terminal session within 7 days; ' +
'reEnrollRatePct is null below 20 eligible), the correction stream attributed to the PRIOR path ' +
'(overriddenByAdmin / overriddenByLateCompletion / overriddenOther — a path that is frequently ' +
'overridden was wrong), and a 7d-vs-28d share trend (window7, baseline28, lift; lift null without ' +
'baseline; window/baseline denominators are the single `trend.windowSessions` / `trend.baselineSessions`). ' +
'reEnrollRatePct and lift are always present — null means withheld, not missing. Read it as calibration: ' +
'a Succeeded path whose re-enrollment rate sits at Failed level reconciles too boldly; an Incomplete ' +
'path whose rate matches the Succeeded background is too cautious; a rising sweep:*/maxlife:* share ' +
'is an agent-liveness signal, not a classifier one. Omit tenantId for the cross-tenant aggregate, or ' +
'pass tenantId to scope to one tenant. minSharePct / top trim the long tail of one-session paths ' +
'(rows carrying overrides are never trimmed); what was dropped is reported in `omitted`, totals and ' +
'trend always cover the full window. Rows come from the 2-hourly maintenance sweep (computedAt); ' +
'empty before its first run.',
inputSchema: {
tenantId: z.string().optional().describe('Optional — scope to one tenant. Omit for the cross-tenant aggregate.'),
days: z.coerce.number().int().min(1).max(180).optional()
.describe('Window in days (default 30, max 180 — aggregate retention). The 7d/28d trend is always today-anchored.'),
minSharePct: z.coerce.number().min(0).max(100).optional()
.describe('Drop paths whose share of the window is below this percentage (e.g. 1). Paths with overrides are always kept.'),
top: z.coerce.number().int().min(1).max(100).optional()
.describe('Keep only the N largest paths by session count. Paths with overrides are always kept.'),
},
annotations: READ_ONLY,
},
async (args) => withToolTelemetry('get_verdict_calibration', args, async () => {
try {
const { tenantId, days, minSharePct, top } = args;
const data = await apiFetch(`/api/global/metrics/verdict-calibration${buildQuery({ tenantId, days })}`);
return toolResultText(shapeVerdictCalibration(data, { minSharePct, top }), MAX_RESULT_SIZE_CHARS.adminStream);
} catch (error: unknown) {
return toolError('get_verdict_calibration', args, error);
}
})
);
// Tool 18: list_session_reports — Global Admin only; not registered for normal users.
if (ga) server.registerTool(
'list_session_reports',
@@ -0,0 +1,102 @@
/**
* LLM-facing shaping of the verdict-calibration matrix (GET /api/global/metrics/verdict-calibration).
*
* The backend wire shape is shared with the admin metrics page and stays untouched; this layer only
* trims what a model does not need and makes absent values explicit:
* - `window7.sessions` / `baseline28.sessions` repeat `trend.windowSessions` / `trend.baselineSessions`
* on every row — dropped (the denominators live once in `trend`).
* - `reEnrollRatePct` / `lift` are omitted by the backend's WhenWritingNull serializer when null;
* a missing key is easy to misread as "not applicable", so both are always present (null = withheld).
* - `minSharePct` / `top` cut the long tail of one-session paths. Rows carrying overrides are never
* cut (an override on a rare path is the signal the tool exists for). What was dropped is reported
* in `omitted` so a filtered response never reads as complete coverage.
*/
interface TrendCell {
count: number;
sessions?: number;
sharePct: number;
}
export interface VerdictCalibrationPathRow {
verdictPath: string;
status: string;
count: number;
sharePct: number;
overriddenByAdmin: number;
overriddenByLateCompletion: number;
overriddenOther: number;
reEnrollRatePct?: number | null;
lift?: number | null;
window7: TrendCell;
baseline28: TrendCell;
[key: string]: unknown;
}
export interface VerdictCalibrationShapeOptions {
/** Drop rows whose share (in the window) is below this percentage. */
minSharePct?: number;
/** Keep only the first N rows (the backend orders by count descending). */
top?: number;
}
export interface VerdictCalibrationOmitted {
paths: number;
sessions: number;
reason: string;
}
function hasOverrides(row: VerdictCalibrationPathRow): boolean {
return (row.overriddenByAdmin ?? 0) + (row.overriddenByLateCompletion ?? 0) + (row.overriddenOther ?? 0) > 0;
}
function trimCell(cell: TrendCell | undefined): TrendCell {
if (!cell) return { count: 0, sharePct: 0 };
const { sessions: _sessions, ...rest } = cell;
return rest;
}
/**
* Pure: returns a new object; the input is not mutated. Non-object or unsuccessful payloads
* (error envelopes) pass through unchanged so the caller's error handling still sees them.
*/
export function shapeVerdictCalibration(data: unknown, options: VerdictCalibrationShapeOptions = {}): unknown {
if (!data || typeof data !== 'object' || !Array.isArray((data as { paths?: unknown }).paths)) return data;
const payload = data as { paths: VerdictCalibrationPathRow[] } & Record<string, unknown>;
const { minSharePct, top } = options;
const kept: VerdictCalibrationPathRow[] = [];
let omittedPaths = 0;
let omittedSessions = 0;
payload.paths.forEach((row, index) => {
const belowShare = minSharePct != null && row.sharePct < minSharePct;
const beyondTop = top != null && index >= top;
if ((belowShare || beyondTop) && !hasOverrides(row)) {
omittedPaths++;
omittedSessions += row.count ?? 0;
return;
}
kept.push({
...row,
reEnrollRatePct: row.reEnrollRatePct ?? null,
lift: row.lift ?? null,
window7: trimCell(row.window7),
baseline28: trimCell(row.baseline28),
});
});
const shaped: Record<string, unknown> = { ...payload, paths: kept };
if (omittedPaths > 0) {
const criteria = [
minSharePct != null ? `share < ${minSharePct}%` : null,
top != null ? `rank > ${top}` : null,
].filter(Boolean).join(' or ');
shaped.omitted = {
paths: omittedPaths,
sessions: omittedSessions,
reason: `${omittedPaths} path(s) with ${criteria} dropped (rows carrying overrides are always kept); totals and trend still cover the full window.`,
} satisfies VerdictCalibrationOmitted;
}
return shaped;
}
@@ -988,6 +988,11 @@ namespace AutopilotMonitor.Shared
// like UsageMetrics.
public const string DeviceHistories = "DeviceHistories";
public const string DeviceJourneyAggregates = "DeviceJourneyAggregates";
// VerdictCalibrationAggregates: PK=TenantId (or "global"), RK="{yyyy-MM-dd}" — daily
// per-verdict-path buckets (count, derived, re-enrolled-within-7d, overridden-by-admin/
// late-completion) the operator calibration matrix + drift radar read
// (docs/backend/verdict-calibration.md). Additive across days; 180d retention; regenerable.
public const string VerdictCalibrationAggregates = "VerdictCalibrationAggregates";
// Configuration tables
public const string TenantConfiguration = "TenantConfiguration";
@@ -1198,6 +1203,7 @@ namespace AutopilotMonitor.Shared
TimeAttributionAggregates,
DeviceHistories,
DeviceJourneyAggregates,
VerdictCalibrationAggregates,
TenantConfiguration,
AdminConfiguration,
GlobalAdmins,
@@ -55,6 +55,20 @@ namespace AutopilotMonitor.Shared.DataAccess
/// <summary>Active regression episodes of one tenant (RowKey-prefix scan; empty on failure — fail-soft reads).</summary>
Task<List<RuleRegressionAlert>> GetRuleRegressionsAsync(string tenantId);
// --- Verdict-calibration drift (RowKey "verdictcalibration|{kind}|{path}|{status}") ---
/// <summary>Atomically opens a verdict-calibration episode; true = this pass owns it (fires the ops event once). Fail-closed.</summary>
Task<bool> TryRegisterVerdictCalibrationAlertAsync(string tenantId, VerdictCalibrationAlert alert);
/// <summary>Refreshes an active episode's numbers; FirstNotifiedAt carried unchanged. Fail-soft.</summary>
Task RefreshVerdictCalibrationAlertAsync(string tenantId, VerdictCalibrationAlert alert);
/// <summary>Closes an episode (re-armed). 404-tolerant.</summary>
Task DeleteVerdictCalibrationAlertAsync(string tenantId, string kind, string verdictPath, string status);
/// <summary>Active verdict-calibration episodes of one tenant partition ("global" allowed); empty on failure.</summary>
Task<List<VerdictCalibrationAlert>> GetVerdictCalibrationAlertsAsync(string tenantId);
// --- App-version duration regressions (RowKey "appversionregression|{app}|{version}") ---
/// <summary>
@@ -80,6 +80,13 @@ namespace AutopilotMonitor.Shared.DataAccess
/// Fields outside the projection come back as defaults — callers must not read them.
/// </summary>
Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null);
/// <summary>
/// Column-projected cross-tenant variant of <see cref="GetSessionsByDateRangeAsync"/> that
/// the maintenance tick drains ONCE and shares across its rolling sweeps (time attribution,
/// device journeys, verdict calibration, calibration radar discovery). Fields outside the
/// projection come back as defaults — sweeps must not read them.
/// </summary>
Task<List<SessionSummary>> GetMaintenanceWindowSessionsAsync(DateTime startDate, DateTime endDate);
Task<List<SessionSummary>> GetStalledSessionsAsync(string tenantId, DateTime cutoffTime);
Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff);
@@ -128,6 +128,14 @@ namespace AutopilotMonitor.Shared.DataAccess
/// <summary>Deletes one daily FTR row — the sweep's stale-bucket reconcile.</summary>
Task DeleteDeviceJourneyAggregateAsync(string tenantId, string dateKey);
Task<int> DeleteDeviceJourneyAggregatesOlderThanAsync(DateTime cutoffDate);
// --- Verdict calibration (docs/backend/verdict-calibration.md) ---
Task<bool> SaveVerdictCalibrationAggregateAsync(VerdictCalibrationDailyAggregate aggregate);
/// <summary>Daily verdict-path rows of one tenant partition ("global" allowed), inclusive date range; counts are additive across days.</summary>
Task<List<VerdictCalibrationDailyAggregate>> GetVerdictCalibrationAggregatesAsync(string tenantId, DateTime startDate, DateTime endDate);
/// <summary>Deletes one daily row — the sweep's stale-bucket reconcile.</summary>
Task DeleteVerdictCalibrationAggregateAsync(string tenantId, string dateKey);
Task<int> DeleteVerdictCalibrationAggregatesOlderThanAsync(DateTime cutoffDate);
}
/// <summary>
@@ -83,8 +83,14 @@ namespace AutopilotMonitor.Shared.DataAccess
string? tenantIdFilter, int days, IReadOnlyCollection<string>? allowedTenantIds = null);
// --- Session Updates ---
/// <summary>
/// Single status-write seam. <paramref name="verdictPath"/> is REQUIRED: every writer names
/// the code path that produced the status (<see cref="VerdictPaths"/>), stamped for all
/// statuses so the verdict calibration aggregate can count each path. Overriding a prior
/// verdict preserves it in PriorStatus/PriorVerdictPath.
/// </summary>
Task<bool> UpdateSessionStatusAsync(
string tenantId, string sessionId, SessionStatus status,
string tenantId, string sessionId, SessionStatus status, string verdictPath,
EnrollmentPhase? currentPhase = null, string? failureReason = null,
DateTime? completedAt = null, DateTime? earliestEventTimestamp = null,
DateTime? latestEventTimestamp = null, bool? isPreProvisioned = null,
@@ -123,7 +129,7 @@ namespace AutopilotMonitor.Shared.DataAccess
Task UpdateSessionDiagnosticsBlobAsync(
string tenantId, string sessionId, string blobName, string? destination = null);
Task SetSessionPreProvisionedAsync(string tenantId, string sessionId, bool isPreProvisioned,
SessionStatus? status = null, bool? isUserDriven = null);
SessionStatus? status = null, bool? isUserDriven = null, string? verdictPath = null);
Task UpdateSessionGeoAsync(string tenantId, string sessionId,
string? country, string? region, string? city, string? loc);
Task UpdateSessionImeAgentVersionAsync(string tenantId, string sessionId, string version);
@@ -8,6 +8,14 @@ namespace AutopilotMonitor.Shared.DataAccess
/// </summary>
public interface IStorageInitializer
{
/// <summary>Startup path: cheap when the storage schema is already known to be current.</summary>
Task InitializeAsync();
/// <summary>
/// Unconditional full pass that (re)creates every registered table. Used by daily
/// maintenance to repair tables that were deleted out-of-band after the startup
/// fast path stopped checking them.
/// </summary>
Task EnsureAllAsync();
}
}
@@ -0,0 +1,59 @@
using System;
namespace AutopilotMonitor.Shared.Models
{
/// <summary>Kinds of verdict-calibration drift the radar detects (docs/backend/verdict-calibration.md).</summary>
public static class VerdictCalibrationAlertKinds
{
/// <summary>One verdict path's share of all sessions rose ≥2× over its 28-day baseline (Wilson-separated).</summary>
public const string ShareRegression = "share_regression";
/// <summary>The combined sweep:* + maxlife:* share (agent went silent, backend had to decide) rose ≥2× — an agent-liveness signal, not a classifier one.</summary>
public const string SilenceShareRegression = "silence_share_regression";
/// <summary>The pure fallthrough rule (r6) decides ≥20 % of classifier verdicts — the evidence the rules need is missing.</summary>
public const string EvidenceGap = "evidence_gap";
}
/// <summary>
/// One ACTIVE verdict-calibration alert episode (operator-only). Persisted as the
/// <c>verdictcalibration|{kind}|{path}|{status}</c> keyspace of the notification tracker —
/// the row IS the dedup (one ops event per episode) and the <c>alerts[]</c> payload of the
/// calibration endpoint. Deleted when the signal re-arms (share back under 1.5× baseline,
/// path stops occurring, evidence-gap share back under 15 %) or by the tracker's 30-day
/// retention sweep. Numbers refresh on every radar pass; <see cref="FirstNotifiedAt"/> never moves.
/// Dimension concentration is CORRELATION only — every consumer says so.
/// </summary>
public class VerdictCalibrationAlert
{
public string TenantId { get; set; } = string.Empty;
/// <summary>One of <see cref="VerdictCalibrationAlertKinds"/>.</summary>
public string Kind { get; set; } = string.Empty;
/// <summary>Verdict path, or the group label for group kinds ("sweep+maxlife", "r6").</summary>
public string VerdictPath { get; set; } = string.Empty;
/// <summary>Status the path produced, or "*" for group kinds.</summary>
public string Status { get; set; } = string.Empty;
public int WindowHitCount { get; set; }
public int WindowSessionCount { get; set; }
public int BaselineHitCount { get; set; }
public int BaselineSessionCount { get; set; }
public double WindowRatePct { get; set; }
public double BaselineRatePct { get; set; }
/// <summary>Window rate ÷ baseline rate. Null when the baseline rate is 0 or the kind is absolute (evidence gap).</summary>
public double? Lift { get; set; }
public string WindowStartDate { get; set; } = string.Empty;
public string WindowEndDate { get; set; } = string.Empty;
/// <summary>Dimension concentration captured when the alert FIRST fired; null = no clear concentration.</summary>
public RuleRegressionDimension? Dimension { get; set; }
public DateTime FirstNotifiedAt { get; set; }
public DateTime LastEvaluatedAt { get; set; }
}
}
@@ -0,0 +1,78 @@
using System;
using System.Collections.Generic;
namespace AutopilotMonitor.Shared.Models
{
/// <summary>
/// One (verdict path × status) bucket of a daily verdict-calibration row. <c>Count</c> is the
/// sessions whose CURRENT status was produced by this path; the <c>Overridden*</c> counters
/// are sessions that once carried this path and were then overridden (their
/// <c>PriorVerdictPath</c> names this bucket) — the correction stream. Re-enrollment is the
/// delayed proxy signal: of the sessions old enough to judge (<see cref="Eligible7d"/>), how
/// many saw the same device register another terminal session within 7 days.
/// </summary>
public class VerdictCalibrationBucket
{
/// <summary><see cref="VerdictPaths"/> value, or the read-side derivation for pre-instrumentation rows.</summary>
public string VerdictPath { get; set; } = string.Empty;
/// <summary><see cref="SessionStatus"/> name the path produced.</summary>
public string Status { get; set; } = string.Empty;
/// <summary>Sessions currently on this path/status.</summary>
public int Count { get; set; }
/// <summary>Of <see cref="Count"/>: rows without a stamped VerdictPath, attributed by <c>VerdictPathDerivation</c> (weaker evidence).</summary>
public int DerivedCount { get; set; }
/// <summary>Of <see cref="Count"/>: sessions whose terminal moment lies ≥ 7 days before the compute time — the re-enrollment denominator.</summary>
public int Eligible7d { get; set; }
/// <summary>Of <see cref="Eligible7d"/>: the device registered another terminal session within 7 days of this one's end.</summary>
public int ReEnrolled7d { get; set; }
/// <summary>Sessions that carried this path and were then flipped by an administrator (MarkSessionSucceeded/Failed).</summary>
public int OverriddenByAdmin { get; set; }
/// <summary>Sessions that carried this path and were then upgraded to Succeeded by a late agent completion report.</summary>
public int OverriddenByLateCompletion { get; set; }
/// <summary>Sessions that carried this path and were overridden by any other writer (retro-reclassification, grace expiry, supersede).</summary>
public int OverriddenOther { get; set; }
}
/// <summary>
/// Daily verdict-calibration rollup for one (tenant, date) — the operator's thermometer for the
/// rule classifier (docs/backend/verdict-calibration.md). A session buckets on its StartedAt
/// date (same convention as every other daily aggregate; the rolling sweep re-buckets
/// late-terminating sessions idempotently). Every status is counted — non-terminal rows are
/// part of the picture (a growing Stalled share is a signal) — and
/// <see cref="TerminalSessionCount"/> carries the share denominator for terminal paths.
/// Counts are additive across days; a window is the sum of daily rows. "global" mirror rows
/// sum all tenants. Regenerable from Sessions + DeviceHistories at any time.
/// </summary>
public class VerdictCalibrationDailyAggregate
{
/// <summary>Bucketing/derivation algorithm version — a definition change never silently mixes semantics.</summary>
public const int CurrentVersion = 1;
/// <summary>Calendar day (UTC, "yyyy-MM-dd") the sessions STARTED.</summary>
public string Date { get; set; } = string.Empty;
/// <summary>Tenant GUID, or "global" for the cross-tenant row.</summary>
public string TenantId { get; set; } = string.Empty;
public int Version { get; set; } = CurrentVersion;
/// <summary>All sessions started on this date (any status).</summary>
public int SessionCount { get; set; }
/// <summary>Sessions whose status is Succeeded / Failed / Incomplete — the share denominator for terminal paths.</summary>
public int TerminalSessionCount { get; set; }
/// <summary>Ordered by VerdictPath, then Status (ordinal) for deterministic output.</summary>
public List<VerdictCalibrationBucket> Buckets { get; set; } = new List<VerdictCalibrationBucket>();
public DateTime ComputedAt { get; set; }
}
}
@@ -220,14 +220,37 @@ namespace AutopilotMonitor.Shared.Models
public string FailureReason { get; set; } = default!;
/// <summary>
/// Origin of a Failed status. Values:
/// Origin of a Failed status — persisted ONLY on a Failed write (a non-Failed verdict
/// discards it; see <see cref="VerdictPath"/> for the status-independent attribution). Values:
/// - "" / null: agent-reported (default; terminal enrollment_failed event)
/// - "rule:&lt;RuleId&gt;": session failed because an analyze rule with MarkSessionAsFailed fired
/// - "manual": operator flipped the session via the portal
/// - "max_lifetime_watchdog": the agent's max-lifetime shutdown verdict was Failed
/// Consumers use this to render rule-based failures distinctly (badge + link to rule).
/// </summary>
public string FailureSource { get; set; } = string.Empty;
/// <summary>
/// Machine-readable origin of the CURRENT <see cref="Status"/> — which code path wrote it
/// (<see cref="VerdictPaths"/> vocabulary, e.g. <c>agent:complete</c>, <c>sweep:r5_incomplete</c>,
/// <c>manual:failed</c>). Stamped on every status write regardless of status, so the verdict
/// calibration aggregate can count each path. Null on rows written before instrumentation
/// (read-side derivation covers those).
/// </summary>
public string? VerdictPath { get; set; }
/// <summary>
/// The verdict this session carried BEFORE the current status overrode it — set only when a
/// write replaces a prior Succeeded/Failed/Incomplete/AwaitingUser (admin mark, late agent
/// completion upgrading a sweep verdict, retro-reclassification). Together with
/// <see cref="PriorVerdictPath"/> this is the correction stream: "path X was overridden".
/// Null when the status never overrode a verdict.
/// </summary>
public string? PriorStatus { get; set; }
/// <summary>The <see cref="VerdictPath"/> of the overridden prior verdict; see <see cref="PriorStatus"/>.</summary>
public string? PriorVerdictPath { get; set; }
/// <summary>
/// Non-empty only when the BACKEND (not the agent) declared this session Succeeded:
/// either the maintenance timeout sweep reconciled it (e.g. "user completed setup —
@@ -0,0 +1,103 @@
using System;
namespace AutopilotMonitor.Shared.Models
{
/// <summary>
/// Machine-readable vocabulary for <see cref="SessionSummary.VerdictPath"/> — WHICH code path
/// produced the session's current <see cref="SessionStatus"/>. Format is
/// <c>origin:detail</c>, lower-case, stable: values are a persisted contract that the verdict
/// calibration aggregate (docs/backend/verdict-calibration.md) counts over days and tenants, so
/// entries are append-only — never rename or reuse one. Unlike <c>FailureSource</c> (written
/// only on Failed) the path is stamped for every status write, so a sweep "Incomplete", a late
/// agent upgrade and an admin override are all countable.
/// </summary>
public static class VerdictPaths
{
// ── Agent-declared (ingest of the agent's own terminal/phase events) ──────────────────
public const string AgentComplete = "agent:complete";
public const string AgentCompleteSoft = "agent:complete_soft";
public const string AgentFailed = "agent:failed";
public const string AgentEspFailureFallback = "agent:esp_failure_fallback";
public const string AgentGatherComplete = "agent:gather_complete";
public const string AgentWhiteGlovePending = "agent:whiteglove_pending";
public const string AgentWhiteGloveResumed = "agent:whiteglove_resumed";
public const string AgentStallProbe = "agent:stall_probe";
public const string AgentStallHeal = "agent:stall_heal";
// ── Backend-declared, non-classifier ─────────────────────────────────────────────────
public const string IngestWhiteGloveAwaiting = "ingest:wg_awaiting";
public const string SweepStalled = "sweep:stalled";
public const string SweepWhiteGloveAwaiting = "sweep:wg_awaiting";
public const string SweepSelfDeployingReconcile = "sweep:sd_reconcile";
public const string RetroSelfDeployingReconcile = "retro:sd_reconcile";
public const string RetroSuperseded = "retro:superseded";
public const string RegisterSuperseded = "register:superseded";
/// <summary>Status InProgress set by a fresh session registration.</summary>
public const string RegisterNew = "register:new";
/// <summary>WhiteGlove Part-2 re-registration flipped Pending to InProgress.</summary>
public const string RegisterWhiteGloveResume = "register:whiteglove_resume";
public const string ManualFailed = "manual:failed";
public const string ManualSucceeded = "manual:succeeded";
/// <summary>Read-side derivation could not attribute a pre-instrumentation row. Never written.</summary>
public const string LegacyUnknown = "legacy:unknown";
// ── Origins whose detail is a classifier rule id (see <see cref="ClassifierRules"/>) ──
public const string OriginMaxLifetime = "maxlife";
public const string OriginLateReconcile = "late";
public const string OriginSweep = "sweep";
public const string OriginRetro = "retro";
public const string OriginRule = "rule";
/// <summary><c>{origin}:{detail}</c> — e.g. <c>sweep:r5_incomplete</c>, <c>rule:ANALYZE-ESP-001</c>.</summary>
public static string Compose(string origin, string detail)
{
if (string.IsNullOrWhiteSpace(origin)) throw new ArgumentException("origin required", nameof(origin));
if (string.IsNullOrWhiteSpace(detail)) throw new ArgumentException("detail required", nameof(detail));
return origin + ":" + detail;
}
/// <summary>The <c>origin</c> half of a path (<c>"sweep"</c> for <c>"sweep:r6"</c>); the whole string when there is no colon.</summary>
public static string Origin(string path)
{
if (string.IsNullOrEmpty(path)) return string.Empty;
var i = path.IndexOf(':');
return i < 0 ? path : path.Substring(0, i);
}
/// <summary>
/// True for a verdict produced by a silence/timeout classifier RULE (<c>sweep:r5_incomplete</c>,
/// <c>maxlife:r1</c>, <c>late:r4</c>, <c>retro:r6</c>) — the rows whose calibration drives rule
/// tuning. Non-rule verdicts from the same origins (<c>sweep:stalled</c>, <c>retro:superseded</c>)
/// are excluded.
/// </summary>
public static bool IsClassifierPath(string path)
{
if (string.IsNullOrEmpty(path)) return false;
var origin = Origin(path);
if (origin != OriginSweep && origin != OriginMaxLifetime && origin != OriginLateReconcile && origin != OriginRetro)
return false;
var detail = path.Length > origin.Length + 1 ? path.Substring(origin.Length + 1) : string.Empty;
return detail.Length >= 2 && detail[0] == 'r' && char.IsDigit(detail[1]);
}
}
/// <summary>
/// Rule ids of <c>EnrollmentTimeoutClassifier.ClassifyTimedOutSession</c> — the <c>detail</c> half
/// of a classifier verdict path. One id per <c>return</c> so the calibration matrix attributes a
/// verdict to the exact rule, including the grace-dependent forks (1b, 5).
/// </summary>
public static class ClassifierRules
{
public const string R1ExplicitFailure = "r1";
public const string R1bWhiteGloveAwaiting = "r1b_awaiting";
public const string R1bWhiteGloveSucceeded = "r1b_succeeded";
public const string R1cSelfDeploying = "r1c";
public const string R2AccountSetupComplete = "r2";
public const string R3EmergencyBreak = "r3";
public const string R4DesktopHello = "r4";
public const string R5DeviceSetupAwaiting = "r5_awaiting";
public const string R5DeviceSetupIncomplete = "r5_incomplete";
public const string R6Fallthrough = "r6";
}
}
@@ -0,0 +1,130 @@
using System;
namespace AutopilotMonitor.Shared.Security
{
/// <summary>
/// Microsoft device-certificate extension OIDs and the shared ASN.1 decoding for the
/// GUID-bearing ones. Canonical for both the agent (local identity probing) and the
/// backend (client-cert tenant binding) so the two cannot drift apart.
/// </summary>
/// <remarks>
/// Two certificate families carry these, and they are NOT interchangeable:
/// <list type="bullet">
/// <item><description>
/// <b>MS-Organization-Access</b> (Entra/AAD-Join, <c>1.2.840.113556.1.5.284.*</c>) — installed
/// on AAD-join. The agent reads it locally to learn its own TenantId; it is never used for
/// backend authentication.
/// </description></item>
/// <item><description>
/// <b>Microsoft Intune MDM Device CA</b> (<c>1.2.840.113556.5.*</c>) — this is the cert the
/// agent presents as the mTLS client certificate, so it is the only one whose contents the
/// backend can trust as proof of possession.
/// </description></item>
/// </list>
/// Values verified against a real field certificate (issued 2026-05-08) — see
/// <c>Functions.Tests/device-cert-sample.pem</c>. Note that the Intune <b>Account</b> ID
/// (<c>5.6</c>) is a DIFFERENT GUID from the Entra tenant ID (<c>5.14</c>); they must never
/// be compared against each other.
/// </remarks>
public static class MsDeviceCertificateOids
{
/// <summary>Issuer CN of the Intune MDM device certificate used for mTLS client auth.</summary>
public const string IntuneMdmIssuer = "CN=Microsoft Intune MDM Device CA";
/// <summary>
/// Intune MDM device certificate → Entra (Azure AD) TenantId. Encoded as a nested
/// ASN.1 OCTET STRING wrapping a 16-byte little-endian GUID.
/// </summary>
public const string IntuneCertEntraTenantIdOid = "1.2.840.113556.5.14";
/// <summary>
/// Intune MDM device certificate → Intune <b>Account</b> ID. Distinct from the Entra
/// tenant ID — do not use it for tenant comparison.
/// </summary>
public const string IntuneCertAccountIdOid = "1.2.840.113556.5.6";
/// <summary>
/// Intune MDM device certificate → Intune device ID (matches the certificate's Subject CN).
/// Unlike the other two this extension holds the 16 GUID bytes <b>raw</b>, without the
/// nested OCTET STRING wrapper — decode it with <see cref="TryParseGuid"/>, which accepts
/// both shapes.
/// </summary>
public const string IntuneCertDeviceIdOid = "1.2.840.113556.5.4";
/// <summary>MS-Organization-Access (Entra join) certificate → TenantId.</summary>
public const string EntraJoinCertTenantIdOid = "1.2.840.113556.1.5.284.5";
/// <summary>MS-Organization-Access (Entra join) certificate → DeviceId.</summary>
public const string EntraJoinCertDeviceIdOid = "1.2.840.113556.1.5.284.2";
/// <summary>
/// Decodes a 16-byte Microsoft binary GUID out of an X.509 extension value
/// (<c>X509Extension.RawData</c>).
/// <para>
/// Accepts both encodings seen in the wild: the GUID wrapped in a nested ASN.1 OCTET
/// STRING (short form <c>04 10 …</c> and long form <c>04 81 10 …</c>), and the 16 raw
/// bytes with no wrapper at all (Intune OID <c>5.4</c>).
/// </para>
/// The byte order is Microsoft's — little-endian for the first three groups, i.e. exactly
/// what <see cref="Guid(byte[])"/> expects.
/// </summary>
/// <returns><c>true</c> and the decoded GUID, or <c>false</c> on any structural mismatch. Never throws.</returns>
public static bool TryParseGuid(byte[] raw, out Guid value)
{
value = Guid.Empty;
if (raw == null) return false;
// Unwrapped form: exactly the 16 GUID bytes (Intune OID 5.4).
if (raw.Length == 16)
return TryFromBytes(raw, 0, out value);
if (raw.Length < 2) return false;
// ASN.1 OCTET STRING tag.
if (raw[0] != 0x04) return false;
int contentStart;
int contentLen;
if (raw[1] < 0x80)
{
// short-form length
contentLen = raw[1];
contentStart = 2;
}
else
{
// long-form length: lower 7 bits = number of following length bytes.
int numLenBytes = raw[1] & 0x7F;
if (numLenBytes < 1 || numLenBytes > 4 || raw.Length < 2 + numLenBytes) return false;
contentLen = 0;
for (int i = 0; i < numLenBytes; i++)
contentLen = (contentLen << 8) | raw[2 + i];
contentStart = 2 + numLenBytes;
}
if (contentLen != 16 || contentStart + 16 > raw.Length) return false;
return TryFromBytes(raw, contentStart, out value);
}
private static bool TryFromBytes(byte[] source, int offset, out Guid value)
{
value = Guid.Empty;
var guidBytes = new byte[16];
Buffer.BlockCopy(source, offset, guidBytes, 0, 16);
try
{
value = new Guid(guidBytes);
return true;
}
catch
{
return false;
}
}
}
}
@@ -18,6 +18,11 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
"AppHomingFlippedWithEntraRoles",
],
Maintenance: [
// Verdict-calibration drift radar (operator-only classifier diagnostics): a verdict path's
// session share doubled over its 28d baseline, the silence share doubled, or the r6
// fallthrough decides ≥20% of classifier verdicts. Warning-tier, once per episode
// (tracker-deduped). Dual-register per memory feedback_ops_event_types_dual_register.
"VerdictCalibrationDrift",
"MaintenanceCompleted",
"MaintenanceFailed",
// Soft watchdog (Warning): a maintenance run completed but exceeded the 10min threshold and is
@@ -5,11 +5,13 @@ import { type MetricsSectionId } from "../metricsNavSections";
import { SectionAgentMetrics } from "../sections/SectionAgentMetrics";
import { SectionPlatformUsage } from "../sections/SectionPlatformUsage";
import { SectionMcpUsage } from "../sections/SectionMcpUsage";
import { SectionVerdictCalibration } from "../sections/SectionVerdictCalibration";
const SECTION_COMPONENTS: Record<MetricsSectionId, React.ComponentType> = {
"platform-metrics": SectionAgentMetrics,
"usage": SectionPlatformUsage,
"mcp-usage": SectionMcpUsage,
"verdict-calibration": SectionVerdictCalibration,
};
export function SectionClient({ section }: { section: string }) {
@@ -2,6 +2,7 @@ export const METRICS_NAV_SECTIONS = [
{ id: "platform-metrics", label: "Platform Metrics", description: "Agent performance, delivery latency, crash rates, and platform health" },
{ id: "usage", label: "Platform Usage", description: "Platform usage statistics across all tenants" },
{ id: "mcp-usage", label: "MCP Usage", description: "MCP API usage metrics across all users" },
{ id: "verdict-calibration", label: "Verdict Calibration", description: "Which code path produced each session verdict, overrides and re-enrollment — classifier diagnostics" },
] as const;
export type MetricsSectionId = (typeof METRICS_NAV_SECTIONS)[number]["id"];
@@ -0,0 +1,267 @@
"use client";
import { useCallback, useEffect, useState } from "react";
import { useAuth } from "../../../../contexts/AuthContext";
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
import { api } from "@/lib/api";
import { useAggregatedAdminScope } from "@/hooks";
import { TenantScopeSelector } from "@/components/TenantScopeSelector";
import { SessionStatusBadge } from "@/components/SessionStatusBadge";
import { SegmentedControl, TIME_RANGE_OPTIONS } from "@/components/SegmentedControl";
import {
ALERT_KIND_LABELS,
VERDICT_PATH_ORIGIN_LABELS,
alertsByPath,
describeAlert,
groupPathsByOrigin,
trendGlyph,
type VerdictCalibrationAlert,
type VerdictCalibrationPath,
type VerdictCalibrationResponse,
} from "./verdictCalibrationLogic";
type DateRange = "7d" | "30d" | "90d";
const RANGE_DAYS: Record<DateRange, number> = { "7d": 7, "30d": 30, "90d": 90 };
export function SectionVerdictCalibration() {
const { getAccessToken } = useAuth();
const scope = useAggregatedAdminScope({ defaultAggregated: true });
const { selectedTenantId, scopeInitialized } = scope;
const [data, setData] = useState<VerdictCalibrationResponse | null>(null);
const [loading, setLoading] = useState(true);
const [error, setError] = useState<string | null>(null);
const [dateRange, setDateRange] = useState<DateRange>("30d");
const fetchData = useCallback(
async (range: DateRange, tenantId: string) => {
setLoading(true);
setError(null);
try {
const res = await authenticatedFetch(
api.metrics.globalVerdictCalibration(RANGE_DAYS[range], tenantId || undefined),
getAccessToken
);
if (res.status === 404) {
// Deploy skew: backend without the route yet — reads as "no rows", never as an error.
setData(null);
return;
}
if (!res.ok) throw new Error(`Verdict calibration: ${res.status}`);
setData((await res.json()) as VerdictCalibrationResponse);
} catch (err) {
if (err instanceof TokenExpiredError) {
setError("Session expired. Please refresh the page.");
} else {
setError(err instanceof Error ? err.message : "Failed to load verdict calibration");
}
} finally {
setLoading(false);
}
},
[getAccessToken]
);
useEffect(() => {
if (!scopeInitialized) return;
const run = async () => {
await fetchData(dateRange, selectedTenantId);
};
void run();
}, [fetchData, dateRange, selectedTenantId, scopeInitialized]);
const groups = data ? groupPathsByOrigin(data.paths) : [];
const pathAlerts = alertsByPath(data?.alerts ?? []);
return (
<div className="max-w-7xl mx-auto px-4 sm:px-6 lg:px-8 py-8 space-y-6">
<div className="flex flex-col gap-3 sm:flex-row sm:items-center sm:justify-between">
<div>
<h2 className="text-lg font-semibold text-gray-900">Verdict Calibration</h2>
<p className="text-sm text-gray-500">
Which code path produced each session verdict, how often it was overridden, and whether the device came back.
Platform-internal classifier diagnostics — correlations only, never causal claims.
</p>
</div>
<div className="flex flex-wrap items-center gap-3">
<TenantScopeSelector scope={scope} allowAggregated />
<SegmentedControl options={TIME_RANGE_OPTIONS} value={dateRange} onChange={(v) => setDateRange(v as DateRange)} />
<button
onClick={() => fetchData(dateRange, selectedTenantId)}
disabled={loading}
className="px-3 py-1.5 text-sm bg-white border border-gray-300 rounded-md hover:bg-gray-50 disabled:opacity-50"
>
Refresh
</button>
</div>
</div>
{error && (
<div className="bg-red-50 border border-red-200 text-red-700 text-sm rounded-md px-4 py-3">{error}</div>
)}
{data && (
<div className="grid grid-cols-2 md:grid-cols-4 gap-4">
<Stat label="Sessions in window" value={data.totals.sessions.toLocaleString()} hint={`${data.totals.days} day rows · ${data.windowStart} – ${data.windowEnd}`} />
<Stat label="Terminal" value={data.totals.terminal.toLocaleString()} hint="Succeeded / Failed / Incomplete" />
<Stat label="Derived attribution" value={data.totals.derived.toLocaleString()} hint="rows without a stamped path (pre-instrumentation)" />
<Stat
label="Trend basis"
value={`${data.trend.windowSessions.toLocaleString()} / ${data.trend.baselineSessions.toLocaleString()}`}
hint={`last ${data.trend.windowDays}d vs prior ${data.trend.baselineDays}d sessions`}
/>
</div>
)}
{data && data.alerts.length > 0 && (
<div className="bg-white shadow rounded-lg p-6">
<h3 className="text-sm font-semibold text-gray-900">Active drift episodes</h3>
<p className="text-xs text-gray-500 mb-3">
Fired once per episode as a VerdictCalibrationDrift ops event; cleared when the share re-arms. Dimensions are
correlated, not necessarily causal.
</p>
<ul className="divide-y divide-gray-100">
{data.alerts.map((a) => (
<AlertRow key={`${a.kind}|${a.verdictPath}|${a.status}`} alert={a} />
))}
</ul>
</div>
)}
<div className="bg-white shadow rounded-lg p-6">
{loading && !data ? (
<p className="text-sm text-gray-500">Loading…</p>
) : !data || data.paths.length === 0 ? (
<p className="text-sm text-gray-500">
No calibration rows yet. The maintenance sweep writes them every two hours; the first rows appear after its
next pass.
</p>
) : (
<div className="overflow-x-auto">
<table className="min-w-full divide-y divide-gray-200 text-sm">
<thead>
<tr className="text-left text-xs font-medium text-gray-500 uppercase tracking-wider">
<th className="px-3 py-2">Verdict path</th>
<th className="px-3 py-2">Status</th>
<th className="px-3 py-2 text-right">Sessions</th>
<th className="px-3 py-2 text-right">Share</th>
<th className="px-3 py-2 text-right" title="Share in the last 7 days vs the prior 28 days (lift = window share ÷ baseline share)">
7d trend
</th>
<th className="px-3 py-2 text-right" title="Of terminal sessions at least 7 days old: the same device registered another terminal session within 7 days. Blank below 20 eligible sessions.">
Re-enrolled ≤7d
</th>
<th className="px-3 py-2 text-right" title="Sessions that carried this path and were then overridden: by an administrator / by a late agent completion / by another writer">
Overridden (admin / late / other)
</th>
</tr>
</thead>
<tbody className="divide-y divide-gray-100">
{groups.map((group) => (
<GroupRows key={group.origin} origin={group.origin} rows={group.rows} alerts={pathAlerts} />
))}
</tbody>
</table>
</div>
)}
{data?.computedAt && (
<p className="mt-3 text-xs text-gray-400">
Rows computed {new Date(data.computedAt).toLocaleString()} · algorithm version{" "}
{data.versions.join(", ") || "—"}
</p>
)}
</div>
</div>
);
}
function Stat({ label, value, hint }: { label: string; value: string; hint: string }) {
return (
<div className="bg-white shadow rounded-lg p-4">
<p className="text-xs font-medium text-gray-500 uppercase tracking-wider">{label}</p>
<p className="mt-1 text-2xl font-semibold text-gray-900">{value}</p>
<p className="text-xs text-gray-400">{hint}</p>
</div>
);
}
function AlertRow({ alert }: { alert: VerdictCalibrationAlert }) {
return (
<li className="py-2 text-sm">
<span className="inline-flex items-center px-2 py-0.5 rounded text-xs font-medium bg-red-100 text-red-800 mr-2">
↑ {ALERT_KIND_LABELS[alert.kind] ?? alert.kind}
</span>
<span className="font-mono text-xs text-gray-800">{alert.verdictPath}</span>
{alert.status !== "*" && <span className="text-gray-500"> / {alert.status}</span>}
<span className="text-gray-600"> — {describeAlert(alert)}</span>
<span className="text-xs text-gray-400"> · since {new Date(alert.firstNotifiedAt).toLocaleDateString()}</span>
</li>
);
}
function GroupRows({ origin, rows, alerts }: { origin: string; rows: VerdictCalibrationPath[]; alerts: Map<string, VerdictCalibrationAlert> }) {
return (
<>
<tr className="bg-gray-50">
<td colSpan={7} className="px-3 py-1.5 text-xs font-semibold text-gray-600">
{VERDICT_PATH_ORIGIN_LABELS[origin] ?? origin}
</td>
</tr>
{rows.map((row) => (
<PathRow key={`${row.verdictPath}|${row.status}`} row={row} alert={alerts.get(`${row.verdictPath}|${row.status}`)} />
))}
</>
);
}
function PathRow({ row, alert }: { row: VerdictCalibrationPath; alert?: VerdictCalibrationAlert }) {
const trend = trendGlyph(row);
const overridden = row.overriddenByAdmin + row.overriddenByLateCompletion + row.overriddenOther;
return (
<tr>
<td className="px-3 py-2 font-mono text-xs text-gray-800 whitespace-nowrap">
{row.verdictPath}
{alert && (
<span
className="ml-2 inline-flex items-center px-2 py-0.5 rounded text-xs font-medium bg-red-100 text-red-800"
title={describeAlert(alert)}
>
↑ Drift
</span>
)}
{row.derivedCount > 0 && (
<span
className="ml-2 inline-flex items-center px-2 py-0.5 rounded text-xs font-medium bg-gray-100 text-gray-600 border border-gray-200"
title={`${row.derivedCount} of ${row.count} attributed read-side (no stamped path)`}
>
derived {row.derivedCount}
</span>
)}
</td>
<td className="px-3 py-2">
<SessionStatusBadge status={row.status} />
</td>
<td className="px-3 py-2 text-right tabular-nums">{row.count.toLocaleString()}</td>
<td className="px-3 py-2 text-right tabular-nums">{row.sharePct.toFixed(1)}%</td>
<td
className={`px-3 py-2 text-right tabular-nums ${trend.className}`}
title={`last 7d: ${row.window7.count} of ${row.window7.sessions} (${row.window7.sharePct.toFixed(1)}%) · prior 28d: ${row.baseline28.count} of ${row.baseline28.sessions} (${row.baseline28.sharePct.toFixed(1)}%)`}
>
{trend.text}
</td>
<td
className="px-3 py-2 text-right tabular-nums"
title={`${row.reEnrolled7d} of ${row.eligible7d} eligible`}
>
{row.reEnrollRatePct == null ? <span className="text-gray-400">— (n={row.eligible7d})</span> : `${row.reEnrollRatePct.toFixed(1)}%`}
</td>
<td className="px-3 py-2 text-right tabular-nums" title={overridden === 0 ? undefined : `${overridden} overridden in total`}>
{overridden === 0 ? (
<span className="text-gray-400">—</span>
) : (
`${row.overriddenByAdmin} / ${row.overriddenByLateCompletion} / ${row.overriddenOther}`
)}
</td>
</tr>
);
}
@@ -0,0 +1,76 @@
import { describe, expect, it } from "vitest";
import {
VERDICT_PATH_ORIGIN_LABELS,
groupPathsByOrigin,
pathOrigin,
trendGlyph,
type VerdictCalibrationPath,
} from "../verdictCalibrationLogic";
function row(verdictPath: string, overrides: Partial<VerdictCalibrationPath> = {}): VerdictCalibrationPath {
return {
verdictPath,
status: "Succeeded",
count: 10,
sharePct: 10,
derivedCount: 0,
eligible7d: 10,
reEnrolled7d: 1,
reEnrollRatePct: null,
overriddenByAdmin: 0,
overriddenByLateCompletion: 0,
overriddenOther: 0,
window7: { count: 5, sessions: 50, sharePct: 10 },
baseline28: { count: 20, sessions: 200, sharePct: 10 },
lift: 1,
...overrides,
};
}
describe("pathOrigin", () => {
it("splits on the first colon and tolerates rule ids with colons-free detail", () => {
expect(pathOrigin("sweep:r5_incomplete")).toBe("sweep");
expect(pathOrigin("rule:ANALYZE-ESP-001")).toBe("rule");
expect(pathOrigin("legacy:unknown")).toBe("legacy");
expect(pathOrigin("weird")).toBe("weird");
});
});
describe("groupPathsByOrigin", () => {
it("orders groups by the documented origin order, unknown origins last, rows untouched", () => {
const groups = groupPathsByOrigin([
row("zzz:x"),
row("manual:failed"),
row("sweep:r6"),
row("sweep:r5_incomplete"),
row("agent:complete"),
]);
expect(groups.map((g) => g.origin)).toEqual(["agent", "sweep", "manual", "zzz"]);
expect(groups[1].rows.map((r) => r.verdictPath)).toEqual(["sweep:r6", "sweep:r5_incomplete"]);
});
it("has a label for every origin in the order list", () => {
for (const origin of ["agent", "ingest", "sweep", "maxlife", "late", "retro", "register", "rule", "manual", "legacy"]) {
expect(VERDICT_PATH_ORIGIN_LABELS[origin]).toBeTruthy();
}
});
});
describe("trendGlyph", () => {
it("flags a doubled share with an up arrow and a halved share with a down arrow", () => {
expect(trendGlyph(row("sweep:r6", { lift: 2.4 })).text).toBe("↑ 2.4×");
expect(trendGlyph(row("sweep:r6", { lift: 0.4 })).text).toBe("↓ 0.4×");
expect(trendGlyph(row("sweep:r6", { lift: 1.3 })).text).toBe("≈ 1.3×");
});
it("withholds the arrow below five window hits — one session can double a tiny share", () => {
const g = trendGlyph(row("sweep:r6", { lift: 3, window7: { count: 3, sessions: 50, sharePct: 6 } }));
expect(g.text).toBe("≈ 3.0×");
expect(g.className).toContain("text-gray-400");
});
it("never invents a lift without a baseline", () => {
expect(trendGlyph(row("sweep:r6", { lift: null, baseline28: { count: 0, sessions: 200, sharePct: 0 } })).text).toBe("new");
expect(trendGlyph(row("sweep:r6", { lift: null, window7: { count: 0, sessions: 0, sharePct: 0 }, baseline28: { count: 0, sessions: 0, sharePct: 0 } })).text).toBe("—");
});
});
@@ -0,0 +1,163 @@
/**
* Pure helpers for the Verdict Calibration matrix (docs/backend/verdict-calibration.md).
* Wire shape mirrors GetVerdictCalibrationFunction / VerdictCalibrationResponse (camelCase).
*/
export interface TrendWindow {
count: number;
sessions: number;
sharePct: number;
}
export interface VerdictCalibrationPath {
verdictPath: string;
status: string;
count: number;
sharePct: number;
derivedCount: number;
eligible7d: number;
reEnrolled7d: number;
/** null below 20 eligible sessions — never a rate on a handful */
reEnrollRatePct: number | null;
overriddenByAdmin: number;
overriddenByLateCompletion: number;
overriddenOther: number;
window7: TrendWindow;
baseline28: TrendWindow;
/** window share ÷ baseline share; null without a baseline */
lift: number | null;
}
export interface VerdictCalibrationDimension {
dimension: string;
value: string;
hitCount: number;
hitSharePct: number;
allSharePct: number;
lift: number;
}
/** One ACTIVE drift episode (mirrors VerdictCalibrationAlert). */
export interface VerdictCalibrationAlert {
tenantId: string;
kind: "share_regression" | "silence_share_regression" | "evidence_gap" | string;
verdictPath: string;
status: string;
windowHitCount: number;
windowSessionCount: number;
baselineHitCount: number;
baselineSessionCount: number;
windowRatePct: number;
baselineRatePct: number;
lift: number | null;
windowStartDate: string;
windowEndDate: string;
dimension: VerdictCalibrationDimension | null;
firstNotifiedAt: string;
lastEvaluatedAt: string;
}
export const ALERT_KIND_LABELS: Record<string, string> = {
share_regression: "Share regression",
silence_share_regression: "Silence share regression (agent liveness)",
evidence_gap: "Evidence gap (r6 fallthrough)",
};
/** One-line alert summary; the dimension clause keeps the rule-radar wording contract. */
export function describeAlert(a: VerdictCalibrationAlert): string {
const base =
`${a.windowHitCount} of ${a.windowSessionCount} (${a.windowRatePct.toFixed(1)}%) over 7d vs ` +
`${a.baselineRatePct.toFixed(1)}% baseline (${a.baselineHitCount} of ${a.baselineSessionCount} over 28d)` +
(a.lift != null ? ` — lift ${a.lift.toFixed(1)}×` : "");
if (!a.dimension) return base;
return `${base}. ${a.dimension.value} carries ${a.dimension.hitSharePct.toFixed(0)}% of hits vs ${a.dimension.allSharePct.toFixed(0)}% of all sessions — correlated, not necessarily causal.`;
}
/** Index active per-path alerts by "path|status" for the matrix pill. */
export function alertsByPath(alerts: VerdictCalibrationAlert[]): Map<string, VerdictCalibrationAlert> {
const map = new Map<string, VerdictCalibrationAlert>();
for (const a of alerts) {
if (a.kind === "share_regression") map.set(`${a.verdictPath}|${a.status}`, a);
}
return map;
}
export interface VerdictCalibrationResponse {
success: boolean;
tenantId: string;
windowDays: number;
windowStart: string;
windowEnd: string;
computedAt: string | null;
versions: number[];
totals: { sessions: number; terminal: number; derived: number; days: number };
trend: { windowDays: number; baselineDays: number; windowSessions: number; baselineSessions: number };
paths: VerdictCalibrationPath[];
alerts: VerdictCalibrationAlert[];
}
/** The `origin` half of a verdict path (`sweep` for `sweep:r6`); the whole string without a colon. */
export function pathOrigin(path: string): string {
const i = path.indexOf(":");
return i < 0 ? path : path.slice(0, i);
}
/** Display order + labels for the origin groups; unknown origins sort last, alphabetically. */
export const VERDICT_PATH_ORIGIN_LABELS: Record<string, string> = {
agent: "Agent-declared",
ingest: "Ingest mapping",
sweep: "Maintenance sweep (silence classifier)",
maxlife: "Agent max-lifetime shutdown (silence classifier)",
late: "Late telemetry reconcile (silence classifier)",
retro: "Retro reclassification",
register: "Session registration",
rule: "Analyze rule (MarkSessionAsFailed)",
manual: "Administrator",
legacy: "Pre-instrumentation (derived, origin unknown)",
};
const ORIGIN_ORDER = Object.keys(VERDICT_PATH_ORIGIN_LABELS);
export interface OriginGroup {
origin: string;
rows: VerdictCalibrationPath[];
}
/** Groups rows by origin in display order; rows inside a group keep the server order (count desc). */
export function groupPathsByOrigin(paths: VerdictCalibrationPath[]): OriginGroup[] {
const byOrigin = new Map<string, VerdictCalibrationPath[]>();
for (const p of paths) {
const origin = pathOrigin(p.verdictPath);
const list = byOrigin.get(origin);
if (list) list.push(p);
else byOrigin.set(origin, [p]);
}
return Array.from(byOrigin.entries())
.sort(([a], [b]) => {
const ia = ORIGIN_ORDER.indexOf(a);
const ib = ORIGIN_ORDER.indexOf(b);
if (ia !== -1 || ib !== -1) return (ia === -1 ? ORIGIN_ORDER.length : ia) - (ib === -1 ? ORIGIN_ORDER.length : ib);
return a.localeCompare(b);
})
.map(([origin, rows]) => ({ origin, rows }));
}
/**
* 7d-vs-28d trend cell. Lift ≥ 2 (share at least doubled) and ≤ 0.5 (halved) get the arrow +
* color; anything in between is a neutral "≈". No baseline → "new" (a new signal has no finite
* lift — never invented). Below 5 window hits the arrow is withheld: one session can double a
* tiny share.
*/
export function trendGlyph(row: VerdictCalibrationPath): { text: string; className: string } {
if (row.lift == null) {
return row.window7.count > 0 && row.baseline28.sessions > 0
? { text: "new", className: "text-gray-500" }
: { text: "—", className: "text-gray-400" };
}
const lift = row.lift;
const label = `${lift.toFixed(1)}×`;
if (row.window7.count < 5) return { text: `≈ ${label}`, className: "text-gray-400" };
if (lift >= 2) return { text: `↑ ${label}`, className: "text-red-600 font-medium" };
if (lift <= 0.5) return { text: `↓ ${label}`, className: "text-green-700" };
return { text: `≈ ${label}`, className: "text-gray-500" };
}
+4
View File
@@ -292,6 +292,10 @@ export const api = {
`${API_BASE_URL}/api/metrics/device-journeys${qs({ days: String(days) })}`,
globalDeviceJourneys: (days: number, tenantId?: string) =>
`${API_BASE_URL}/api/global/metrics/device-journeys${qs({ days: String(days), tenantId })}`,
// Verdict calibration matrix (operator-only; GlobalReadOrAdmin). Omit tenantId for the
// cross-tenant aggregate. Daily rows are additive, so the window follows days honestly.
globalVerdictCalibration: (days: number, tenantId?: string) =>
`${API_BASE_URL}/api/global/metrics/verdict-calibration${qs({ days: String(days), tenantId })}`,
geographic: (tenantId: string, days: number, groupBy: string) =>
`${API_BASE_URL}/api/metrics/geographic${qs({ tenantId, days: String(days), groupBy })}`,
globalGeographic: (days: number, groupBy: string, tenantId?: string) =>

Some files were not shown because too many files have changed in this diff Show More