mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-25 21:19:38 +00:00
Compare commits
48 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 8e8bb66371 | |||
| 7e1389cf32 | |||
| 3a89319472 | |||
| b37e6f975c | |||
| cacbd24667 | |||
| a157d98a40 | |||
| 7fa50d09c1 | |||
| 94afc02e62 | |||
| 5e4c5e80be | |||
| b535973531 | |||
| 7e0c578a59 | |||
| 1257e0d195 | |||
| 98f8114d90 | |||
| 7b3230d507 | |||
| 87b7a20223 | |||
| 4c19be1de4 | |||
| 93de130cd1 | |||
| b8b95c7e76 | |||
| e57e081cec | |||
| 3a60186824 | |||
| c35595503f | |||
| 838da4cc8a | |||
| a9937e0f5d | |||
| 1dc2cc9a54 | |||
| dd7ae299a8 | |||
| d96d10ccec | |||
| 457fff65f5 | |||
| dc45694611 | |||
| 7569061983 | |||
| 0f5e36db94 | |||
| e1de5a8971 | |||
| a2d93e884c | |||
| 425229deee | |||
| 8675e7ee06 | |||
| e857e99436 | |||
| 6435d5f290 | |||
| 445ce4e75b | |||
| 4e54ffe4b4 | |||
| 65d45a616e | |||
| 4aa1f07fbb | |||
| ede616444a | |||
| 61a359ee54 | |||
| babc89c776 | |||
| a2f448cad0 | |||
| f15e115dea | |||
| 977c5fb447 | |||
| 27fdd19568 | |||
| 72f7adcb3d |
+1
-1
Submodule .claude/commands updated: 2866e07882...a115d243a3
@@ -0,0 +1,147 @@
|
||||
---
|
||||
type: concept
|
||||
title: Diagnostics Package
|
||||
description: What the agent's diagnostics ZIP contains and why — the built-in section catalog shared with the portal, the RealmJoin and Device Preparation collection gates, the configured-path guards, caps, and the package manifest.
|
||||
resource: agent
|
||||
tags: [diagnostics, realmjoin, device-preparation, guardrails, portal]
|
||||
timestamp: 2026-08-23
|
||||
---
|
||||
|
||||
# Schema
|
||||
|
||||
The diagnostics package is the agent's last-resort support artifact: a ZIP built in memory
|
||||
by `DiagnosticsPackageService.BuildArchiveBytes` and uploaded through a short-lived SAS
|
||||
URL at the end of a session, at the WhiteGlove seal (`Always` mode) or on a server-requested
|
||||
on-demand collection ([server actions](../backend/server-actions-on-demand-diagnostics.md)).
|
||||
|
||||
## Archive layout
|
||||
|
||||
```
|
||||
sessioninfo.txt session/tenant/device/outcome, hardware via WMI
|
||||
<built-in sections> catalog order, see below
|
||||
AdditionalLogs/<last folder>/ admin-configured paths (global ∪ tenant)
|
||||
package-manifest.txt every packaging decision (always written)
|
||||
_TRUNCATED.txt only when a cap skipped something
|
||||
```
|
||||
|
||||
## The built-in catalog is the single source of truth
|
||||
|
||||
Everything the package collects **before** any configured path is data, not code paths:
|
||||
`DiagnosticsBuiltInSections.All` in Shared (`Models/Diagnostics/DiagnosticsBuiltInSections.cs`).
|
||||
The agent iterates it in order; the backend serves the same list to the portal
|
||||
(`GET /api/diagnostics/paths`, MemberRead) so what administrators see is exactly what the
|
||||
agent does. There is no second copy to drift.
|
||||
|
||||
| Id | ZIP folder | Source (unexpanded) | Patterns | Recursive | Condition |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| AgentLogs | `AgentLogs` | `%ProgramData%\AutopilotMonitor\Logs` | log patterns | no | Always |
|
||||
| ImeLogs | `ImeLogs` | `%ProgramData%\Microsoft\IntuneManagementExtension\Logs` | log patterns | no | Always |
|
||||
| ImeBootstrapperEventLog | `ImeLogs` | `C:\Windows\System32\winevt\Logs` | `BootstrapperAgentServiceLogProvider.evtx` | no | DevicePreparation |
|
||||
| AgentState | `AgentState` | `%ProgramData%\AutopilotMonitor\State` | log patterns + `*.complete`, `*.marker` | yes | Always |
|
||||
| AgentSpool | `AgentSpool` | `%ProgramData%\AutopilotMonitor\Spool` | `*.jsonl`, `*.json` | no | Always |
|
||||
| AgentMarkers | `AgentMarkers` | `%ProgramData%\AutopilotMonitor` | `*.complete`, `*.marker` | no | Always |
|
||||
| RealmJoinWindows | `RealmJoinLogs/Windows` | `C:\Windows\Logs` | `realmjoin*.log` | no | RealmJoinWatcher |
|
||||
| RealmJoinPackages | `RealmJoinLogs/Windows/RealmJoin` | `C:\Windows\Logs\RealmJoin` | `*.log` | yes | RealmJoinWatcher |
|
||||
| RealmJoinChoco | `RealmJoinLogs/Choco` | `%ProgramData%\RealmJoin\choco\logs` | `*.log` | yes | RealmJoinWatcher |
|
||||
| RealmJoinUserTray | `RealmJoinLogs/User` | `%LOGGED_ON_USER_PROFILE%\AppData\Local\RealmJoin` | `tray*.log` | no | RealmJoinWatcher |
|
||||
| RealmJoinUserLogs | `RealmJoinLogs/User/Logs` | `%LOGGED_ON_USER_PROFILE%\AppData\Local\RealmJoin\Logs` | `*.log` | yes | RealmJoinWatcher |
|
||||
|
||||
"Log patterns" = `*.log *.txt *.json *.jsonl *.etl *.evtx *.xml *.csv *.cab`. The five
|
||||
`Always` sections are the historical layout (PR1-B forensics) and are lock-tested in order;
|
||||
the RealmJoin ZIP layout mirrors the disk (`C:\Windows\Logs\{realmjoin*.log, RealmJoin\…}` →
|
||||
`RealmJoinLogs/Windows/…`, `%LOCALAPPDATA%\RealmJoin\{tray*.log, Logs\…}` →
|
||||
`RealmJoinLogs/User/…`), so a flat section and a tree section never produce the same entry
|
||||
name. Two sections may share a ZIP folder only when they read different source folders
|
||||
(`ImeLogs` + the bootstrapper evtx) — `(ZipFolder, SourceFolder)` pairs are unique.
|
||||
|
||||
### Conditions
|
||||
|
||||
`DiagnosticsSectionCondition` is evaluated by the agent (`IsSectionActive`, pure) against its
|
||||
effective configuration and the enrollment scenario; unknown values fail closed.
|
||||
|
||||
* **RealmJoinWatcher** — the tenant's RealmJoin Watcher toggle
|
||||
(`AnalyzerConfiguration.EnableRealmJoinWatcher`). The package service only sees
|
||||
`AgentConfiguration`, so `RemoteConfigMerger` mirrors that one Analyzers knob onto
|
||||
`AgentConfiguration.EnableRealmJoinWatcher` (the RealmJoin host keeps reading the remote
|
||||
block). Deliberately NOT additionally gated on the compile-time `RealmJoinTrackingEnabled`
|
||||
kill switch: if the live watcher is ever disabled, the logs become more useful, not less.
|
||||
No ConfigVersion bump — no new wire field.
|
||||
* **DevicePreparation** — `EnrollmentRegistryDetector.IsDeterministicDevicePreparation()`,
|
||||
the same deterministic `DevicePreparation\BootstrapperAgent\ExecutionContext` marker every
|
||||
other WDP gate keys on ([WDP scenario gates](wdp-scenario-gates.md)); CloudAssigned*
|
||||
fallback rules count as Classic. The probe is a registry read, evaluated once per build
|
||||
(`SCENARIO: devicePreparation=… realmJoinWatcher=…` in the manifest); a probe failure is
|
||||
treated as Classic and logged — the evtx export is skipped, never the package. The channel
|
||||
is exclusively locked while active, so the existing `wevtutil epl` export path
|
||||
(`ResolveChannelForEvtxFile`, field case 2026-08-17) handles it.
|
||||
|
||||
Inactive sections are never silent: `BUILT-IN SKIPPED (RealmJoin Watcher disabled): <Id>`,
|
||||
`BUILT-IN SKIPPED (not a Device Preparation enrollment): <Id>`,
|
||||
`BUILT-IN SKIPPED (no user session for token): <Id> '<source>'` (the
|
||||
`%LOGGED_ON_USER_PROFILE%` token resolves via `UserProfileResolver` only once an
|
||||
interactive user exists — identical to configured paths). Active ones log
|
||||
`BUILT-IN: <Id> -> folder='…' zip='…' recursive=…` followed by the per-pattern
|
||||
`ADDED / NO MATCH / FOLDER MISSING` lines.
|
||||
|
||||
## Built-ins bypass the configured-path guards — on purpose
|
||||
|
||||
`DiagnosticsPathGuards` ([gather-rule guardrails](../rules/gather-rule-guardrails.md))
|
||||
validates **admin-typed** paths: allowlisted prefixes, wildcard only in the last segment,
|
||||
`C:\Users` blocked except the token's `AppData\Local|Roaming`. The catalog is reviewed code,
|
||||
exactly like the five sections that never went through the guard before — and
|
||||
`%ProgramData%\RealmJoin` is not on the allowlist at all. What does apply to every section:
|
||||
the reparse-point skip (files and directories), the `.evtx` channel export, and the caps
|
||||
(100 MB per file, 500 MB total, 5000 files) tracked globally across sections by
|
||||
`BudgetTracker`, with every skip recorded in the manifest and `_TRUNCATED.txt`.
|
||||
|
||||
## Configured paths: global ∪ tenant
|
||||
|
||||
`GetAgentConfigFunction.MergeDiagnosticsLogPaths` sends global paths (Global Admin,
|
||||
`AdminConfiguration.DiagnosticsGlobalLogPathsJson`) first, then tenant paths
|
||||
(`TenantConfiguration.DiagnosticsLogPathsJson`), dropping blank entries and collapsing
|
||||
duplicates (trimmed, case-insensitive) onto the first occurrence — a path present in both
|
||||
lists used to produce duplicate ZIP entry names. The agent keys each entry's ZIP folder on
|
||||
the last directory name (`AdditionalLogs/<folder>`), so two configured paths ending in a
|
||||
same-named folder share one ZIP folder.
|
||||
|
||||
## Portal surfaces
|
||||
|
||||
* Global Admin → Settings → Diagnostics Log Paths: the built-in catalog (collapsed, read-only,
|
||||
neutral condition pills) above the editable global list; add-row first.
|
||||
* Tenant → Settings → Agent → Diagnostics Package: built-in catalog (header carries the
|
||||
tenant's persisted RealmJoin Watcher state, RealmJoin rows show on/off), the global paths
|
||||
read-only, then the tenant's own entries; every role with a tenant membership can read the
|
||||
catalog (the route is MemberRead, JWT-scoped). One line per entry (`DiagnosticsPathRow`),
|
||||
full values in tooltips.
|
||||
|
||||
## Test seams
|
||||
|
||||
* `DiagnosticsPackageService` internal ctor: the five historical folder overrides map onto
|
||||
catalog ids; `sectionFolderOverrides` covers any other section; `devicePreparationProbe`
|
||||
replaces the registry read (seam callers default to "not WDP", so existing rigs never touch
|
||||
the registry). `UserProfileResolver.SetForTesting(path|null)` / `Reset()` drive the token.
|
||||
* Catalog invariants live in the backend test project (`DiagnosticsBuiltInSectionsTests`),
|
||||
because Shared has no test project.
|
||||
|
||||
# Examples
|
||||
|
||||
A RealmJoin tenant (watcher on) with a signed-in user: `RealmJoinLogs/Windows/realmjoin.log`,
|
||||
`RealmJoinLogs/Windows/RealmJoin/Packages/<pkg>/<timestamp>_install.log`,
|
||||
`RealmJoinLogs/Choco/<pkg>/<timestamp>_install.log`, `RealmJoinLogs/User/tray.log`,
|
||||
`RealmJoinLogs/User/Logs/RjImeHost.log`. The same tenant's package collected before the
|
||||
desktop: the two `User` sections appear as `BUILT-IN SKIPPED (no user session for token)`.
|
||||
A Classic enrollment: `BUILT-IN SKIPPED (not a Device Preparation enrollment):
|
||||
ImeBootstrapperEventLog`. A tenant that had added
|
||||
`%LOGGED_ON_USER_PROFILE%\AppData\Local\RealmJoin\Logs\*.log` manually keeps working — the
|
||||
files simply appear twice (`AdditionalLogs/Logs/…` and `RealmJoinLogs/User/Logs/…`) until the
|
||||
entry is removed.
|
||||
|
||||
# Citations
|
||||
|
||||
* `src/Shared/AutopilotMonitor.Shared/Models/Diagnostics/DiagnosticsBuiltInSections.cs` — the catalog
|
||||
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Runtime/DiagnosticsPackageService.cs` — `BuildArchiveBytes`, `AddBuiltInSection`, `IsSectionActive`, caps, evtx export
|
||||
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Configuration/RemoteConfigMerger.cs` — `EnableRealmJoinWatcher` mirror
|
||||
* `src/Backend/AutopilotMonitor.Functions/Functions/Diagnostics/GetDiagnosticsPathsFunction.cs` — `GET diagnostics/paths`
|
||||
* `src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs` — `MergeDiagnosticsLogPaths`
|
||||
* `src/Web/autopilot-monitor-web/components/diagnostics/` — `DiagnosticsPathRow`, `BuiltInSectionsList`, `builtInSectionDisplay`
|
||||
* Tests: `DiagnosticsPackageServiceTests` (RealmJoin on/off, no user session, bootstrapper evtx, probe failure), `DiagnosticsBuiltInSectionsTests`, `GetDiagnosticsPathsPayloadTests`, `GetAgentConfigFunctionTests.MergeDiagnosticsLogPaths_*`
|
||||
@@ -0,0 +1,113 @@
|
||||
---
|
||||
type: Concept
|
||||
title: DO GroupId Auto-Set — Delivery Optimization peering group from a network fingerprint
|
||||
description: EnableDoGroupIdAutoSet (ConfigVersion 40) makes the agent set the DOGroupId policy value to a deterministic GUID derived from the default gateway's IPv4 address and MAC (RealmJoin-compatible byte layout), so devices on the same local network form one DO peering group during enrollment; foreign DOGroupId/DOGroupIdSource policies always win.
|
||||
resource: /src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Transport/DoGroupIdService.cs
|
||||
tags:
|
||||
- agent
|
||||
- delivery-optimization
|
||||
- network
|
||||
- tenant-configuration
|
||||
timestamp: 2026-08-21T00:00:00+02:00
|
||||
---
|
||||
|
||||
# DO GroupId Auto-Set
|
||||
|
||||
## Schema
|
||||
|
||||
### The problem
|
||||
|
||||
With Delivery Optimization download mode Group (2), peering is scoped by GroupId. When no
|
||||
GroupId is configured, Windows derives it from AD site → authenticated domain SID → Entra
|
||||
tenant ID — for a cloud-native tenant that means **one tenant-wide group**, so "peers on
|
||||
the same network" degrades to "peers anywhere in the tenant" and DO's cloud service has to
|
||||
match them by public IP. A GroupId derived from the local network makes the group exactly
|
||||
"devices behind the same gateway" — including branch offices behind NAT. RealmJoin ships
|
||||
this as its network-fingerprint GroupId; our agent is on the device even earlier (during
|
||||
ESP, when the big app downloads actually happen), so it can seed the same value for
|
||||
Autopilot enrollments.
|
||||
|
||||
### The setting
|
||||
|
||||
`TenantConfiguration.EnableDoGroupIdAutoSet` (`bool?`, default null→false), served as
|
||||
`AgentConfigResponse.EnableDoGroupIdAutoSet` — **ConfigVersion 40**. Portal: Settings →
|
||||
Agent → Settings, toggle "Set Delivery Optimization Group ID", stored via the usual
|
||||
tenant-config roundtrip (`TableConfigRepository` Store+Map). Also shown in the admin
|
||||
tenant-config report. `RemoteConfigMerger` maps the bool 1:1; it survives the offline
|
||||
config cache like every plain feature flag.
|
||||
|
||||
The agent does NOT configure `DODownloadMode` — the fingerprint GroupId only has an
|
||||
effect when the tenant already deploys download mode Group (2) via Intune/GPO.
|
||||
|
||||
### The fingerprint
|
||||
|
||||
`DoGroupIdService.ComputeGroupId(ipv4Bytes, macBytes)` builds a 16-byte GUID:
|
||||
|
||||
```
|
||||
{101,48,6} + {0,0,0} + {ip[1], 0, ip[2], ip[3]} + mac[0..5] → new Guid(byte[16])
|
||||
```
|
||||
|
||||
Gateway discovery reuses the shared `NetworkInterfaceLocator` heuristic (Up, not
|
||||
Loopback/Tunnel, non-0.0.0.0 gateway; IPv4 gateway required), the gateway MAC comes from
|
||||
`ArpNativeMethods` (`iphlpapi!SendARP`). The byte layout — including the IP shuffle and
|
||||
the zero padding — is **RealmJoin-compatible and must never change**: devices managed by
|
||||
RealmJoin on the same network compute the identical GUID and peer with ours. The registry
|
||||
value is written as `Guid.ToString()` (braceless lowercase), identical to RealmJoin's
|
||||
`SetValue(name, Guid)` string form. Deliberate divergence from RealmJoin: **no
|
||||
random-GUID fallback** — a random per-device ID would isolate the device in a one-member
|
||||
group, which is worse than the OS default grouping, so no gateway/no MAC means skip.
|
||||
|
||||
### Ownership + conflict rules (never fight a foreign policy)
|
||||
|
||||
Target: `HKLM\SOFTWARE\Policies\Microsoft\Windows\DeliveryOptimization\DOGroupId`
|
||||
(64-bit registry view — the Policies store is WOW64-redirected; key paths reused from
|
||||
`DoThrottlePolicyReader`). Only this GPO-store value is written; the legacy pre-1607
|
||||
`GroupId` value name is intentionally not.
|
||||
|
||||
Skip conditions (Info event with `skipReason`, nothing written):
|
||||
|
||||
| skipReason | Meaning |
|
||||
|---|---|
|
||||
| `no_ipv4_gateway` | No active interface with an IPv4 default gateway |
|
||||
| `arp_failed` | Gateway MAC could not be resolved via SendARP |
|
||||
| `group_id_source_policy` | `DOGroupIdSource` configured in GPO or MDM store (dynamic group source wins) |
|
||||
| `mdm_policy_present` | `DOGroupId` managed via the Policy CSP store (Intune owns it) |
|
||||
| `foreign_value_present` | Existing GPO value neither matches the computed GUID nor the one we last wrote |
|
||||
|
||||
Ownership is tracked through the `StartupEventGate` fingerprint for the
|
||||
`do_group_id_auto_set` key: `written:<guid>` records the GUID we last wrote, so after a
|
||||
gateway change the agent recognizes its own stale value and updates it — while a value
|
||||
written by anyone else is never overwritten (`DecideAction`: empty→write, equal→no-op,
|
||||
ours→update, else→skip; unparseable existing values count as foreign). Every write is
|
||||
verified by readback.
|
||||
|
||||
### The event
|
||||
|
||||
One `do_group_id_auto_set` event per outcome change (Source `StartupEnvironmentProbes`,
|
||||
gated on the fingerprint — unchanged reruns across reboots stay silent; errors never
|
||||
latch, so each failing start keeps its retry trail). Severity: Info for
|
||||
success/skip, Warning for errors. Data: `groupId`, `previousValue`, `gatewayIp`,
|
||||
`gatewayMac`, `written`, `skipped`, `skipReason`, `success`, `error`.
|
||||
|
||||
## Examples
|
||||
|
||||
First boot on a network with gateway `192.168.178.1` / MAC `00-11-22-33-44-55`:
|
||||
|
||||
```
|
||||
do_group_id_auto_set (Info): Delivery Optimization GroupId set to
|
||||
00063065-0000-00a8-b201-001122334455 (gateway 192.168.178.1)
|
||||
```
|
||||
|
||||
Tenant already deploys a DO GroupId via Intune:
|
||||
|
||||
```
|
||||
do_group_id_auto_set (Info): Delivery Optimization GroupId not set: mdm_policy_present
|
||||
```
|
||||
|
||||
## Citations
|
||||
|
||||
* Service (fingerprint, ownership, write+readback): `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Transport/DoGroupIdService.cs`
|
||||
* ARP interop: `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Interop/ArpNativeMethods.cs`
|
||||
* Probe wiring + event + fingerprint dedup: `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Runtime/StartupEnvironmentProbes.cs`
|
||||
* Backend serve + ConfigVersion: `../../src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs`
|
||||
* DO policy stores (key paths, Registry64 rationale): `../../src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Telemetry/Periodic/DoThrottlePolicyReader.cs`
|
||||
@@ -5,10 +5,12 @@ How the on-device agent works:
|
||||
* [Runtime Overview](overview.md) - lifecycle (install → boot guards → runtime → termination), collector hosts, the single-rail event pipeline to the backend, config & kill switch, on-disk layout.
|
||||
* [Decision Engine](decision-engine.md) - the pure-reducer state machine (DecisionCore): signals, state, effects, the A/B/C completion arms, invariants, threading.
|
||||
* [Logs, Persistence & Crash Recovery](logs-and-persistence.md) - every persisted file and why it exists; signal log vs spool; snapshot/journal; the replay-based recovery flow and quarantine.
|
||||
* [Diagnostics Package](diagnostics-package.md) - what the diagnostics ZIP contains: the built-in section catalog shared with the portal (Shared `DiagnosticsBuiltInSections`, served by GET diagnostics/paths), the RealmJoin Watcher and Device Preparation collection gates, why built-ins bypass the configured-path guards, global ∪ tenant path dedup, caps and the package manifest.
|
||||
* [Hello Wizard Un-Skip](hello-wizard-unskip.md) - why a single "WHfB disabled" policy read never decides completion: the HelloWizardStarted signal (prevention + cure) and the HelloTracker confirmation second read.
|
||||
* [Autopilot ZTD Diagnostics](autopilot-ztd-diagnostics.md) - Windows' own diagnostic surfaces for the profile-download flow: ModernDeployment event IDs (807/815/164/...), the Diagnostics\Autopilot registry key, deployment-service endpoints, and the known-issue error-code map.
|
||||
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated 2800 observations (neutral, flush-time watermark) + ANALYZE-ESP-005 gated on an actually-observed reboot.
|
||||
* [Power-State Watcher](power-state-watcher.md) - live AC↔battery transitions and 50/30/15% downward threshold crossings via a WMI push subscription (no polling): power_state_change semantics (latch/debounce/cap, never armed without a battery) and the single-condition analyze rules ANALYZE-DEV-009/-010 on top.
|
||||
* [DO GroupId Auto-Set](do-group-id-auto-set.md) - the optional Delivery Optimization peering-group seed: deterministic GUID from default gateway IPv4 + MAC (RealmJoin-compatible), written to the DOGroupId policy value with readback, ownership-tracked so foreign Intune/GPO values are never overwritten; skip reasons and the do_group_id_auto_set event.
|
||||
* [System Timeline Watcher](system-timeline-watcher.md) - clock steps (system_clock_changed: oldTime/newTime/timeDeltaMs/process, micro-slew floor) and completed sleep episodes (system_sleep_episode: sleep/hibernate/Modern Standby with authoritative enteredAt/exitedAt) from one System-channel EventLogWatcher with pre-agent backfill; feeds the web clock-set jump cause, the standby duration note, time-attribution SleepSpans and ANALYZE-DEV-012.
|
||||
|
||||
Reading order for newcomers: overview → decision engine → persistence.
|
||||
|
||||
@@ -78,7 +78,8 @@ Durations shown for attempts are the sessions' authoritative `DurationSeconds` v
|
||||
completed journeys, first-time-right count, attempt histogram, junk-serial exclusion
|
||||
count, `JourneyVersion`. A journey buckets on the StartedAt date of its completing
|
||||
success session (the platform's StartedAt-date convention; late terminals converge via
|
||||
the sweep). All counts are **additive**, so a window rate is the ratio of summed daily
|
||||
the sweep, which slices the tick's shared projected window scan — see
|
||||
`time-attribution.md`). All counts are **additive**, so a window rate is the ratio of summed daily
|
||||
rows — there is no rolling-window row (unlike the median-based time-attribution
|
||||
aggregates, which cannot merge). 180-day retention. Rows exist below the ≥20 UI gate
|
||||
on purpose (the UI needs the n for "insufficient data (n=…)").
|
||||
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
type: concept
|
||||
title: Plan Downgrade Policy & the Retention Grace Period
|
||||
description: What happens when a tenant loses Pro (explicit downgrade or trial expiry) — every entitlement gates read-time and reversibly, except retention, which gets a 30-day grace before the Community cap starts hard-deleting; the anchor model, the ops events, and the deliberate non-grace of everything else.
|
||||
resource: backend
|
||||
tags: [plan, edition, downgrade, retention, grace-period, trial, entitlements]
|
||||
timestamp: 2026-08-22
|
||||
---
|
||||
|
||||
# Schema
|
||||
|
||||
## The downgrade model: gate, don't destroy
|
||||
|
||||
Edition is resolved at READ time (`FeatureEntitlementCatalog.ResolveEdition`): Pro ⇔
|
||||
permanent tier `pro`/legacy `enterprise` OR an active trial. A plan mutation is a single
|
||||
field write (`PATCH config/{tenantId}/plan`, GA-only) — there is no downgrade event
|
||||
handler, no cascade. On Pro → Community every entitlement simply gates on the next read
|
||||
(≤ ~5 min config cache):
|
||||
|
||||
* **Unrestricted Mode** — effective value flips false; the GA on-request flag and the
|
||||
tenant toggle survive untouched (inert). Re-upgrade restores it verbatim.
|
||||
* **Delegated (MSP) admin** — scope suppressed at the home-tenant gate
|
||||
(`DelegatedAdminService.ApplyHomeTenantGateAsync`); grant rows / tenant groups /
|
||||
assignments are never touched. Resurrects on re-upgrade.
|
||||
* **OOBE bootstrap** — endpoints 403/404 unless the additive GA per-tenant flag
|
||||
(`BootstrapTokenEnabled`) is set; `BootstrapSessions` rows are kept.
|
||||
* **Rate limits / MCP quota** — pure computed-at-read floors/plan names.
|
||||
|
||||
**Retention is the one exception**: the retention sweep (12-hourly
|
||||
`SessionDeletionMaintenance` → `SessionRetentionFanoutService`) hard-deletes sessions
|
||||
older than `GetEffectiveRetentionDays` = `min(stored DataRetentionDays, edition cap)`.
|
||||
Without protection, a downgrade (or trial expiry) with stored retention 365 would start
|
||||
irreversibly cascade-deleting the 90–365-day band on the next tick.
|
||||
|
||||
## The retention grace period (30 days)
|
||||
|
||||
`FeatureEntitlementCatalog.RetentionDowngradeGraceDays = 30`. For 30 days after losing
|
||||
Pro, `GetEffectiveRetentionDays` keeps clamping to the **Pro** cap (365); only after the
|
||||
grace does the Community cap (90) bite. The stored `DataRetentionDays` is never rewritten.
|
||||
`days <= 0` (GA-only "infinite") short-circuits before any cap — those tenants are never
|
||||
swept, grace or not.
|
||||
|
||||
**Anchor** — "when was Pro lost" (`TenantEntitlementService.GetRetentionGraceEndUtc`,
|
||||
returns null while effectively Pro; grace end = latest anchor + 30d):
|
||||
|
||||
* **Explicit downgrade**: `TenantConfiguration.ProDowngradedUtc` (backend-only field, no
|
||||
agent ConfigVersion impact), maintained exclusively by the plan endpoint's pure mutation
|
||||
core `PlanManagementFunction.ApplyPlanChanges`. Stamped when the EFFECTIVE edition
|
||||
(before vs. after, same `nowUtc`) transitions Pro → Community — this covers both a
|
||||
planTier downgrade and a GA explicitly ending a trial (`trialExpiresUtc: null`, where
|
||||
the trial timestamp disappears and could not anchor). Cleared on any transition back to
|
||||
effectively Pro (PATCH pro, trial grant, self-service trial start), so a re-upgrade also
|
||||
resets the grace clock.
|
||||
* **Trial expiry**: no write happens — the stored `TrialExpiresUtc` itself is the anchor
|
||||
(read-time, like edition resolution). A planTier downgrade under a still-active trial
|
||||
deliberately does NOT stamp: the edition is still Pro, and the later trial expiry
|
||||
anchors the grace then.
|
||||
|
||||
The field is on the patch-endpoint `BaseDeniedFields` and `RevertProtectedFields` lists
|
||||
(a config revert must not time-travel the anchor) and roundtrips the table-serialization
|
||||
contract (Store+Map + roundtrip tests). Legacy rows read null → no anchor → no grace,
|
||||
which is correct: no tenant had been downgraded before the feature existed.
|
||||
|
||||
## Ops visibility (dual-registered event types)
|
||||
|
||||
* `TenantPlanDowngraded` (Warning) — emitted by the PATCH handler on an effective
|
||||
Pro→Community transition; carries `retentionGraceEndsUtc` + stored retention days. The
|
||||
response body also returns `retentionGraceEndsUtc`.
|
||||
* `TenantRetentionGraceExpiring` / `TenantRetentionGraceEnded` (Warning) — emitted by the
|
||||
daily `TrialExpirySweepFunction` (03:30 UTC) with the same stateless window mechanics as
|
||||
the trial events (≤3-day heads-up re-fired daily; ended once via the 24h look-back), and
|
||||
ONLY when data is actually at risk: stored `DataRetentionDays` above the Community cap
|
||||
(0 = infinite ⇒ never). Re-upgraded tenants go silent automatically (grace resolves null).
|
||||
|
||||
All three are registered in the web `OPS_EVENT_TYPES` catalog (OpsAlertRulesSection.tsx)
|
||||
so Telegram alert rules can route them.
|
||||
|
||||
## Deliberate non-goals
|
||||
|
||||
* No grace for Unrestricted/MSP/bootstrap/limits — reversible gates need none.
|
||||
* The retention WRITE cap (`TenantConfigValidation`) does not honor the grace: during the
|
||||
grace a tenant admin cannot SET a value above the Community cap. The grace protects
|
||||
existing data; new writes follow the new plan.
|
||||
* The feature-flags endpoint keeps publishing the plain edition cap
|
||||
(`retentionCapDays`) — it feeds the settings input maximum, which is the write cap.
|
||||
|
||||
# Citations
|
||||
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/TenantEntitlementService.cs` — grace + effective retention
|
||||
* `src/Backend/AutopilotMonitor.Functions/Functions/Config/PlanManagementFunction.cs` — `ApplyPlanChanges` anchor lifecycle
|
||||
* `src/Backend/AutopilotMonitor.Functions/Functions/Maintenance/TrialExpirySweepFunction.cs` — grace event windows
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/Deletion/SessionRetentionFanoutService.cs` — enforcement point
|
||||
* Tests: `TenantEntitlementServiceTests`, `PlanManagementTransitionTests`, `TrialExpirySweepFunctionTests`, `SessionRetentionFanoutServiceTests`, `TenantConfigTableSerializationTests`
|
||||
@@ -0,0 +1,110 @@
|
||||
---
|
||||
type: Concept
|
||||
title: Silent-Session Classification (Sweep, Max-Lifetime, Retro-Reconcile)
|
||||
description: How a session whose agent went silent gets its honest status — the two-stage maintenance sweep (2h Stalled marker, 5h timeout classification), the EnrollmentTimeoutClassifier rule order incl. the WhiteGlove Part-2 and self-deploying gates, which ingest paths share the classifier, and the admin retro-reconcile modes.
|
||||
resource: src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs
|
||||
tags:
|
||||
- backend
|
||||
- maintenance
|
||||
- session-status
|
||||
- classification
|
||||
- whiteglove
|
||||
- self-deploying
|
||||
timestamp: 2026-08-23T18:00:00+02:00
|
||||
---
|
||||
|
||||
# Silent-Session Classification
|
||||
|
||||
A session only ever becomes terminal through evidence. When the agent stops reporting
|
||||
(power-off, reboot without relaunch, Wi-Fi never re-associating, max-lifetime watchdog),
|
||||
the backend must decide the status from what it last saw — it never hard-fails a session
|
||||
for being silent. One pure function holds that decision,
|
||||
`EnrollmentTimeoutClassifier.ClassifyTimedOutSession`, and every silent-session path calls it
|
||||
so a session gets the same verdict regardless of which path noticed the silence first.
|
||||
|
||||
# Schema
|
||||
|
||||
## Inputs (`EspProvisioningRollup` + session flags)
|
||||
|
||||
`ExtractRollup(events)` distills the event stream into facts: Device/Account Setup
|
||||
subcategory counts (incl. the 30s fallback-confirmed messages), `HasExplicitFailure`,
|
||||
`HasEnrollmentComplete` vs. `HasTerminalComplete` (the Part-1 `whiteglove_complete`),
|
||||
`HasAgentEmergencyBreak`, `DesktopArrived`, `HelloResolved`, `HelloPolicyDisabled`,
|
||||
`SkipUserEsp`, RealmJoin detection/resolution. Session-row flags are passed alongside:
|
||||
`IsPreProvisioned` + `ResumedAt` (WhiteGlove Part 2) and `IsSelfDeployingProfile`
|
||||
(registry-confirmed `CloudAssignedOobeConfig 0x20|0x40`, set at registration).
|
||||
|
||||
## Rule order (first match wins)
|
||||
|
||||
| # | Condition | Verdict |
|
||||
| --- | --- | --- |
|
||||
| 1 | `HasExplicitFailure` | Failed |
|
||||
| 1b | `IsWhiteGloveAwaitingUser`: pre-provisioned + resumed + Device Setup done + **no** user evidence since resume + no explicit failure/enrollment_complete | AwaitingUser within grace; Succeeded (honest WG reason) past grace |
|
||||
| 1c | `IsSelfDeployingProvisioned`: self-deploying profile + Device Setup done + no explicit failure | **Succeeded** (reconcile) — no grace, no AwaitingUser: the profile has no user phase |
|
||||
| 2 | Account Setup all-succeeded, or `enrollment_complete`, or `whiteglove_complete` on a **non-resumed** session | Succeeded |
|
||||
| 3 | `HasAgentEmergencyBreak` | Incomplete immediately (agent self-destructed, nothing can arrive) |
|
||||
| 4 | `DesktopArrived` + (Hello terminal, or Hello disabled + SkipUserEsp) | Succeeded (user provably finished) |
|
||||
| 5 | Device Setup done | AwaitingUser within grace, Incomplete past grace |
|
||||
| 6 | otherwise | Incomplete |
|
||||
|
||||
Grace = `ResolveGraceHours(SessionGraceHours, AbsoluteMaxSessionHours)` (~51h default),
|
||||
measured from `ResumedAt ?? StartedAt`. Every Succeeded-reconcile reason carries the
|
||||
silence-transparency clause (`AppendReconcileTiming`: last agent contact, silence, verdict time).
|
||||
|
||||
## Why rule 1c exists (kiosk tenant audit 2026-08-23)
|
||||
|
||||
On a self-deploying profile Device ESP all-succeeded IS the enrollment's end; the agent's
|
||||
own SelfDeploying terminal only adds a 5-min confirmation window
|
||||
(`DecisionEngine.SelfDeploying.s_deviceOnlyEspDetectionWindow`). Tenant `aebdce78` (school,
|
||||
Wi-Fi ThinkBooks, a language-change app forcing a reboot right after Device ESP) lost that
|
||||
window on 836 of ~970 sessions in one week: the agent died 0–4 min after
|
||||
`DeviceSetup fallback_confirmed`, the reboot relaunch never registered (30s registration
|
||||
budget on a Wi-Fi link still associating), and rule 5 labeled finished kiosks
|
||||
"awaiting user / Account Setup 0/5" → Incomplete 51h later. Nobody signs in on these devices
|
||||
(prepared during the holidays, boxed, in service) — AwaitingUser was factually wrong and
|
||||
Incomplete turned a working rollout into a red wall. Other self-deploying tenants completed
|
||||
normally in the same window, so this is a classification gap, not a profile-wide agent bug.
|
||||
Agent-side companion fix: `SessionRegistrationHelper` now waits up to 15s for a network link
|
||||
and retries 6× (~62s backoff) instead of 5× (~30s).
|
||||
|
||||
## Callers (all pass both gate inputs)
|
||||
|
||||
| Path | File | Notes |
|
||||
| --- | --- | --- |
|
||||
| Sweep stage 1 (2h agent-silent marker) | `MaintenanceService.MarkStalledSessionsAsTimedOutAsync` | Default: InProgress → Stalled. Gates run BEFORE the Stalled write: `TryMarkWhiteGloveAwaitingUserAsync` (AwaitingUser) and `TryReconcileSelfDeployingAsync` (Succeeded). Only flagged sessions pay the event read. |
|
||||
| Sweep stage 2 (SessionTimeoutHours, 5h default) | same | InProgress/Stalled/AwaitingUser past the window and past the 2h silence cutoff → full classification; AwaitingUser within grace is skipped without an event read. |
|
||||
| Agent max-lifetime (`enrollment_failed`/max_lifetime, `agent_shutting_down`/max_lifetime) | `EventIngestProcessor.ApplyMaxLifetimeVerdictAsync` | Watchdog = "agent stopped waiting", not a verdict; classified from evidence. Pending sessions exempt. |
|
||||
| Late-telemetry reconcile | `EventIngestProcessor.TryLateTelemetryReconcileAsync` | Heal-only: applies a Succeeded verdict to Incomplete/AwaitingUser rows when straggler events arrive. |
|
||||
| Agent stall probe (`session_stalled`) | `EventIngestProcessor.TryMapStallToWhiteGloveAwaitingUserAsync` | Only the WG gate (1b); a stall probe on a self-deploying device with Device Setup done means the agent is alive and something else blocks — stays Stalled. |
|
||||
| Admin retro-reconcile | `LegacyReclassificationService` | Modes `legacy_timeouts`, `pending_orphans`, `self_deploying_silent` (Incomplete/AwaitingUser/Stalled rows with the profile flag, heal-only to Succeeded, admin-marked rows skipped). `POST /api/maintenance/reclassify-legacy?mode=…&dryRun=false&tenantId=…&maxSessions=…`, dry-run by default. |
|
||||
|
||||
## Known gap (decided, not built)
|
||||
|
||||
After max-lifetime the agent writes `enrollment-complete.marker` and self-destructs, so an
|
||||
AwaitingUser verdict from that path can never heal — it is a 51h countdown to Incomplete.
|
||||
The emergency-break path (rule 3) already terminalizes immediately; max-lifetime still goes
|
||||
through the grace window by design ("user simply absent"). Revisit if customers read
|
||||
AwaitingUser as "still possible".
|
||||
|
||||
# Examples
|
||||
|
||||
* Session `195593e2` (tenant aebdce78): `DeviceSetup 4/4` + `fallback_confirmed`, `AccountSetup 0/5`
|
||||
(IME false positive), Hello disabled, SkipUser=True, silent after 13:01 → rule 1c →
|
||||
Succeeded "self-deploying profile — Device Setup fully provisioned … agent went silent
|
||||
before confirming completion. Agent last reported 2026-08-21 13:01 UTC …".
|
||||
* Same event shape on a user-driven profile → rule 5 → AwaitingUser (within grace) /
|
||||
Incomplete (past grace). Guarded by
|
||||
`Classify_same_stream_without_self_deploying_flag_keeps_user_phase_verdicts`.
|
||||
* fairstone WG Part 2 reseal-reboot, technician powers off at logon screen → rule 1b →
|
||||
AwaitingUser; user unboxes days later → ingest completion heals to Succeeded.
|
||||
|
||||
# Citations
|
||||
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs` — rollup + rule order, `IsWhiteGloveAwaitingUser`, `IsSelfDeployingProvisioned`, reason helpers.
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs` — `MarkStalledSessionsAsTimedOutAsync`, `TryMarkWhiteGloveAwaitingUserAsync`, `TryReconcileSelfDeployingAsync`.
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs` — max-lifetime verdict, late-telemetry reconcile, stall mapping.
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/LegacyReclassificationService.cs`, `Functions/Admin/ReclassifyLegacySessionsFunction.cs` — retro modes.
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Maintenance.cs` — `GetAgentSilentSessionsAsync`, `GetStalledSessionsAsync`, `GetSelfDeployingSilentSessionsAsync`.
|
||||
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Security/SessionRegistrationHelper.cs` — link wait + 6-attempt registration budget.
|
||||
* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.SelfDeploying.cs` — the agent-side 5-min SelfDeploying terminal window.
|
||||
* `src/Backend/AutopilotMonitor.Functions.Tests/EnrollmentTimeoutClassifierTests.cs` — rule coverage incl. the 2026-08-23 self-deploying block.
|
||||
@@ -0,0 +1,67 @@
|
||||
---
|
||||
type: Concept
|
||||
title: Table Schema Sentinel (startup table initialization)
|
||||
description: Why a cold start no longer issues one CreateTableIfNotExists per registered table - a hash over TableNames.All stored in AdminConfiguration gates the full pass, the SessionsIndex backfill only runs after a full pass and behind a conditional-insert claim, and daily maintenance repairs out-of-band deletions.
|
||||
resource: /src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.cs
|
||||
tags:
|
||||
- backend
|
||||
- table-storage
|
||||
- startup
|
||||
- maintenance
|
||||
timestamp: 2026-08-23
|
||||
---
|
||||
|
||||
# Schema
|
||||
|
||||
## Problem
|
||||
|
||||
`TableInitializerService` (hosted service) called `InitializeTablesAsync` on every process start, which issued `CreateTableIfNotExistsAsync` for every entry of `Constants.TableNames.All` (66 at the time of writing, DoP 8). Each call is a Create POST whose 409 is swallowed - on Flex Consumption with frequent cold starts that was ~9 serial round-trip waves before the first request. It was followed by a SessionsIndex emptiness query whose *error* path returned "empty", so a transient storage error triggered a full Sessions scan with serial upserts - on every scaled-out instance at once.
|
||||
|
||||
## Sentinel
|
||||
|
||||
* Row `AdminConfiguration` / PK `SchemaSentinel` / RK `tables`, property `TableSchemaHash` (+ `TableCount`, `UpdatedUtc`).
|
||||
* Value = `ComputeTableSchemaHash(TableNames.All)`: SHA-256 over the ordinal-sorted, newline-joined table names. Derived, never hand-maintained: adding a table to `TableNames.All` changes the hash by construction (`TableLifecycleBucketTests` already forces every constant into `All`).
|
||||
* `InitializeTablesAsync` (returns `bool fullPassRan`):
|
||||
1. Point-read the sentinel. Match -> mark initialized, **zero** table calls, return `false`.
|
||||
2. Missing / stale / unreadable (fresh storage has no AdminConfiguration table yet) -> `EnsureAllTablesAsync` (the old full pass), then upsert the sentinel **only if no table failed**, return `true`. A failed pass leaves `_tablesInitialized = false`, so the next call retries.
|
||||
* Scale-out: two instances that both see a stale sentinel both run the idempotent pass - harmless.
|
||||
|
||||
## SessionsIndex backfill
|
||||
|
||||
* Only evaluated when the full pass ran (`fullPassRan == true`): an empty index can only exist on fresh storage, which is exactly the case without a sentinel.
|
||||
* `IsSessionIndexEmptyAsync` now returns `false` on error (skip), not `true`.
|
||||
* `TryClaimSessionIndexBackfillAsync`: conditional `AddEntityAsync` on `SchemaSentinel` / `sessionIndexBackfill` - only the winner scans; 409 or any error -> skip. The manual maintenance backfill (`RunManualAsync`) remains the safety net.
|
||||
|
||||
## Known gap and repair
|
||||
|
||||
A table deleted out-of-band (cleanup skill, portal, tests) is not noticed at startup anymore. `IStorageInitializer.EnsureAllAsync` runs as the first step of the daily timer maintenance (`MaintenanceService.RunAllAsync`), recreating it within 24h. Storage helpers are fail-soft in the meantime.
|
||||
|
||||
## Measuring it
|
||||
|
||||
`StartupTelemetryService` (hosted service) emits two App Insights metrics on `IHostApplicationLifetime.ApplicationStarted` - i.e. after every hosted service finished `StartAsync`:
|
||||
|
||||
* `BackendStartupMs` - process start (`Process.StartTime`) to host ready; includes runtime, DI and all hosted services.
|
||||
* `BackendTableInitMs` - the `InitializeTablesAsync` slice alone (`TableStorageService.LastInitialization`).
|
||||
|
||||
Dimensions: `version`, `commit`, `tableInitFullPass`, `instance`. Metrics bypass adaptive sampling and the worker log level, so they arrive even though Information traces do not. The MCP server logs the same idea as one `{"type":"boot","readyMs":…}` stderr line per start (ContainerAppConsoleLogs_CL, see `.claude/commands/mcp-telemetry.md`).
|
||||
|
||||
```kusto
|
||||
customMetrics
|
||||
| where name in ("BackendStartupMs", "BackendTableInitMs")
|
||||
| summarize p50 = percentile(value, 50), p95 = percentile(value, 95), n = count()
|
||||
by name, version = tostring(customDimensions.version), fullPass = tostring(customDimensions.tableInitFullPass)
|
||||
```
|
||||
|
||||
# Examples
|
||||
|
||||
* Deploy that adds a table: first cold start after the deploy runs the full pass once (log: "Table schema sentinel missing or stale"), every later start logs "Table schema sentinel matches (...)".
|
||||
* Fresh storage account: sentinel read throws TableNotFound -> full pass -> index empty -> one instance claims and backfills.
|
||||
|
||||
# Citations
|
||||
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.cs` - sentinel, `InitializeTablesAsync`, `EnsureAllTablesAsync`, `TryClaimSessionIndexBackfillAsync`
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/TableInitializerService.cs` - startup gating
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs` - daily `EnsureAllTablesAsync`
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/StartupTelemetryService.cs` - `BackendStartupMs` / `BackendTableInitMs`
|
||||
* `src/McpServer/autopilot-monitor-mcp/src/index.ts` - `{"type":"boot"}` line
|
||||
* `src/Backend/AutopilotMonitor.Functions.Tests/TableSchemaSentinelTests.cs`, `StartupTelemetryServiceTests.cs`
|
||||
@@ -4,7 +4,7 @@ title: Tenant Activation & the Once-Only Welcome Mail
|
||||
description: How a signup becomes an activated tenant (manual approve or auto-approve queue), and how the welcome mail is sent exactly once even though activation and the user-provided address race each other.
|
||||
resource: backend
|
||||
tags: [activation, auto-approve, welcome-email, preview-whitelist, race]
|
||||
timestamp: 2026-08-02
|
||||
timestamp: 2026-08-23
|
||||
---
|
||||
|
||||
# Schema
|
||||
@@ -58,6 +58,46 @@ The Global-Admin resend endpoint (`POST /api/preview/send-welcome-email/{tenantI
|
||||
always sends (explicit intent) and consumes the marker best-effort so the automatic
|
||||
paths never duplicate afterwards.
|
||||
|
||||
## Mail transport
|
||||
|
||||
Both transactional mails (welcome, post-offboarding farewell) go through the
|
||||
provider-neutral `EmailService` (`IEmailService` for the welcome path,
|
||||
`IOffboardFarewellEmailSender` for the farewell path). The provider is an implementation
|
||||
detail confined to one private transport method plus the `Email:*` configuration
|
||||
section — a provider swap must not rename classes, DI registrations or callers.
|
||||
|
||||
* Current provider: **Mailchimp Transactional (Mandrill)**, `POST {Email:Endpoint}`
|
||||
(`messages/send`), API key in the JSON body, typed `HttpClient` with the shared
|
||||
`ResiliencePolicies.Notification` retry policy (15 s timeout).
|
||||
* Settings (App Settings use `Email__Key`): `Email:ApiKey` (required — empty means every
|
||||
send is a logged no-op, never an error), `Email:Endpoint`, `Email:FromAddress`
|
||||
(default `noreply@autopilotmonitor.com`), `Email:FromName` (default `Autopilot Monitor`).
|
||||
* Result handling: per-recipient status `sent`/`queued`/`scheduled` = success;
|
||||
`rejected`/`invalid` (with `reject_reason`), non-2xx or malformed bodies = warning.
|
||||
Never throws — both callers rely on that fail-soft contract.
|
||||
* `track_opens`/`track_clicks` are explicitly `false` and `auto_text` is `true`: the provider
|
||||
receives only the recipient address and the tenant domain, which is exactly the claim on
|
||||
the privacy page and the trust data-flows page. Changing tracking = changing those pages.
|
||||
|
||||
## Template overrides (operator-editable)
|
||||
|
||||
Global Admins can replace either HTML body without a deployment (portal: Settings →
|
||||
Email Templates). `EmailTemplateService` resolves the body for every send: the stored
|
||||
override wins, otherwise the built-in `EmailTemplates.*`. Overrides live in the
|
||||
`PreviewConfig` table (partition `EmailTemplates`, row key `welcome` | `farewell`,
|
||||
properties `Html`, `UpdatedBy`, `UpdatedUtc`), cached 5 minutes, invalidated on save/reset.
|
||||
|
||||
* The only placeholder is `{{domainName}}`; an empty domain renders as "your organization",
|
||||
matching the built-ins. Subjects are not editable (pinned constants).
|
||||
* Limit 30 000 characters (Table string property cap is 32K UTF-16 chars).
|
||||
* The send path is fail-soft: a storage error falls back to the built-in body. The admin
|
||||
read path is not — a broken store surfaces as an error instead of silently showing the
|
||||
built-in.
|
||||
* Endpoints (all GlobalAdminOnly): `GET|PUT|DELETE /api/global/email-templates/{kind}` and
|
||||
`POST /api/global/email-templates/{kind}/test`. The test send goes to the caller's tenant
|
||||
contact address (activation notification email → `ContactEmail` → caller UPN) with the
|
||||
caller's tenant domain, optionally with an unsaved draft body; provider tag `test`.
|
||||
|
||||
# Examples
|
||||
|
||||
* Auto-approve wins the race: approve at T+64 s finds no address → logs "deferred to the
|
||||
@@ -73,3 +113,8 @@ paths never duplicate afterwards.
|
||||
* `src/Backend/AutopilotMonitor.Functions/Functions/Rules/PreviewWhitelistFunction.cs` — save-path trigger, GA resend
|
||||
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableConfigRepository.cs` — conditional inserts (approved / welcome-email-sent)
|
||||
* `src/Backend/AutopilotMonitor.Functions.Tests/TenantApprovalServiceTests.cs` — ordering + dedup pins
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/EmailService.cs` — provider-neutral transport (Mandrill wire format, result handling, tracking off)
|
||||
* `src/Backend/AutopilotMonitor.Functions.Tests/EmailServiceTests.cs` — wire-format, fail-soft and gate-ordering pins
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/EmailTemplateService.cs` — override resolution, placeholder rendering, cache
|
||||
* `src/Backend/AutopilotMonitor.Functions/Functions/Admin/EmailTemplatesFunction.cs` — GA endpoints (read / override / reset / test send)
|
||||
* `src/Web/autopilot-monitor-web/app/admin/settings/sections/SectionEmailTemplates.tsx` — portal editor, preview frame, test send
|
||||
|
||||
@@ -105,6 +105,11 @@ the partition).
|
||||
* Self-healing 30-day maintenance sweep (2h tick): backfills missing breakdowns, folds
|
||||
late-terminating sessions into their StartedAt-date aggregates, and enforces version
|
||||
purity — a stale-version row whose events aged out counts as missing, never mixed.
|
||||
Reads the tick's **shared window scan** (`MaintenanceService.LoadSweepWindowSessionsAsync`,
|
||||
one projected 35-day cross-tenant drain via `MaintenanceSweepSessionProjection` that
|
||||
also feeds the device-journey and verdict-calibration sweeps) — the StartedAt-only
|
||||
filter is a full-table read in Table Storage, so it is done once per tick, not once per
|
||||
sweep; `MaintenanceSweepProjectionEquivalenceTests` pins the column set.
|
||||
* `TimeAttributionAggregates` (PK = TenantId or `global`, RK = `{yyyy-MM-dd}|{class}` and
|
||||
`rolling30|{class}`): per enrollment class (`user_driven`, `whiteglove`, `self_deploying`,
|
||||
`device_preparation` — never mixed) median/p75/p90 per segment over the fixed six-segment
|
||||
|
||||
@@ -0,0 +1,164 @@
|
||||
---
|
||||
type: Concept
|
||||
title: Verdict Calibration (classifier thermometer)
|
||||
description: How every session status write declares its VerdictPath, how overriding a verdict preserves it as PriorStatus/PriorVerdictPath, how the daily calibration aggregate counts paths with the re-enrollment proxy and the correction stream, and how the drift radar (share / silence / evidence-gap) turns that into operator-only ops events.
|
||||
resource: src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.VerdictCalibration.cs
|
||||
tags:
|
||||
- backend
|
||||
- sessions
|
||||
- classifier
|
||||
- calibration
|
||||
- regression-radar
|
||||
timestamp: 2026-08-23T18:00:00+02:00
|
||||
---
|
||||
|
||||
# Verdict Calibration (classifier thermometer)
|
||||
|
||||
A session's terminal status is written by ~20 code paths: the agent's own terminal events,
|
||||
the silence classifier (`EnrollmentTimeoutClassifier`, rules 1–6) reached via the
|
||||
maintenance sweep / the agent's max-lifetime shutdown / late-telemetry heal / retro
|
||||
reclassification, the analyze-rule engine, admin marks and session superseding. Verdict
|
||||
calibration makes every one of those paths **countable**, tracks how often a path's verdict
|
||||
was later **overridden**, attaches the delayed **re-enrollment proxy**, and raises an
|
||||
operator-only alarm when a path's share **drifts**. It is deterministic, fact-based tuning
|
||||
input for the rules — explicitly not a learned model, and never a customer-facing feature.
|
||||
|
||||
# Schema
|
||||
|
||||
## VerdictPath on the session row
|
||||
|
||||
`SessionSummary.VerdictPath` (`VerdictPaths` vocabulary, `origin:detail`, append-only) is
|
||||
stamped by **every** status write — `UpdateSessionStatusAsync` takes it as a required
|
||||
parameter, `StoreSessionAsync` stamps `register:new` / `register:whiteglove_resume` and
|
||||
preserves an existing path through the re-registration Replace, `SetSessionPreProvisionedAsync`
|
||||
requires it whenever it writes a status. Unlike `FailureSource` (persisted only on `Failed`),
|
||||
the path survives for Incomplete, AwaitingUser, Stalled and Pending.
|
||||
|
||||
| Origin | Paths |
|
||||
| --- | --- |
|
||||
| `agent` | `complete`, `complete_soft` (Continue-Anyway), `failed`, `esp_failure_fallback`, `gather_complete`, `whiteglove_pending`, `whiteglove_resumed`, `stall_probe`, `stall_heal` |
|
||||
| `ingest` | `wg_awaiting` (stall probe mapped to WhiteGlove AwaitingUser) |
|
||||
| `sweep` | `stalled`, `wg_awaiting`, `sd_reconcile`, `r{rule}` (stage-2 classifier) |
|
||||
| `maxlife` / `late` / `retro` | `r{rule}` (classifier via max-lifetime shutdown / late-telemetry heal / retro reclassification); `retro:sd_reconcile`, `retro:superseded` |
|
||||
| `register` | `new`, `whiteglove_resume`, `superseded` |
|
||||
| `rule` | `rule:{ruleId}` (analyze rule with MarkSessionAsFailed) |
|
||||
| `manual` | `failed`, `succeeded` |
|
||||
| `legacy` | read-side derivation only (`legacy:{rule}` when the rule is recoverable from the reason literal, `legacy:unknown` otherwise) — never written |
|
||||
|
||||
Classifier rule ids (`ClassifierRules`): `r1`, `r1b_awaiting`, `r1b_succeeded`, `r1c`, `r2`,
|
||||
`r3`, `r4`, `r5_awaiting`, `r5_incomplete`, `r6`. `ClassifyTimedOutSession` returns
|
||||
`(Status, Reason, Rule)` so each `return` names its rule.
|
||||
|
||||
## PriorStatus / PriorVerdictPath — the correction stream
|
||||
|
||||
When a write replaces a prior **verdict** (existing status ∈ Succeeded/Failed/Incomplete/
|
||||
AwaitingUser with a stamped path, incoming status different), the storage seam preserves the
|
||||
pair (`TableStorageService.ComputePriorVerdict`, both the ETag path and the force path). A
|
||||
stall marker healing to InProgress, a same-status refresh, or an unstamped legacy row never
|
||||
sets it. Admin marks, late agent completions and retro reclassifications all flow through it.
|
||||
|
||||
## Read-side derivation for pre-instrumentation rows
|
||||
|
||||
`VerdictPathDerivation.Derive(session)` returns the stamped path verbatim, otherwise derives
|
||||
from `AdminMarkedAction`, `FailureSource`, `ReconcileReason` and `FailureReason` using the
|
||||
writers' fixed literals (classifier rule prefixes, the `max-lifetime watchdog` / `Retro-
|
||||
reclassified` suffixes, `Rule: `, `Superseded by session`, `Late completion report received`).
|
||||
Unambiguous rows get the real path (`agent:complete`, `manual:failed`, `rule:…`,
|
||||
`maxlife:r6`), ambiguous-but-rule-known rows `legacy:{rule}`, the rest `legacy:unknown`. No
|
||||
mass backfill: the aggregate counts derived rows with `DerivedCount` so the matrix shows
|
||||
history while flagging its weaker evidence. Tests pin the literals against the classifier
|
||||
output so a wording change fails loudly.
|
||||
|
||||
## Daily aggregate — `VerdictCalibrationAggregates`
|
||||
|
||||
PK = `TenantId` | `"global"`, RK = `yyyy-MM-dd` (session **StartedAt** date, like every daily
|
||||
aggregate). `VerdictCalibrationDailyAggregate { Version, SessionCount, TerminalSessionCount,
|
||||
Buckets[], ComputedAt }` with one bucket per (VerdictPath, Status):
|
||||
|
||||
| Field | Meaning |
|
||||
| --- | --- |
|
||||
| `Count` / `DerivedCount` | sessions currently on the path; of those, derived read-side |
|
||||
| `Eligible7d` | terminal sessions whose end (`CompletedAt ?? LastEventAt ?? StartedAt`) lies ≥ 7 d before compute time |
|
||||
| `ReEnrolled7d` | of the eligible: the device's `DeviceHistories` chain shows the next terminal session starting < 7 d after this one's end |
|
||||
| `OverriddenByAdmin` / `OverriddenByLateCompletion` / `OverriddenOther` | sessions whose **PriorVerdictPath** is this bucket — attributed to the path that was overridden, so a bucket can carry overrides with `Count = 0` |
|
||||
|
||||
`MaintenanceService.SweepVerdictCalibrationAsync` (2-hourly, after the device-journey sweep
|
||||
so the chains are fresh; manual maintenance parity; kill switch
|
||||
`VerdictCalibrationSweepDisabled`) slices the tick's shared projected window scan
|
||||
(`LoadSweepWindowSessionsAsync` — one 35-day cross-tenant drain feeds the time-attribution,
|
||||
device-journey and calibration sweeps plus the radar's tenant discovery; the StartedAt-only
|
||||
filter is a full-table read, so never per sweep), recomputes the rolling 30-day window whole (Replace),
|
||||
deletes stale date rows it did not regenerate, and the unbounded-tables cleanup applies the
|
||||
180-day retention. Sessions inside a deletion cascade are skipped; the table is regenerable
|
||||
and is wiped with the tenant on offboarding.
|
||||
|
||||
## Read surfaces
|
||||
|
||||
* `GET global/metrics/verdict-calibration?days=&tenantId=` (GlobalReadOrAdmin,
|
||||
TenantScoping.QueryParam; `VerdictCalibrationResponse.Build`): per path — count, share of
|
||||
all window sessions, derived count, eligible/re-enrolled with `reEnrollRatePct` **null below
|
||||
20 eligible**, the three override counters, a today-anchored 7d-vs-28d share trend
|
||||
(`window7`, `baseline28`, `lift` null without baseline) — plus totals, the trend basis and
|
||||
the active `alerts[]`. Empty but well-formed before the first sweep.
|
||||
* MCP `get_verdict_calibration` (platform scope: GA + Global Reader). Same endpoint, LLM-shaped in
|
||||
`verdict-calibration-shape.ts`: per-row `window7.sessions` / `baseline28.sessions` are dropped (the
|
||||
denominators live once in `trend`), `reEnrollRatePct` / `lift` are always present (`null` = withheld),
|
||||
and `minSharePct` / `top` trim the one-session tail — rows carrying overrides are never trimmed and
|
||||
everything dropped is reported in `omitted`. The wire shape the admin page consumes is untouched.
|
||||
* Portal: Global Admin → Metrics → Verdict Calibration (`/admin/metrics/verdict-calibration`):
|
||||
matrix grouped by origin, status pill, `derived n` gray pill, trend glyph (arrow only from
|
||||
5 window hits and lift ≥ 2 / ≤ 0.5), re-enrollment rate or `— (n=…)`, overrides
|
||||
`admin / late / other`, `↑ Drift` pill on rows with an active episode, and the active
|
||||
episodes list.
|
||||
|
||||
## Drift radar — `VerdictCalibrationRadar`
|
||||
|
||||
Same statistics as the [rule regression radar](rule-regression-radar.md) (7-day window vs the
|
||||
prior 28 days, ≥ 5 window hits, ≥ 20 window sessions, lift ≥ 2, `MetricsMath.
|
||||
RateIncreaseSeparated`), anchored on yesterday, over every partition with rows (each tenant
|
||||
and `global`):
|
||||
|
||||
| Kind | Signal | Re-arm |
|
||||
| --- | --- | --- |
|
||||
| `share_regression` | one **backend-decided** path's share of all sessions doubled (`sweep`/`maxlife`/`late`/`retro`/`rule`/`manual`/`ingest`; `agent:*` and `register:*` mirror customer workflow mix — first prod pass fired on `agent:whiteglove_pending` rollout weeks — and `legacy:*` is derived, all excluded) | share < 1.5× baseline, the path stops occurring, or the path is no longer eligible |
|
||||
| `silence_share_regression` | the `sweep:*` + `maxlife:*` share doubled — the backend had to decide more often because the agent went silent (a liveness signal, not a classifier one) | same |
|
||||
| `evidence_gap` | absolute: `r6` (pure fallthrough) decides ≥ 20 % of the window's classifier verdicts, ≥ 20 verdicts | < 15 % or < 20 verdicts |
|
||||
|
||||
Episodes are tracker rows `verdictcalibration|{kind}|{path}|{status}` (register = `AddEntity`,
|
||||
409 = already burning, fail-closed; refresh carries `FirstNotifiedAt`; 30-day tracker
|
||||
retention re-arms long burners). A new episode fires **one** `VerdictCalibrationDrift` ops
|
||||
event (category Maintenance, Warning; dual-registered in the web `OPS_EVENT_TYPES`) — there
|
||||
is deliberately **no tenant bell**. Per-path findings on a tenant partition get the on-fire
|
||||
dimension correlation (`RuleRegressionRadar.ComputeDimensionConcentration` over the window's
|
||||
sessions whose derived path matches), reported with the shared wording contract "correlated,
|
||||
not necessarily causal"; group kinds and the global partition carry no dimension claim.
|
||||
Kill switch `VerdictCalibrationRadarDisabled`.
|
||||
|
||||
# Examples
|
||||
|
||||
* **Reading the thermometer (prod, 2026-08-23 hand analysis that motivated this):**
|
||||
re-enrollment within 7 days after Succeeded 8.9 % (lab background), after Failed 24.3 %,
|
||||
after Incomplete 10.6 % — Incomplete behaves like Succeeded, so rules 5/6 are probably too
|
||||
cautious; the matrix now shows this per rule instead of per status.
|
||||
* **Correction stream:** a session swept to `sweep:r5_incomplete`, later completed by the
|
||||
agent → row `agent:complete` with `PriorVerdictPath = sweep:r5_incomplete`; the aggregate
|
||||
increments `OverriddenByLateCompletion` on the `sweep:r5_incomplete / Incomplete` bucket.
|
||||
Four admin marks since launch were invisible before — now each one counts against the path
|
||||
it corrected.
|
||||
* **Evidence gap:** a tenant whose agent build stopped emitting the Device Setup rollup line
|
||||
drives its sweep verdicts into `r6`; the absolute gate fires at 20 % and names the
|
||||
tenant — the fix is agent-side, the classifier is only the messenger.
|
||||
|
||||
# Citations
|
||||
|
||||
* `src/Shared/AutopilotMonitor.Shared/Models/VerdictPaths.cs` — vocabulary + `ClassifierRules`
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Sessions.cs` — `ComputePriorVerdict`, `ApplyVerdictPath`, re-registration preserve
|
||||
* `src/Backend/AutopilotMonitor.Functions/Helpers/VerdictPathDerivation.cs`
|
||||
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.VerdictCalibration.cs` — sweep + `BuildVerdictCalibrationAggregates`
|
||||
* `src/Backend/AutopilotMonitor.Functions/Functions/Metrics/GetVerdictCalibrationFunction.cs`
|
||||
* `src/Backend/AutopilotMonitor.Functions/Helpers/VerdictCalibrationRadar.cs`, `Services/MaintenanceService.VerdictCalibrationRadar.cs`
|
||||
* `src/McpServer/autopilot-monitor-mcp/src/tools/admin.ts` — `get_verdict_calibration`
|
||||
* `src/McpServer/autopilot-monitor-mcp/src/verdict-calibration-shape.ts` — LLM-facing response shaping (`minSharePct` / `top`, explicit nulls)
|
||||
* `src/Web/autopilot-monitor-web/app/admin/metrics/sections/SectionVerdictCalibration.tsx`
|
||||
* Tests: `VerdictPathTests`, `VerdictCalibrationTests`, `VerdictCalibrationRadarTests`
|
||||
* Related: [Silent-Session Classification](silent-session-classification.md), [Device Journeys](device-journeys.md), [Rule Regression Radar](rule-regression-radar.md)
|
||||
+6
-1
@@ -19,7 +19,7 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
|
||||
# Agent
|
||||
|
||||
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
|
||||
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [diagnostics package](agent/diagnostics-package.md) (built-in section catalog shared with the portal, RealmJoin/Device Preparation collection gates, caps, manifest), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
|
||||
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
|
||||
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
|
||||
* [Gather Rule Debug Log](agent/gather-rule-debug-log.md) - EnableGatherRuleDebugLog (ConfigVersion 36): the per-tenant local trace file that explains every gather-rule evaluation decision — scope skips, on_change suppression, empty collector results, logparser position/parse-mode outcomes — for rules that produce no timeline events.
|
||||
@@ -32,6 +32,7 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
* [WDP Scenario Gates & Completion Backstop](agent/wdp-scenario-gates.md) - How the agent gates Autopilot/ESP-only machinery off on Windows Device Preparation sessions (deterministic ExecutionContext marker, fail-toward-Classic), which noise it suppresses (ZTD probe, esp_config unknown, ModernDeployment 100/417/418/1005, starved sweep), and the WDP completion model: High-confidence seed, desktop-based arm D, and the 30-min device_prep_completion backstop.
|
||||
* [Power-State Watcher](agent/power-state-watcher.md) - Live AC↔battery transitions and 50/30/15% downward battery-threshold crossings during enrollment via a WMI Win32_PowerManagementEvent push subscription (zero polling, never armed on battery-less devices): the power_state_change event (transition field, per-level lifetime latch, 5s debounce, emission cap) and the analyze rules ANALYZE-DEV-009 (battery <15%, high) / ANALYZE-DEV-010 (AC→battery, warning) with on_event interim triggers.
|
||||
* [System Timeline Watcher](agent/system-timeline-watcher.md) - Ground truth for two timeline blind spots from one System-channel EventLogWatcher with pre-agent backfill: system_clock_changed (Kernel-General 1 — oldTime/newTime/signed timeDeltaMs/reason/setting process; 2 s micro-slew floor, ≥5 min Warning+immediate) and system_sleep_episode (Power-Troubleshooter 1 sleep/hibernate + Kernel-Power 507 Modern Standby; authoritative enteredAt/exitedAt payload instants, 60 s Modern-Standby floor). Consumed by the TimeJumpBadge clock-set cause, the SessionInfoCard standby note, time-attribution SleepSpans (AttributionVersion 3) and ANALYZE-DEV-012.
|
||||
* [DO GroupId Auto-Set](agent/do-group-id-auto-set.md) - EnableDoGroupIdAutoSet (ConfigVersion 40): the agent seeds the Delivery Optimization DOGroupId policy with a deterministic network-fingerprint GUID (default gateway IPv4 + SendARP MAC, RealmJoin-compatible byte layout) so same-network devices form one DO peering group during ESP; ownership-tracked via the StartupEventGate fingerprint, foreign DOGroupId/DOGroupIdSource policies always win, no random-GUID fallback.
|
||||
* [ESP Policy-Provider Stall Detection](agent/esp-policy-provider-stall.md) - The EnrollmentStatusTracking CSP's no-timeout Setup/Apps wait is keyed to the Intune-registered "Sidecar" provider by NAME: co-management's "ConfigMgr" parks the user ESP at "Apps (Identifying)" for days even with TrackingPoliciesCreated=1 (issue #106); the always-on two-arm esp_policy_provider_stalled tripwire (provider_incomplete | sidecar_provider_missing) and the SOFTWARE\Microsoft\Windows\Autopilot gather-rule allowlist entry that surface it.
|
||||
|
||||
# Rules
|
||||
@@ -44,14 +45,18 @@ https://docs.autopilotmonitor.com (separate repository).
|
||||
|
||||
# Backend
|
||||
|
||||
* [Table Schema Sentinel](backend/table-schema-sentinel.md) - Startup no longer issues one CreateTableIfNotExists per table: a SHA-256 over TableNames.All in AdminConfiguration gates the full pass (derived, never hand-maintained), the SessionsIndex backfill runs only after a full pass behind a conditional-insert claim, and daily maintenance repairs out-of-band deletions.
|
||||
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
|
||||
* [Send-Time Frame Separation](backend/send-time-frame-separation.md) - One device-clock send timestamp per upload batch (X-Send-Time-Utc → SentAt on Events) splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset; the clock_skew condition measures SentAt batches directly (spool-immune), the CMTrace tripwire deliberately stays on its differential math.
|
||||
* [OOBE Bootstrap go-Code Flow](backend/oobe-bootstrap-go-flow.md) - The bootstrap short-code lifecycle (portal-created session → `irm go.autopilotmonitor.com/CODE | iex` at Shift+F10 → GetBootstrapScriptFunction), the always-HTTP-200 text/plain contract, the two-line injection defense (GUID/URL/expiry validator + single-quoted PS literals, golden-pinned against the legacy TS template), and the per-IP rate-limit story behind Front Door.
|
||||
* [Silent-Session Classification](backend/silent-session-classification.md) - How a session whose agent went silent gets its honest status: the two-stage maintenance sweep (2h Stalled marker with the WhiteGlove Part-2 AwaitingUser gate and the self-deploying Succeeded gate, 5h timeout classification), the EnrollmentTimeoutClassifier rule order (explicit failure → WG Part 2 → self-deploying → completion evidence → emergency break → desktop+Hello → Device Setup/grace), the ingest paths sharing it (max-lifetime, late-telemetry heal, stall probe) and the admin retro-reconcile modes incl. self_deploying_silent.
|
||||
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
|
||||
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
|
||||
* [Attempt-Scoped App Install Durations](backend/app-install-duration-attempt.md) - AppInstallSummary.DurationSeconds = status-defining (last) install attempt via the LastAttemptStartedAt max-fold anchor instead of the first-observation→last-terminal span that billed multi-pass IME gaps to trivial apps; InstallPassCount, the weaker-terminal guard, the radar's cutover gate, and time attribution's union-of-attempt-segments occupancy.
|
||||
* [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract.
|
||||
* [Verdict Calibration](backend/verdict-calibration.md) - How every session status write declares its VerdictPath (origin:detail vocabulary incl. classifier rule ids), how overriding a verdict preserves PriorStatus/PriorVerdictPath as the correction stream, the daily per-path calibration aggregate with the 7-day re-enrollment proxy and read-side legacy derivation, the GA endpoint / MCP tool / admin matrix, and the drift radar (share, silence-share, r6 evidence gap) with its operator-only ops events.
|
||||
* [App Version Duration Regression Radar](backend/app-version-regression-radar.md) - How app versions whose median install duration regressed vs. the previous version are detected (first-seen-ordered versions, median lift ≥2× plus a 5-minute absolute floor over measured installs only), how alert episodes live in the appversionregression tracker keyspace with drain/median re-arm semantics, and the bell/ops-event/analytics surfaces.
|
||||
* [Plan Downgrade Policy & the Retention Grace Period](backend/plan-downgrade-retention-grace.md) - What happens when a tenant loses Pro: every entitlement gates read-time and reversibly ("gate, don't destroy"), except retention — a 30-day grace keeps the Pro cap before the Community cap starts hard-deleting; the ProDowngradedUtc/trial-expiry anchor model, the TenantPlanDowngraded/RetentionGrace* ops events, and the deliberate non-grace of everything else.
|
||||
* [Tenant Activation & the Once-Only Welcome Mail](backend/tenant-activation-welcome-mail.md) - How a signup becomes an activated tenant (manual approve or the auto-approve queue), and the write-then-read + conditional-marker arbitration that sends the welcome mail exactly once even though activation and the user-typed notification address race each other.
|
||||
* [W365 Cloud PC Device Validation](backend/cloudpc-device-validation.md) - Why Cloud PCs can never pass the Autopilot serial lookup, and the cert-CN-bound CloudPc fallback validator: Subject CN (= Intune device id) matched against the tenant's Graph virtualEndpoint/cloudPCs inventory — admits exactly the service-provisioned Cloud PCs, nothing else.
|
||||
* [ServerActions & On-Demand Diagnostics](backend/server-actions-on-demand-diagnostics.md) - The queue-on-session, deliver-on-ingest action channel, the TenantAdminOrOperator tier with in-function per-type re-gating, the portal's Collect Logs flow (quick-config orders rotate_config before request_diagnostics), and the hosted-prefix sweep in the offboarding cascade.
|
||||
|
||||
+21
@@ -1,7 +1,28 @@
|
||||
# Log
|
||||
|
||||
## 2026-08-23
|
||||
|
||||
* **Update**: `backend/table-schema-sentinel.md` — "Measuring it": `StartupTelemetryService` emits `BackendStartupMs` (process start → ApplicationStarted) and `BackendTableInitMs` as App Insights custom metrics with version/commit/fullPass dimensions; the MCP server writes one `{"type":"boot","readyMs"}` stderr line per start (query in `.claude/commands/mcp-telemetry.md` §A2).
|
||||
* **New**: `backend/table-schema-sentinel.md` — startup table initialization gated by a schema sentinel (SHA-256 over `TableNames.All` in `AdminConfiguration`/`SchemaSentinel`/`tables`): matching sentinel ⇒ zero `CreateTableIfNotExists` calls on a cold start, missing/stale ⇒ one full pass + sentinel rewrite. SessionsIndex startup backfill only after a full pass, `IsSessionIndexEmptyAsync` no longer treats errors as "empty", conditional-insert claim so one scaled-out instance backfills. Daily maintenance runs `IStorageInitializer.EnsureAllAsync` to repair out-of-band table deletions.
|
||||
* **Update**: `backend/verdict-calibration.md` — MCP `get_verdict_calibration` response shaping: per-row trend denominators dropped, explicit nulls, `minSharePct` / `top` with an `omitted` report.
|
||||
* **Update**: `backend/verdict-calibration.md` — share-regression radar restricted to backend-decided paths; agent-declared/registration paths (customer workflow mix) and derived legacy paths no longer alert, stale episodes re-arm themselves.
|
||||
* **Update**: `backend/time-attribution.md`, `backend/device-journeys.md`, `backend/verdict-calibration.md` — the rolling maintenance sweeps (time attribution, device journeys, verdict calibration, calibration-radar discovery) now share ONE projected 35-day cross-tenant Sessions scan per tick (`LoadSweepWindowSessionsAsync` + `MaintenanceSweepSessionProjection`, pinned by `MaintenanceSweepProjectionEquivalenceTests`) instead of four full-table drains; each sweep slices its own window by StartedAt.
|
||||
* **New**: `backend/verdict-calibration.md` — the classifier thermometer: every status write now stamps `VerdictPath` (required parameter on `UpdateSessionStatusAsync`; `EnrollmentTimeoutClassifier` returns its rule id), overriding a verdict preserves `PriorStatus`/`PriorVerdictPath`, `VerdictPathDerivation` attributes pre-instrumentation rows read-side (`legacy:*` when ambiguous). New daily aggregate `VerdictCalibrationAggregates` (per-path count/derived/eligible7d/reEnrolled7d/overridden-by-admin|late|other, sweep after the device-journey pass, 180d retention), `GET global/metrics/verdict-calibration`, MCP `get_verdict_calibration`, admin metrics section, and the `VerdictCalibrationRadar` (share / silence-share / r6 evidence-gap episodes in the tracker keyspace `verdictcalibration|`, `VerdictCalibrationDrift` ops event, no tenant bell).
|
||||
* **New**: `backend/silent-session-classification.md` — one place for the silent-session verdict: `EnrollmentTimeoutClassifier` rule order incl. the new rule 1c `IsSelfDeployingProvisioned` (self-deploying profile + Device Setup all-succeeded + no explicit failure ⇒ Succeeded reconcile, no grace/AwaitingUser — kiosk tenant aebdce78 audit: 836 finished devices parked AwaitingUser/Incomplete), the sweep stage-1 gates (`TryMarkWhiteGloveAwaitingUserAsync`, new `TryReconcileSelfDeployingAsync`), the ingest callers (max-lifetime, late-telemetry heal, stall probe) and the new admin retro mode `self_deploying_silent` (`GetSelfDeployingSilentSessionsAsync`). Agent companion: `SessionRegistrationHelper` waits up to 15s for a network link and retries 6× (~62s) instead of 5× (~30s) so the reboot relaunch on Wi-Fi no longer exits 7 before the link is back.
|
||||
* **New**: `agent/diagnostics-package.md` — the diagnostics ZIP's built-in collection is now ONE catalog in Shared (`DiagnosticsBuiltInSections.All`): the agent iterates it in order and the backend serves the same list to the portal (`GET diagnostics/paths`, MemberRead), so what administrators see is exactly what the agent collects. New conditional sections: RealmJoin client/package/Chocolatey/per-user logs behind the tenant's RealmJoin Watcher toggle (mirrored onto `AgentConfiguration.EnableRealmJoinWatcher` by the merger) and the Device Preparation bootstrapper event log behind the deterministic WDP marker (probe once per build, fail toward Classic); every skip is a manifest line. Built-ins bypass `DiagnosticsPathGuards` by design (reviewed code, `%ProgramData%\RealmJoin` is not allowlisted); caps/reparse/evtx export still apply. `MergeDiagnosticsLogPaths` dedups global ∪ tenant paths. Portal: collapsed read-only catalog + one-line rows in both cards, tenant context dropped the GA-only global fetch.
|
||||
* **Update**: `rules/gather-rule-guardrails.md` — `DiagnosticsPathGuards` scope clarified: configured diagnostics paths only, never the built-in catalog.
|
||||
* **Update**: `backend/tenant-activation-welcome-mail.md` — new "Template overrides" section: Global Admins can replace the welcome/farewell HTML in the portal (Settings → Email Templates); `EmailTemplateService` resolves override-or-built-in per send (PreviewConfig table, partition `EmailTemplates`, 5-min cache, `{{domainName}}` placeholder, 30k limit, fail-soft send path), GA endpoints `global/email-templates/{kind}` (+`/test` to the caller's tenant contact address). Also: the mail header is image-free (CSS-drawn brand mark, gradient stripes and clip-path edge as progressive enhancement).
|
||||
|
||||
## 2026-08-22
|
||||
|
||||
* **Update**: `backend/tenant-activation-welcome-mail.md` — new "Mail transport" section: transactional mail now goes through the provider-neutral `EmailService` (`IEmailService` + `IOffboardFarewellEmailSender`) with the provider confined to one private transport method and the `Email:*` settings (`ApiKey`, `Endpoint`, `FromAddress`, `FromName`); current provider Mailchimp Transactional (Mandrill) replacing Resend (NuGet dropped, typed `HttpClient` + `ResiliencePolicies.Notification`). Result-array semantics (`sent`/`queued`/`scheduled` ok, `rejected`/`invalid` warn), open/click tracking explicitly off because the privacy/trust claim depends on it.
|
||||
|
||||
* **New**: `backend/plan-downgrade-retention-grace.md` — plan downgrade policy: every Pro entitlement gates read-time and reversibly on downgrade ("gate, don't destroy"), except retention, which now gets a 30-day grace (`FeatureEntitlementCatalog.RetentionDowngradeGraceDays`) before the Community cap starts hard-deleting the 90–365-day band. Anchor model: `ProDowngradedUtc` stamped/cleared by the plan endpoint's pure `ApplyPlanChanges` core on EFFECTIVE-edition transitions (covers explicitly ended trials; planTier downgrade under an active trial deliberately doesn't stamp), trial expiry anchors read-time via `TrialExpiresUtc`. New Warning ops events `TenantPlanDowngraded` (PATCH handler) and `TenantRetentionGraceExpiring`/`TenantRetentionGraceEnded` (TrialExpirySweep window mechanics, only when stored retention exceeds the Community cap). Field is patch-denied + revert-protected; retention WRITE cap deliberately ignores the grace.
|
||||
|
||||
## 2026-08-21
|
||||
|
||||
* **New**: `agent/do-group-id-auto-set.md` — optional DO peering-group seed (`EnableDoGroupIdAutoSet`, default off, ConfigVersion 40): `DoGroupIdService` derives a deterministic GUID from the default gateway's IPv4 + SendARP MAC in RealmJoin's exact byte layout (cross-product peering) and writes `DOGroupId` in the 64-bit GPO Policies store with readback verification. Conservative by design: no random-GUID fallback (skip on missing gateway/MAC), `DOGroupIdSource`/MDM-managed values/foreign GPO values always win (skip reasons in the `do_group_id_auto_set` event), own stale values are recognized via the `written:<guid>` StartupEventGate fingerprint and updated after a gateway change. Gateway heuristic extracted into the shared `NetworkInterfaceLocator` (dedup of NetworkChangeDetector + DeviceInfoCollector copies).
|
||||
|
||||
* **New**: `agent/system-timeline-watcher.md` — clock-step and sleep-episode ground truth from the System event log: one `EventLogWatcher` (Kernel-General 1, Power-Troubleshooter 1, Kernel-Power 507; 506 deliberately unconsumed) emits `system_clock_changed` (oldTime/newTime/signed timeDeltaMs/reason/process; 2 s micro-slew floor, ≥5 min = Warning+immediate) and `system_sleep_episode` (kind sleep/hibernate/modern_standby, authoritative `enteredAt`/`exitedAt` payload instants, 60 s Modern-Standby floor). WindowsUpdateTracker-style backfill (24 h default, `system-timeline-watermark.json`) recovers pre-agent records; ConfigVersion 39 (`Collectors.EnableSystemTimelineWatcher`). Consumers: TimeJumpBadge cause `clock-set` (ground truth beats inference), SessionInfoCard standby note next to the unchanged wall-clock duration, `SleepSpan`s in time attribution (AttributionVersion 3, RebootSpan clone), and ANALYZE-DEV-012 (info, single, ≥5 min episode; deliberately the only new rule — clock issues stay with DEV-008).
|
||||
|
||||
* **New**: `agent/power-state-watcher.md` — live AC/battery observation during enrollment: `PowerStateWatcherHost` subscribes to the WMI push `Win32_PowerManagementEvent EventCode 10` (no polling; re-probes `GetSystemPowerStatus` per push, 5 s trailing-edge debounce, never armed on battery-less devices, `Collectors.EnablePowerStateWatcher` kill-switch, ConfigVersion 38). The pure `PowerStateTransitionTracker` emits `power_state_change` with `transition` = ac_to_battery (Warning, immediate) / battery_to_ac (Info) / threshold_crossed at 50/30/15% (each level latched once per lifetime, lowest-crossed-only on jumps, baseline-below-level covered, emission cap 20). Classified non-activity on both sides (SignalActivityClassifier + IsPeriodicOrStallEvent). New analyze rules ANALYZE-DEV-009 (thresholdPercent=15, high) and ANALYZE-DEV-010 (transition=ac_to_battery, warning), both with `on_event:power_state_change` interim triggers and deliberately single-required-condition (event_data conditions have no same-instance join).
|
||||
|
||||
@@ -31,6 +31,12 @@ item in `AutopilotMonitor.Agent.V2.Core.csproj`) and regenerated into the portal
|
||||
| `IsCommandAllowed` | `allowedCommands` | **exact** match, trimmed — never prefix |
|
||||
| `IsEventLogChannelAllowed` | `eventLogChannels` | prefix, boundary `/` or end |
|
||||
|
||||
`DiagnosticsPathGuards.IsDiagnosticsPathAllowed` (allowlist `diagnosticsPathPrefixes`, a C#
|
||||
literal mirrored in the JSON) applies the same hard blocks and the same user-profile
|
||||
exception to **admin-configured diagnostics paths** only. The diagnostics package's built-in
|
||||
sections are reviewed code and never pass through it — see
|
||||
[Diagnostics Package](../agent/diagnostics-package.md).
|
||||
|
||||
**Hard blocks live in code, not in the JSON**, so a parse error cannot lift them — on a
|
||||
parse failure the allowlists fall back to empty and everything is blocked (fail-closed),
|
||||
but the hard blocks must hold regardless of configuration:
|
||||
|
||||
+4
-2
@@ -37,6 +37,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
|
||||
nameof(AgentConfigResponse.EnrollmentSummaryLaunchRetrySeconds),
|
||||
nameof(AgentConfigResponse.NtpServer),
|
||||
nameof(AgentConfigResponse.EnableTimezoneAutoSet),
|
||||
nameof(AgentConfigResponse.EnableDoGroupIdAutoSet),
|
||||
nameof(AgentConfigResponse.DiagnosticsUploadEnabled),
|
||||
nameof(AgentConfigResponse.DiagnosticsUploadMode),
|
||||
nameof(AgentConfigResponse.DiagnosticsLogPaths),
|
||||
@@ -52,6 +53,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
|
||||
// Merge reads two Collectors knobs (AgentMaxLifetimeMinutes, HelloWaitTimeoutSeconds);
|
||||
// the remaining CollectorConfiguration flags flow untouched to DefaultComponentFactory.
|
||||
nameof(AgentConfigResponse.Collectors),
|
||||
// Merge reads one Analyzers knob (EnableRealmJoinWatcher → diagnostics RealmJoin sections);
|
||||
// the remaining flags flow untouched to DefaultComponentFactory + AgentAnalyzerManager.
|
||||
nameof(AgentConfigResponse.Analyzers),
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
@@ -69,8 +73,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
|
||||
nameof(AgentConfigResponse.ImeLogPatterns),
|
||||
// Passed by AgentRuntimeHost into IComponentFactory.CreateCollectorHosts → ImeLogHost sealing re-emit.
|
||||
nameof(AgentConfigResponse.WhiteGloveSealingPatternIds),
|
||||
// Consumed by DefaultComponentFactory (RealmJoin/KeepAwake/ConsoleBypass host gates) + AgentAnalyzerManager.
|
||||
nameof(AgentConfigResponse.Analyzers),
|
||||
// Self-update ZIP integrity second trust channel — BinaryIntegrityVerifier / Program update path.
|
||||
nameof(AgentConfigResponse.LatestAgentSha256),
|
||||
// Running-EXE integrity verification — BinaryIntegrityVerifier post-config check.
|
||||
|
||||
+29
@@ -43,6 +43,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
|
||||
EnrollmentSummaryLaunchRetrySeconds = 300,
|
||||
NtpServer = "pool.ntp.org",
|
||||
EnableTimezoneAutoSet = true,
|
||||
EnableDoGroupIdAutoSet = true,
|
||||
DiagnosticsUploadEnabled = true,
|
||||
DiagnosticsUploadMode = "OnFailure",
|
||||
DiagnosticsLogPaths = new List<DiagnosticsLogPath>
|
||||
@@ -83,6 +84,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
|
||||
Assert.Equal(300, agent.EnrollmentSummaryLaunchRetrySeconds);
|
||||
Assert.Equal("pool.ntp.org", agent.NtpServer);
|
||||
Assert.True(agent.EnableTimezoneAutoSet);
|
||||
Assert.True(agent.EnableDoGroupIdAutoSet);
|
||||
Assert.True(agent.DiagnosticsUploadEnabled);
|
||||
Assert.Equal("OnFailure", agent.DiagnosticsUploadMode);
|
||||
Assert.Single(agent.DiagnosticsLogPaths);
|
||||
@@ -366,6 +368,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
|
||||
Assert.Equal(30, agent.HelloWaitTimeoutSeconds);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Merge_mirrors_analyzers_realmjoin_watcher_onto_agent_config()
|
||||
{
|
||||
// The diagnostics package gates its RealmJoin sections on this flag but never
|
||||
// sees the remote response — the merger carries the one Analyzers knob over.
|
||||
var agent = NewAgentConfig();
|
||||
var remote = FullRemote();
|
||||
remote.Analyzers = new AnalyzerConfiguration { EnableRealmJoinWatcher = true };
|
||||
|
||||
RemoteConfigMerger.Merge(agent, remote);
|
||||
|
||||
Assert.True(agent.EnableRealmJoinWatcher);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Merge_clears_realmjoin_watcher_when_remote_analyzers_missing()
|
||||
{
|
||||
var agent = NewAgentConfig();
|
||||
agent.EnableRealmJoinWatcher = true; // stale cached true must not survive
|
||||
var remote = FullRemote();
|
||||
remote.Analyzers = null!;
|
||||
|
||||
RemoteConfigMerger.Merge(agent, remote);
|
||||
|
||||
Assert.False(agent.EnableRealmJoinWatcher);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Merge_replaces_diagnostics_log_paths_with_empty_list_when_remote_null()
|
||||
{
|
||||
|
||||
+188
-2
@@ -1,9 +1,11 @@
|
||||
#nullable enable
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.IO.Compression;
|
||||
using System.Linq;
|
||||
using System.Text;
|
||||
using AutopilotMonitor.Agent.V2.Core.Configuration;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
|
||||
@@ -46,7 +48,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
|
||||
Directory.CreateDirectory(SpoolFolder);
|
||||
}
|
||||
|
||||
public DiagnosticsPackageService Build(System.Action<AgentConfiguration>? mutateConfig = null)
|
||||
public DiagnosticsPackageService Build(
|
||||
System.Action<AgentConfiguration>? mutateConfig = null,
|
||||
IReadOnlyDictionary<string, string>? sectionFolderOverrides = null,
|
||||
System.Func<bool>? devicePreparationProbe = null)
|
||||
{
|
||||
// BackendApiClient is required by the public ctor but BuildArchiveBytes
|
||||
// never touches it — construct with a throwaway HttpClient. Tests only
|
||||
@@ -71,7 +76,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
|
||||
imeLogFolderOverride: ImeTmp.Path,
|
||||
agentStateFolderOverride: StateFolder,
|
||||
agentSpoolFolderOverride: SpoolFolder,
|
||||
agentDataFolderOverride: DataFolder);
|
||||
agentDataFolderOverride: DataFolder,
|
||||
sectionFolderOverrides: sectionFolderOverrides,
|
||||
devicePreparationProbe: devicePreparationProbe);
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
@@ -287,6 +294,185 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
|
||||
Assert.DoesNotContain(entries, e => e.StartsWith("AgentSpool/"));
|
||||
}
|
||||
|
||||
// ===================================================== built-in catalog gates ====
|
||||
// The RealmJoin sections ride behind the tenant's RealmJoin Watcher toggle, the
|
||||
// Device Preparation bootstrapper event log behind the deterministic WDP marker.
|
||||
// Every conditional section is skipped WITH a manifest line, never silently.
|
||||
|
||||
/// <summary>RealmJoin system-side folders redirected to temp (user-side goes via UserProfileResolver).</summary>
|
||||
private sealed class RealmJoinRig : System.IDisposable
|
||||
{
|
||||
public TempDirectory WindowsLogs { get; } = new TempDirectory();
|
||||
public TempDirectory Choco { get; } = new TempDirectory();
|
||||
public TempDirectory UserProfile { get; } = new TempDirectory();
|
||||
public string PackagesFolder => Path.Combine(WindowsLogs.Path, "RealmJoin");
|
||||
|
||||
public RealmJoinRig()
|
||||
{
|
||||
File.WriteAllText(Path.Combine(WindowsLogs.Path, "realmjoin.log"), "rj");
|
||||
File.WriteAllText(Path.Combine(WindowsLogs.Path, "realmjoin2.log"), "rj2");
|
||||
var pkg = Path.Combine(PackagesFolder, "Packages", "generic-7zip");
|
||||
Directory.CreateDirectory(pkg);
|
||||
File.WriteAllText(Path.Combine(pkg, "install.log"), "pkg");
|
||||
var choco = Path.Combine(Choco.Path, "generic-7zip");
|
||||
Directory.CreateDirectory(choco);
|
||||
File.WriteAllText(Path.Combine(choco, "2026-06-30_install.log"), "choco");
|
||||
var userRj = Path.Combine(UserProfile.Path, "AppData", "Local", "RealmJoin");
|
||||
Directory.CreateDirectory(Path.Combine(userRj, "Logs", "generic-office"));
|
||||
File.WriteAllText(Path.Combine(userRj, "tray.log"), "tray");
|
||||
File.WriteAllText(Path.Combine(userRj, "config.pjson"), "not a log");
|
||||
File.WriteAllText(Path.Combine(userRj, "Logs", "RjImeHost.log"), "host");
|
||||
File.WriteAllText(Path.Combine(userRj, "Logs", "generic-office", "usersettings.log"), "pkg-user");
|
||||
}
|
||||
|
||||
public IReadOnlyDictionary<string, string> Overrides => new Dictionary<string, string>
|
||||
{
|
||||
["RealmJoinWindows"] = WindowsLogs.Path,
|
||||
["RealmJoinPackages"] = PackagesFolder,
|
||||
["RealmJoinChoco"] = Choco.Path,
|
||||
};
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
WindowsLogs.Dispose();
|
||||
Choco.Dispose();
|
||||
UserProfile.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildArchiveBytes_skips_realmjoin_sections_when_watcher_disabled()
|
||||
{
|
||||
using var rig = new Rig();
|
||||
using var rj = new RealmJoinRig();
|
||||
UserProfileResolver.SetForTesting(rj.UserProfile.Path);
|
||||
try
|
||||
{
|
||||
var bytes = rig.Build(sectionFolderOverrides: rj.Overrides).BuildArchiveBytes(enrollmentSucceeded: true);
|
||||
|
||||
var entries = ZipEntryNames(bytes);
|
||||
Assert.DoesNotContain(entries, e => e.StartsWith("RealmJoinLogs/"));
|
||||
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
|
||||
Assert.Contains("SCENARIO: devicePreparation=False realmJoinWatcher=False", manifest);
|
||||
Assert.Contains("BUILT-IN SKIPPED (RealmJoin Watcher disabled): RealmJoinWindows", manifest);
|
||||
Assert.Contains("BUILT-IN SKIPPED (RealmJoin Watcher disabled): RealmJoinUserLogs", manifest);
|
||||
}
|
||||
finally
|
||||
{
|
||||
UserProfileResolver.Reset();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildArchiveBytes_includes_realmjoin_sections_when_watcher_enabled()
|
||||
{
|
||||
using var rig = new Rig();
|
||||
using var rj = new RealmJoinRig();
|
||||
UserProfileResolver.SetForTesting(rj.UserProfile.Path);
|
||||
try
|
||||
{
|
||||
var bytes = rig.Build(cfg => cfg.EnableRealmJoinWatcher = true, rj.Overrides)
|
||||
.BuildArchiveBytes(enrollmentSucceeded: false);
|
||||
|
||||
var entries = ZipEntryNames(bytes);
|
||||
// ZIP layout mirrors the disk layout: flat client logs + package tree under
|
||||
// Windows/, tray logs + per-user Logs tree under User/.
|
||||
Assert.Contains("RealmJoinLogs/Windows/realmjoin.log", entries);
|
||||
Assert.Contains("RealmJoinLogs/Windows/realmjoin2.log", entries);
|
||||
Assert.Contains("RealmJoinLogs/Windows/RealmJoin/Packages/generic-7zip/install.log", entries);
|
||||
Assert.Contains("RealmJoinLogs/Choco/generic-7zip/2026-06-30_install.log", entries);
|
||||
Assert.Contains("RealmJoinLogs/User/tray.log", entries);
|
||||
Assert.Contains("RealmJoinLogs/User/Logs/RjImeHost.log", entries);
|
||||
Assert.Contains("RealmJoinLogs/User/Logs/generic-office/usersettings.log", entries);
|
||||
// Only tray*.log is collected from the user RealmJoin root — config stays out.
|
||||
Assert.DoesNotContain(entries, e => e.EndsWith("config.pjson"));
|
||||
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
|
||||
Assert.Contains("BUILT-IN: RealmJoinUserLogs -> folder=", manifest);
|
||||
Assert.DoesNotContain("BUILT-IN SKIPPED (RealmJoin Watcher disabled)", manifest);
|
||||
}
|
||||
finally
|
||||
{
|
||||
UserProfileResolver.Reset();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildArchiveBytes_skips_realmjoin_user_sections_without_user_session()
|
||||
{
|
||||
using var rig = new Rig();
|
||||
using var rj = new RealmJoinRig();
|
||||
// Token present, no interactive user detected → user-profile sections skip with a
|
||||
// manifest line; the system-side sections are unaffected.
|
||||
UserProfileResolver.SetForTesting(null!);
|
||||
try
|
||||
{
|
||||
var bytes = rig.Build(cfg => cfg.EnableRealmJoinWatcher = true, rj.Overrides)
|
||||
.BuildArchiveBytes(enrollmentSucceeded: true);
|
||||
|
||||
var entries = ZipEntryNames(bytes);
|
||||
Assert.Contains("RealmJoinLogs/Windows/realmjoin.log", entries);
|
||||
Assert.DoesNotContain(entries, e => e.StartsWith("RealmJoinLogs/User"));
|
||||
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
|
||||
Assert.Contains("BUILT-IN SKIPPED (no user session for token): RealmJoinUserTray", manifest);
|
||||
Assert.Contains("BUILT-IN SKIPPED (no user session for token): RealmJoinUserLogs", manifest);
|
||||
}
|
||||
finally
|
||||
{
|
||||
UserProfileResolver.Reset();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildArchiveBytes_collects_bootstrapper_event_log_only_on_device_preparation()
|
||||
{
|
||||
using var rig = new Rig();
|
||||
using var winevt = new TempDirectory();
|
||||
// Synthetic evtx: no registered channel claims the file, so the export falls
|
||||
// back to a raw copy — the point here is the scenario gate, not wevtutil.
|
||||
File.WriteAllBytes(Path.Combine(winevt.Path, "BootstrapperAgentServiceLogProvider.evtx"), new byte[] { 1, 2, 3 });
|
||||
var overrides = new Dictionary<string, string> { ["ImeBootstrapperEventLog"] = winevt.Path };
|
||||
|
||||
var classic = rig.Build(sectionFolderOverrides: overrides, devicePreparationProbe: () => false)
|
||||
.BuildArchiveBytes(enrollmentSucceeded: true);
|
||||
Assert.DoesNotContain("ImeLogs/BootstrapperAgentServiceLogProvider.evtx", ZipEntryNames(classic));
|
||||
Assert.Contains("BUILT-IN SKIPPED (not a Device Preparation enrollment): ImeBootstrapperEventLog",
|
||||
ReadZipEntryText(classic, "package-manifest.txt"));
|
||||
|
||||
var wdp = rig.Build(sectionFolderOverrides: overrides, devicePreparationProbe: () => true)
|
||||
.BuildArchiveBytes(enrollmentSucceeded: true);
|
||||
Assert.Contains("ImeLogs/BootstrapperAgentServiceLogProvider.evtx", ZipEntryNames(wdp));
|
||||
Assert.Contains("SCENARIO: devicePreparation=True", ReadZipEntryText(wdp, "package-manifest.txt"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildArchiveBytes_treats_failing_device_preparation_probe_as_classic()
|
||||
{
|
||||
using var rig = new Rig();
|
||||
var bytes = rig.Build(devicePreparationProbe: () => throw new System.InvalidOperationException("registry boom"))
|
||||
.BuildArchiveBytes(enrollmentSucceeded: true);
|
||||
|
||||
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
|
||||
Assert.Contains("SCENARIO PROBE FAILED (treated as Classic): registry boom", manifest);
|
||||
Assert.Contains("BUILT-IN SKIPPED (not a Device Preparation enrollment): ImeBootstrapperEventLog", manifest);
|
||||
Assert.Contains("sessioninfo.txt", ZipEntryNames(bytes)); // archive still built
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(DiagnosticsSectionCondition.Always, false, false, true)]
|
||||
[InlineData(DiagnosticsSectionCondition.RealmJoinWatcher, false, true, false)]
|
||||
[InlineData(DiagnosticsSectionCondition.RealmJoinWatcher, true, false, true)]
|
||||
[InlineData(DiagnosticsSectionCondition.DevicePreparation, true, false, false)]
|
||||
[InlineData(DiagnosticsSectionCondition.DevicePreparation, false, true, true)]
|
||||
[InlineData((DiagnosticsSectionCondition)999, true, true, false)]
|
||||
public void IsSectionActive_gates_on_condition(DiagnosticsSectionCondition condition, bool watcher, bool wdp, bool expected)
|
||||
{
|
||||
var section = new DiagnosticsBuiltInSection("X", "X", @"C:\X", new[] { "*.log" }, false, "x", condition);
|
||||
var cfg = Cfg();
|
||||
cfg.EnableRealmJoinWatcher = watcher;
|
||||
|
||||
Assert.Equal(expected, DiagnosticsPackageService.IsSectionActive(section, cfg, wdp));
|
||||
}
|
||||
|
||||
private static string[] ZipEntryNames(byte[] zipBytes)
|
||||
{
|
||||
using var ms = new MemoryStream(zipBytes);
|
||||
|
||||
+126
@@ -0,0 +1,126 @@
|
||||
using System;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
|
||||
{
|
||||
public class DoGroupIdServiceTests
|
||||
{
|
||||
// -------------------------------------------------------------------
|
||||
// ComputeGroupId — RealmJoin byte-layout compatibility
|
||||
// -------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public void ComputeGroupId_MatchesHandComputedVector()
|
||||
{
|
||||
// Layout: {101,48,6} + {0,0,0} + {ip[1],0,ip[2],ip[3]} + mac — through new Guid(byte[])
|
||||
// the first three groups print little-endian, the last eight bytes verbatim.
|
||||
var guid = DoGroupIdService.ComputeGroupId(
|
||||
new byte[] { 192, 168, 178, 1 },
|
||||
new byte[] { 0x00, 0x11, 0x22, 0x33, 0x44, 0x55 });
|
||||
|
||||
Assert.Equal("00063065-0000-00a8-b201-001122334455", guid.ToString());
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(new byte[] { 10, 0, 0, 1 }, new byte[] { 0xAA, 0xBB, 0xCC, 0xDD, 0xEE, 0xFF })]
|
||||
[InlineData(new byte[] { 192, 168, 1, 254 }, new byte[] { 0x00, 0x50, 0x56, 0x01, 0x02, 0x03 })]
|
||||
[InlineData(new byte[] { 172, 16, 254, 3 }, new byte[] { 0xF0, 0x9F, 0xC2, 0x00, 0x00, 0x01 })]
|
||||
public void ComputeGroupId_MatchesRealmJoinReferenceImplementation(byte[] ip, byte[] mac)
|
||||
{
|
||||
Assert.Equal(RealmJoinReferenceGuid(ip, mac), DoGroupIdService.ComputeGroupId(ip, mac));
|
||||
}
|
||||
|
||||
/// <summary>Verbatim port of RealmJoin's ConvertToGuid — the compatibility contract.</summary>
|
||||
private static Guid RealmJoinReferenceGuid(byte[] ipBytes, byte[] address)
|
||||
{
|
||||
byte[] prefix = { 101, 48, 6 };
|
||||
var stuff = new byte[4];
|
||||
stuff[0] = ipBytes[1];
|
||||
stuff[2] = ipBytes[2];
|
||||
stuff[3] = ipBytes[3];
|
||||
|
||||
return new Guid(prefix
|
||||
.Concat(new byte[16 - prefix.Length - stuff.Length - address.Length])
|
||||
.Concat(stuff)
|
||||
.Concat(address)
|
||||
.ToArray());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputeGroupId_RejectsNonIpv4AndBadMacLengths()
|
||||
{
|
||||
var ip4 = new byte[] { 10, 0, 0, 1 };
|
||||
var mac = new byte[] { 1, 2, 3, 4, 5, 6 };
|
||||
|
||||
Assert.Throws<ArgumentException>(() => DoGroupIdService.ComputeGroupId(new byte[16], mac));
|
||||
Assert.Throws<ArgumentException>(() => DoGroupIdService.ComputeGroupId(null!, mac));
|
||||
Assert.Throws<ArgumentException>(() => DoGroupIdService.ComputeGroupId(ip4, new byte[8]));
|
||||
Assert.Throws<ArgumentException>(() => DoGroupIdService.ComputeGroupId(ip4, null!));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputeGroupId_RegistryFormat_IsBracelessLowercase()
|
||||
{
|
||||
// RealmJoin writes SetValue(name, Guid) which stringifies via Guid.ToString() ("d"
|
||||
// format) — our written value must be the identical string form.
|
||||
var text = DoGroupIdService.ComputeGroupId(
|
||||
new byte[] { 10, 1, 2, 3 }, new byte[] { 0xAB, 0xCD, 0xEF, 0x01, 0x02, 0x03 }).ToString();
|
||||
|
||||
Assert.DoesNotContain("{", text);
|
||||
Assert.Equal(text.ToLowerInvariant(), text);
|
||||
Assert.Equal(36, text.Length);
|
||||
}
|
||||
|
||||
// -------------------------------------------------------------------
|
||||
// DecideAction — ownership matrix
|
||||
// -------------------------------------------------------------------
|
||||
|
||||
private const string Computed = "00063065-0000-00a8-b201-001122334455";
|
||||
private const string Other = "11111111-2222-3333-4444-555555555555";
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
public void DecideAction_NoExistingValue_Writes(string? existing)
|
||||
{
|
||||
Assert.Equal(DoGroupIdAction.Write, DoGroupIdService.DecideAction(existing, Computed, null));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DecideAction_ExistingEqualsComputed_IsNoOp()
|
||||
{
|
||||
Assert.Equal(DoGroupIdAction.NoOp, DoGroupIdService.DecideAction(Computed, Computed, null));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DecideAction_ExistingEqualsComputed_ToleratesBracesAndCase()
|
||||
{
|
||||
// A value written by another tool in "{...}"/uppercase form is still semantically ours.
|
||||
var braced = "{" + Computed.ToUpperInvariant() + "}";
|
||||
Assert.Equal(DoGroupIdAction.NoOp, DoGroupIdService.DecideAction(braced, Computed, null));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DecideAction_ExistingIsOurStaleValue_WritesUpdate()
|
||||
{
|
||||
// Gateway changed: the registry still holds the GUID we wrote for the old network.
|
||||
Assert.Equal(DoGroupIdAction.Write, DoGroupIdService.DecideAction(Other, Computed, Other));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DecideAction_ForeignValue_Skips()
|
||||
{
|
||||
Assert.Equal(DoGroupIdAction.SkipForeign, DoGroupIdService.DecideAction(Other, Computed, null));
|
||||
Assert.Equal(DoGroupIdAction.SkipForeign, DoGroupIdService.DecideAction(Other, Computed, Computed));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DecideAction_NonGuidGarbageExisting_SkipsAsForeign()
|
||||
{
|
||||
// Never overwrite a value we can't even parse — conservative by design.
|
||||
Assert.Equal(DoGroupIdAction.SkipForeign, DoGroupIdService.DecideAction("not-a-guid", Computed, null));
|
||||
}
|
||||
}
|
||||
}
|
||||
+14
-2
@@ -1,5 +1,6 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Agent.V2.Core.Orchestration;
|
||||
using AutopilotMonitor.DecisionCore.Signals;
|
||||
|
||||
@@ -11,8 +12,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
|
||||
internal sealed class FakeSignalIngressSink : ISignalIngressSink
|
||||
{
|
||||
private readonly List<PostedSignal> _posted = new List<PostedSignal>();
|
||||
private readonly object _gate = new object();
|
||||
|
||||
public IReadOnlyList<PostedSignal> Posted => _posted;
|
||||
/// <summary>
|
||||
/// Snapshot, not the live list: producers post from timer / worker threads (e.g. the
|
||||
/// parked-dwell tripwire) while tests poll this in a SpinWait loop. Enumerating the live
|
||||
/// list across an Add throws "Collection was modified" — a rare CI-only failure
|
||||
/// (run 32644425162) with no assertion text.
|
||||
/// </summary>
|
||||
public IReadOnlyList<PostedSignal> Posted
|
||||
{
|
||||
get { lock (_gate) return _posted.ToArray(); }
|
||||
}
|
||||
|
||||
/// <summary>When non-null, <see cref="Post"/> throws instead of capturing.</summary>
|
||||
public Exception? ThrowOnPost { get; set; }
|
||||
@@ -27,7 +38,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
|
||||
object? typedPayload = null)
|
||||
{
|
||||
if (ThrowOnPost != null) throw ThrowOnPost;
|
||||
_posted.Add(new PostedSignal(kind, occurredAtUtc, sourceOrigin, evidence, payload, kindSchemaVersion, typedPayload));
|
||||
lock (_gate)
|
||||
_posted.Add(new PostedSignal(kind, occurredAtUtc, sourceOrigin, evidence, payload, kindSchemaVersion, typedPayload));
|
||||
}
|
||||
|
||||
internal sealed class PostedSignal
|
||||
|
||||
+108
@@ -339,6 +339,114 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Runtime
|
||||
Assert.False(StartupEnvironmentProbes.IsNtpWithinTolerance(null!));
|
||||
}
|
||||
|
||||
// ================================================================= DO GroupId auto-set
|
||||
// Builder/fingerprint tests only — TrySetGroupId itself does live registry + ARP I/O and
|
||||
// must never run in a unit test (it would write HKLM on the build machine).
|
||||
|
||||
[Fact]
|
||||
public void BuildDoGroupIdEvent_written_success_is_info_with_full_data()
|
||||
{
|
||||
var result = new DoGroupIdSetResult
|
||||
{
|
||||
Success = true,
|
||||
Written = true,
|
||||
GroupId = "00063065-0000-00a8-b201-001122334455",
|
||||
PreviousValue = null,
|
||||
GatewayIp = "192.168.178.1",
|
||||
GatewayMac = "00-11-22-33-44-55",
|
||||
};
|
||||
|
||||
var evt = StartupEnvironmentProbes.BuildDoGroupIdEvent(Config(), result);
|
||||
|
||||
Assert.Equal("do_group_id_auto_set", evt.EventType);
|
||||
Assert.Equal(EventSeverity.Info, evt.Severity);
|
||||
Assert.Equal("StartupEnvironmentProbes", evt.Source);
|
||||
Assert.Equal("S1", evt.SessionId);
|
||||
Assert.Equal("T1", evt.TenantId);
|
||||
Assert.Contains("00063065-0000-00a8-b201-001122334455", evt.Message);
|
||||
Assert.Equal("192.168.178.1", evt.Data["gatewayIp"]);
|
||||
Assert.Equal("00-11-22-33-44-55", evt.Data["gatewayMac"]);
|
||||
Assert.Equal(true, evt.Data["written"]);
|
||||
Assert.Equal(true, evt.Data["success"]);
|
||||
Assert.Equal(false, evt.Data["skipped"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildDoGroupIdEvent_noop_success_says_already_set()
|
||||
{
|
||||
var result = new DoGroupIdSetResult
|
||||
{
|
||||
Success = true,
|
||||
Written = false,
|
||||
GroupId = "00063065-0000-00a8-b201-001122334455",
|
||||
};
|
||||
|
||||
var evt = StartupEnvironmentProbes.BuildDoGroupIdEvent(Config(), result);
|
||||
|
||||
Assert.Equal(EventSeverity.Info, evt.Severity);
|
||||
Assert.Contains("already set", evt.Message);
|
||||
Assert.Equal(false, evt.Data["written"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildDoGroupIdEvent_skip_is_info_with_reason()
|
||||
{
|
||||
var result = new DoGroupIdSetResult
|
||||
{
|
||||
Skipped = true,
|
||||
SkipReason = "foreign_value_present",
|
||||
PreviousValue = "11111111-2222-3333-4444-555555555555",
|
||||
};
|
||||
|
||||
var evt = StartupEnvironmentProbes.BuildDoGroupIdEvent(Config(), result);
|
||||
|
||||
Assert.Equal(EventSeverity.Info, evt.Severity);
|
||||
Assert.Contains("foreign_value_present", evt.Message);
|
||||
Assert.Equal("foreign_value_present", evt.Data["skipReason"]);
|
||||
Assert.Equal("11111111-2222-3333-4444-555555555555", evt.Data["previousValue"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildDoGroupIdEvent_error_is_warning()
|
||||
{
|
||||
var result = new DoGroupIdSetResult { Error = "readback mismatch" };
|
||||
|
||||
var evt = StartupEnvironmentProbes.BuildDoGroupIdEvent(Config(), result);
|
||||
|
||||
Assert.Equal(EventSeverity.Warning, evt.Severity);
|
||||
Assert.Contains("readback mismatch", evt.Message);
|
||||
Assert.Equal("readback mismatch", evt.Data["error"]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DoGroupIdFingerprint_roundtrips_ownership_marker_for_success()
|
||||
{
|
||||
var fp = StartupEnvironmentProbes.ComputeDoGroupIdFingerprint(new DoGroupIdSetResult
|
||||
{
|
||||
Success = true,
|
||||
GroupId = "00063065-0000-00a8-b201-001122334455",
|
||||
});
|
||||
|
||||
Assert.Equal("written:00063065-0000-00a8-b201-001122334455", fp);
|
||||
Assert.Equal("00063065-0000-00a8-b201-001122334455", StartupEnvironmentProbes.ExtractLastWrittenGroupId(fp));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DoGroupIdFingerprint_skip_carries_no_ownership_and_errors_never_dedupe()
|
||||
{
|
||||
var skipFp = StartupEnvironmentProbes.ComputeDoGroupIdFingerprint(new DoGroupIdSetResult
|
||||
{
|
||||
Skipped = true,
|
||||
SkipReason = "mdm_policy_present",
|
||||
});
|
||||
Assert.Equal("skipped:mdm_policy_present", skipFp);
|
||||
Assert.Null(StartupEnvironmentProbes.ExtractLastWrittenGroupId(skipFp));
|
||||
|
||||
// Errors: no fingerprint → the probe re-runs AND re-emits each failing start.
|
||||
Assert.Null(StartupEnvironmentProbes.ComputeDoGroupIdFingerprint(new DoGroupIdSetResult { Error = "boom" }));
|
||||
Assert.Null(StartupEnvironmentProbes.ExtractLastWrittenGroupId(null));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task RunAsync_skips_geo_timezone_and_ntp_once_all_succeeded_only_power_state_emits()
|
||||
{
|
||||
|
||||
+45
-3
@@ -97,7 +97,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Fails_after_five_exceptions_without_auth_error()
|
||||
public async Task Fails_after_six_exceptions_without_auth_error()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var logger = NewLogger(tmp);
|
||||
@@ -111,8 +111,50 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
|
||||
backoffDelay: _ => Task.CompletedTask);
|
||||
|
||||
Assert.Equal(SessionRegistrationOutcome.Failed, result.Outcome);
|
||||
Assert.Equal(5, apiClient.CallCount);
|
||||
Assert.Contains("transient attempt 5", result.ErrorMessage);
|
||||
Assert.Equal(6, apiClient.CallCount);
|
||||
Assert.Contains("transient attempt 6", result.ErrorMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Waits_for_network_link_before_first_attempt()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var logger = NewLogger(tmp);
|
||||
var config = NewConfig();
|
||||
|
||||
var order = new System.Collections.Generic.List<string>();
|
||||
var apiClient = new FakeApiClient(_ =>
|
||||
{
|
||||
order.Add("register");
|
||||
return new RegisterSessionResponse { Success = true, SessionId = config.SessionId };
|
||||
});
|
||||
|
||||
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
|
||||
apiClient, config, agentVersion: "0.0.0", logger: logger,
|
||||
backoffDelay: _ => Task.CompletedTask,
|
||||
networkLinkWait: _ => { order.Add("link-wait"); return Task.CompletedTask; });
|
||||
|
||||
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
|
||||
Assert.Equal(new[] { "link-wait", "register" }, order);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Network_link_wait_failure_does_not_block_registration()
|
||||
{
|
||||
using var tmp = new TempDirectory();
|
||||
var logger = NewLogger(tmp);
|
||||
var config = NewConfig();
|
||||
|
||||
var apiClient = new FakeApiClient(_ =>
|
||||
new RegisterSessionResponse { Success = true, SessionId = config.SessionId });
|
||||
|
||||
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
|
||||
apiClient, config, agentVersion: "0.0.0", logger: logger,
|
||||
backoffDelay: _ => Task.CompletedTask,
|
||||
networkLinkWait: _ => throw new InvalidOperationException("probe broken"));
|
||||
|
||||
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
|
||||
Assert.Equal(1, apiClient.CallCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -109,6 +109,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
|
||||
/// </summary>
|
||||
public bool EnableTimezoneAutoSet { get; set; } = false;
|
||||
|
||||
/// <summary>
|
||||
/// Whether to set the Delivery Optimization group ID (DOGroupId policy value) from a
|
||||
/// deterministic network fingerprint (default gateway IP + MAC, RealmJoin-compatible).
|
||||
/// Existing DOGroupId/DOGroupIdSource policies are never overwritten.
|
||||
/// Default: false
|
||||
/// </summary>
|
||||
public bool EnableDoGroupIdAutoSet { get; set; } = false;
|
||||
|
||||
/// <summary>
|
||||
/// Continue-Anyway observation mode (operator-set tenant opt-in, delivered via
|
||||
/// remote config): stamped into the esp_config_detected decision-signal payload so
|
||||
@@ -254,6 +262,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
|
||||
/// </summary>
|
||||
public bool UnrestrictedMode { get; set; } = false;
|
||||
|
||||
/// <summary>
|
||||
/// Mirror of <c>AnalyzerConfiguration.EnableRealmJoinWatcher</c> (remote, per tenant).
|
||||
/// Gates the RealmJoin built-in sections of the diagnostics package — the package
|
||||
/// service only sees this configuration, not the remote response. The RealmJoin host
|
||||
/// itself keeps reading the remote Analyzers block. Default: false.
|
||||
/// </summary>
|
||||
public bool EnableRealmJoinWatcher { get; set; } = false;
|
||||
|
||||
/// <summary>
|
||||
/// Sanitized command-line arguments for the agent_started event.
|
||||
/// Secrets (e.g. bootstrap token) are redacted before assignment.
|
||||
|
||||
@@ -94,12 +94,17 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
|
||||
// a tenant admin can clear a previously-applied custom NTP server.
|
||||
agentConfig.NtpServer = remoteConfig.NtpServer;
|
||||
agentConfig.EnableTimezoneAutoSet = remoteConfig.EnableTimezoneAutoSet;
|
||||
agentConfig.EnableDoGroupIdAutoSet = remoteConfig.EnableDoGroupIdAutoSet;
|
||||
|
||||
// ---------------- Diagnostics — remote wins (including DiagnosticsUploadMode="Off"
|
||||
// which must clear any CLI-initial default).
|
||||
agentConfig.DiagnosticsUploadEnabled = remoteConfig.DiagnosticsUploadEnabled;
|
||||
agentConfig.DiagnosticsUploadMode = remoteConfig.DiagnosticsUploadMode;
|
||||
agentConfig.DiagnosticsLogPaths = remoteConfig.DiagnosticsLogPaths ?? new List<DiagnosticsLogPath>();
|
||||
// The diagnostics package gates its RealmJoin sections on the tenant's watcher
|
||||
// toggle but never sees the remote response — mirror the one Analyzers knob here.
|
||||
// The remaining Analyzers flags flow untouched to DefaultComponentFactory.
|
||||
agentConfig.EnableRealmJoinWatcher = remoteConfig.Analyzers?.EnableRealmJoinWatcher ?? false;
|
||||
|
||||
agentConfig.SendTraceEvents = remoteConfig.SendTraceEvents;
|
||||
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Net;
|
||||
using System.Net.Sockets;
|
||||
using System.Runtime.InteropServices;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Interop
|
||||
{
|
||||
/// <summary>
|
||||
/// ARP resolution via <c>iphlpapi!SendARP</c> — resolves the MAC address of an on-link IPv4
|
||||
/// host (in practice: the default gateway). IPv4 only by API design; the OS answers from its
|
||||
/// ARP cache or sends a real ARP request, so the target must be on the local subnet.
|
||||
/// </summary>
|
||||
internal static class ArpNativeMethods
|
||||
{
|
||||
[DllImport("iphlpapi.dll", ExactSpelling = true)]
|
||||
private static extern int SendARP(uint destIP, uint srcIP, byte[] macAddress, ref uint macAddressLength);
|
||||
|
||||
/// <summary>
|
||||
/// Fail-soft MAC resolution. Returns false (with a diagnostic in <paramref name="error"/>)
|
||||
/// for non-IPv4 addresses, SendARP failures, or unexpected reply lengths — never throws.
|
||||
/// </summary>
|
||||
internal static bool TryResolveMac(IPAddress? ipv4Address, out byte[] mac, out string? error)
|
||||
{
|
||||
mac = new byte[6];
|
||||
error = null;
|
||||
try
|
||||
{
|
||||
if (ipv4Address == null || ipv4Address.AddressFamily != AddressFamily.InterNetwork)
|
||||
{
|
||||
error = "not an IPv4 address";
|
||||
return false;
|
||||
}
|
||||
|
||||
// SendARP takes the IPv4 address as a uint in network byte order (first octet in
|
||||
// the lowest byte).
|
||||
var addressBytes = ipv4Address.GetAddressBytes();
|
||||
uint dest = ((uint)addressBytes[3] << 24)
|
||||
| ((uint)addressBytes[2] << 16)
|
||||
| ((uint)addressBytes[1] << 8)
|
||||
| addressBytes[0];
|
||||
|
||||
uint len = (uint)mac.Length;
|
||||
var err = SendARP(dest, 0, mac, ref len);
|
||||
if (err != 0)
|
||||
{
|
||||
error = $"SendARP returned {err}";
|
||||
return false;
|
||||
}
|
||||
if (len != 6)
|
||||
{
|
||||
error = $"SendARP returned unexpected MAC length {len}";
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
error = ex.Message;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
#nullable enable
|
||||
using System.Net;
|
||||
using System.Net.NetworkInformation;
|
||||
using System.Net.Sockets;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Monitoring
|
||||
{
|
||||
/// <summary>
|
||||
/// Shared "which NIC is the device actually using" heuristic: Up, not Loopback/Tunnel,
|
||||
/// carries a non-0.0.0.0 default gateway. Fail-soft — any enumeration error yields null.
|
||||
/// </summary>
|
||||
internal static class NetworkInterfaceLocator
|
||||
{
|
||||
/// <summary>
|
||||
/// Finds the active network interface: Up, not Loopback/Tunnel, has a non-0.0.0.0 gateway.
|
||||
/// </summary>
|
||||
internal static NetworkInterface? FindActiveNetworkInterface()
|
||||
{
|
||||
try
|
||||
{
|
||||
var interfaces = NetworkInterface.GetAllNetworkInterfaces();
|
||||
foreach (var nic in interfaces)
|
||||
{
|
||||
if (nic.OperationalStatus != OperationalStatus.Up)
|
||||
continue;
|
||||
if (nic.NetworkInterfaceType == NetworkInterfaceType.Loopback ||
|
||||
nic.NetworkInterfaceType == NetworkInterfaceType.Tunnel)
|
||||
continue;
|
||||
|
||||
var ipProps = nic.GetIPProperties();
|
||||
foreach (var gw in ipProps.GatewayAddresses)
|
||||
{
|
||||
if (gw.Address.ToString() != "0.0.0.0")
|
||||
return nic;
|
||||
}
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Caller handles null
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>First IPv4 default gateway of the interface (0.0.0.0 excluded), or null.</summary>
|
||||
internal static IPAddress? GetIpv4Gateway(NetworkInterface nic)
|
||||
{
|
||||
try
|
||||
{
|
||||
foreach (var gw in nic.GetIPProperties().GatewayAddresses)
|
||||
{
|
||||
var address = gw?.Address;
|
||||
if (address != null
|
||||
&& address.AddressFamily == AddressFamily.InterNetwork
|
||||
&& address.ToString() != "0.0.0.0")
|
||||
{
|
||||
return address;
|
||||
}
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Caller handles null
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
}
|
||||
+125
-88
@@ -10,7 +10,9 @@ using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
|
||||
using AutopilotMonitor.Agent.V2.Core.Security;
|
||||
using AutopilotMonitor.Shared;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
|
||||
@@ -124,62 +126,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
|
||||
}
|
||||
}
|
||||
|
||||
private readonly string _imeLogFolder;
|
||||
private readonly string _agentLogFolder;
|
||||
private readonly string _agentStateFolder;
|
||||
private readonly string _agentSpoolFolder;
|
||||
private readonly string _agentDataFolder;
|
||||
// Built-in sections come from the Shared catalog (DiagnosticsBuiltInSections.All) — the
|
||||
// same list the backend serves to the portal, so what administrators see is exactly
|
||||
// what this service collects. Test seam: section id → folder; production leaves the
|
||||
// map empty and every section resolves from the catalog's (unexpanded) source folder.
|
||||
private readonly IReadOnlyDictionary<string, string> _sectionFolderOverrides;
|
||||
|
||||
private static readonly string DefaultImeLogFolder =
|
||||
Environment.ExpandEnvironmentVariables(@"%ProgramData%\Microsoft\IntuneManagementExtension\Logs");
|
||||
|
||||
private static readonly string DefaultAgentLogFolder =
|
||||
Environment.ExpandEnvironmentVariables(Constants.LogDirectory);
|
||||
|
||||
private static readonly string DefaultAgentStateFolder =
|
||||
Environment.ExpandEnvironmentVariables(Constants.StateDirectory);
|
||||
|
||||
private static readonly string DefaultAgentSpoolFolder =
|
||||
Environment.ExpandEnvironmentVariables(Constants.SpoolDirectory);
|
||||
|
||||
private static readonly string DefaultAgentDataFolder =
|
||||
Environment.ExpandEnvironmentVariables(@"%ProgramData%\AutopilotMonitor");
|
||||
|
||||
/// <summary>
|
||||
/// File extensions collected from built-in log folders (Agent + IME).
|
||||
/// Covers standard logs, structured data, traces, event logs, and diagnostics archives.
|
||||
/// </summary>
|
||||
private static readonly string[] LogFilePatterns = new[]
|
||||
{
|
||||
"*.log", "*.txt", "*.json", "*.jsonl",
|
||||
"*.etl", "*.evtx", "*.xml", "*.csv", "*.cab"
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Patterns collected from the agent state folder. Adds completion markers on top of
|
||||
/// the standard log patterns so files like <c>enrollment-complete.marker</c> and
|
||||
/// <c>whiteglove-backfill-state.json</c> reach the diagnostics archive.
|
||||
/// </summary>
|
||||
private static readonly string[] StateFilePatterns = new[]
|
||||
{
|
||||
"*.log", "*.txt", "*.json", "*.jsonl",
|
||||
"*.etl", "*.evtx", "*.xml", "*.csv", "*.cab",
|
||||
"*.complete", "*.marker"
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Marker patterns from the top-level data directory. Kept tight on purpose: only
|
||||
/// completion/exit markers, no session.id / bootstrap.json (those are config, not
|
||||
/// forensic state).
|
||||
/// </summary>
|
||||
private static readonly string[] RootMarkerPatterns = new[] { "*.complete", "*.marker" };
|
||||
|
||||
/// <summary>
|
||||
/// Telemetry spool: only the JSON-shaped files. The spool folder may also contain
|
||||
/// quarantine subfolders, so subfolders are excluded to keep the archive focused
|
||||
/// on what is actually pending upload.
|
||||
/// </summary>
|
||||
private static readonly string[] SpoolFilePatterns = new[] { "*.jsonl", "*.json" };
|
||||
// Scenario oracle for DiagnosticsSectionCondition.DevicePreparation. Production reads
|
||||
// the deterministic WDP registry marker (the same gate every other WDP switch uses);
|
||||
// tests inject a constant so they never touch the registry.
|
||||
private readonly Func<bool> _devicePreparationProbe;
|
||||
|
||||
public DiagnosticsPackageService(AgentConfiguration configuration, AgentLogger logger, BackendApiClient apiClient)
|
||||
: this(configuration, logger, apiClient, null, null, null, null, null)
|
||||
@@ -188,7 +144,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
|
||||
|
||||
// Test seam: allows xUnit fixtures to redirect log/state/spool/data folders to a
|
||||
// temp dir without touching real %ProgramData% paths. Production callers always go
|
||||
// through the public ctor.
|
||||
// through the public ctor. The five named overrides map onto the catalog's section
|
||||
// ids; sectionFolderOverrides covers every other section (RealmJoin*, ImeBootstrapper…).
|
||||
internal DiagnosticsPackageService(
|
||||
AgentConfiguration configuration,
|
||||
AgentLogger logger,
|
||||
@@ -197,18 +154,38 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
|
||||
string imeLogFolderOverride,
|
||||
string agentStateFolderOverride,
|
||||
string agentSpoolFolderOverride,
|
||||
string agentDataFolderOverride)
|
||||
string agentDataFolderOverride,
|
||||
IReadOnlyDictionary<string, string> sectionFolderOverrides = null,
|
||||
Func<bool> devicePreparationProbe = null)
|
||||
{
|
||||
_configuration = configuration;
|
||||
_logger = logger;
|
||||
_apiClient = apiClient;
|
||||
_httpClient = new HttpClient { Timeout = TimeSpan.FromMinutes(5) };
|
||||
|
||||
_agentLogFolder = agentLogFolderOverride ?? DefaultAgentLogFolder;
|
||||
_imeLogFolder = imeLogFolderOverride ?? DefaultImeLogFolder;
|
||||
_agentStateFolder = agentStateFolderOverride ?? DefaultAgentStateFolder;
|
||||
_agentSpoolFolder = agentSpoolFolderOverride ?? DefaultAgentSpoolFolder;
|
||||
_agentDataFolder = agentDataFolderOverride ?? DefaultAgentDataFolder;
|
||||
var overrides = new Dictionary<string, string>(StringComparer.Ordinal);
|
||||
void Map(string sectionId, string folder)
|
||||
{
|
||||
if (folder != null) overrides[sectionId] = folder;
|
||||
}
|
||||
Map("AgentLogs", agentLogFolderOverride);
|
||||
Map("ImeLogs", imeLogFolderOverride);
|
||||
Map("AgentState", agentStateFolderOverride);
|
||||
Map("AgentSpool", agentSpoolFolderOverride);
|
||||
Map("AgentMarkers", agentDataFolderOverride);
|
||||
if (sectionFolderOverrides != null)
|
||||
{
|
||||
foreach (var kv in sectionFolderOverrides)
|
||||
overrides[kv.Key] = kv.Value;
|
||||
}
|
||||
_sectionFolderOverrides = overrides;
|
||||
|
||||
// Tests that pass folder overrides but no probe must stay registry-free: a seam
|
||||
// caller gets "not WDP" by default, the production ctor gets the real marker.
|
||||
_devicePreparationProbe = devicePreparationProbe
|
||||
?? (overrides.Count == 0
|
||||
? (Func<bool>)EnrollmentRegistryDetector.IsDeterministicDevicePreparation
|
||||
: () => false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -370,33 +347,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
|
||||
// 1. sessioninfo.txt — informational metadata, not subject to caps.
|
||||
AddSessionInfo(archive, enrollmentSucceeded);
|
||||
|
||||
// 2. Agent logs
|
||||
foreach (var pattern in LogFilePatterns)
|
||||
AddLogFiles(archive, _agentLogFolder, "AgentLogs", pattern, tracker);
|
||||
// 2. Built-in sections — ONE catalog shared with the backend (the portal
|
||||
// shows exactly this list): agent logs, IME logs (+ the Device Preparation
|
||||
// bootstrapper event log on WDP), agent state (recursive: `.quarantine`
|
||||
// and `.part1-<utc>/` buckets ride along), telemetry spool, top-level
|
||||
// markers, and the RealmJoin logs behind the tenant's watcher toggle.
|
||||
// The scenario probe is a registry read — evaluate it once per build.
|
||||
var isDevicePreparation = ProbeDevicePreparation();
|
||||
ManifestLine($"SCENARIO: devicePreparation={isDevicePreparation} realmJoinWatcher={_configuration.EnableRealmJoinWatcher}");
|
||||
foreach (var section in DiagnosticsBuiltInSections.All)
|
||||
AddBuiltInSection(archive, section, tracker, isDevicePreparation);
|
||||
|
||||
// 3. IME logs (all relevant files in the IME log folder)
|
||||
foreach (var pattern in LogFilePatterns)
|
||||
AddLogFiles(archive, _imeLogFolder, "ImeLogs", pattern, tracker);
|
||||
|
||||
// 4. Agent state: snapshot, journal, signal-log, ime-tracker-state,
|
||||
// enrollment-complete.marker, final-status.json, whiteglove-backfill-state.json.
|
||||
// Recursive to include `.quarantine` subfolders AND `.part1-<utc>/` buckets
|
||||
// that StateArchiver creates on WG Part-2 boot — both ride along under
|
||||
// AgentState/ via the same recursive walk.
|
||||
foreach (var pattern in StateFilePatterns)
|
||||
AddLogFiles(archive, _agentStateFolder, "AgentState", pattern, tracker, includeSubfolders: true);
|
||||
|
||||
// 5. Telemetry spool — pending uploads + upload cursor (forensic: what didn't ship?)
|
||||
foreach (var pattern in SpoolFilePatterns)
|
||||
AddLogFiles(archive, _agentSpoolFolder, "AgentSpool", pattern, tracker, includeSubfolders: false);
|
||||
|
||||
// 6. Top-level markers (whiteglove.complete, clean-exit marker) — Part-1/Part-2
|
||||
// bridge + ghost-restart guard. Top-directory only to avoid re-zipping
|
||||
// State/Spool/Logs/Agent which already have their own sections.
|
||||
foreach (var pattern in RootMarkerPatterns)
|
||||
AddLogFiles(archive, _agentDataFolder, "AgentMarkers", pattern, tracker, includeSubfolders: false);
|
||||
|
||||
// 7. Configured additional log paths (global + tenant, validated by guards)
|
||||
// 3. Configured additional log paths (global + tenant, validated by guards)
|
||||
foreach (var entry in _configuration.DiagnosticsLogPaths ?? new System.Collections.Generic.List<Shared.Models.DiagnosticsLogPath>())
|
||||
{
|
||||
// Resolve %LOGGED_ON_USER_PROFILE% token and get profile path for guard exception
|
||||
@@ -447,6 +409,81 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
|
||||
}
|
||||
}
|
||||
|
||||
private bool ProbeDevicePreparation()
|
||||
{
|
||||
try
|
||||
{
|
||||
return _devicePreparationProbe();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// Fail toward Classic like every other WDP gate: the evtx export is skipped,
|
||||
// never the whole package.
|
||||
_logger.Warning($"Device Preparation probe failed — treating as Classic: {ex.Message}");
|
||||
ManifestLine($"SCENARIO PROBE FAILED (treated as Classic): {ex.Message}");
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pure gate: is this built-in section collected under the given configuration and
|
||||
/// enrollment scenario? Unknown conditions (a newer Shared catalog than this agent)
|
||||
/// fail closed.
|
||||
/// </summary>
|
||||
internal static bool IsSectionActive(DiagnosticsBuiltInSection section, AgentConfiguration configuration, bool isDevicePreparation)
|
||||
{
|
||||
switch (section.Condition)
|
||||
{
|
||||
case DiagnosticsSectionCondition.Always:
|
||||
return true;
|
||||
case DiagnosticsSectionCondition.RealmJoinWatcher:
|
||||
return configuration.EnableRealmJoinWatcher;
|
||||
case DiagnosticsSectionCondition.DevicePreparation:
|
||||
return isDevicePreparation;
|
||||
default:
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private static string DescribeInactiveCondition(DiagnosticsSectionCondition condition)
|
||||
{
|
||||
switch (condition)
|
||||
{
|
||||
case DiagnosticsSectionCondition.RealmJoinWatcher:
|
||||
return "RealmJoin Watcher disabled";
|
||||
case DiagnosticsSectionCondition.DevicePreparation:
|
||||
return "not a Device Preparation enrollment";
|
||||
default:
|
||||
return $"unknown condition {condition}";
|
||||
}
|
||||
}
|
||||
|
||||
private void AddBuiltInSection(ZipArchive archive, DiagnosticsBuiltInSection section, BudgetTracker tracker, bool isDevicePreparation)
|
||||
{
|
||||
if (!IsSectionActive(section, _configuration, isDevicePreparation))
|
||||
{
|
||||
ManifestLine($"BUILT-IN SKIPPED ({DescribeInactiveCondition(section.Condition)}): {section.Id}");
|
||||
return;
|
||||
}
|
||||
|
||||
// Override (test seam) or catalog source folder with %ProgramData% /
|
||||
// %LOGGED_ON_USER_PROFILE% expanded; null means the token is present but no
|
||||
// interactive user has been detected yet — same skip as configured paths.
|
||||
var folder = _sectionFolderOverrides.TryGetValue(section.Id, out var overrideFolder)
|
||||
? overrideFolder
|
||||
: UserProfileResolver.ExpandCustomTokens(section.SourceFolder);
|
||||
if (folder == null)
|
||||
{
|
||||
_logger.Warning($"Built-in diagnostics section skipped (no user session for token): {section.Id}");
|
||||
ManifestLine($"BUILT-IN SKIPPED (no user session for token): {section.Id} '{section.SourceFolder}'");
|
||||
return;
|
||||
}
|
||||
|
||||
ManifestLine($"BUILT-IN: {section.Id} -> folder='{folder}' zip='{section.ZipFolder}' recursive={section.IncludeSubfolders}");
|
||||
foreach (var pattern in section.Patterns)
|
||||
AddLogFiles(archive, folder, section.ZipFolder, pattern, tracker, section.IncludeSubfolders);
|
||||
}
|
||||
|
||||
private void AddSessionInfo(ZipArchive archive, bool? enrollmentSucceeded)
|
||||
{
|
||||
var sb = new StringBuilder();
|
||||
|
||||
@@ -20,7 +20,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
|
||||
/// </summary>
|
||||
public static class UserProfileResolver
|
||||
{
|
||||
public const string Token = "%LOGGED_ON_USER_PROFILE%";
|
||||
// Spelled once, in Shared: the built-in diagnostics catalog uses the same token.
|
||||
public const string Token = Shared.Models.DiagnosticsBuiltInSections.UserProfileToken;
|
||||
|
||||
private static readonly object Lock = new object();
|
||||
private static bool _resolved;
|
||||
|
||||
+2
-30
@@ -922,36 +922,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Finds the active network interface: Up, not Loopback/Tunnel, has a non-0.0.0.0 gateway.
|
||||
/// </summary>
|
||||
private static NetworkInterface FindActiveNetworkInterface()
|
||||
{
|
||||
try
|
||||
{
|
||||
var interfaces = NetworkInterface.GetAllNetworkInterfaces();
|
||||
foreach (var nic in interfaces)
|
||||
{
|
||||
if (nic.OperationalStatus != OperationalStatus.Up)
|
||||
continue;
|
||||
if (nic.NetworkInterfaceType == NetworkInterfaceType.Loopback ||
|
||||
nic.NetworkInterfaceType == NetworkInterfaceType.Tunnel)
|
||||
continue;
|
||||
|
||||
var ipProps = nic.GetIPProperties();
|
||||
foreach (var gw in ipProps.GatewayAddresses)
|
||||
{
|
||||
if (gw.Address.ToString() != "0.0.0.0")
|
||||
return nic;
|
||||
}
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Caller handles null
|
||||
}
|
||||
return null;
|
||||
}
|
||||
private static NetworkInterface FindActiveNetworkInterface() =>
|
||||
Monitoring.NetworkInterfaceLocator.FindActiveNetworkInterface();
|
||||
|
||||
/// <summary>
|
||||
/// Collects WiFi signal info via the native WLAN API (<see cref="WifiInfoReader"/>,
|
||||
|
||||
+2
-30
@@ -420,36 +420,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
|
||||
// Helpers
|
||||
// -------------------------------------------------------------------
|
||||
|
||||
/// <summary>
|
||||
/// Finds the active network interface: Up, not Loopback/Tunnel, has a non-0.0.0.0 gateway.
|
||||
/// </summary>
|
||||
private static NetworkInterface FindActiveNetworkInterface()
|
||||
{
|
||||
try
|
||||
{
|
||||
var interfaces = NetworkInterface.GetAllNetworkInterfaces();
|
||||
foreach (var nic in interfaces)
|
||||
{
|
||||
if (nic.OperationalStatus != OperationalStatus.Up)
|
||||
continue;
|
||||
if (nic.NetworkInterfaceType == NetworkInterfaceType.Loopback ||
|
||||
nic.NetworkInterfaceType == NetworkInterfaceType.Tunnel)
|
||||
continue;
|
||||
|
||||
var ipProps = nic.GetIPProperties();
|
||||
foreach (var gw in ipProps.GatewayAddresses)
|
||||
{
|
||||
if (gw.Address.ToString() != "0.0.0.0")
|
||||
return nic;
|
||||
}
|
||||
}
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Caller handles null
|
||||
}
|
||||
return null;
|
||||
}
|
||||
private static NetworkInterface FindActiveNetworkInterface() =>
|
||||
NetworkInterfaceLocator.FindActiveNetworkInterface();
|
||||
|
||||
private static string BuildChangeMessage(NetworkStateSnapshot before, NetworkStateSnapshot after)
|
||||
{
|
||||
|
||||
@@ -0,0 +1,217 @@
|
||||
#nullable enable
|
||||
using System;
|
||||
using System.Net;
|
||||
using System.Net.Sockets;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
|
||||
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic;
|
||||
using Microsoft.Win32;
|
||||
|
||||
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
|
||||
{
|
||||
public class DoGroupIdSetResult
|
||||
{
|
||||
/// <summary>The DOGroupId value is the computed one after this run (written or already there).</summary>
|
||||
public bool Success { get; set; }
|
||||
|
||||
/// <summary>True when this run actually wrote the registry value (false for the idempotent no-op).</summary>
|
||||
public bool Written { get; set; }
|
||||
|
||||
/// <summary>Deliberate non-action (existing foreign policy/value, no usable network) — not an error.</summary>
|
||||
public bool Skipped { get; set; }
|
||||
public string? SkipReason { get; set; }
|
||||
|
||||
public string? GroupId { get; set; }
|
||||
public string? PreviousValue { get; set; }
|
||||
public string? GatewayIp { get; set; }
|
||||
public string? GatewayMac { get; set; }
|
||||
public string? Error { get; set; }
|
||||
}
|
||||
|
||||
internal enum DoGroupIdAction
|
||||
{
|
||||
Write,
|
||||
NoOp,
|
||||
SkipForeign,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Sets the Delivery Optimization group ID policy value (<c>DOGroupId</c> under the GPO
|
||||
/// DeliveryOptimization key) from a network fingerprint: a deterministic GUID derived from the
|
||||
/// default gateway's IPv4 address and MAC address, so all devices behind the same gateway land
|
||||
/// in the same DO peering group. The byte layout is RealmJoin-compatible — devices managed by
|
||||
/// RealmJoin on the same network compute the identical GUID and peer with ours.
|
||||
/// <para>
|
||||
/// Deliberately conservative where RealmJoin is not: no random-GUID fallback (a per-device
|
||||
/// random ID would isolate the device in a one-member group — worse than the OS default
|
||||
/// grouping), and existing foreign configuration always wins: a DOGroupIdSource policy, an
|
||||
/// MDM-managed DOGroupId, or a GPO value we did not write ourselves all result in a skip.
|
||||
/// Never throws.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public static class DoGroupIdService
|
||||
{
|
||||
internal const string GroupIdValueName = "DOGroupId";
|
||||
internal const string GroupIdSourceValueName = "DOGroupIdSource";
|
||||
|
||||
// RealmJoin's fingerprint-GUID prefix — part of the compatible byte layout, do not change.
|
||||
private static readonly byte[] GuidPrefix = { 101, 48, 6 };
|
||||
|
||||
/// <summary>
|
||||
/// Computes and applies the fingerprint GroupId. <paramref name="lastWrittenGroupId"/> is
|
||||
/// the GUID a previous agent run wrote (null when none is recorded) — it lets us update our
|
||||
/// own value after a gateway change while never touching a foreign one.
|
||||
/// </summary>
|
||||
public static DoGroupIdSetResult TrySetGroupId(string? lastWrittenGroupId, AgentLogger logger)
|
||||
{
|
||||
var result = new DoGroupIdSetResult();
|
||||
|
||||
try
|
||||
{
|
||||
// ---- Network fingerprint -------------------------------------------------
|
||||
var nic = NetworkInterfaceLocator.FindActiveNetworkInterface();
|
||||
var gateway = nic == null ? null : NetworkInterfaceLocator.GetIpv4Gateway(nic);
|
||||
if (gateway == null)
|
||||
{
|
||||
result.Skipped = true;
|
||||
result.SkipReason = "no_ipv4_gateway";
|
||||
logger.Info("DO GroupId auto-set: no active interface with an IPv4 default gateway — skipping.");
|
||||
return result;
|
||||
}
|
||||
result.GatewayIp = gateway.ToString();
|
||||
|
||||
if (!ArpNativeMethods.TryResolveMac(gateway, out var mac, out var arpError))
|
||||
{
|
||||
result.Skipped = true;
|
||||
result.SkipReason = "arp_failed";
|
||||
result.Error = arpError;
|
||||
logger.Warning($"DO GroupId auto-set: could not resolve gateway MAC ({arpError}) — skipping.");
|
||||
return result;
|
||||
}
|
||||
result.GatewayMac = BitConverter.ToString(mac);
|
||||
|
||||
var computed = ComputeGroupId(gateway.GetAddressBytes(), mac).ToString();
|
||||
result.GroupId = computed;
|
||||
|
||||
// ---- Foreign-configuration checks ----------------------------------------
|
||||
// DOGroupIdSource (either store) selects a dynamic group source — a fingerprint
|
||||
// GroupId would fight it. An MDM-managed DOGroupId means Intune owns the setting.
|
||||
if (ReadPolicyValue(DoThrottlePolicyReader.GpoKeyPath, GroupIdSourceValueName) != null
|
||||
|| ReadPolicyValue(DoThrottlePolicyReader.MdmKeyPath, GroupIdSourceValueName) != null)
|
||||
{
|
||||
result.Skipped = true;
|
||||
result.SkipReason = "group_id_source_policy";
|
||||
logger.Info("DO GroupId auto-set: DOGroupIdSource policy is configured — skipping.");
|
||||
return result;
|
||||
}
|
||||
if (ReadPolicyValue(DoThrottlePolicyReader.MdmKeyPath, GroupIdValueName) != null)
|
||||
{
|
||||
result.Skipped = true;
|
||||
result.SkipReason = "mdm_policy_present";
|
||||
logger.Info("DO GroupId auto-set: DOGroupId is managed via MDM policy — skipping.");
|
||||
return result;
|
||||
}
|
||||
|
||||
// ---- Ownership decision ---------------------------------------------------
|
||||
var existing = ReadPolicyValue(DoThrottlePolicyReader.GpoKeyPath, GroupIdValueName) as string;
|
||||
result.PreviousValue = existing;
|
||||
|
||||
switch (DecideAction(existing, computed, lastWrittenGroupId))
|
||||
{
|
||||
case DoGroupIdAction.NoOp:
|
||||
result.Success = true;
|
||||
logger.Info($"DO GroupId auto-set: already set to {computed} — nothing to do.");
|
||||
return result;
|
||||
|
||||
case DoGroupIdAction.SkipForeign:
|
||||
result.Skipped = true;
|
||||
result.SkipReason = "foreign_value_present";
|
||||
logger.Info($"DO GroupId auto-set: existing DOGroupId '{existing}' was not written by us — skipping.");
|
||||
return result;
|
||||
}
|
||||
|
||||
// ---- Write + readback -----------------------------------------------------
|
||||
using (var hklm = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, RegistryView.Registry64))
|
||||
using (var key = hklm.CreateSubKey(DoThrottlePolicyReader.GpoKeyPath))
|
||||
{
|
||||
key.SetValue(GroupIdValueName, computed, RegistryValueKind.String);
|
||||
}
|
||||
|
||||
var readback = ReadPolicyValue(DoThrottlePolicyReader.GpoKeyPath, GroupIdValueName) as string;
|
||||
if (!string.Equals(readback, computed, StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
result.Error = $"readback mismatch: wrote '{computed}', read '{readback ?? "<null>"}'";
|
||||
logger.Warning($"DO GroupId auto-set failed: {result.Error}");
|
||||
return result;
|
||||
}
|
||||
|
||||
result.Success = true;
|
||||
result.Written = true;
|
||||
logger.Info($"DO GroupId auto-set: DOGroupId set to {computed} (gateway {result.GatewayIp}, previous: {existing ?? "<none>"})");
|
||||
return result;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
result.Error = ex.Message;
|
||||
logger.Warning($"DO GroupId auto-set failed: {ex.Message}");
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// RealmJoin-compatible fingerprint GUID: 16 bytes laid out as
|
||||
/// prefix {101,48,6} + {0,0,0} + {ip[1],0,ip[2],ip[3]} + mac[6], passed to
|
||||
/// <c>new Guid(byte[])</c>. The IP byte shuffle mirrors RealmJoin's "swap for
|
||||
/// new Guid()" step — the layout must stay bit-identical for cross-product peering.
|
||||
/// </summary>
|
||||
internal static Guid ComputeGroupId(byte[] ipv4Bytes, byte[] macBytes)
|
||||
{
|
||||
if (ipv4Bytes == null || ipv4Bytes.Length != 4)
|
||||
throw new ArgumentException("IPv4 address bytes required", nameof(ipv4Bytes));
|
||||
if (macBytes == null || macBytes.Length != 6)
|
||||
throw new ArgumentException("6-byte MAC required", nameof(macBytes));
|
||||
|
||||
var bytes = new byte[16];
|
||||
Array.Copy(GuidPrefix, 0, bytes, 0, GuidPrefix.Length);
|
||||
// bytes[3..5] stay zero (RealmJoin's zero padding)
|
||||
bytes[6] = ipv4Bytes[1];
|
||||
// bytes[7] stays zero
|
||||
bytes[8] = ipv4Bytes[2];
|
||||
bytes[9] = ipv4Bytes[3];
|
||||
Array.Copy(macBytes, 0, bytes, 10, macBytes.Length);
|
||||
return new Guid(bytes);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pure ownership decision. GUID comparison tolerates case and braces so a value written
|
||||
/// by another tool in "{...}" form still matches semantically.
|
||||
/// </summary>
|
||||
internal static DoGroupIdAction DecideAction(string? existing, string computed, string? lastWrittenByUs)
|
||||
{
|
||||
if (string.IsNullOrEmpty(existing))
|
||||
return DoGroupIdAction.Write;
|
||||
if (GuidEquals(existing, computed))
|
||||
return DoGroupIdAction.NoOp;
|
||||
if (GuidEquals(existing, lastWrittenByUs))
|
||||
return DoGroupIdAction.Write; // our own stale value — gateway changed
|
||||
return DoGroupIdAction.SkipForeign;
|
||||
}
|
||||
|
||||
private static bool GuidEquals(string? a, string? b)
|
||||
{
|
||||
if (string.IsNullOrEmpty(a) || string.IsNullOrEmpty(b)) return false;
|
||||
return Guid.TryParse(a, out var ga) && Guid.TryParse(b, out var gb) && ga == gb;
|
||||
}
|
||||
|
||||
private static object? ReadPolicyValue(string keyPath, string valueName)
|
||||
{
|
||||
// Registry64 explicitly — SOFTWARE\Policies is WOW64-redirected, and the policy
|
||||
// stores live in the 64-bit view regardless of the agent's bitness.
|
||||
using (var hklm = RegistryKey.OpenBaseKey(RegistryHive.LocalMachine, RegistryView.Registry64))
|
||||
using (var key = hklm.OpenSubKey(keyPath))
|
||||
{
|
||||
return key?.GetValue(valueName);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -124,6 +124,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Persistence
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Read-only peek of the key's current fingerprint (possibly restored from a previous
|
||||
/// agent run), or null when none is recorded. Like <see cref="HasFingerprint"/> this makes
|
||||
/// NO claim — use it when the fingerprint carries state the caller needs to read back
|
||||
/// (e.g. the DO GroupId ownership marker).
|
||||
/// </summary>
|
||||
public string? GetFingerprint(string key)
|
||||
{
|
||||
if (string.IsNullOrEmpty(key)) return null;
|
||||
lock (_lock)
|
||||
{
|
||||
return _entries.TryGetValue(key, out var entry) ? entry.Fingerprint : null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Retry-until-success: true when <see cref="MarkSucceeded"/> latched this key in any run.</summary>
|
||||
public bool AlreadySucceeded(string key)
|
||||
{
|
||||
|
||||
@@ -25,6 +25,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Runtime
|
||||
/// Always emitted — offset >60s is Warning, smaller is Info, failure is Warning.</item>
|
||||
/// <item><b>power_state_check</b> — AC vs. battery snapshot via Win32 <c>GetSystemPowerStatus</c>.
|
||||
/// On battery with <80% charge is Warning, otherwise Info, probe failure is Warning.</item>
|
||||
/// <item><b>do_group_id_auto_set</b> — only when <c>EnableDoGroupIdAutoSet=true</c>. Sets the
|
||||
/// Delivery Optimization <c>DOGroupId</c> policy value from a network fingerprint via
|
||||
/// <see cref="DoGroupIdService"/>; skips (Info) rather than fight foreign policy values.</item>
|
||||
/// </list>
|
||||
/// <para>
|
||||
/// Each probe runs best-effort: failures are logged + optionally emitted as a Warning event,
|
||||
@@ -78,6 +81,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Runtime
|
||||
await RunNtpCheck(configuration, logger, post, gate).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
if (configuration.EnableDoGroupIdAutoSet)
|
||||
RunDoGroupIdAutoSet(configuration, logger, post, gate);
|
||||
|
||||
RunPowerStateCheck(configuration, logger, post);
|
||||
}
|
||||
|
||||
@@ -331,6 +337,99 @@ namespace AutopilotMonitor.Agent.V2.Core.Runtime
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// One-shot DO GroupId apply with change-based restart dedup: the gate fingerprint doubles
|
||||
/// as the ownership marker ("written:<guid>" records the GUID we last wrote, so a later
|
||||
/// run can distinguish our own stale value from a foreign one after a gateway change).
|
||||
/// Unchanged outcomes (same GUID, same skip reason) re-run silently without a duplicate
|
||||
/// event; errors never claim a fingerprint so each failing start keeps its retry trail.
|
||||
/// </summary>
|
||||
private static void RunDoGroupIdAutoSet(
|
||||
AgentConfiguration configuration,
|
||||
AgentLogger logger,
|
||||
Orchestration.InformationalEventPost post,
|
||||
Persistence.StartupEventGate? gate)
|
||||
{
|
||||
try
|
||||
{
|
||||
var lastWritten = ExtractLastWrittenGroupId(gate?.GetFingerprint(Constants.EventTypes.DoGroupIdAutoSet));
|
||||
var result = DoGroupIdService.TrySetGroupId(lastWritten, logger);
|
||||
|
||||
var fingerprint = ComputeDoGroupIdFingerprint(result);
|
||||
if (fingerprint != null && gate != null
|
||||
&& !gate.ShouldEmit(Constants.EventTypes.DoGroupIdAutoSet, fingerprint))
|
||||
{
|
||||
logger.Debug("StartupEnvironmentProbes: DO GroupId outcome unchanged — skipping duplicate event (restart dedup).");
|
||||
return;
|
||||
}
|
||||
|
||||
SafeEmit(post, logger, BuildDoGroupIdEvent(configuration, result));
|
||||
|
||||
if (fingerprint != null)
|
||||
gate?.MarkEmitted(Constants.EventTypes.DoGroupIdAutoSet);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
logger.Warning($"StartupEnvironmentProbes: DO GroupId auto-set threw: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
internal const string DoGroupIdWrittenFingerprintPrefix = "written:";
|
||||
|
||||
/// <summary>
|
||||
/// Success (written or already correct) fingerprints as "written:<guid>", a skip as
|
||||
/// "skipped:<reason>"; errors return null so they are never deduped.
|
||||
/// </summary>
|
||||
internal static string? ComputeDoGroupIdFingerprint(DoGroupIdSetResult result)
|
||||
{
|
||||
if (result.Success) return DoGroupIdWrittenFingerprintPrefix + result.GroupId;
|
||||
if (result.Skipped) return "skipped:" + result.SkipReason;
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>Ownership marker readback: the GUID from a "written:" fingerprint, else null.</summary>
|
||||
internal static string? ExtractLastWrittenGroupId(string? fingerprint) =>
|
||||
fingerprint != null && fingerprint.StartsWith(DoGroupIdWrittenFingerprintPrefix, StringComparison.Ordinal)
|
||||
? fingerprint.Substring(DoGroupIdWrittenFingerprintPrefix.Length)
|
||||
: null;
|
||||
|
||||
internal static EnrollmentEvent BuildDoGroupIdEvent(AgentConfiguration configuration, DoGroupIdSetResult result)
|
||||
{
|
||||
string message;
|
||||
if (result.Success && result.Written)
|
||||
message = $"Delivery Optimization GroupId set to {result.GroupId} (gateway {result.GatewayIp})";
|
||||
else if (result.Success)
|
||||
message = $"Delivery Optimization GroupId already set to {result.GroupId}";
|
||||
else if (result.Skipped)
|
||||
message = $"Delivery Optimization GroupId not set: {result.SkipReason}";
|
||||
else
|
||||
message = $"Delivery Optimization GroupId auto-set failed: {result.Error}";
|
||||
|
||||
return new EnrollmentEvent
|
||||
{
|
||||
SessionId = configuration.SessionId,
|
||||
TenantId = configuration.TenantId,
|
||||
EventType = Constants.EventTypes.DoGroupIdAutoSet,
|
||||
Severity = result.Success || result.Skipped ? EventSeverity.Info : EventSeverity.Warning,
|
||||
Source = "StartupEnvironmentProbes",
|
||||
Phase = EnrollmentPhase.Unknown,
|
||||
Timestamp = DateTime.UtcNow,
|
||||
Message = message,
|
||||
Data = new Dictionary<string, object>
|
||||
{
|
||||
{ "groupId", result.GroupId ?? "" },
|
||||
{ "previousValue", result.PreviousValue ?? "" },
|
||||
{ "gatewayIp", result.GatewayIp ?? "" },
|
||||
{ "gatewayMac", result.GatewayMac ?? "" },
|
||||
{ "written", result.Written },
|
||||
{ "skipped", result.Skipped },
|
||||
{ "skipReason", result.SkipReason ?? "" },
|
||||
{ "success", result.Success },
|
||||
{ "error", result.Error ?? "" },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
private static void RunPowerStateCheck(
|
||||
AgentConfiguration configuration,
|
||||
AgentLogger logger,
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using System;
|
||||
using System.Net.NetworkInformation;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Agent.V2.Core.Configuration;
|
||||
using AutopilotMonitor.Agent.V2.Core.Logging;
|
||||
@@ -19,7 +20,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
/// <b>Retry contract (V1 <c>MonitoringService.RegisterSessionAsync</c>):</b>
|
||||
/// </para>
|
||||
/// <list type="bullet">
|
||||
/// <item>5 attempts total; between attempts <c>2^attempt</c> seconds (2s, 4s, 8s, 16s).</item>
|
||||
/// <item>Boot-time NIC grace before the first attempt: when no network link is up yet
|
||||
/// (BootTrigger relaunch after a mid-enrollment reboot, Wi-Fi still associating) wait
|
||||
/// up to <see cref="NetworkLinkWaitMax"/> for the link-level signal. Best-effort and
|
||||
/// free on the normal path — a live link returns immediately.</item>
|
||||
/// <item>6 attempts total; between attempts <c>2^attempt</c> seconds (2s, 4s, 8s, 16s, 32s —
|
||||
/// ~62 s of backoff). The original V1 budget of 5 attempts / ~30 s was too short for the
|
||||
/// reboot-relaunch on Wi-Fi kiosks: the relaunched agent gave up and exited 7 before
|
||||
/// the network came back, leaving the session silent forever (tenant aebdce78,
|
||||
/// 2026-08-23 audit). A relaunch that misses here is lost — nothing retries later.</item>
|
||||
/// <item>On <c>response.Success == true</c> → stop immediately.</item>
|
||||
/// <item>On <see cref="BackendAuthException"/> (401/403) → report to the
|
||||
/// <see cref="AuthFailureTracker"/> (which fires the first-failure distress + may trip
|
||||
@@ -37,16 +46,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
/// </summary>
|
||||
public static class SessionRegistrationHelper
|
||||
{
|
||||
private const int MaxAttempts = 5;
|
||||
private const int MaxAttempts = 6;
|
||||
|
||||
/// <summary>Upper bound for the pre-registration link wait (see class remarks).</summary>
|
||||
internal static readonly TimeSpan NetworkLinkWaitMax = TimeSpan.FromSeconds(15);
|
||||
|
||||
/// <summary>
|
||||
/// Executes the 5-retry register-session handshake against the backend. Returns the
|
||||
/// Executes the retrying register-session handshake against the backend. Returns the
|
||||
/// response + classified outcome. Never throws.
|
||||
/// </summary>
|
||||
/// <param name="backoffDelay">
|
||||
/// Optional delay-provider used by tests to avoid real-time waits. Production callers
|
||||
/// pass <c>null</c> → V1 parity <c>Task.Delay(2^attempt * 1000)</c> between attempts.
|
||||
/// </param>
|
||||
/// <param name="networkLinkWait">
|
||||
/// Optional pre-registration link wait used by tests. Production callers pass
|
||||
/// <c>null</c> → <see cref="WaitForNetworkLinkAsync"/> (bounded by <see cref="NetworkLinkWaitMax"/>).
|
||||
/// </param>
|
||||
public static async Task<SessionRegistrationResult> RegisterWithRetryAsync(
|
||||
BackendApiClient apiClient,
|
||||
AgentConfiguration agentConfig,
|
||||
@@ -56,14 +72,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
EmergencyReporter emergencyReporter = null,
|
||||
Func<int, Task> backoffDelay = null,
|
||||
Func<Exception, Task> onTerminalTransportFailure = null,
|
||||
(string Manufacturer, string Model, string SerialNumber)? deviceHardware = null)
|
||||
(string Manufacturer, string Model, string SerialNumber)? deviceHardware = null,
|
||||
Func<AgentLogger, Task> networkLinkWait = null)
|
||||
{
|
||||
if (apiClient == null) throw new ArgumentNullException(nameof(apiClient));
|
||||
if (agentConfig == null) throw new ArgumentNullException(nameof(agentConfig));
|
||||
if (logger == null) throw new ArgumentNullException(nameof(logger));
|
||||
|
||||
backoffDelay ??= DefaultBackoffDelay;
|
||||
networkLinkWait ??= WaitForNetworkLinkAsync;
|
||||
var registration = BuildRegistration(agentConfig, agentVersion, deviceHardware);
|
||||
|
||||
try { await networkLinkWait(logger).ConfigureAwait(false); }
|
||||
catch (Exception ex) { logger.Debug($"Pre-registration network wait failed ({ex.Message}) — registering anyway."); }
|
||||
string lastError = null;
|
||||
Exception lastException = null;
|
||||
|
||||
@@ -113,7 +134,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
|
||||
if (attempt < MaxAttempts)
|
||||
{
|
||||
var delaySeconds = (int)Math.Pow(2, attempt); // 2, 4, 8, 16
|
||||
var delaySeconds = (int)Math.Pow(2, attempt); // 2, 4, 8, 16, 32
|
||||
logger.Info($"Retrying session registration in {delaySeconds}s");
|
||||
await backoffDelay(attempt).ConfigureAwait(false);
|
||||
}
|
||||
@@ -143,6 +164,31 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
private static Task DefaultBackoffDelay(int attempt)
|
||||
=> Task.Delay(((int)Math.Pow(2, attempt)) * 1000);
|
||||
|
||||
/// <summary>
|
||||
/// Boot-time NIC grace (same pattern as the emergency-break reporter): polls the cheap
|
||||
/// link-level signal once per second for at most <see cref="NetworkLinkWaitMax"/>. It
|
||||
/// cannot prove backend reachability — the retry loop handles that — it only keeps the
|
||||
/// first attempts from being burned into a link that is provably still down. Probe
|
||||
/// errors end the wait, never the registration.
|
||||
/// </summary>
|
||||
internal static async Task WaitForNetworkLinkAsync(AgentLogger logger)
|
||||
{
|
||||
if (NetworkInterface.GetIsNetworkAvailable()) return;
|
||||
|
||||
logger?.Info($"Session registration: no network link yet — waiting up to {NetworkLinkWaitMax.TotalSeconds:F0}s.");
|
||||
var deadline = DateTime.UtcNow + NetworkLinkWaitMax;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
await Task.Delay(1000).ConfigureAwait(false);
|
||||
if (NetworkInterface.GetIsNetworkAvailable())
|
||||
{
|
||||
logger?.Info("Session registration: network link is up.");
|
||||
return;
|
||||
}
|
||||
}
|
||||
logger?.Warning($"Session registration: still no network link after {NetworkLinkWaitMax.TotalSeconds:F0}s — attempting anyway.");
|
||||
}
|
||||
|
||||
private static SessionRegistration BuildRegistration(
|
||||
AgentConfiguration agentConfig,
|
||||
string agentVersion,
|
||||
@@ -194,7 +240,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
|
||||
/// <summary>401/403 from the backend — cert/token rejected. No retry was attempted.</summary>
|
||||
AuthFailed = 1,
|
||||
|
||||
/// <summary>All 5 attempts failed with non-auth errors (network, 5xx, malformed response).</summary>
|
||||
/// <summary>All attempts failed with non-auth errors (network, 5xx, malformed response).</summary>
|
||||
Failed = 2,
|
||||
}
|
||||
|
||||
|
||||
@@ -11,7 +11,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
|
||||
{
|
||||
/// <summary>
|
||||
/// Phase 6 of <see cref="Program"/>'s <c>RunAgent</c>: blocking
|
||||
/// <c>POST /api/agent/register-session</c> with 5-retry exponential backoff before the
|
||||
/// <c>POST /api/agent/register-session</c> with link-wait + 6-retry exponential backoff before the
|
||||
/// orchestrator starts. V1 parity (MonitoringService.RegisterSessionAsync) — without
|
||||
/// this call the backend's Sessions table never gets a row for this session, so
|
||||
/// <c>IncrementSessionEventCountAsync</c> / <c>UpdateSessionStatusAsync</c> silently
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Invariants of the built-in diagnostics catalog (<see cref="DiagnosticsBuiltInSections"/>).
|
||||
/// Shared has no test project, so the catalog is pinned here: the agent iterates it to build the
|
||||
/// archive and the backend serves it to the portal, so a malformed entry would either break
|
||||
/// collection on every device or mislead every administrator reading the list.
|
||||
/// </summary>
|
||||
public class DiagnosticsBuiltInSectionsTests
|
||||
{
|
||||
[Fact]
|
||||
public void SectionIds_AreUnique()
|
||||
{
|
||||
var ids = DiagnosticsBuiltInSections.All.Select(s => s.Id).ToArray();
|
||||
Assert.Equal(ids.Length, ids.Distinct(StringComparer.Ordinal).Count());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ZipFolderAndSourceFolderPairs_AreUnique()
|
||||
{
|
||||
// Two sections may share a ZIP folder (ImeLogs + the bootstrapper evtx) only when they
|
||||
// read from different source folders — the same (zip, source) pair would double-add
|
||||
// every file under duplicate entry names.
|
||||
var pairs = DiagnosticsBuiltInSections.All
|
||||
.Select(s => $"{s.ZipFolder}|{s.SourceFolder}".ToLowerInvariant())
|
||||
.ToArray();
|
||||
Assert.Equal(pairs.Length, pairs.Distinct().Count());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void AlwaysSections_KeepHistoricalArchiveLayoutAndOrder()
|
||||
{
|
||||
// Archive-layout lock: these five sections (and their order) are what every diagnostics
|
||||
// ZIP has carried since the PR1-B forensics work; tooling and support habits rely on it.
|
||||
var always = DiagnosticsBuiltInSections.All
|
||||
.Where(s => s.Condition == DiagnosticsSectionCondition.Always)
|
||||
.Select(s => s.Id)
|
||||
.ToArray();
|
||||
Assert.Equal(new[] { "AgentLogs", "ImeLogs", "AgentState", "AgentSpool", "AgentMarkers" }, always);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ConditionalSections_CarryTheExpectedGate()
|
||||
{
|
||||
foreach (var section in DiagnosticsBuiltInSections.All)
|
||||
{
|
||||
if (section.Id.StartsWith("RealmJoin", StringComparison.Ordinal))
|
||||
Assert.Equal(DiagnosticsSectionCondition.RealmJoinWatcher, section.Condition);
|
||||
if (section.Id == "ImeBootstrapperEventLog")
|
||||
Assert.Equal(DiagnosticsSectionCondition.DevicePreparation, section.Condition);
|
||||
}
|
||||
Assert.Contains(DiagnosticsBuiltInSections.All, s => s.Id == "ImeBootstrapperEventLog");
|
||||
Assert.Equal(5, DiagnosticsBuiltInSections.All.Count(s => s.Condition == DiagnosticsSectionCondition.RealmJoinWatcher));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void UserProfileTokenSections_AreNeverAlwaysOn()
|
||||
{
|
||||
// The token resolves only once an interactive user exists; an Always section reading
|
||||
// from the user profile would be skipped on every pre-desktop package and mislead the
|
||||
// portal's "collected by every agent" wording.
|
||||
var tokenSections = DiagnosticsBuiltInSections.All
|
||||
.Where(s => s.SourceFolder.Contains(DiagnosticsBuiltInSections.UserProfileToken, StringComparison.OrdinalIgnoreCase))
|
||||
.ToArray();
|
||||
Assert.NotEmpty(tokenSections);
|
||||
Assert.All(tokenSections, s => Assert.NotEqual(DiagnosticsSectionCondition.Always, s.Condition));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Patterns_AreFileNamesWithoutPathSeparators()
|
||||
{
|
||||
Assert.All(DiagnosticsBuiltInSections.All, s =>
|
||||
{
|
||||
Assert.NotEmpty(s.Patterns);
|
||||
Assert.All(s.Patterns, p =>
|
||||
{
|
||||
Assert.False(string.IsNullOrWhiteSpace(p));
|
||||
Assert.DoesNotContain('\\', p);
|
||||
Assert.DoesNotContain('/', p);
|
||||
});
|
||||
Assert.False(string.IsNullOrWhiteSpace(s.Description));
|
||||
Assert.False(s.ZipFolder.StartsWith("/") || s.ZipFolder.EndsWith("/"));
|
||||
Assert.DoesNotContain('\\', s.ZipFolder);
|
||||
});
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NoBuiltInSection_UsesTheConfiguredPathsZipFolder()
|
||||
{
|
||||
// AdditionalLogs/<folder> is reserved for admin-configured entries.
|
||||
Assert.All(DiagnosticsBuiltInSections.All,
|
||||
s => Assert.False(s.ZipFolder.StartsWith("AdditionalLogs", StringComparison.OrdinalIgnoreCase)));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
using System.Text.Json;
|
||||
using AutopilotMonitor.Functions.Services.Notifications;
|
||||
using AutopilotMonitor.Shared.Models.Notifications;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Discord embed rendering: payload shape, color mapping, and the Discord embed limits
|
||||
/// (25 fields, 256/1024 name/value, 4096 description, 6000 total).
|
||||
/// </summary>
|
||||
public class DiscordRendererTests
|
||||
{
|
||||
private static JsonElement RenderEmbed(NotificationAlert alert)
|
||||
{
|
||||
var json = new DiscordRenderer().RenderToJson(alert);
|
||||
var doc = JsonDocument.Parse(json);
|
||||
var embeds = doc.RootElement.GetProperty("embeds");
|
||||
Assert.Equal(1, embeds.GetArrayLength());
|
||||
return embeds[0];
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RendersSingleEmbed_WithTitleColorFieldsAndLinks()
|
||||
{
|
||||
var alert = new NotificationAlert
|
||||
{
|
||||
Title = "Enrollment Succeeded",
|
||||
Summary = "Enrollment Succeeded: DESK-001",
|
||||
Severity = NotificationSeverity.Success,
|
||||
ThemeColor = "00B050",
|
||||
Facts = { new NotificationFact { Name = "Device", Value = "DESK-001" } },
|
||||
Sections = { new NotificationSection { Title = "Details", Text = "All apps installed." } },
|
||||
Actions = { new NotificationAction { Type = "openUrl", Title = "Open session", Url = "https://portal.example.com/sessions/abc" } },
|
||||
};
|
||||
|
||||
var embed = RenderEmbed(alert);
|
||||
|
||||
Assert.Equal("✅ Enrollment Succeeded", embed.GetProperty("title").GetString());
|
||||
Assert.Equal(0x00B050, embed.GetProperty("color").GetInt32());
|
||||
|
||||
var description = embed.GetProperty("description").GetString()!;
|
||||
Assert.StartsWith("Enrollment Succeeded: DESK-001", description);
|
||||
Assert.Contains("**Details**\nAll apps installed.", description);
|
||||
// Webhooks cannot post buttons — actions become markdown links.
|
||||
Assert.Contains("[Open session](https://portal.example.com/sessions/abc)", description);
|
||||
|
||||
var fields = embed.GetProperty("fields");
|
||||
Assert.Equal(1, fields.GetArrayLength());
|
||||
Assert.Equal("Device", fields[0].GetProperty("name").GetString());
|
||||
Assert.Equal("DESK-001", fields[0].GetProperty("value").GetString());
|
||||
Assert.True(fields[0].GetProperty("inline").GetBoolean());
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(NotificationSeverity.Success, 0x2ECC71)]
|
||||
[InlineData(NotificationSeverity.Error, 0xE74C3C)]
|
||||
[InlineData(NotificationSeverity.Warning, 0xF1C40F)]
|
||||
[InlineData(NotificationSeverity.Info, 0x3498DB)]
|
||||
public void Color_FallsBackToSeverity_WhenThemeColorMissingOrInvalid(NotificationSeverity severity, int expected)
|
||||
{
|
||||
foreach (var themeColor in new[] { "", "not-hex" })
|
||||
{
|
||||
var embed = RenderEmbed(new NotificationAlert { Title = "T", Severity = severity, ThemeColor = themeColor });
|
||||
Assert.Equal(expected, embed.GetProperty("color").GetInt32());
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Fields_CappedAt25_AndEmptyFactsSkipped()
|
||||
{
|
||||
var alert = new NotificationAlert { Title = "T", Summary = "S" };
|
||||
alert.Facts.Add(new NotificationFact { Name = "", Value = "dropped" });
|
||||
alert.Facts.Add(new NotificationFact { Name = "dropped", Value = " " });
|
||||
for (var i = 0; i < 30; i++)
|
||||
alert.Facts.Add(new NotificationFact { Name = $"f{i}", Value = "v" });
|
||||
|
||||
var fields = RenderEmbed(alert).GetProperty("fields");
|
||||
|
||||
Assert.Equal(25, fields.GetArrayLength());
|
||||
Assert.Equal("f0", fields[0].GetProperty("name").GetString()); // empty facts skipped, not counted
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FieldValue_TruncatedTo1024_WithEllipsis()
|
||||
{
|
||||
var alert = new NotificationAlert { Title = "T" };
|
||||
alert.Facts.Add(new NotificationFact { Name = "long", Value = new string('x', 2000) });
|
||||
|
||||
var value = RenderEmbed(alert).GetProperty("fields")[0].GetProperty("value").GetString()!;
|
||||
|
||||
Assert.Equal(1024, value.Length);
|
||||
Assert.EndsWith("…", value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void TotalEmbedLength_CappedAt6000_ByDroppingTrailingFields()
|
||||
{
|
||||
var alert = new NotificationAlert { Title = "T", Summary = "S" };
|
||||
for (var i = 0; i < 25; i++)
|
||||
alert.Facts.Add(new NotificationFact { Name = $"f{i}", Value = new string('v', 1000) });
|
||||
|
||||
var embed = RenderEmbed(alert);
|
||||
var fields = embed.GetProperty("fields");
|
||||
|
||||
var total = embed.GetProperty("title").GetString()!.Length
|
||||
+ embed.GetProperty("description").GetString()!.Length;
|
||||
for (var i = 0; i < fields.GetArrayLength(); i++)
|
||||
total += fields[i].GetProperty("name").GetString()!.Length + fields[i].GetProperty("value").GetString()!.Length;
|
||||
|
||||
Assert.True(fields.GetArrayLength() < 25);
|
||||
Assert.True(total <= 6000);
|
||||
Assert.Equal("f0", fields[0].GetProperty("name").GetString()); // leading fields kept
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EmptyDescription_Omitted_AndProviderTypeIsDiscord()
|
||||
{
|
||||
var embed = RenderEmbed(new NotificationAlert { Title = "T", Severity = NotificationSeverity.Info });
|
||||
|
||||
Assert.False(embed.TryGetProperty("description", out _));
|
||||
Assert.False(embed.TryGetProperty("fields", out _));
|
||||
Assert.Equal(WebhookProviderType.Discord, new DiscordRenderer().ProviderType);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,315 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Net;
|
||||
using System.Net.Http;
|
||||
using System.Text.Json;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Coverage for the provider-neutral <see cref="EmailService"/> (welcome + farewell mails).
|
||||
/// The provider HTTP call is driven through a stub handler, so the Mandrill wire format,
|
||||
/// the result-array interpretation and the fail-soft contract are pinned without network.
|
||||
/// <list type="bullet">
|
||||
/// <item>Neither send method ever throws — not on a missing key, missing recipient,
|
||||
/// HTTP 500, a rejected recipient, nor a malformed body.</item>
|
||||
/// <item>Gate ordering: missing-key short-circuits before missing-recipient.</item>
|
||||
/// <item>Tracking is off on the wire (trust-page claim: provider receives only the
|
||||
/// recipient address and tenant domain).</item>
|
||||
/// <item>The farewell template is final: no [DRAFT]/placeholder residue, domain is
|
||||
/// interpolated, empty domain falls back to "your organization".</item>
|
||||
/// </list>
|
||||
/// </summary>
|
||||
public sealed class EmailServiceTests
|
||||
{
|
||||
private const string TenantId = "88888888-8888-8888-8888-888888888888";
|
||||
|
||||
// ----- configuration gates -----
|
||||
|
||||
[Fact]
|
||||
public async Task SendAsync_EmptyApiKey_NoOps_AndLogsMissingKeyDebugLine()
|
||||
{
|
||||
var (sut, handler, logger) = Build(apiKey: "");
|
||||
|
||||
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId, CancellationToken.None);
|
||||
|
||||
Assert.Equal(0, handler.CallCount);
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Debug && e.Message.Contains("Email:ApiKey not configured"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SendAsync_EmptyRecipient_NoOps_AndLogsMissingEmailDebugLine()
|
||||
{
|
||||
var (sut, handler, logger) = Build(apiKey: "any-non-empty-key");
|
||||
|
||||
await sut.SendAsync("", "contoso.invalid", TenantId);
|
||||
|
||||
Assert.Equal(0, handler.CallCount);
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Debug && e.Message.Contains("No notification email captured"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SendAsync_EmptyApiKey_ShortCircuitsBeforeRecipientCheck()
|
||||
{
|
||||
// Pins the gate ordering: missing-key > missing-email. Both are empty here; only
|
||||
// the key line must show up.
|
||||
var (sut, _, logger) = Build(apiKey: "");
|
||||
|
||||
await sut.SendAsync("", "contoso.invalid", TenantId);
|
||||
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Debug && e.Message.Contains("Email:ApiKey not configured"));
|
||||
Assert.DoesNotContain(logger.Entries,
|
||||
e => e.Message.Contains("No notification email captured"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SendPreviewApprovedEmailAsync_EmptyApiKey_NoOps()
|
||||
{
|
||||
var (sut, handler, logger) = Build(apiKey: "");
|
||||
|
||||
await sut.SendPreviewApprovedEmailAsync("it@contoso.invalid", "contoso.invalid");
|
||||
|
||||
Assert.Equal(0, handler.CallCount);
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Debug && e.Message.Contains("Email:ApiKey not configured"));
|
||||
}
|
||||
|
||||
// ----- wire format -----
|
||||
|
||||
[Fact]
|
||||
public async Task SendPreviewApprovedEmailAsync_PostsMandrillPayload_WithTrackingOff()
|
||||
{
|
||||
var (sut, handler, logger) = Build(apiKey: "md-test-key");
|
||||
|
||||
await sut.SendPreviewApprovedEmailAsync("it@contoso.invalid", "contoso.invalid");
|
||||
|
||||
Assert.Equal(1, handler.CallCount);
|
||||
Assert.Equal(HttpMethod.Post, handler.LastRequest!.Method);
|
||||
Assert.Equal(EmailService.DefaultEndpoint, handler.LastRequest.RequestUri!.ToString());
|
||||
|
||||
using var doc = JsonDocument.Parse(handler.LastBody!);
|
||||
var root = doc.RootElement;
|
||||
Assert.Equal("md-test-key", root.GetProperty("key").GetString());
|
||||
|
||||
var message = root.GetProperty("message");
|
||||
Assert.Equal(EmailService.DefaultFromAddress, message.GetProperty("from_email").GetString());
|
||||
Assert.Equal(EmailService.DefaultFromName, message.GetProperty("from_name").GetString());
|
||||
Assert.Equal("it@contoso.invalid", message.GetProperty("to")[0].GetProperty("email").GetString());
|
||||
Assert.Equal("to", message.GetProperty("to")[0].GetProperty("type").GetString());
|
||||
Assert.Equal(EmailTemplates.PreviewApprovedSubject, message.GetProperty("subject").GetString());
|
||||
Assert.Contains("contoso.invalid", message.GetProperty("html").GetString());
|
||||
Assert.True(message.GetProperty("auto_text").GetBoolean());
|
||||
Assert.False(message.GetProperty("track_opens").GetBoolean());
|
||||
Assert.False(message.GetProperty("track_clicks").GetBoolean());
|
||||
Assert.Equal("welcome", message.GetProperty("tags")[0].GetString());
|
||||
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Information && e.Message.Contains("Preview approval email sent"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SendAsync_UsesFarewellTemplate_AndFarewellTag()
|
||||
{
|
||||
var (sut, handler, logger) = Build(apiKey: "md-test-key");
|
||||
|
||||
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId);
|
||||
|
||||
using var doc = JsonDocument.Parse(handler.LastBody!);
|
||||
var message = doc.RootElement.GetProperty("message");
|
||||
Assert.Equal(EmailTemplates.OffboardingFarewellSubject, message.GetProperty("subject").GetString());
|
||||
Assert.Equal("offboarding-farewell", message.GetProperty("tags")[0].GetString());
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Information && e.Message.Contains("Offboard farewell email sent"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ConfiguredEndpointAndSender_OverrideDefaults()
|
||||
{
|
||||
var (sut, handler, _) = Build(apiKey: "md-test-key", extra: new Dictionary<string, string?>
|
||||
{
|
||||
["Email:Endpoint"] = "https://mail.example.invalid/send",
|
||||
["Email:FromAddress"] = "hello@example.invalid",
|
||||
["Email:FromName"] = "Example Sender",
|
||||
});
|
||||
|
||||
await sut.SendPreviewApprovedEmailAsync("it@contoso.invalid", "contoso.invalid");
|
||||
|
||||
Assert.Equal("https://mail.example.invalid/send", handler.LastRequest!.RequestUri!.ToString());
|
||||
using var doc = JsonDocument.Parse(handler.LastBody!);
|
||||
var message = doc.RootElement.GetProperty("message");
|
||||
Assert.Equal("hello@example.invalid", message.GetProperty("from_email").GetString());
|
||||
Assert.Equal("Example Sender", message.GetProperty("from_name").GetString());
|
||||
}
|
||||
|
||||
// ----- provider responses / fail-soft -----
|
||||
|
||||
[Fact]
|
||||
public async Task ProviderHttpError_DoesNotThrow_AndLogsWarning_WithoutSuccessLine()
|
||||
{
|
||||
var (sut, _, logger) = Build(apiKey: "md-test-key",
|
||||
responder: _ => Json(HttpStatusCode.InternalServerError,
|
||||
"{\"status\":\"error\",\"code\":-1,\"name\":\"Invalid_Key\",\"message\":\"Invalid API key\"}"));
|
||||
|
||||
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId);
|
||||
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Warning && e.Message.Contains("500") && e.Message.Contains("Invalid_Key"));
|
||||
Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Information);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ProviderRejectsRecipient_LogsWarningWithReason_WithoutSuccessLine()
|
||||
{
|
||||
var (sut, _, logger) = Build(apiKey: "md-test-key",
|
||||
responder: _ => Json(HttpStatusCode.OK,
|
||||
"[{\"email\":\"ops@contoso.invalid\",\"status\":\"rejected\",\"reject_reason\":\"hard-bounce\",\"_id\":\"abc\"}]"));
|
||||
|
||||
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId);
|
||||
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Warning && e.Message.Contains("rejected") && e.Message.Contains("hard-bounce"));
|
||||
Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Information);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("sent")]
|
||||
[InlineData("queued")]
|
||||
[InlineData("scheduled")]
|
||||
public async Task ProviderAcceptedStatuses_CountAsSuccess(string status)
|
||||
{
|
||||
var (sut, _, logger) = Build(apiKey: "md-test-key",
|
||||
responder: _ => Json(HttpStatusCode.OK,
|
||||
$"[{{\"email\":\"ops@contoso.invalid\",\"status\":\"{status}\",\"_id\":\"abc\"}}]"));
|
||||
|
||||
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId);
|
||||
|
||||
Assert.Contains(logger.Entries, e => e.Level == LogLevel.Information);
|
||||
Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Warning);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MalformedProviderBody_DoesNotThrow_AndLogsWarning()
|
||||
{
|
||||
var (sut, _, logger) = Build(apiKey: "md-test-key",
|
||||
responder: _ => Json(HttpStatusCode.OK, "not json"));
|
||||
|
||||
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId);
|
||||
|
||||
Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning);
|
||||
Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Information);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task TransportException_DoesNotThrow_AndLogsWarning()
|
||||
{
|
||||
var (sut, _, logger) = Build(apiKey: "md-test-key",
|
||||
responder: _ => throw new HttpRequestException("simulated provider outage"));
|
||||
|
||||
await sut.SendPreviewApprovedEmailAsync("it@contoso.invalid", "contoso.invalid");
|
||||
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Warning && e.Message.Contains("Failed to send welcome mail"));
|
||||
}
|
||||
|
||||
// ----- templates -----
|
||||
|
||||
[Fact]
|
||||
public void FarewellTemplate_IsFinal_NoDraftResidue()
|
||||
{
|
||||
Assert.DoesNotContain("DRAFT", EmailTemplates.OffboardingFarewellSubject);
|
||||
|
||||
var html = EmailTemplates.GetOffboardingFarewellHtml("contoso.invalid");
|
||||
Assert.DoesNotContain("DRAFT", html);
|
||||
Assert.DoesNotContain("TEMPLATE NOT FINALISED", html);
|
||||
Assert.DoesNotContain("TODO", html);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FarewellTemplate_InterpolatesDomain_AndLinksFeedbackChannels()
|
||||
{
|
||||
var html = EmailTemplates.GetOffboardingFarewellHtml("contoso.invalid");
|
||||
|
||||
Assert.Contains("contoso.invalid", html);
|
||||
// The mail is sent from a noreply address after portal access is gone, so the
|
||||
// feedback pointers must be external channels the recipient can actually reach.
|
||||
Assert.Contains("https://github.com/okieselbach/AutopilotMonitor/issues", html);
|
||||
Assert.Contains("linkedin.com", html);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FarewellTemplate_EmptyDomain_FallsBackToGenericLabel()
|
||||
{
|
||||
var html = EmailTemplates.GetOffboardingFarewellHtml("");
|
||||
Assert.Contains("your organization", html);
|
||||
}
|
||||
|
||||
// ----- harness -----
|
||||
|
||||
private static HttpResponseMessage Json(HttpStatusCode status, string body)
|
||||
=> new(status) { Content = new StringContent(body, System.Text.Encoding.UTF8, "application/json") };
|
||||
|
||||
private static (EmailService sut, StubHandler handler, CapturingLogger<EmailService> logger) Build(
|
||||
string apiKey,
|
||||
Dictionary<string, string?>? extra = null,
|
||||
System.Func<HttpRequestMessage, HttpResponseMessage>? responder = null)
|
||||
{
|
||||
var settings = new Dictionary<string, string?> { ["Email:ApiKey"] = apiKey };
|
||||
if (extra is not null)
|
||||
foreach (var kv in extra) settings[kv.Key] = kv.Value;
|
||||
|
||||
var config = new ConfigurationBuilder().AddInMemoryCollection(settings).Build();
|
||||
var handler = new StubHandler();
|
||||
if (responder is not null) handler.Responder = responder;
|
||||
var logger = new CapturingLogger<EmailService>();
|
||||
var templates = new EmailTemplateService(
|
||||
Moq.Mock.Of<AutopilotMonitor.Shared.DataAccess.IConfigRepository>(),
|
||||
new Microsoft.Extensions.Caching.Memory.MemoryCache(new Microsoft.Extensions.Caching.Memory.MemoryCacheOptions()),
|
||||
Microsoft.Extensions.Logging.Abstractions.NullLogger<EmailTemplateService>.Instance);
|
||||
var sut = new EmailService(new HttpClient(handler, disposeHandler: false), config, templates, logger);
|
||||
return (sut, handler, logger);
|
||||
}
|
||||
|
||||
private sealed class StubHandler : HttpMessageHandler
|
||||
{
|
||||
public int CallCount { get; private set; }
|
||||
public HttpRequestMessage? LastRequest { get; private set; }
|
||||
public string? LastBody { get; private set; }
|
||||
public System.Func<HttpRequestMessage, HttpResponseMessage> Responder { get; set; } = _ =>
|
||||
Json(HttpStatusCode.OK, "[{\"email\":\"x@y.invalid\",\"status\":\"sent\",\"_id\":\"abc\"}]");
|
||||
|
||||
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
|
||||
{
|
||||
CallCount++;
|
||||
LastRequest = request;
|
||||
LastBody = request.Content is null ? null : await request.Content.ReadAsStringAsync(cancellationToken);
|
||||
return Responder(request);
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class CapturingLogger<T> : ILogger<T>
|
||||
{
|
||||
public readonly List<(LogLevel Level, string Message)> Entries = new();
|
||||
public IDisposable BeginScope<TState>(TState state) where TState : notnull => NullScope.Instance;
|
||||
public bool IsEnabled(LogLevel logLevel) => true;
|
||||
public void Log<TState>(
|
||||
LogLevel logLevel, EventId eventId, TState state, System.Exception? exception,
|
||||
System.Func<TState, System.Exception?, string> formatter)
|
||||
{
|
||||
Entries.Add((logLevel, formatter(state, exception)));
|
||||
}
|
||||
|
||||
private sealed class NullScope : IDisposable
|
||||
{
|
||||
public static readonly NullScope Instance = new();
|
||||
public void Dispose() { }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Moq;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins the operator-override layer over the built-in transactional templates:
|
||||
/// override wins when stored, built-in otherwise; placeholder rendering matches the
|
||||
/// built-ins' empty-domain fallback; the send path is fail-soft on storage errors while the
|
||||
/// admin path is not; save validates and invalidates the cache; reset is idempotent.
|
||||
/// </summary>
|
||||
public sealed class EmailTemplateServiceTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task NoOverride_ReturnsBuiltIn_Rendered()
|
||||
{
|
||||
var (sut, _) = Build();
|
||||
|
||||
var html = await sut.GetHtmlAsync(EmailTemplateKind.Welcome, "contoso.invalid");
|
||||
|
||||
Assert.Equal(EmailTemplates.GetPreviewApprovedHtml("contoso.invalid"), html);
|
||||
Assert.DoesNotContain(EmailTemplateService.DomainPlaceholder, html);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Override_WinsAndRendersPlaceholder()
|
||||
{
|
||||
var (sut, repo) = Build();
|
||||
repo.Setup(r => r.GetEmailTemplateOverrideAsync("farewell"))
|
||||
.ReturnsAsync(new EmailTemplateOverride { Kind = "farewell", Html = "<p>Bye {{domainName}}</p>" });
|
||||
|
||||
var html = await sut.GetHtmlAsync(EmailTemplateKind.Farewell, "contoso.invalid");
|
||||
|
||||
Assert.Equal("<p>Bye contoso.invalid</p>", html);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Override_EmptyDomain_FallsBackToGenericLabel_LikeBuiltIn()
|
||||
{
|
||||
var (sut, repo) = Build();
|
||||
repo.Setup(r => r.GetEmailTemplateOverrideAsync("welcome"))
|
||||
.ReturnsAsync(new EmailTemplateOverride { Kind = "welcome", Html = "Hi {{domainName}}" });
|
||||
|
||||
Assert.Equal("Hi your organization", await sut.GetHtmlAsync(EmailTemplateKind.Welcome, ""));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StorageError_SendPath_FallsBackToBuiltIn_AdminPath_Throws()
|
||||
{
|
||||
var (sut, repo) = Build();
|
||||
repo.Setup(r => r.GetEmailTemplateOverrideAsync(It.IsAny<string>()))
|
||||
.ThrowsAsync(new System.InvalidOperationException("storage down"));
|
||||
|
||||
var html = await sut.GetHtmlAsync(EmailTemplateKind.Welcome, "contoso.invalid");
|
||||
Assert.Contains("contoso.invalid", html);
|
||||
|
||||
await Assert.ThrowsAsync<System.InvalidOperationException>(() => sut.GetOverrideAsync(EmailTemplateKind.Welcome));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Save_Validates_ThenInvalidatesCache()
|
||||
{
|
||||
var (sut, repo) = Build();
|
||||
repo.Setup(r => r.GetEmailTemplateOverrideAsync("welcome")).ReturnsAsync((EmailTemplateOverride?)null);
|
||||
|
||||
// warm the cache with "no override"
|
||||
Assert.Null(await sut.GetOverrideAsync(EmailTemplateKind.Welcome));
|
||||
|
||||
await Assert.ThrowsAsync<System.ArgumentException>(() => sut.SaveOverrideAsync(EmailTemplateKind.Welcome, " ", "ga@contoso.invalid"));
|
||||
await Assert.ThrowsAsync<System.ArgumentException>(() =>
|
||||
sut.SaveOverrideAsync(EmailTemplateKind.Welcome, new string('x', EmailTemplateService.MaxHtmlLength + 1), "ga@contoso.invalid"));
|
||||
repo.Verify(r => r.SaveEmailTemplateOverrideAsync(It.IsAny<EmailTemplateOverride>()), Times.Never);
|
||||
|
||||
EmailTemplateOverride? stored = null;
|
||||
repo.Setup(r => r.SaveEmailTemplateOverrideAsync(It.IsAny<EmailTemplateOverride>()))
|
||||
.Callback<EmailTemplateOverride>(e => stored = e).Returns(Task.CompletedTask);
|
||||
repo.Setup(r => r.GetEmailTemplateOverrideAsync("welcome")).ReturnsAsync(() => stored);
|
||||
|
||||
var saved = await sut.SaveOverrideAsync(EmailTemplateKind.Welcome, "<p>{{domainName}}</p>", "ga@contoso.invalid");
|
||||
|
||||
Assert.Equal("welcome", saved.Kind);
|
||||
Assert.Equal("ga@contoso.invalid", saved.UpdatedBy);
|
||||
// cache was invalidated: the next read sees the stored override, not the cached null
|
||||
Assert.NotNull(await sut.GetOverrideAsync(EmailTemplateKind.Welcome));
|
||||
Assert.Equal("<p>contoso.invalid</p>", await sut.GetHtmlAsync(EmailTemplateKind.Welcome, "contoso.invalid"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Delete_ClearsCache_ReturnsToBuiltIn()
|
||||
{
|
||||
var (sut, repo) = Build();
|
||||
var stored = new EmailTemplateOverride { Kind = "farewell", Html = "custom {{domainName}}" };
|
||||
repo.Setup(r => r.GetEmailTemplateOverrideAsync("farewell")).ReturnsAsync(() => stored);
|
||||
Assert.Equal("custom contoso.invalid", await sut.GetHtmlAsync(EmailTemplateKind.Farewell, "contoso.invalid"));
|
||||
|
||||
repo.Setup(r => r.DeleteEmailTemplateOverrideAsync("farewell")).Callback(() => stored = null!).Returns(Task.CompletedTask);
|
||||
await sut.DeleteOverrideAsync(EmailTemplateKind.Farewell, "ga@contoso.invalid");
|
||||
|
||||
Assert.Equal(EmailTemplates.GetOffboardingFarewellHtml("contoso.invalid"),
|
||||
await sut.GetHtmlAsync(EmailTemplateKind.Farewell, "contoso.invalid"));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("welcome", true)]
|
||||
[InlineData("Farewell", true)]
|
||||
[InlineData("invoice", false)]
|
||||
[InlineData(null, false)]
|
||||
public void TryParseKind_AcceptsOnlyKnownKinds(string? value, bool expected)
|
||||
=> Assert.Equal(expected, EmailTemplateService.TryParseKind(value, out _));
|
||||
|
||||
[Fact]
|
||||
public void BuiltInRaw_ContainsPlaceholder_ForBothKinds()
|
||||
{
|
||||
Assert.Contains(EmailTemplateService.DomainPlaceholder, EmailTemplateService.BuiltInRaw(EmailTemplateKind.Welcome));
|
||||
Assert.Contains(EmailTemplateService.DomainPlaceholder, EmailTemplateService.BuiltInRaw(EmailTemplateKind.Farewell));
|
||||
}
|
||||
|
||||
private static (EmailTemplateService sut, Mock<IConfigRepository> repo) Build()
|
||||
{
|
||||
var repo = new Mock<IConfigRepository>(MockBehavior.Loose);
|
||||
var sut = new EmailTemplateService(repo.Object, new MemoryCache(new MemoryCacheOptions()), NullLogger<EmailTemplateService>.Instance);
|
||||
return (sut, repo);
|
||||
}
|
||||
}
|
||||
+15
@@ -191,6 +191,21 @@ public class EndpointPolicyCatalogCompletenessTests
|
||||
Assert.Equal(EndpointPolicy.GlobalAdminOnly, reseed.Policy);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The built-in diagnostics catalog + global paths are platform-wide, secret-free data that
|
||||
/// every settings viewer needs (what is collected from the tenant's devices). Member tier,
|
||||
/// JWT-scoped — an upgrade to a global tier would hide it from tenant admins again.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void DiagnosticsPaths_IsMemberReadJwtScoped()
|
||||
{
|
||||
var entry = EndpointAccessPolicyCatalog.FindPolicy("GET", "/api/diagnostics/paths");
|
||||
|
||||
Assert.NotNull(entry);
|
||||
Assert.Equal(EndpointPolicy.MemberRead, entry.Policy);
|
||||
Assert.Equal(TenantScoping.Jwt, entry.TenantScoping);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Unregistered routes must return null (fail-closed).
|
||||
/// </summary>
|
||||
|
||||
@@ -163,6 +163,12 @@ public class EnrollmentTimeoutClassifierTests
|
||||
// -------- ClassifyTimedOutSession --------
|
||||
|
||||
private static (SessionStatus, string) Classify(IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 6, int grace = 72)
|
||||
{
|
||||
var (status, reason, _) = ClassifyWithRule(events, hoursSinceStart, grace);
|
||||
return (status, reason);
|
||||
}
|
||||
|
||||
private static (SessionStatus, string, string) ClassifyWithRule(IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 6, int grace = 72)
|
||||
{
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
|
||||
var now = Start.AddHours(hoursSinceStart);
|
||||
@@ -220,7 +226,7 @@ public class EnrollmentTimeoutClassifierTests
|
||||
Esp(DeviceSetup44), Esp(AccountSetup15),
|
||||
Evt("desktop_arrived"), Evt("hello_provisioning_completed"), Evt("realmjoin_detected"),
|
||||
});
|
||||
var (status, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
var (status, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, Start, now, graceHours: 72, lastEventAtUtc: lastEvent);
|
||||
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
@@ -471,4 +477,230 @@ public class EnrollmentTimeoutClassifierTests
|
||||
Assert.NotEqual(SessionStatus.Failed, status);
|
||||
}
|
||||
}
|
||||
|
||||
// -------- WhiteGlove Part-2 awaiting-user gate (fairstone.ca analysis 2026-08-21) --------
|
||||
|
||||
private static readonly DateTime Resumed = Start.AddMinutes(20);
|
||||
|
||||
private static (SessionStatus, string) ClassifyWhiteGlove(
|
||||
IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 6, int grace = 51)
|
||||
{
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
|
||||
var now = Start.AddHours(hoursSinceStart);
|
||||
var (status, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, Start, now, grace, isPreProvisioned: true, resumedAt: Resumed);
|
||||
return (status, reason);
|
||||
}
|
||||
|
||||
/// <summary>The fairstone event shape: Part 1 sealed, Part 2 resumed, nobody signed in.</summary>
|
||||
private static EnrollmentEvent[] WgParkedEvents() => new[]
|
||||
{
|
||||
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup05),
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void ExtractRollup_separates_enrollment_complete_from_whiteglove_complete()
|
||||
{
|
||||
var wgOnly = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("whiteglove_complete") });
|
||||
Assert.True(wgOnly.HasTerminalComplete);
|
||||
Assert.False(wgOnly.HasEnrollmentComplete);
|
||||
|
||||
var real = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("enrollment_complete") });
|
||||
Assert.True(real.HasTerminalComplete);
|
||||
Assert.True(real.HasEnrollmentComplete);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WhiteGloveGate_requires_preprovisioned_and_resume_and_device_setup()
|
||||
{
|
||||
var parked = EnrollmentTimeoutClassifier.ExtractRollup(WgParkedEvents());
|
||||
Assert.True(EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(parked, true, Resumed));
|
||||
Assert.False(EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(parked, false, Resumed));
|
||||
Assert.False(EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(parked, true, null));
|
||||
|
||||
// Powered off before Device Setup ever provisioned → not the parking state.
|
||||
var early = EnrollmentTimeoutClassifier.ExtractRollup(new[]
|
||||
{
|
||||
Esp("ESP provisioning status: DeviceSetup — 3 of 4 subcategories completed"),
|
||||
});
|
||||
Assert.False(EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(early, true, Resumed));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void WhiteGloveGate_any_user_evidence_or_terminal_disarms_it()
|
||||
{
|
||||
foreach (var evidence in new[]
|
||||
{
|
||||
Esp(AccountSetup15), // Account Setup progressed → user was there
|
||||
Evt("desktop_arrived"), // real-user desktop
|
||||
Evt("hello_provisioning_completed"), // positive Hello terminal
|
||||
Evt("enrollment_complete"), // real completion
|
||||
Evt("enrollment_failed"), // explicit failure
|
||||
})
|
||||
{
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(new[]
|
||||
{
|
||||
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup05), evidence,
|
||||
});
|
||||
Assert.False(EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(rollup, true, Resumed));
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_whiteglove_part2_parked_within_grace_is_AwaitingUser()
|
||||
{
|
||||
// The fairstone shape: technician powers the device off at the logon screen minutes
|
||||
// after the reseal-reboot. Part-1 whiteglove_complete must NOT reconcile to Succeeded.
|
||||
var (status, reason) = ClassifyWhiteGlove(WgParkedEvents(), hoursSinceStart: 6);
|
||||
Assert.Equal(SessionStatus.AwaitingUser, status);
|
||||
Assert.Contains("sealed or powered off awaiting the end user", reason);
|
||||
Assert.Contains("(Account Setup 0/5)", reason);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_whiteglove_part2_parked_past_grace_reconciles_with_honest_reason()
|
||||
{
|
||||
var (status, reason) = ClassifyWhiteGlove(WgParkedEvents(), hoursSinceStart: 60, grace: 51);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
Assert.Contains("pre-provisioning (WhiteGlove Part 1) completed", reason);
|
||||
Assert.DoesNotContain("Account Setup completed", reason);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_whiteglove_part2_with_partial_user_progress_is_not_reconciled_on_part1_evidence()
|
||||
{
|
||||
// User signed in and Account Setup progressed to 1/5, then silence. The Part-1
|
||||
// whiteglove_complete previously reconciled this to Succeeded claiming "Account Setup
|
||||
// completed" — on a resumed session only a real enrollment_complete proves the outcome,
|
||||
// so this honest shape falls through to the AwaitingUser/Incomplete rules.
|
||||
var events = new[]
|
||||
{
|
||||
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup15),
|
||||
};
|
||||
var (withinGrace, _) = ClassifyWhiteGlove(events, hoursSinceStart: 6);
|
||||
Assert.Equal(SessionStatus.AwaitingUser, withinGrace);
|
||||
|
||||
var (pastGrace, reason) = ClassifyWhiteGlove(events, hoursSinceStart: 60, grace: 51);
|
||||
Assert.Equal(SessionStatus.Incomplete, pastGrace);
|
||||
Assert.Contains("1/5", reason);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_whiteglove_part2_with_desktop_and_hello_still_reconciles_via_user_evidence()
|
||||
{
|
||||
// Both Classic completion prerequisites in → the user provably finished; the
|
||||
// user-completed reconcile (rule 4) owns this, not the parking gate.
|
||||
var (status, reason) = ClassifyWhiteGlove(new[]
|
||||
{
|
||||
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup15),
|
||||
Evt("desktop_arrived"), Evt("hello_provisioning_completed"),
|
||||
}, hoursSinceStart: 6);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
Assert.Contains("user completed setup", reason);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_whiteglove_part2_real_enrollment_complete_still_reconciles_to_Succeeded()
|
||||
{
|
||||
var (status, _) = ClassifyWhiteGlove(new[]
|
||||
{
|
||||
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup05), Evt("enrollment_complete"),
|
||||
}, hoursSinceStart: 6);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_unresumed_session_keeps_terminal_complete_reconcile()
|
||||
{
|
||||
// No resumedAt (plain sessions, or a sealed Part 1 whose Pending write was lost):
|
||||
// whiteglove_complete keeps reconciling to Succeeded exactly as before.
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(new[]
|
||||
{
|
||||
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup05),
|
||||
});
|
||||
var (status, _, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, Start, Start.AddHours(6), 51, isPreProvisioned: true, resumedAt: null);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
}
|
||||
|
||||
// -------- Self-deploying profile gate (kiosk tenant aebdce78, audit 2026-08-23) --------
|
||||
|
||||
private static (SessionStatus, string) ClassifySelfDeploying(
|
||||
IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 3, int grace = 51)
|
||||
{
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
|
||||
var (status, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, Start, Start.AddHours(hoursSinceStart), grace,
|
||||
lastEventAtUtc: Start.AddMinutes(4), isSelfDeployingProfile: true);
|
||||
return (status, reason);
|
||||
}
|
||||
|
||||
// Session 195593e2 replay: DeviceSetup 4/4 (fallback-confirmed), AccountSetup registry
|
||||
// at 0/5 (the IME false positive — user ESP never runs on this profile), Hello disabled,
|
||||
// SkipUser=True, then silence after the post-ESP reboot. No user will ever sign in.
|
||||
private static readonly EnrollmentEvent[] KioskSilentStream =
|
||||
{
|
||||
Esp(DeviceSetup44), Esp(AccountSetup05), Esp(DeviceSetupFallback),
|
||||
HelloPolicy(enabled: false), EspConfig(skipUser: true),
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public void IsSelfDeployingProvisioned_requires_flag_device_setup_and_no_failure()
|
||||
{
|
||||
var provisioned = EnrollmentTimeoutClassifier.ExtractRollup(KioskSilentStream);
|
||||
Assert.True(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(provisioned, isSelfDeployingProfile: true));
|
||||
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(provisioned, isSelfDeployingProfile: false));
|
||||
|
||||
var beforeDeviceSetup = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(AccountSetup05) });
|
||||
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(beforeDeviceSetup, isSelfDeployingProfile: true));
|
||||
|
||||
var failed = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(DeviceSetup44), Evt("enrollment_failed") });
|
||||
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(failed, isSelfDeployingProfile: true));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_self_deploying_silent_after_device_setup_reconciles_to_Succeeded_within_grace()
|
||||
{
|
||||
var (status, reason) = ClassifySelfDeploying(KioskSilentStream, hoursSinceStart: 3);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
Assert.Contains("self-deploying profile", reason);
|
||||
Assert.Contains("no user / Account Setup phase", reason);
|
||||
Assert.Contains("Agent last reported 2026-07-06 15:04 UTC", reason);
|
||||
Assert.DoesNotContain("Account Setup completed", reason);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_self_deploying_silent_after_device_setup_reconciles_to_Succeeded_past_grace()
|
||||
{
|
||||
// Grace is a user-phase concept; it does not apply when there is no user phase.
|
||||
var (status, _) = ClassifySelfDeploying(KioskSilentStream, hoursSinceStart: 60, grace: 51);
|
||||
Assert.Equal(SessionStatus.Succeeded, status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_same_stream_without_self_deploying_flag_keeps_user_phase_verdicts()
|
||||
{
|
||||
// Guard against the gate leaking into user-driven sessions: the identical event shape
|
||||
// on a user-driven profile is still "awaiting user" within grace / Incomplete past it.
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(KioskSilentStream);
|
||||
var (within, _, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(3), 51);
|
||||
var (past, _, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(60), 51);
|
||||
Assert.Equal(SessionStatus.AwaitingUser, within);
|
||||
Assert.Equal(SessionStatus.Incomplete, past);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_self_deploying_explicit_failure_stays_Failed()
|
||||
{
|
||||
var (status, _) = ClassifySelfDeploying(new[] { Esp(DeviceSetup44), Evt("enrollment_failed") });
|
||||
Assert.Equal(SessionStatus.Failed, status);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classify_self_deploying_silent_before_device_setup_is_Incomplete()
|
||||
{
|
||||
// The profile flag alone proves nothing — Device ESP must have finished.
|
||||
var (status, _) = ClassifySelfDeploying(new[] { Esp(AccountSetup05), Evt("agent_started") });
|
||||
Assert.Equal(SessionStatus.Incomplete, status);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
using AutopilotMonitor.Functions.Functions.Config;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
@@ -32,6 +33,44 @@ public class GetAgentConfigFunctionTests
|
||||
Assert.Equal(expectedMajor, actual);
|
||||
}
|
||||
|
||||
// ── MergeDiagnosticsLogPaths — global ∪ tenant without duplicate ZIP entries ──────────
|
||||
|
||||
private static DiagnosticsLogPath P(string path, string description = "") =>
|
||||
new() { Path = path, Description = description };
|
||||
|
||||
[Fact]
|
||||
public void MergeDiagnosticsLogPaths_GlobalFirst_OrderPreserved_DuplicatesDroppedCaseInsensitively()
|
||||
{
|
||||
var global = new[] { P(@"C:\Windows\Panther\setupact.log", "global"), P(@"C:\Windows\Logs\DISM\dism.log") };
|
||||
var tenant = new[]
|
||||
{
|
||||
P(@"c:\windows\panther\SETUPACT.LOG", "tenant copy"), // same path, different case → dropped
|
||||
P(@" C:\Windows\Logs\DISM\dism.log "), // trimmed duplicate → dropped
|
||||
P(@"C:\Windows\SetupDiag\*.log", "tenant"),
|
||||
};
|
||||
|
||||
var merged = GetAgentConfigFunction.MergeDiagnosticsLogPaths(global, tenant);
|
||||
|
||||
Assert.Equal(new[]
|
||||
{
|
||||
@"C:\Windows\Panther\setupact.log",
|
||||
@"C:\Windows\Logs\DISM\dism.log",
|
||||
@"C:\Windows\SetupDiag\*.log",
|
||||
}, merged.Select(p => p.Path));
|
||||
Assert.Equal("global", merged[0].Description); // first occurrence (global) wins
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void MergeDiagnosticsLogPaths_DropsBlankEntries()
|
||||
{
|
||||
var merged = GetAgentConfigFunction.MergeDiagnosticsLogPaths(
|
||||
new[] { P(""), P(" "), P(@"C:\Install\Log\*.log") },
|
||||
new DiagnosticsLogPath[] { null!, P(@"C:\Install\Log\*.log") });
|
||||
|
||||
Assert.Single(merged);
|
||||
Assert.Equal(@"C:\Install\Log\*.log", merged[0].Path);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
// CustomerSas: gated on the per-tenant SAS URL being present.
|
||||
[InlineData("https://acct.blob.core.windows.net/c?sv=...", "CustomerSas", true)]
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
using System.Text.Json;
|
||||
using AutopilotMonitor.Functions.Functions.Diagnostics;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Wire shape of <c>GET /api/diagnostics/paths</c> (<see cref="GetDiagnosticsPathsFunction.BuildPayload"/>).
|
||||
/// The route is MemberRead, so the field set is locked: the payload must stay "what is
|
||||
/// collected" — never grow admin-only config.
|
||||
/// </summary>
|
||||
public class GetDiagnosticsPathsPayloadTests
|
||||
{
|
||||
// Mirrors the worker's serializer (Program.cs: PropertyNamingPolicy = CamelCase) so the
|
||||
// nested DiagnosticsLogPath class serializes the way the portal actually receives it.
|
||||
private static readonly JsonSerializerOptions WorkerJson = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase };
|
||||
|
||||
private static JsonElement Serialize(AdminConfiguration config)
|
||||
{
|
||||
var json = JsonSerializer.Serialize(GetDiagnosticsPathsFunction.BuildPayload(config), WorkerJson);
|
||||
return JsonDocument.Parse(json).RootElement;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Payload_HasExactlyBuiltInAndGlobalPaths()
|
||||
{
|
||||
var element = Serialize(new AdminConfiguration());
|
||||
|
||||
var fieldNames = element.EnumerateObject().Select(p => p.Name).OrderBy(n => n).ToArray();
|
||||
Assert.Equal(new[] { "builtIn", "globalPaths" }, fieldNames);
|
||||
Assert.Equal(JsonValueKind.Array, element.GetProperty("globalPaths").ValueKind);
|
||||
Assert.Equal(0, element.GetProperty("globalPaths").GetArrayLength());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuiltIn_MirrorsTheCatalog_WithConditionAsName()
|
||||
{
|
||||
var builtIn = Serialize(new AdminConfiguration()).GetProperty("builtIn");
|
||||
|
||||
Assert.Equal(DiagnosticsBuiltInSections.All.Count, builtIn.GetArrayLength());
|
||||
var first = builtIn[0];
|
||||
Assert.Equal("AgentLogs", first.GetProperty("id").GetString());
|
||||
Assert.Equal("AgentLogs", first.GetProperty("zipFolder").GetString());
|
||||
Assert.Equal(AutopilotMonitor.Shared.Constants.LogDirectory, first.GetProperty("sourceFolder").GetString());
|
||||
Assert.Equal(DiagnosticsBuiltInSections.LogFilePatterns.Length, first.GetProperty("patterns").GetArrayLength());
|
||||
Assert.False(first.GetProperty("includeSubfolders").GetBoolean());
|
||||
Assert.Equal("Always", first.GetProperty("condition").GetString());
|
||||
|
||||
var conditions = builtIn.EnumerateArray().Select(s => s.GetProperty("condition").GetString()).ToArray();
|
||||
Assert.Contains("RealmJoinWatcher", conditions);
|
||||
Assert.Contains("DevicePreparation", conditions);
|
||||
Assert.All(conditions, c => Assert.False(int.TryParse(c, out _), "condition must serialize as the enum name"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GlobalPaths_RoundTripTheStoredJson()
|
||||
{
|
||||
var config = new AdminConfiguration
|
||||
{
|
||||
DiagnosticsGlobalLogPathsJson =
|
||||
"[{\"path\":\"C:\\\\Windows\\\\Panther\\\\setupact.log\",\"description\":\"Setup\",\"isBuiltIn\":true,\"includeSubfolders\":false}]",
|
||||
};
|
||||
|
||||
var globalPaths = Serialize(config).GetProperty("globalPaths");
|
||||
|
||||
Assert.Equal(1, globalPaths.GetArrayLength());
|
||||
Assert.Equal(@"C:\Windows\Panther\setupact.log", globalPaths[0].GetProperty("path").GetString());
|
||||
Assert.Equal("Setup", globalPaths[0].GetProperty("description").GetString());
|
||||
}
|
||||
}
|
||||
+166
@@ -0,0 +1,166 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using Azure.Data.Tables;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Moq;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Pins that the ONE projected window scan the maintenance tick shares across its rolling
|
||||
/// sweeps (<c>GetMaintenanceWindowSessionsAsync</c> / <c>MaintenanceSweepSessionProjection</c>)
|
||||
/// carries every field the time-attribution, device-journey and verdict-calibration sweeps
|
||||
/// read. In Azure Table Storage a <c>$select</c> returns ONLY the listed properties, so the
|
||||
/// projected row here is built from the production array — dropping a column there fails the
|
||||
/// sweep that consumes it. Mirrors <see cref="UsageMetricsProjectionEquivalenceTests"/>.
|
||||
/// </summary>
|
||||
public class MaintenanceSweepProjectionEquivalenceTests
|
||||
{
|
||||
private const string TenantId = "00000000-0000-0000-0000-000000000abc";
|
||||
private static readonly DateTime Started = new(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private static readonly TableStorageService Sut =
|
||||
new(new Mock<TableServiceClient>().Object, NullLogger<TableStorageService>.Instance);
|
||||
|
||||
/// <summary>Full wide Sessions row with every column the sweeps read set to a distinct non-default value, plus noise.</summary>
|
||||
private static TableEntity FullRow() => new(TenantId, "11111111-1111-1111-1111-111111111111")
|
||||
{
|
||||
["Status"] = "Succeeded",
|
||||
["StartedAt"] = new DateTimeOffset(Started),
|
||||
["CompletedAt"] = new DateTimeOffset(Started.AddHours(1)),
|
||||
["LastEventAt"] = new DateTimeOffset(Started.AddHours(2)),
|
||||
["DurationSeconds"] = 3600,
|
||||
["EventCount"] = 321,
|
||||
["DeletionState"] = "None",
|
||||
["SerialNumber"] = "SN-1234",
|
||||
["Manufacturer"] = "Contoso",
|
||||
["Model"] = "Model-X",
|
||||
["EnrollmentType"] = "v2",
|
||||
["IsPreProvisioned"] = true,
|
||||
["IsSelfDeployingProfile"] = true,
|
||||
["IsUserDriven"] = false,
|
||||
["ResumedAt"] = new DateTimeOffset(Started.AddMinutes(30)),
|
||||
["AdminMarkedAction"] = "Succeeded",
|
||||
["VerdictPath"] = "manual:succeeded",
|
||||
["PriorStatus"] = "Incomplete",
|
||||
["PriorVerdictPath"] = "sweep:r6",
|
||||
["FailureReason"] = "No Device Setup completion or explicit failure signal observed before timeout",
|
||||
["ReconcileReason"] = "Reconciled at timeout: user completed setup (desktop + Windows Hello)",
|
||||
["FailureSource"] = "manual",
|
||||
["EspSoftFailure"] = true,
|
||||
// Noise the sweeps never read — present on the wide row, dropped by the projection.
|
||||
["DeviceName"] = "PC-FULL",
|
||||
["OsName"] = "Windows 11",
|
||||
["OsBuild"] = "26200.1234",
|
||||
["GeoCountry"] = "DE",
|
||||
["AgentVersion"] = "2.9.0",
|
||||
["DiagnosticsBlobName"] = "diag.zip",
|
||||
["FailureSnapshotJson"] = "{\"big\":\"" + new string('x', 2000) + "\"}",
|
||||
};
|
||||
|
||||
private static TableEntity Project(TableEntity full)
|
||||
{
|
||||
var keep = new HashSet<string>(TableStorageService.MaintenanceSweepSessionProjection, StringComparer.Ordinal);
|
||||
var projected = new TableEntity(full.PartitionKey, full.RowKey);
|
||||
foreach (var kv in full)
|
||||
{
|
||||
if (keep.Contains(kv.Key)) projected[kv.Key] = kv.Value;
|
||||
}
|
||||
return projected;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Projection_drops_the_wide_columns()
|
||||
{
|
||||
var projected = Project(FullRow());
|
||||
Assert.False(projected.ContainsKey("FailureSnapshotJson"));
|
||||
Assert.False(projected.ContainsKey("GeoCountry"));
|
||||
Assert.False(projected.ContainsKey("OsBuild"));
|
||||
Assert.True(projected.Count < FullRow().Count);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Every_field_the_sweeps_read_survives_the_projection()
|
||||
{
|
||||
var full = Sut.MapToSessionSummary(FullRow());
|
||||
var proj = Sut.MapToSessionSummary(Project(FullRow()));
|
||||
|
||||
// Shared / structural
|
||||
Assert.Equal(full.TenantId, proj.TenantId);
|
||||
Assert.Equal(full.SessionId, proj.SessionId);
|
||||
Assert.Equal(full.Status, proj.Status);
|
||||
Assert.Equal(full.StartedAt, proj.StartedAt);
|
||||
Assert.Equal(full.CompletedAt, proj.CompletedAt);
|
||||
Assert.Equal(full.DeletionState, proj.DeletionState);
|
||||
|
||||
// Time attribution: Status/DurationSeconds/DeletionState/EventCount + enrollment class
|
||||
Assert.Equal(full.DurationSeconds, proj.DurationSeconds);
|
||||
Assert.Equal(full.EventCount, proj.EventCount);
|
||||
Assert.Equal(TimeAttributionCalculator.GetEnrollmentClass(full), TimeAttributionCalculator.GetEnrollmentClass(proj));
|
||||
|
||||
// Device journeys: BuildSessionRef + BuildDeviceHistoryRow display fields
|
||||
Assert.Equal(full.SerialNumber, proj.SerialNumber);
|
||||
Assert.Equal(full.Manufacturer, proj.Manufacturer);
|
||||
Assert.Equal(full.Model, proj.Model);
|
||||
Assert.Equal(full.EnrollmentType, proj.EnrollmentType);
|
||||
Assert.Equal(full.IsPreProvisioned, proj.IsPreProvisioned);
|
||||
Assert.Equal(full.AdminMarkedAction, proj.AdminMarkedAction);
|
||||
var refFull = DeviceJourneyCalculator.BuildSessionRef(full)!;
|
||||
var refProj = DeviceJourneyCalculator.BuildSessionRef(proj)!;
|
||||
Assert.Equal(System.Text.Json.JsonSerializer.Serialize(refFull), System.Text.Json.JsonSerializer.Serialize(refProj));
|
||||
|
||||
// Verdict calibration: attribution + legacy derivation inputs + proxy timestamps
|
||||
Assert.Equal(full.LastEventAt, proj.LastEventAt);
|
||||
Assert.Equal(full.VerdictPath, proj.VerdictPath);
|
||||
Assert.Equal(full.PriorStatus, proj.PriorStatus);
|
||||
Assert.Equal(full.PriorVerdictPath, proj.PriorVerdictPath);
|
||||
Assert.Equal(full.FailureReason, proj.FailureReason);
|
||||
Assert.Equal(full.ReconcileReason, proj.ReconcileReason);
|
||||
Assert.Equal(full.FailureSource, proj.FailureSource);
|
||||
Assert.Equal(full.EspSoftFailure, proj.EspSoftFailure);
|
||||
Assert.Equal(VerdictPathDerivation.Derive(full), VerdictPathDerivation.Derive(proj));
|
||||
// Unstamped variant exercises the derivation on the projected inputs.
|
||||
var unstampedFull = FullRow(); unstampedFull.Remove("VerdictPath"); unstampedFull.Remove("AdminMarkedAction"); unstampedFull.Remove("FailureSource");
|
||||
Assert.Equal(VerdictPathDerivation.Derive(Sut.MapToSessionSummary(unstampedFull)),
|
||||
VerdictPathDerivation.Derive(Sut.MapToSessionSummary(Project(unstampedFull))));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Aggregation_cores_produce_identical_rows_from_the_projected_scan()
|
||||
{
|
||||
var now = new DateTime(2026, 8, 23, 12, 0, 0, DateTimeKind.Utc);
|
||||
var fullList = new List<SessionSummary> { Sut.MapToSessionSummary(FullRow()) };
|
||||
var projList = new List<SessionSummary> { Sut.MapToSessionSummary(Project(FullRow())) };
|
||||
var histories = new Dictionary<string, List<DeviceHistory>>();
|
||||
|
||||
var a = MaintenanceService.BuildVerdictCalibrationAggregates(fullList, histories, now);
|
||||
var b = MaintenanceService.BuildVerdictCalibrationAggregates(projList, histories, now);
|
||||
Assert.Equal(System.Text.Json.JsonSerializer.Serialize(a), System.Text.Json.JsonSerializer.Serialize(b));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SliceSweepWindow_keeps_sessions_from_the_window_start()
|
||||
{
|
||||
var window = new List<SessionSummary>
|
||||
{
|
||||
new() { SessionId = "old", StartedAt = Started.AddDays(-40) },
|
||||
new() { SessionId = "edge", StartedAt = Started },
|
||||
new() { SessionId = "new", StartedAt = Started.AddDays(1) },
|
||||
};
|
||||
var sliced = MaintenanceService.SliceSweepWindow(window, Started);
|
||||
Assert.Equal(new[] { "edge", "new" }, sliced.Select(s => s.SessionId));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Shared_window_covers_every_sweeps_horizon()
|
||||
{
|
||||
// 7d window + 28d baseline anchored on yesterday = 35 days back ≥ the 30-day rolling sweeps.
|
||||
Assert.Equal(35, MaintenanceService.SweepWindowDays);
|
||||
Assert.True(MaintenanceService.SweepWindowDays >= 30);
|
||||
}
|
||||
}
|
||||
@@ -1,4 +1,5 @@
|
||||
using AutopilotMonitor.Functions.Functions.Config;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Shared;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using AutopilotMonitor.Shared.Models.Notifications;
|
||||
@@ -64,6 +65,17 @@ public class NotificationChannelsTests
|
||||
Assert.Empty(slack.GetCustomHeaders());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetSigningSecret_OnlyForGenericJson()
|
||||
{
|
||||
var secret = "0123456789abcdef";
|
||||
Assert.Equal(secret, new NotificationChannel { Id = "g", ProviderType = 20, SigningSecret = secret }.GetSigningSecret());
|
||||
Assert.Null(new NotificationChannel { Id = "s", ProviderType = 10, SigningSecret = secret }.GetSigningSecret());
|
||||
Assert.Null(new NotificationChannel { Id = "d", ProviderType = 30, SigningSecret = secret }.GetSigningSecret());
|
||||
Assert.Null(new NotificationChannel { Id = "w", ProviderType = 20, SigningSecret = " " }.GetSigningSecret());
|
||||
Assert.Null(new NotificationChannel { Id = "n", ProviderType = 20 }.GetSigningSecret());
|
||||
}
|
||||
|
||||
// ── Legacy synthesis (GetNotificationChannels) ────────────────────────
|
||||
|
||||
[Fact]
|
||||
@@ -141,7 +153,7 @@ public class NotificationChannelsTests
|
||||
var cfg = new TenantConfiguration
|
||||
{
|
||||
NotificationChannelsJson = "[" +
|
||||
ChannelJson("ch-1", extra: ",\"customHeadersJson\":\"{\\\"X-Api-Key\\\":\\\"k\\\"}\"") + "," +
|
||||
ChannelJson("ch-1", extra: ",\"customHeadersJson\":\"{\\\"X-Api-Key\\\":\\\"k\\\"}\",\"signingSecret\":\"0123456789abcdef\"") + "," +
|
||||
ChannelJson("ch-2", providerType: 10, url: "https://slack.example/h") + "]",
|
||||
};
|
||||
|
||||
@@ -151,8 +163,10 @@ public class NotificationChannelsTests
|
||||
Assert.Equal("ch-1", redacted[0].Id); // structure visible
|
||||
Assert.Equal(Constants.RedactedSecretPlaceholder, redacted[0].Url); // secrets masked
|
||||
Assert.Equal(Constants.RedactedSecretPlaceholder, redacted[0].CustomHeadersJson);
|
||||
Assert.Equal(Constants.RedactedSecretPlaceholder, redacted[0].SigningSecret);
|
||||
Assert.Equal(Constants.RedactedSecretPlaceholder, redacted[1].Url);
|
||||
Assert.Null(redacted[1].CustomHeadersJson); // unset stays unset
|
||||
Assert.Null(redacted[1].SigningSecret);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -169,7 +183,7 @@ public class NotificationChannelsTests
|
||||
{
|
||||
NotificationChannelsJson = "[" +
|
||||
ChannelJson("ch-1", url: "https://real.example/hook",
|
||||
extra: ",\"customHeadersJson\":\"{\\\"X-Api-Key\\\":\\\"real\\\"}\"") + "]",
|
||||
extra: ",\"customHeadersJson\":\"{\\\"X-Api-Key\\\":\\\"real\\\"}\",\"signingSecret\":\"real-secret-0123\"") + "]",
|
||||
};
|
||||
|
||||
// Round-trip the redacted view back onto a save (defense-in-depth path).
|
||||
@@ -179,6 +193,7 @@ public class NotificationChannelsTests
|
||||
var ch = Assert.Single(NotificationChannel.ParseList(incoming.NotificationChannelsJson));
|
||||
Assert.Equal("https://real.example/hook", ch.Url);
|
||||
Assert.Equal("{\"X-Api-Key\":\"real\"}", ch.CustomHeadersJson);
|
||||
Assert.Equal("real-secret-0123", ch.SigningSecret);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -244,4 +259,40 @@ public class NotificationChannelsTests
|
||||
Assert.NotNull(error);
|
||||
Assert.Contains("headers", error);
|
||||
}
|
||||
|
||||
// ── Signing-secret validation ─────────────────────────────────────────
|
||||
|
||||
[Theory]
|
||||
[InlineData(null)]
|
||||
[InlineData("")]
|
||||
[InlineData(" ")]
|
||||
[InlineData("0123456789abcdef")] // 16 chars (minimum)
|
||||
public void ValidateWebhookSigningSecret_AcceptsAbsentOrValid(string? secret)
|
||||
{
|
||||
Assert.Null(TenantConfigValidation.ValidateWebhookSigningSecret(secret));
|
||||
// 128 chars (maximum) and a typical generated 64-char hex secret
|
||||
Assert.Null(TenantConfigValidation.ValidateWebhookSigningSecret(new string('a', 128)));
|
||||
Assert.Null(TenantConfigValidation.ValidateWebhookSigningSecret(new string('f', 64)));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("short-15-chars.")] // below minimum
|
||||
[InlineData("has a space in the middle!")] // whitespace
|
||||
[InlineData("line\nbreak-0123456789")] // control char
|
||||
[InlineData("umlauts-äöü-0123456789")] // non-ASCII
|
||||
[InlineData("***REDACTED***")] // unrestored sentinel (too short anyway)
|
||||
public void ValidateWebhookSigningSecret_RejectsInvalid(string secret)
|
||||
{
|
||||
Assert.NotNull(TenantConfigValidation.ValidateWebhookSigningSecret(secret));
|
||||
Assert.NotNull(TenantConfigValidation.ValidateWebhookSigningSecret(new string('a', 129)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ValidateNotificationChannels_RejectsBadPerChannelSigningSecret()
|
||||
{
|
||||
var json = "[" + ChannelJson("ch-1", extra: ",\"signingSecret\":\"too-short\"") + "]";
|
||||
var error = UpdateTenantConfigurationFunction.ValidateNotificationChannels(json);
|
||||
Assert.NotNull(error);
|
||||
Assert.Contains("signing secret", error);
|
||||
}
|
||||
}
|
||||
|
||||
+1
-1
@@ -7,7 +7,7 @@ namespace AutopilotMonitor.Functions.Tests.Offboarding;
|
||||
/// <summary>
|
||||
/// Test double for <see cref="IOffboardFarewellEmailSender"/>. Records every invocation so
|
||||
/// the handler-side tests can assert "did the farewell send fire after History → Completed?"
|
||||
/// without booting the real Resend client. Default = silent noop; set <see cref="ThrowOnSend"/>
|
||||
/// without booting the real mail provider client. Default = silent noop; set <see cref="ThrowOnSend"/>
|
||||
/// to simulate downstream failures and prove the handler's fail-soft contract.
|
||||
/// </summary>
|
||||
internal sealed class FakeOffboardFarewellEmailSender : IOffboardFarewellEmailSender
|
||||
|
||||
+4
-4
@@ -221,7 +221,7 @@ public class TenantOffboardingHandlerOrderingTests
|
||||
public async Task PostDrain_FarewellEmail_Skipped_WhenNotificationEmailNull()
|
||||
{
|
||||
// Tenants that never set a preview-notification email get null capture; the handler
|
||||
// skip-gates the send so no Resend call is even attempted.
|
||||
// skip-gates the send so no provider call is even attempted.
|
||||
var harness = Harness.New();
|
||||
Assert.Null(harness.Repo.History[HistoryRowKey].NotificationEmail);
|
||||
|
||||
@@ -235,7 +235,7 @@ public class TenantOffboardingHandlerOrderingTests
|
||||
public async Task PostDrain_FarewellEmail_Skipped_WhenNotificationEmailWhitespace()
|
||||
{
|
||||
// Defensive: legacy History rows may have a non-null empty/whitespace value. Treat
|
||||
// these as "no email captured" so we never call Resend with a useless string.
|
||||
// these as "no email captured" so we never call the mail provider with a useless string.
|
||||
var harness = Harness.New();
|
||||
harness.Repo.History[HistoryRowKey].NotificationEmail = " ";
|
||||
|
||||
@@ -248,11 +248,11 @@ public class TenantOffboardingHandlerOrderingTests
|
||||
[Fact]
|
||||
public async Task PostDrain_FarewellEmail_SenderThrows_HandlerSurvives_HistoryStillCompleted()
|
||||
{
|
||||
// The send is best-effort. A Resend outage / template bug / serializer crash must
|
||||
// The send is best-effort. A mail-provider outage / template bug / serializer crash must
|
||||
// NOT propagate to the queue worker — that would re-poison a Completed offboard.
|
||||
var harness = Harness.New();
|
||||
harness.Repo.History[HistoryRowKey].NotificationEmail = "ops@contoso.invalid";
|
||||
harness.FarewellEmail.ThrowOnSend = new InvalidOperationException("simulated Resend outage");
|
||||
harness.FarewellEmail.ThrowOnSend = new InvalidOperationException("simulated mail-provider outage");
|
||||
|
||||
await harness.Sut.HandleAsync(harness.Envelope());
|
||||
|
||||
|
||||
@@ -112,9 +112,11 @@ public class OffboardingSessionEnumeratorTests
|
||||
public Task<List<SessionSummary>> GetSessionsOlderThanAsync(string tenantId, DateTime cutoffDate, int maxResults = int.MaxValue, bool excludeInFlightDeletions = false) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetSessionsByDateRangeAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetUsageWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetMaintenanceWindowSessionsAsync(DateTime startDate, DateTime endDate) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetStalledSessionsAsync(string tenantId, DateTime cutoffTime) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId) => throw new NotImplementedException();
|
||||
public Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff) => throw new NotImplementedException();
|
||||
public Task<List<string>> GetAllTenantIdsAsync() => throw new NotImplementedException();
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
using AutopilotMonitor.Functions.Functions.Config;
|
||||
using AutopilotMonitor.Functions.Security;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Tests for <see cref="PlanManagementFunction.ApplyPlanChanges"/> — the pure mutation core
|
||||
/// of the PATCH plan endpoint. Pins the retention-grace anchor lifecycle: an EFFECTIVE
|
||||
/// Pro→Community transition stamps <see cref="TenantConfiguration.ProDowngradedUtc"/>, any
|
||||
/// effectively-Pro outcome clears it, and a planTier downgrade under an active trial does
|
||||
/// NOT stamp (the edition is still Pro — the later trial expiry is the anchor then).
|
||||
/// </summary>
|
||||
public class PlanManagementTransitionTests
|
||||
{
|
||||
private static readonly DateTime Now = new(2026, 8, 22, 12, 0, 0, DateTimeKind.Utc);
|
||||
private const string Caller = "ga@operator.example";
|
||||
|
||||
private static (TenantEdition Before, TenantEdition After, Dictionary<string, string> Changes) Apply(
|
||||
TenantConfiguration config, string? planTier = null, bool trialProvided = false, DateTime? trialExpiresUtc = null)
|
||||
{
|
||||
var changes = new Dictionary<string, string>();
|
||||
var (before, after) = PlanManagementFunction.ApplyPlanChanges(
|
||||
config, planTier, trialProvided, trialExpiresUtc, Caller, Now, changes);
|
||||
return (before, after, changes);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Downgrade_ProToCommunity_StampsAnchorAndRecordsChange()
|
||||
{
|
||||
var config = new TenantConfiguration { TenantId = "t1", PlanTier = "pro" };
|
||||
|
||||
var (before, after, changes) = Apply(config, planTier: "community");
|
||||
|
||||
Assert.Equal(TenantEdition.Pro, before);
|
||||
Assert.Equal(TenantEdition.Community, after);
|
||||
Assert.Equal(Now, config.ProDowngradedUtc);
|
||||
Assert.True(changes.ContainsKey("ProDowngradedUtc"));
|
||||
Assert.True(changes.ContainsKey("PlanTier"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Upgrade_CommunityToPro_ClearsAnchor()
|
||||
{
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = "t1",
|
||||
PlanTier = "community",
|
||||
ProDowngradedUtc = Now.AddDays(-10)
|
||||
};
|
||||
|
||||
var (_, after, changes) = Apply(config, planTier: "pro");
|
||||
|
||||
Assert.Equal(TenantEdition.Pro, after);
|
||||
Assert.Null(config.ProDowngradedUtc);
|
||||
Assert.True(changes.ContainsKey("ProDowngradedUtc"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Downgrade_WithActiveTrial_DoesNotStamp_EditionStaysPro()
|
||||
{
|
||||
// planTier drops but the trial keeps the EFFECTIVE edition Pro: no anchor now —
|
||||
// the trial's own expiry timestamp anchors the grace later, read-time.
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = "t1",
|
||||
PlanTier = "pro",
|
||||
TrialExpiresUtc = Now.AddDays(5)
|
||||
};
|
||||
|
||||
var (before, after, changes) = Apply(config, planTier: "community");
|
||||
|
||||
Assert.Equal(TenantEdition.Pro, before);
|
||||
Assert.Equal(TenantEdition.Pro, after);
|
||||
Assert.Null(config.ProDowngradedUtc);
|
||||
Assert.False(changes.ContainsKey("ProDowngradedUtc"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void EndingTrialExplicitly_StampsAnchor()
|
||||
{
|
||||
// GA sets trialExpiresUtc: null on a trial-Pro tenant — TrialExpiresUtc is gone, so
|
||||
// WITHOUT the explicit stamp the grace would have no anchor at all.
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = "t1",
|
||||
PlanTier = "free",
|
||||
TrialExpiresUtc = Now.AddDays(5)
|
||||
};
|
||||
|
||||
var (before, after, _) = Apply(config, trialProvided: true, trialExpiresUtc: null);
|
||||
|
||||
Assert.Equal(TenantEdition.Pro, before);
|
||||
Assert.Equal(TenantEdition.Community, after);
|
||||
Assert.Equal(Now, config.ProDowngradedUtc);
|
||||
Assert.Null(config.TrialExpiresUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GrantingTrial_ToDowngradedTenant_ClearsAnchor()
|
||||
{
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = "t1",
|
||||
PlanTier = "community",
|
||||
ProDowngradedUtc = Now.AddDays(-10)
|
||||
};
|
||||
|
||||
var (_, after, _) = Apply(config, trialProvided: true, trialExpiresUtc: Now.AddDays(30));
|
||||
|
||||
Assert.Equal(TenantEdition.Pro, after);
|
||||
Assert.Null(config.ProDowngradedUtc);
|
||||
Assert.Equal(Caller, config.TrialGrantedBy);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NoOpPatch_SameTier_RecordsNothing()
|
||||
{
|
||||
var config = new TenantConfiguration { TenantId = "t1", PlanTier = "community" };
|
||||
|
||||
var (before, after, changes) = Apply(config, planTier: "community");
|
||||
|
||||
Assert.Equal(TenantEdition.Community, before);
|
||||
Assert.Equal(TenantEdition.Community, after);
|
||||
Assert.Empty(changes);
|
||||
Assert.Null(config.ProDowngradedUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RepeatedDowngradeAfterReUpgrade_RestampsWithNewTimestamp()
|
||||
{
|
||||
var config = new TenantConfiguration { TenantId = "t1", PlanTier = "pro" };
|
||||
|
||||
Apply(config, planTier: "community");
|
||||
Assert.Equal(Now, config.ProDowngradedUtc);
|
||||
|
||||
Apply(config, planTier: "pro");
|
||||
Assert.Null(config.ProDowngradedUtc);
|
||||
|
||||
var changes = new Dictionary<string, string>();
|
||||
var later = Now.AddDays(40);
|
||||
PlanManagementFunction.ApplyPlanChanges(config, "community", false, null, Caller, later, changes);
|
||||
Assert.Equal(later, config.ProDowngradedUtc);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,76 @@
|
||||
using System.Collections.Concurrent;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Both sliding-window limiters keep the request history and its lock in one cache entry.
|
||||
/// Previously the entry was re-Set without its eviction callback, which (a) leaked per-key lock
|
||||
/// objects and (b) let two threads mutate the same List under different locks. These tests hammer
|
||||
/// one key from many threads: the count of allowed requests must be exactly the limit and nothing
|
||||
/// may throw (a List mutated concurrently throws InvalidOperationException or corrupts Count).
|
||||
/// </summary>
|
||||
public class RateLimitServiceConcurrencyTests
|
||||
{
|
||||
private const int Threads = 16;
|
||||
private const int RequestsPerThread = 500;
|
||||
|
||||
[Fact]
|
||||
public void CheckRateLimit_ManyThreadsSameKey_AllowsExactlyTheLimit()
|
||||
{
|
||||
var sut = new RateLimitService(new MemoryCache(new MemoryCacheOptions()), NullLogger<RateLimitService>.Instance);
|
||||
const int limit = 1000;
|
||||
var allowed = 0;
|
||||
var errors = new ConcurrentBag<Exception>();
|
||||
|
||||
RunInParallel(() =>
|
||||
{
|
||||
try
|
||||
{
|
||||
if (sut.CheckRateLimit("thumb-shared", limit).IsAllowed)
|
||||
Interlocked.Increment(ref allowed);
|
||||
}
|
||||
catch (Exception ex) { errors.Add(ex); }
|
||||
});
|
||||
|
||||
Assert.Empty(errors);
|
||||
Assert.Equal(limit, allowed);
|
||||
Assert.False(sut.CheckRateLimit("thumb-shared", limit).IsAllowed);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DistressCheck_ManyThreadsSameIp_AllowsExactlyTheIpLimit()
|
||||
{
|
||||
var sut = new DistressRateLimitService(new MemoryCache(new MemoryCacheOptions()), NullLogger<DistressRateLimitService>.Instance);
|
||||
var allowed = 0;
|
||||
var errors = new ConcurrentBag<Exception>();
|
||||
|
||||
RunInParallel(() =>
|
||||
{
|
||||
try
|
||||
{
|
||||
if (sut.Check("10.0.0.9", "a1b2c3d4-e5f6-7890-abcd-ef1234567890").IsAllowed)
|
||||
Interlocked.Increment(ref allowed);
|
||||
}
|
||||
catch (Exception ex) { errors.Add(ex); }
|
||||
});
|
||||
|
||||
Assert.Empty(errors);
|
||||
Assert.Equal(5, allowed); // MaxPerIp
|
||||
}
|
||||
|
||||
private static void RunInParallel(Action body)
|
||||
{
|
||||
using var gate = new ManualResetEventSlim(false);
|
||||
var tasks = Enumerable.Range(0, Threads).Select(_ => Task.Run(() =>
|
||||
{
|
||||
gate.Wait();
|
||||
for (var i = 0; i < RequestsPerThread; i++) body();
|
||||
})).ToArray();
|
||||
gate.Set();
|
||||
Task.WaitAll(tasks);
|
||||
}
|
||||
}
|
||||
@@ -1,134 +0,0 @@
|
||||
using System.Collections.Generic;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Coverage for the post-offboarding farewell-email send path. The actual Resend HTTP
|
||||
/// call is not exercised here — every test drives one of the no-op gates. Pins:
|
||||
/// <list type="bullet">
|
||||
/// <item><see cref="ResendEmailService.SendAsync"/> never throws — not when the API key
|
||||
/// is empty and not when the recipient is missing. The handler's fail-soft
|
||||
/// try/catch is a belt; this is the suspenders.</item>
|
||||
/// <item>Gate ordering: missing-key short-circuits before missing-recipient.</item>
|
||||
/// <item>The farewell template is final: no [DRAFT]/placeholder residue, domain is
|
||||
/// interpolated, empty domain falls back to "your organization".</item>
|
||||
/// </list>
|
||||
/// </summary>
|
||||
public sealed class ResendEmailServiceTests
|
||||
{
|
||||
[Fact]
|
||||
public async Task SendAsync_EmptyApiKey_NoOps_AndLogsMissingKeyDebugLine()
|
||||
{
|
||||
var logger = new CapturingLogger<ResendEmailService>();
|
||||
var sut = Build(apiKey: "", logger);
|
||||
|
||||
await sut.SendAsync(
|
||||
toEmail: "ops@contoso.invalid",
|
||||
domainName: "contoso.invalid",
|
||||
tenantId: "88888888-8888-8888-8888-888888888888",
|
||||
ct: CancellationToken.None);
|
||||
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Debug && e.Message.Contains("RESEND_API_KEY not configured"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SendAsync_EmptyRecipient_NoOps_AndLogsMissingEmailDebugLine()
|
||||
{
|
||||
var logger = new CapturingLogger<ResendEmailService>();
|
||||
var sut = Build(apiKey: "any-non-empty-key", logger);
|
||||
|
||||
await sut.SendAsync(
|
||||
toEmail: "",
|
||||
domainName: "contoso.invalid",
|
||||
tenantId: "88888888-8888-8888-8888-888888888888");
|
||||
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Debug && e.Message.Contains("No notification email captured"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SendAsync_EmptyApiKey_ShortCircuitsBeforeRecipientCheck()
|
||||
{
|
||||
// Pins the gate ordering: missing-key > missing-email. Both are empty here; only
|
||||
// the key line must show up.
|
||||
var logger = new CapturingLogger<ResendEmailService>();
|
||||
var sut = Build(apiKey: "", logger);
|
||||
|
||||
await sut.SendAsync(
|
||||
toEmail: "",
|
||||
domainName: "contoso.invalid",
|
||||
tenantId: "88888888-8888-8888-8888-888888888888");
|
||||
|
||||
Assert.Contains(logger.Entries,
|
||||
e => e.Level == LogLevel.Debug && e.Message.Contains("RESEND_API_KEY not configured"));
|
||||
Assert.DoesNotContain(logger.Entries,
|
||||
e => e.Message.Contains("No notification email captured"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FarewellTemplate_IsFinal_NoDraftResidue()
|
||||
{
|
||||
Assert.DoesNotContain("DRAFT", EmailTemplates.OffboardingFarewellSubject);
|
||||
|
||||
var html = EmailTemplates.GetOffboardingFarewellHtml("contoso.invalid");
|
||||
Assert.DoesNotContain("DRAFT", html);
|
||||
Assert.DoesNotContain("TEMPLATE NOT FINALISED", html);
|
||||
Assert.DoesNotContain("TODO", html);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FarewellTemplate_InterpolatesDomain_AndLinksFeedbackChannels()
|
||||
{
|
||||
var html = EmailTemplates.GetOffboardingFarewellHtml("contoso.invalid");
|
||||
|
||||
Assert.Contains("contoso.invalid", html);
|
||||
// The mail is sent from a noreply address after portal access is gone, so the
|
||||
// feedback pointers must be external channels the recipient can actually reach.
|
||||
Assert.Contains("https://github.com/okieselbach/AutopilotMonitor/issues", html);
|
||||
Assert.Contains("linkedin.com", html);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FarewellTemplate_EmptyDomain_FallsBackToGenericLabel()
|
||||
{
|
||||
var html = EmailTemplates.GetOffboardingFarewellHtml("");
|
||||
Assert.Contains("your organization", html);
|
||||
}
|
||||
|
||||
private static ResendEmailService Build(string apiKey, ILogger<ResendEmailService> logger)
|
||||
{
|
||||
var config = new ConfigurationBuilder()
|
||||
.AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["RESEND_API_KEY"] = apiKey,
|
||||
})
|
||||
.Build();
|
||||
return new ResendEmailService(config, logger);
|
||||
}
|
||||
|
||||
private sealed class CapturingLogger<T> : ILogger<T>
|
||||
{
|
||||
public readonly List<(LogLevel Level, string Message)> Entries = new();
|
||||
public IDisposable BeginScope<TState>(TState state) where TState : notnull => NullScope.Instance;
|
||||
public bool IsEnabled(LogLevel logLevel) => true;
|
||||
public void Log<TState>(
|
||||
LogLevel logLevel, EventId eventId, TState state, System.Exception? exception,
|
||||
System.Func<TState, System.Exception?, string> formatter)
|
||||
{
|
||||
Entries.Add((logLevel, formatter(state, exception)));
|
||||
}
|
||||
|
||||
private sealed class NullScope : IDisposable
|
||||
{
|
||||
public static readonly NullScope Instance = new();
|
||||
public void Dispose() { }
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -86,6 +86,9 @@ public class SessionIndexFieldManifestTests
|
||||
["ValidatedBy"] = "AutopilotV1",
|
||||
["FailureSnapshotJson"] = "{\"x\":1}",
|
||||
["AdminMarkedAction"] = "failed",
|
||||
["VerdictPath"] = "sweep:r5_incomplete",
|
||||
["PriorStatus"] = "Incomplete",
|
||||
["PriorVerdictPath"] = "maxlife:r6",
|
||||
["DurationSeconds"] = 360,
|
||||
["DiagnosticsBlobName"] = "diag-0001.zip",
|
||||
["DiagnosticsBlobDestination"] = "Hosted",
|
||||
|
||||
@@ -331,8 +331,9 @@ public class SessionRetentionFanoutServiceTests
|
||||
var baseTime = new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
var deadline = baseTime.AddMinutes(50);
|
||||
var clockCalls = 0;
|
||||
// Call sequence: tenant-A boundary (1), session s0 (2), session s1 (3), session s2 (4 → past deadline).
|
||||
var harness = new Harness(utcNow: () => ++clockCalls >= 4 ? deadline.AddMinutes(1) : baseTime);
|
||||
// Call sequence: tenant-A boundary (1), retention resolution (2), session s0 (3),
|
||||
// session s1 (4), session s2 (5 → past deadline).
|
||||
var harness = new Harness(utcNow: () => ++clockCalls >= 5 ? deadline.AddMinutes(1) : baseTime);
|
||||
|
||||
var sessions = new List<SessionSummary>();
|
||||
for (int i = 0; i < 4; i++) sessions.Add(Summary(TenantA, $"s{i}"));
|
||||
@@ -388,6 +389,38 @@ public class SessionRetentionFanoutServiceTests
|
||||
};
|
||||
}
|
||||
|
||||
// =========================================== Retention downgrade grace ====
|
||||
|
||||
[Fact]
|
||||
public async Task Tenant_InRetentionGrace_KeepsProCutoff()
|
||||
{
|
||||
// Downgraded 10 days ago, stored retention 365: the grace keeps the Pro cap, so the
|
||||
// cutoff must be now-365d — NOT now-90d (which would enqueue the 90–365d band).
|
||||
var now = new DateTime(2026, 8, 22, 12, 0, 0, DateTimeKind.Utc);
|
||||
var harness = new Harness(utcNow: () => now);
|
||||
harness.WithTenant("t1", retentionDays: 365, sessions: Array.Empty<SessionSummary>(),
|
||||
planTier: "community", proDowngradedUtc: now.AddDays(-10));
|
||||
|
||||
await harness.RunAsync();
|
||||
|
||||
harness.MaintenanceRepo.Verify(m => m.GetSessionsOlderThanAsync(
|
||||
"t1", now.AddDays(-365), It.IsAny<int>(), It.IsAny<bool>()), Times.Once);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Tenant_AfterRetentionGrace_CommunityCutoffApplies()
|
||||
{
|
||||
var now = new DateTime(2026, 8, 22, 12, 0, 0, DateTimeKind.Utc);
|
||||
var harness = new Harness(utcNow: () => now);
|
||||
harness.WithTenant("t1", retentionDays: 365, sessions: Array.Empty<SessionSummary>(),
|
||||
planTier: "community", proDowngradedUtc: now.AddDays(-31));
|
||||
|
||||
await harness.RunAsync();
|
||||
|
||||
harness.MaintenanceRepo.Verify(m => m.GetSessionsOlderThanAsync(
|
||||
"t1", now.AddDays(-90), It.IsAny<int>(), It.IsAny<bool>()), Times.Once);
|
||||
}
|
||||
|
||||
// ============================================================ Harness ====
|
||||
|
||||
private sealed class Harness
|
||||
@@ -450,11 +483,18 @@ public class SessionRetentionFanoutServiceTests
|
||||
return result;
|
||||
}
|
||||
|
||||
public void WithTenant(string tenantId, int retentionDays, SessionSummary[] sessions, string planTier = "free")
|
||||
public void WithTenant(string tenantId, int retentionDays, SessionSummary[] sessions, string planTier = "free",
|
||||
DateTime? proDowngradedUtc = null)
|
||||
{
|
||||
_tenantIds.Add(tenantId);
|
||||
TenantConfig.Setup(t => t.GetConfigurationAsync(tenantId))
|
||||
.ReturnsAsync(new TenantConfiguration { TenantId = tenantId, DataRetentionDays = retentionDays, PlanTier = planTier });
|
||||
.ReturnsAsync(new TenantConfiguration
|
||||
{
|
||||
TenantId = tenantId,
|
||||
DataRetentionDays = retentionDays,
|
||||
PlanTier = planTier,
|
||||
ProDowngradedUtc = proDowngradedUtc,
|
||||
});
|
||||
MaintenanceRepo.Setup(m => m.GetSessionsOlderThanAsync(tenantId, It.IsAny<DateTime>(), It.IsAny<int>(), It.IsAny<bool>()))
|
||||
.ReturnsAsync(new List<SessionSummary>(WithTenantId(tenantId, sessions)));
|
||||
}
|
||||
|
||||
+52
-6
@@ -74,7 +74,7 @@ public class SessionStartedAtBackwardShiftGuardTests
|
||||
var harness = new Harness(SessionRow(), earliestStoredEvent: poisoned);
|
||||
|
||||
var ok = await harness.Sut.UpdateSessionStatusAsync(
|
||||
TenantId, SessionId, SessionStatus.Pending,
|
||||
TenantId, SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
|
||||
earliestEventTimestamp: poisoned, latestEventTimestamp: lastEvent);
|
||||
|
||||
Assert.True(ok);
|
||||
@@ -93,7 +93,7 @@ public class SessionStartedAtBackwardShiftGuardTests
|
||||
var harness = new Harness(SessionRow(), earliestStoredEvent: earliest);
|
||||
|
||||
var ok = await harness.Sut.UpdateSessionStatusAsync(
|
||||
TenantId, SessionId, SessionStatus.Pending,
|
||||
TenantId, SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
|
||||
latestEventTimestamp: lastEvent);
|
||||
|
||||
Assert.True(ok);
|
||||
@@ -101,6 +101,52 @@ public class SessionStartedAtBackwardShiftGuardTests
|
||||
Assert.Equal((int)(lastEvent - earliest).TotalSeconds, harness.Written!.GetInt32("DurationSeconds"));
|
||||
}
|
||||
|
||||
// =========================== system-timeline rows are not session anchors ====
|
||||
|
||||
[Fact]
|
||||
public void IsSessionAnchorEligible_excludes_system_timeline_watcher_rows()
|
||||
{
|
||||
Assert.False(TableStorageService.IsSessionAnchorEligible(
|
||||
new TableEntity("p", "r") { ["Source"] = "SystemTimelineWatcher" }));
|
||||
Assert.True(TableStorageService.IsSessionAnchorEligible(
|
||||
new TableEntity("p", "r") { ["Source"] = "ImeLogTracker" }));
|
||||
Assert.True(TableStorageService.IsSessionAnchorEligible(
|
||||
new TableEntity("p", "r"))); // legacy row without Source projection
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Pending_duration_skips_backfilled_system_timeline_row_and_anchors_on_activity()
|
||||
{
|
||||
// Field case d0c5b672: a pre-agent system_clock_changed event (backfilled from the
|
||||
// Windows event log, original timestamp ~14min before the first real activity) was
|
||||
// the partition's earliest row and re-anchored StartedAt. The probe must skip
|
||||
// SystemTimelineWatcher rows and anchor on the first activity event instead —
|
||||
// even when the watcher row sits well inside the 2h plausibility window.
|
||||
var clockEventAt = StartedAt.AddMinutes(-90);
|
||||
var firstActivity = StartedAt.AddMinutes(-5);
|
||||
var lastEvent = StartedAt.AddMinutes(70);
|
||||
var harness = new Harness(SessionRow(), eventRows: new[]
|
||||
{
|
||||
EventRow(clockEventAt, sequence: 18, source: "SystemTimelineWatcher"),
|
||||
EventRow(firstActivity, sequence: 2, source: "ImeLogTracker"),
|
||||
});
|
||||
|
||||
var ok = await harness.Sut.UpdateSessionStatusAsync(
|
||||
TenantId, SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
|
||||
latestEventTimestamp: lastEvent);
|
||||
|
||||
Assert.True(ok);
|
||||
Assert.NotNull(harness.Written);
|
||||
Assert.Equal((int)(lastEvent - firstActivity).TotalSeconds, harness.Written!.GetInt32("DurationSeconds"));
|
||||
}
|
||||
|
||||
private static TableEntity EventRow(DateTime occurredUtc, int sequence, string source) =>
|
||||
new($"{TenantId}_{SessionId}", $"{occurredUtc:yyyyMMddHHmmssfff}_{sequence:D10}")
|
||||
{
|
||||
["OccurredUtc"] = new DateTimeOffset(occurredUtc),
|
||||
["Source"] = source,
|
||||
};
|
||||
|
||||
// ============================================================ Harness ====
|
||||
|
||||
private static TableEntity SessionRow()
|
||||
@@ -126,7 +172,7 @@ public class SessionStartedAtBackwardShiftGuardTests
|
||||
public TableStorageService Sut { get; }
|
||||
public TableEntity? Written { get; private set; }
|
||||
|
||||
public Harness(TableEntity existing, DateTime? earliestStoredEvent = null)
|
||||
public Harness(TableEntity existing, DateTime? earliestStoredEvent = null, TableEntity[]? eventRows = null)
|
||||
{
|
||||
var sessions = new Mock<TableClient>();
|
||||
|
||||
@@ -150,7 +196,7 @@ public class SessionStartedAtBackwardShiftGuardTests
|
||||
});
|
||||
|
||||
var events = new Mock<TableClient>();
|
||||
var eventRows = earliestStoredEvent.HasValue
|
||||
var servedRows = eventRows ?? (earliestStoredEvent.HasValue
|
||||
? new[]
|
||||
{
|
||||
new TableEntity($"{TenantId}_{SessionId}", $"{earliestStoredEvent.Value:yyyyMMddHHmmssfff}_0000000001")
|
||||
@@ -158,12 +204,12 @@ public class SessionStartedAtBackwardShiftGuardTests
|
||||
["OccurredUtc"] = new DateTimeOffset(earliestStoredEvent.Value),
|
||||
},
|
||||
}
|
||||
: Array.Empty<TableEntity>();
|
||||
: Array.Empty<TableEntity>());
|
||||
events.Setup(t => t.QueryAsync<TableEntity>(
|
||||
It.IsAny<string>(), It.IsAny<int?>(), It.IsAny<IEnumerable<string>>(), It.IsAny<CancellationToken>()))
|
||||
.Returns(AsyncPageable<TableEntity>.FromPages(new[]
|
||||
{
|
||||
Page<TableEntity>.FromValues(eventRows, null, new Mock<Response>().Object),
|
||||
Page<TableEntity>.FromValues(servedRows, null, new Mock<Response>().Object),
|
||||
}));
|
||||
|
||||
var serviceClient = new Mock<TableServiceClient>();
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared;
|
||||
using Azure;
|
||||
using Azure.Data.Tables;
|
||||
using Azure.Data.Tables.Models;
|
||||
using Microsoft.ApplicationInsights;
|
||||
using Microsoft.ApplicationInsights.Channel;
|
||||
using Microsoft.ApplicationInsights.DataContracts;
|
||||
using Microsoft.ApplicationInsights.Extensibility;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Moq;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests
|
||||
{
|
||||
public class StartupTelemetryServiceTests
|
||||
{
|
||||
private sealed class CapturingChannel : ITelemetryChannel
|
||||
{
|
||||
public List<ITelemetry> Items { get; } = new();
|
||||
public bool? DeveloperMode { get; set; }
|
||||
public string EndpointAddress { get; set; } = "";
|
||||
public void Send(ITelemetry item) => Items.Add(item);
|
||||
public void Flush() { }
|
||||
public void Dispose() { }
|
||||
}
|
||||
|
||||
private sealed class Lifetime : IHostApplicationLifetime
|
||||
{
|
||||
private readonly CancellationTokenSource _started = new();
|
||||
public CancellationToken ApplicationStarted => _started.Token;
|
||||
public CancellationToken ApplicationStopping => CancellationToken.None;
|
||||
public CancellationToken ApplicationStopped => CancellationToken.None;
|
||||
public void StopApplication() { }
|
||||
public void FireStarted() => _started.Cancel();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ApplicationStarted_EmitsStartupAndTableInitMetrics()
|
||||
{
|
||||
var channel = new CapturingChannel();
|
||||
var telemetry = new TelemetryClient(new TelemetryConfiguration { TelemetryChannel = channel, ConnectionString = "InstrumentationKey=00000000-0000-0000-0000-000000000000" });
|
||||
|
||||
// Table storage: unreadable sentinel → full pass → FullPassRan = true.
|
||||
var service = new Mock<TableServiceClient>();
|
||||
var admin = new Mock<TableClient>();
|
||||
service.Setup(s => s.GetTableClient(It.IsAny<string>())).Returns(admin.Object);
|
||||
admin.Setup(c => c.GetEntityIfExistsAsync<TableEntity>(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<string>>(), It.IsAny<CancellationToken>()))
|
||||
.ThrowsAsync(new RequestFailedException(404, "TableNotFound"));
|
||||
admin.Setup(c => c.UpsertEntityAsync(It.IsAny<TableEntity>(), It.IsAny<TableUpdateMode>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(Mock.Of<Response>());
|
||||
service.Setup(s => s.CreateTableIfNotExistsAsync(It.IsAny<string>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(Response.FromValue(new TableItem("x"), Mock.Of<Response>()));
|
||||
var storage = new TableStorageService(service.Object, NullLogger<TableStorageService>.Instance);
|
||||
await storage.InitializeTablesAsync();
|
||||
|
||||
var lifetime = new Lifetime();
|
||||
var sut = new StartupTelemetryService(lifetime, telemetry, storage, new BackendBuildInfo(), NullLogger<StartupTelemetryService>.Instance);
|
||||
await sut.StartAsync(CancellationToken.None);
|
||||
Assert.Empty(channel.Items);
|
||||
|
||||
lifetime.FireStarted();
|
||||
|
||||
var metrics = channel.Items.ConvertAll(i => (MetricTelemetry)i);
|
||||
Assert.Equal(2, metrics.Count);
|
||||
var startup = metrics.Find(m => m.Name == StartupTelemetryService.StartupMetricName)!;
|
||||
var tableInit = metrics.Find(m => m.Name == StartupTelemetryService.TableInitMetricName)!;
|
||||
Assert.True(startup.Sum > 0);
|
||||
Assert.True(tableInit.Sum >= 0);
|
||||
Assert.Equal("True", startup.Properties["tableInitFullPass"]);
|
||||
Assert.False(string.IsNullOrEmpty(startup.Properties["version"]));
|
||||
Assert.True(startup.Sum >= tableInit.Sum);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
using System;
|
||||
using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared;
|
||||
using Azure;
|
||||
using Azure.Data.Tables;
|
||||
using Azure.Data.Tables.Models;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Moq;
|
||||
using Xunit;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests
|
||||
{
|
||||
/// <summary>
|
||||
/// Startup table initialization is gated by a schema sentinel (hash over TableNames.All)
|
||||
/// stored in AdminConfiguration: a matching sentinel means zero CreateTableIfNotExists
|
||||
/// calls on a cold start; anything else runs the full idempotent pass and rewrites it.
|
||||
/// </summary>
|
||||
public class TableSchemaSentinelTests
|
||||
{
|
||||
private sealed class Harness
|
||||
{
|
||||
public Mock<TableServiceClient> Service { get; } = new();
|
||||
public Mock<TableClient> AdminConfig { get; } = new();
|
||||
public TableEntity? UpsertedSentinel { get; private set; }
|
||||
public int CreateCalls;
|
||||
|
||||
public Harness()
|
||||
{
|
||||
Service.Setup(s => s.GetTableClient(Constants.TableNames.AdminConfiguration)).Returns(AdminConfig.Object);
|
||||
Service.Setup(s => s.CreateTableIfNotExistsAsync(It.IsAny<string>(), It.IsAny<CancellationToken>()))
|
||||
.Returns<string, CancellationToken>((_, _) =>
|
||||
{
|
||||
Interlocked.Increment(ref CreateCalls);
|
||||
return Task.FromResult(Response.FromValue(new TableItem("x"), Mock.Of<Response>()));
|
||||
});
|
||||
AdminConfig.Setup(c => c.UpsertEntityAsync(It.IsAny<TableEntity>(), It.IsAny<TableUpdateMode>(), It.IsAny<CancellationToken>()))
|
||||
.Callback<ITableEntity, TableUpdateMode, CancellationToken>((e, _, _) => UpsertedSentinel = (TableEntity)e)
|
||||
.ReturnsAsync(Mock.Of<Response>());
|
||||
}
|
||||
|
||||
public void SentinelReturns(string? hash)
|
||||
{
|
||||
var response = new Mock<NullableResponse<TableEntity>>();
|
||||
response.SetupGet(r => r.HasValue).Returns(hash != null);
|
||||
if (hash != null)
|
||||
{
|
||||
response.SetupGet(r => r.Value).Returns(new TableEntity(
|
||||
TableStorageService.SchemaSentinelPartitionKey, TableStorageService.SchemaSentinelRowKey)
|
||||
{
|
||||
[TableStorageService.SchemaSentinelHashProperty] = hash
|
||||
});
|
||||
}
|
||||
AdminConfig.Setup(c => c.GetEntityIfExistsAsync<TableEntity>(
|
||||
TableStorageService.SchemaSentinelPartitionKey, TableStorageService.SchemaSentinelRowKey,
|
||||
It.IsAny<System.Collections.Generic.IEnumerable<string>>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(response.Object);
|
||||
}
|
||||
|
||||
public void SentinelThrows()
|
||||
{
|
||||
AdminConfig.Setup(c => c.GetEntityIfExistsAsync<TableEntity>(
|
||||
It.IsAny<string>(), It.IsAny<string>(),
|
||||
It.IsAny<System.Collections.Generic.IEnumerable<string>>(), It.IsAny<CancellationToken>()))
|
||||
.ThrowsAsync(new RequestFailedException(404, "TableNotFound"));
|
||||
}
|
||||
|
||||
public TableStorageService Build() => new(Service.Object, NullLogger<TableStorageService>.Instance);
|
||||
}
|
||||
|
||||
private static string CurrentHash => TableStorageService.ComputeTableSchemaHash(Constants.TableNames.All);
|
||||
|
||||
[Fact]
|
||||
public void Hash_IsOrderIndependent_AndChangesWhenTableAdded()
|
||||
{
|
||||
var a = TableStorageService.ComputeTableSchemaHash(new[] { "B", "A" });
|
||||
var b = TableStorageService.ComputeTableSchemaHash(new[] { "A", "B" });
|
||||
var c = TableStorageService.ComputeTableSchemaHash(new[] { "A", "B", "C" });
|
||||
|
||||
Assert.Equal(a, b);
|
||||
Assert.NotEqual(a, c);
|
||||
Assert.Equal(64, a.Length);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MatchingSentinel_SkipsTableCreation()
|
||||
{
|
||||
var h = new Harness();
|
||||
h.SentinelReturns(CurrentHash);
|
||||
|
||||
var fullPass = await h.Build().InitializeTablesAsync();
|
||||
|
||||
Assert.False(fullPass);
|
||||
Assert.Equal(0, h.CreateCalls);
|
||||
Assert.Null(h.UpsertedSentinel);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task StaleSentinel_RunsFullPass_AndRewritesSentinel()
|
||||
{
|
||||
var h = new Harness();
|
||||
h.SentinelReturns("STALE");
|
||||
|
||||
var fullPass = await h.Build().InitializeTablesAsync();
|
||||
|
||||
Assert.True(fullPass);
|
||||
Assert.Equal(Constants.TableNames.All.Length, h.CreateCalls);
|
||||
Assert.NotNull(h.UpsertedSentinel);
|
||||
Assert.Equal(CurrentHash, h.UpsertedSentinel!.GetString(TableStorageService.SchemaSentinelHashProperty));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task MissingSentinel_RunsFullPass()
|
||||
{
|
||||
var h = new Harness();
|
||||
h.SentinelReturns(null);
|
||||
|
||||
Assert.True(await h.Build().InitializeTablesAsync());
|
||||
Assert.Equal(Constants.TableNames.All.Length, h.CreateCalls);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task UnreadableSentinel_FreshStorage_RunsFullPass()
|
||||
{
|
||||
var h = new Harness();
|
||||
h.SentinelThrows();
|
||||
|
||||
Assert.True(await h.Build().InitializeTablesAsync());
|
||||
Assert.Equal(Constants.TableNames.All.Length, h.CreateCalls);
|
||||
Assert.NotNull(h.UpsertedSentinel);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task FailedTableCreate_DoesNotWriteSentinel_AndRetriesNextCall()
|
||||
{
|
||||
var h = new Harness();
|
||||
h.SentinelReturns(null);
|
||||
var failing = Constants.TableNames.All.First();
|
||||
h.Service.Setup(s => s.CreateTableIfNotExistsAsync(failing, It.IsAny<CancellationToken>()))
|
||||
.ThrowsAsync(new RequestFailedException(500, "boom"));
|
||||
var svc = h.Build();
|
||||
|
||||
Assert.True(await svc.InitializeTablesAsync());
|
||||
Assert.Null(h.UpsertedSentinel);
|
||||
|
||||
// Not marked initialized → a second call runs the pass again instead of short-circuiting.
|
||||
var before = h.CreateCalls;
|
||||
Assert.True(await svc.InitializeTablesAsync());
|
||||
Assert.True(h.CreateCalls > before);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SecondCallAfterSuccess_IsNoOp()
|
||||
{
|
||||
var h = new Harness();
|
||||
h.SentinelReturns(null);
|
||||
var svc = h.Build();
|
||||
|
||||
await svc.InitializeTablesAsync();
|
||||
var calls = h.CreateCalls;
|
||||
Assert.False(await svc.InitializeTablesAsync());
|
||||
Assert.Equal(calls, h.CreateCalls);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task BackfillClaim_WinnerTrue_LoserFalse()
|
||||
{
|
||||
var h = new Harness();
|
||||
h.AdminConfig.SetupSequence(c => c.AddEntityAsync(It.IsAny<TableEntity>(), It.IsAny<CancellationToken>()))
|
||||
.ReturnsAsync(Mock.Of<Response>())
|
||||
.ThrowsAsync(new RequestFailedException(409, "EntityAlreadyExists"));
|
||||
var svc = h.Build();
|
||||
|
||||
Assert.True(await svc.TryClaimSessionIndexBackfillAsync());
|
||||
Assert.False(await svc.TryClaimSessionIndexBackfillAsync());
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -21,7 +21,7 @@ public class TenantApprovalServiceTests
|
||||
|
||||
private readonly Mock<PreviewWhitelistService> _previewMock;
|
||||
private readonly Mock<TenantConfigurationService> _tenantConfigMock;
|
||||
private readonly Mock<ResendEmailService> _resendMock;
|
||||
private readonly Mock<IEmailService> _emailMock;
|
||||
private readonly TenantApprovalService _sut;
|
||||
|
||||
public TenantApprovalServiceTests()
|
||||
@@ -37,9 +37,10 @@ public class TenantApprovalServiceTests
|
||||
configRepo, cache, NullLogger<PreviewWhitelistService>.Instance, _tenantConfigMock.Object)
|
||||
{ CallBase = false };
|
||||
|
||||
_resendMock = new Mock<ResendEmailService>(
|
||||
Mock.Of<Microsoft.Extensions.Configuration.IConfiguration>(),
|
||||
NullLogger<ResendEmailService>.Instance);
|
||||
_emailMock = new Mock<IEmailService>();
|
||||
_emailMock.Setup(x => x.SendPreviewApprovedEmailAsync(
|
||||
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()))
|
||||
.Returns(Task.CompletedTask);
|
||||
|
||||
_sut = new TenantApprovalService(
|
||||
NullLogger<TenantApprovalService>.Instance,
|
||||
@@ -47,7 +48,7 @@ public class TenantApprovalServiceTests
|
||||
_tenantConfigMock.Object,
|
||||
new Mock<TenantAdminsService>(
|
||||
Mock.Of<IAdminRepository>(), cache, NullLogger<TenantAdminsService>.Instance).Object,
|
||||
_resendMock.Object);
|
||||
_emailMock.Object);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -94,7 +95,7 @@ public class TenantApprovalServiceTests
|
||||
|
||||
Assert.False(sent);
|
||||
_previewMock.Verify(x => x.TryMarkWelcomeEmailSentAsync(It.IsAny<string>()), Times.Never);
|
||||
_resendMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
|
||||
_emailMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()), Times.Never);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -109,7 +110,7 @@ public class TenantApprovalServiceTests
|
||||
var sent = await _sut.TrySendWelcomeEmailAsync(TenantId);
|
||||
|
||||
Assert.True(sent);
|
||||
_resendMock.Verify(x => x.SendPreviewApprovedEmailAsync("it@contoso.com", "contoso.com"), Times.Once);
|
||||
_emailMock.Verify(x => x.SendPreviewApprovedEmailAsync("it@contoso.com", "contoso.com", It.IsAny<CancellationToken>()), Times.Once);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -121,7 +122,7 @@ public class TenantApprovalServiceTests
|
||||
var sent = await _sut.TrySendWelcomeEmailAsync(TenantId);
|
||||
|
||||
Assert.False(sent);
|
||||
_resendMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
|
||||
_emailMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()), Times.Never);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -134,7 +135,7 @@ public class TenantApprovalServiceTests
|
||||
var sent = await _sut.TrySendWelcomeEmailAsync(TenantId);
|
||||
|
||||
Assert.False(sent);
|
||||
_resendMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
|
||||
_emailMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()), Times.Never);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -151,6 +152,6 @@ public class TenantApprovalServiceTests
|
||||
var newlyApproved = await _sut.ApproveWithSideEffectsAsync(TenantId, ApprovedBy);
|
||||
|
||||
Assert.True(newlyApproved);
|
||||
_resendMock.Verify(x => x.SendPreviewApprovedEmailAsync("it@contoso.com", It.IsAny<string>()), Times.Once);
|
||||
_emailMock.Verify(x => x.SendPreviewApprovedEmailAsync("it@contoso.com", It.IsAny<string>(), It.IsAny<CancellationToken>()), Times.Once);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -50,9 +50,7 @@ public class TenantAutoApproveHandlerTests
|
||||
_tenantConfigMock.Object,
|
||||
new Mock<TenantAdminsService>(
|
||||
Mock.Of<IAdminRepository>(), cache, NullLogger<TenantAdminsService>.Instance).Object,
|
||||
new Mock<ResendEmailService>(
|
||||
Mock.Of<Microsoft.Extensions.Configuration.IConfiguration>(),
|
||||
NullLogger<ResendEmailService>.Instance).Object)
|
||||
Mock.Of<IEmailService>())
|
||||
{ CallBase = false };
|
||||
|
||||
var opsRepo = new Mock<IOpsEventRepository>();
|
||||
|
||||
+31
-1
@@ -20,6 +20,7 @@ public class TenantConfigTableSerializationTests
|
||||
{
|
||||
var expires = new DateTime(2026, 8, 6, 10, 30, 0, DateTimeKind.Utc);
|
||||
var started = new DateTime(2026, 7, 7, 10, 30, 0, DateTimeKind.Utc);
|
||||
var downgraded = new DateTime(2026, 8, 10, 8, 0, 0, DateTimeKind.Utc);
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = TenantId,
|
||||
@@ -29,7 +30,8 @@ public class TenantConfigTableSerializationTests
|
||||
TrialExpiresUtc = expires,
|
||||
TrialStartedUtc = started,
|
||||
TrialConsumed = true,
|
||||
TrialGrantedBy = "alice@contoso.com"
|
||||
TrialGrantedBy = "alice@contoso.com",
|
||||
ProDowngradedUtc = downgraded
|
||||
};
|
||||
|
||||
var entity = TableConfigRepository.ConvertToTenantTableEntity(config);
|
||||
@@ -40,6 +42,7 @@ public class TenantConfigTableSerializationTests
|
||||
Assert.Equal(started, mapped.TrialStartedUtc);
|
||||
Assert.True(mapped.TrialConsumed);
|
||||
Assert.Equal("alice@contoso.com", mapped.TrialGrantedBy);
|
||||
Assert.Equal(downgraded, mapped.ProDowngradedUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -59,6 +62,7 @@ public class TenantConfigTableSerializationTests
|
||||
Assert.Null(mapped.TrialStartedUtc);
|
||||
Assert.False(mapped.TrialConsumed);
|
||||
Assert.Null(mapped.TrialGrantedBy);
|
||||
Assert.Null(mapped.ProDowngradedUtc);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -77,6 +81,7 @@ public class TenantConfigTableSerializationTests
|
||||
Assert.Null(mapped.TrialExpiresUtc);
|
||||
Assert.False(mapped.TrialConsumed);
|
||||
Assert.Null(mapped.TrialGrantedBy);
|
||||
Assert.Null(mapped.ProDowngradedUtc); // pre-grace row: no anchor, no grace
|
||||
}
|
||||
|
||||
[Fact]
|
||||
@@ -120,6 +125,31 @@ public class TenantConfigTableSerializationTests
|
||||
Assert.Null(mapped.AbsoluteMaxSessionHours);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Roundtrip_EnableDoGroupIdAutoSet_SurvivesStoreAndMap()
|
||||
{
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = TenantId,
|
||||
DomainName = "contoso.com",
|
||||
UpdatedBy = "admin@contoso.com",
|
||||
EnableDoGroupIdAutoSet = true
|
||||
};
|
||||
|
||||
var mapped = TableConfigRepository.ConvertFromTenantTableEntity(
|
||||
TableConfigRepository.ConvertToTenantTableEntity(config));
|
||||
|
||||
Assert.True(mapped.EnableDoGroupIdAutoSet);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Map_LegacyRow_WithoutDoGroupIdColumn_ReadsNull()
|
||||
{
|
||||
// null = "not set" → GetAgentConfigFunction serves the agent default (false).
|
||||
var entity = new TableEntity(TenantId, "config") { { "DomainName", "fabrikam.com" } };
|
||||
Assert.Null(TableConfigRepository.ConvertFromTenantTableEntity(entity).EnableDoGroupIdAutoSet);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Roundtrip_NotificationChannelsJson_SurvivesStoreAndMap()
|
||||
{
|
||||
|
||||
@@ -194,8 +194,10 @@ public class TenantEntitlementServiceTests
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetEffectiveRetentionDays_TrialExpiry_DegradesCapFrom365To90()
|
||||
public void GetEffectiveRetentionDays_TrialExpiry_ProCapHoldsThroughGrace_ThenDegradesTo90()
|
||||
{
|
||||
// Trial expiry is a downgrade: the Pro cap keeps applying for the 30-day retention
|
||||
// grace (anchor = TrialExpiresUtc, no write involved), THEN the Community cap bites.
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = TenantId,
|
||||
@@ -204,6 +206,93 @@ public class TenantEntitlementServiceTests
|
||||
TrialExpiresUtc = Now.AddDays(1)
|
||||
};
|
||||
Assert.Equal(365, TenantEntitlementService.GetEffectiveRetentionDays(config, Now));
|
||||
Assert.Equal(90, TenantEntitlementService.GetEffectiveRetentionDays(config, Now.AddDays(2)));
|
||||
// 1 day after expiry: inside the grace — still the Pro cap.
|
||||
Assert.Equal(365, TenantEntitlementService.GetEffectiveRetentionDays(config, Now.AddDays(2)));
|
||||
// 31 days after expiry: grace over — Community cap enforced.
|
||||
Assert.Equal(90, TenantEntitlementService.GetEffectiveRetentionDays(config, Now.AddDays(32)));
|
||||
}
|
||||
|
||||
// ── Retention downgrade grace (GetRetentionGraceEndUtc + cap interplay) ──────
|
||||
|
||||
[Fact]
|
||||
public void GetRetentionGraceEndUtc_ProTenant_ReturnsNull()
|
||||
{
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = TenantId,
|
||||
PlanTier = "pro",
|
||||
ProDowngradedUtc = Now.AddDays(-5) // stale leftover must not matter while Pro
|
||||
};
|
||||
Assert.Null(TenantEntitlementService.GetRetentionGraceEndUtc(config, Now));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetRetentionGraceEndUtc_NeverLostPro_ReturnsNull()
|
||||
{
|
||||
var config = new TenantConfiguration { TenantId = TenantId, PlanTier = "free" };
|
||||
Assert.Null(TenantEntitlementService.GetRetentionGraceEndUtc(config, Now));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetRetentionGraceEndUtc_ExplicitDowngrade_AnchorPlusGraceDays()
|
||||
{
|
||||
var downgraded = Now.AddDays(-10);
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = TenantId,
|
||||
PlanTier = "community",
|
||||
ProDowngradedUtc = downgraded
|
||||
};
|
||||
Assert.Equal(
|
||||
downgraded.AddDays(FeatureEntitlementCatalog.RetentionDowngradeGraceDays),
|
||||
TenantEntitlementService.GetRetentionGraceEndUtc(config, Now));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetRetentionGraceEndUtc_LatestAnchorWins()
|
||||
{
|
||||
// Downgrade, re-granted trial, trial expired again: the LATER trial expiry anchors.
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = TenantId,
|
||||
PlanTier = "community",
|
||||
ProDowngradedUtc = Now.AddDays(-20),
|
||||
TrialExpiresUtc = Now.AddDays(-3)
|
||||
};
|
||||
Assert.Equal(
|
||||
Now.AddDays(-3).AddDays(FeatureEntitlementCatalog.RetentionDowngradeGraceDays),
|
||||
TenantEntitlementService.GetRetentionGraceEndUtc(config, Now));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetEffectiveRetentionDays_ExplicitDowngrade_ProCapDuringGrace_CommunityCapAfter()
|
||||
{
|
||||
var config = new TenantConfiguration
|
||||
{
|
||||
TenantId = TenantId,
|
||||
PlanTier = "community",
|
||||
DataRetentionDays = 365,
|
||||
ProDowngradedUtc = Now
|
||||
};
|
||||
Assert.Equal(365, TenantEntitlementService.GetEffectiveRetentionDays(config, Now.AddDays(29)));
|
||||
Assert.Equal(90, TenantEntitlementService.GetEffectiveRetentionDays(config, Now.AddDays(31)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GetEffectiveRetentionDays_GraceDoesNotRaiseAnythingElse()
|
||||
{
|
||||
// 0 (infinite) passes through untouched, and a stored value below the Community cap
|
||||
// is unaffected — the grace only swaps which CAP clamps.
|
||||
var zero = new TenantConfiguration
|
||||
{
|
||||
TenantId = TenantId, PlanTier = "community", DataRetentionDays = 0, ProDowngradedUtc = Now
|
||||
};
|
||||
Assert.Equal(0, TenantEntitlementService.GetEffectiveRetentionDays(zero, Now.AddDays(5)));
|
||||
|
||||
var below = new TenantConfiguration
|
||||
{
|
||||
TenantId = TenantId, PlanTier = "community", DataRetentionDays = 60, ProDowngradedUtc = Now
|
||||
};
|
||||
Assert.Equal(60, TenantEntitlementService.GetEffectiveRetentionDays(below, Now.AddDays(5)));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -20,7 +20,8 @@ public class TrialExpirySweepFunctionTests
|
||||
private static readonly DateTime Now = new(2026, 7, 7, 3, 30, 0, DateTimeKind.Utc);
|
||||
|
||||
private static TenantConfiguration Tenant(
|
||||
string id, DateTime? trialExpiresUtc, string planTier = "free", string domain = "contoso.com") => new()
|
||||
string id, DateTime? trialExpiresUtc, string planTier = "free", string domain = "contoso.com",
|
||||
int retentionDays = 90, DateTime? proDowngradedUtc = null) => new()
|
||||
{
|
||||
TenantId = id,
|
||||
DomainName = domain,
|
||||
@@ -28,6 +29,8 @@ public class TrialExpirySweepFunctionTests
|
||||
PlanTier = planTier,
|
||||
TrialExpiresUtc = trialExpiresUtc,
|
||||
TrialConsumed = trialExpiresUtc.HasValue,
|
||||
DataRetentionDays = retentionDays,
|
||||
ProDowngradedUtc = proDowngradedUtc,
|
||||
};
|
||||
|
||||
private static (TrialExpirySweepFunction Sut, List<OpsEventEntry> Events) Build(params TenantConfiguration[] configs)
|
||||
@@ -150,6 +153,88 @@ public class TrialExpirySweepFunctionTests
|
||||
Assert.Equal(2, events.Count);
|
||||
}
|
||||
|
||||
// ── Retention downgrade grace events ─────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public async Task GraceEndingSoon_WithDataAtRisk_EmitsGraceExpiring()
|
||||
{
|
||||
// Downgraded 28 days ago with stored retention 365: grace ends in 2 days.
|
||||
var (sut, events) = Build(Tenant("t1", null, planTier: "community",
|
||||
retentionDays: 365, proDowngradedUtc: Now.AddDays(-28)));
|
||||
|
||||
var result = await sut.RunCoreAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(1, result.GraceExpiringEmitted);
|
||||
var e = Assert.Single(events);
|
||||
Assert.Equal("TenantRetentionGraceExpiring", e.EventType);
|
||||
Assert.Equal(OpsEventSeverity.Warning, e.Severity);
|
||||
Assert.Contains("2 day", e.Message);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task GraceEndedWithinLookBack_EmitsGraceEnded_OlderStaysSilent()
|
||||
{
|
||||
var (sut, events) = Build(
|
||||
// Grace ended 12h ago → emit once.
|
||||
Tenant("just-ended", null, planTier: "community",
|
||||
retentionDays: 365, proDowngradedUtc: Now.AddDays(-30).AddHours(-12)),
|
||||
// Grace ended 3 days ago → reported by a previous run, silent.
|
||||
Tenant("long-ended", null, planTier: "community",
|
||||
retentionDays: 365, proDowngradedUtc: Now.AddDays(-33)));
|
||||
|
||||
var result = await sut.RunCoreAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(1, result.GraceEndedEmitted);
|
||||
var e = Assert.Single(events);
|
||||
Assert.Equal("TenantRetentionGraceEnded", e.EventType);
|
||||
Assert.Equal("just-ended", e.TenantId);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Grace_NoDataAtRisk_StaysSilent()
|
||||
{
|
||||
var (sut, events) = Build(
|
||||
// Stored retention at the Community cap — nothing to lose.
|
||||
Tenant("at-cap", null, planTier: "community",
|
||||
retentionDays: 90, proDowngradedUtc: Now.AddDays(-29)),
|
||||
// 0 = infinite escape hatch — the fanout never deletes, no warning needed.
|
||||
Tenant("infinite", null, planTier: "community",
|
||||
retentionDays: 0, proDowngradedUtc: Now.AddDays(-29)));
|
||||
|
||||
var result = await sut.RunCoreAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(0, result.GraceExpiringEmitted);
|
||||
Assert.Equal(0, result.GraceEndedEmitted);
|
||||
Assert.Empty(events);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Grace_ReUpgradedTenant_StaysSilent()
|
||||
{
|
||||
// Back on Pro: GetRetentionGraceEndUtc returns null even with a stale anchor.
|
||||
var (sut, events) = Build(Tenant("t1", null, planTier: "pro",
|
||||
retentionDays: 365, proDowngradedUtc: Now.AddDays(-29)));
|
||||
|
||||
var result = await sut.RunCoreAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(0, result.GraceExpiringEmitted);
|
||||
Assert.Empty(events);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ExpiredTrial_WithDataAtRisk_GetsBothTrialExpiredAndLaterGraceEvents()
|
||||
{
|
||||
// The trial-expiry anchor drives the grace window too: expired 2h ago fires
|
||||
// TenantTrialExpired now; 28 days later the same tenant enters the grace heads-up.
|
||||
var (sut, events) = Build(Tenant("t1", Now.AddHours(-2), retentionDays: 365));
|
||||
|
||||
var result = await sut.RunCoreAsync(CancellationToken.None);
|
||||
|
||||
Assert.Equal(1, result.ExpiredEmitted);
|
||||
Assert.Equal(0, result.GraceExpiringEmitted); // grace end is ~30d out — beyond heads-up
|
||||
Assert.Single(events);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task ConfigLoadFailure_ReturnsEmptyResult_NeverThrows()
|
||||
{
|
||||
|
||||
@@ -0,0 +1,206 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using AutopilotMonitor.Functions.DataAccess.TableStorage;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Verdict-calibration drift radar (docs/backend/verdict-calibration.md): the three finding
|
||||
/// kinds, the small-n / lift / Wilson gates, re-arm, and the tracker round-trip.
|
||||
/// </summary>
|
||||
public class VerdictCalibrationRadarTests
|
||||
{
|
||||
private static readonly DateTime Target = new(2026, 8, 22, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
/// <summary>35 daily rows ending on Target; per-day buckets chosen by window/baseline membership.</summary>
|
||||
private static List<VerdictCalibrationDailyAggregate> Horizon(
|
||||
Func<bool, (string Path, string Status, int Count)[]> bucketsFor, int sessionsPerDay = 20)
|
||||
{
|
||||
var rows = new List<VerdictCalibrationDailyAggregate>();
|
||||
for (var i = 34; i >= 0; i--)
|
||||
{
|
||||
var inWindow = i < 7;
|
||||
var date = Target.AddDays(-i);
|
||||
rows.Add(new VerdictCalibrationDailyAggregate
|
||||
{
|
||||
TenantId = "t", Date = date.ToString("yyyy-MM-dd"), SessionCount = sessionsPerDay, TerminalSessionCount = sessionsPerDay,
|
||||
Buckets = bucketsFor(inWindow).Select(b => new VerdictCalibrationBucket { VerdictPath = b.Path, Status = b.Status, Count = b.Count }).ToList(),
|
||||
});
|
||||
}
|
||||
return rows;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Share_regression_fires_on_a_doubled_separated_share()
|
||||
{
|
||||
// baseline: 1/20 per day (5%), window: 4/20 per day (20%) → lift 4, clearly separated
|
||||
var rows = Horizon(w => new[] { ("sweep:r5_incomplete", "Incomplete", w ? 4 : 1), ("agent:complete", "Succeeded", w ? 16 : 19) });
|
||||
var findings = VerdictCalibrationRadar.Evaluate(rows, Target);
|
||||
|
||||
var f = Assert.Single(findings, x => x.Kind == VerdictCalibrationAlertKinds.ShareRegression);
|
||||
Assert.Equal("sweep:r5_incomplete", f.VerdictPath);
|
||||
Assert.Equal("Incomplete", f.Status);
|
||||
Assert.Equal(28, f.WindowHitCount);
|
||||
Assert.Equal(140, f.WindowSessionCount);
|
||||
Assert.Equal(28, f.BaselineHitCount);
|
||||
Assert.Equal(560, f.BaselineSessionCount);
|
||||
Assert.Equal(20.0, f.WindowRatePct);
|
||||
Assert.Equal(5.0, f.BaselineRatePct);
|
||||
Assert.Equal(4.0, f.Lift);
|
||||
Assert.Equal("2026-08-16", f.WindowStartDate);
|
||||
Assert.Equal("2026-08-22", f.WindowEndDate);
|
||||
// sweep:* is also the silence group → silence share doubled as well
|
||||
Assert.Contains(findings, x => x.Kind == VerdictCalibrationAlertKinds.SilenceShareRegression && x.VerdictPath == VerdictCalibrationRadar.SilenceGroupPath);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Share_regression_is_suppressed_below_lift_two_or_without_separation()
|
||||
{
|
||||
// 5% → 8%: lift 1.6 — under the gate
|
||||
var mild = Horizon(w => new[] { ("agent:failed", "Failed", w ? 2 : 1), ("agent:complete", "Succeeded", w ? 18 : 19) }, sessionsPerDay: 20);
|
||||
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(mild, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
|
||||
|
||||
// tiny n: 1/2 per day in window vs 0/2 baseline — lift infinite but Wilson intervals overlap / hits < 5
|
||||
var tiny = Horizon(w => new[] { ("agent:failed", "Failed", w ? 1 : 0), ("agent:complete", "Succeeded", w ? 1 : 2) }, sessionsPerDay: 2);
|
||||
Assert.Empty(VerdictCalibrationRadar.Evaluate(tiny, Target));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Agent_and_registration_paths_never_share_alert_but_still_feed_the_silence_group()
|
||||
{
|
||||
// Workflow mix, not classifier drift: a WhiteGlove rollout week doubles the pending share.
|
||||
var wg = Horizon(w => new[] { ("agent:whiteglove_pending", "Pending", w ? 6 : 1), ("agent:complete", "Succeeded", w ? 14 : 19) });
|
||||
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(wg, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
|
||||
var reg = Horizon(w => new[] { ("register:superseded", "Incomplete", w ? 6 : 1), ("agent:complete", "Succeeded", w ? 14 : 19) });
|
||||
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(reg, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
|
||||
// Backend-decided paths stay eligible.
|
||||
Assert.True(VerdictCalibrationRadar.IsShareRegressionEligible("sweep:r6"));
|
||||
Assert.True(VerdictCalibrationRadar.IsShareRegressionEligible("rule:ANALYZE-ESP-001"));
|
||||
Assert.True(VerdictCalibrationRadar.IsShareRegressionEligible("manual:failed"));
|
||||
Assert.False(VerdictCalibrationRadar.IsShareRegressionEligible("agent:complete_soft"));
|
||||
|
||||
// An episode opened by the earlier, broader gate re-arms on the next pass regardless of numbers.
|
||||
var stale = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "agent:whiteglove_pending", Status = "Pending" };
|
||||
Assert.True(VerdictCalibrationRadar.ShouldReArm(stale, wg, Target));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Derived_legacy_paths_never_alert()
|
||||
{
|
||||
var rows = Horizon(w => new[] { ("legacy:r6", "Incomplete", w ? 10 : 1), ("agent:complete", "Succeeded", w ? 10 : 19) });
|
||||
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(rows, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Evidence_gap_fires_on_r6_share_and_ignores_non_rule_sweep_paths()
|
||||
{
|
||||
// classifier verdicts in window: r6 5/day + r5 15/day = 140; r6 share 25% → fires. sweep:stalled is not a rule verdict.
|
||||
var rows = Horizon(w => new[]
|
||||
{
|
||||
("sweep:r6", "Incomplete", 5), ("sweep:r5_incomplete", "Incomplete", 15), ("sweep:stalled", "Stalled", 30),
|
||||
}, sessionsPerDay: 50);
|
||||
var f = Assert.Single(VerdictCalibrationRadar.Evaluate(rows, Target), x => x.Kind == VerdictCalibrationAlertKinds.EvidenceGap);
|
||||
Assert.Equal(VerdictCalibrationRadar.EvidenceGapPath, f.VerdictPath);
|
||||
Assert.Equal(35, f.WindowHitCount);
|
||||
Assert.Equal(140, f.WindowSessionCount);
|
||||
Assert.Equal(25.0, f.WindowRatePct);
|
||||
Assert.Null(f.Lift);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Evidence_gap_needs_twenty_classifier_verdicts()
|
||||
{
|
||||
var rows = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 2 : 0), ("agent:complete", "Succeeded", 18) });
|
||||
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(rows, Target), f => f.Kind == VerdictCalibrationAlertKinds.EvidenceGap);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReArm_when_share_falls_under_one_point_five_or_path_stops()
|
||||
{
|
||||
var alert = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "sweep:r6", Status = "Incomplete" };
|
||||
|
||||
var stillHigh = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 4 : 1), ("agent:complete", "Succeeded", w ? 16 : 19) });
|
||||
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, stillHigh, Target));
|
||||
|
||||
var elevatedButUnderGate = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 2 : 1), ("agent:complete", "Succeeded", w ? 18 : 19) }, sessionsPerDay: 20);
|
||||
// 10% vs 5% = 2.0× — still at/above 1.5× → keep
|
||||
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, elevatedButUnderGate, Target));
|
||||
|
||||
var back = Horizon(w => new[] { ("sweep:r6", "Incomplete", 1), ("agent:complete", "Succeeded", 19) });
|
||||
Assert.True(VerdictCalibrationRadar.ShouldReArm(alert, back, Target));
|
||||
|
||||
var gone = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 0 : 1), ("agent:complete", "Succeeded", w ? 20 : 19) });
|
||||
Assert.True(VerdictCalibrationRadar.ShouldReArm(alert, gone, Target));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Evidence_gap_rearms_under_fifteen_percent()
|
||||
{
|
||||
var alert = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.EvidenceGap, VerdictPath = "r6", Status = "*" };
|
||||
var still = Horizon(w => new[] { ("sweep:r6", "Incomplete", 4), ("sweep:r5_incomplete", "Incomplete", 16) });
|
||||
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, still, Target)); // 20%
|
||||
var between = Horizon(w => new[] { ("sweep:r6", "Incomplete", 17), ("sweep:r5_incomplete", "Incomplete", 83) }, sessionsPerDay: 100);
|
||||
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, between, Target)); // 17%
|
||||
var low = Horizon(w => new[] { ("sweep:r6", "Incomplete", 1), ("sweep:r5_incomplete", "Incomplete", 19) });
|
||||
Assert.True(VerdictCalibrationRadar.ShouldReArm(alert, low, Target)); // 5%
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void CurrentSums_follow_the_alert_kind()
|
||||
{
|
||||
var rows = Horizon(w => new[] { ("sweep:r6", "Incomplete", 2), ("maxlife:r3", "Incomplete", 1), ("agent:complete", "Succeeded", 17) });
|
||||
var share = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "sweep:r6", Status = "Incomplete" }, rows, Target);
|
||||
Assert.Equal((14, 140, 56, 560), share);
|
||||
var silence = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.SilenceShareRegression }, rows, Target);
|
||||
Assert.Equal((21, 140, 84, 560), silence);
|
||||
var gap = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.EvidenceGap }, rows, Target);
|
||||
Assert.Equal((14, 21, 56, 84), gap);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Rows_outside_the_horizon_are_ignored()
|
||||
{
|
||||
var rows = Horizon(w => new[] { ("agent:complete", "Succeeded", 20) });
|
||||
rows.Add(new VerdictCalibrationDailyAggregate { TenantId = "t", Date = "2026-08-23", SessionCount = 1000, Buckets = { new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = 1000 } } });
|
||||
rows.Add(new VerdictCalibrationDailyAggregate { TenantId = "t", Date = "2026-07-01", SessionCount = 1000, Buckets = { new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = 1000 } } });
|
||||
Assert.Empty(VerdictCalibrationRadar.Evaluate(rows, Target));
|
||||
}
|
||||
|
||||
// ---- tracker round-trip ----
|
||||
|
||||
[Fact]
|
||||
public void Alert_round_trips_through_the_tracker_entity()
|
||||
{
|
||||
var alert = new VerdictCalibrationAlert
|
||||
{
|
||||
TenantId = "11111111-1111-1111-1111-111111111111", Kind = VerdictCalibrationAlertKinds.ShareRegression,
|
||||
VerdictPath = "rule:ANALYZE-ESP-001", Status = "Failed",
|
||||
WindowHitCount = 7, WindowSessionCount = 40, BaselineHitCount = 3, BaselineSessionCount = 160,
|
||||
WindowRatePct = 17.5, BaselineRatePct = 1.9, Lift = 9.3, WindowStartDate = "2026-08-16", WindowEndDate = "2026-08-22",
|
||||
Dimension = new RuleRegressionDimension { Dimension = "osBuild", Value = "26200.1234", HitCount = 6, HitSharePct = 85.7, AllSharePct = 20.0, Lift = 4.3 },
|
||||
FirstNotifiedAt = new DateTime(2026, 8, 20, 6, 0, 0, DateTimeKind.Utc), LastEvaluatedAt = new DateTime(2026, 8, 23, 6, 0, 0, DateTimeKind.Utc),
|
||||
};
|
||||
var entity = TableHardwareRejectionNotificationTracker.BuildVerdictCalibrationEntity(alert.TenantId, alert);
|
||||
Assert.Equal("verdictcalibration|share_regression|rule:analyze-esp-001|failed", entity.RowKey);
|
||||
Assert.Equal(alert.TenantId, entity.PartitionKey);
|
||||
|
||||
var back = TableHardwareRejectionNotificationTracker.MapToVerdictCalibrationAlert(entity);
|
||||
Assert.Equal(JsonSerializer.Serialize(alert), JsonSerializer.Serialize(back));
|
||||
|
||||
// Tri-states: no lift / no dimension → absent columns → null on read, never 0 / a guessed dimension.
|
||||
var bare = new VerdictCalibrationAlert
|
||||
{
|
||||
TenantId = alert.TenantId, Kind = VerdictCalibrationAlertKinds.EvidenceGap, VerdictPath = "r6", Status = "*",
|
||||
FirstNotifiedAt = alert.FirstNotifiedAt, LastEvaluatedAt = alert.LastEvaluatedAt,
|
||||
};
|
||||
var bareBack = TableHardwareRejectionNotificationTracker.MapToVerdictCalibrationAlert(
|
||||
TableHardwareRejectionNotificationTracker.BuildVerdictCalibrationEntity(bare.TenantId, bare));
|
||||
Assert.Null(bareBack.Lift);
|
||||
Assert.Null(bareBack.Dimension);
|
||||
Assert.Equal("verdictcalibration|evidence_gap|r6|*", TableHardwareRejectionNotificationTracker.BuildVerdictCalibrationRowKey(bare.Kind, bare.VerdictPath, bare.Status));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,301 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text.Json;
|
||||
using AutopilotMonitor.Functions.Functions.Metrics;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Verdict calibration aggregate (docs/backend/verdict-calibration.md): the pure bucketing core,
|
||||
/// the persistence round-trip and the matrix arithmetic of the read endpoint.
|
||||
/// </summary>
|
||||
public class VerdictCalibrationTests
|
||||
{
|
||||
private const string TenantA = "11111111-1111-1111-1111-111111111111";
|
||||
private const string TenantB = "22222222-2222-2222-2222-222222222222";
|
||||
private static readonly DateTime Now = new(2026, 8, 23, 12, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private static SessionSummary Session(string tenant, string id, SessionStatus status, DateTime started,
|
||||
string? path = null, string? priorPath = null, string? priorStatus = null, string? adminMarked = null,
|
||||
DateTime? completed = null, string serial = "SN-1", string? failureReason = null, string? reconcileReason = null) => new()
|
||||
{
|
||||
TenantId = tenant, SessionId = id, SerialNumber = serial, Status = status, StartedAt = started,
|
||||
CompletedAt = completed, VerdictPath = path, PriorVerdictPath = priorPath, PriorStatus = priorStatus,
|
||||
AdminMarkedAction = adminMarked, FailureReason = failureReason ?? string.Empty,
|
||||
ReconcileReason = reconcileReason ?? string.Empty, FailureSource = string.Empty,
|
||||
};
|
||||
|
||||
private static DeviceHistory History(string tenant, string serial, params (string Id, DateTime Started, DateTime? Completed, string Status)[] refs) => new()
|
||||
{
|
||||
TenantId = tenant, SerialKey = serial.ToLowerInvariant(), SerialNumber = serial,
|
||||
Chain = refs.Select(r => new DeviceSessionRef { SessionId = r.Id, StartedAt = r.Started, CompletedAt = r.Completed, Status = r.Status }).ToList(),
|
||||
};
|
||||
|
||||
private static VerdictCalibrationBucket Bucket(VerdictCalibrationDailyAggregate row, string path, string status)
|
||||
=> Assert.Single(row.Buckets, b => b.VerdictPath == path && b.Status == status);
|
||||
|
||||
// ---- BuildVerdictCalibrationAggregates ----
|
||||
|
||||
[Fact]
|
||||
public void Buckets_by_tenant_date_path_and_status_with_global_mirror()
|
||||
{
|
||||
var d1 = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
|
||||
var sessions = new List<SessionSummary>
|
||||
{
|
||||
Session(TenantA, "a1", SessionStatus.Succeeded, d1, path: "agent:complete", completed: d1.AddHours(1)),
|
||||
Session(TenantA, "a2", SessionStatus.Succeeded, d1.AddHours(2), path: "agent:complete", completed: d1.AddHours(3)),
|
||||
Session(TenantA, "a3", SessionStatus.Incomplete, d1, path: "sweep:r6", completed: d1.AddHours(6)),
|
||||
Session(TenantA, "a4", SessionStatus.Stalled, d1.AddDays(1), path: "sweep:stalled"),
|
||||
Session(TenantB, "b1", SessionStatus.Failed, d1, path: "agent:failed", completed: d1.AddHours(1)),
|
||||
};
|
||||
var rows = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, new Dictionary<string, List<DeviceHistory>>(), Now);
|
||||
|
||||
var a10 = Assert.Single(rows, r => r.TenantId == TenantA && r.Date == "2026-08-10");
|
||||
Assert.Equal(3, a10.SessionCount);
|
||||
Assert.Equal(3, a10.TerminalSessionCount);
|
||||
Assert.Equal(2, Bucket(a10, "agent:complete", "Succeeded").Count);
|
||||
Assert.Equal(1, Bucket(a10, "sweep:r6", "Incomplete").Count);
|
||||
|
||||
var a11 = Assert.Single(rows, r => r.TenantId == TenantA && r.Date == "2026-08-11");
|
||||
Assert.Equal(1, a11.SessionCount);
|
||||
Assert.Equal(0, a11.TerminalSessionCount); // Stalled is not terminal
|
||||
Assert.Equal(1, Bucket(a11, "sweep:stalled", "Stalled").Count);
|
||||
|
||||
var g10 = Assert.Single(rows, r => r.TenantId == "global" && r.Date == "2026-08-10");
|
||||
Assert.Equal(4, g10.SessionCount);
|
||||
Assert.Equal(1, Bucket(g10, "agent:failed", "Failed").Count);
|
||||
Assert.Equal(VerdictCalibrationDailyAggregate.CurrentVersion, g10.Version);
|
||||
Assert.Equal(Now, g10.ComputedAt);
|
||||
|
||||
// Deterministic ordering: rows by tenant then date, buckets by path then status.
|
||||
Assert.Equal(rows.Select(r => (r.TenantId, r.Date)).OrderBy(k => k.TenantId, StringComparer.Ordinal).ThenBy(k => k.Date, StringComparer.Ordinal), rows.Select(r => (r.TenantId, r.Date)));
|
||||
Assert.Equal(a10.Buckets.Select(b => b.VerdictPath).OrderBy(p => p, StringComparer.Ordinal), a10.Buckets.Select(b => b.VerdictPath));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Unstamped_rows_are_derived_and_counted_as_derived()
|
||||
{
|
||||
var d = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
|
||||
var sessions = new List<SessionSummary>
|
||||
{
|
||||
Session(TenantA, "a1", SessionStatus.Succeeded, d, path: null, completed: d.AddHours(1)), // legacy agent completion
|
||||
Session(TenantA, "a2", SessionStatus.Incomplete, d, path: null, completed: d.AddHours(5),
|
||||
failureReason: "No Device Setup completion or explicit failure signal observed before timeout"),
|
||||
Session(TenantA, "a3", SessionStatus.Succeeded, d, path: "agent:complete", completed: d.AddHours(1)),
|
||||
};
|
||||
var row = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, new Dictionary<string, List<DeviceHistory>>(), Now)
|
||||
.Single(r => r.TenantId == TenantA);
|
||||
|
||||
var complete = Bucket(row, "agent:complete", "Succeeded");
|
||||
Assert.Equal(2, complete.Count);
|
||||
Assert.Equal(1, complete.DerivedCount);
|
||||
var r6 = Bucket(row, "legacy:r6", "Incomplete");
|
||||
Assert.Equal(1, r6.Count);
|
||||
Assert.Equal(1, r6.DerivedCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Overrides_are_attributed_to_the_prior_path()
|
||||
{
|
||||
var d = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
|
||||
var sessions = new List<SessionSummary>
|
||||
{
|
||||
// sweep said Incomplete, agent later completed → late completion against sweep:r5_incomplete
|
||||
Session(TenantA, "a1", SessionStatus.Succeeded, d, path: "agent:complete", priorPath: "sweep:r5_incomplete", priorStatus: "Incomplete", completed: d.AddHours(1)),
|
||||
// admin flipped an agent success to Failed
|
||||
Session(TenantA, "a2", SessionStatus.Failed, d, path: "manual:failed", priorPath: "agent:complete", priorStatus: "Succeeded", adminMarked: "Failed", completed: d.AddHours(1)),
|
||||
// retro reclassification Failed → Incomplete
|
||||
Session(TenantA, "a3", SessionStatus.Incomplete, d, path: "retro:r6", priorPath: "agent:failed", priorStatus: "Failed", completed: d.AddHours(1)),
|
||||
// admin mark to Succeeded counts as admin, not late completion
|
||||
Session(TenantA, "a4", SessionStatus.Succeeded, d, path: "manual:succeeded", priorPath: "sweep:r6", priorStatus: "Incomplete", adminMarked: "Succeeded", completed: d.AddHours(1)),
|
||||
};
|
||||
var row = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, new Dictionary<string, List<DeviceHistory>>(), Now)
|
||||
.Single(r => r.TenantId == TenantA);
|
||||
|
||||
var priorSweep = Bucket(row, "sweep:r5_incomplete", "Incomplete");
|
||||
Assert.Equal(0, priorSweep.Count); // nobody is on it any more
|
||||
Assert.Equal(1, priorSweep.OverriddenByLateCompletion);
|
||||
|
||||
var priorAgent = Bucket(row, "agent:complete", "Succeeded");
|
||||
Assert.Equal(1, priorAgent.Count); // a1 is on it now
|
||||
Assert.Equal(1, priorAgent.OverriddenByAdmin); // a2 used to be
|
||||
|
||||
Assert.Equal(1, Bucket(row, "agent:failed", "Failed").OverriddenOther);
|
||||
Assert.Equal(1, Bucket(row, "sweep:r6", "Incomplete").OverriddenByAdmin);
|
||||
Assert.Equal(0, Bucket(row, "manual:succeeded", "Succeeded").OverriddenByAdmin);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReEnrollment_proxy_counts_only_eligible_terminal_sessions()
|
||||
{
|
||||
var old = Now.AddDays(-20);
|
||||
var young = Now.AddDays(-2);
|
||||
var sessions = new List<SessionSummary>
|
||||
{
|
||||
// old, re-enrolled 3 days later → eligible + re-enrolled
|
||||
Session(TenantA, "a1", SessionStatus.Incomplete, old, path: "sweep:r6", completed: old.AddHours(5), serial: "SN-A"),
|
||||
// old, next attempt 10 days later → eligible, not re-enrolled
|
||||
Session(TenantA, "a2", SessionStatus.Failed, old, path: "agent:failed", completed: old.AddHours(1), serial: "SN-B"),
|
||||
// old, no next attempt → eligible, not re-enrolled
|
||||
Session(TenantA, "a3", SessionStatus.Succeeded, old, path: "agent:complete", completed: old.AddHours(1), serial: "SN-C"),
|
||||
// too young to judge → not eligible even though the chain shows a follow-up
|
||||
Session(TenantA, "a4", SessionStatus.Failed, young, path: "agent:failed", completed: young.AddHours(1), serial: "SN-D"),
|
||||
// non-terminal → never eligible
|
||||
Session(TenantA, "a5", SessionStatus.AwaitingUser, old, path: "sweep:r5_awaiting", serial: "SN-E"),
|
||||
};
|
||||
var histories = new Dictionary<string, List<DeviceHistory>>
|
||||
{
|
||||
[TenantA] = new()
|
||||
{
|
||||
History(TenantA, "SN-A", ("a1", old, old.AddHours(5), "Incomplete"), ("x1", old.AddDays(3), old.AddDays(3).AddHours(1), "Succeeded")),
|
||||
History(TenantA, "SN-B", ("a2", old, old.AddHours(1), "Failed"), ("x2", old.AddDays(10), null, "Succeeded")),
|
||||
History(TenantA, "SN-C", ("a3", old, old.AddHours(1), "Succeeded")),
|
||||
History(TenantA, "SN-D", ("a4", young, young.AddHours(1), "Failed"), ("x4", young.AddHours(2), null, "Succeeded")),
|
||||
},
|
||||
};
|
||||
var rows = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, histories, Now);
|
||||
|
||||
var oldRow = rows.Single(r => r.TenantId == TenantA && r.Date == old.ToString("yyyy-MM-dd"));
|
||||
var r6 = Bucket(oldRow, "sweep:r6", "Incomplete");
|
||||
Assert.Equal((1, 1), (r6.Eligible7d, r6.ReEnrolled7d));
|
||||
var failed = Bucket(oldRow, "agent:failed", "Failed");
|
||||
Assert.Equal((1, 0), (failed.Eligible7d, failed.ReEnrolled7d));
|
||||
var complete = Bucket(oldRow, "agent:complete", "Succeeded");
|
||||
Assert.Equal((1, 0), (complete.Eligible7d, complete.ReEnrolled7d));
|
||||
var awaiting = Bucket(oldRow, "sweep:r5_awaiting", "AwaitingUser");
|
||||
Assert.Equal((0, 0), (awaiting.Eligible7d, awaiting.ReEnrolled7d));
|
||||
|
||||
var youngRow = rows.Single(r => r.TenantId == TenantA && r.Date == young.ToString("yyyy-MM-dd"));
|
||||
var youngFailed = Bucket(youngRow, "agent:failed", "Failed");
|
||||
Assert.Equal((0, 0), (youngFailed.Eligible7d, youngFailed.ReEnrolled7d));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Sessions_inside_a_deletion_cascade_are_skipped()
|
||||
{
|
||||
var d = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
|
||||
var deleting = Session(TenantA, "a1", SessionStatus.Succeeded, d, path: "agent:complete");
|
||||
deleting.DeletionState = "Pending";
|
||||
var rows = MaintenanceService.BuildVerdictCalibrationAggregates(new[] { deleting }, new Dictionary<string, List<DeviceHistory>>(), Now);
|
||||
Assert.Empty(rows);
|
||||
}
|
||||
|
||||
// ---- Persistence round-trip ----
|
||||
|
||||
[Fact]
|
||||
public void Aggregate_round_trips_through_the_table_entity()
|
||||
{
|
||||
var aggregate = new VerdictCalibrationDailyAggregate
|
||||
{
|
||||
TenantId = TenantA, Date = "2026-08-10", Version = 1, SessionCount = 5, TerminalSessionCount = 4, ComputedAt = Now,
|
||||
Buckets =
|
||||
{
|
||||
new VerdictCalibrationBucket { VerdictPath = "agent:complete", Status = "Succeeded", Count = 3, DerivedCount = 1, Eligible7d = 3, ReEnrolled7d = 1, OverriddenByAdmin = 1, OverriddenByLateCompletion = 0, OverriddenOther = 2 },
|
||||
new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = 1 },
|
||||
},
|
||||
};
|
||||
var entity = TableStorageService.BuildVerdictCalibrationAggregateEntity(aggregate);
|
||||
Assert.Equal(TenantA, entity.PartitionKey);
|
||||
Assert.Equal("2026-08-10", entity.RowKey);
|
||||
|
||||
var back = TableStorageService.MapToVerdictCalibrationAggregate(entity);
|
||||
Assert.Equal(JsonSerializer.Serialize(aggregate), JsonSerializer.Serialize(back));
|
||||
}
|
||||
|
||||
// ---- Response builder ----
|
||||
|
||||
private static VerdictCalibrationDailyAggregate Day(string tenant, DateTime date, int sessions, params (string Path, string Status, int Count)[] buckets) => new()
|
||||
{
|
||||
TenantId = tenant, Date = date.ToString("yyyy-MM-dd"), SessionCount = sessions, TerminalSessionCount = sessions, Version = 1, ComputedAt = Now,
|
||||
Buckets = buckets.Select(b => new VerdictCalibrationBucket { VerdictPath = b.Path, Status = b.Status, Count = b.Count, Eligible7d = b.Count, ReEnrolled7d = b.Count / 5 }).ToList(),
|
||||
};
|
||||
|
||||
// Same naming policy as the worker host (Program.cs: camelCase) so the assertions pin the wire shape.
|
||||
private static readonly JsonSerializerOptions Wire = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase };
|
||||
|
||||
private static JsonElement BuildJson(IReadOnlyList<VerdictCalibrationDailyAggregate> daily, DateTime today, int days)
|
||||
=> JsonSerializer.SerializeToElement(VerdictCalibrationResponse.Build(daily, "global", today, days), Wire);
|
||||
|
||||
[Fact]
|
||||
public void Response_sums_the_window_and_computes_share_rate_and_trend()
|
||||
{
|
||||
var today = new DateTime(2026, 8, 23, 0, 0, 0, DateTimeKind.Utc);
|
||||
var daily = new List<VerdictCalibrationDailyAggregate>();
|
||||
// 35 days: baseline days carry 10 sessions (1 sweep:r6), the last 7 days carry 10 sessions (3 sweep:r6) → share 10% → 30%, lift 3.0
|
||||
for (var i = 34; i >= 0; i--)
|
||||
{
|
||||
var date = today.AddDays(-i);
|
||||
var r6 = i < 7 ? 3 : 1;
|
||||
daily.Add(Day("global", date, 10, ("agent:complete", "Succeeded", 10 - r6), ("sweep:r6", "Incomplete", r6)));
|
||||
}
|
||||
var json = BuildJson(daily, today, days: 30);
|
||||
|
||||
Assert.True(json.GetProperty("success").GetBoolean());
|
||||
Assert.Equal(30, json.GetProperty("windowDays").GetInt32());
|
||||
Assert.Equal(300, json.GetProperty("totals").GetProperty("sessions").GetInt32());
|
||||
Assert.Equal(30, json.GetProperty("totals").GetProperty("days").GetInt32());
|
||||
|
||||
var paths = json.GetProperty("paths").EnumerateArray().ToList();
|
||||
var r6Row = paths.Single(p => p.GetProperty("verdictPath").GetString() == "sweep:r6");
|
||||
Assert.Equal(7 * 3 + 23 * 1, r6Row.GetProperty("count").GetInt32());
|
||||
Assert.Equal(Math.Round(100.0 * 44 / 300, 1), r6Row.GetProperty("sharePct").GetDouble());
|
||||
Assert.Equal(30.0, r6Row.GetProperty("window7").GetProperty("sharePct").GetDouble());
|
||||
Assert.Equal(10.0, r6Row.GetProperty("baseline28").GetProperty("sharePct").GetDouble());
|
||||
Assert.Equal(3.0, r6Row.GetProperty("lift").GetDouble());
|
||||
// Eligible 44 ≥ 20 → rate stated; ReEnrolled7d = count/5 per day → 7*0 + 23*0 = 0 → 0%
|
||||
Assert.Equal(0.0, r6Row.GetProperty("reEnrollRatePct").GetDouble());
|
||||
|
||||
// Sorted by count desc → agent:complete first.
|
||||
Assert.Equal("agent:complete", paths[0].GetProperty("verdictPath").GetString());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Response_withholds_rate_below_twenty_eligible_and_lift_without_baseline()
|
||||
{
|
||||
var today = new DateTime(2026, 8, 23, 0, 0, 0, DateTimeKind.Utc);
|
||||
var daily = new List<VerdictCalibrationDailyAggregate>
|
||||
{
|
||||
Day("global", today, 5, ("maxlife:r3", "Incomplete", 5)),
|
||||
};
|
||||
var json = BuildJson(daily, today, days: 7);
|
||||
var row = json.GetProperty("paths").EnumerateArray().Single();
|
||||
Assert.Equal(JsonValueKind.Null, row.GetProperty("reEnrollRatePct").ValueKind);
|
||||
Assert.Equal(JsonValueKind.Null, row.GetProperty("lift").ValueKind); // no baseline rows → never invented
|
||||
Assert.Equal(100.0, row.GetProperty("sharePct").GetDouble());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Response_keeps_a_zero_count_bucket_that_carries_overrides()
|
||||
{
|
||||
var today = new DateTime(2026, 8, 23, 0, 0, 0, DateTimeKind.Utc);
|
||||
var day = Day("global", today, 1, ("agent:complete", "Succeeded", 1));
|
||||
day.Buckets.Add(new VerdictCalibrationBucket { VerdictPath = "sweep:r5_incomplete", Status = "Incomplete", Count = 0, OverriddenByLateCompletion = 1 });
|
||||
var json = BuildJson(new[] { day }, today, days: 7);
|
||||
var rows = json.GetProperty("paths").EnumerateArray().ToList();
|
||||
var prior = rows.Single(p => p.GetProperty("verdictPath").GetString() == "sweep:r5_incomplete");
|
||||
Assert.Equal(0, prior.GetProperty("count").GetInt32());
|
||||
Assert.Equal(1, prior.GetProperty("overriddenByLateCompletion").GetInt32());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Response_is_empty_but_well_formed_before_the_first_sweep()
|
||||
{
|
||||
var json = BuildJson(Array.Empty<VerdictCalibrationDailyAggregate>(), new DateTime(2026, 8, 23), 30);
|
||||
Assert.True(json.GetProperty("success").GetBoolean());
|
||||
Assert.Empty(json.GetProperty("paths").EnumerateArray());
|
||||
Assert.Equal(0, json.GetProperty("totals").GetProperty("sessions").GetInt32());
|
||||
Assert.Equal(JsonValueKind.Null, json.GetProperty("computedAt").ValueKind);
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData(null, 30)]
|
||||
[InlineData("abc", 30)]
|
||||
[InlineData("0", 1)]
|
||||
[InlineData("500", 180)]
|
||||
[InlineData("14", 14)]
|
||||
public void ClampDays_bounds_the_window(string? raw, int expected)
|
||||
=> Assert.Equal(expected, VerdictCalibrationResponse.ClampDays(raw));
|
||||
}
|
||||
@@ -0,0 +1,201 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Verdict calibration instrumentation (docs/backend/verdict-calibration.md): every status write
|
||||
/// declares its origin, overriding a prior verdict preserves it, the classifier names the rule
|
||||
/// behind each verdict, and pre-instrumentation rows derive an honest path from their literals.
|
||||
/// </summary>
|
||||
public class VerdictPathTests
|
||||
{
|
||||
private static readonly DateTime Start = new(2026, 8, 20, 8, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
// ---- ComputePriorVerdict: the correction stream ----
|
||||
|
||||
[Theory]
|
||||
[InlineData("Succeeded", "agent:complete", "Failed", true)] // admin mark over an agent success
|
||||
[InlineData("Incomplete", "sweep:r5_incomplete", "Succeeded", true)] // late agent completion upgrades a sweep verdict
|
||||
[InlineData("AwaitingUser", "sweep:r5_awaiting", "Incomplete", true)] // grace expired
|
||||
[InlineData("Failed", "agent:failed", "Incomplete", true)] // retro reclassification
|
||||
[InlineData("Stalled", "sweep:stalled", "InProgress", false)] // stall heal is not a correction
|
||||
[InlineData("Stalled", "sweep:stalled", "Succeeded", false)]
|
||||
[InlineData("InProgress", "register:new", "Succeeded", false)]
|
||||
[InlineData("Pending", "agent:whiteglove_pending", "InProgress", false)]
|
||||
[InlineData("AwaitingUser", "sweep:wg_awaiting", "AwaitingUser", false)] // same-status refresh keeps the original prior
|
||||
public void Prior_is_preserved_only_when_a_verdict_is_overridden(string existing, string existingPath, string incoming, bool expectPrior)
|
||||
{
|
||||
var prior = TableStorageService.ComputePriorVerdict(existing, existingPath, Enum.Parse<SessionStatus>(incoming));
|
||||
if (expectPrior)
|
||||
{
|
||||
Assert.NotNull(prior);
|
||||
Assert.Equal(existing, prior!.Value.PriorStatus);
|
||||
Assert.Equal(existingPath, prior.Value.PriorVerdictPath);
|
||||
}
|
||||
else
|
||||
{
|
||||
Assert.Null(prior);
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Prior_needs_a_stamped_path_on_the_existing_row()
|
||||
{
|
||||
// Pre-instrumentation rows carry no path — there is nothing trustworthy to preserve.
|
||||
Assert.Null(TableStorageService.ComputePriorVerdict("Incomplete", null, SessionStatus.Succeeded));
|
||||
Assert.Null(TableStorageService.ComputePriorVerdict("Incomplete", "", SessionStatus.Succeeded));
|
||||
Assert.Null(TableStorageService.ComputePriorVerdict(null, "sweep:r6", SessionStatus.Succeeded));
|
||||
}
|
||||
|
||||
// ---- VerdictPaths vocabulary ----
|
||||
|
||||
[Fact]
|
||||
public void Compose_and_Origin_round_trip()
|
||||
{
|
||||
var path = VerdictPaths.Compose(VerdictPaths.OriginSweep, ClassifierRules.R5DeviceSetupIncomplete);
|
||||
Assert.Equal("sweep:r5_incomplete", path);
|
||||
Assert.Equal("sweep", VerdictPaths.Origin(path));
|
||||
Assert.True(VerdictPaths.IsClassifierPath(path));
|
||||
Assert.True(VerdictPaths.IsClassifierPath("maxlife:r1"));
|
||||
Assert.True(VerdictPaths.IsClassifierPath("late:r4"));
|
||||
Assert.True(VerdictPaths.IsClassifierPath("retro:r6"));
|
||||
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.AgentComplete));
|
||||
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.SweepStalled));
|
||||
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.RetroSuperseded));
|
||||
Assert.False(VerdictPaths.IsClassifierPath("legacy:r6"));
|
||||
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.ManualFailed));
|
||||
Assert.Throws<ArgumentException>(() => VerdictPaths.Compose("", "r1"));
|
||||
Assert.Throws<ArgumentException>(() => VerdictPaths.Compose("sweep", " "));
|
||||
}
|
||||
|
||||
// ---- Classifier: every return names its rule ----
|
||||
|
||||
private static EnrollmentEvent Evt(string type, string? message = null) => new()
|
||||
{
|
||||
EventType = type, Timestamp = Start, Source = "test", Message = message ?? type,
|
||||
};
|
||||
|
||||
private static EnrollmentEvent Esp(string message) => Evt("esp_provisioning_status", message);
|
||||
private const string DeviceSetup44 = "ESP provisioning status: DeviceSetup — 4 of 4 subcategories completed";
|
||||
private const string AccountSetup55 = "ESP provisioning status: AccountSetup — 5 of 5 subcategories completed";
|
||||
private const string AccountSetup15 = "ESP provisioning status: AccountSetup — 1 of 5 subcategories completed";
|
||||
|
||||
private static string RuleOf(IReadOnlyList<EnrollmentEvent> events, double hours = 6, int grace = 72,
|
||||
bool preProvisioned = false, DateTime? resumedAt = null, bool selfDeploying = false)
|
||||
{
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
|
||||
var (_, _, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, Start, Start.AddHours(hours), grace, isPreProvisioned: preProvisioned,
|
||||
resumedAt: resumedAt, isSelfDeployingProfile: selfDeploying);
|
||||
return rule;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Classifier_returns_the_rule_id_behind_each_verdict()
|
||||
{
|
||||
Assert.Equal(ClassifierRules.R1ExplicitFailure, RuleOf(new[] { Evt("enrollment_failed") }));
|
||||
Assert.Equal(ClassifierRules.R1bWhiteGloveAwaiting, RuleOf(
|
||||
new[] { Esp(DeviceSetup44), Evt("whiteglove_complete") }, hours: 6, grace: 51, preProvisioned: true, resumedAt: Start.AddHours(1)));
|
||||
Assert.Equal(ClassifierRules.R1bWhiteGloveSucceeded, RuleOf(
|
||||
new[] { Esp(DeviceSetup44), Evt("whiteglove_complete") }, hours: 60, grace: 51, preProvisioned: true, resumedAt: Start.AddHours(1)));
|
||||
Assert.Equal(ClassifierRules.R1cSelfDeploying, RuleOf(new[] { Esp(DeviceSetup44) }, selfDeploying: true));
|
||||
Assert.Equal(ClassifierRules.R2AccountSetupComplete, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup55) }));
|
||||
Assert.Equal(ClassifierRules.R3EmergencyBreak, RuleOf(new[] { Esp(DeviceSetup44), Evt("agent_emergency_break") }));
|
||||
Assert.Equal(ClassifierRules.R4DesktopHello, RuleOf(new[] { Esp(DeviceSetup44), Evt("desktop_arrived"), Evt("hello_provisioning_completed") }));
|
||||
Assert.Equal(ClassifierRules.R5DeviceSetupAwaiting, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 6, grace: 72));
|
||||
Assert.Equal(ClassifierRules.R5DeviceSetupIncomplete, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 80, grace: 72));
|
||||
Assert.Equal(ClassifierRules.R6Fallthrough, RuleOf(Array.Empty<EnrollmentEvent>()));
|
||||
}
|
||||
|
||||
// ---- Derivation for pre-instrumentation rows ----
|
||||
|
||||
private static SessionSummary Row(SessionStatus status, string? failureReason = null, string? reconcileReason = null,
|
||||
string? failureSource = null, string? adminMarked = null, bool soft = false, string? verdictPath = null) => new()
|
||||
{
|
||||
SessionId = "s", TenantId = "t", Status = status,
|
||||
FailureReason = failureReason ?? string.Empty, ReconcileReason = reconcileReason ?? string.Empty,
|
||||
FailureSource = failureSource ?? string.Empty, AdminMarkedAction = adminMarked, EspSoftFailure = soft,
|
||||
VerdictPath = verdictPath,
|
||||
};
|
||||
|
||||
private static string Classified(IReadOnlyList<EnrollmentEvent> events, double hours = 80, int grace = 72, bool selfDeploying = false)
|
||||
{
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
|
||||
var (_, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, Start, Start.AddHours(hours), grace, isSelfDeployingProfile: selfDeploying);
|
||||
return reason;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Stamped_rows_are_returned_verbatim()
|
||||
{
|
||||
var (path, derived) = VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete, failureReason: "anything", verdictPath: "sweep:r6"));
|
||||
Assert.Equal("sweep:r6", path);
|
||||
Assert.False(derived);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Derivation_uses_the_real_classifier_literals()
|
||||
{
|
||||
// Reasons come straight from the classifier so a wording change there surfaces here.
|
||||
Assert.Equal(("legacy:r6", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
|
||||
failureReason: Classified(Array.Empty<EnrollmentEvent>()))));
|
||||
Assert.Equal(("legacy:r5_incomplete", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
|
||||
failureReason: Classified(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }))));
|
||||
Assert.Equal(("legacy:r5_awaiting", true), VerdictPathDerivation.Derive(Row(SessionStatus.AwaitingUser,
|
||||
failureReason: Classified(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 6))));
|
||||
Assert.Equal(("legacy:r3", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
|
||||
failureReason: Classified(new[] { Esp(DeviceSetup44), Evt("agent_emergency_break") }))));
|
||||
Assert.Equal(("legacy:r2", true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
|
||||
reconcileReason: Classified(new[] { Esp(DeviceSetup44), Esp(AccountSetup55) }))));
|
||||
Assert.Equal(("legacy:r4", true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
|
||||
reconcileReason: Classified(new[] { Esp(DeviceSetup44), Evt("desktop_arrived"), Evt("hello_provisioning_completed") }))));
|
||||
Assert.Equal(("legacy:r1c", true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
|
||||
reconcileReason: Classified(new[] { Esp(DeviceSetup44) }, selfDeploying: true))));
|
||||
Assert.Equal(("legacy:r1", true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed,
|
||||
failureReason: Classified(new[] { Evt("enrollment_failed") }))));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Derivation_recovers_the_origin_from_writer_suffixes()
|
||||
{
|
||||
var r6 = Classified(Array.Empty<EnrollmentEvent>());
|
||||
Assert.Equal(("maxlife:r6", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
|
||||
failureReason: r6 + " Verdict triggered by the agent's max-lifetime watchdog shutdown.")));
|
||||
Assert.Equal(("retro:r6", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
|
||||
failureReason: r6 + " Retro-reclassified from the legacy blanket timeout verdict.")));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Derivation_attributes_unambiguous_non_classifier_paths()
|
||||
{
|
||||
Assert.Equal((VerdictPaths.ManualFailed, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed,
|
||||
failureReason: "Manually marked as failed by administrator", failureSource: "manual", adminMarked: "Failed")));
|
||||
Assert.Equal((VerdictPaths.ManualSucceeded, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded, adminMarked: "Succeeded")));
|
||||
Assert.Equal(("rule:ANALYZE-ESP-001", true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed,
|
||||
failureReason: "Rule: ESP timeout", failureSource: "rule:ANALYZE-ESP-001")));
|
||||
Assert.Equal((VerdictPaths.AgentComplete, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded)));
|
||||
Assert.Equal((VerdictPaths.AgentComplete, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
|
||||
reconcileReason: "Late completion report received — upgraded prior 'Incomplete' verdict")));
|
||||
Assert.Equal((VerdictPaths.AgentCompleteSoft, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded, soft: true)));
|
||||
Assert.Equal((VerdictPaths.AgentFailed, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed, failureReason: "ESP timeout (0x800705B4)")));
|
||||
Assert.Equal((VerdictPaths.AgentEspFailureFallback, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed, failureReason: "ESP failure (backend fallback)")));
|
||||
Assert.Equal((VerdictPaths.SweepStalled, true), VerdictPathDerivation.Derive(Row(SessionStatus.Stalled, failureReason: "Agent silent for 130min (detected by maintenance sweep)")));
|
||||
Assert.Equal((VerdictPaths.AgentStallProbe, true), VerdictPathDerivation.Derive(Row(SessionStatus.Stalled, failureReason: "Agent reported stall after 60min without progress (stall_probe)")));
|
||||
Assert.Equal(("legacy:superseded", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete, failureReason: "Superseded by session abc: re-registered")));
|
||||
Assert.Equal((VerdictPaths.AgentWhiteGlovePending, true), VerdictPathDerivation.Derive(Row(SessionStatus.Pending)));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Derivation_never_invents_a_path()
|
||||
{
|
||||
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed, failureReason: "Session timed out after 5 hours (started at 2026-01-01 UTC)")));
|
||||
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete)));
|
||||
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.InProgress)));
|
||||
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded, reconcileReason: "Some future wording")));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
using AutopilotMonitor.Functions.Services.Notifications;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// HMAC request-signature scheme for generic JSON webhooks: known-vector stability (the
|
||||
/// scheme is a published contract receivers implement against), determinism, and
|
||||
/// secret/message sensitivity.
|
||||
/// </summary>
|
||||
public class WebhookSignatureTests
|
||||
{
|
||||
[Fact]
|
||||
public void ComputeSignature_MatchesKnownVector()
|
||||
{
|
||||
// Independently computed: HMACSHA256("whsec_test_0123456789abcdef", "1700000000.{"a":1}")
|
||||
var signature = WebhookSignatureCalculator.ComputeSignature(
|
||||
"whsec_test_0123456789abcdef", "1700000000", "{\"a\":1}");
|
||||
|
||||
Assert.Equal("sha256=07728d7e7b1736bac3de36a46c9bc0b660d59aa4a6f251544124fede1dc42a6c", signature);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ComputeSignature_IsDeterministic_AndSensitiveToEveryInput()
|
||||
{
|
||||
var baseline = WebhookSignatureCalculator.ComputeSignature("secret-0123456789", "1700000000", "{}");
|
||||
|
||||
Assert.Equal(baseline, WebhookSignatureCalculator.ComputeSignature("secret-0123456789", "1700000000", "{}"));
|
||||
Assert.NotEqual(baseline, WebhookSignatureCalculator.ComputeSignature("secret-0123456789X", "1700000000", "{}"));
|
||||
Assert.NotEqual(baseline, WebhookSignatureCalculator.ComputeSignature("secret-0123456789", "1700000001", "{}"));
|
||||
Assert.NotEqual(baseline, WebhookSignatureCalculator.ComputeSignature("secret-0123456789", "1700000000", "{ }"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void HeaderNames_AreTheDocumentedContract()
|
||||
{
|
||||
Assert.Equal("X-AutopilotMonitor-Timestamp", WebhookSignatureCalculator.TimestampHeader);
|
||||
Assert.Equal("X-AutopilotMonitor-Signature", WebhookSignatureCalculator.SignatureHeader);
|
||||
}
|
||||
}
|
||||
@@ -56,7 +56,6 @@
|
||||
<PackageReference Include="Microsoft.Identity.Web" Version="4.3.0" />
|
||||
<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.10" />
|
||||
<PackageReference Include="Microsoft.Extensions.Http.Polly" Version="8.0.25" />
|
||||
<PackageReference Include="Resend" Version="0.2.1" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
||||
+52
@@ -494,6 +494,54 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
}
|
||||
}
|
||||
|
||||
// --- Email Template Overrides ---
|
||||
|
||||
private const string EmailTemplatesPartition = "EmailTemplates";
|
||||
|
||||
public async Task<EmailTemplateOverride?> GetEmailTemplateOverrideAsync(string kind)
|
||||
{
|
||||
try
|
||||
{
|
||||
var entity = await _previewConfigTableClient.GetEntityAsync<TableEntity>(EmailTemplatesPartition, kind);
|
||||
var html = entity.Value.GetString("Html");
|
||||
if (string.IsNullOrEmpty(html)) return null;
|
||||
return new EmailTemplateOverride
|
||||
{
|
||||
Kind = kind,
|
||||
Html = html,
|
||||
UpdatedBy = entity.Value.GetString("UpdatedBy") ?? string.Empty,
|
||||
UpdatedUtc = entity.Value.GetDateTime("UpdatedUtc") ?? DateTime.MinValue,
|
||||
};
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 404)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public async Task SaveEmailTemplateOverrideAsync(EmailTemplateOverride overrideEntry)
|
||||
{
|
||||
var entity = new TableEntity(EmailTemplatesPartition, overrideEntry.Kind)
|
||||
{
|
||||
["Html"] = overrideEntry.Html,
|
||||
["UpdatedBy"] = overrideEntry.UpdatedBy,
|
||||
["UpdatedUtc"] = overrideEntry.UpdatedUtc == default ? DateTime.UtcNow : overrideEntry.UpdatedUtc,
|
||||
};
|
||||
await _previewConfigTableClient.UpsertEntityAsync(entity, TableUpdateMode.Replace);
|
||||
}
|
||||
|
||||
public async Task DeleteEmailTemplateOverrideAsync(string kind)
|
||||
{
|
||||
try
|
||||
{
|
||||
await _previewConfigTableClient.DeleteEntityAsync(EmailTemplatesPartition, kind);
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 404)
|
||||
{
|
||||
// already gone — reset is idempotent
|
||||
}
|
||||
}
|
||||
|
||||
// --- Preview Notification Email ---
|
||||
|
||||
public async Task<string?> GetNotificationEmailAsync(string tenantId)
|
||||
@@ -620,6 +668,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
{ "RebootDelaySeconds", config.RebootDelaySeconds },
|
||||
{ "EnableGeoLocation", config.EnableGeoLocation },
|
||||
{ "EnableTimezoneAutoSet", config.EnableTimezoneAutoSet },
|
||||
{ "EnableDoGroupIdAutoSet", config.EnableDoGroupIdAutoSet },
|
||||
{ "NtpServer", config.NtpServer },
|
||||
{ "EnableImeMatchLog", config.EnableImeMatchLog },
|
||||
{ "EnableGatherRuleDebugLog", config.EnableGatherRuleDebugLog },
|
||||
@@ -670,6 +719,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
{ "TrialStartedUtc", config.TrialStartedUtc },
|
||||
{ "TrialConsumed", config.TrialConsumed },
|
||||
{ "TrialGrantedBy", config.TrialGrantedBy },
|
||||
{ "ProDowngradedUtc", config.ProDowngradedUtc },
|
||||
// SLA targets
|
||||
{ "SlaTargetSuccessRate", config.SlaTargetSuccessRate.HasValue ? (double)config.SlaTargetSuccessRate.Value : (double?)null },
|
||||
{ "SlaTargetMaxDurationMinutes", config.SlaTargetMaxDurationMinutes },
|
||||
@@ -727,6 +777,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
RebootDelaySeconds = entity.GetInt32("RebootDelaySeconds"),
|
||||
EnableGeoLocation = entity.GetBoolean("EnableGeoLocation"),
|
||||
EnableTimezoneAutoSet = entity.GetBoolean("EnableTimezoneAutoSet"),
|
||||
EnableDoGroupIdAutoSet = entity.GetBoolean("EnableDoGroupIdAutoSet"),
|
||||
NtpServer = string.IsNullOrWhiteSpace(entity.GetString("NtpServer")) ? "time.windows.com" : entity.GetString("NtpServer"),
|
||||
EnableImeMatchLog = entity.GetBoolean("EnableImeMatchLog"),
|
||||
EnableGatherRuleDebugLog = entity.GetBoolean("EnableGatherRuleDebugLog"),
|
||||
@@ -781,6 +832,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
TrialStartedUtc = entity.GetDateTime("TrialStartedUtc"),
|
||||
TrialConsumed = entity.GetBoolean("TrialConsumed") ?? false,
|
||||
TrialGrantedBy = entity.GetString("TrialGrantedBy"),
|
||||
ProDowngradedUtc = entity.GetDateTime("ProDowngradedUtc"),
|
||||
// SLA targets
|
||||
SlaTargetSuccessRate = entity.GetDouble("SlaTargetSuccessRate") != null ? (decimal)entity.GetDouble("SlaTargetSuccessRate")! : null,
|
||||
SlaTargetMaxDurationMinutes = entity.GetInt32("SlaTargetMaxDurationMinutes"),
|
||||
|
||||
+146
@@ -422,6 +422,152 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
};
|
||||
}
|
||||
|
||||
// ===== VERDICT CALIBRATION DRIFT (docs/backend/verdict-calibration.md) =====
|
||||
|
||||
internal const string VerdictCalibrationRowKeyPrefix = "verdictcalibration|";
|
||||
|
||||
internal static string BuildVerdictCalibrationRowKey(string kind, string verdictPath, string status)
|
||||
{
|
||||
// Paths are a closed vocabulary plus rule ids (ANALYZE-...), statuses are enum names and
|
||||
// the group marker "*" - none contain table-key-hostile characters, but sanitize anyway.
|
||||
var k = SanitizeTableKey((kind ?? string.Empty).Trim().ToLowerInvariant());
|
||||
var p = SanitizeTableKey((verdictPath ?? string.Empty).Trim().ToLowerInvariant());
|
||||
var s = SanitizeTableKey((status ?? string.Empty).Trim().ToLowerInvariant());
|
||||
return $"{VerdictCalibrationRowKeyPrefix}{k}|{p}|{s}";
|
||||
}
|
||||
|
||||
public async Task<bool> TryRegisterVerdictCalibrationAlertAsync(string tenantId, VerdictCalibrationAlert alert)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(alert?.Kind) || string.IsNullOrWhiteSpace(alert?.VerdictPath))
|
||||
return false;
|
||||
try
|
||||
{
|
||||
await _table.AddEntityAsync(BuildVerdictCalibrationEntity(tenantId, alert!));
|
||||
_logger.LogInformation(
|
||||
"VerdictCalibration tracker registered: tenant={TenantId} kind={Kind} path={Path}", tenantId, alert!.Kind, alert.VerdictPath);
|
||||
return true;
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 409)
|
||||
{
|
||||
return false; // episode already active - one ops event per episode
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "VerdictCalibration tracker failed: tenant={TenantId} kind={Kind} path={Path}", tenantId, alert!.Kind, alert.VerdictPath);
|
||||
return false; // fail closed - never double-fire
|
||||
}
|
||||
}
|
||||
|
||||
public async Task RefreshVerdictCalibrationAlertAsync(string tenantId, VerdictCalibrationAlert alert)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(alert?.Kind) || string.IsNullOrWhiteSpace(alert?.VerdictPath))
|
||||
return;
|
||||
try
|
||||
{
|
||||
await _table.UpsertEntityAsync(BuildVerdictCalibrationEntity(tenantId, alert!), TableUpdateMode.Replace);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "VerdictCalibration tracker refresh failed: tenant={TenantId} kind={Kind} path={Path}", tenantId, alert!.Kind, alert.VerdictPath);
|
||||
}
|
||||
}
|
||||
|
||||
public async Task DeleteVerdictCalibrationAlertAsync(string tenantId, string kind, string verdictPath, string status)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(kind) || string.IsNullOrWhiteSpace(verdictPath))
|
||||
return;
|
||||
try
|
||||
{
|
||||
await _table.DeleteEntityAsync(tenantId.ToLowerInvariant(), BuildVerdictCalibrationRowKey(kind, verdictPath, status));
|
||||
}
|
||||
catch (RequestFailedException ex) when (ex.Status == 404)
|
||||
{
|
||||
// already gone - idempotent
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "VerdictCalibration tracker delete failed: tenant={TenantId} kind={Kind} path={Path}", tenantId, kind, verdictPath);
|
||||
}
|
||||
}
|
||||
|
||||
public async Task<List<VerdictCalibrationAlert>> GetVerdictCalibrationAlertsAsync(string tenantId)
|
||||
{
|
||||
var results = new List<VerdictCalibrationAlert>();
|
||||
if (string.IsNullOrWhiteSpace(tenantId)) return results;
|
||||
try
|
||||
{
|
||||
var partitionKey = tenantId.ToLowerInvariant();
|
||||
// '}' sorts directly after '|' - prefix range over "verdictcalibration|...".
|
||||
var filter = $"PartitionKey eq '{partitionKey}' and RowKey ge '{VerdictCalibrationRowKeyPrefix}' and RowKey lt 'verdictcalibration}}'";
|
||||
await foreach (var entity in _table.QueryAsync<TableEntity>(filter: filter))
|
||||
{
|
||||
results.Add(MapToVerdictCalibrationAlert(entity));
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "VerdictCalibration tracker list failed for tenant {TenantId}", tenantId);
|
||||
}
|
||||
return results;
|
||||
}
|
||||
|
||||
// internal static: pinned by round-trip unit tests (table-storage-serialization rule).
|
||||
internal static TableEntity BuildVerdictCalibrationEntity(string tenantId, VerdictCalibrationAlert alert)
|
||||
{
|
||||
var entity = new TableEntity(tenantId.ToLowerInvariant(), BuildVerdictCalibrationRowKey(alert.Kind, alert.VerdictPath, alert.Status))
|
||||
{
|
||||
["TenantId"] = tenantId,
|
||||
["Kind"] = alert.Kind,
|
||||
["VerdictPath"] = alert.VerdictPath,
|
||||
["Status"] = alert.Status ?? string.Empty,
|
||||
["WindowHitCount"] = alert.WindowHitCount,
|
||||
["WindowSessionCount"] = alert.WindowSessionCount,
|
||||
["BaselineHitCount"] = alert.BaselineHitCount,
|
||||
["BaselineSessionCount"] = alert.BaselineSessionCount,
|
||||
["WindowRatePct"] = alert.WindowRatePct,
|
||||
["BaselineRatePct"] = alert.BaselineRatePct,
|
||||
["WindowStartDate"] = alert.WindowStartDate ?? string.Empty,
|
||||
["WindowEndDate"] = alert.WindowEndDate ?? string.Empty,
|
||||
["FirstNotifiedAt"] = alert.FirstNotifiedAt,
|
||||
["LastEvaluatedAt"] = alert.LastEvaluatedAt,
|
||||
};
|
||||
if (alert.Lift.HasValue)
|
||||
entity["Lift"] = alert.Lift.Value;
|
||||
if (alert.Dimension != null)
|
||||
entity["DimensionJson"] = JsonSerializer.Serialize(alert.Dimension);
|
||||
return entity;
|
||||
}
|
||||
|
||||
internal static VerdictCalibrationAlert MapToVerdictCalibrationAlert(TableEntity entity)
|
||||
{
|
||||
RuleRegressionDimension? dimension = null;
|
||||
var dimensionJson = entity.GetString("DimensionJson");
|
||||
if (!string.IsNullOrEmpty(dimensionJson))
|
||||
{
|
||||
try { dimension = JsonSerializer.Deserialize<RuleRegressionDimension>(dimensionJson!); }
|
||||
catch (JsonException) { dimension = null; }
|
||||
}
|
||||
return new VerdictCalibrationAlert
|
||||
{
|
||||
TenantId = entity.GetString("TenantId") ?? entity.PartitionKey,
|
||||
Kind = entity.GetString("Kind") ?? string.Empty,
|
||||
VerdictPath = entity.GetString("VerdictPath") ?? string.Empty,
|
||||
Status = entity.GetString("Status") ?? string.Empty,
|
||||
WindowHitCount = entity.GetInt32("WindowHitCount") ?? 0,
|
||||
WindowSessionCount = entity.GetInt32("WindowSessionCount") ?? 0,
|
||||
BaselineHitCount = entity.GetInt32("BaselineHitCount") ?? 0,
|
||||
BaselineSessionCount = entity.GetInt32("BaselineSessionCount") ?? 0,
|
||||
WindowRatePct = entity.GetDouble("WindowRatePct") ?? 0,
|
||||
BaselineRatePct = entity.GetDouble("BaselineRatePct") ?? 0,
|
||||
Lift = entity.GetDouble("Lift"),
|
||||
WindowStartDate = entity.GetString("WindowStartDate") ?? string.Empty,
|
||||
WindowEndDate = entity.GetString("WindowEndDate") ?? string.Empty,
|
||||
Dimension = dimension,
|
||||
FirstNotifiedAt = entity.GetDateTimeOffset("FirstNotifiedAt")?.UtcDateTime ?? DateTime.MinValue,
|
||||
LastEvaluatedAt = entity.GetDateTimeOffset("LastEvaluatedAt")?.UtcDateTime ?? DateTime.MinValue,
|
||||
};
|
||||
}
|
||||
|
||||
internal const string AppVersionRegressionRowKeyPrefix = "appversionregression|";
|
||||
|
||||
/// <summary>
|
||||
|
||||
+6
@@ -60,6 +60,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
public Task<List<SessionSummary>> GetUsageWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null)
|
||||
=> _storage.GetUsageWindowSessionsAsync(startDate, endDate, tenantId);
|
||||
|
||||
public Task<List<SessionSummary>> GetMaintenanceWindowSessionsAsync(DateTime startDate, DateTime endDate)
|
||||
=> _storage.GetMaintenanceWindowSessionsAsync(startDate, endDate);
|
||||
|
||||
public Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null)
|
||||
=> _storage.GetGeoWindowSessionsAsync(startDate, endDate, tenantId);
|
||||
|
||||
@@ -72,6 +75,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults)
|
||||
=> _storage.GetLegacyTimeoutFailedSessionsAsync(tenantId, maxResults);
|
||||
|
||||
public Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults)
|
||||
=> _storage.GetSelfDeployingSilentSessionsAsync(tenantId, maxResults);
|
||||
|
||||
public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId)
|
||||
=> _storage.GetSessionsLeanAsync(tenantId);
|
||||
|
||||
|
||||
+12
@@ -159,5 +159,17 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
|
||||
public Task<int> DeleteDeviceJourneyAggregatesOlderThanAsync(DateTime cutoffDate)
|
||||
=> _storage.DeleteDeviceJourneyAggregatesOlderThanAsync(cutoffDate);
|
||||
|
||||
public Task<bool> SaveVerdictCalibrationAggregateAsync(VerdictCalibrationDailyAggregate aggregate)
|
||||
=> _storage.SaveVerdictCalibrationAggregateAsync(aggregate);
|
||||
|
||||
public Task<List<VerdictCalibrationDailyAggregate>> GetVerdictCalibrationAggregatesAsync(string tenantId, DateTime startDate, DateTime endDate)
|
||||
=> _storage.GetVerdictCalibrationAggregatesAsync(tenantId, startDate, endDate);
|
||||
|
||||
public Task DeleteVerdictCalibrationAggregateAsync(string tenantId, string dateKey)
|
||||
=> _storage.DeleteVerdictCalibrationAggregateAsync(tenantId, dateKey);
|
||||
|
||||
public Task<int> DeleteVerdictCalibrationAggregatesOlderThanAsync(DateTime cutoffDate)
|
||||
=> _storage.DeleteVerdictCalibrationAggregatesOlderThanAsync(cutoffDate);
|
||||
}
|
||||
}
|
||||
|
||||
+4
-4
@@ -58,7 +58,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
=> _storage.GetAllSessionStatsAsync(tenantIdFilter, days, allowedTenantIds);
|
||||
|
||||
public async Task<bool> UpdateSessionStatusAsync(
|
||||
string tenantId, string sessionId, SessionStatus status,
|
||||
string tenantId, string sessionId, SessionStatus status, string verdictPath,
|
||||
EnrollmentPhase? currentPhase = null, string? failureReason = null,
|
||||
DateTime? completedAt = null, DateTime? earliestEventTimestamp = null,
|
||||
DateTime? latestEventTimestamp = null, bool? isPreProvisioned = null,
|
||||
@@ -68,7 +68,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
string? failureSnapshotJson = null, bool allowTerminalReclassification = false,
|
||||
bool espSoftFailure = false, string? completionSource = null)
|
||||
{
|
||||
var transitioned = await _storage.UpdateSessionStatusAsync(tenantId, sessionId, status,
|
||||
var transitioned = await _storage.UpdateSessionStatusAsync(tenantId, sessionId, status, verdictPath,
|
||||
currentPhase, failureReason, completedAt, earliestEventTimestamp,
|
||||
latestEventTimestamp, isPreProvisioned, isUserDriven, resumedAt,
|
||||
stalledAt, clearStalledAt, clearFailureReason, failureSource, adminMarkedAction,
|
||||
@@ -142,8 +142,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
=> _storage.UpdateSessionDiagnosticsBlobAsync(tenantId, sessionId, blobName, destination);
|
||||
|
||||
public Task SetSessionPreProvisionedAsync(string tenantId, string sessionId, bool isPreProvisioned,
|
||||
SessionStatus? status = null, bool? isUserDriven = null)
|
||||
=> _storage.SetSessionPreProvisionedAsync(tenantId, sessionId, isPreProvisioned, status, isUserDriven);
|
||||
SessionStatus? status = null, bool? isUserDriven = null, string? verdictPath = null)
|
||||
=> _storage.SetSessionPreProvisionedAsync(tenantId, sessionId, isPreProvisioned, status, isUserDriven, verdictPath);
|
||||
|
||||
public Task UpdateSessionGeoAsync(string tenantId, string sessionId,
|
||||
string? country, string? region, string? city, string? loc)
|
||||
|
||||
+3
@@ -18,5 +18,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
|
||||
|
||||
public Task InitializeAsync()
|
||||
=> _storage.InitializeTablesAsync();
|
||||
|
||||
public Task EnsureAllAsync()
|
||||
=> _storage.EnsureAllTablesAsync();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,163 @@
|
||||
using System.Net;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using Microsoft.Azure.Functions.Worker;
|
||||
using Microsoft.Azure.Functions.Worker.Http;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Functions.Admin;
|
||||
|
||||
/// <summary>
|
||||
/// Global-Admin management of the two transactional email templates (welcome, farewell):
|
||||
/// read the effective template, store/reset an HTML override, and send a test mail.
|
||||
/// All routes are GlobalAdminOnly via <c>EndpointAccessPolicyCatalog</c>.
|
||||
/// </summary>
|
||||
public class EmailTemplatesFunction
|
||||
{
|
||||
private readonly ILogger<EmailTemplatesFunction> _logger;
|
||||
private readonly EmailTemplateService _templates;
|
||||
private readonly IEmailService _email;
|
||||
private readonly PreviewWhitelistService _previewWhitelistService;
|
||||
private readonly TenantConfigurationService _tenantConfigurationService;
|
||||
|
||||
public EmailTemplatesFunction(
|
||||
ILogger<EmailTemplatesFunction> logger,
|
||||
EmailTemplateService templates,
|
||||
IEmailService email,
|
||||
PreviewWhitelistService previewWhitelistService,
|
||||
TenantConfigurationService tenantConfigurationService)
|
||||
{
|
||||
_logger = logger;
|
||||
_templates = templates;
|
||||
_email = email;
|
||||
_previewWhitelistService = previewWhitelistService;
|
||||
_tenantConfigurationService = tenantConfigurationService;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GET /api/global/email-templates/{kind}
|
||||
/// Effective template: the override when stored, otherwise the built-in (raw, with the
|
||||
/// {{domainName}} placeholder so it can be edited as-is).
|
||||
/// </summary>
|
||||
[Function("GetEmailTemplate")]
|
||||
public async Task<HttpResponseData> Get(
|
||||
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "global/email-templates/{kind}")] HttpRequestData req,
|
||||
string kind)
|
||||
{
|
||||
if (!EmailTemplateService.TryParseKind(kind, out var templateKind))
|
||||
return await BadRequest(req, "Unknown template kind. Use 'welcome' or 'farewell'.");
|
||||
|
||||
var overrideEntry = await _templates.GetOverrideAsync(templateKind);
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new
|
||||
{
|
||||
kind = EmailTemplateService.KindKey(templateKind),
|
||||
subject = EmailTemplateService.Subject(templateKind),
|
||||
isOverridden = overrideEntry is not null,
|
||||
html = overrideEntry?.Html ?? EmailTemplateService.BuiltInRaw(templateKind),
|
||||
builtInHtml = EmailTemplateService.BuiltInRaw(templateKind),
|
||||
updatedBy = overrideEntry?.UpdatedBy,
|
||||
updatedUtc = overrideEntry?.UpdatedUtc,
|
||||
placeholder = EmailTemplateService.DomainPlaceholder,
|
||||
maxLength = EmailTemplateService.MaxHtmlLength,
|
||||
});
|
||||
return response;
|
||||
}
|
||||
|
||||
/// <summary>PUT /api/global/email-templates/{kind} — body { html } stores an override.</summary>
|
||||
[Function("SaveEmailTemplate")]
|
||||
public async Task<HttpResponseData> Save(
|
||||
[HttpTrigger(AuthorizationLevel.Anonymous, "put", Route = "global/email-templates/{kind}")] HttpRequestData req,
|
||||
string kind)
|
||||
{
|
||||
if (!EmailTemplateService.TryParseKind(kind, out var templateKind))
|
||||
return await BadRequest(req, "Unknown template kind. Use 'welcome' or 'farewell'.");
|
||||
|
||||
var body = await req.ReadFromJsonAsync<EmailTemplateRequest>();
|
||||
var error = EmailTemplateService.Validate(body?.Html);
|
||||
if (error is not null)
|
||||
return await BadRequest(req, error);
|
||||
|
||||
var upn = TenantHelper.GetUserIdentifier(req);
|
||||
var saved = await _templates.SaveOverrideAsync(templateKind, body!.Html!, upn);
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new { kind = saved.Kind, isOverridden = true, updatedBy = saved.UpdatedBy, updatedUtc = saved.UpdatedUtc });
|
||||
return response;
|
||||
}
|
||||
|
||||
/// <summary>DELETE /api/global/email-templates/{kind} — resets to the built-in template.</summary>
|
||||
[Function("ResetEmailTemplate")]
|
||||
public async Task<HttpResponseData> Reset(
|
||||
[HttpTrigger(AuthorizationLevel.Anonymous, "delete", Route = "global/email-templates/{kind}")] HttpRequestData req,
|
||||
string kind)
|
||||
{
|
||||
if (!EmailTemplateService.TryParseKind(kind, out var templateKind))
|
||||
return await BadRequest(req, "Unknown template kind. Use 'welcome' or 'farewell'.");
|
||||
|
||||
await _templates.DeleteOverrideAsync(templateKind, TenantHelper.GetUserIdentifier(req));
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new { kind = EmailTemplateService.KindKey(templateKind), isOverridden = false });
|
||||
return response;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// POST /api/global/email-templates/{kind}/test — body { html? } sends the effective template
|
||||
/// (or the unsaved draft) to the caller's tenant contact address: the activation
|
||||
/// notification email, then TenantConfiguration.ContactEmail, then the caller's UPN.
|
||||
/// Domain = the caller's tenant domain.
|
||||
/// </summary>
|
||||
[Function("SendEmailTemplateTest")]
|
||||
public async Task<HttpResponseData> SendTest(
|
||||
[HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "global/email-templates/{kind}/test")] HttpRequestData req,
|
||||
string kind)
|
||||
{
|
||||
if (!EmailTemplateService.TryParseKind(kind, out var templateKind))
|
||||
return await BadRequest(req, "Unknown template kind. Use 'welcome' or 'farewell'.");
|
||||
|
||||
var body = await req.ReadFromJsonAsync<EmailTemplateRequest>();
|
||||
var draft = string.IsNullOrWhiteSpace(body?.Html) ? null : body!.Html;
|
||||
if (draft is not null)
|
||||
{
|
||||
var error = EmailTemplateService.Validate(draft);
|
||||
if (error is not null) return await BadRequest(req, error);
|
||||
}
|
||||
|
||||
var tenantId = TenantHelper.GetTenantId(req);
|
||||
var upn = TenantHelper.GetUserIdentifier(req);
|
||||
var tenantConfig = await _tenantConfigurationService.GetConfigurationAsync(tenantId);
|
||||
|
||||
var toEmail = await _previewWhitelistService.GetNotificationEmailAsync(tenantId);
|
||||
if (string.IsNullOrWhiteSpace(toEmail)) toEmail = tenantConfig.ContactEmail;
|
||||
if (string.IsNullOrWhiteSpace(toEmail)) toEmail = upn.Contains('@') ? upn : null;
|
||||
if (string.IsNullOrWhiteSpace(toEmail))
|
||||
return await BadRequest(req, "No contact email found for your tenant — set a notification email or ContactEmail first.");
|
||||
|
||||
var sent = await _email.SendTestAsync(templateKind, toEmail, tenantConfig.DomainName, draft);
|
||||
_logger.LogInformation("Email template test {Kind} requested by {Upn} → {ToEmail}: {Result}", templateKind, upn, toEmail, sent ? "accepted" : "failed");
|
||||
|
||||
if (!sent)
|
||||
{
|
||||
var failed = req.CreateResponse(HttpStatusCode.BadGateway);
|
||||
await failed.WriteAsJsonAsync(new { error = "The email provider did not accept the message. Check the backend log for the provider response.", sentTo = toEmail });
|
||||
return failed;
|
||||
}
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new { sentTo = toEmail, domainName = tenantConfig.DomainName, draft = draft is not null });
|
||||
return response;
|
||||
}
|
||||
|
||||
private static async Task<HttpResponseData> BadRequest(HttpRequestData req, string error)
|
||||
{
|
||||
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
||||
await bad.WriteAsJsonAsync(new { error });
|
||||
return bad;
|
||||
}
|
||||
}
|
||||
|
||||
public class EmailTemplateRequest
|
||||
{
|
||||
public string? Html { get; set; }
|
||||
}
|
||||
+5
-1
@@ -35,6 +35,8 @@ namespace AutopilotMonitor.Functions.Functions.Admin
|
||||
/// Query parameters:
|
||||
/// - mode: "legacy_timeouts" (default) — re-classify pre-classifier "Session timed out" Failed rows;
|
||||
/// "pending_orphans" — resolve Pending rows superseded by a later session of the same device.
|
||||
/// "self_deploying_silent" — heal self-deploying (kiosk) rows parked Incomplete/AwaitingUser/Stalled
|
||||
/// after Device ESP provisioned to Succeeded (classifier rule 1c, 2026-08-23).
|
||||
/// - dryRun: "false" to actually write; anything else (or absent) reports only.
|
||||
/// - tenantId: optional single-tenant scope (GUID); absent = all tenants.
|
||||
/// - maxSessions: per-invocation examination cap (default 200, max 2000). Re-run to continue.
|
||||
@@ -82,6 +84,8 @@ namespace AutopilotMonitor.Functions.Functions.Admin
|
||||
await _service.ReclassifyLegacyTimeoutsAsync(tenantIdScope, dryRun, maxSessions, userEmail),
|
||||
LegacyReclassificationService.ModePendingOrphans =>
|
||||
await _service.ResolvePendingOrphansAsync(tenantIdScope, dryRun, maxSessions, userEmail),
|
||||
LegacyReclassificationService.ModeSelfDeployingSilent =>
|
||||
await _service.ReconcileSelfDeployingSilentAsync(tenantIdScope, dryRun, maxSessions, userEmail),
|
||||
_ => null,
|
||||
};
|
||||
|
||||
@@ -90,7 +94,7 @@ namespace AutopilotMonitor.Functions.Functions.Admin
|
||||
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
||||
await bad.WriteAsJsonAsync(new
|
||||
{
|
||||
error = $"Unknown mode '{mode}'. Valid: {LegacyReclassificationService.ModeLegacyTimeouts}, {LegacyReclassificationService.ModePendingOrphans}"
|
||||
error = $"Unknown mode '{mode}'. Valid: {LegacyReclassificationService.ModeLegacyTimeouts}, {LegacyReclassificationService.ModePendingOrphans}, {LegacyReclassificationService.ModeSelfDeployingSilent}"
|
||||
});
|
||||
return bad;
|
||||
}
|
||||
|
||||
@@ -142,6 +142,23 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
|| string.Equals(destination, "Hosted", StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Global ∪ tenant diagnostics paths: global first, order preserved, blank paths dropped,
|
||||
/// duplicates (trimmed, case-insensitive) collapsed onto the first occurrence. The agent
|
||||
/// keys ZIP entries on the path, so a path present in both lists would otherwise produce
|
||||
/// duplicate entry names inside the archive.
|
||||
/// </summary>
|
||||
internal static List<DiagnosticsLogPath> MergeDiagnosticsLogPaths(
|
||||
IEnumerable<DiagnosticsLogPath> global,
|
||||
IEnumerable<DiagnosticsLogPath> tenant)
|
||||
{
|
||||
return global.Concat(tenant)
|
||||
.Where(p => !string.IsNullOrWhiteSpace(p?.Path))
|
||||
.GroupBy(p => p.Path.Trim(), StringComparer.OrdinalIgnoreCase)
|
||||
.Select(g => g.First())
|
||||
.ToList();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Returns the (ZipSha256, ExeSha256) pair appropriate for the calling agent.
|
||||
/// Reads the X-Agent-Version header, parses the major, and dispatches to the
|
||||
@@ -242,10 +259,10 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
// Get active IME log patterns for this tenant (from Table Storage)
|
||||
var imeLogPatterns = await _imeLogPatternService.GetActivePatternsForTenantAsync(tenantId);
|
||||
|
||||
// Merge global + tenant-specific diagnostics log paths
|
||||
var globalDiagPaths = adminConfig.GetDiagnosticsGlobalLogPaths();
|
||||
var tenantDiagPaths = tenantConfig.GetDiagnosticsLogPaths();
|
||||
var diagLogPaths = globalDiagPaths.Concat(tenantDiagPaths).ToList();
|
||||
// Merge global + tenant-specific diagnostics log paths (the built-in sections are
|
||||
// compiled into the agent — DiagnosticsBuiltInSections — and never travel here)
|
||||
var diagLogPaths = MergeDiagnosticsLogPaths(
|
||||
adminConfig.GetDiagnosticsGlobalLogPaths(), tenantConfig.GetDiagnosticsLogPaths());
|
||||
|
||||
// Select per-line hash oracle from the X-Agent-Version header (parametric per major).
|
||||
// The wire response keeps generic field names (LatestAgentSha256 / LatestAgentExeSha256)
|
||||
@@ -274,7 +291,7 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new AgentConfigResponse
|
||||
{
|
||||
ConfigVersion = 39, // Collectors.EnableSystemTimelineWatcher + SystemEventBackfillLookbackMinutes (clock changes + sleep episodes)
|
||||
ConfigVersion = 40, // EnableDoGroupIdAutoSet (Delivery Optimization group ID from network fingerprint)
|
||||
UploadIntervalSeconds = Shared.Constants.DefaultUploadIntervalSeconds,
|
||||
SelfDestructOnComplete = tenantConfig.SelfDestructOnComplete ?? true,
|
||||
KeepLogFile = tenantConfig.KeepLogFile ?? false,
|
||||
@@ -312,6 +329,7 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
AllowAgentDowngrade = adminConfig.AllowAgentDowngrade,
|
||||
NtpServer = string.IsNullOrEmpty(tenantConfig.NtpServer) ? "time.windows.com" : tenantConfig.NtpServer,
|
||||
EnableTimezoneAutoSet = tenantConfig.EnableTimezoneAutoSet ?? false,
|
||||
EnableDoGroupIdAutoSet = tenantConfig.EnableDoGroupIdAutoSet ?? false,
|
||||
SendTraceEvents = tenantConfig.SendTraceEvents,
|
||||
UnrestrictedMode = TenantEntitlementService.IsUnrestrictedModeActive(tenantConfig, DateTime.UtcNow),
|
||||
GatherRules = gatherRules,
|
||||
|
||||
@@ -27,14 +27,16 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
private readonly TenantConfigurationService _configService;
|
||||
private readonly AdminConfigurationService _adminConfigService;
|
||||
private readonly IMaintenanceRepository _maintenanceRepo;
|
||||
private readonly OpsEventService _opsEvents;
|
||||
private readonly TimeProvider _time;
|
||||
|
||||
public PlanManagementFunction(
|
||||
ILogger<PlanManagementFunction> logger,
|
||||
TenantConfigurationService configService,
|
||||
AdminConfigurationService adminConfigService,
|
||||
IMaintenanceRepository maintenanceRepo)
|
||||
: this(logger, configService, adminConfigService, maintenanceRepo, TimeProvider.System)
|
||||
IMaintenanceRepository maintenanceRepo,
|
||||
OpsEventService opsEvents)
|
||||
: this(logger, configService, adminConfigService, maintenanceRepo, opsEvents, TimeProvider.System)
|
||||
{
|
||||
}
|
||||
|
||||
@@ -44,12 +46,14 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
TenantConfigurationService configService,
|
||||
AdminConfigurationService adminConfigService,
|
||||
IMaintenanceRepository maintenanceRepo,
|
||||
OpsEventService opsEvents,
|
||||
TimeProvider time)
|
||||
{
|
||||
_logger = logger;
|
||||
_configService = configService;
|
||||
_adminConfigService = adminConfigService;
|
||||
_maintenanceRepo = maintenanceRepo;
|
||||
_opsEvents = opsEvents;
|
||||
_time = time;
|
||||
}
|
||||
|
||||
@@ -139,22 +143,8 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
var nowUtc = _time.GetUtcNow().UtcDateTime;
|
||||
var changes = new Dictionary<string, string>();
|
||||
|
||||
if (newPlanTier != null && !string.Equals(config.PlanTier, newPlanTier, StringComparison.Ordinal))
|
||||
{
|
||||
changes["PlanTier"] = $"{config.PlanTier} -> {newPlanTier}";
|
||||
config.PlanTier = newPlanTier;
|
||||
}
|
||||
|
||||
if (trialProvided && config.TrialExpiresUtc != newTrialExpiresUtc)
|
||||
{
|
||||
changes["TrialExpiresUtc"] = $"{FormatUtc(config.TrialExpiresUtc)} -> {FormatUtc(newTrialExpiresUtc)}";
|
||||
config.TrialExpiresUtc = newTrialExpiresUtc;
|
||||
if (newTrialExpiresUtc.HasValue)
|
||||
{
|
||||
config.TrialStartedUtc ??= nowUtc;
|
||||
config.TrialGrantedBy = caller;
|
||||
}
|
||||
}
|
||||
var (editionBefore, editionAfter) =
|
||||
ApplyPlanChanges(config, newPlanTier, trialProvided, newTrialExpiresUtc, caller, nowUtc, changes);
|
||||
|
||||
if (changes.Count > 0)
|
||||
{
|
||||
@@ -171,9 +161,17 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
requestCtx.TargetTenantId,
|
||||
caller,
|
||||
changes);
|
||||
}
|
||||
|
||||
var effectiveEdition = FeatureEntitlementCatalog.ResolveEdition(config.PlanTier, config.TrialExpiresUtc, nowUtc);
|
||||
if (editionBefore == TenantEdition.Pro && editionAfter == TenantEdition.Community)
|
||||
{
|
||||
await _opsEvents.RecordTenantPlanDowngradedAsync(
|
||||
requestCtx.TargetTenantId,
|
||||
config.DomainName,
|
||||
caller,
|
||||
TenantEntitlementService.GetRetentionGraceEndUtc(config, nowUtc),
|
||||
config.DataRetentionDays);
|
||||
}
|
||||
}
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new
|
||||
@@ -182,7 +180,8 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
planTier = config.PlanTier,
|
||||
trialExpiresUtc = config.TrialExpiresUtc,
|
||||
trialConsumed = config.TrialConsumed,
|
||||
effectiveEdition = effectiveEdition.ToString().ToLowerInvariant()
|
||||
effectiveEdition = editionAfter.ToString().ToLowerInvariant(),
|
||||
retentionGraceEndsUtc = TenantEntitlementService.GetRetentionGraceEndUtc(config, nowUtc)
|
||||
});
|
||||
return response;
|
||||
}
|
||||
@@ -195,6 +194,61 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pure mutation core of the PATCH plan endpoint — no I/O, unit-testable. Applies the
|
||||
/// plan/trial changes to <paramref name="config"/>, records human-readable entries in
|
||||
/// <paramref name="changes"/>, and returns the EFFECTIVE edition before/after (same
|
||||
/// <paramref name="nowUtc"/> for both, so only the mutation itself can flip it).
|
||||
///
|
||||
/// Maintains the retention downgrade grace anchor: an effective Pro→Community
|
||||
/// transition stamps <see cref="TenantConfiguration.ProDowngradedUtc"/> (covers both a
|
||||
/// planTier downgrade and an explicitly ended trial); any state that is effectively Pro
|
||||
/// afterwards clears the anchor, so a re-upgrade also resets the grace clock.
|
||||
/// </summary>
|
||||
internal static (TenantEdition Before, TenantEdition After) ApplyPlanChanges(
|
||||
TenantConfiguration config,
|
||||
string? newPlanTier,
|
||||
bool trialProvided,
|
||||
DateTime? newTrialExpiresUtc,
|
||||
string caller,
|
||||
DateTime nowUtc,
|
||||
Dictionary<string, string> changes)
|
||||
{
|
||||
var before = FeatureEntitlementCatalog.ResolveEdition(config.PlanTier, config.TrialExpiresUtc, nowUtc);
|
||||
|
||||
if (newPlanTier != null && !string.Equals(config.PlanTier, newPlanTier, StringComparison.Ordinal))
|
||||
{
|
||||
changes["PlanTier"] = $"{config.PlanTier} -> {newPlanTier}";
|
||||
config.PlanTier = newPlanTier;
|
||||
}
|
||||
|
||||
if (trialProvided && config.TrialExpiresUtc != newTrialExpiresUtc)
|
||||
{
|
||||
changes["TrialExpiresUtc"] = $"{FormatUtc(config.TrialExpiresUtc)} -> {FormatUtc(newTrialExpiresUtc)}";
|
||||
config.TrialExpiresUtc = newTrialExpiresUtc;
|
||||
if (newTrialExpiresUtc.HasValue)
|
||||
{
|
||||
config.TrialStartedUtc ??= nowUtc;
|
||||
config.TrialGrantedBy = caller;
|
||||
}
|
||||
}
|
||||
|
||||
var after = FeatureEntitlementCatalog.ResolveEdition(config.PlanTier, config.TrialExpiresUtc, nowUtc);
|
||||
|
||||
if (before == TenantEdition.Pro && after == TenantEdition.Community)
|
||||
{
|
||||
changes["ProDowngradedUtc"] = $"{FormatUtc(config.ProDowngradedUtc)} -> {FormatUtc(nowUtc)}";
|
||||
config.ProDowngradedUtc = nowUtc;
|
||||
}
|
||||
else if (after == TenantEdition.Pro && config.ProDowngradedUtc.HasValue)
|
||||
{
|
||||
changes["ProDowngradedUtc"] = $"{FormatUtc(config.ProDowngradedUtc)} -> (none)";
|
||||
config.ProDowngradedUtc = null;
|
||||
}
|
||||
|
||||
return (before, after);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// POST /api/config/{tenantId}/trial — TenantAdminOrGA (catalog-enforced). Self-service
|
||||
/// 30-day Pro trial, exactly once per tenant. 409 when the trial was already
|
||||
@@ -262,6 +316,8 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
config.TrialExpiresUtc = nowUtc.AddDays(SelfServiceTrialDays);
|
||||
config.TrialConsumed = true;
|
||||
config.TrialGrantedBy = caller;
|
||||
// Effectively Pro again — a leftover retention grace anchor is obsolete.
|
||||
config.ProDowngradedUtc = null;
|
||||
config.UpdatedBy = caller;
|
||||
|
||||
// Fail-loud: throws when the write did not persist (cache invalidated in finally).
|
||||
|
||||
+1
-1
@@ -82,7 +82,7 @@ namespace AutopilotMonitor.Functions.Functions.Config
|
||||
|
||||
var testAlert = NotificationAlertBuilder.BuildTestAlert();
|
||||
var result = await _webhookNotificationService.SendNotificationWithResultAsync(
|
||||
channel.Url, (WebhookProviderType)channel.ProviderType, testAlert, channel.GetCustomHeaders());
|
||||
channel.Url, (WebhookProviderType)channel.ProviderType, testAlert, channel.GetCustomHeaders(), channel.GetSigningSecret());
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(new
|
||||
|
||||
+77
@@ -0,0 +1,77 @@
|
||||
using System;
|
||||
using System.Linq;
|
||||
using System.Net;
|
||||
using System.Threading.Tasks;
|
||||
using AutopilotMonitor.Functions.Services;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Azure.Functions.Worker;
|
||||
using Microsoft.Azure.Functions.Worker.Http;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Functions.Diagnostics
|
||||
{
|
||||
public class GetDiagnosticsPathsFunction
|
||||
{
|
||||
private readonly ILogger<GetDiagnosticsPathsFunction> _logger;
|
||||
private readonly AdminConfigurationService _adminConfigService;
|
||||
|
||||
public GetDiagnosticsPathsFunction(
|
||||
ILogger<GetDiagnosticsPathsFunction> logger,
|
||||
AdminConfigurationService adminConfigService)
|
||||
{
|
||||
_logger = logger;
|
||||
_adminConfigService = adminConfigService;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// GET /api/diagnostics/paths
|
||||
/// What every diagnostics package collects before a tenant's own entries: the built-in
|
||||
/// section catalog (compiled into the agent, shared through
|
||||
/// <see cref="DiagnosticsBuiltInSections"/>) and the platform-wide global paths set by
|
||||
/// Global Admins. Member-readable by design — the catalog is code, the global list is
|
||||
/// non-secret admin data, and every settings viewer needs to see what is collected from
|
||||
/// the tenant's devices. Tenant-less: the payload is platform-wide, not tenant data.
|
||||
/// </summary>
|
||||
[Function("GetDiagnosticsPaths")]
|
||||
public async Task<HttpResponseData> Run(
|
||||
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "diagnostics/paths")] HttpRequestData req)
|
||||
{
|
||||
try
|
||||
{
|
||||
// Authentication + MemberRead authorization enforced by PolicyEnforcementMiddleware
|
||||
var adminConfig = await _adminConfigService.GetConfigurationAsync();
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(BuildPayload(adminConfig));
|
||||
return response;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Error getting diagnostics paths");
|
||||
var response = req.CreateResponse(HttpStatusCode.InternalServerError);
|
||||
await response.WriteAsJsonAsync(new { error = "Internal server error" });
|
||||
return response;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Static projection so unit tests pin the wire shape without an HttpRequestData mock.
|
||||
/// <c>condition</c> travels as the enum NAME ("Always" | "RealmJoinWatcher" |
|
||||
/// "DevicePreparation"), never the integer — the web switches on the string.
|
||||
/// </summary>
|
||||
internal static object BuildPayload(AdminConfiguration adminConfig) => new
|
||||
{
|
||||
builtIn = DiagnosticsBuiltInSections.All.Select(s => new
|
||||
{
|
||||
id = s.Id,
|
||||
zipFolder = s.ZipFolder,
|
||||
sourceFolder = s.SourceFolder,
|
||||
patterns = s.Patterns,
|
||||
includeSubfolders = s.IncludeSubfolders,
|
||||
description = s.Description,
|
||||
condition = s.Condition.ToString(),
|
||||
}).ToArray(),
|
||||
globalPaths = adminConfig.GetDiagnosticsGlobalLogPaths(),
|
||||
};
|
||||
}
|
||||
}
|
||||
+85
-31
@@ -10,15 +10,21 @@ using Microsoft.Extensions.Logging;
|
||||
namespace AutopilotMonitor.Functions.Functions.Maintenance;
|
||||
|
||||
/// <summary>
|
||||
/// Daily informational sweep over tenant Pro trials. NOT load-bearing — trial expiry
|
||||
/// degrades a tenant to Community at READ time (FeatureEntitlementCatalog.ResolveEdition);
|
||||
/// this timer only surfaces the transitions as ops events so the operator has visibility:
|
||||
/// Daily informational sweep over the tenant plan lifecycle. NOT load-bearing — trial expiry
|
||||
/// degrades a tenant to Community at READ time (FeatureEntitlementCatalog.ResolveEdition),
|
||||
/// and the retention grace is likewise resolved read-time by the retention fanout; this timer
|
||||
/// only surfaces the transitions as ops events so the operator has visibility:
|
||||
/// <list type="bullet">
|
||||
/// <item><c>TenantTrialExpired</c> (Warning) — the trial ended within the last sweep window
|
||||
/// (24h look-back matching the daily cadence).</item>
|
||||
/// <item><c>TenantTrialExpiring</c> (Info) — the trial ends within the next
|
||||
/// <see cref="ExpiringHeadsUpDays"/> days (re-emitted each daily run until expiry —
|
||||
/// acceptable for an Info-tier heads-up, no dedupe state to maintain).</item>
|
||||
/// <item><c>TenantRetentionGraceExpiring</c> / <c>TenantRetentionGraceEnded</c> (Warning) —
|
||||
/// same window mechanics for the retention downgrade grace
|
||||
/// (TenantEntitlementService.GetRetentionGraceEndUtc). Emitted only when data is
|
||||
/// actually at risk: stored retention above the Community cap (0 = infinite ⇒ never
|
||||
/// at risk, the fanout skips those tenants).</item>
|
||||
/// </list>
|
||||
/// Tenants whose stored <c>PlanTier</c> is already a permanent Pro tier ("pro"/legacy "enterprise") are skipped —
|
||||
/// their trial timestamps are inert leftovers and expiry changes nothing.
|
||||
@@ -90,45 +96,93 @@ public class TrialExpirySweepFunction
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
|
||||
if (config.TrialExpiresUtc is not DateTime expiry)
|
||||
continue;
|
||||
|
||||
// Permanent Pro: the trial timestamps are inert leftovers — expiry changes nothing.
|
||||
if (FeatureEntitlementCatalog.IsPermanentProTier(config.PlanTier))
|
||||
continue;
|
||||
|
||||
result.TrialsSeen++;
|
||||
|
||||
if (expiry <= now)
|
||||
{
|
||||
if (expiry > now - ExpiredLookBack)
|
||||
{
|
||||
await _opsEvents.RecordTenantTrialExpiredAsync(config.TenantId, config.DomainName, expiry)
|
||||
.ConfigureAwait(false);
|
||||
result.ExpiredEmitted++;
|
||||
}
|
||||
// Older expiries were reported by a previous run — stay silent.
|
||||
}
|
||||
else if (expiry <= now + ExpiringHeadsUp)
|
||||
{
|
||||
var daysLeft = Math.Max(1, (int)Math.Ceiling((expiry - now).TotalDays));
|
||||
await _opsEvents.RecordTenantTrialExpiringAsync(config.TenantId, config.DomainName, expiry, daysLeft)
|
||||
.ConfigureAwait(false);
|
||||
result.ExpiringEmitted++;
|
||||
}
|
||||
await SweepTrialAsync(config, now, result).ConfigureAwait(false);
|
||||
await SweepRetentionGraceAsync(config, now, result).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"TrialExpirySweep completed: trialsSeen={TrialsSeen} expiredEmitted={ExpiredEmitted} expiringEmitted={ExpiringEmitted}",
|
||||
result.TrialsSeen, result.ExpiredEmitted, result.ExpiringEmitted);
|
||||
"TrialExpirySweep completed: trialsSeen={TrialsSeen} expiredEmitted={ExpiredEmitted} expiringEmitted={ExpiringEmitted} " +
|
||||
"graceExpiringEmitted={GraceExpiringEmitted} graceEndedEmitted={GraceEndedEmitted}",
|
||||
result.TrialsSeen, result.ExpiredEmitted, result.ExpiringEmitted,
|
||||
result.GraceExpiringEmitted, result.GraceEndedEmitted);
|
||||
return result;
|
||||
}
|
||||
|
||||
private async Task SweepTrialAsync(
|
||||
AutopilotMonitor.Shared.Models.TenantConfiguration config, DateTime now, SweepResult result)
|
||||
{
|
||||
if (config.TrialExpiresUtc is not DateTime expiry)
|
||||
return;
|
||||
|
||||
// Permanent Pro: the trial timestamps are inert leftovers — expiry changes nothing.
|
||||
if (FeatureEntitlementCatalog.IsPermanentProTier(config.PlanTier))
|
||||
return;
|
||||
|
||||
result.TrialsSeen++;
|
||||
|
||||
if (expiry <= now)
|
||||
{
|
||||
if (expiry > now - ExpiredLookBack)
|
||||
{
|
||||
await _opsEvents.RecordTenantTrialExpiredAsync(config.TenantId, config.DomainName, expiry)
|
||||
.ConfigureAwait(false);
|
||||
result.ExpiredEmitted++;
|
||||
}
|
||||
// Older expiries were reported by a previous run — stay silent.
|
||||
}
|
||||
else if (expiry <= now + ExpiringHeadsUp)
|
||||
{
|
||||
var daysLeft = Math.Max(1, (int)Math.Ceiling((expiry - now).TotalDays));
|
||||
await _opsEvents.RecordTenantTrialExpiringAsync(config.TenantId, config.DomainName, expiry, daysLeft)
|
||||
.ConfigureAwait(false);
|
||||
result.ExpiringEmitted++;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Retention downgrade grace visibility: warn shortly before and once after the grace ends,
|
||||
/// but only when data is actually at risk — stored retention above the Community cap.
|
||||
/// (0 = the GA-only infinite escape hatch: the fanout skips those tenants, nothing is ever
|
||||
/// deleted, so the comparison excludes it naturally.) GetRetentionGraceEndUtc returns null
|
||||
/// for effectively-Pro tenants, so re-upgraded tenants go silent automatically.
|
||||
/// </summary>
|
||||
private async Task SweepRetentionGraceAsync(
|
||||
AutopilotMonitor.Shared.Models.TenantConfiguration config, DateTime now, SweepResult result)
|
||||
{
|
||||
var communityCap = FeatureEntitlementCatalog.Get(TenantEdition.Community).RetentionCapDays;
|
||||
if (config.DataRetentionDays <= communityCap)
|
||||
return;
|
||||
|
||||
if (TenantEntitlementService.GetRetentionGraceEndUtc(config, now) is not DateTime graceEnd)
|
||||
return;
|
||||
|
||||
if (graceEnd <= now)
|
||||
{
|
||||
if (graceEnd > now - ExpiredLookBack)
|
||||
{
|
||||
await _opsEvents.RecordTenantRetentionGraceEndedAsync(
|
||||
config.TenantId, config.DomainName, graceEnd, config.DataRetentionDays)
|
||||
.ConfigureAwait(false);
|
||||
result.GraceEndedEmitted++;
|
||||
}
|
||||
}
|
||||
else if (graceEnd <= now + ExpiringHeadsUp)
|
||||
{
|
||||
var daysLeft = Math.Max(1, (int)Math.Ceiling((graceEnd - now).TotalDays));
|
||||
await _opsEvents.RecordTenantRetentionGraceExpiringAsync(
|
||||
config.TenantId, config.DomainName, graceEnd, daysLeft, config.DataRetentionDays)
|
||||
.ConfigureAwait(false);
|
||||
result.GraceExpiringEmitted++;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Summary counters returned from <see cref="RunCoreAsync"/> so tests can assert outcomes.</summary>
|
||||
internal sealed class SweepResult
|
||||
{
|
||||
public int TrialsSeen { get; set; }
|
||||
public int ExpiredEmitted { get; set; }
|
||||
public int ExpiringEmitted { get; set; }
|
||||
public int GraceExpiringEmitted { get; set; }
|
||||
public int GraceEndedEmitted { get; set; }
|
||||
}
|
||||
}
|
||||
|
||||
+232
@@ -0,0 +1,232 @@
|
||||
using System.Globalization;
|
||||
using System.Net;
|
||||
using AutopilotMonitor.Functions.Helpers;
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Azure.Functions.Worker;
|
||||
using Microsoft.Azure.Functions.Worker.Http;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Functions.Metrics
|
||||
{
|
||||
/// <summary>
|
||||
/// Operator-only verdict calibration matrix (docs/backend/verdict-calibration.md): per verdict
|
||||
/// path, how many sessions it produced in the window, its share, how often it was overridden
|
||||
/// (admin / late completion / other), the 7-day re-enrollment proxy, and a 7d-vs-28d trend.
|
||||
/// Platform scope ("global" partition) or one tenant via ?tenantId=. Serves the admin
|
||||
/// metrics page and the <c>get_verdict_calibration</c> MCP tool. Never customer-facing —
|
||||
/// these are platform-internal classifier diagnostics, not product features.
|
||||
/// </summary>
|
||||
public class GetVerdictCalibrationFunction
|
||||
{
|
||||
private readonly ILogger<GetVerdictCalibrationFunction> _logger;
|
||||
private readonly IMetricsRepository _metricsRepo;
|
||||
private readonly IHardwareRejectionNotificationTracker _tracker;
|
||||
|
||||
public GetVerdictCalibrationFunction(
|
||||
ILogger<GetVerdictCalibrationFunction> logger,
|
||||
IMetricsRepository metricsRepo,
|
||||
IHardwareRejectionNotificationTracker tracker)
|
||||
{
|
||||
_logger = logger;
|
||||
_metricsRepo = metricsRepo;
|
||||
_tracker = tracker;
|
||||
}
|
||||
|
||||
[Function("GetVerdictCalibration")]
|
||||
public async Task<HttpResponseData> Run(
|
||||
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "global/metrics/verdict-calibration")] HttpRequestData req)
|
||||
{
|
||||
try
|
||||
{
|
||||
// Authentication + GlobalReadOrAdmin authorization enforced by PolicyEnforcementMiddleware
|
||||
var query = System.Web.HttpUtility.ParseQueryString(req.Url.Query);
|
||||
var tenantIdFilter = query["tenantId"];
|
||||
var days = VerdictCalibrationResponse.ClampDays(query["days"]);
|
||||
var partition = "global";
|
||||
if (!string.IsNullOrWhiteSpace(tenantIdFilter))
|
||||
{
|
||||
if (!Guid.TryParse(tenantIdFilter, out var parsed))
|
||||
{
|
||||
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
|
||||
await bad.WriteAsJsonAsync(new { success = false, message = "tenantId must be a GUID" });
|
||||
return bad;
|
||||
}
|
||||
partition = parsed.ToString("D");
|
||||
}
|
||||
|
||||
var today = DateTime.UtcNow.Date;
|
||||
// One read covers both the requested window and the trend horizon (7d window + 28d baseline).
|
||||
var readDays = Math.Max(days, VerdictCalibrationResponse.TrendHorizonDays);
|
||||
var daily = await _metricsRepo.GetVerdictCalibrationAggregatesAsync(
|
||||
partition, VerdictCalibrationResponse.InclusiveWindowStart(today, readDays), today);
|
||||
// Active drift episodes of this partition (tracker keyspace; fail-soft empty).
|
||||
var alerts = await _tracker.GetVerdictCalibrationAlertsAsync(partition);
|
||||
|
||||
var response = req.CreateResponse(HttpStatusCode.OK);
|
||||
await response.WriteAsJsonAsync(VerdictCalibrationResponse.Build(daily, partition, today, days, alerts));
|
||||
return response;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Error fetching verdict calibration");
|
||||
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
|
||||
await errorResponse.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
|
||||
return errorResponse;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Pure response builder — internal static so the matrix arithmetic is pinned by tests.</summary>
|
||||
internal static class VerdictCalibrationResponse
|
||||
{
|
||||
internal const int DefaultWindowDays = 30;
|
||||
internal const int MaxWindowDays = 180; // aggregate retention
|
||||
internal const int TrendWindowDays = 7;
|
||||
internal const int TrendBaselineDays = 28;
|
||||
internal const int TrendHorizonDays = TrendWindowDays + TrendBaselineDays;
|
||||
|
||||
internal static int ClampDays(string? raw)
|
||||
{
|
||||
if (!int.TryParse(raw, out var days)) return DefaultWindowDays;
|
||||
return Math.Clamp(days, 1, MaxWindowDays);
|
||||
}
|
||||
|
||||
/// <summary>"Last N days" = exactly N calendar day keys including today (both range ends inclusive).</summary>
|
||||
internal static DateTime InclusiveWindowStart(DateTime today, int days) => today.AddDays(-(days - 1));
|
||||
|
||||
internal sealed class PathRow
|
||||
{
|
||||
public string VerdictPath { get; set; } = string.Empty;
|
||||
public string Status { get; set; } = string.Empty;
|
||||
public int Count { get; set; }
|
||||
public double SharePct { get; set; }
|
||||
public int DerivedCount { get; set; }
|
||||
public int Eligible7d { get; set; }
|
||||
public int ReEnrolled7d { get; set; }
|
||||
/// <summary>Null below the minimum eligible count — never a rate on a handful of sessions.</summary>
|
||||
public double? ReEnrollRatePct { get; set; }
|
||||
public int OverriddenByAdmin { get; set; }
|
||||
public int OverriddenByLateCompletion { get; set; }
|
||||
public int OverriddenOther { get; set; }
|
||||
public TrendWindow Window7 { get; set; } = new();
|
||||
public TrendWindow Baseline28 { get; set; } = new();
|
||||
/// <summary>Window share ÷ baseline share; null when the baseline share is 0 (a new path has no finite lift — never invented).</summary>
|
||||
public double? Lift { get; set; }
|
||||
}
|
||||
|
||||
internal sealed class TrendWindow
|
||||
{
|
||||
public int Count { get; set; }
|
||||
public int Sessions { get; set; }
|
||||
public double SharePct { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>Minimum eligible sessions before a re-enrollment rate is stated (truthfulness: no rate on n<20).</summary>
|
||||
internal const int MinEligibleForRate = 20;
|
||||
|
||||
internal static object Build(
|
||||
IReadOnlyList<VerdictCalibrationDailyAggregate> daily, string partition, DateTime today, int days,
|
||||
IReadOnlyList<VerdictCalibrationAlert>? alerts = null)
|
||||
{
|
||||
var windowStart = Key(InclusiveWindowStart(today, days));
|
||||
var trendWindowStart = Key(InclusiveWindowStart(today, TrendWindowDays));
|
||||
var baselineStart = Key(InclusiveWindowStart(today, TrendHorizonDays));
|
||||
|
||||
var rows = new Dictionary<(string Path, string Status), PathRow>();
|
||||
int sessions = 0, terminal = 0, derived = 0;
|
||||
int trendSessions = 0, baselineSessions = 0;
|
||||
DateTime? computedAt = null;
|
||||
var versions = new HashSet<int>();
|
||||
|
||||
PathRow Row(VerdictCalibrationBucket b)
|
||||
{
|
||||
var key = (b.VerdictPath, b.Status);
|
||||
if (!rows.TryGetValue(key, out var row))
|
||||
{
|
||||
row = new PathRow { VerdictPath = b.VerdictPath, Status = b.Status };
|
||||
rows[key] = row;
|
||||
}
|
||||
return row;
|
||||
}
|
||||
|
||||
foreach (var day in daily)
|
||||
{
|
||||
var inWindow = string.CompareOrdinal(day.Date, windowStart) >= 0;
|
||||
var inTrendWindow = string.CompareOrdinal(day.Date, trendWindowStart) >= 0;
|
||||
var inBaseline = !inTrendWindow && string.CompareOrdinal(day.Date, baselineStart) >= 0;
|
||||
|
||||
if (inWindow)
|
||||
{
|
||||
sessions += day.SessionCount;
|
||||
terminal += day.TerminalSessionCount;
|
||||
versions.Add(day.Version);
|
||||
if (computedAt == null || day.ComputedAt > computedAt) computedAt = day.ComputedAt;
|
||||
}
|
||||
if (inTrendWindow) trendSessions += day.SessionCount;
|
||||
if (inBaseline) baselineSessions += day.SessionCount;
|
||||
|
||||
foreach (var b in day.Buckets)
|
||||
{
|
||||
// Overrides are attributed to the prior path and can exist on a bucket with Count=0.
|
||||
var row = Row(b);
|
||||
if (inWindow)
|
||||
{
|
||||
row.Count += b.Count;
|
||||
row.DerivedCount += b.DerivedCount;
|
||||
row.Eligible7d += b.Eligible7d;
|
||||
row.ReEnrolled7d += b.ReEnrolled7d;
|
||||
row.OverriddenByAdmin += b.OverriddenByAdmin;
|
||||
row.OverriddenByLateCompletion += b.OverriddenByLateCompletion;
|
||||
row.OverriddenOther += b.OverriddenOther;
|
||||
derived += b.DerivedCount;
|
||||
}
|
||||
if (inTrendWindow) row.Window7.Count += b.Count;
|
||||
if (inBaseline) row.Baseline28.Count += b.Count;
|
||||
}
|
||||
}
|
||||
|
||||
var list = rows.Values
|
||||
.Where(r => r.Count > 0 || r.OverriddenByAdmin + r.OverriddenByLateCompletion + r.OverriddenOther > 0)
|
||||
.OrderByDescending(r => r.Count)
|
||||
.ThenBy(r => r.VerdictPath, StringComparer.Ordinal)
|
||||
.ThenBy(r => r.Status, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
|
||||
foreach (var r in list)
|
||||
{
|
||||
r.SharePct = Pct(r.Count, sessions);
|
||||
r.ReEnrollRatePct = r.Eligible7d >= MinEligibleForRate ? Pct(r.ReEnrolled7d, r.Eligible7d) : null;
|
||||
r.Window7.Sessions = trendSessions;
|
||||
r.Window7.SharePct = Pct(r.Window7.Count, trendSessions);
|
||||
r.Baseline28.Sessions = baselineSessions;
|
||||
r.Baseline28.SharePct = Pct(r.Baseline28.Count, baselineSessions);
|
||||
r.Lift = r.Baseline28.SharePct > 0 && trendSessions > 0
|
||||
? Math.Round(r.Window7.SharePct / r.Baseline28.SharePct, 2)
|
||||
: null;
|
||||
}
|
||||
|
||||
return new
|
||||
{
|
||||
success = true,
|
||||
tenantId = partition,
|
||||
windowDays = days,
|
||||
windowStart,
|
||||
windowEnd = Key(today),
|
||||
computedAt,
|
||||
versions = versions.OrderBy(v => v).ToArray(),
|
||||
totals = new { sessions, terminal, derived, days = daily.Count(d => string.CompareOrdinal(d.Date, windowStart) >= 0) },
|
||||
trend = new { windowDays = TrendWindowDays, baselineDays = TrendBaselineDays, windowSessions = trendSessions, baselineSessions },
|
||||
paths = list,
|
||||
// Wording contract shared with the rule radar: a dimension is correlated, not causal.
|
||||
alerts = (alerts ?? Array.Empty<VerdictCalibrationAlert>())
|
||||
.OrderByDescending(a => a.FirstNotifiedAt)
|
||||
.ToList(),
|
||||
};
|
||||
}
|
||||
|
||||
private static string Key(DateTime d) => d.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture);
|
||||
|
||||
private static double Pct(int part, int whole) => whole <= 0 ? 0 : Math.Round(100.0 * part / whole, 1);
|
||||
}
|
||||
}
|
||||
@@ -19,20 +19,20 @@ public class PreviewWhitelistFunction
|
||||
private readonly PreviewWhitelistService _previewWhitelistService;
|
||||
private readonly TenantApprovalService _tenantApprovalService;
|
||||
private readonly TenantConfigurationService _tenantConfigurationService;
|
||||
private readonly ResendEmailService _resendEmailService;
|
||||
private readonly IEmailService _emailService;
|
||||
|
||||
public PreviewWhitelistFunction(
|
||||
ILogger<PreviewWhitelistFunction> logger,
|
||||
PreviewWhitelistService previewWhitelistService,
|
||||
TenantApprovalService tenantApprovalService,
|
||||
TenantConfigurationService tenantConfigurationService,
|
||||
ResendEmailService resendEmailService)
|
||||
IEmailService emailService)
|
||||
{
|
||||
_logger = logger;
|
||||
_previewWhitelistService = previewWhitelistService;
|
||||
_tenantApprovalService = tenantApprovalService;
|
||||
_tenantConfigurationService = tenantConfigurationService;
|
||||
_resendEmailService = resendEmailService;
|
||||
_emailService = emailService;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -221,7 +221,7 @@ public class PreviewWhitelistFunction
|
||||
}
|
||||
|
||||
var tenantConfig = await _tenantConfigurationService.GetConfigurationAsync(tenantId);
|
||||
await _resendEmailService.SendPreviewApprovedEmailAsync(email, tenantConfig.DomainName);
|
||||
await _emailService.SendPreviewApprovedEmailAsync(email, tenantConfig.DomainName);
|
||||
|
||||
// Explicit GA send always sends; consume the once-only marker (best-effort) so
|
||||
// the automatic paths won't produce a duplicate afterwards.
|
||||
|
||||
@@ -49,6 +49,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
|
||||
tenantId,
|
||||
sessionId,
|
||||
SessionStatus.Failed,
|
||||
VerdictPaths.ManualFailed,
|
||||
currentPhase: null, // Keep current phase
|
||||
failureReason: "Manually marked as failed by administrator",
|
||||
failureSource: "manual",
|
||||
|
||||
+1
@@ -49,6 +49,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
|
||||
tenantId,
|
||||
sessionId,
|
||||
SessionStatus.Succeeded,
|
||||
VerdictPaths.ManualSucceeded,
|
||||
currentPhase: null, // Keep current phase
|
||||
failureReason: "Manually marked as succeeded by administrator",
|
||||
adminMarkedAction: "Succeeded"
|
||||
|
||||
+1
-1
@@ -305,7 +305,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
|
||||
$"enrollment session at {registration.StartedAt:yyyy-MM-dd HH:mm} UTC before this one " +
|
||||
"reached a terminal state.";
|
||||
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
registration.TenantId, stale.SessionId, SessionStatus.Incomplete,
|
||||
registration.TenantId, stale.SessionId, SessionStatus.Incomplete, VerdictPaths.RegisterSuperseded,
|
||||
failureReason: reason, failureSource: "superseded_by_reregistration");
|
||||
if (!transitioned)
|
||||
continue;
|
||||
|
||||
@@ -182,6 +182,35 @@ namespace AutopilotMonitor.Functions.Helpers
|
||||
var headerError = ValidateWebhookCustomHeaders(channel.CustomHeadersJson);
|
||||
if (headerError != null)
|
||||
return $"channel \"{label}\" headers: {headerError}";
|
||||
|
||||
var secretError = ValidateWebhookSigningSecret(channel.SigningSecret);
|
||||
if (secretError != null)
|
||||
return $"channel \"{label}\" signing secret: {secretError}";
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Validates a generic-webhook HMAC signing secret. Returns an error message, or null when
|
||||
/// valid/absent. A short key is brute-forceable as an HMAC key; the printable-ASCII rule
|
||||
/// keeps the secret copy-paste safe and (as a side effect) rejects a stray unrestored
|
||||
/// redaction sentinel, which is shorter than the minimum. No provider gate here — a secret
|
||||
/// on a non-generic channel is stored but inert at dispatch (GetSigningSecret), matching
|
||||
/// how CustomHeadersJson behaves.
|
||||
/// </summary>
|
||||
internal static string? ValidateWebhookSigningSecret(string? secret)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(secret))
|
||||
return null;
|
||||
|
||||
if (secret.Length < 16 || secret.Length > 128)
|
||||
return "must be 16-128 characters.";
|
||||
|
||||
foreach (var ch in secret)
|
||||
{
|
||||
if (ch < 0x21 || ch > 0x7E)
|
||||
return "must be printable ASCII without spaces or line breaks.";
|
||||
}
|
||||
|
||||
return null;
|
||||
|
||||
@@ -0,0 +1,281 @@
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Helpers;
|
||||
|
||||
/// <summary>
|
||||
/// One drift finding of the verdict-calibration radar (pure output; the maintenance partial
|
||||
/// turns it into a tracker episode + ops event).
|
||||
/// </summary>
|
||||
public class VerdictCalibrationFinding
|
||||
{
|
||||
public string Kind { get; set; } = string.Empty;
|
||||
public string VerdictPath { get; set; } = string.Empty;
|
||||
public string Status { get; set; } = string.Empty;
|
||||
public int WindowHitCount { get; set; }
|
||||
public int WindowSessionCount { get; set; }
|
||||
public int BaselineHitCount { get; set; }
|
||||
public int BaselineSessionCount { get; set; }
|
||||
public double WindowRatePct { get; set; }
|
||||
public double BaselineRatePct { get; set; }
|
||||
public double? Lift { get; set; }
|
||||
public string WindowStartDate { get; set; } = string.Empty;
|
||||
public string WindowEndDate { get; set; } = string.Empty;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Pure drift detection over one tenant's verdict-calibration daily rows
|
||||
/// (docs/backend/verdict-calibration.md) — the same statistics as the rule-regression radar
|
||||
/// (7-day window vs the prior 28 days, lift ≥2×, Wilson-separated via
|
||||
/// <see cref="MetricsMath.RateIncreaseSeparated"/>, small-n gates) applied to three questions:
|
||||
/// <list type="bullet">
|
||||
/// <item><b>share_regression</b> — one verdict path's share of all sessions doubled. Points at a rule
|
||||
/// (or an agent path) whose conditions suddenly match much more often.</item>
|
||||
/// <item><b>silence_share_regression</b> — the share of sessions the backend had to decide because
|
||||
/// the agent went silent (sweep:* + maxlife:*) doubled. An agent-liveness signal, not a classifier one.</item>
|
||||
/// <item><b>evidence_gap</b> — absolute gate: the pure fallthrough rule (r6) decides ≥20 % of the
|
||||
/// window's classifier verdicts. The evidence the other rules need is not arriving.</item>
|
||||
/// </list>
|
||||
/// Derived (<c>legacy:*</c>) paths are excluded from all three — pre-instrumentation attribution
|
||||
/// is too weak to alert on. Agent-declared paths (<c>agent:*</c>, <c>register:*</c>) are excluded
|
||||
/// from the per-path share regression: their share tracks the customer's workflow mix (a
|
||||
/// WhiteGlove rollout week doubles <c>agent:whiteglove_pending</c>; a Continue-Anyway tenant
|
||||
/// lives on <c>agent:complete_soft</c>) — real signals, but not about the BACKEND classifier
|
||||
/// this radar calibrates. First production pass (2026-08-23) fired exactly those three. Re-arm
|
||||
/// mirrors the rule radar: share back under 1.5× baseline or the path stops occurring; the
|
||||
/// evidence gap re-arms under 15 %.
|
||||
/// </summary>
|
||||
public static class VerdictCalibrationRadar
|
||||
{
|
||||
public const int WindowDays = RuleRegressionRadar.WindowDays;
|
||||
public const int BaselineDays = RuleRegressionRadar.BaselineDays;
|
||||
public const int MinWindowHits = RuleRegressionRadar.MinWindowHitSessions;
|
||||
public const int MinWindowSessions = RuleRegressionRadar.MinWindowSessions;
|
||||
public const double MinLift = RuleRegressionRadar.MinLift;
|
||||
public const double ReArmLiftFactor = RuleRegressionRadar.ReArmLiftFactor;
|
||||
|
||||
/// <summary>Evidence gap fires at ≥20 % r6 share of the window's classifier verdicts (≥20 verdicts) and re-arms under 15 %.</summary>
|
||||
public const double EvidenceGapFirePct = 20.0;
|
||||
public const double EvidenceGapReArmPct = 15.0;
|
||||
public const int EvidenceGapMinVerdicts = 20;
|
||||
|
||||
public const string SilenceGroupPath = "sweep+maxlife";
|
||||
public const string EvidenceGapPath = "r6";
|
||||
public const string GroupStatus = "*";
|
||||
|
||||
private const string LegacyOrigin = "legacy";
|
||||
|
||||
/// <summary>
|
||||
/// True when a path is eligible for the per-path share regression: backend-decided paths only
|
||||
/// (sweep / maxlife / late / retro / rule / manual / ingest). Agent-declared and registration
|
||||
/// paths mirror customer workflow, legacy paths are derived.
|
||||
/// </summary>
|
||||
public static bool IsShareRegressionEligible(string path)
|
||||
{
|
||||
var origin = VerdictPaths.Origin(path);
|
||||
return origin != LegacyOrigin
|
||||
&& origin != "agent"
|
||||
&& origin != "register";
|
||||
}
|
||||
|
||||
public static List<VerdictCalibrationFinding> Evaluate(IReadOnlyList<VerdictCalibrationDailyAggregate> tenantRows, DateTime targetDateUtc)
|
||||
{
|
||||
var findings = new List<VerdictCalibrationFinding>();
|
||||
var horizon = Summarize(tenantRows, targetDateUtc);
|
||||
var windowStart = targetDateUtc.Date.AddDays(-(WindowDays - 1)).ToString("yyyy-MM-dd");
|
||||
var windowEnd = targetDateUtc.Date.ToString("yyyy-MM-dd");
|
||||
|
||||
// 1. Per-path share regression.
|
||||
foreach (var pair in horizon.Paths)
|
||||
{
|
||||
if (!IsShareRegressionEligible(pair.Key.Path)) continue;
|
||||
var f = TryShareRegression(VerdictCalibrationAlertKinds.ShareRegression, pair.Key.Path, pair.Key.Status,
|
||||
pair.Value.WindowHits, horizon.WindowSessions, pair.Value.BaselineHits, horizon.BaselineSessions, windowStart, windowEnd);
|
||||
if (f != null) findings.Add(f);
|
||||
}
|
||||
|
||||
// 2. Silence-share regression (group).
|
||||
{
|
||||
var f = TryShareRegression(VerdictCalibrationAlertKinds.SilenceShareRegression, SilenceGroupPath, GroupStatus,
|
||||
horizon.SilenceWindowHits, horizon.WindowSessions, horizon.SilenceBaselineHits, horizon.BaselineSessions, windowStart, windowEnd);
|
||||
if (f != null) findings.Add(f);
|
||||
}
|
||||
|
||||
// 3. Evidence gap (absolute).
|
||||
if (horizon.ClassifierWindowVerdicts >= EvidenceGapMinVerdicts)
|
||||
{
|
||||
var pct = 100.0 * horizon.R6WindowHits / horizon.ClassifierWindowVerdicts;
|
||||
if (pct >= EvidenceGapFirePct)
|
||||
{
|
||||
findings.Add(new VerdictCalibrationFinding
|
||||
{
|
||||
Kind = VerdictCalibrationAlertKinds.EvidenceGap,
|
||||
VerdictPath = EvidenceGapPath,
|
||||
Status = GroupStatus,
|
||||
WindowHitCount = horizon.R6WindowHits,
|
||||
WindowSessionCount = horizon.ClassifierWindowVerdicts,
|
||||
BaselineHitCount = horizon.R6BaselineHits,
|
||||
BaselineSessionCount = horizon.ClassifierBaselineVerdicts,
|
||||
WindowRatePct = Math.Round(pct, 1),
|
||||
BaselineRatePct = horizon.ClassifierBaselineVerdicts > 0
|
||||
? Math.Round(100.0 * horizon.R6BaselineHits / horizon.ClassifierBaselineVerdicts, 1) : 0,
|
||||
Lift = null,
|
||||
WindowStartDate = windowStart,
|
||||
WindowEndDate = windowEnd,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
return findings
|
||||
.OrderByDescending(f => f.WindowRatePct)
|
||||
.ThenBy(f => f.Kind, StringComparer.Ordinal)
|
||||
.ThenBy(f => f.VerdictPath, StringComparer.Ordinal)
|
||||
.ToList();
|
||||
}
|
||||
|
||||
/// <summary>True when an ACTIVE alert may re-arm given the current horizon sums (or when its path is no longer eligible — episodes from an earlier, broader gate clear themselves).</summary>
|
||||
public static bool ShouldReArm(VerdictCalibrationAlert alert, IReadOnlyList<VerdictCalibrationDailyAggregate> tenantRows, DateTime targetDateUtc)
|
||||
{
|
||||
if (alert.Kind == VerdictCalibrationAlertKinds.ShareRegression && !IsShareRegressionEligible(alert.VerdictPath))
|
||||
return true;
|
||||
|
||||
var horizon = Summarize(tenantRows, targetDateUtc);
|
||||
switch (alert.Kind)
|
||||
{
|
||||
case VerdictCalibrationAlertKinds.EvidenceGap:
|
||||
if (horizon.ClassifierWindowVerdicts < EvidenceGapMinVerdicts) return true;
|
||||
return 100.0 * horizon.R6WindowHits / horizon.ClassifierWindowVerdicts < EvidenceGapReArmPct;
|
||||
|
||||
case VerdictCalibrationAlertKinds.SilenceShareRegression:
|
||||
return ReArmByRate(horizon.SilenceWindowHits, horizon.WindowSessions, horizon.SilenceBaselineHits, horizon.BaselineSessions);
|
||||
|
||||
default:
|
||||
horizon.Paths.TryGetValue((alert.VerdictPath, alert.Status), out var sums);
|
||||
return ReArmByRate(sums.WindowHits, horizon.WindowSessions, sums.BaselineHits, horizon.BaselineSessions);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Current numbers for an ongoing episode (refresh without re-firing).</summary>
|
||||
public static (int WindowHits, int WindowSessions, int BaselineHits, int BaselineSessions) CurrentSums(
|
||||
VerdictCalibrationAlert alert, IReadOnlyList<VerdictCalibrationDailyAggregate> tenantRows, DateTime targetDateUtc)
|
||||
{
|
||||
var horizon = Summarize(tenantRows, targetDateUtc);
|
||||
switch (alert.Kind)
|
||||
{
|
||||
case VerdictCalibrationAlertKinds.EvidenceGap:
|
||||
return (horizon.R6WindowHits, horizon.ClassifierWindowVerdicts, horizon.R6BaselineHits, horizon.ClassifierBaselineVerdicts);
|
||||
case VerdictCalibrationAlertKinds.SilenceShareRegression:
|
||||
return (horizon.SilenceWindowHits, horizon.WindowSessions, horizon.SilenceBaselineHits, horizon.BaselineSessions);
|
||||
default:
|
||||
horizon.Paths.TryGetValue((alert.VerdictPath, alert.Status), out var sums);
|
||||
return (sums.WindowHits, horizon.WindowSessions, sums.BaselineHits, horizon.BaselineSessions);
|
||||
}
|
||||
}
|
||||
|
||||
private static bool ReArmByRate(int windowHits, int windowSessions, int baselineHits, int baselineSessions)
|
||||
{
|
||||
if (windowHits == 0 || windowSessions == 0) return true;
|
||||
if (baselineSessions == 0 || baselineHits == 0) return false; // zero-baseline episode re-arms only via hits-stopped
|
||||
var windowRate = (double)windowHits / windowSessions;
|
||||
var baselineRate = (double)baselineHits / baselineSessions;
|
||||
return windowRate < ReArmLiftFactor * baselineRate;
|
||||
}
|
||||
|
||||
private static VerdictCalibrationFinding? TryShareRegression(string kind, string path, string status,
|
||||
int windowHits, int windowSessions, int baselineHits, int baselineSessions, string windowStart, string windowEnd)
|
||||
{
|
||||
if (windowHits < MinWindowHits) return null;
|
||||
if (windowSessions < MinWindowSessions) return null;
|
||||
if (baselineSessions <= 0) return null;
|
||||
|
||||
var windowRate = (double)windowHits / windowSessions;
|
||||
var baselineRate = (double)baselineHits / baselineSessions;
|
||||
if (windowRate < MinLift * baselineRate) return null;
|
||||
if (!MetricsMath.RateIncreaseSeparated(windowHits, windowSessions, baselineHits, baselineSessions)) return null;
|
||||
|
||||
return new VerdictCalibrationFinding
|
||||
{
|
||||
Kind = kind,
|
||||
VerdictPath = path,
|
||||
Status = status,
|
||||
WindowHitCount = windowHits,
|
||||
WindowSessionCount = windowSessions,
|
||||
BaselineHitCount = baselineHits,
|
||||
BaselineSessionCount = baselineSessions,
|
||||
WindowRatePct = Math.Round(windowRate * 100, 1),
|
||||
BaselineRatePct = Math.Round(baselineRate * 100, 1),
|
||||
Lift = baselineRate > 0 ? Math.Round(windowRate / baselineRate, 1) : null,
|
||||
WindowStartDate = windowStart,
|
||||
WindowEndDate = windowEnd,
|
||||
};
|
||||
}
|
||||
|
||||
internal readonly record struct PathSums(int WindowHits, int BaselineHits);
|
||||
|
||||
internal sealed class Horizon
|
||||
{
|
||||
public int WindowSessions;
|
||||
public int BaselineSessions;
|
||||
public int SilenceWindowHits;
|
||||
public int SilenceBaselineHits;
|
||||
public int R6WindowHits;
|
||||
public int R6BaselineHits;
|
||||
public int ClassifierWindowVerdicts;
|
||||
public int ClassifierBaselineVerdicts;
|
||||
public readonly Dictionary<(string Path, string Status), PathSums> Paths = new();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Sums the daily rows into the trailing window [target−6d, target] and the prior baseline
|
||||
/// [target−34d, target−7d] (ISO-string date compare, rows outside the horizon ignored).
|
||||
/// Silence group = sweep:* + maxlife:* (incl. non-rule sweep paths like sweep:stalled — the
|
||||
/// question is "how often did the backend have to decide", not which rule did). Classifier
|
||||
/// verdicts = stamped rule paths (<see cref="VerdictPaths.IsClassifierPath"/>); r6 = their
|
||||
/// fallthrough subset.
|
||||
/// </summary>
|
||||
internal static Horizon Summarize(IReadOnlyList<VerdictCalibrationDailyAggregate> rows, DateTime targetDateUtc)
|
||||
{
|
||||
var windowStart = targetDateUtc.Date.AddDays(-(WindowDays - 1)).ToString("yyyy-MM-dd");
|
||||
var baselineStart = targetDateUtc.Date.AddDays(-(WindowDays - 1 + BaselineDays)).ToString("yyyy-MM-dd");
|
||||
var target = targetDateUtc.Date.ToString("yyyy-MM-dd");
|
||||
|
||||
var h = new Horizon();
|
||||
foreach (var row in rows)
|
||||
{
|
||||
if (string.CompareOrdinal(row.Date, target) > 0) continue;
|
||||
bool inWindow;
|
||||
if (string.CompareOrdinal(row.Date, windowStart) >= 0) inWindow = true;
|
||||
else if (string.CompareOrdinal(row.Date, baselineStart) >= 0) inWindow = false;
|
||||
else continue;
|
||||
|
||||
if (inWindow) h.WindowSessions += row.SessionCount; else h.BaselineSessions += row.SessionCount;
|
||||
|
||||
foreach (var b in row.Buckets)
|
||||
{
|
||||
if (b.Count == 0) continue;
|
||||
var key = (b.VerdictPath, b.Status);
|
||||
h.Paths.TryGetValue(key, out var sums);
|
||||
h.Paths[key] = inWindow
|
||||
? new PathSums(sums.WindowHits + b.Count, sums.BaselineHits)
|
||||
: new PathSums(sums.WindowHits, sums.BaselineHits + b.Count);
|
||||
|
||||
var origin = VerdictPaths.Origin(b.VerdictPath);
|
||||
if (origin == VerdictPaths.OriginSweep || origin == VerdictPaths.OriginMaxLifetime)
|
||||
{
|
||||
if (inWindow) h.SilenceWindowHits += b.Count; else h.SilenceBaselineHits += b.Count;
|
||||
}
|
||||
if (VerdictPaths.IsClassifierPath(b.VerdictPath))
|
||||
{
|
||||
if (inWindow) h.ClassifierWindowVerdicts += b.Count; else h.ClassifierBaselineVerdicts += b.Count;
|
||||
if (b.VerdictPath.EndsWith(":" + ClassifierRules.R6Fallthrough, StringComparison.Ordinal))
|
||||
{
|
||||
if (inWindow) h.R6WindowHits += b.Count; else h.R6BaselineHits += b.Count;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return h;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
using System;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Helpers
|
||||
{
|
||||
/// <summary>
|
||||
/// Read-side attribution for sessions written BEFORE <see cref="SessionSummary.VerdictPath"/>
|
||||
/// was stamped. No mass backfill: the calibration aggregate calls <see cref="Derive"/> and
|
||||
/// counts the result with <c>Derived=true</c>, so pre-instrumentation history still shows up
|
||||
/// in the matrix and is visibly weaker. The derivation reads the fixed reason literals the
|
||||
/// writers emit (classifier rule prefixes, the max-lifetime / retro suffixes, the manual and
|
||||
/// rule attributions) — when the origin is ambiguous but the rule is known the path is
|
||||
/// <c>legacy:{rule}</c>; when nothing is attributable it is <see cref="VerdictPaths.LegacyUnknown"/>.
|
||||
/// A stamped row is returned verbatim with <c>Derived=false</c>.
|
||||
/// </summary>
|
||||
public static class VerdictPathDerivation
|
||||
{
|
||||
private const string OriginLegacy = "legacy";
|
||||
|
||||
// Literals are owned by EnrollmentTimeoutClassifier / the writers; pinned by
|
||||
// VerdictPathDerivationTests so a wording change there fails here, not silently.
|
||||
private const string R1Prefix = "Enrollment reported an explicit failure before timeout";
|
||||
private const string R1bAwaitingPrefix = "Pre-provisioning (WhiteGlove) completed; no user sign-in since Part 2 resumed";
|
||||
private const string R1bSucceededPrefix = "Reconciled at timeout: pre-provisioning (WhiteGlove Part 1) completed";
|
||||
private const string R1cPrefix = "Reconciled: self-deploying profile";
|
||||
private const string R2Prefix = "Reconciled at timeout: Account Setup completed";
|
||||
private const string R3Prefix = "Agent emergency break fired";
|
||||
private const string R4Prefix = "Reconciled at timeout: user completed setup";
|
||||
private const string R5AwaitingPrefix = "Device Setup completed; awaiting user / Account Setup phase";
|
||||
private const string R5IncompletePrefix = "No completion signal within";
|
||||
private const string R6Prefix = "No Device Setup completion or explicit failure signal observed before timeout";
|
||||
private const string SupersededPrefix = "Superseded by session";
|
||||
private const string SweepStalledPrefix = "Agent silent for";
|
||||
private const string AgentStallProbePrefix = "Agent reported stall after";
|
||||
private const string EspFallbackLiteral = "ESP failure (backend fallback)";
|
||||
/// <summary>Pre-classifier blanket timeout verdict (LegacyReclassificationService) — never agent-reported.</summary>
|
||||
private const string LegacyBlanketTimeoutPrefix = "Session timed out";
|
||||
private const string MaxLifetimeSuffix = "max-lifetime watchdog";
|
||||
private const string RetroSuffix = "Retro-reclassified";
|
||||
|
||||
public static (string Path, bool Derived) Derive(SessionSummary s)
|
||||
{
|
||||
if (s == null) throw new ArgumentNullException(nameof(s));
|
||||
if (!string.IsNullOrWhiteSpace(s.VerdictPath))
|
||||
return (s.VerdictPath!, false);
|
||||
|
||||
return (DeriveLegacy(s), true);
|
||||
}
|
||||
|
||||
private static string DeriveLegacy(SessionSummary s)
|
||||
{
|
||||
var failureReason = s.FailureReason ?? string.Empty;
|
||||
var reconcileReason = s.ReconcileReason ?? string.Empty;
|
||||
var failureSource = s.FailureSource ?? string.Empty;
|
||||
|
||||
// Unambiguous attributions first — independent of status text.
|
||||
if (string.Equals(s.AdminMarkedAction, "Succeeded", StringComparison.OrdinalIgnoreCase))
|
||||
return VerdictPaths.ManualSucceeded;
|
||||
if (string.Equals(s.AdminMarkedAction, "Failed", StringComparison.OrdinalIgnoreCase))
|
||||
return VerdictPaths.ManualFailed;
|
||||
if (failureSource.StartsWith("rule:", StringComparison.OrdinalIgnoreCase))
|
||||
return VerdictPaths.Compose(VerdictPaths.OriginRule, failureSource.Substring(5));
|
||||
|
||||
switch (s.Status)
|
||||
{
|
||||
case SessionStatus.Succeeded:
|
||||
if (reconcileReason.Length == 0
|
||||
|| reconcileReason.StartsWith("Late completion report received", StringComparison.Ordinal))
|
||||
return s.EspSoftFailure ? VerdictPaths.AgentCompleteSoft : VerdictPaths.AgentComplete;
|
||||
if (Starts(reconcileReason, R1bSucceededPrefix)) return Classifier(reconcileReason, ClassifierRules.R1bWhiteGloveSucceeded);
|
||||
if (Starts(reconcileReason, R1cPrefix)) return Classifier(reconcileReason, ClassifierRules.R1cSelfDeploying);
|
||||
if (Starts(reconcileReason, R2Prefix)) return Classifier(reconcileReason, ClassifierRules.R2AccountSetupComplete);
|
||||
if (Starts(reconcileReason, R4Prefix)) return Classifier(reconcileReason, ClassifierRules.R4DesktopHello);
|
||||
return VerdictPaths.LegacyUnknown;
|
||||
|
||||
case SessionStatus.Failed:
|
||||
if (Starts(failureReason, R1Prefix)) return Classifier(failureReason, ClassifierRules.R1ExplicitFailure);
|
||||
if (string.Equals(failureSource, "max_lifetime_watchdog", StringComparison.OrdinalIgnoreCase))
|
||||
return VerdictPaths.Compose(VerdictPaths.OriginMaxLifetime, ClassifierRules.R1ExplicitFailure);
|
||||
if (failureReason.Contains(EspFallbackLiteral, StringComparison.Ordinal)) return VerdictPaths.AgentEspFailureFallback;
|
||||
if (Starts(failureReason, LegacyBlanketTimeoutPrefix)) return VerdictPaths.LegacyUnknown;
|
||||
if (failureReason.Length > 0 && failureSource.Length == 0) return VerdictPaths.AgentFailed;
|
||||
return VerdictPaths.LegacyUnknown;
|
||||
|
||||
case SessionStatus.Incomplete:
|
||||
if (Starts(failureReason, R3Prefix)) return Classifier(failureReason, ClassifierRules.R3EmergencyBreak);
|
||||
if (Starts(failureReason, R5IncompletePrefix)) return Classifier(failureReason, ClassifierRules.R5DeviceSetupIncomplete);
|
||||
if (Starts(failureReason, R6Prefix)) return Classifier(failureReason, ClassifierRules.R6Fallthrough);
|
||||
if (Starts(failureReason, SupersededPrefix)) return VerdictPaths.Compose(OriginLegacy, "superseded");
|
||||
return VerdictPaths.LegacyUnknown;
|
||||
|
||||
case SessionStatus.AwaitingUser:
|
||||
if (Starts(failureReason, R1bAwaitingPrefix)) return VerdictPaths.Compose(OriginLegacy, "wg_awaiting");
|
||||
if (Starts(failureReason, R5AwaitingPrefix)) return Classifier(failureReason, ClassifierRules.R5DeviceSetupAwaiting);
|
||||
return VerdictPaths.LegacyUnknown;
|
||||
|
||||
case SessionStatus.Stalled:
|
||||
if (Starts(failureReason, SweepStalledPrefix)) return VerdictPaths.SweepStalled;
|
||||
if (Starts(failureReason, AgentStallProbePrefix)) return VerdictPaths.AgentStallProbe;
|
||||
return VerdictPaths.LegacyUnknown;
|
||||
|
||||
case SessionStatus.Pending:
|
||||
return VerdictPaths.AgentWhiteGlovePending;
|
||||
|
||||
default:
|
||||
return VerdictPaths.LegacyUnknown;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Classifier verdict with a known rule: the origin is recoverable only from the suffixes
|
||||
/// the max-lifetime and retro writers append; sweep / late-reconcile share the bare text
|
||||
/// and stay <c>legacy:{rule}</c>.
|
||||
/// </summary>
|
||||
private static string Classifier(string reason, string rule)
|
||||
{
|
||||
if (reason.Contains(MaxLifetimeSuffix, StringComparison.Ordinal))
|
||||
return VerdictPaths.Compose(VerdictPaths.OriginMaxLifetime, rule);
|
||||
if (reason.Contains(RetroSuffix, StringComparison.Ordinal))
|
||||
return VerdictPaths.Compose(VerdictPaths.OriginRetro, rule);
|
||||
return VerdictPaths.Compose(OriginLegacy, rule);
|
||||
}
|
||||
|
||||
private static bool Starts(string text, string prefix)
|
||||
=> text.StartsWith(prefix, StringComparison.Ordinal);
|
||||
}
|
||||
}
|
||||
@@ -226,6 +226,7 @@ builder.Services.AddHostedService<AutopilotMonitor.Functions.Services.Offboardin
|
||||
// no AddHostedService call needed. DI for its dependencies comes from the singletons above.
|
||||
|
||||
builder.Services.AddHostedService<TableInitializerService>(); // Initialize all tables at startup
|
||||
builder.Services.AddHostedService<StartupTelemetryService>(); // BackendStartupMs / BackendTableInitMs metrics on ApplicationStarted
|
||||
|
||||
// Data Access Layer — repository interfaces backed by Table Storage.
|
||||
// To switch to Cosmos DB: replace AddTableStorageDataAccess() with AddCosmosDataAccess().
|
||||
@@ -437,8 +438,15 @@ builder.Services.AddHttpClient<AutopilotMonitor.Functions.Services.Notifications
|
||||
.AddPolicyHandler((sp, _) => sp.GetRequiredService<ResiliencePolicies>().Notification);
|
||||
builder.Services.AddHttpClient<TelegramNotificationService>()
|
||||
.AddPolicyHandler((sp, _) => sp.GetRequiredService<ResiliencePolicies>().Notification);
|
||||
builder.Services.AddSingleton<ResendEmailService>();
|
||||
builder.Services.AddSingleton<IOffboardFarewellEmailSender>(sp => sp.GetRequiredService<ResendEmailService>());
|
||||
// Transactional email (welcome / farewell). Typed client like the other outbound notifiers;
|
||||
// the provider is an implementation detail of EmailService + the Email:* settings.
|
||||
builder.Services.AddSingleton<EmailTemplateService>();
|
||||
builder.Services.AddSingleton<IEmailTemplateProvider>(sp => sp.GetRequiredService<EmailTemplateService>());
|
||||
builder.Services.AddHttpClient<EmailService>()
|
||||
.ConfigureHttpClient(client => client.Timeout = TimeSpan.FromSeconds(15))
|
||||
.AddPolicyHandler((sp, _) => sp.GetRequiredService<ResiliencePolicies>().Notification);
|
||||
builder.Services.AddTransient<IEmailService>(sp => sp.GetRequiredService<EmailService>());
|
||||
builder.Services.AddTransient<IOffboardFarewellEmailSender>(sp => sp.GetRequiredService<EmailService>());
|
||||
builder.Services.AddSingleton<GlobalNotificationService>();
|
||||
builder.Services.AddSingleton<TenantNotificationService>();
|
||||
|
||||
|
||||
@@ -282,6 +282,9 @@ public static class EndpointAccessPolicyCatalog
|
||||
new("GET", "audit/tpm-pss-unsupported", EndpointPolicy.MemberRead),
|
||||
new("GET", "diagnostics/download-url", EndpointPolicy.MemberRead, TenantScoping.QueryParam),
|
||||
new("POST", "diagnostics/download-ticket", EndpointPolicy.MemberRead, TenantScoping.QueryParam),
|
||||
// Platform-defined diagnostics collection (built-in catalog + global paths): read-only,
|
||||
// no secrets, needed by every settings viewer — member tier, JWT-scoped (no tenant parameter).
|
||||
new("GET", "diagnostics/paths", EndpointPolicy.MemberRead, TenantScoping.Jwt),
|
||||
new("GET", "rules/gather", EndpointPolicy.MemberRead),
|
||||
new("GET", "rules/analyze", EndpointPolicy.MemberRead),
|
||||
new("GET", "rules/ime-log-patterns", EndpointPolicy.MemberRead),
|
||||
@@ -461,6 +464,7 @@ public static class EndpointAccessPolicyCatalog
|
||||
new("GET", "global/metrics/fleet-health", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
|
||||
new("GET", "global/metrics/time-attribution", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
|
||||
new("GET", "global/metrics/device-journeys", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
|
||||
new("GET", "global/metrics/verdict-calibration", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
|
||||
new("GET", "global/apps/list", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
|
||||
new("GET", "global/apps/{appName}/analytics", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
|
||||
new("GET", "global/apps/{appName}/sessions", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
|
||||
@@ -497,6 +501,12 @@ public static class EndpointAccessPolicyCatalog
|
||||
new("POST", "versions/block", EndpointPolicy.GlobalAdminOnly),
|
||||
new("DELETE", "versions/block/{encodedPattern}", EndpointPolicy.GlobalAdminOnly),
|
||||
new("POST", "maintenance/trigger", EndpointPolicy.GlobalAdminOnly),
|
||||
// Transactional email templates (welcome / farewell): read, override, reset, test send.
|
||||
// Read stays GA-only too — the override HTML is operator content, not tenant data.
|
||||
new("GET", "global/email-templates/{kind}", EndpointPolicy.GlobalAdminOnly),
|
||||
new("PUT", "global/email-templates/{kind}", EndpointPolicy.GlobalAdminOnly),
|
||||
new("DELETE", "global/email-templates/{kind}", EndpointPolicy.GlobalAdminOnly),
|
||||
new("POST", "global/email-templates/{kind}/test", EndpointPolicy.GlobalAdminOnly),
|
||||
// One-time retro-reconcile of misclassified historical sessions (audit 2026-07-16) —
|
||||
// mutating, dry-run by default, GA only.
|
||||
new("POST", "maintenance/reclassify-legacy", EndpointPolicy.GlobalAdminOnly),
|
||||
|
||||
@@ -95,6 +95,16 @@ namespace AutopilotMonitor.Functions.Security
|
||||
/// </summary>
|
||||
public const string LegacyEnterpriseTierName = "enterprise";
|
||||
|
||||
/// <summary>
|
||||
/// Retention downgrade grace period: for this many days after a tenant loses Pro
|
||||
/// (explicit downgrade or trial expiry) the retention sweep keeps enforcing the PRO
|
||||
/// cap, so the downgrade does not immediately hard-delete data older than the
|
||||
/// Community cap. Retention is the only entitlement with a grace — everything else
|
||||
/// gates read-time immediately because it is reversible without data loss.
|
||||
/// Resolved via <see cref="Services.TenantEntitlementService.GetRetentionGraceEndUtc"/>.
|
||||
/// </summary>
|
||||
public const int RetentionDowngradeGraceDays = 30;
|
||||
|
||||
private static readonly EditionEntitlements Community = new()
|
||||
{
|
||||
Edition = TenantEdition.Community,
|
||||
|
||||
+4
-3
@@ -162,16 +162,17 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
// trial expired, or whose stored value predates the cap, is enforced here even though
|
||||
// the stored DataRetentionDays is left untouched. days <= 0 stays the GA-only
|
||||
// "infinite" escape hatch (skipped below, never clamped).
|
||||
var nowUtc = _utcNow();
|
||||
var storedDays = config?.DataRetentionDays ?? 90;
|
||||
var retentionDays = config == null
|
||||
? 90
|
||||
: TenantEntitlementService.GetEffectiveRetentionDays(config, DateTime.UtcNow);
|
||||
: TenantEntitlementService.GetEffectiveRetentionDays(config, nowUtc);
|
||||
if (retentionDays > 0 && retentionDays < storedDays)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Tenant {TenantId}: DataRetentionDays={Stored} exceeds the {Edition}-edition cap — enforcing {Effective} days",
|
||||
tenantId, storedDays,
|
||||
TenantEntitlementService.ResolveEdition(config!, DateTime.UtcNow), retentionDays);
|
||||
TenantEntitlementService.ResolveEdition(config!, nowUtc), retentionDays);
|
||||
}
|
||||
|
||||
if (retentionDays <= 0)
|
||||
@@ -180,7 +181,7 @@ namespace AutopilotMonitor.Functions.Services.Deletion
|
||||
return;
|
||||
}
|
||||
|
||||
var cutoffUtc = DateTime.UtcNow.AddDays(-retentionDays);
|
||||
var cutoffUtc = nowUtc.AddDays(-retentionDays);
|
||||
|
||||
// Server-bounded read: fetch one more than the per-run dispatch cap. The loop below
|
||||
// only ever advances MaxEnqueuesPerTenantPerRun sessions, so loading the whole backlog
|
||||
|
||||
@@ -1,5 +1,4 @@
|
||||
using System;
|
||||
using System.Collections.Concurrent;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
@@ -21,8 +20,15 @@ namespace AutopilotMonitor.Functions.Services
|
||||
private readonly IMemoryCache _cache;
|
||||
private readonly ILogger<DistressRateLimitService> _logger;
|
||||
|
||||
// Per-key locks (same proven pattern as RateLimitService)
|
||||
private readonly ConcurrentDictionary<string, object> _locks = new();
|
||||
// History + lock in one cache entry (same pattern as RateLimitService): same list ⇒ same lock.
|
||||
private sealed class RequestBucket
|
||||
{
|
||||
public readonly List<DateTime> History = new();
|
||||
public readonly object Sync = new();
|
||||
}
|
||||
|
||||
// Serializes bucket creation (GetOrCreate is not atomic); hot path is a lock-free TryGetValue.
|
||||
private readonly object _createLock = new();
|
||||
|
||||
// Layer 1: Per-IP
|
||||
private const int MaxPerIp = 5;
|
||||
@@ -91,17 +97,12 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// </summary>
|
||||
private bool CheckSlidingWindow(string cacheKey, int maxRequests, TimeSpan window, DateTime now)
|
||||
{
|
||||
var lockObj = _locks.GetOrAdd(cacheKey, _ => new object());
|
||||
var history = _cache.GetOrCreate(cacheKey, entry =>
|
||||
{
|
||||
entry.SlidingExpiration = window;
|
||||
entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(2);
|
||||
entry.RegisterPostEvictionCallback((key, _, _, _) => _locks.TryRemove((string)key, out _));
|
||||
return new List<DateTime>();
|
||||
})!;
|
||||
var bucket = GetOrCreateBucket(cacheKey, window);
|
||||
|
||||
lock (lockObj)
|
||||
lock (bucket.Sync)
|
||||
{
|
||||
var history = bucket.History;
|
||||
|
||||
// Remove entries outside the window
|
||||
var windowStart = now.Subtract(window);
|
||||
history.RemoveAll(ts => ts < windowStart);
|
||||
@@ -113,16 +114,26 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
history.Add(now);
|
||||
|
||||
_cache.Set(cacheKey, history, new MemoryCacheEntryOptions
|
||||
{
|
||||
SlidingExpiration = window,
|
||||
AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(2)
|
||||
});
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
private RequestBucket GetOrCreateBucket(string cacheKey, TimeSpan window)
|
||||
{
|
||||
if (_cache.TryGetValue(cacheKey, out RequestBucket? existing) && existing != null)
|
||||
return existing;
|
||||
|
||||
lock (_createLock)
|
||||
{
|
||||
return _cache.GetOrCreate(cacheKey, entry =>
|
||||
{
|
||||
entry.SlidingExpiration = window;
|
||||
entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(2);
|
||||
return new RequestBucket();
|
||||
})!;
|
||||
}
|
||||
}
|
||||
|
||||
private bool IsCircuitOpen(DateTime now)
|
||||
{
|
||||
lock (_globalLock)
|
||||
|
||||
@@ -0,0 +1,259 @@
|
||||
using System.Net.Http.Json;
|
||||
using System.Text.Json;
|
||||
using System.Text.Json.Serialization;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Sends the product's transactional emails (tenant-activation welcome, post-offboarding
|
||||
/// farewell). Best-effort: failures are logged as warnings and never propagated.
|
||||
/// <para>
|
||||
/// The class and its callers are provider-neutral. Everything provider-specific is confined
|
||||
/// to the <c>Email:*</c> configuration section and <see cref="SendViaMandrillAsync"/>, so a
|
||||
/// future provider swap touches the transport method and the app settings only.
|
||||
/// Current provider: Mailchimp Transactional (Mandrill) — <c>POST {Email:Endpoint}</c> with the
|
||||
/// API key in the JSON body. Open/click tracking is explicitly disabled so the provider
|
||||
/// receives nothing beyond the recipient address and the tenant domain (trust-page claim).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Configuration (App Settings use <c>Email__Key</c>):
|
||||
/// <c>Email:ApiKey</c> (required; empty ⇒ every send is a logged no-op),
|
||||
/// <c>Email:Endpoint</c> (default <see cref="DefaultEndpoint"/>),
|
||||
/// <c>Email:FromAddress</c> (default <see cref="DefaultFromAddress"/>),
|
||||
/// <c>Email:FromName</c> (default <see cref="DefaultFromName"/>).
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class EmailService : IEmailService, IOffboardFarewellEmailSender
|
||||
{
|
||||
public const string ApiKeyConfigKey = "Email:ApiKey";
|
||||
public const string DefaultEndpoint = "https://mandrillapp.com/api/1.0/messages/send";
|
||||
public const string DefaultFromAddress = "noreply@autopilotmonitor.com";
|
||||
public const string DefaultFromName = "Autopilot Monitor";
|
||||
|
||||
private static readonly JsonSerializerOptions JsonOptions = new()
|
||||
{
|
||||
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
|
||||
};
|
||||
|
||||
private readonly HttpClient _http;
|
||||
private readonly IEmailTemplateProvider _templates;
|
||||
private readonly ILogger<EmailService> _logger;
|
||||
private readonly string _apiKey;
|
||||
private readonly string _endpoint;
|
||||
private readonly string _fromAddress;
|
||||
private readonly string _fromName;
|
||||
|
||||
public EmailService(
|
||||
HttpClient http,
|
||||
IConfiguration configuration,
|
||||
IEmailTemplateProvider templates,
|
||||
ILogger<EmailService> logger)
|
||||
{
|
||||
_http = http;
|
||||
_templates = templates;
|
||||
_logger = logger;
|
||||
_apiKey = configuration[ApiKeyConfigKey] ?? string.Empty;
|
||||
_endpoint = FirstNonBlank(configuration["Email:Endpoint"], DefaultEndpoint);
|
||||
_fromAddress = FirstNonBlank(configuration["Email:FromAddress"], DefaultFromAddress);
|
||||
_fromName = FirstNonBlank(configuration["Email:FromName"], DefaultFromName);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task SendPreviewApprovedEmailAsync(string toEmail, string domainName, CancellationToken ct = default)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(_apiKey))
|
||||
{
|
||||
_logger.LogDebug("{ConfigKey} not configured — skipping preview approval email", ApiKeyConfigKey);
|
||||
return;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(toEmail))
|
||||
{
|
||||
_logger.LogDebug("No notification email set — skipping preview approval email for {Domain}", domainName);
|
||||
return;
|
||||
}
|
||||
|
||||
var sent = await SendViaMandrillAsync(
|
||||
toEmail,
|
||||
EmailTemplates.PreviewApprovedSubject,
|
||||
await _templates.GetHtmlAsync(EmailTemplateKind.Welcome, domainName, ct),
|
||||
tag: "welcome",
|
||||
ct);
|
||||
|
||||
if (sent)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Preview approval email sent to {ToEmail} for domain {Domain}",
|
||||
toEmail, domainName);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Sends the post-offboarding "sorry to see you go" farewell email.
|
||||
/// No-op if the API key or recipient email is not configured. Best-effort: failures
|
||||
/// are logged as warnings and never propagated (the offboarding correctness contract
|
||||
/// does not depend on email delivery).
|
||||
/// </summary>
|
||||
public async Task SendAsync(string toEmail, string domainName, string tenantId, CancellationToken ct = default)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(_apiKey))
|
||||
{
|
||||
_logger.LogDebug(
|
||||
"{ConfigKey} not configured — skipping offboard farewell email for tenant {TenantId}",
|
||||
ApiKeyConfigKey, tenantId);
|
||||
return;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(toEmail))
|
||||
{
|
||||
_logger.LogDebug(
|
||||
"No notification email captured — skipping offboard farewell email for tenant {TenantId} ({Domain})",
|
||||
tenantId, domainName);
|
||||
return;
|
||||
}
|
||||
|
||||
var sent = await SendViaMandrillAsync(
|
||||
toEmail,
|
||||
EmailTemplates.OffboardingFarewellSubject,
|
||||
await _templates.GetHtmlAsync(EmailTemplateKind.Farewell, domainName, ct),
|
||||
tag: "offboarding-farewell",
|
||||
ct);
|
||||
|
||||
if (sent)
|
||||
{
|
||||
_logger.LogInformation(
|
||||
"Offboard farewell email sent to {ToEmail} for tenant {TenantId} ({Domain})",
|
||||
toEmail, tenantId, domainName);
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<bool> SendTestAsync(EmailTemplateKind kind, string toEmail, string domainName, string? draftHtml, CancellationToken ct = default)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(_apiKey))
|
||||
{
|
||||
_logger.LogWarning("{ConfigKey} not configured — cannot send {Kind} test email", ApiKeyConfigKey, kind);
|
||||
return false;
|
||||
}
|
||||
|
||||
var html = draftHtml is null
|
||||
? await _templates.GetHtmlAsync(kind, domainName, ct)
|
||||
: EmailTemplateService.Render(draftHtml, domainName);
|
||||
|
||||
var sent = await SendViaMandrillAsync(toEmail, EmailTemplateService.Subject(kind), html, tag: "test", ct);
|
||||
if (sent)
|
||||
_logger.LogInformation("{Kind} test email sent to {ToEmail} (draft={IsDraft})", kind, toEmail, draftHtml is not null);
|
||||
return sent;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The only provider-specific code path. Posts a Mandrill <c>messages/send</c> request and
|
||||
/// interprets the per-recipient result array: <c>sent</c>/<c>queued</c>/<c>scheduled</c>
|
||||
/// count as success, <c>rejected</c>/<c>invalid</c> as failure (with the provider's
|
||||
/// <c>reject_reason</c> in the warning). Never throws.
|
||||
/// </summary>
|
||||
private async Task<bool> SendViaMandrillAsync(string toEmail, string subject, string html, string tag, CancellationToken ct)
|
||||
{
|
||||
try
|
||||
{
|
||||
var request = new MandrillSendRequest
|
||||
{
|
||||
Key = _apiKey,
|
||||
Message = new MandrillMessage
|
||||
{
|
||||
FromEmail = _fromAddress,
|
||||
FromName = _fromName,
|
||||
To = new[] { new MandrillRecipient { Email = toEmail, Type = "to" } },
|
||||
Subject = subject,
|
||||
Html = html,
|
||||
AutoText = true,
|
||||
TrackOpens = false,
|
||||
TrackClicks = false,
|
||||
Tags = new[] { tag },
|
||||
},
|
||||
};
|
||||
|
||||
using var response = await _http.PostAsJsonAsync(_endpoint, request, JsonOptions, ct);
|
||||
var body = await response.Content.ReadAsStringAsync(ct);
|
||||
|
||||
if (!response.IsSuccessStatusCode)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Email provider returned {StatusCode} for {Tag} mail to {ToEmail}: {Body}",
|
||||
(int)response.StatusCode, tag, toEmail, Truncate(body));
|
||||
return false;
|
||||
}
|
||||
|
||||
var results = JsonSerializer.Deserialize<MandrillSendResult[]>(body, JsonOptions);
|
||||
var result = results?.FirstOrDefault();
|
||||
if (result is null)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Email provider returned an empty result for {Tag} mail to {ToEmail}: {Body}",
|
||||
tag, toEmail, Truncate(body));
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!IsAcceptedStatus(result.Status))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Email provider did not accept {Tag} mail to {ToEmail}: status={Status} reason={Reason}",
|
||||
tag, toEmail, result.Status, result.RejectReason ?? "n/a");
|
||||
return false;
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to send {Tag} mail to {ToEmail}", tag, toEmail);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private static bool IsAcceptedStatus(string? status)
|
||||
=> status is "sent" or "queued" or "scheduled";
|
||||
|
||||
private static string FirstNonBlank(string? value, string fallback)
|
||||
=> string.IsNullOrWhiteSpace(value) ? fallback : value.Trim();
|
||||
|
||||
private static string Truncate(string value)
|
||||
=> value.Length <= 500 ? value : value[..500] + "…";
|
||||
|
||||
// --- Mandrill wire format (https://mailchimp.com/developer/transactional/api/messages/send-new-message/) ---
|
||||
|
||||
private sealed class MandrillSendRequest
|
||||
{
|
||||
[JsonPropertyName("key")] public string Key { get; set; } = string.Empty;
|
||||
[JsonPropertyName("message")] public MandrillMessage Message { get; set; } = new();
|
||||
}
|
||||
|
||||
private sealed class MandrillMessage
|
||||
{
|
||||
[JsonPropertyName("from_email")] public string FromEmail { get; set; } = string.Empty;
|
||||
[JsonPropertyName("from_name")] public string FromName { get; set; } = string.Empty;
|
||||
[JsonPropertyName("to")] public MandrillRecipient[] To { get; set; } = Array.Empty<MandrillRecipient>();
|
||||
[JsonPropertyName("subject")] public string Subject { get; set; } = string.Empty;
|
||||
[JsonPropertyName("html")] public string Html { get; set; } = string.Empty;
|
||||
[JsonPropertyName("auto_text")] public bool AutoText { get; set; }
|
||||
[JsonPropertyName("track_opens")] public bool TrackOpens { get; set; }
|
||||
[JsonPropertyName("track_clicks")] public bool TrackClicks { get; set; }
|
||||
[JsonPropertyName("tags")] public string[] Tags { get; set; } = Array.Empty<string>();
|
||||
}
|
||||
|
||||
private sealed class MandrillRecipient
|
||||
{
|
||||
[JsonPropertyName("email")] public string Email { get; set; } = string.Empty;
|
||||
[JsonPropertyName("type")] public string Type { get; set; } = "to";
|
||||
}
|
||||
|
||||
private sealed class MandrillSendResult
|
||||
{
|
||||
[JsonPropertyName("email")] public string? Email { get; set; }
|
||||
[JsonPropertyName("status")] public string? Status { get; set; }
|
||||
[JsonPropertyName("reject_reason")] public string? RejectReason { get; set; }
|
||||
[JsonPropertyName("_id")] public string? Id { get; set; }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,151 @@
|
||||
using AutopilotMonitor.Shared.DataAccess;
|
||||
using AutopilotMonitor.Shared.Models;
|
||||
using Microsoft.Extensions.Caching.Memory;
|
||||
using Microsoft.Extensions.Logging;
|
||||
|
||||
namespace AutopilotMonitor.Functions.Services;
|
||||
|
||||
/// <summary>The two transactional mails the product sends.</summary>
|
||||
public enum EmailTemplateKind
|
||||
{
|
||||
Welcome,
|
||||
Farewell,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resolves the HTML body for a transactional mail: the operator override when one is
|
||||
/// stored, otherwise the built-in template. Consumed by <see cref="EmailService"/>.
|
||||
/// </summary>
|
||||
public interface IEmailTemplateProvider
|
||||
{
|
||||
Task<string> GetHtmlAsync(EmailTemplateKind kind, string domainName, CancellationToken ct = default);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Operator-editable email templates. Overrides live in the PreviewConfig table (partition
|
||||
/// "EmailTemplates", row key = kind) and are cached for 5 minutes; the built-in templates in
|
||||
/// <see cref="EmailTemplates"/> remain the fallback. Subjects are not editable — they are pinned
|
||||
/// constants.
|
||||
/// <para>
|
||||
/// Rendering contract: the only placeholder is <c>{{domainName}}</c>; an empty domain renders as
|
||||
/// "your organization", exactly like the built-ins. Resolution is fail-soft — a storage error
|
||||
/// falls back to the built-in template so activation/offboarding mails keep going out.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public class EmailTemplateService : IEmailTemplateProvider
|
||||
{
|
||||
public const string DomainPlaceholder = "{{domainName}}";
|
||||
public const string EmptyDomainLabel = "your organization";
|
||||
/// <summary>Azure Table string properties cap at 32K UTF-16 chars; keep headroom.</summary>
|
||||
public const int MaxHtmlLength = 30_000;
|
||||
|
||||
private static readonly TimeSpan CacheDuration = TimeSpan.FromMinutes(5);
|
||||
|
||||
private readonly IConfigRepository _configRepo;
|
||||
private readonly IMemoryCache _cache;
|
||||
private readonly ILogger<EmailTemplateService> _logger;
|
||||
|
||||
public EmailTemplateService(IConfigRepository configRepo, IMemoryCache cache, ILogger<EmailTemplateService> logger)
|
||||
{
|
||||
_configRepo = configRepo;
|
||||
_cache = cache;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public static bool TryParseKind(string? value, out EmailTemplateKind kind)
|
||||
{
|
||||
switch (value?.Trim().ToLowerInvariant())
|
||||
{
|
||||
case "welcome": kind = EmailTemplateKind.Welcome; return true;
|
||||
case "farewell": kind = EmailTemplateKind.Farewell; return true;
|
||||
default: kind = default; return false;
|
||||
}
|
||||
}
|
||||
|
||||
public static string KindKey(EmailTemplateKind kind) => kind == EmailTemplateKind.Welcome ? "welcome" : "farewell";
|
||||
|
||||
public static string Subject(EmailTemplateKind kind) => kind == EmailTemplateKind.Welcome
|
||||
? EmailTemplates.PreviewApprovedSubject
|
||||
: EmailTemplates.OffboardingFarewellSubject;
|
||||
|
||||
/// <summary>Built-in template with the placeholder left in place (for the editor).</summary>
|
||||
public static string BuiltInRaw(EmailTemplateKind kind) => kind == EmailTemplateKind.Welcome
|
||||
? EmailTemplates.GetPreviewApprovedHtml(DomainPlaceholder)
|
||||
: EmailTemplates.GetOffboardingFarewellHtml(DomainPlaceholder);
|
||||
|
||||
/// <summary>Substitutes the placeholder; an empty domain becomes "your organization".</summary>
|
||||
public static string Render(string rawHtml, string? domainName)
|
||||
{
|
||||
var display = string.IsNullOrWhiteSpace(domainName) ? EmptyDomainLabel : domainName.Trim();
|
||||
return rawHtml.Replace(DomainPlaceholder, display, StringComparison.Ordinal);
|
||||
}
|
||||
|
||||
/// <summary>Returns null when the HTML is acceptable, otherwise the validation message.</summary>
|
||||
public static string? Validate(string? html)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(html)) return "Template HTML must not be empty.";
|
||||
if (html.Length > MaxHtmlLength) return $"Template HTML exceeds {MaxHtmlLength:N0} characters ({html.Length:N0}).";
|
||||
return null;
|
||||
}
|
||||
|
||||
public async Task<string> GetHtmlAsync(EmailTemplateKind kind, string domainName, CancellationToken ct = default)
|
||||
{
|
||||
var overrideEntry = await GetOverrideAsync(kind, failSoft: true);
|
||||
return overrideEntry is null
|
||||
? Render(BuiltInRaw(kind), domainName)
|
||||
: Render(overrideEntry.Html, domainName);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The stored override, or null for built-in. <paramref name="failSoft"/> swallows storage
|
||||
/// errors (send path must never depend on the override store); the admin endpoints pass
|
||||
/// false so a broken store surfaces as an error instead of silently showing the built-in.
|
||||
/// </summary>
|
||||
public virtual async Task<EmailTemplateOverride?> GetOverrideAsync(EmailTemplateKind kind, bool failSoft = false)
|
||||
{
|
||||
var key = CacheKey(kind);
|
||||
if (_cache.TryGetValue(key, out EmailTemplateOverride? cached))
|
||||
return cached;
|
||||
|
||||
try
|
||||
{
|
||||
var entry = await _configRepo.GetEmailTemplateOverrideAsync(KindKey(kind));
|
||||
// null is a valid (and the common) value — cache it too so the send path
|
||||
// doesn't hit storage on every mail.
|
||||
_cache.Set(key, entry, CacheDuration);
|
||||
return entry;
|
||||
}
|
||||
catch (Exception ex) when (failSoft)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to read {Kind} email template override — using built-in template", kind);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
public virtual async Task<EmailTemplateOverride> SaveOverrideAsync(EmailTemplateKind kind, string html, string updatedBy)
|
||||
{
|
||||
var error = Validate(html);
|
||||
if (error is not null) throw new ArgumentException(error, nameof(html));
|
||||
|
||||
var entry = new EmailTemplateOverride
|
||||
{
|
||||
Kind = KindKey(kind),
|
||||
Html = html,
|
||||
UpdatedBy = updatedBy,
|
||||
UpdatedUtc = DateTime.UtcNow,
|
||||
};
|
||||
await _configRepo.SaveEmailTemplateOverrideAsync(entry);
|
||||
_cache.Remove(CacheKey(kind));
|
||||
_logger.LogInformation("Email template override saved: {Kind} by {UpdatedBy} ({Length} chars)", kind, updatedBy, html.Length);
|
||||
return entry;
|
||||
}
|
||||
|
||||
public virtual async Task DeleteOverrideAsync(EmailTemplateKind kind, string deletedBy)
|
||||
{
|
||||
await _configRepo.DeleteEmailTemplateOverrideAsync(KindKey(kind));
|
||||
_cache.Remove(CacheKey(kind));
|
||||
_logger.LogInformation("Email template override reset to built-in: {Kind} by {DeletedBy}", kind, deletedBy);
|
||||
}
|
||||
|
||||
private static string CacheKey(EmailTemplateKind kind) => $"email-template-override:{KindKey(kind)}";
|
||||
}
|
||||
@@ -3,7 +3,7 @@ using AutopilotMonitor.Shared;
|
||||
namespace AutopilotMonitor.Functions.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Email templates for Resend notifications.
|
||||
/// Templates for the product's transactional emails (sent via <see cref="EmailService"/>).
|
||||
/// Keep all email content here for easy maintenance.
|
||||
/// </summary>
|
||||
public static class EmailTemplates
|
||||
@@ -37,12 +37,33 @@ public static class EmailTemplates
|
||||
<tr><td align=""center"">
|
||||
<table width=""600"" cellpadding=""0"" cellspacing=""0"" style=""background-color:#ffffff; border-radius:12px; overflow:hidden; box-shadow:0 4px 6px rgba(0,0,0,0.07);"">
|
||||
|
||||
<!-- Header -->
|
||||
<!-- Header: image-free. Base for every client: solid bgcolor band, text wordmark, brand mark
|
||||
drawn with CSS borders (trapezoid + a green wedge for the slanted right edge; must stay
|
||||
<div>s — Outlook paints transparent <td> borders white). Progressive enhancement where
|
||||
supported: ONE linear-gradient (hex colours, % stops — the only form Outlook web/iOS
|
||||
renders; multiple layers or rgba stops are dropped) forms the facets, and clip-path cuts
|
||||
the slanted bottom edge (Apple Mail only). No bgcolor inside the mark: Outlook dark mode
|
||||
inverts cell backgrounds but leaves border colours alone.
|
||||
Geometry is the approved reference (scratch header-render.html): facets
|
||||
light right of (40%,0)->(70%,100%), dark right of (55%,0)->(85%,100%) and left of
|
||||
(78%,0)->(100%,60%) — the last one is dropped for the single-gradient constraint.
|
||||
Change only against a side-by-side render. -->
|
||||
<tr>
|
||||
<td style=""background:linear-gradient(135deg,#1e8a4c,#33b161); padding:32px 40px; text-align:center;"">
|
||||
<h1 style=""color:#ffffff; margin:0; font-size:24px; font-weight:700;"">Autopilot Monitor</h1>
|
||||
<p style=""color:#c9efd8; margin:8px 0 0; font-size:14px;"">Windows Autopilot monitoring</p>
|
||||
</td>
|
||||
<td bgcolor=""#33b161"" style=""background-color:#33b161; background-image:linear-gradient(39.8deg, #33b161 0%, #33b161 53.8%, #3cbb6a 53.8%, #3cbb6a 65.4%, #2ea85a 65.4%, #2ea85a 100%); clip-path:polygon(0 0, 100% 0, 100% 100%, 0 82%); padding:28px 32px 52px; line-height:0; font-size:0;"">
|
||||
<table cellpadding=""0"" cellspacing=""0"" border=""0""><tr>
|
||||
<td valign=""middle"" style=""padding:0 12px 0 0; line-height:0; font-size:0;"">
|
||||
<!-- brand mark, 34x33, three non-overlapping rows (Outlook ignores negative margins and
|
||||
paints transparent td borders white, so: div triangles + one white td only) -->
|
||||
<div style=""width:0; height:0; border-left:31px solid #ffffff; border-top:13px solid transparent; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div>
|
||||
<table cellpadding=""0"" cellspacing=""0"" border=""0"" style=""border-collapse:collapse;""><tr>
|
||||
<td style=""padding:0; font-size:0; line-height:0;""><div style=""width:0; height:18px; border-left:31px solid #ffffff; border-top:0; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div></td>
|
||||
<td style=""padding:0; font-size:0; line-height:0;""><div style=""width:0; height:0; border-left:3px solid #ffffff; border-top:18px solid transparent; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div></td>
|
||||
</tr></table>
|
||||
<div style=""width:0; height:0; border-right:34px solid #ffffff; border-bottom:2px solid transparent; border-left:0; border-top:0; display:block; font-size:0; line-height:0;""></div>
|
||||
</td>
|
||||
<td valign=""middle"" style=""color:#ffffff; font-size:22px; font-weight:600; letter-spacing:-0.2px; line-height:1; white-space:nowrap; font-family:-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;"">Autopilot Monitor</td>
|
||||
</tr></table>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<!-- Body -->
|
||||
@@ -56,14 +77,14 @@ public static class EmailTemplates
|
||||
|
||||
<p style=""color:#374151; font-size:15px; line-height:1.6; margin:0 0 24px;"">
|
||||
Thank you for taking the time to try Autopilot Monitor. Every organization that
|
||||
puts it to work in a real environment helps shape what it becomes — we're
|
||||
puts it to work in a real environment helps shape what it becomes – we're
|
||||
genuinely grateful you gave it a chance.
|
||||
</p>
|
||||
|
||||
<!-- Gentle feedback ask -->
|
||||
<p style=""color:#374151; font-size:15px; line-height:1.6; margin:0 0 12px;"">
|
||||
If something didn't fit — a missing feature, a bug that got in the way, or
|
||||
simply a better alternative — we'd love to hear about it. A couple of honest
|
||||
If something didn't fit – a missing feature, a bug that got in the way, or
|
||||
simply a better alternative – we'd love to hear about it. A couple of honest
|
||||
sentences help more than you'd think:
|
||||
</p>
|
||||
|
||||
@@ -78,7 +99,7 @@ public static class EmailTemplates
|
||||
|
||||
<p style=""color:#6b7280; font-size:13px; line-height:1.6; margin:0;"">
|
||||
This is the last email you'll receive from us. This mailbox doesn't accept
|
||||
replies — please use the links above if you'd like to get in touch.
|
||||
replies – please use the links above if you'd like to get in touch.
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
@@ -115,12 +136,33 @@ public static class EmailTemplates
|
||||
<tr><td align=""center"">
|
||||
<table width=""600"" cellpadding=""0"" cellspacing=""0"" style=""background-color:#ffffff; border-radius:12px; overflow:hidden; box-shadow:0 4px 6px rgba(0,0,0,0.07);"">
|
||||
|
||||
<!-- Header -->
|
||||
<!-- Header: image-free. Base for every client: solid bgcolor band, text wordmark, brand mark
|
||||
drawn with CSS borders (trapezoid + a green wedge for the slanted right edge; must stay
|
||||
<div>s — Outlook paints transparent <td> borders white). Progressive enhancement where
|
||||
supported: ONE linear-gradient (hex colours, % stops — the only form Outlook web/iOS
|
||||
renders; multiple layers or rgba stops are dropped) forms the facets, and clip-path cuts
|
||||
the slanted bottom edge (Apple Mail only). No bgcolor inside the mark: Outlook dark mode
|
||||
inverts cell backgrounds but leaves border colours alone.
|
||||
Geometry is the approved reference (scratch header-render.html): facets
|
||||
light right of (40%,0)->(70%,100%), dark right of (55%,0)->(85%,100%) and left of
|
||||
(78%,0)->(100%,60%) — the last one is dropped for the single-gradient constraint.
|
||||
Change only against a side-by-side render. -->
|
||||
<tr>
|
||||
<td style=""background:linear-gradient(135deg,#1e8a4c,#33b161); padding:32px 40px; text-align:center;"">
|
||||
<h1 style=""color:#ffffff; margin:0; font-size:24px; font-weight:700;"">Autopilot Monitor</h1>
|
||||
<p style=""color:#c9efd8; margin:8px 0 0; font-size:14px;"">Windows Autopilot monitoring</p>
|
||||
</td>
|
||||
<td bgcolor=""#33b161"" style=""background-color:#33b161; background-image:linear-gradient(39.8deg, #33b161 0%, #33b161 53.8%, #3cbb6a 53.8%, #3cbb6a 65.4%, #2ea85a 65.4%, #2ea85a 100%); clip-path:polygon(0 0, 100% 0, 100% 100%, 0 82%); padding:28px 32px 52px; line-height:0; font-size:0;"">
|
||||
<table cellpadding=""0"" cellspacing=""0"" border=""0""><tr>
|
||||
<td valign=""middle"" style=""padding:0 12px 0 0; line-height:0; font-size:0;"">
|
||||
<!-- brand mark, 34x33, three non-overlapping rows (Outlook ignores negative margins and
|
||||
paints transparent td borders white, so: div triangles + one white td only) -->
|
||||
<div style=""width:0; height:0; border-left:31px solid #ffffff; border-top:13px solid transparent; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div>
|
||||
<table cellpadding=""0"" cellspacing=""0"" border=""0"" style=""border-collapse:collapse;""><tr>
|
||||
<td style=""padding:0; font-size:0; line-height:0;""><div style=""width:0; height:18px; border-left:31px solid #ffffff; border-top:0; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div></td>
|
||||
<td style=""padding:0; font-size:0; line-height:0;""><div style=""width:0; height:0; border-left:3px solid #ffffff; border-top:18px solid transparent; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div></td>
|
||||
</tr></table>
|
||||
<div style=""width:0; height:0; border-right:34px solid #ffffff; border-bottom:2px solid transparent; border-left:0; border-top:0; display:block; font-size:0; line-height:0;""></div>
|
||||
</td>
|
||||
<td valign=""middle"" style=""color:#ffffff; font-size:22px; font-weight:600; letter-spacing:-0.2px; line-height:1; white-space:nowrap; font-family:-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;"">Autopilot Monitor</td>
|
||||
</tr></table>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
<!-- Body -->
|
||||
@@ -129,8 +171,8 @@ public static class EmailTemplates
|
||||
<h2 style=""color:#111827; margin:0 0 16px; font-size:20px;"">Welcome to Autopilot Monitor!</h2>
|
||||
|
||||
<p style=""color:#374151; font-size:15px; line-height:1.6; margin:0 0 16px;"">
|
||||
Great news — access for <strong>{displayDomain}</strong> has been activated and is ready to use.
|
||||
You can now sign in and start monitoring your Windows Autopilot enrollments in real time.
|
||||
Great news – access for <strong>{displayDomain}</strong> has been activated and is ready to use.
|
||||
You can now <a href=""{Constants.PortalBaseUrl}"" target=""_blank"" style=""color:#1e8a4c; text-decoration:underline;"">sign in</a> and start monitoring your Windows Autopilot enrollments in real time.
|
||||
</p>
|
||||
|
||||
<p style=""color:#374151; font-size:15px; line-height:1.6; margin:0 0 24px;"">
|
||||
@@ -149,13 +191,6 @@ public static class EmailTemplates
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
<!-- Active development note -->
|
||||
<div style=""background-color:#fef3c7; border:1px solid #fde68a; border-radius:8px; padding:16px 20px; margin:0 0 24px;"">
|
||||
<p style=""color:#92400e; font-size:14px; line-height:1.5; margin:0;"">
|
||||
<strong>Please note:</strong> Autopilot Monitor is in active development. Some features are still being built
|
||||
and things may occasionally not work as expected. Your patience and understanding are greatly appreciated!
|
||||
</p>
|
||||
</div>
|
||||
|
||||
<!-- Feedback -->
|
||||
<p style=""color:#374151; font-size:15px; line-height:1.6; margin:0 0 12px;"">
|
||||
@@ -169,7 +204,7 @@ public static class EmailTemplates
|
||||
</ul>
|
||||
|
||||
<p style=""color:#6b7280; font-size:14px; line-height:1.6; margin:0;"">
|
||||
Thanks for being part of the journey — enjoy Autopilot Monitor!
|
||||
Thanks for being part of the journey – enjoy Autopilot Monitor!
|
||||
</p>
|
||||
</td>
|
||||
</tr>
|
||||
|
||||
@@ -92,6 +92,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
bool AccountSetupAllSucceeded,
|
||||
bool HasExplicitFailure,
|
||||
bool HasTerminalComplete,
|
||||
bool HasEnrollmentComplete,
|
||||
bool HasAgentEmergencyBreak,
|
||||
bool DesktopArrived,
|
||||
bool HelloResolved,
|
||||
@@ -113,7 +114,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
bool deviceAll = false;
|
||||
int acctBestN = 0, acctBestM = 0;
|
||||
bool acctFallbackAll = false;
|
||||
bool hasFailure = false, hasComplete = false, hasEmergencyBreak = false;
|
||||
bool hasFailure = false, hasComplete = false, hasEnrollmentComplete = false, hasEmergencyBreak = false;
|
||||
bool desktopArrived = false, helloResolved = false;
|
||||
bool realmJoinDetected = false, realmJoinResolved = false;
|
||||
bool sawHelloPolicyEnabled = false, sawHelloPolicyDisabled = false;
|
||||
@@ -135,7 +136,12 @@ namespace AutopilotMonitor.Functions.Services
|
||||
hasFailure = true;
|
||||
}
|
||||
else if (Eq(type, "enrollment_complete") || Eq(type, "whiteglove_complete"))
|
||||
{
|
||||
hasComplete = true;
|
||||
// whiteglove_complete only proves Part 1 (technician flow) — the WhiteGlove
|
||||
// awaiting-user gate needs to know whether a REAL enrollment completion exists.
|
||||
if (Eq(type, "enrollment_complete")) hasEnrollmentComplete = true;
|
||||
}
|
||||
else if (Eq(type, "agent_emergency_break"))
|
||||
hasEmergencyBreak = true;
|
||||
else if (Eq(type, "desktop_arrived"))
|
||||
@@ -217,6 +223,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
AccountSetupAllSucceeded: acctAll,
|
||||
HasExplicitFailure: hasFailure,
|
||||
HasTerminalComplete: hasComplete,
|
||||
HasEnrollmentComplete: hasEnrollmentComplete,
|
||||
HasAgentEmergencyBreak: hasEmergencyBreak,
|
||||
DesktopArrived: desktopArrived,
|
||||
HelloResolved: helloResolved,
|
||||
@@ -226,6 +233,74 @@ namespace AutopilotMonitor.Functions.Services
|
||||
RealmJoinResolved: realmJoinResolved);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// WhiteGlove Part-2 awaiting-user gate (fairstone.ca analysis 2026-08-21). After the
|
||||
/// reseal-reboot the session is BY DEFINITION in the user-driven final phase: the agent
|
||||
/// resumed (<c>whiteglove_resumed</c> → ResumedAt, the only writer of that column), the
|
||||
/// Part-2 Device ESP re-confirms in minutes, and the device sits at the logon screen.
|
||||
/// Technicians routinely power the machine off there to box it — well before the 10-min
|
||||
/// hybrid-login evidence window — so agent silence in this state means "sealed / powered
|
||||
/// off awaiting the end user", not a stall. The gate demands the full evidence set:
|
||||
/// pre-provisioned + resumed + Device Setup provisioned, and NO trace of a real user
|
||||
/// since the resume (no Account Setup progress, no real-user desktop, no Hello terminal,
|
||||
/// no enrollment completion) and no explicit failure. Any user evidence keeps the honest
|
||||
/// Stalled/timeout handling — a stall after the user showed up is a real stall.
|
||||
/// Shared by the maintenance sweep stage 1, the <c>session_stalled</c> ingest mapping and
|
||||
/// <see cref="ClassifyTimedOutSession"/> so all paths apply one evidence rule.
|
||||
/// </summary>
|
||||
public static bool IsWhiteGloveAwaitingUser(
|
||||
in EspProvisioningRollup rollup, bool isPreProvisioned, DateTime? resumedAt)
|
||||
=> isPreProvisioned
|
||||
&& resumedAt.HasValue
|
||||
&& rollup.DeviceSetupAllSucceeded
|
||||
&& !rollup.HasExplicitFailure
|
||||
&& !rollup.HasEnrollmentComplete
|
||||
&& rollup.AccountSetupSucceededCount == 0
|
||||
&& !rollup.DesktopArrived
|
||||
&& !rollup.HelloResolved;
|
||||
|
||||
/// <summary>
|
||||
/// Customer-facing reason for a WhiteGlove Part-2 session parked as AwaitingUser by the
|
||||
/// silence/stall paths. One authoring spot so the sweep, the ingest mapping and the
|
||||
/// timeout classifier show the identical text.
|
||||
/// </summary>
|
||||
public static string WhiteGloveAwaitingUserReason(in EspProvisioningRollup rollup)
|
||||
{
|
||||
var acct = rollup.AccountSetupTotal > 0
|
||||
? $" (Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal})"
|
||||
: " (Account Setup not yet started)";
|
||||
return "Pre-provisioning (WhiteGlove) completed; no user sign-in since Part 2 resumed — " +
|
||||
$"device presumably sealed or powered off awaiting the end user{acct}";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Self-deploying (kiosk / shared-device) profile gate. Such a profile has NO user phase:
|
||||
/// Device ESP reaching all-succeeded IS the enrollment's end, and the agent's own
|
||||
/// SelfDeploying terminal only adds a 5-min confirmation window on top of it. When the
|
||||
/// agent goes silent after that point (reboot into the kiosk autologon, powered off and
|
||||
/// boxed for the school term, WiFi never re-associating for the relaunch — tenant
|
||||
/// aebdce78, 836 sessions 2026-08-17..21), "awaiting user / Account Setup" is a factual
|
||||
/// error and the later Incomplete("Account Setup 0/5") is worse: nobody will ever sign
|
||||
/// in, the devices are finished and in service. Demands the registration-time
|
||||
/// registry-confirmed profile flag (CloudAssignedOobeConfig 0x20|0x40), Device Setup
|
||||
/// all-succeeded and no explicit failure. Shared by the sweep (stage 1 + 2), the
|
||||
/// max-lifetime ingest verdict and the retro-reclassification so one evidence rule applies.
|
||||
/// </summary>
|
||||
public static bool IsSelfDeployingProvisioned(in EspProvisioningRollup rollup, bool isSelfDeployingProfile)
|
||||
=> isSelfDeployingProfile
|
||||
&& rollup.DeviceSetupAllSucceeded
|
||||
&& !rollup.HasExplicitFailure;
|
||||
|
||||
/// <summary>
|
||||
/// Customer-facing Succeeded-reconcile reason for <see cref="IsSelfDeployingProvisioned"/>.
|
||||
/// Carries the silence-transparency clause like every other reconcile reason.
|
||||
/// </summary>
|
||||
public static string SelfDeployingReconcileReason(DateTime startedAtUtc, DateTime nowUtc, DateTime? lastEventAtUtc)
|
||||
=> AppendReconcileTiming(
|
||||
"Reconciled: self-deploying profile — Device Setup fully provisioned (this profile has no user / Account Setup phase); " +
|
||||
"agent went silent before confirming completion",
|
||||
startedAtUtc, nowUtc, lastEventAtUtc);
|
||||
|
||||
/// <summary>
|
||||
/// Decide the honest terminal (or non-terminal) state for a timed-out session. See the
|
||||
/// decision table in the design note. Returns the target status and a human-readable
|
||||
@@ -242,27 +317,66 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// mechanism declared success too early" — session efbc17ff. Falls back to
|
||||
/// <paramref name="startedAtUtc"/> when unknown (same fallback the stalled-marker uses).
|
||||
/// </param>
|
||||
public static (SessionStatus Status, string Reason) ClassifyTimedOutSession(
|
||||
/// <param name="isPreProvisioned">Session's IsPreProvisioned flag (WhiteGlove Part-2 gate input).</param>
|
||||
/// <param name="resumedAt">Session's ResumedAt — set exclusively by the whiteglove_resumed ingest.</param>
|
||||
/// <param name="isSelfDeployingProfile">Session's IsSelfDeployingProfile flag (rule 1c gate input).</param>
|
||||
public static (SessionStatus Status, string Reason, string Rule) ClassifyTimedOutSession(
|
||||
EspProvisioningRollup rollup, DateTime startedAtUtc, DateTime nowUtc, int graceHours,
|
||||
DateTime? lastEventAtUtc = null)
|
||||
DateTime? lastEventAtUtc = null, bool isPreProvisioned = false, DateTime? resumedAt = null,
|
||||
bool isSelfDeployingProfile = false)
|
||||
{
|
||||
// 1. An explicit terminal failure event is a real failure (defensive — such a session
|
||||
// would normally already be terminal via ingest and not reach the sweep).
|
||||
if (rollup.HasExplicitFailure)
|
||||
return (SessionStatus.Failed, "Enrollment reported an explicit failure before timeout");
|
||||
return (SessionStatus.Failed, "Enrollment reported an explicit failure before timeout", ClassifierRules.R1ExplicitFailure);
|
||||
|
||||
// 1b. WhiteGlove Part 2 with zero user evidence since the resume: the device is parked
|
||||
// between technician and end user (sealed box / powered off at the logon screen).
|
||||
// Must run BEFORE rule 2 — the Part-1 whiteglove_complete in the same event stream
|
||||
// sets HasTerminalComplete and would otherwise reconcile the session to Succeeded
|
||||
// with a reason claiming Account Setup completed, which never happened. Within
|
||||
// grace the honest label is AwaitingUser; past grace the Part-1 success stands as
|
||||
// the session outcome (the user phase is tracked by the session the device
|
||||
// registers when it is finally unboxed), stated with an honest WhiteGlove reason.
|
||||
if (IsWhiteGloveAwaitingUser(rollup, isPreProvisioned, resumedAt))
|
||||
{
|
||||
if ((nowUtc - startedAtUtc).TotalHours < graceHours)
|
||||
return (SessionStatus.AwaitingUser, WhiteGloveAwaitingUserReason(rollup), ClassifierRules.R1bWhiteGloveAwaiting);
|
||||
|
||||
return (SessionStatus.Succeeded, AppendReconcileTiming(
|
||||
$"Reconciled at timeout: pre-provisioning (WhiteGlove Part 1) completed — no user sign-in observed within {graceHours}h grace",
|
||||
startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R1bWhiteGloveSucceeded);
|
||||
}
|
||||
|
||||
// 1c. Self-deploying profile with Device Setup fully provisioned: there is no user phase
|
||||
// to wait for, so neither AwaitingUser nor the grace window apply — the device is
|
||||
// done (see IsSelfDeployingProvisioned). Runs before rule 2 so the reason names the
|
||||
// profile instead of claiming "Account Setup completed"; real user evidence cannot
|
||||
// contradict it (a kiosk autologon is not a user) and an explicit failure already
|
||||
// returned in rule 1.
|
||||
if (IsSelfDeployingProvisioned(rollup, isSelfDeployingProfile))
|
||||
return (SessionStatus.Succeeded, SelfDeployingReconcileReason(startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R1cSelfDeploying);
|
||||
|
||||
// 2. Account Setup fully succeeded (or a terminal completion event) → reconcile to success.
|
||||
if (rollup.AccountSetupAllSucceeded || rollup.HasTerminalComplete)
|
||||
// Once Part 2 has resumed, the Part-1 whiteglove_complete in the same stream no longer
|
||||
// proves the session's outcome — only a real enrollment_complete does. Without this,
|
||||
// a resumed session with partial user progress (e.g. Account Setup 1/5, then silence)
|
||||
// reconciled to Succeeded on Part-1 evidence, with a reason claiming Account Setup
|
||||
// completed; rules 4-6 below hold the honest verdicts for that shape.
|
||||
var terminalCompleteProvesOutcome = rollup.HasEnrollmentComplete
|
||||
|| (rollup.HasTerminalComplete && !resumedAt.HasValue);
|
||||
if (rollup.AccountSetupAllSucceeded || terminalCompleteProvesOutcome)
|
||||
return (SessionStatus.Succeeded, AppendReconcileTiming(
|
||||
"Reconciled at timeout: Account Setup completed (all subcategories succeeded / enrollment_complete observed)",
|
||||
startedAtUtc, nowUtc, lastEventAtUtc));
|
||||
startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R2AccountSetupComplete);
|
||||
|
||||
// 3. The agent's absolute-age emergency break fired — the agent has cleaned up and exited, so
|
||||
// nothing more will ever arrive for this session. Terminalize NOW (skip the AwaitingUser grace):
|
||||
// it did not complete (2) and did not explicitly fail (1), so the honest verdict is Incomplete.
|
||||
if (rollup.HasAgentEmergencyBreak)
|
||||
return (SessionStatus.Incomplete,
|
||||
"Agent emergency break fired (absolute session-age cap) — agent gone without completion");
|
||||
"Agent emergency break fired (absolute session-age cap) — agent gone without completion",
|
||||
ClassifierRules.R3EmergencyBreak);
|
||||
|
||||
// 4. The user demonstrably finished setup: a real-user desktop was observed AND
|
||||
// Windows Hello reached a positive terminal (provisioned or skipped). Those are
|
||||
@@ -297,7 +411,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
: "agent went silent before reporting completion";
|
||||
return (SessionStatus.Succeeded, AppendReconcileTiming(
|
||||
$"Reconciled at timeout: user completed setup ({evidence}) — {detail}",
|
||||
startedAtUtc, nowUtc, lastEventAtUtc));
|
||||
startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R4DesktopHello);
|
||||
}
|
||||
|
||||
// 5. Device Setup fully provisioned (device is AADJ + MDM-enrolled), user phase pending.
|
||||
@@ -310,7 +424,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
? $" (Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal})"
|
||||
: " (Account Setup not yet started)";
|
||||
return (SessionStatus.AwaitingUser,
|
||||
$"Device Setup completed; awaiting user / Account Setup phase — agent silent, within {graceHours}h grace{acct}");
|
||||
$"Device Setup completed; awaiting user / Account Setup phase — agent silent, within {graceHours}h grace{acct}",
|
||||
ClassifierRules.R5DeviceSetupAwaiting);
|
||||
}
|
||||
|
||||
// "0/0" would suggest a parsed rollup that never existed — when no Account Setup
|
||||
@@ -319,12 +434,14 @@ namespace AutopilotMonitor.Functions.Services
|
||||
? $"last Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal}"
|
||||
: "Account Setup progress never observed";
|
||||
return (SessionStatus.Incomplete,
|
||||
$"No completion signal within {graceHours}h grace after Device Setup completed ({acctDetail})");
|
||||
$"No completion signal within {graceHours}h grace after Device Setup completed ({acctDetail})",
|
||||
ClassifierRules.R5DeviceSetupIncomplete);
|
||||
}
|
||||
|
||||
// 6. Silent before Device Setup completed, with no explicit failure → unknown, not a failure.
|
||||
return (SessionStatus.Incomplete,
|
||||
"No Device Setup completion or explicit failure signal observed before timeout");
|
||||
"No Device Setup completion or explicit failure signal observed before timeout",
|
||||
ClassifierRules.R6Fallthrough);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
+89
-15
@@ -14,7 +14,15 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
foreach (var evt in storedEvents)
|
||||
{
|
||||
if (!classification.EarliestEventTimestamp.HasValue || evt.Timestamp < classification.EarliestEventTimestamp.Value)
|
||||
// System-timeline-watcher rows (clock steps, sleep episodes) are backfilled
|
||||
// from the Windows event log with original timestamps up to 24h before the
|
||||
// agent started — environment observation, not enrollment activity. They must
|
||||
// not pull the session anchor toward pre-enrollment OOBE idle time (mirrors
|
||||
// TableStorageService.IsSessionAnchorEligible on the Events-table probe;
|
||||
// field case d0c5b672).
|
||||
var anchorEligible = evt.Source != "SystemTimelineWatcher";
|
||||
if (anchorEligible &&
|
||||
(!classification.EarliestEventTimestamp.HasValue || evt.Timestamp < classification.EarliestEventTimestamp.Value))
|
||||
classification.EarliestEventTimestamp = evt.Timestamp;
|
||||
if (!classification.LatestEventTimestamp.HasValue || evt.Timestamp > classification.LatestEventTimestamp.Value)
|
||||
classification.LatestEventTimestamp = evt.Timestamp;
|
||||
@@ -278,8 +286,10 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var effectiveStart = session?.ResumedAt ?? session?.StartedAt ?? triggerEvent.Timestamp;
|
||||
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
|
||||
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session?.LastEventAt ?? c.LatestEventTimestamp);
|
||||
var (targetStatus, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session?.LastEventAt ?? c.LatestEventTimestamp,
|
||||
isPreProvisioned: session?.IsPreProvisioned == true, resumedAt: session?.ResumedAt,
|
||||
isSelfDeployingProfile: session?.IsSelfDeployingProfile == true);
|
||||
|
||||
// Keep the max-lifetime trigger visible: the Succeeded reasons already carry their own
|
||||
// silence-transparency clause, everything else gets the watchdog context appended.
|
||||
@@ -305,7 +315,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
}
|
||||
|
||||
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, targetStatus, triggerEvent.Phase, reason,
|
||||
request.TenantId, request.SessionId, targetStatus, VerdictPaths.Compose(VerdictPaths.OriginMaxLifetime, rule),
|
||||
triggerEvent.Phase, reason,
|
||||
completedAt: isTerminalNonSuccess ? triggerEvent.Timestamp : (DateTime?)null,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
|
||||
failureSource: isTerminalNonSuccess ? "max_lifetime_watchdog" : null,
|
||||
@@ -347,14 +358,17 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
var now = DateTime.UtcNow;
|
||||
var effectiveStart = session.ResumedAt ?? session.StartedAt;
|
||||
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session.LastEventAt ?? c.LatestEventTimestamp);
|
||||
var (targetStatus, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session.LastEventAt ?? c.LatestEventTimestamp,
|
||||
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
|
||||
isSelfDeployingProfile: session.IsSelfDeployingProfile);
|
||||
|
||||
if (targetStatus != SessionStatus.Succeeded)
|
||||
return;
|
||||
|
||||
var healed = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, SessionStatus.Succeeded,
|
||||
VerdictPaths.Compose(VerdictPaths.OriginLateReconcile, rule),
|
||||
failureReason: reason,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
|
||||
if (healed)
|
||||
@@ -368,6 +382,49 @@ namespace AutopilotMonitor.Functions.Services
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// WhiteGlove Part-2 awaiting-user gate for the agent-reported stall
|
||||
/// (<c>session_stalled</c>). Pre-provisioned sessions with a ResumedAt whose event
|
||||
/// stream shows no user evidence since the resume are parked as AwaitingUser instead of
|
||||
/// Stalled — same evidence rule as the maintenance sweep
|
||||
/// (<see cref="EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser"/>). Returns true
|
||||
/// when the AwaitingUser write was applied; false (also on any error, fail-soft) lets
|
||||
/// the caller run the normal Stalled mapping.
|
||||
/// </summary>
|
||||
private async Task<bool> TryMapStallToWhiteGloveAwaitingUserAsync(
|
||||
IngestEventsRequest request, string sessionPrefix, EventClassification c)
|
||||
{
|
||||
try
|
||||
{
|
||||
var session = await _sessionRepo.GetSessionAsync(request.TenantId, request.SessionId);
|
||||
if (session?.IsPreProvisioned != true || !session.ResumedAt.HasValue)
|
||||
return false;
|
||||
|
||||
var sessionEvents = await _sessionRepo.GetSessionEventsAsync(
|
||||
request.TenantId, request.SessionId, maxResults: 1000);
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
|
||||
if (!EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(
|
||||
rollup, isPreProvisioned: true, session.ResumedAt))
|
||||
return false;
|
||||
|
||||
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, SessionStatus.AwaitingUser, VerdictPaths.IngestWhiteGloveAwaiting,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
|
||||
failureReason: EnrollmentTimeoutClassifier.WhiteGloveAwaitingUserReason(rollup));
|
||||
if (transitioned)
|
||||
_logger.LogInformation(
|
||||
"{SessionPrefix} Status: AwaitingUser (agent stall probe on WhiteGlove Part 2 with no user evidence since resume)",
|
||||
sessionPrefix);
|
||||
return true;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"{SessionPrefix} WhiteGlove awaiting-user gate failed for session_stalled; falling back to Stalled", sessionPrefix);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private static bool IsPeriodicOrStallEvent(string? eventType) => eventType switch
|
||||
{
|
||||
"performance_snapshot" => true,
|
||||
@@ -410,7 +467,9 @@ namespace AutopilotMonitor.Functions.Services
|
||||
? c.CompletionEvent.Data["completionSource"]?.ToString() : null;
|
||||
|
||||
statusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, SessionStatus.Succeeded, c.CompletionEvent.Phase,
|
||||
request.TenantId, request.SessionId, SessionStatus.Succeeded,
|
||||
espSoftFailure ? VerdictPaths.AgentCompleteSoft : VerdictPaths.AgentComplete,
|
||||
c.CompletionEvent.Phase,
|
||||
completedAt: c.CompletionEvent.Timestamp,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
|
||||
espSoftFailure: espSoftFailure, completionSource: completionSource);
|
||||
@@ -437,7 +496,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
: c.FailureEvent.Message;
|
||||
|
||||
statusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, SessionStatus.Failed, c.FailureEvent.Phase, failureReason,
|
||||
request.TenantId, request.SessionId, SessionStatus.Failed, VerdictPaths.AgentFailed,
|
||||
c.FailureEvent.Phase, failureReason,
|
||||
completedAt: c.FailureEvent.Timestamp,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
|
||||
_logger.LogWarning("{SessionPrefix} Status: Failed - {FailureReason} (transitioned={Transitioned})", sessionPrefix, failureReason, statusTransitioned);
|
||||
@@ -447,7 +507,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
{
|
||||
failureReason = c.EspFailureEvent.Message ?? "ESP failure (backend fallback)";
|
||||
statusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, SessionStatus.Failed, c.EspFailureEvent.Phase, failureReason,
|
||||
request.TenantId, request.SessionId, SessionStatus.Failed, VerdictPaths.AgentEspFailureFallback,
|
||||
c.EspFailureEvent.Phase, failureReason,
|
||||
completedAt: c.EspFailureEvent.Timestamp,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
|
||||
_logger.LogWarning("{SessionPrefix} Status: Failed via esp_failure fallback - {FailureReason} (transitioned={Transitioned})",
|
||||
@@ -456,7 +517,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
else if (c.GatherCompletionEvent != null)
|
||||
{
|
||||
await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, SessionStatus.Succeeded, c.GatherCompletionEvent.Phase,
|
||||
request.TenantId, request.SessionId, SessionStatus.Succeeded, VerdictPaths.AgentGatherComplete,
|
||||
c.GatherCompletionEvent.Phase,
|
||||
completedAt: c.GatherCompletionEvent.Timestamp,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
|
||||
_logger.LogInformation("{SessionPrefix} Status: Succeeded (gather_rules)", sessionPrefix);
|
||||
@@ -464,7 +526,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
else if (c.WhiteGloveEvent != null)
|
||||
{
|
||||
whiteGloveStatusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, SessionStatus.Pending, EnrollmentPhase.AppsDevice,
|
||||
request.TenantId, request.SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
|
||||
EnrollmentPhase.AppsDevice,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
|
||||
isPreProvisioned: true, isUserDriven: false);
|
||||
|
||||
@@ -485,7 +548,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
_logger.LogWarning("{SessionPrefix} WhiteGlove UpdateSessionStatusAsync failed, attempting unconditional fallback for IsPreProvisioned + Status", sessionPrefix);
|
||||
try
|
||||
{
|
||||
await _sessionRepo.SetSessionPreProvisionedAsync(request.TenantId, request.SessionId, true, SessionStatus.Pending, isUserDriven: false);
|
||||
await _sessionRepo.SetSessionPreProvisionedAsync(request.TenantId, request.SessionId, true, SessionStatus.Pending, isUserDriven: false, verdictPath: VerdictPaths.AgentWhiteGlovePending);
|
||||
whiteGloveStatusTransitioned = true;
|
||||
_logger.LogInformation("{SessionPrefix} WhiteGlove fallback succeeded: IsPreProvisioned + Status=Pending set via unconditional merge", sessionPrefix);
|
||||
}
|
||||
@@ -525,7 +588,8 @@ namespace AutopilotMonitor.Functions.Services
|
||||
if (currentSession?.Status == SessionStatus.Pending)
|
||||
{
|
||||
await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, SessionStatus.InProgress, c.WhiteGloveResumedEvent.Phase,
|
||||
request.TenantId, request.SessionId, SessionStatus.InProgress, VerdictPaths.AgentWhiteGloveResumed,
|
||||
c.WhiteGloveResumedEvent.Phase,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
|
||||
isUserDriven: true, resumedAt: c.WhiteGloveResumedEvent.Timestamp);
|
||||
_logger.LogInformation("{SessionPrefix} Status: InProgress (WhiteGlove Part 2 resumed, IsUserDriven=true)", sessionPrefix);
|
||||
@@ -559,9 +623,19 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
if (c.SessionStalledEvent != null)
|
||||
{
|
||||
// WhiteGlove Part-2 awaiting-user gate (fairstone.ca analysis 2026-08-21): the
|
||||
// agent-side stall probe also fires on a resumed Part-2 device that simply sits
|
||||
// at the logon screen with nobody signing in. With zero user evidence since the
|
||||
// resume that is the expected parking state between technician and end user, so
|
||||
// the honest label is AwaitingUser, not Stalled. Rare path (one probe batch per
|
||||
// stalling session), so the session + events reads are acceptable; fail-soft —
|
||||
// any error falls back to the plain Stalled mapping.
|
||||
if (await TryMapStallToWhiteGloveAwaitingUserAsync(request, sessionPrefix, c))
|
||||
return (statusTransitioned, whiteGloveStatusTransitioned, failureReason);
|
||||
|
||||
var stalledReason = "Agent reported stall after 60min without progress (stall_probe)";
|
||||
var stalledTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, SessionStatus.Stalled,
|
||||
request.TenantId, request.SessionId, SessionStatus.Stalled, VerdictPaths.AgentStallProbe,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
|
||||
stalledAt: c.SessionStalledEvent.Timestamp, failureReason: stalledReason);
|
||||
if (stalledTransitioned)
|
||||
@@ -584,7 +658,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
if (currentStatus == SessionStatus.Stalled)
|
||||
{
|
||||
var healed = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
request.TenantId, request.SessionId, SessionStatus.InProgress,
|
||||
request.TenantId, request.SessionId, SessionStatus.InProgress, VerdictPaths.AgentStallHeal,
|
||||
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
|
||||
clearStalledAt: true, clearFailureReason: true);
|
||||
if (healed)
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
namespace AutopilotMonitor.Functions.Services;
|
||||
|
||||
/// <summary>
|
||||
/// Provider-neutral transactional email contract for the tenant-activation welcome mail.
|
||||
/// <para>
|
||||
/// Always fail-soft — implementations must NOT throw; activation correctness never depends
|
||||
/// on email delivery (the Global Admin "Send welcome email" button is the manual fallback).
|
||||
/// The farewell mail after offboarding lives on the sibling
|
||||
/// <see cref="IOffboardFarewellEmailSender"/> contract; both are served by <see cref="EmailService"/>.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public interface IEmailService
|
||||
{
|
||||
/// <summary>
|
||||
/// Sends the tenant-activation welcome email. No-op when the provider is not configured
|
||||
/// or the recipient is empty.
|
||||
/// </summary>
|
||||
Task SendPreviewApprovedEmailAsync(string toEmail, string domainName, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Operator test send: the effective template of <paramref name="kind"/> (or an unsaved
|
||||
/// <paramref name="draftHtml"/>, placeholder-rendered) to <paramref name="toEmail"/>.
|
||||
/// Returns true when the provider accepted the message. Never throws.
|
||||
/// </summary>
|
||||
Task<bool> SendTestAsync(EmailTemplateKind kind, string toEmail, string domainName, string? draftHtml, CancellationToken ct = default);
|
||||
}
|
||||
@@ -18,6 +18,11 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// registered a LATER session (Part 2 under a fresh session id, or re-provisioning) are
|
||||
/// resolved as Incomplete("Superseded by ..."). Forward-looking coverage lives in the
|
||||
/// registration supersede pass; this mode clears the existing backlog.</item>
|
||||
/// <item><b>self_deploying_silent</b> — self-deploying-profile rows parked as
|
||||
/// Incomplete / AwaitingUser / Stalled by the pre-2026-08-23 classifier (which had no
|
||||
/// notion of a profile without a user phase) are re-run through the classifier; only a
|
||||
/// Succeeded verdict (<see cref="EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned"/>)
|
||||
/// is applied, everything else keeps its verdict.</item>
|
||||
/// </list>
|
||||
/// Hand-marked sessions (AdminMarkedAction) are never touched.
|
||||
/// </summary>
|
||||
@@ -30,6 +35,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
public const string ModeLegacyTimeouts = "legacy_timeouts";
|
||||
public const string ModePendingOrphans = "pending_orphans";
|
||||
public const string ModeSelfDeployingSilent = "self_deploying_silent";
|
||||
private const int MaxSamplesInResponse = 100;
|
||||
|
||||
public LegacyReclassificationService(
|
||||
@@ -108,8 +114,10 @@ namespace AutopilotMonitor.Functions.Services
|
||||
var events = await _sessionRepo.GetSessionEventsAsync(tenantId, session.SessionId, maxResults: 1000);
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
|
||||
var effectiveStart = session.ResumedAt ?? session.StartedAt;
|
||||
var (target, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session.LastEventAt);
|
||||
var (target, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session.LastEventAt,
|
||||
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
|
||||
isSelfDeployingProfile: session.IsSelfDeployingProfile);
|
||||
|
||||
if (target == SessionStatus.Failed)
|
||||
{
|
||||
@@ -137,7 +145,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
}
|
||||
|
||||
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
tenantId, session.SessionId, target,
|
||||
tenantId, session.SessionId, target, VerdictPaths.Compose(VerdictPaths.OriginRetro, rule),
|
||||
failureReason: reason,
|
||||
failureSource: target == SessionStatus.Incomplete ? "legacy_timeout_retro_reconcile" : null,
|
||||
allowTerminalReclassification: true);
|
||||
@@ -164,6 +172,99 @@ namespace AutopilotMonitor.Functions.Services
|
||||
return result;
|
||||
}
|
||||
|
||||
public async Task<ReclassificationResult> ReconcileSelfDeployingSilentAsync(
|
||||
string? tenantIdScope, bool dryRun, int maxSessions, string triggeredBy)
|
||||
{
|
||||
var result = new ReclassificationResult { Mode = ModeSelfDeployingSilent, DryRun = dryRun };
|
||||
var now = DateTime.UtcNow;
|
||||
|
||||
foreach (var tenantId in await ResolveTenantsAsync(tenantIdScope))
|
||||
{
|
||||
if (result.SessionsExamined >= maxSessions) { result.CapReached = true; break; }
|
||||
result.TenantsExamined++;
|
||||
|
||||
int? tenantGraceHours = null, absoluteMaxHours = null;
|
||||
try
|
||||
{
|
||||
var config = await _configService.GetConfigurationAsync(tenantId);
|
||||
tenantGraceHours = config?.SessionGraceHours;
|
||||
absoluteMaxHours = config?.AbsoluteMaxSessionHours;
|
||||
}
|
||||
catch { /* defaults */ }
|
||||
var graceHours = EnrollmentTimeoutClassifier.ResolveGraceHours(tenantGraceHours, absoluteMaxHours);
|
||||
|
||||
var candidates = await _maintenanceRepo.GetSelfDeployingSilentSessionsAsync(
|
||||
tenantId, maxSessions - result.SessionsExamined + 1);
|
||||
if (candidates.Count > maxSessions - result.SessionsExamined)
|
||||
{
|
||||
result.CapReached = true;
|
||||
candidates = candidates.Take(maxSessions - result.SessionsExamined).ToList();
|
||||
}
|
||||
|
||||
foreach (var session in candidates)
|
||||
{
|
||||
result.SessionsExamined++;
|
||||
try
|
||||
{
|
||||
if (!string.IsNullOrEmpty(session.AdminMarkedAction))
|
||||
{
|
||||
result.Skipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
var events = await _sessionRepo.GetSessionEventsAsync(tenantId, session.SessionId, maxResults: 1000);
|
||||
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
|
||||
var effectiveStart = session.ResumedAt ?? session.StartedAt;
|
||||
var (target, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
|
||||
rollup, effectiveStart, now, graceHours, session.LastEventAt,
|
||||
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
|
||||
isSelfDeployingProfile: session.IsSelfDeployingProfile);
|
||||
|
||||
// Heal-only: a verdict other than Succeeded means the device never reached
|
||||
// Device ESP all-succeeded (or failed explicitly) — the existing label stands.
|
||||
if (target != SessionStatus.Succeeded)
|
||||
{
|
||||
result.Skipped++;
|
||||
continue;
|
||||
}
|
||||
|
||||
RecordSample(result, tenantId, session, target, reason, oldStatus: session.Status.ToString());
|
||||
|
||||
if (dryRun)
|
||||
{
|
||||
result.WouldChange++;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Succeeded may upgrade Incomplete / AwaitingUser / Stalled through the normal
|
||||
// reconcile gate (no terminal-reclassification override needed); the storage
|
||||
// layer additionally refuses admin-marked rows.
|
||||
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
tenantId, session.SessionId, SessionStatus.Succeeded, VerdictPaths.RetroSelfDeployingReconcile,
|
||||
failureReason: reason);
|
||||
if (transitioned)
|
||||
{
|
||||
result.Changed++;
|
||||
result.ToSucceeded++;
|
||||
}
|
||||
else
|
||||
{
|
||||
result.Skipped++;
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
result.Errors++;
|
||||
_logger.LogWarning(ex, "Self-deploying retro-reconcile failed for session {SessionId} (tenant {TenantId})",
|
||||
session.SessionId, tenantId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
await WriteAuditAsync(result, tenantIdScope, triggeredBy);
|
||||
return result;
|
||||
}
|
||||
|
||||
public async Task<ReclassificationResult> ResolvePendingOrphansAsync(
|
||||
string? tenantIdScope, bool dryRun, int maxSessions, string triggeredBy)
|
||||
{
|
||||
@@ -213,7 +314,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
|
||||
// Pending is non-terminal — the standard transition guard covers this write.
|
||||
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
|
||||
tenantId, pending.SessionId, SessionStatus.Incomplete,
|
||||
tenantId, pending.SessionId, SessionStatus.Incomplete, VerdictPaths.RetroSuperseded,
|
||||
failureReason: reason, failureSource: "superseded_by_reregistration");
|
||||
if (transitioned) { result.Changed++; result.ToIncomplete++; }
|
||||
else result.Skipped++;
|
||||
|
||||
@@ -289,14 +289,13 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// Sessions with a deletion cascade in flight are skipped so the sweep cannot write a
|
||||
/// breakdown row after the session's deletion manifest was snapshotted.
|
||||
/// </summary>
|
||||
private async Task SweepTimeAttributionAsync()
|
||||
private async Task SweepTimeAttributionAsync(IReadOnlyList<SessionSummary> sweepWindow)
|
||||
{
|
||||
try
|
||||
{
|
||||
var sw = Stopwatch.StartNew();
|
||||
var today = DateTime.UtcNow.Date;
|
||||
var sessions = await _maintenanceRepo.GetSessionsByDateRangeAsync(
|
||||
today.AddDays(-TimeAttributionSweepDays), today.AddDays(1));
|
||||
var sessions = SliceSweepWindow(sweepWindow, today.AddDays(-TimeAttributionSweepDays));
|
||||
|
||||
var pairs = new List<(SessionSummary Session, SessionTimeBreakdown Breakdown)>();
|
||||
var missing = new List<SessionSummary>();
|
||||
@@ -1102,6 +1101,19 @@ namespace AutopilotMonitor.Functions.Services
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to cleanup old device-journey aggregates");
|
||||
}
|
||||
|
||||
// Verdict-calibration daily aggregates: same 180d window (regenerable; the radar
|
||||
// needs 35d, the matrix serves up to 180d).
|
||||
try
|
||||
{
|
||||
var deleted = await _metricsRepo.DeleteVerdictCalibrationAggregatesOlderThanAsync(now.AddDays(-usageMetricsRetentionDays));
|
||||
if (deleted > 0)
|
||||
_logger.LogInformation("Verdict-calibration aggregate cleanup: deleted {Count} rows older than {Days} days", deleted, usageMetricsRetentionDays);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex, "Failed to cleanup old verdict-calibration aggregates");
|
||||
}
|
||||
}
|
||||
/// <summary>
|
||||
/// Detects and cleans up orphaned events — events stored in the Events table
|
||||
|
||||
+2
-2
@@ -36,7 +36,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
/// per window device per tick — same order as the F1 sweep; negligible at current fleet
|
||||
/// size (the scaling lever is gating the per-device RMW on new terminal sessions).
|
||||
/// </summary>
|
||||
private async Task SweepDeviceJourneysAsync()
|
||||
private async Task SweepDeviceJourneysAsync(IReadOnlyList<SessionSummary> sweepWindow)
|
||||
{
|
||||
try
|
||||
{
|
||||
@@ -84,7 +84,7 @@ namespace AutopilotMonitor.Functions.Services
|
||||
}
|
||||
|
||||
// ---- Phase 2: merge the window's terminal sessions into their device chains. ----
|
||||
var sessions = await _maintenanceRepo.GetSessionsByDateRangeAsync(windowStart, windowEnd);
|
||||
var sessions = SliceSweepWindow(sweepWindow, windowStart);
|
||||
var excludedSessions = new List<SessionSummary>();
|
||||
var byDevice = new Dictionary<(string TenantId, string SerialKey), List<SessionSummary>>();
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user