Compare commits

...

49 Commits

Author SHA1 Message Date
Oliver Kieselbach c26867ced8 fix(agent): network-link gate before initial config fetch + post-registration recovery
Root cause (edu.kosh audit 2026-08-24, session ba828038): on boot-time relaunches
the agent starts before the Wi-Fi link is up. The Phase 4 config fetch had no
network gate (98f8114d only guarded Phase 6 registration), burned its 10/30/60s
retry budget into a dead link and stranded the whole session on built-in defaults
(ConfigVersion=0) — up to ~50% of that tenant's sessions, with no self-heal short
of a manual rotate_config action. Not a version regression: rates varied wildly
per day within the same agent build; the failed requests never reached the backend.

Two complementary layers:
- RemoteConfigService gates the initial fetch on the shared NetworkLinkWait poll
  (60s cap — longer than registration's 15s because the fetch runs first), so the
  retry budget starts on a live link. rotate_config stays ungated (latency-bound).
- Phase 6.5: after registration proves the backend reachable, TryRecoverConfigAsync
  refetches once and AgentRuntimeConfig.TryRecoverAfterRegistration re-applies the
  live config (same merge/integrity pipeline as Phase 4). LastFetch* keeps the
  original failure record; remote_config_fetch_failed reports it with a recovered
  flag at Info severity, agent_started flags remoteConfigRecovered.

Deliberately not re-run in recovery: kill-signal check (telemetry-channel control
signals cover post-registration kills) and endpoint migration (next start).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-24 11:49:37 +02:00
Oliver Kieselbach 3af94d9dcd infra(telemetry): DCR also drops host-emitted successful SignalR dependencies
Live-verified after 0d5e2572 deploy: the worker filter removes every 404/412/409
and worker SignalR row, but the SignalR output-binding calls are emitted by the
host process (SDK rdddsc) and only reachable at ingestion.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 19:15:41 +02:00
Oliver Kieselbach 0d5e2572aa perf(telemetry): one canonical request record; drop expected storage outcomes and SignalR success deps
- RequestTelemetryMiddleware: SamplingPercentage=100 per item (worker copy was
  adaptively sampled, ItemCount 2-5 on weekdays, despite host.json excludedTypes)
- StorageDependencyFilterProcessor: also drop 404/412/409 storage outcomes
  (InProc + HTTP shape) and successful *.service.signalr.net REST calls
- infra/appinsights-workspace-transforms.bicep: DCR dropping the host HTTP
  request duplicate at ingestion (timer/queue rows kept)
- docs/backend/telemetry-ingest-shaping.md; skills updated

Measured baseline 7d: ~1.3 GB/week removed, requests now 100% with tenant context.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 18:31:32 +02:00
Oliver Kieselbach 793e907e60 fix(cveindex): merge findings per CVE before upsert; bounded parallelism
One CVE commonly appears under many findings of the same report (e.g. one
RD CVE across 14 installed RD components). Each occurrence produced its own
concurrent upsert on the same PK/RK, so the stored SoftwareName/OverallRisk
was race-dependent. Collapse to one entity per CVE (highest CVSS wins,
IsKev OR-ed, OverallRisk max) and write with Parallel.ForEachAsync(8) and
per-entity fail-soft logging instead of Task.WhenAll + one outer catch.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 17:52:02 +02:00
Oliver Kieselbach 8e8bb66371 test(agent): thread-safe FakeSignalIngressSink (parked-tripwire CI flake)
Timer-thread Post() appended to a bare List while the test polled it in a
SpinWait foreach -> 'Collection was modified' (run 32644425162, no assert
text). Posted now returns a locked snapshot.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 16:28:58 +02:00
Oliver Kieselbach 7e1389cf32 fix(ratelimit): keep history and lock in one cache entry; drop leaking per-key lock map
Both sliding-window limiters re-Set the cache entry without its eviction callback.
The replaced entry fired the callback asynchronously, removing the per-key lock while
a new one was already handed out, so two threads could mutate the same List under
different locks; the replacement entry never cleaned its lock, leaking one object per
key for the process lifetime.

The history now lives in a RequestBucket together with its own lock: same list ⇒ same
lock, no callback, no dictionary. Bucket creation is serialized (GetOrCreate is not
atomic) behind a lock-free TryGetValue fast path. The redundant re-Set is gone — the
lookup already refreshes the sliding expiration.

Concurrency tests (16 threads × 500 requests, one key) fail 3/3 on the old code and
pass on the new one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 16:26:44 +02:00
Oliver Kieselbach 3a89319472 feat(telemetry): startup duration metrics for backend and MCP
StartupTelemetryService emits BackendStartupMs (process start ->
ApplicationStarted) and BackendTableInitMs as App Insights custom
metrics with version/commit/tableInitFullPass/instance dimensions;
TableStorageService exposes LastInitialization. MCP server writes one
{"type":"boot","readyMs"} stderr line per start next to the
tool_call lines so cold-start time is queryable as a trend.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 16:04:31 +02:00
Oliver Kieselbach b37e6f975c perf(startup): table schema sentinel gates CreateTableIfNotExists pass
Cold start point-reads a SHA-256 over TableNames.All stored in
AdminConfiguration/SchemaSentinel/tables instead of issuing 66
CreateTableIfNotExists calls. Missing/stale sentinel runs the full
idempotent pass once and rewrites it. SessionsIndex startup backfill
only after a full pass, behind a conditional-insert claim (one instance
under scale-out); IsSessionIndexEmptyAsync no longer treats errors as
empty. Daily maintenance runs IStorageInitializer.EnsureAllAsync to
repair out-of-band table deletions.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 15:33:45 +02:00
Oliver Kieselbach cacbd24667 feat(mcp): get_verdict_calibration response shaping (explicit nulls, minSharePct/top, omitted report)
Drop per-row window7/baseline28.sessions (duplicated trend denominators), make
reEnrollRatePct/lift always present with null = withheld, add minSharePct/top
tail trimming that never drops override-carrying rows and reports what was
omitted. Backend wire shape and admin page untouched.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 12:51:17 +02:00
Oliver Kieselbach a157d98a40 fix(radar): verdict share regression only for backend-decided paths
First production pass alerted on agent:whiteglove_pending (two tenants in a
WhiteGlove rollout week) and agent:complete_soft (a Continue-Anyway tenant) —
customer workflow mix, not classifier drift. agent:* and register:* are now
ineligible for share_regression (still in the matrix and the silence group);
episodes opened by the broader gate re-arm on the next pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 12:32:00 +02:00
Oliver Kieselbach 7fa50d09c1 perf(maintenance): one projected 35d Sessions scan shared by all rolling sweeps
The cross-tenant StartedAt-only date-range query is a full-table drain in
Table Storage; time attribution, device journeys, verdict calibration and the
calibration radar's tenant discovery each ran their own — four identical reads
per 2h tick. RunAllAsync / RunManualAsync now load the window once through
GetMaintenanceWindowSessionsAsync (MaintenanceSweepSessionProjection: the union
of the sweeps' columns, drops FailureSnapshotJson/geo/OS/diagnostics) and the
sweeps slice it by StartedAt. MaintenanceSweepProjectionEquivalenceTests pins
that every consumed field survives the projection.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 12:19:41 +02:00
Oliver Kieselbach 94afc02e62 docs(okf): verdict calibration concept (VerdictPath, prior verdicts, aggregate, radar)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:40:34 +02:00
Oliver Kieselbach 5e4c5e80be feat: verdict calibration drift radar (share / silence / evidence-gap) with ops events
Applies the rule-regression statistics (7d vs 28d, lift >=2, Wilson-separated,
small-n gates) to the verdict-calibration rows: per-path share regression,
sweep+maxlife silence-share regression (agent liveness signal) and an
absolute r6 evidence-gap gate. Episodes live in the notification tracker
(verdictcalibration|kind|path|status), fire one VerdictCalibrationDrift ops
event (operator-only, no tenant bell), re-arm under 1.5x / 15%, and surface as
alerts[] + a Drift pill in the calibration matrix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:39:06 +02:00
Oliver Kieselbach b535973531 feat: verdict calibration aggregate, GA endpoint, MCP tool and operator matrix
Daily (tenant x StartedAt date) buckets per verdict path: count, derived
attribution, 7-day re-enrollment proxy (eligible/re-enrolled), and the
correction stream attributed to the PRIOR path (admin / late completion /
other). Written by the 2h maintenance sweep after the device-journey pass,
180d retention, regenerable. Read via GET global/metrics/verdict-calibration
(GlobalReadOrAdmin, 7d-vs-28d share trend, rates withheld below n=20),
the get_verdict_calibration MCP tool, and the admin metrics section.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:22:17 +02:00
Oliver Kieselbach 7e0c578a59 feat(backend): VerdictPath + PriorStatus/PriorVerdictPath on every session status write
Every status writer now declares the code path that produced the status
(VerdictPaths vocabulary, stamped for all statuses — unlike FailureSource
which is Failed-only), the timeout classifier returns its rule id, and a
write that overrides a prior verdict preserves it as PriorStatus/
PriorVerdictPath (admin mark, late completion, retro). Pre-instrumentation
rows derive a path read-side (VerdictPathDerivation, legacy:* when
ambiguous). Foundation for the verdict calibration aggregate + radar.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 11:06:37 +02:00
Oliver Kieselbach 1257e0d195 Merge branch 'main' of https://github.com/okieselbach/Autopilot-Monitor 2026-08-23 09:57:39 +02:00
Oliver Kieselbach 98f8114d90 fix(classification): self-deploying profiles reconcile to Succeeded after Device Setup, never AwaitingUser
Kiosk tenant audit 2026-08-23: 836 of ~970 self-deploying sessions in one week
were parked AwaitingUser ("Account Setup 0/5") and graduated to Incomplete 51h
later although the devices were finished — the profile has no user phase, so the
classifier's rule 5 was asking for evidence that can never arrive.

- EnrollmentTimeoutClassifier: new rule 1c IsSelfDeployingProvisioned (profile flag +
  Device Setup all-succeeded + no explicit failure) => Succeeded reconcile with an
  honest reason; isSelfDeployingProfile passed by every caller (sweep stage 1+2,
  max-lifetime verdict, late-telemetry heal, legacy retro mode).
- MaintenanceService stage 1: TryReconcileSelfDeployingAsync gate before the Stalled
  marker (mirrors the WhiteGlove Part-2 gate), audit counter SessionsSelfDeployingReconciled.
- LegacyReclassificationService: new admin mode self_deploying_silent (heal-only,
  dry-run by default) + GetSelfDeployingSilentSessionsAsync for the existing backlog.
- Agent SessionRegistrationHelper: wait up to 15s for a network link and retry 6x
  (~62s) instead of 5x (~30s) so the BootTrigger relaunch on Wi-Fi no longer exits 7
  before the link is back (the mechanism that left these sessions silent).
- Tests: 6 classifier cases (gate, within/past grace, user-driven guard, failure,
  pre-Device-Setup), 2 registration tests; docs/backend/silent-session-classification.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 09:55:06 +02:00
Oliver Kieselbach 7b3230d507 fix(web): mobile wrapping for App Details header, Software tables and SLA banner (#133) 2026-08-23 05:19:20 +02:00
Oliver Kieselbach 87b7a20223 fix(web): diagnostics path rows wrap the full path on narrow screens
Below `sm` the path owns the first line(s) and wraps in full (flex-wrap,
basis-full, break-all) with badges and controls flowing onto the next line —
a path truncated to "C:\…" is useless on a phone and a tooltip cannot be
hovered there. From `sm` up the row stays a single truncated line. The
"N file types" pill is shown on every width again now that the badge line
has room.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 04:30:10 +02:00
Oliver Kieselbach 4c19be1de4 feat(web): drop Experimental/Pre-Release badges from Software Inventory & Vulnerability Analyzer
Settings section loses both pills; the vulnerability report footer keeps the
best-effort CVE matching note without the Experimental prefix.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 04:26:55 +02:00
Oliver Kieselbach 93de130cd1 feat(diagnostics): built-in collection catalog visible in the portal, RealmJoin + Device Preparation sections gated
The diagnostics package's built-in collection is now ONE catalog in Shared
(DiagnosticsBuiltInSections.All): the agent iterates it when building the ZIP and
the backend serves the same list to the portal (GET /api/diagnostics/paths,
MemberRead), so what administrators see is exactly what the agent collects.

Agent
- New conditional sections: RealmJoin client/package/Chocolatey/per-user logs
  behind the tenant's RealmJoin Watcher toggle (mirrored onto
  AgentConfiguration.EnableRealmJoinWatcher by RemoteConfigMerger) and the Device
  Preparation bootstrapper event log behind the deterministic WDP marker (probe
  once per build, fail toward Classic). ZIP layout mirrors the disk
  (RealmJoinLogs/Windows|Choco|User). Every skip is a package-manifest line.
- Built-ins bypass DiagnosticsPathGuards by design (reviewed code); caps,
  reparse-point skip and evtx export still apply. No ConfigVersion bump.

Backend
- GET diagnostics/paths (MemberRead, JWT-scoped) with the catalog + global paths.
- MergeDiagnosticsLogPaths: global ∪ tenant dedup (trimmed, case-insensitive) —
  duplicates used to produce duplicate ZIP entry names.

Web
- Collapsed read-only built-in list (RealmJoin on/off in the tenant header,
  neutral in the Global-Admin view), one-line DiagnosticsPathRow for every entry,
  add-row above the lists; tenant context no longer fetches /api/global/config.

Tests: agent RealmJoin on/off, no user session, bootstrapper evtx, probe failure,
merger mirror; backend catalog invariants, payload shape, dedup, policy; web
builtInSectionDisplay. Docs: docs/agent/diagnostics-package.md (+index/log).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 04:09:50 +02:00
Oliver Kieselbach b8b95c7e76 refactor(web): move RealmJoin Watcher and Keep Awake from Agent Analyzers to Agent Settings
Both toggles gate collector hosts / device behavior, not IAgentAnalyzers, and
tenants looked for them under Agent Settings. RealmJoin Watcher now sits in the
Agent Collectors card, Keep Awake During User-ESP in Agent Parameters; OOBE
console detection stays with the security analyzers. Section ownership in
SECTION_FIELD_MAP and the per-section reset handlers follow, the Agent Settings
Save/Reset bar moves below both cards it saves, and the Analyzers sidebar
description names all four checks. Wire contract (AnalyzerConfiguration) and
agent are unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 04:06:39 +02:00
Oliver Kieselbach e57e081cec docs(web): Discord in notification-provider copy on about, privacy, settings and rule card
Privacy page last-updated bumped to 23 August 2026.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 02:47:52 +02:00
Oliver Kieselbach 3a60186824 revert(email): drop the gradient slant row and corner cell — solid fallbacks made non-gradient clients worse
Back to the 838da4cc header: one facet gradient, three-row brand mark, 113px
band, clip-path edge for clients that support it. Clients without gradient
support get a flat band instead of band-coloured blocks.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 02:30:40 +02:00
Oliver Kieselbach c35595503f feat(email): header slant and light corner facet without clip-path — gradients Outlook can draw
Row 1 keeps the single facet gradient and adds a top-right cell with its own
dark/light split (the corner facet); row 2 draws the slanted bottom edge as a
green/white gradient in three band-coloured cells. Pixel-measured against the
reference render: band boundaries identical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 02:22:44 +02:00
Oliver Kieselbach 838da4cc8a fix(email): single Outlook-safe facet gradient, no bgcolor inside the brand mark
Outlook web/iOS render exactly one linear-gradient with hex colours and % stops
(multiple layers / rgba stops are dropped), and dark mode inverts cell
backgrounds but not border colours — the white cell in the mark became a black
bar. Facets are now one gradient (top-right light corner dropped), the mark's
middle row is a white-bordered div.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 02:11:17 +02:00
Oliver Kieselbach a9937e0f5d fix(email): brand mark as three non-overlapping rows — Outlook ignores negative margins
Triangle / white cell + triangle / triangle, no overlays, no transparent td
borders; matches the reference mark in a side-by-side zoom render.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 02:02:14 +02:00
Oliver Kieselbach 1dc2cc9a54 fix(email): header rebuilt 1:1 from the approved banner — faceted gradients and four-edge brand mark
Verified side-by-side against the reference render (scratch header-render.html):
three layered half-plane gradients reproduce the light/dark facets, the mark is
a CSS trapezoid plus two green wedges for the slanted right edge and the
descending bottom edge. Still no images in the mail.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 01:44:26 +02:00
Oliver Kieselbach dd7ae299a8 fix(email): header restored to the approved rendering (e1de5a89 geometry)
112deg stripe gradient, 13/20/2 border-drawn mark, clip-path edge — the version
the user signed off on; later 'improvements' drifted from it.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 01:23:22 +02:00
Oliver Kieselbach d96d10ccec fix(email): header matches the reference banner — mark 12/21/0 trapezoid, stripes thin/wide/thin in the proven 8-stop gradient form
Verified against the original SVG mark in a Chromium render; the gradient keeps
the exact stop count that rendered in Outlook earlier.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 01:05:12 +02:00
Oliver Kieselbach 457fff65f5 fix(email): brand mark back to the div border trick (Outlook paints transparent td borders white)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:56:09 +02:00
Oliver Kieselbach dc45694611 docs(okf): email template overrides and image-free mail header
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:50:41 +02:00
Oliver Kieselbach 7569061983 feat(admin): editable email templates — preview, test send and HTML override in Settings
- EmailTemplateService: override-or-built-in resolution per send (PreviewConfig
  table, partition EmailTemplates, 5-min cache, {{domainName}} placeholder,
  30k limit); send path fail-soft, admin read path strict
- GA endpoints global/email-templates/{kind} (GET/PUT/DELETE) and /test, which
  mails the effective template or an unsaved draft to the caller's tenant
  contact address
- portal: Settings → Email Templates with preview frame, test send, HTML editor
- mail header: brand mark drawn with CSS borders, no images left in the mails

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:50:10 +02:00
Oliver Kieselbach 0f5e36db94 fix(email): exact brand mark as PNG again, stripes direction and widths match the reference
The CSS border trapezoid did not reproduce the mark's shape across clients; the
mark is brand identity, so it is the one image in the mail (empty alt). Stripes
now run top-left to bottom-right, thin/wide/thin as in the reference banner.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:40:15 +02:00
Oliver Kieselbach e1de5a8971 feat(email): image-free header — CSS stripes, slanted edge and border-drawn brand mark
Progressive enhancement: solid bgcolor band + text wordmark everywhere; the
brand mark is a CSS border trapezoid; diagonal stripes (linear-gradient) and the
slanted bottom edge (clip-path) render where supported and degrade to a flat
band elsewhere. No remote images left in either mail.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:22:26 +02:00
Oliver Kieselbach a2d93e884c fix(email): native HTML header that survives blocked images
Solid bgcolor band with the wordmark as real text; only the 36px brand mark is
an image (empty alt). Replaces the full-width header PNG, which collapsed to a
bare green bar on clients that block remote images (iOS Mail).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-23 00:15:45 +02:00
Oliver Kieselbach 425229deee feat(email): branded header image, sign-in link, drop preview banner in welcome/farewell mails
- header is a hosted PNG (www/email/header.png, rendered from the brand mark +
  wordmark) on a solid bgcolor band; mail clients drop CSS gradients, which left
  the old header white
- welcome mail links "sign in" to the portal, active-development banner removed
- em dashes replaced with en dashes

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 23:55:53 +02:00
Oliver Kieselbach 8675e7ee06 feat(backend): provider-neutral EmailService on Mailchimp Transactional (Mandrill), replaces Resend
- EmailService + IEmailService: Mandrill messages/send via typed HttpClient with the
  Notification resilience policy; provider confined to one transport method and the
  Email:* settings (ApiKey, Endpoint, FromAddress, FromName). Resend NuGet dropped.
- open/click tracking explicitly off; sent/queued/scheduled = ok, rejected/invalid = warn
- callers (TenantApprovalService, PreviewWhitelistFunction) depend on IEmailService
- EmailServiceTests pin wire format, result semantics and fail-soft contract
- privacy page + OKF welcome-mail doc updated for the new sub-processor

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 23:26:32 +02:00
Oliver Kieselbach e857e99436 fix(web): list Generic JSON provider last in channel provider dropdown
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 21:54:34 +02:00
Oliver Kieselbach 6435d5f290 feat(notifications): Discord webhook provider + HMAC signing for generic webhooks
Discord (provider 30) renders alerts as a single embed with Discord's
field/length caps; actions become markdown links since webhooks cannot
post buttons. Generic JSON channels gain an optional per-channel signing
secret: requests carry X-AutopilotMonitor-Timestamp and
X-AutopilotMonitor-Signature (sha256 hex HMAC over "{timestamp}.{body}"),
computed over the exact posted JSON with same-named custom headers
stripped first. Secret is redacted for readers and restored by channel
id on save; validation enforces 16-128 printable ASCII characters.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 21:09:12 +02:00
Oliver Kieselbach 445ce4e75b feat(backend): 30-day retention grace period on Pro downgrade
Retention was the only enforcement point that destroyed state on a
Pro -> Community downgrade: the 12-hourly sweep clamps to the Community
cap at read time and would immediately start hard-deleting the
90-365-day band (trial expiry identical, silently).

New policy: for 30 days after losing Pro
(FeatureEntitlementCatalog.RetentionDowngradeGraceDays) the Pro cap
keeps applying. Anchor model: ProDowngradedUtc (backend-only field,
patch-denied + revert-protected) stamped/cleared by the plan endpoint's
new pure ApplyPlanChanges core on EFFECTIVE-edition transitions (covers
explicitly ended trials; a planTier downgrade under an active trial
deliberately does not stamp); trial expiry anchors read-time via
TrialExpiresUtc. days<=0 (infinite) stays untouched.

Ops visibility: TenantPlanDowngraded (PATCH handler, new OpsEventService
dep) plus TenantRetentionGraceExpiring/Ended from the daily
TrialExpirySweep with the same stateless window mechanics, emitted only
when stored retention exceeds the Community cap; all dual-registered in
the web OPS_EVENT_TYPES catalog. Fanout now uses its _utcNow seam
instead of inline DateTime.UtcNow. OKF doc
backend/plan-downgrade-retention-grace.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 21:00:53 +02:00
Oliver Kieselbach 4e54ffe4b4 feat(web): shared plan cards, public /plans and /buy pages
Community/Pro comparison cards extracted into a shared module
(components/plans: planData.ts as single copy source, PlanCards.tsx with
marketing/portal surfaces) so the portal Plan section and the new public
/plans page can never drift. Pro price is one constant (null renders
"Pricing - announced soon").

New public pages following the /help pattern: /plans (free-first
messaging, nav item "Plans") and /buy (direct purchase + Microsoft
Marketplace as coming-soon channels). Landing nav, footer, FinalCta and
get-started now link to /plans instead of the docs plans page; portal
Plan section links to /buy via absolute www URL. Purple utilities
neutralized inside .dark .landing-v2; sitemap/robots/lastmod cover
/plans, /buy and the previously missing /help.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 19:46:44 +02:00
Oliver Kieselbach 65d45a616e feat(web): Community Edition tag under navbar wordmark
Community tenants see a small "Community Edition" line under the brand
wordmark as a standing reminder (and nudge toward upgrade options); Pro
tenants (incl. trial) keep clean chrome. Label renders only after the
feature-flags fetch confirms the edition - no flash for Pro while loading.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 19:38:19 +02:00
Oliver Kieselbach 4aa1f07fbb feat(web): green welcome accent + session-dismissable dev banner with telemetry
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 19:17:53 +02:00
Oliver Kieselbach ede616444a fix(web): honest skip/uninstall/pending semantics in Download/Install Progress
IME reports skips and uninstall enforcements through the same completed
events, so both panels counted them as successful installs/downloads and
their pills contradicted the app_tracking_summary headers (session 502274b4:
"2 of 8 installed" vs "6 completed").

- Install fold: app_install_completed with state Skipped -> Skipped;
  successful uninstall intent -> new terminal state Uninstalled with its own
  pill and neutral row badge; completed now counts only real installs.
- Download fold: rows are classified by observed byte activity, never by
  intent (Win32 uninstalls re-download the full package and stay visible
  with an Uninstall badge); 0-byte rows sit behind a "no download" toggle.
- Both panels render tracker-known pending apps (summary pendingNames)
  behind a "pending" toggle, so every "X of Y" header fully adds up.
- Headers wrap on narrow screens (flex-wrap + gap instead of space-x).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 18:42:23 +02:00
Oliver Kieselbach 61a359ee54 feat(web): public /help page for support channels
Adds www.autopilotmonitor.com/help as a mail-linkable page describing how
to get help: Report Session in the portal for session-specific problems,
GitHub Issues for everything else, LinkedIn for private contact, plus
self-service links (docs, FAQ, announcements, changelog) and the private
security-advisory path. Registered as public path in hostRouting (guard
test pinned) and linked from the site footer.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-22 18:12:09 +02:00
Oliver Kieselbach babc89c776 fix(backend): park silent WhiteGlove Part-2 sessions as AwaitingUser instead of Stalled
fairstone.ca analysis 2026-08-21: after the reseal-reboot technicians routinely
power devices off at the logon screen to box them. Both silence paths (2h
maintenance sweep, agent stall probe) then labeled the expected parking state
"Stalled - Agent silent", and stage 2 later reconciled it to Succeeded on the
Part-1 whiteglove_complete with a reason claiming Account Setup completed.

- EnrollmentTimeoutClassifier: shared pure gate IsWhiteGloveAwaitingUser
  (pre-provisioned + ResumedAt + Device Setup provisioned + zero user evidence
  since resume) with one canonical customer-facing reason; new rollup field
  HasEnrollmentComplete separates real completions from Part-1 whiteglove_complete
- ClassifyTimedOutSession: gated sessions are AwaitingUser within grace and
  reconcile past grace with an honest WhiteGlove reason; on resumed sessions
  whiteglove_complete no longer counts as terminal-complete evidence (rule 2),
  so partial user progress falls through to the honest AwaitingUser/Incomplete rules
- MaintenanceService stage 1 + session_stalled ingest mapping apply the same
  gate (fail-soft to Stalled); sweep audit now carries SessionsAwaitingUser
- Callsites (stage 2, max-lifetime verdict, late-telemetry reconcile, legacy
  reclassification) pass IsPreProvisioned/ResumedAt through

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 22:30:29 +02:00
Oliver Kieselbach a2f448cad0 fix(web): day-scale duration formatting in network timeline
fmtDuration rolls hours into days above 24h ("768d 8h" instead of
"18440h 58m") - minutes are noise at day scale. Surfaced by the first
fleet session with a CMOS clock years off (0bd4af1c, 2024 -> 2026 step);
also keeps the slanted clock-marker label inside its headroom. Test
pinned on the real fleet delta.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 21:05:04 +02:00
Oliver Kieselbach f15e115dea fix(sessions): backfilled system-timeline events no longer anchor StartedAt/duration
Field case d0c5b672: a pre-agent system_clock_changed event (backfilled from
the Windows event log with its original timestamp, ~14min before the first
real activity) became the partition's earliest row and re-anchored the
session's StartedAt, inflating the duration toward pre-enrollment OOBE idle
time. With the watcher's 24h backfill lookback, the 2h backward-shift guard
only caps this effect, it does not prevent it.

SystemTimelineWatcher rows (clock steps, sleep episodes) are environment
observation, not enrollment activity, and are now excluded from all three
anchor derivations: the Events-table earliest-row probe (new
IsSessionAnchorEligible, probe scans past ineligible rows), the ingest
batch's EarliestEventTimestamp fold, and the web's client-side
enrollment-duration derivation. The events themselves stay on the timeline
unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-08-21 20:18:57 +02:00
223 changed files with 12698 additions and 1636 deletions
+147
View File
@@ -0,0 +1,147 @@
---
type: concept
title: Diagnostics Package
description: What the agent's diagnostics ZIP contains and why — the built-in section catalog shared with the portal, the RealmJoin and Device Preparation collection gates, the configured-path guards, caps, and the package manifest.
resource: agent
tags: [diagnostics, realmjoin, device-preparation, guardrails, portal]
timestamp: 2026-08-23
---
# Schema
The diagnostics package is the agent's last-resort support artifact: a ZIP built in memory
by `DiagnosticsPackageService.BuildArchiveBytes` and uploaded through a short-lived SAS
URL at the end of a session, at the WhiteGlove seal (`Always` mode) or on a server-requested
on-demand collection ([server actions](../backend/server-actions-on-demand-diagnostics.md)).
## Archive layout
```
sessioninfo.txt session/tenant/device/outcome, hardware via WMI
<built-in sections> catalog order, see below
AdditionalLogs/<last folder>/ admin-configured paths (global ∪ tenant)
package-manifest.txt every packaging decision (always written)
_TRUNCATED.txt only when a cap skipped something
```
## The built-in catalog is the single source of truth
Everything the package collects **before** any configured path is data, not code paths:
`DiagnosticsBuiltInSections.All` in Shared (`Models/Diagnostics/DiagnosticsBuiltInSections.cs`).
The agent iterates it in order; the backend serves the same list to the portal
(`GET /api/diagnostics/paths`, MemberRead) so what administrators see is exactly what the
agent does. There is no second copy to drift.
| Id | ZIP folder | Source (unexpanded) | Patterns | Recursive | Condition |
| --- | --- | --- | --- | --- | --- |
| AgentLogs | `AgentLogs` | `%ProgramData%\AutopilotMonitor\Logs` | log patterns | no | Always |
| ImeLogs | `ImeLogs` | `%ProgramData%\Microsoft\IntuneManagementExtension\Logs` | log patterns | no | Always |
| ImeBootstrapperEventLog | `ImeLogs` | `C:\Windows\System32\winevt\Logs` | `BootstrapperAgentServiceLogProvider.evtx` | no | DevicePreparation |
| AgentState | `AgentState` | `%ProgramData%\AutopilotMonitor\State` | log patterns + `*.complete`, `*.marker` | yes | Always |
| AgentSpool | `AgentSpool` | `%ProgramData%\AutopilotMonitor\Spool` | `*.jsonl`, `*.json` | no | Always |
| AgentMarkers | `AgentMarkers` | `%ProgramData%\AutopilotMonitor` | `*.complete`, `*.marker` | no | Always |
| RealmJoinWindows | `RealmJoinLogs/Windows` | `C:\Windows\Logs` | `realmjoin*.log` | no | RealmJoinWatcher |
| RealmJoinPackages | `RealmJoinLogs/Windows/RealmJoin` | `C:\Windows\Logs\RealmJoin` | `*.log` | yes | RealmJoinWatcher |
| RealmJoinChoco | `RealmJoinLogs/Choco` | `%ProgramData%\RealmJoin\choco\logs` | `*.log` | yes | RealmJoinWatcher |
| RealmJoinUserTray | `RealmJoinLogs/User` | `%LOGGED_ON_USER_PROFILE%\AppData\Local\RealmJoin` | `tray*.log` | no | RealmJoinWatcher |
| RealmJoinUserLogs | `RealmJoinLogs/User/Logs` | `%LOGGED_ON_USER_PROFILE%\AppData\Local\RealmJoin\Logs` | `*.log` | yes | RealmJoinWatcher |
"Log patterns" = `*.log *.txt *.json *.jsonl *.etl *.evtx *.xml *.csv *.cab`. The five
`Always` sections are the historical layout (PR1-B forensics) and are lock-tested in order;
the RealmJoin ZIP layout mirrors the disk (`C:\Windows\Logs\{realmjoin*.log, RealmJoin\…}` →
`RealmJoinLogs/Windows/…`, `%LOCALAPPDATA%\RealmJoin\{tray*.log, Logs\…}` →
`RealmJoinLogs/User/…`), so a flat section and a tree section never produce the same entry
name. Two sections may share a ZIP folder only when they read different source folders
(`ImeLogs` + the bootstrapper evtx) — `(ZipFolder, SourceFolder)` pairs are unique.
### Conditions
`DiagnosticsSectionCondition` is evaluated by the agent (`IsSectionActive`, pure) against its
effective configuration and the enrollment scenario; unknown values fail closed.
* **RealmJoinWatcher** — the tenant's RealmJoin Watcher toggle
(`AnalyzerConfiguration.EnableRealmJoinWatcher`). The package service only sees
`AgentConfiguration`, so `RemoteConfigMerger` mirrors that one Analyzers knob onto
`AgentConfiguration.EnableRealmJoinWatcher` (the RealmJoin host keeps reading the remote
block). Deliberately NOT additionally gated on the compile-time `RealmJoinTrackingEnabled`
kill switch: if the live watcher is ever disabled, the logs become more useful, not less.
No ConfigVersion bump — no new wire field.
* **DevicePreparation** — `EnrollmentRegistryDetector.IsDeterministicDevicePreparation()`,
the same deterministic `DevicePreparation\BootstrapperAgent\ExecutionContext` marker every
other WDP gate keys on ([WDP scenario gates](wdp-scenario-gates.md)); CloudAssigned*
fallback rules count as Classic. The probe is a registry read, evaluated once per build
(`SCENARIO: devicePreparation=… realmJoinWatcher=…` in the manifest); a probe failure is
treated as Classic and logged — the evtx export is skipped, never the package. The channel
is exclusively locked while active, so the existing `wevtutil epl` export path
(`ResolveChannelForEvtxFile`, field case 2026-08-17) handles it.
Inactive sections are never silent: `BUILT-IN SKIPPED (RealmJoin Watcher disabled): <Id>`,
`BUILT-IN SKIPPED (not a Device Preparation enrollment): <Id>`,
`BUILT-IN SKIPPED (no user session for token): <Id> '<source>'` (the
`%LOGGED_ON_USER_PROFILE%` token resolves via `UserProfileResolver` only once an
interactive user exists — identical to configured paths). Active ones log
`BUILT-IN: <Id> -> folder='…' zip='…' recursive=…` followed by the per-pattern
`ADDED / NO MATCH / FOLDER MISSING` lines.
## Built-ins bypass the configured-path guards — on purpose
`DiagnosticsPathGuards` ([gather-rule guardrails](../rules/gather-rule-guardrails.md))
validates **admin-typed** paths: allowlisted prefixes, wildcard only in the last segment,
`C:\Users` blocked except the token's `AppData\Local|Roaming`. The catalog is reviewed code,
exactly like the five sections that never went through the guard before — and
`%ProgramData%\RealmJoin` is not on the allowlist at all. What does apply to every section:
the reparse-point skip (files and directories), the `.evtx` channel export, and the caps
(100 MB per file, 500 MB total, 5000 files) tracked globally across sections by
`BudgetTracker`, with every skip recorded in the manifest and `_TRUNCATED.txt`.
## Configured paths: global ∪ tenant
`GetAgentConfigFunction.MergeDiagnosticsLogPaths` sends global paths (Global Admin,
`AdminConfiguration.DiagnosticsGlobalLogPathsJson`) first, then tenant paths
(`TenantConfiguration.DiagnosticsLogPathsJson`), dropping blank entries and collapsing
duplicates (trimmed, case-insensitive) onto the first occurrence — a path present in both
lists used to produce duplicate ZIP entry names. The agent keys each entry's ZIP folder on
the last directory name (`AdditionalLogs/<folder>`), so two configured paths ending in a
same-named folder share one ZIP folder.
## Portal surfaces
* Global Admin → Settings → Diagnostics Log Paths: the built-in catalog (collapsed, read-only,
neutral condition pills) above the editable global list; add-row first.
* Tenant → Settings → Agent → Diagnostics Package: built-in catalog (header carries the
tenant's persisted RealmJoin Watcher state, RealmJoin rows show on/off), the global paths
read-only, then the tenant's own entries; every role with a tenant membership can read the
catalog (the route is MemberRead, JWT-scoped). One line per entry (`DiagnosticsPathRow`),
full values in tooltips.
## Test seams
* `DiagnosticsPackageService` internal ctor: the five historical folder overrides map onto
catalog ids; `sectionFolderOverrides` covers any other section; `devicePreparationProbe`
replaces the registry read (seam callers default to "not WDP", so existing rigs never touch
the registry). `UserProfileResolver.SetForTesting(path|null)` / `Reset()` drive the token.
* Catalog invariants live in the backend test project (`DiagnosticsBuiltInSectionsTests`),
because Shared has no test project.
# Examples
A RealmJoin tenant (watcher on) with a signed-in user: `RealmJoinLogs/Windows/realmjoin.log`,
`RealmJoinLogs/Windows/RealmJoin/Packages/<pkg>/<timestamp>_install.log`,
`RealmJoinLogs/Choco/<pkg>/<timestamp>_install.log`, `RealmJoinLogs/User/tray.log`,
`RealmJoinLogs/User/Logs/RjImeHost.log`. The same tenant's package collected before the
desktop: the two `User` sections appear as `BUILT-IN SKIPPED (no user session for token)`.
A Classic enrollment: `BUILT-IN SKIPPED (not a Device Preparation enrollment):
ImeBootstrapperEventLog`. A tenant that had added
`%LOGGED_ON_USER_PROFILE%\AppData\Local\RealmJoin\Logs\*.log` manually keeps working — the
files simply appear twice (`AdditionalLogs/Logs/…` and `RealmJoinLogs/User/Logs/…`) until the
entry is removed.
# Citations
* `src/Shared/AutopilotMonitor.Shared/Models/Diagnostics/DiagnosticsBuiltInSections.cs` — the catalog
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Monitoring/Runtime/DiagnosticsPackageService.cs` — `BuildArchiveBytes`, `AddBuiltInSection`, `IsSectionActive`, caps, evtx export
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Configuration/RemoteConfigMerger.cs` — `EnableRealmJoinWatcher` mirror
* `src/Backend/AutopilotMonitor.Functions/Functions/Diagnostics/GetDiagnosticsPathsFunction.cs` — `GET diagnostics/paths`
* `src/Backend/AutopilotMonitor.Functions/Functions/Config/GetAgentConfigFunction.cs` — `MergeDiagnosticsLogPaths`
* `src/Web/autopilot-monitor-web/components/diagnostics/` — `DiagnosticsPathRow`, `BuiltInSectionsList`, `builtInSectionDisplay`
* Tests: `DiagnosticsPackageServiceTests` (RealmJoin on/off, no user session, bootstrapper evtx, probe failure), `DiagnosticsBuiltInSectionsTests`, `GetDiagnosticsPathsPayloadTests`, `GetAgentConfigFunctionTests.MergeDiagnosticsLogPaths_*`
+1
View File
@@ -5,6 +5,7 @@ How the on-device agent works:
* [Runtime Overview](overview.md) - lifecycle (install → boot guards → runtime → termination), collector hosts, the single-rail event pipeline to the backend, config & kill switch, on-disk layout.
* [Decision Engine](decision-engine.md) - the pure-reducer state machine (DecisionCore): signals, state, effects, the A/B/C completion arms, invariants, threading.
* [Logs, Persistence & Crash Recovery](logs-and-persistence.md) - every persisted file and why it exists; signal log vs spool; snapshot/journal; the replay-based recovery flow and quarantine.
* [Diagnostics Package](diagnostics-package.md) - what the diagnostics ZIP contains: the built-in section catalog shared with the portal (Shared `DiagnosticsBuiltInSections`, served by GET diagnostics/paths), the RealmJoin Watcher and Device Preparation collection gates, why built-ins bypass the configured-path guards, global ∪ tenant path dedup, caps and the package manifest.
* [Hello Wizard Un-Skip](hello-wizard-unskip.md) - why a single "WHfB disabled" policy read never decides completion: the HelloWizardStarted signal (prevention + cure) and the HelloTracker confirmation second read.
* [Autopilot ZTD Diagnostics](autopilot-ztd-diagnostics.md) - Windows' own diagnostic surfaces for the profile-download flow: ModernDeployment event IDs (807/815/164/...), the Diagnostics\Autopilot registry key, deployment-service endpoints, and the known-issue error-code map.
* [MDM Reboot Coalescing](mdm-reboot-coalescing.md) - attributing the mid-ESP coalesced reboot ("second sign-in") to the device-assigned policy URIs that forced it: aggregated 2800 observations (neutral, flush-time watermark) + ANALYZE-ESP-005 gated on an actually-observed reboot.
+2 -1
View File
@@ -78,7 +78,8 @@ Durations shown for attempts are the sessions' authoritative `DurationSeconds` v
completed journeys, first-time-right count, attempt histogram, junk-serial exclusion
count, `JourneyVersion`. A journey buckets on the StartedAt date of its completing
success session (the platform's StartedAt-date convention; late terminals converge via
the sweep). All counts are **additive**, so a window rate is the ratio of summed daily
the sweep, which slices the tick's shared projected window scan — see
`time-attribution.md`). All counts are **additive**, so a window rate is the ratio of summed daily
rows — there is no rolling-window row (unlike the median-based time-attribution
aggregates, which cannot merge). 180-day retention. Rows exist below the ≥20 UI gate
on purpose (the UI needs the n for "insufficient data (n=…)").
@@ -0,0 +1,93 @@
---
type: concept
title: Plan Downgrade Policy & the Retention Grace Period
description: What happens when a tenant loses Pro (explicit downgrade or trial expiry) — every entitlement gates read-time and reversibly, except retention, which gets a 30-day grace before the Community cap starts hard-deleting; the anchor model, the ops events, and the deliberate non-grace of everything else.
resource: backend
tags: [plan, edition, downgrade, retention, grace-period, trial, entitlements]
timestamp: 2026-08-22
---
# Schema
## The downgrade model: gate, don't destroy
Edition is resolved at READ time (`FeatureEntitlementCatalog.ResolveEdition`): Pro ⇔
permanent tier `pro`/legacy `enterprise` OR an active trial. A plan mutation is a single
field write (`PATCH config/{tenantId}/plan`, GA-only) — there is no downgrade event
handler, no cascade. On Pro → Community every entitlement simply gates on the next read
(≤ ~5 min config cache):
* **Unrestricted Mode** — effective value flips false; the GA on-request flag and the
tenant toggle survive untouched (inert). Re-upgrade restores it verbatim.
* **Delegated (MSP) admin** — scope suppressed at the home-tenant gate
(`DelegatedAdminService.ApplyHomeTenantGateAsync`); grant rows / tenant groups /
assignments are never touched. Resurrects on re-upgrade.
* **OOBE bootstrap** — endpoints 403/404 unless the additive GA per-tenant flag
(`BootstrapTokenEnabled`) is set; `BootstrapSessions` rows are kept.
* **Rate limits / MCP quota** — pure computed-at-read floors/plan names.
**Retention is the one exception**: the retention sweep (12-hourly
`SessionDeletionMaintenance` → `SessionRetentionFanoutService`) hard-deletes sessions
older than `GetEffectiveRetentionDays` = `min(stored DataRetentionDays, edition cap)`.
Without protection, a downgrade (or trial expiry) with stored retention 365 would start
irreversibly cascade-deleting the 90–365-day band on the next tick.
## The retention grace period (30 days)
`FeatureEntitlementCatalog.RetentionDowngradeGraceDays = 30`. For 30 days after losing
Pro, `GetEffectiveRetentionDays` keeps clamping to the **Pro** cap (365); only after the
grace does the Community cap (90) bite. The stored `DataRetentionDays` is never rewritten.
`days <= 0` (GA-only "infinite") short-circuits before any cap — those tenants are never
swept, grace or not.
**Anchor** — "when was Pro lost" (`TenantEntitlementService.GetRetentionGraceEndUtc`,
returns null while effectively Pro; grace end = latest anchor + 30d):
* **Explicit downgrade**: `TenantConfiguration.ProDowngradedUtc` (backend-only field, no
agent ConfigVersion impact), maintained exclusively by the plan endpoint's pure mutation
core `PlanManagementFunction.ApplyPlanChanges`. Stamped when the EFFECTIVE edition
(before vs. after, same `nowUtc`) transitions Pro → Community — this covers both a
planTier downgrade and a GA explicitly ending a trial (`trialExpiresUtc: null`, where
the trial timestamp disappears and could not anchor). Cleared on any transition back to
effectively Pro (PATCH pro, trial grant, self-service trial start), so a re-upgrade also
resets the grace clock.
* **Trial expiry**: no write happens — the stored `TrialExpiresUtc` itself is the anchor
(read-time, like edition resolution). A planTier downgrade under a still-active trial
deliberately does NOT stamp: the edition is still Pro, and the later trial expiry
anchors the grace then.
The field is on the patch-endpoint `BaseDeniedFields` and `RevertProtectedFields` lists
(a config revert must not time-travel the anchor) and roundtrips the table-serialization
contract (Store+Map + roundtrip tests). Legacy rows read null → no anchor → no grace,
which is correct: no tenant had been downgraded before the feature existed.
## Ops visibility (dual-registered event types)
* `TenantPlanDowngraded` (Warning) — emitted by the PATCH handler on an effective
Pro→Community transition; carries `retentionGraceEndsUtc` + stored retention days. The
response body also returns `retentionGraceEndsUtc`.
* `TenantRetentionGraceExpiring` / `TenantRetentionGraceEnded` (Warning) — emitted by the
daily `TrialExpirySweepFunction` (03:30 UTC) with the same stateless window mechanics as
the trial events (≤3-day heads-up re-fired daily; ended once via the 24h look-back), and
ONLY when data is actually at risk: stored `DataRetentionDays` above the Community cap
(0 = infinite ⇒ never). Re-upgraded tenants go silent automatically (grace resolves null).
All three are registered in the web `OPS_EVENT_TYPES` catalog (OpsAlertRulesSection.tsx)
so Telegram alert rules can route them.
## Deliberate non-goals
* No grace for Unrestricted/MSP/bootstrap/limits — reversible gates need none.
* The retention WRITE cap (`TenantConfigValidation`) does not honor the grace: during the
grace a tenant admin cannot SET a value above the Community cap. The grace protects
existing data; new writes follow the new plan.
* The feature-flags endpoint keeps publishing the plain edition cap
(`retentionCapDays`) — it feeds the settings input maximum, which is the write cap.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Services/TenantEntitlementService.cs` — grace + effective retention
* `src/Backend/AutopilotMonitor.Functions/Functions/Config/PlanManagementFunction.cs` — `ApplyPlanChanges` anchor lifecycle
* `src/Backend/AutopilotMonitor.Functions/Functions/Maintenance/TrialExpirySweepFunction.cs` — grace event windows
* `src/Backend/AutopilotMonitor.Functions/Services/Deletion/SessionRetentionFanoutService.cs` — enforcement point
* Tests: `TenantEntitlementServiceTests`, `PlanManagementTransitionTests`, `TrialExpirySweepFunctionTests`, `SessionRetentionFanoutServiceTests`, `TenantConfigTableSerializationTests`
@@ -0,0 +1,110 @@
---
type: Concept
title: Silent-Session Classification (Sweep, Max-Lifetime, Retro-Reconcile)
description: How a session whose agent went silent gets its honest status — the two-stage maintenance sweep (2h Stalled marker, 5h timeout classification), the EnrollmentTimeoutClassifier rule order incl. the WhiteGlove Part-2 and self-deploying gates, which ingest paths share the classifier, and the admin retro-reconcile modes.
resource: src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs
tags:
- backend
- maintenance
- session-status
- classification
- whiteglove
- self-deploying
timestamp: 2026-08-23T18:00:00+02:00
---
# Silent-Session Classification
A session only ever becomes terminal through evidence. When the agent stops reporting
(power-off, reboot without relaunch, Wi-Fi never re-associating, max-lifetime watchdog),
the backend must decide the status from what it last saw — it never hard-fails a session
for being silent. One pure function holds that decision,
`EnrollmentTimeoutClassifier.ClassifyTimedOutSession`, and every silent-session path calls it
so a session gets the same verdict regardless of which path noticed the silence first.
# Schema
## Inputs (`EspProvisioningRollup` + session flags)
`ExtractRollup(events)` distills the event stream into facts: Device/Account Setup
subcategory counts (incl. the 30s fallback-confirmed messages), `HasExplicitFailure`,
`HasEnrollmentComplete` vs. `HasTerminalComplete` (the Part-1 `whiteglove_complete`),
`HasAgentEmergencyBreak`, `DesktopArrived`, `HelloResolved`, `HelloPolicyDisabled`,
`SkipUserEsp`, RealmJoin detection/resolution. Session-row flags are passed alongside:
`IsPreProvisioned` + `ResumedAt` (WhiteGlove Part 2) and `IsSelfDeployingProfile`
(registry-confirmed `CloudAssignedOobeConfig 0x20|0x40`, set at registration).
## Rule order (first match wins)
| # | Condition | Verdict |
| --- | --- | --- |
| 1 | `HasExplicitFailure` | Failed |
| 1b | `IsWhiteGloveAwaitingUser`: pre-provisioned + resumed + Device Setup done + **no** user evidence since resume + no explicit failure/enrollment_complete | AwaitingUser within grace; Succeeded (honest WG reason) past grace |
| 1c | `IsSelfDeployingProvisioned`: self-deploying profile + Device Setup done + no explicit failure | **Succeeded** (reconcile) — no grace, no AwaitingUser: the profile has no user phase |
| 2 | Account Setup all-succeeded, or `enrollment_complete`, or `whiteglove_complete` on a **non-resumed** session | Succeeded |
| 3 | `HasAgentEmergencyBreak` | Incomplete immediately (agent self-destructed, nothing can arrive) |
| 4 | `DesktopArrived` + (Hello terminal, or Hello disabled + SkipUserEsp) | Succeeded (user provably finished) |
| 5 | Device Setup done | AwaitingUser within grace, Incomplete past grace |
| 6 | otherwise | Incomplete |
Grace = `ResolveGraceHours(SessionGraceHours, AbsoluteMaxSessionHours)` (~51h default),
measured from `ResumedAt ?? StartedAt`. Every Succeeded-reconcile reason carries the
silence-transparency clause (`AppendReconcileTiming`: last agent contact, silence, verdict time).
## Why rule 1c exists (kiosk tenant audit 2026-08-23)
On a self-deploying profile Device ESP all-succeeded IS the enrollment's end; the agent's
own SelfDeploying terminal only adds a 5-min confirmation window
(`DecisionEngine.SelfDeploying.s_deviceOnlyEspDetectionWindow`). Tenant `aebdce78` (school,
Wi-Fi ThinkBooks, a language-change app forcing a reboot right after Device ESP) lost that
window on 836 of ~970 sessions in one week: the agent died 0–4 min after
`DeviceSetup fallback_confirmed`, the reboot relaunch never registered (30s registration
budget on a Wi-Fi link still associating), and rule 5 labeled finished kiosks
"awaiting user / Account Setup 0/5" → Incomplete 51h later. Nobody signs in on these devices
(prepared during the holidays, boxed, in service) — AwaitingUser was factually wrong and
Incomplete turned a working rollout into a red wall. Other self-deploying tenants completed
normally in the same window, so this is a classification gap, not a profile-wide agent bug.
Agent-side companion fix: `SessionRegistrationHelper` now waits up to 15s for a network link
and retries 6× (~62s backoff) instead of 5× (~30s).
## Callers (all pass both gate inputs)
| Path | File | Notes |
| --- | --- | --- |
| Sweep stage 1 (2h agent-silent marker) | `MaintenanceService.MarkStalledSessionsAsTimedOutAsync` | Default: InProgress → Stalled. Gates run BEFORE the Stalled write: `TryMarkWhiteGloveAwaitingUserAsync` (AwaitingUser) and `TryReconcileSelfDeployingAsync` (Succeeded). Only flagged sessions pay the event read. |
| Sweep stage 2 (SessionTimeoutHours, 5h default) | same | InProgress/Stalled/AwaitingUser past the window and past the 2h silence cutoff → full classification; AwaitingUser within grace is skipped without an event read. |
| Agent max-lifetime (`enrollment_failed`/max_lifetime, `agent_shutting_down`/max_lifetime) | `EventIngestProcessor.ApplyMaxLifetimeVerdictAsync` | Watchdog = "agent stopped waiting", not a verdict; classified from evidence. Pending sessions exempt. |
| Late-telemetry reconcile | `EventIngestProcessor.TryLateTelemetryReconcileAsync` | Heal-only: applies a Succeeded verdict to Incomplete/AwaitingUser rows when straggler events arrive. |
| Agent stall probe (`session_stalled`) | `EventIngestProcessor.TryMapStallToWhiteGloveAwaitingUserAsync` | Only the WG gate (1b); a stall probe on a self-deploying device with Device Setup done means the agent is alive and something else blocks — stays Stalled. |
| Admin retro-reconcile | `LegacyReclassificationService` | Modes `legacy_timeouts`, `pending_orphans`, `self_deploying_silent` (Incomplete/AwaitingUser/Stalled rows with the profile flag, heal-only to Succeeded, admin-marked rows skipped). `POST /api/maintenance/reclassify-legacy?mode=…&dryRun=false&tenantId=…&maxSessions=…`, dry-run by default. |
## Known gap (decided, not built)
After max-lifetime the agent writes `enrollment-complete.marker` and self-destructs, so an
AwaitingUser verdict from that path can never heal — it is a 51h countdown to Incomplete.
The emergency-break path (rule 3) already terminalizes immediately; max-lifetime still goes
through the grace window by design ("user simply absent"). Revisit if customers read
AwaitingUser as "still possible".
# Examples
* Session `195593e2` (tenant aebdce78): `DeviceSetup 4/4` + `fallback_confirmed`, `AccountSetup 0/5`
(IME false positive), Hello disabled, SkipUser=True, silent after 13:01 → rule 1c →
Succeeded "self-deploying profile — Device Setup fully provisioned … agent went silent
before confirming completion. Agent last reported 2026-08-21 13:01 UTC …".
* Same event shape on a user-driven profile → rule 5 → AwaitingUser (within grace) /
Incomplete (past grace). Guarded by
`Classify_same_stream_without_self_deploying_flag_keeps_user_phase_verdicts`.
* fairstone WG Part 2 reseal-reboot, technician powers off at logon screen → rule 1b →
AwaitingUser; user unboxes days later → ingest completion heals to Succeeded.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Services/EnrollmentTimeoutClassifier.cs` — rollup + rule order, `IsWhiteGloveAwaitingUser`, `IsSelfDeployingProvisioned`, reason helpers.
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs` — `MarkStalledSessionsAsTimedOutAsync`, `TryMarkWhiteGloveAwaitingUserAsync`, `TryReconcileSelfDeployingAsync`.
* `src/Backend/AutopilotMonitor.Functions/Services/EventIngestProcessor.Classification.cs` — max-lifetime verdict, late-telemetry reconcile, stall mapping.
* `src/Backend/AutopilotMonitor.Functions/Services/LegacyReclassificationService.cs`, `Functions/Admin/ReclassifyLegacySessionsFunction.cs` — retro modes.
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Maintenance.cs` — `GetAgentSilentSessionsAsync`, `GetStalledSessionsAsync`, `GetSelfDeployingSilentSessionsAsync`.
* `src/Agent/AutopilotMonitor.Agent.V2.Core/Security/SessionRegistrationHelper.cs` — link wait + 6-attempt registration budget.
* `src/Shared/AutopilotMonitor.DecisionCore/Engine/DecisionEngine.SelfDeploying.cs` — the agent-side 5-min SelfDeploying terminal window.
* `src/Backend/AutopilotMonitor.Functions.Tests/EnrollmentTimeoutClassifierTests.cs` — rule coverage incl. the 2026-08-23 self-deploying block.
+67
View File
@@ -0,0 +1,67 @@
---
type: Concept
title: Table Schema Sentinel (startup table initialization)
description: Why a cold start no longer issues one CreateTableIfNotExists per registered table - a hash over TableNames.All stored in AdminConfiguration gates the full pass, the SessionsIndex backfill only runs after a full pass and behind a conditional-insert claim, and daily maintenance repairs out-of-band deletions.
resource: /src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.cs
tags:
- backend
- table-storage
- startup
- maintenance
timestamp: 2026-08-23
---
# Schema
## Problem
`TableInitializerService` (hosted service) called `InitializeTablesAsync` on every process start, which issued `CreateTableIfNotExistsAsync` for every entry of `Constants.TableNames.All` (66 at the time of writing, DoP 8). Each call is a Create POST whose 409 is swallowed - on Flex Consumption with frequent cold starts that was ~9 serial round-trip waves before the first request. It was followed by a SessionsIndex emptiness query whose *error* path returned "empty", so a transient storage error triggered a full Sessions scan with serial upserts - on every scaled-out instance at once.
## Sentinel
* Row `AdminConfiguration` / PK `SchemaSentinel` / RK `tables`, property `TableSchemaHash` (+ `TableCount`, `UpdatedUtc`).
* Value = `ComputeTableSchemaHash(TableNames.All)`: SHA-256 over the ordinal-sorted, newline-joined table names. Derived, never hand-maintained: adding a table to `TableNames.All` changes the hash by construction (`TableLifecycleBucketTests` already forces every constant into `All`).
* `InitializeTablesAsync` (returns `bool fullPassRan`):
1. Point-read the sentinel. Match -> mark initialized, **zero** table calls, return `false`.
2. Missing / stale / unreadable (fresh storage has no AdminConfiguration table yet) -> `EnsureAllTablesAsync` (the old full pass), then upsert the sentinel **only if no table failed**, return `true`. A failed pass leaves `_tablesInitialized = false`, so the next call retries.
* Scale-out: two instances that both see a stale sentinel both run the idempotent pass - harmless.
## SessionsIndex backfill
* Only evaluated when the full pass ran (`fullPassRan == true`): an empty index can only exist on fresh storage, which is exactly the case without a sentinel.
* `IsSessionIndexEmptyAsync` now returns `false` on error (skip), not `true`.
* `TryClaimSessionIndexBackfillAsync`: conditional `AddEntityAsync` on `SchemaSentinel` / `sessionIndexBackfill` - only the winner scans; 409 or any error -> skip. The manual maintenance backfill (`RunManualAsync`) remains the safety net.
## Known gap and repair
A table deleted out-of-band (cleanup skill, portal, tests) is not noticed at startup anymore. `IStorageInitializer.EnsureAllAsync` runs as the first step of the daily timer maintenance (`MaintenanceService.RunAllAsync`), recreating it within 24h. Storage helpers are fail-soft in the meantime.
## Measuring it
`StartupTelemetryService` (hosted service) emits two App Insights metrics on `IHostApplicationLifetime.ApplicationStarted` - i.e. after every hosted service finished `StartAsync`:
* `BackendStartupMs` - process start (`Process.StartTime`) to host ready; includes runtime, DI and all hosted services.
* `BackendTableInitMs` - the `InitializeTablesAsync` slice alone (`TableStorageService.LastInitialization`).
Dimensions: `version`, `commit`, `tableInitFullPass`, `instance`. Metrics bypass adaptive sampling and the worker log level, so they arrive even though Information traces do not. The MCP server logs the same idea as one `{"type":"boot","readyMs":…}` stderr line per start (ContainerAppConsoleLogs_CL, see `.claude/commands/mcp-telemetry.md`).
```kusto
customMetrics
| where name in ("BackendStartupMs", "BackendTableInitMs")
| summarize p50 = percentile(value, 50), p95 = percentile(value, 95), n = count()
by name, version = tostring(customDimensions.version), fullPass = tostring(customDimensions.tableInitFullPass)
```
# Examples
* Deploy that adds a table: first cold start after the deploy runs the full pass once (log: "Table schema sentinel missing or stale"), every later start logs "Table schema sentinel matches (...)".
* Fresh storage account: sentinel read throws TableNotFound -> full pass -> index empty -> one instance claims and backfills.
# Citations
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.cs` - sentinel, `InitializeTablesAsync`, `EnsureAllTablesAsync`, `TryClaimSessionIndexBackfillAsync`
* `src/Backend/AutopilotMonitor.Functions/Services/TableInitializerService.cs` - startup gating
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.cs` - daily `EnsureAllTablesAsync`
* `src/Backend/AutopilotMonitor.Functions/Services/StartupTelemetryService.cs` - `BackendStartupMs` / `BackendTableInitMs`
* `src/McpServer/autopilot-monitor-mcp/src/index.ts` - `{"type":"boot"}` line
* `src/Backend/AutopilotMonitor.Functions.Tests/TableSchemaSentinelTests.cs`, `StartupTelemetryServiceTests.cs`
+89
View File
@@ -0,0 +1,89 @@
---
type: Concept
title: Telemetry Ingest Shaping (App Insights / Log Analytics)
description: What the backend deliberately keeps out of Application Insights and why - one canonical request record (worker copy, per-item sampling bypass; host HTTP duplicate dropped by a workspace transformation), expected storage outcomes (404/412/409) and successful SignalR REST calls filtered in the worker, and the query rules that follow.
resource: /src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs
tags:
- backend
- telemetry
- application-insights
- cost
timestamp: 2026-08-23
---
# Telemetry Ingest Shaping
The backend is storage-I/O heavy and runs on the Functions isolated worker, which emits telemetry from TWO processes (host and worker). Left alone, Log Analytics ingestion is dominated by rows nobody reads. This document records what is removed, where, and what that means for queries.
# Schema
## Request telemetry — one canonical record
| Source | Emitted by | Content | Fate |
|---|---|---|---|
| Host HTTP request (`Properties.Source` empty, `Url` set) | Functions host process | Name, duration, status — no tenant/user/correlation | **Dropped at ingestion** by the workspace transformation on `AppRequests` (`infra/appinsights-workspace-transforms.bicep`) |
| Host non-HTTP invocation (`Url` empty) | Functions host process | Timer / queue executions | Kept (the worker middleware never sees them) |
| Worker request (`Properties.Source == 'WorkerMiddleware'`) | `RequestTelemetryMiddleware` | TenantId, UserId, UserRole, CorrelationId, ClientSource, McpToolName, HttpPath | **Kept, unsampled** — the canonical record |
Why two mechanisms:
* The host request item cannot be suppressed from worker code or from `host.json` (`Host.Results` controls log traces, not the request item; verified 2026-06-09). The only lever is a Log Analytics workspace transformation: `source | where tostring(Properties.Source) == 'WorkerMiddleware' or isempty(Url)`.
* `host.json` `samplingSettings.excludedTypes: "Request;Event"` only applies to the host process. The worker's `AddApplicationInsightsTelemetryWorkerService` runs its own adaptive sampling; before 2026-08-23 worker request rows carried `ItemCount` 2–5 on weekdays (≈30 % of individual requests missing, `count()` undercounting by the same share — `sum(ItemCount)` matched the host copy to 1.00 per day). Re-configuring the worker sampling chain did not take effect in the isolated worker (see the L4 note in `Program.cs`). The per-item opt-out does: the middleware sets `SamplingPercentage = 100` on its own `RequestTelemetry`, and the SDK sampling processor passes through every item whose sampling percentage is already set.
## Dependency telemetry — keep only signal
`StorageDependencyFilterProcessor` (worker pipeline) drops:
* successful Azure Storage calls (Table/Queue/Blob) in both shapes — the HTTP shape (`Azure table`, target `*.table.core.windows.net`) and the Azure SDK ActivitySource shape (`InProc | Microsoft.Tables`, `InProc | Microsoft.Storage`);
* storage calls with an **expected outcome**: `404` (point-read miss — EventTypeIndex, BlockedDevices lookups), `412` (ETag precondition), `409` (idempotent insert conflict). HTTP-shape rows carry the code in `ResultCode`; InProc rows only in `Properties["Error"]` (`Status: 404 (`). Before this filter these were 100 % of the remaining "failed" storage rows (~540 MB/week, two rows per call);
* successful Azure SignalR Service REST calls the worker makes itself (`*.service.signalr.net`).
The bulk of SignalR REST traffic (group add/remove per connection, ~470 MB/week) is emitted by the **host** process for the SignalR output binding (`SDKVersion rdddsc:*`) and never passes the worker pipeline — live-verified after the 2026-08-23 deploy (worker rows: 0, host rows: ~560/h). Those are removed by the second data flow of the same workspace transformation: `AppDependencies | where not(Success == true and Target endswith '.service.signalr.net')`.
Everything else passes: storage 429/5xx/auth/timeouts, failed SignalR calls, all HTTP/Graph/NVD/MSRC dependencies, the worker's own `InProc Invoke` span, and every non-dependency item.
## Measured baseline (7 days before the change, `_BilledSize`, `autopilotmonitor-api-eu`)
| Table | Rows | MB | After |
|---|---|---|---|
| AppDependencies | 976k | 1248 | ≈ 240 (SignalR + expected-outcome rows gone) |
| AppRequests host copy | 274k | 323 | ≈ 0 (only timer/queue rows) |
| AppRequests worker copy | 189k | 208 | ≈ 300 (now 100 % of requests) |
| AppTraces | 142k | 137 | unchanged |
| AppEvents | 6k | 4 | unchanged — not worth touching |
Net ≈ −1.3 GB/week with MORE usable data (every request now has tenant context).
# Examples
Counting requests (exact since 2026-08-23; for older data replace `count()` with `sum(itemCount)`):
```kql
requests
| where customDimensions.Source == 'WorkerMiddleware'
| summarize total=count(), failed=countif(success == false) by name
```
Real storage trouble only:
```kql
dependencies
| where type startswith 'InProc | Microsoft.Tables' or target endswith '.table.core.windows.net'
| summarize count() by resultCode, tostring(customDimensions.Error)
```
Post-deploy verification of the per-item sampling bypass (must be `ItemCount == 1` only):
```kql
AppRequests
| where TimeGenerated > ago(1h) and tostring(Properties.Source) == 'WorkerMiddleware'
| summarize rows=count() by ItemCount
```
# Citations
* `src/Backend/AutopilotMonitor.Functions/Middleware/RequestTelemetryMiddleware.cs` — `SamplingPercentage = 100` before `TrackRequest`.
* `src/Backend/AutopilotMonitor.Functions/Telemetry/StorageDependencyFilterProcessor.cs` + `StorageDependencyFilterProcessorTests.cs` — the filter contract.
* `src/Backend/AutopilotMonitor.Functions/Program.cs` — the failed L4 worker-sampling attempt and why it must not be retried.
* `infra/appinsights-workspace-transforms.bicep` — the AppRequests transformation DCR, apply/verify/rollback commands.
* `.claude/commands/backend-logs.md`, `session-debug.md` — query conventions.
+46 -1
View File
@@ -4,7 +4,7 @@ title: Tenant Activation & the Once-Only Welcome Mail
description: How a signup becomes an activated tenant (manual approve or auto-approve queue), and how the welcome mail is sent exactly once even though activation and the user-provided address race each other.
resource: backend
tags: [activation, auto-approve, welcome-email, preview-whitelist, race]
timestamp: 2026-08-02
timestamp: 2026-08-23
---
# Schema
@@ -58,6 +58,46 @@ The Global-Admin resend endpoint (`POST /api/preview/send-welcome-email/{tenantI
always sends (explicit intent) and consumes the marker best-effort so the automatic
paths never duplicate afterwards.
## Mail transport
Both transactional mails (welcome, post-offboarding farewell) go through the
provider-neutral `EmailService` (`IEmailService` for the welcome path,
`IOffboardFarewellEmailSender` for the farewell path). The provider is an implementation
detail confined to one private transport method plus the `Email:*` configuration
section — a provider swap must not rename classes, DI registrations or callers.
* Current provider: **Mailchimp Transactional (Mandrill)**, `POST {Email:Endpoint}`
(`messages/send`), API key in the JSON body, typed `HttpClient` with the shared
`ResiliencePolicies.Notification` retry policy (15 s timeout).
* Settings (App Settings use `Email__Key`): `Email:ApiKey` (required — empty means every
send is a logged no-op, never an error), `Email:Endpoint`, `Email:FromAddress`
(default `noreply@autopilotmonitor.com`), `Email:FromName` (default `Autopilot Monitor`).
* Result handling: per-recipient status `sent`/`queued`/`scheduled` = success;
`rejected`/`invalid` (with `reject_reason`), non-2xx or malformed bodies = warning.
Never throws — both callers rely on that fail-soft contract.
* `track_opens`/`track_clicks` are explicitly `false` and `auto_text` is `true`: the provider
receives only the recipient address and the tenant domain, which is exactly the claim on
the privacy page and the trust data-flows page. Changing tracking = changing those pages.
## Template overrides (operator-editable)
Global Admins can replace either HTML body without a deployment (portal: Settings →
Email Templates). `EmailTemplateService` resolves the body for every send: the stored
override wins, otherwise the built-in `EmailTemplates.*`. Overrides live in the
`PreviewConfig` table (partition `EmailTemplates`, row key `welcome` | `farewell`,
properties `Html`, `UpdatedBy`, `UpdatedUtc`), cached 5 minutes, invalidated on save/reset.
* The only placeholder is `{{domainName}}`; an empty domain renders as "your organization",
matching the built-ins. Subjects are not editable (pinned constants).
* Limit 30 000 characters (Table string property cap is 32K UTF-16 chars).
* The send path is fail-soft: a storage error falls back to the built-in body. The admin
read path is not — a broken store surfaces as an error instead of silently showing the
built-in.
* Endpoints (all GlobalAdminOnly): `GET|PUT|DELETE /api/global/email-templates/{kind}` and
`POST /api/global/email-templates/{kind}/test`. The test send goes to the caller's tenant
contact address (activation notification email → `ContactEmail` → caller UPN) with the
caller's tenant domain, optionally with an unsaved draft body; provider tag `test`.
# Examples
* Auto-approve wins the race: approve at T+64 s finds no address → logs "deferred to the
@@ -73,3 +113,8 @@ paths never duplicate afterwards.
* `src/Backend/AutopilotMonitor.Functions/Functions/Rules/PreviewWhitelistFunction.cs` — save-path trigger, GA resend
* `src/Backend/AutopilotMonitor.Functions/DataAccess/TableStorage/TableConfigRepository.cs` — conditional inserts (approved / welcome-email-sent)
* `src/Backend/AutopilotMonitor.Functions.Tests/TenantApprovalServiceTests.cs` — ordering + dedup pins
* `src/Backend/AutopilotMonitor.Functions/Services/EmailService.cs` — provider-neutral transport (Mandrill wire format, result handling, tracking off)
* `src/Backend/AutopilotMonitor.Functions.Tests/EmailServiceTests.cs` — wire-format, fail-soft and gate-ordering pins
* `src/Backend/AutopilotMonitor.Functions/Services/EmailTemplateService.cs` — override resolution, placeholder rendering, cache
* `src/Backend/AutopilotMonitor.Functions/Functions/Admin/EmailTemplatesFunction.cs` — GA endpoints (read / override / reset / test send)
* `src/Web/autopilot-monitor-web/app/admin/settings/sections/SectionEmailTemplates.tsx` — portal editor, preview frame, test send
+5
View File
@@ -105,6 +105,11 @@ the partition).
* Self-healing 30-day maintenance sweep (2h tick): backfills missing breakdowns, folds
late-terminating sessions into their StartedAt-date aggregates, and enforces version
purity — a stale-version row whose events aged out counts as missing, never mixed.
Reads the tick's **shared window scan** (`MaintenanceService.LoadSweepWindowSessionsAsync`,
one projected 35-day cross-tenant drain via `MaintenanceSweepSessionProjection` that
also feeds the device-journey and verdict-calibration sweeps) — the StartedAt-only
filter is a full-table read in Table Storage, so it is done once per tick, not once per
sweep; `MaintenanceSweepProjectionEquivalenceTests` pins the column set.
* `TimeAttributionAggregates` (PK = TenantId or `global`, RK = `{yyyy-MM-dd}|{class}` and
`rolling30|{class}`): per enrollment class (`user_driven`, `whiteglove`, `self_deploying`,
`device_preparation` — never mixed) median/p75/p90 per segment over the fixed six-segment
+164
View File
@@ -0,0 +1,164 @@
---
type: Concept
title: Verdict Calibration (classifier thermometer)
description: How every session status write declares its VerdictPath, how overriding a verdict preserves it as PriorStatus/PriorVerdictPath, how the daily calibration aggregate counts paths with the re-enrollment proxy and the correction stream, and how the drift radar (share / silence / evidence-gap) turns that into operator-only ops events.
resource: src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.VerdictCalibration.cs
tags:
- backend
- sessions
- classifier
- calibration
- regression-radar
timestamp: 2026-08-23T18:00:00+02:00
---
# Verdict Calibration (classifier thermometer)
A session's terminal status is written by ~20 code paths: the agent's own terminal events,
the silence classifier (`EnrollmentTimeoutClassifier`, rules 1–6) reached via the
maintenance sweep / the agent's max-lifetime shutdown / late-telemetry heal / retro
reclassification, the analyze-rule engine, admin marks and session superseding. Verdict
calibration makes every one of those paths **countable**, tracks how often a path's verdict
was later **overridden**, attaches the delayed **re-enrollment proxy**, and raises an
operator-only alarm when a path's share **drifts**. It is deterministic, fact-based tuning
input for the rules — explicitly not a learned model, and never a customer-facing feature.
# Schema
## VerdictPath on the session row
`SessionSummary.VerdictPath` (`VerdictPaths` vocabulary, `origin:detail`, append-only) is
stamped by **every** status write — `UpdateSessionStatusAsync` takes it as a required
parameter, `StoreSessionAsync` stamps `register:new` / `register:whiteglove_resume` and
preserves an existing path through the re-registration Replace, `SetSessionPreProvisionedAsync`
requires it whenever it writes a status. Unlike `FailureSource` (persisted only on `Failed`),
the path survives for Incomplete, AwaitingUser, Stalled and Pending.
| Origin | Paths |
| --- | --- |
| `agent` | `complete`, `complete_soft` (Continue-Anyway), `failed`, `esp_failure_fallback`, `gather_complete`, `whiteglove_pending`, `whiteglove_resumed`, `stall_probe`, `stall_heal` |
| `ingest` | `wg_awaiting` (stall probe mapped to WhiteGlove AwaitingUser) |
| `sweep` | `stalled`, `wg_awaiting`, `sd_reconcile`, `r{rule}` (stage-2 classifier) |
| `maxlife` / `late` / `retro` | `r{rule}` (classifier via max-lifetime shutdown / late-telemetry heal / retro reclassification); `retro:sd_reconcile`, `retro:superseded` |
| `register` | `new`, `whiteglove_resume`, `superseded` |
| `rule` | `rule:{ruleId}` (analyze rule with MarkSessionAsFailed) |
| `manual` | `failed`, `succeeded` |
| `legacy` | read-side derivation only (`legacy:{rule}` when the rule is recoverable from the reason literal, `legacy:unknown` otherwise) — never written |
Classifier rule ids (`ClassifierRules`): `r1`, `r1b_awaiting`, `r1b_succeeded`, `r1c`, `r2`,
`r3`, `r4`, `r5_awaiting`, `r5_incomplete`, `r6`. `ClassifyTimedOutSession` returns
`(Status, Reason, Rule)` so each `return` names its rule.
## PriorStatus / PriorVerdictPath — the correction stream
When a write replaces a prior **verdict** (existing status ∈ Succeeded/Failed/Incomplete/
AwaitingUser with a stamped path, incoming status different), the storage seam preserves the
pair (`TableStorageService.ComputePriorVerdict`, both the ETag path and the force path). A
stall marker healing to InProgress, a same-status refresh, or an unstamped legacy row never
sets it. Admin marks, late agent completions and retro reclassifications all flow through it.
## Read-side derivation for pre-instrumentation rows
`VerdictPathDerivation.Derive(session)` returns the stamped path verbatim, otherwise derives
from `AdminMarkedAction`, `FailureSource`, `ReconcileReason` and `FailureReason` using the
writers' fixed literals (classifier rule prefixes, the `max-lifetime watchdog` / `Retro-
reclassified` suffixes, `Rule: `, `Superseded by session`, `Late completion report received`).
Unambiguous rows get the real path (`agent:complete`, `manual:failed`, `rule:…`,
`maxlife:r6`), ambiguous-but-rule-known rows `legacy:{rule}`, the rest `legacy:unknown`. No
mass backfill: the aggregate counts derived rows with `DerivedCount` so the matrix shows
history while flagging its weaker evidence. Tests pin the literals against the classifier
output so a wording change fails loudly.
## Daily aggregate — `VerdictCalibrationAggregates`
PK = `TenantId` | `"global"`, RK = `yyyy-MM-dd` (session **StartedAt** date, like every daily
aggregate). `VerdictCalibrationDailyAggregate { Version, SessionCount, TerminalSessionCount,
Buckets[], ComputedAt }` with one bucket per (VerdictPath, Status):
| Field | Meaning |
| --- | --- |
| `Count` / `DerivedCount` | sessions currently on the path; of those, derived read-side |
| `Eligible7d` | terminal sessions whose end (`CompletedAt ?? LastEventAt ?? StartedAt`) lies ≥ 7 d before compute time |
| `ReEnrolled7d` | of the eligible: the device's `DeviceHistories` chain shows the next terminal session starting < 7 d after this one's end |
| `OverriddenByAdmin` / `OverriddenByLateCompletion` / `OverriddenOther` | sessions whose **PriorVerdictPath** is this bucket — attributed to the path that was overridden, so a bucket can carry overrides with `Count = 0` |
`MaintenanceService.SweepVerdictCalibrationAsync` (2-hourly, after the device-journey sweep
so the chains are fresh; manual maintenance parity; kill switch
`VerdictCalibrationSweepDisabled`) slices the tick's shared projected window scan
(`LoadSweepWindowSessionsAsync` — one 35-day cross-tenant drain feeds the time-attribution,
device-journey and calibration sweeps plus the radar's tenant discovery; the StartedAt-only
filter is a full-table read, so never per sweep), recomputes the rolling 30-day window whole (Replace),
deletes stale date rows it did not regenerate, and the unbounded-tables cleanup applies the
180-day retention. Sessions inside a deletion cascade are skipped; the table is regenerable
and is wiped with the tenant on offboarding.
## Read surfaces
* `GET global/metrics/verdict-calibration?days=&tenantId=` (GlobalReadOrAdmin,
TenantScoping.QueryParam; `VerdictCalibrationResponse.Build`): per path — count, share of
all window sessions, derived count, eligible/re-enrolled with `reEnrollRatePct` **null below
20 eligible**, the three override counters, a today-anchored 7d-vs-28d share trend
(`window7`, `baseline28`, `lift` null without baseline) — plus totals, the trend basis and
the active `alerts[]`. Empty but well-formed before the first sweep.
* MCP `get_verdict_calibration` (platform scope: GA + Global Reader). Same endpoint, LLM-shaped in
`verdict-calibration-shape.ts`: per-row `window7.sessions` / `baseline28.sessions` are dropped (the
denominators live once in `trend`), `reEnrollRatePct` / `lift` are always present (`null` = withheld),
and `minSharePct` / `top` trim the one-session tail — rows carrying overrides are never trimmed and
everything dropped is reported in `omitted`. The wire shape the admin page consumes is untouched.
* Portal: Global Admin → Metrics → Verdict Calibration (`/admin/metrics/verdict-calibration`):
matrix grouped by origin, status pill, `derived n` gray pill, trend glyph (arrow only from
5 window hits and lift ≥ 2 / ≤ 0.5), re-enrollment rate or `— (n=…)`, overrides
`admin / late / other`, `↑ Drift` pill on rows with an active episode, and the active
episodes list.
## Drift radar — `VerdictCalibrationRadar`
Same statistics as the [rule regression radar](rule-regression-radar.md) (7-day window vs the
prior 28 days, ≥ 5 window hits, ≥ 20 window sessions, lift ≥ 2, `MetricsMath.
RateIncreaseSeparated`), anchored on yesterday, over every partition with rows (each tenant
and `global`):
| Kind | Signal | Re-arm |
| --- | --- | --- |
| `share_regression` | one **backend-decided** path's share of all sessions doubled (`sweep`/`maxlife`/`late`/`retro`/`rule`/`manual`/`ingest`; `agent:*` and `register:*` mirror customer workflow mix — first prod pass fired on `agent:whiteglove_pending` rollout weeks — and `legacy:*` is derived, all excluded) | share < 1.5× baseline, the path stops occurring, or the path is no longer eligible |
| `silence_share_regression` | the `sweep:*` + `maxlife:*` share doubled — the backend had to decide more often because the agent went silent (a liveness signal, not a classifier one) | same |
| `evidence_gap` | absolute: `r6` (pure fallthrough) decides ≥ 20 % of the window's classifier verdicts, ≥ 20 verdicts | < 15 % or < 20 verdicts |
Episodes are tracker rows `verdictcalibration|{kind}|{path}|{status}` (register = `AddEntity`,
409 = already burning, fail-closed; refresh carries `FirstNotifiedAt`; 30-day tracker
retention re-arms long burners). A new episode fires **one** `VerdictCalibrationDrift` ops
event (category Maintenance, Warning; dual-registered in the web `OPS_EVENT_TYPES`) — there
is deliberately **no tenant bell**. Per-path findings on a tenant partition get the on-fire
dimension correlation (`RuleRegressionRadar.ComputeDimensionConcentration` over the window's
sessions whose derived path matches), reported with the shared wording contract "correlated,
not necessarily causal"; group kinds and the global partition carry no dimension claim.
Kill switch `VerdictCalibrationRadarDisabled`.
# Examples
* **Reading the thermometer (prod, 2026-08-23 hand analysis that motivated this):**
re-enrollment within 7 days after Succeeded 8.9 % (lab background), after Failed 24.3 %,
after Incomplete 10.6 % — Incomplete behaves like Succeeded, so rules 5/6 are probably too
cautious; the matrix now shows this per rule instead of per status.
* **Correction stream:** a session swept to `sweep:r5_incomplete`, later completed by the
agent → row `agent:complete` with `PriorVerdictPath = sweep:r5_incomplete`; the aggregate
increments `OverriddenByLateCompletion` on the `sweep:r5_incomplete / Incomplete` bucket.
Four admin marks since launch were invisible before — now each one counts against the path
it corrected.
* **Evidence gap:** a tenant whose agent build stopped emitting the Device Setup rollup line
drives its sweep verdicts into `r6`; the absolute gate fires at 20 % and names the
tenant — the fix is agent-side, the classifier is only the messenger.
# Citations
* `src/Shared/AutopilotMonitor.Shared/Models/VerdictPaths.cs` — vocabulary + `ClassifierRules`
* `src/Backend/AutopilotMonitor.Functions/Services/TableStorageService.Sessions.cs` — `ComputePriorVerdict`, `ApplyVerdictPath`, re-registration preserve
* `src/Backend/AutopilotMonitor.Functions/Helpers/VerdictPathDerivation.cs`
* `src/Backend/AutopilotMonitor.Functions/Services/MaintenanceService.VerdictCalibration.cs` — sweep + `BuildVerdictCalibrationAggregates`
* `src/Backend/AutopilotMonitor.Functions/Functions/Metrics/GetVerdictCalibrationFunction.cs`
* `src/Backend/AutopilotMonitor.Functions/Helpers/VerdictCalibrationRadar.cs`, `Services/MaintenanceService.VerdictCalibrationRadar.cs`
* `src/McpServer/autopilot-monitor-mcp/src/tools/admin.ts` — `get_verdict_calibration`
* `src/McpServer/autopilot-monitor-mcp/src/verdict-calibration-shape.ts` — LLM-facing response shaping (`minSharePct` / `top`, explicit nulls)
* `src/Web/autopilot-monitor-web/app/admin/metrics/sections/SectionVerdictCalibration.tsx`
* Tests: `VerdictPathTests`, `VerdictCalibrationTests`, `VerdictCalibrationRadarTests`
* Related: [Silent-Session Classification](silent-session-classification.md), [Device Journeys](device-journeys.md), [Rule Regression Radar](rule-regression-radar.md)
+6 -1
View File
@@ -19,7 +19,7 @@ https://docs.autopilotmonitor.com (separate repository).
# Agent
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
* [V2 Agent](agent/index.md) - How the on-device agent works: [runtime overview](agent/overview.md), [decision engine](agent/decision-engine.md) (reducer, signals, completion arms), [logs & persistence](agent/logs-and-persistence.md) (signal log, snapshot, crash recovery), [diagnostics package](agent/diagnostics-package.md) (built-in section catalog shared with the portal, RealmJoin/Device Preparation collection gates, caps, manifest), [Hello wizard un-skip](agent/hello-wizard-unskip.md) (why a single "WHfB disabled" read never decides completion), [Autopilot ZTD diagnostics](agent/autopilot-ztd-diagnostics.md) (event IDs, diagnostic registry, endpoints, error-code map).
* [Build Counter Blob](agent/build-counter-blob.md) - How agent build numbers are minted: shared counter blob with ETag-CAS so local and CI builds never collide; CI additionally attests build provenance (Sigstore).
* [Agent Endpoint Migration](agent/endpoint-migration.md) - How the backend re-homes agents to a new API base URL via the config channel (backend move or per-tenant region move): live-fetch-only, allowlist-validated on both sides, one hop, kill wins.
* [Gather Rule Debug Log](agent/gather-rule-debug-log.md) - EnableGatherRuleDebugLog (ConfigVersion 36): the per-tenant local trace file that explains every gather-rule evaluation decision — scope skips, on_change suppression, empty collector results, logparser position/parse-mode outcomes — for rules that produce no timeline events.
@@ -45,14 +45,19 @@ https://docs.autopilotmonitor.com (separate repository).
# Backend
* [Telemetry Ingest Shaping](backend/telemetry-ingest-shaping.md) - One canonical App Insights request record (worker copy with tenant context, per-item sampling bypass; host HTTP duplicate dropped by a workspace transformation DCR), expected storage outcomes 404/412/409 and successful SignalR REST calls filtered in the worker, and the count()/sum(itemCount) query rules that follow.
* [Table Schema Sentinel](backend/table-schema-sentinel.md) - Startup no longer issues one CreateTableIfNotExists per table: a SHA-256 over TableNames.All in AdminConfiguration gates the full pass (derived, never hand-maintained), the SessionsIndex backfill runs only after a full pass behind a conditional-insert claim, and daily maintenance repairs out-of-band deletions.
* [Business Timestamps](backend/business-timestamps.md) - Why the Azure Tables system Timestamp is never authoritative (migrations reset it), and the OccurredUtc + RowKey-decode compensation for AuditLogs/OpsEvents/Events after the 2026-07-18 migration.
* [Send-Time Frame Separation](backend/send-time-frame-separation.md) - One device-clock send timestamp per upload batch (X-Send-Time-Utc → SentAt on Events) splits ReceivedAt − OccurredUtc into pure spool delay and pure device-vs-server clock offset; the clock_skew condition measures SentAt batches directly (spool-immune), the CMTrace tripwire deliberately stays on its differential math.
* [OOBE Bootstrap go-Code Flow](backend/oobe-bootstrap-go-flow.md) - The bootstrap short-code lifecycle (portal-created session → `irm go.autopilotmonitor.com/CODE | iex` at Shift+F10 → GetBootstrapScriptFunction), the always-HTTP-200 text/plain contract, the two-line injection defense (GUID/URL/expiry validator + single-quoted PS literals, golden-pinned against the legacy TS template), and the per-IP rate-limit story behind Front Door.
* [Silent-Session Classification](backend/silent-session-classification.md) - How a session whose agent went silent gets its honest status: the two-stage maintenance sweep (2h Stalled marker with the WhiteGlove Part-2 AwaitingUser gate and the self-deploying Succeeded gate, 5h timeout classification), the EnrollmentTimeoutClassifier rule order (explicit failure → WG Part 2 → self-deploying → completion evidence → emergency break → desktop+Hello → Device Setup/grace), the ingest paths sharing it (max-lifetime, late-telemetry heal, stall probe) and the admin retro-reconcile modes incl. self_deploying_silent.
* [Device Journeys & First-Time-Right](backend/device-journeys.md) - How a device's terminal enrollment attempts are grouped into journeys (trim+casefold device key with junk-serial exclusion, first-success end, 30-day gap, WhiteGlove = one attempt), the DeviceHistories chain with its tombstone-driven deleted-session pruning, the additive daily FTR aggregates, and the banner/fleet/MCP surfaces.
* [Enrollment Time Attribution](backend/time-attribution.md) - How a terminal session's authoritative wall clock (DurationSeconds, WhiteGlove pause excluded) is partitioned into named segments with an explicit unattributed remainder, ESP-blocking apps as positive evidence with "up to" what-if bounds, reboot spans from event gaps, and the breakdown/aggregate tables the sweep maintains.
* [Attempt-Scoped App Install Durations](backend/app-install-duration-attempt.md) - AppInstallSummary.DurationSeconds = status-defining (last) install attempt via the LastAttemptStartedAt max-fold anchor instead of the first-observation→last-terminal span that billed multi-pass IME gaps to trivial apps; InstallPassCount, the weaker-terminal guard, the radar's cutover gate, and time attribution's union-of-attempt-segments occupancy.
* [Rule Regression Radar](backend/rule-regression-radar.md) - How analyze-rule hit-rate regressions are detected (7-day window ≥2× the 28-day baseline with disjoint Wilson intervals), which suppressions keep the radar quiet (grace period, edit-in-window, kill switch), how alert episodes live in the ruleregression tracker keyspace with re-arm semantics, and the on-fire dimension correlation with its "correlated, not necessarily causal" wording contract.
* [Verdict Calibration](backend/verdict-calibration.md) - How every session status write declares its VerdictPath (origin:detail vocabulary incl. classifier rule ids), how overriding a verdict preserves PriorStatus/PriorVerdictPath as the correction stream, the daily per-path calibration aggregate with the 7-day re-enrollment proxy and read-side legacy derivation, the GA endpoint / MCP tool / admin matrix, and the drift radar (share, silence-share, r6 evidence gap) with its operator-only ops events.
* [App Version Duration Regression Radar](backend/app-version-regression-radar.md) - How app versions whose median install duration regressed vs. the previous version are detected (first-seen-ordered versions, median lift ≥2× plus a 5-minute absolute floor over measured installs only), how alert episodes live in the appversionregression tracker keyspace with drain/median re-arm semantics, and the bell/ops-event/analytics surfaces.
* [Plan Downgrade Policy & the Retention Grace Period](backend/plan-downgrade-retention-grace.md) - What happens when a tenant loses Pro: every entitlement gates read-time and reversibly ("gate, don't destroy"), except retention — a 30-day grace keeps the Pro cap before the Community cap starts hard-deleting; the ProDowngradedUtc/trial-expiry anchor model, the TenantPlanDowngraded/RetentionGrace* ops events, and the deliberate non-grace of everything else.
* [Tenant Activation & the Once-Only Welcome Mail](backend/tenant-activation-welcome-mail.md) - How a signup becomes an activated tenant (manual approve or the auto-approve queue), and the write-then-read + conditional-marker arbitration that sends the welcome mail exactly once even though activation and the user-typed notification address race each other.
* [W365 Cloud PC Device Validation](backend/cloudpc-device-validation.md) - Why Cloud PCs can never pass the Autopilot serial lookup, and the cert-CN-bound CloudPc fallback validator: Subject CN (= Intune device id) matched against the tenant's Graph virtualEndpoint/cloudPCs inventory — admits exactly the service-provisioned Cloud PCs, nothing else.
* [ServerActions & On-Demand Diagnostics](backend/server-actions-on-demand-diagnostics.md) - The queue-on-session, deliver-on-ingest action channel, the TenantAdminOrOperator tier with in-function per-type re-gating, the portal's Collect Logs flow (quick-config orders rotate_config before request_diagnostics), and the hosted-prefix sweep in the offboarding cascade.
+20
View File
@@ -1,5 +1,25 @@
# Log
## 2026-08-23
* **New**: `backend/telemetry-ingest-shaping.md` — `RequestTelemetryMiddleware` sets `SamplingPercentage=100` on its own item (worker request rows were adaptively sampled, ItemCount 2–5 on weekdays, despite host.json `excludedTypes`), the redundant host HTTP request copy is dropped by a workspace transformation DCR (`infra/appinsights-workspace-transforms.bicep`, timer/queue rows kept), and `StorageDependencyFilterProcessor` now also drops expected storage outcomes (404/412/409, both InProc and HTTP shapes) and successful SignalR REST calls; the host-emitted SignalR binding calls (bulk of the volume, SDK `rdddsc`) are dropped by a second `AppDependencies` flow in the same DCR. Measured ≈ −1.3 GB/week.
* **Update**: `backend/table-schema-sentinel.md` — "Measuring it": `StartupTelemetryService` emits `BackendStartupMs` (process start → ApplicationStarted) and `BackendTableInitMs` as App Insights custom metrics with version/commit/fullPass dimensions; the MCP server writes one `{"type":"boot","readyMs"}` stderr line per start (query in `.claude/commands/mcp-telemetry.md` §A2).
* **New**: `backend/table-schema-sentinel.md` — startup table initialization gated by a schema sentinel (SHA-256 over `TableNames.All` in `AdminConfiguration`/`SchemaSentinel`/`tables`): matching sentinel ⇒ zero `CreateTableIfNotExists` calls on a cold start, missing/stale ⇒ one full pass + sentinel rewrite. SessionsIndex startup backfill only after a full pass, `IsSessionIndexEmptyAsync` no longer treats errors as "empty", conditional-insert claim so one scaled-out instance backfills. Daily maintenance runs `IStorageInitializer.EnsureAllAsync` to repair out-of-band table deletions.
* **Update**: `backend/verdict-calibration.md` — MCP `get_verdict_calibration` response shaping: per-row trend denominators dropped, explicit nulls, `minSharePct` / `top` with an `omitted` report.
* **Update**: `backend/verdict-calibration.md` — share-regression radar restricted to backend-decided paths; agent-declared/registration paths (customer workflow mix) and derived legacy paths no longer alert, stale episodes re-arm themselves.
* **Update**: `backend/time-attribution.md`, `backend/device-journeys.md`, `backend/verdict-calibration.md` — the rolling maintenance sweeps (time attribution, device journeys, verdict calibration, calibration-radar discovery) now share ONE projected 35-day cross-tenant Sessions scan per tick (`LoadSweepWindowSessionsAsync` + `MaintenanceSweepSessionProjection`, pinned by `MaintenanceSweepProjectionEquivalenceTests`) instead of four full-table drains; each sweep slices its own window by StartedAt.
* **New**: `backend/verdict-calibration.md` — the classifier thermometer: every status write now stamps `VerdictPath` (required parameter on `UpdateSessionStatusAsync`; `EnrollmentTimeoutClassifier` returns its rule id), overriding a verdict preserves `PriorStatus`/`PriorVerdictPath`, `VerdictPathDerivation` attributes pre-instrumentation rows read-side (`legacy:*` when ambiguous). New daily aggregate `VerdictCalibrationAggregates` (per-path count/derived/eligible7d/reEnrolled7d/overridden-by-admin|late|other, sweep after the device-journey pass, 180d retention), `GET global/metrics/verdict-calibration`, MCP `get_verdict_calibration`, admin metrics section, and the `VerdictCalibrationRadar` (share / silence-share / r6 evidence-gap episodes in the tracker keyspace `verdictcalibration|`, `VerdictCalibrationDrift` ops event, no tenant bell).
* **New**: `backend/silent-session-classification.md` — one place for the silent-session verdict: `EnrollmentTimeoutClassifier` rule order incl. the new rule 1c `IsSelfDeployingProvisioned` (self-deploying profile + Device Setup all-succeeded + no explicit failure ⇒ Succeeded reconcile, no grace/AwaitingUser — kiosk tenant aebdce78 audit: 836 finished devices parked AwaitingUser/Incomplete), the sweep stage-1 gates (`TryMarkWhiteGloveAwaitingUserAsync`, new `TryReconcileSelfDeployingAsync`), the ingest callers (max-lifetime, late-telemetry heal, stall probe) and the new admin retro mode `self_deploying_silent` (`GetSelfDeployingSilentSessionsAsync`). Agent companion: `SessionRegistrationHelper` waits up to 15s for a network link and retries 6× (~62s) instead of 5× (~30s) so the reboot relaunch on Wi-Fi no longer exits 7 before the link is back.
* **New**: `agent/diagnostics-package.md` — the diagnostics ZIP's built-in collection is now ONE catalog in Shared (`DiagnosticsBuiltInSections.All`): the agent iterates it in order and the backend serves the same list to the portal (`GET diagnostics/paths`, MemberRead), so what administrators see is exactly what the agent collects. New conditional sections: RealmJoin client/package/Chocolatey/per-user logs behind the tenant's RealmJoin Watcher toggle (mirrored onto `AgentConfiguration.EnableRealmJoinWatcher` by the merger) and the Device Preparation bootstrapper event log behind the deterministic WDP marker (probe once per build, fail toward Classic); every skip is a manifest line. Built-ins bypass `DiagnosticsPathGuards` by design (reviewed code, `%ProgramData%\RealmJoin` is not allowlisted); caps/reparse/evtx export still apply. `MergeDiagnosticsLogPaths` dedups global ∪ tenant paths. Portal: collapsed read-only catalog + one-line rows in both cards, tenant context dropped the GA-only global fetch.
* **Update**: `rules/gather-rule-guardrails.md` — `DiagnosticsPathGuards` scope clarified: configured diagnostics paths only, never the built-in catalog.
* **Update**: `backend/tenant-activation-welcome-mail.md` — new "Template overrides" section: Global Admins can replace the welcome/farewell HTML in the portal (Settings → Email Templates); `EmailTemplateService` resolves override-or-built-in per send (PreviewConfig table, partition `EmailTemplates`, 5-min cache, `{{domainName}}` placeholder, 30k limit, fail-soft send path), GA endpoints `global/email-templates/{kind}` (+`/test` to the caller's tenant contact address). Also: the mail header is image-free (CSS-drawn brand mark, gradient stripes and clip-path edge as progressive enhancement).
## 2026-08-22
* **Update**: `backend/tenant-activation-welcome-mail.md` — new "Mail transport" section: transactional mail now goes through the provider-neutral `EmailService` (`IEmailService` + `IOffboardFarewellEmailSender`) with the provider confined to one private transport method and the `Email:*` settings (`ApiKey`, `Endpoint`, `FromAddress`, `FromName`); current provider Mailchimp Transactional (Mandrill) replacing Resend (NuGet dropped, typed `HttpClient` + `ResiliencePolicies.Notification`). Result-array semantics (`sent`/`queued`/`scheduled` ok, `rejected`/`invalid` warn), open/click tracking explicitly off because the privacy/trust claim depends on it.
* **New**: `backend/plan-downgrade-retention-grace.md` — plan downgrade policy: every Pro entitlement gates read-time and reversibly on downgrade ("gate, don't destroy"), except retention, which now gets a 30-day grace (`FeatureEntitlementCatalog.RetentionDowngradeGraceDays`) before the Community cap starts hard-deleting the 90–365-day band. Anchor model: `ProDowngradedUtc` stamped/cleared by the plan endpoint's pure `ApplyPlanChanges` core on EFFECTIVE-edition transitions (covers explicitly ended trials; planTier downgrade under an active trial deliberately doesn't stamp), trial expiry anchors read-time via `TrialExpiresUtc`. New Warning ops events `TenantPlanDowngraded` (PATCH handler) and `TenantRetentionGraceExpiring`/`TenantRetentionGraceEnded` (TrialExpirySweep window mechanics, only when stored retention exceeds the Community cap). Field is patch-denied + revert-protected; retention WRITE cap deliberately ignores the grace.
## 2026-08-21
* **New**: `agent/do-group-id-auto-set.md` — optional DO peering-group seed (`EnableDoGroupIdAutoSet`, default off, ConfigVersion 40): `DoGroupIdService` derives a deterministic GUID from the default gateway's IPv4 + SendARP MAC in RealmJoin's exact byte layout (cross-product peering) and writes `DOGroupId` in the 64-bit GPO Policies store with readback verification. Conservative by design: no random-GUID fallback (skip on missing gateway/MAC), `DOGroupIdSource`/MDM-managed values/foreign GPO values always win (skip reasons in the `do_group_id_auto_set` event), own stale values are recognized via the `written:<guid>` StartupEventGate fingerprint and updated after a gateway change. Gateway heuristic extracted into the shared `NetworkInterfaceLocator` (dedup of NetworkChangeDetector + DeviceInfoCollector copies).
+6
View File
@@ -31,6 +31,12 @@ item in `AutopilotMonitor.Agent.V2.Core.csproj`) and regenerated into the portal
| `IsCommandAllowed` | `allowedCommands` | **exact** match, trimmed — never prefix |
| `IsEventLogChannelAllowed` | `eventLogChannels` | prefix, boundary `/` or end |
`DiagnosticsPathGuards.IsDiagnosticsPathAllowed` (allowlist `diagnosticsPathPrefixes`, a C#
literal mirrored in the JSON) applies the same hard blocks and the same user-profile
exception to **admin-configured diagnostics paths** only. The diagnostics package's built-in
sections are reviewed code and never pass through it — see
[Diagnostics Package](../agent/diagnostics-package.md).
**Hard blocks live in code, not in the JSON**, so a parse error cannot lift them — on a
parse failure the allowlists fall back to empty and everything is blocked (fail-closed),
but the hard blocks must hold regardless of configuration:
@@ -0,0 +1,78 @@
// Autopilot-Monitor — Log Analytics workspace transformation (ingestion-time filter)
//
// Two ingestion-time filters. (2) Successful Azure SignalR REST dependencies emitted by the
// HOST process for the SignalR output binding (SDKVersion rdddsc:*) — the worker-side
// StorageDependencyFilterProcessor never sees those. Failed SignalR calls are kept.
// (1) Drops the redundant HOST-side HTTP request record from AppRequests so that exactly one
// canonical request row remains: the worker-side copy emitted by RequestTelemetryMiddleware
// (Properties.Source == 'WorkerMiddleware', carries TenantId/UserId/CorrelationId and is
// sampling-exempt per item). Non-HTTP host invocations (timer/queue triggers) have an empty
// Url and are kept — the worker middleware never sees them.
//
// Why ingestion-time: the host request item is emitted by the Functions host process and
// cannot be suppressed from worker code or host.json (verified 2026-06-09, see Program.cs).
// Live numbers 2026-08-23 (7d): host copy 274k rows / 323 MB, worker copy 189k rows / 208 MB.
//
// Cost note: transformation filtering is free up to 50% of the table's incoming volume; the
// portion beyond that (here ~11% of AppRequests) is billed at the (cheap) data-processing rate.
//
// Apply (two steps; the link is a CLI update = GET+PUT, which keeps retention/SKU intact):
// az deployment group create -g rg-autopilotmonitor-prd-gwc \
// --template-file infra/appinsights-workspace-transforms.bicep \
// --query properties.outputs.dcrId.value -o tsv
// az monitor log-analytics workspace update -g rg-autopilotmonitor-prd-gwc -n AutopilotMonitor \
// --data-collection-rule <dcrId-from-above>
//
// Verify after ~20 min (ingestion latency):
// AppRequests | where TimeGenerated > ago(15m) | where AppRoleName == 'autopilotmonitor-api-eu'
// | summarize rows=count() by source=tostring(Properties.Source), hasUrl=isnotempty(Url)
// → only (WorkerMiddleware, true) and ('', false) buckets remain.
//
// Rollback: az monitor log-analytics workspace update -g <rg> -n AutopilotMonitor
// --set properties.defaultDataCollectionRuleResourceId=null (then delete the DCR)
@description('Log Analytics workspace that backs the workspace-based Application Insights resources.')
param workspaceName string = 'AutopilotMonitor'
param location string = resourceGroup().location
resource workspace 'Microsoft.OperationalInsights/workspaces@2023-09-01' existing = {
name: workspaceName
}
resource transforms 'Microsoft.Insights/dataCollectionRules@2023-03-11' = {
name: 'dcr-autopilotmonitor-ws-transforms'
location: location
kind: 'WorkspaceTransforms'
properties: {
description: 'AppRequests: keep the enriched worker request record (Properties.Source == WorkerMiddleware) and non-HTTP host invocations (empty Url); drop the host HTTP duplicate.'
destinations: {
logAnalytics: [
{
workspaceResourceId: workspace.id
name: 'ws'
}
]
}
dataFlows: [
{
streams: [ 'Microsoft-Table-AppRequests' ]
destinations: [ 'ws' ]
transformKql: 'source | where tostring(Properties.Source) == \'WorkerMiddleware\' or isempty(Url)'
}
{
// Successful Azure SignalR REST calls (group add/remove per connection) are emitted by
// the HOST process for the SignalR output binding (SDKVersion rdddsc:*), so the worker's
// StorageDependencyFilterProcessor never sees them. Failed SignalR calls are kept.
streams: [ 'Microsoft-Table-AppDependencies' ]
destinations: [ 'ws' ]
transformKql: 'source | where not(Success == true and Target endswith \'.service.signalr.net\')'
}
]
}
}
// A workspace has exactly one transformation DCR, activated by linking it as the workspace's
// defaultDataCollectionRuleResourceId — deliberately NOT modelled here (see Apply above).
output dcrId string = transforms.id
@@ -53,6 +53,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// Merge reads two Collectors knobs (AgentMaxLifetimeMinutes, HelloWaitTimeoutSeconds);
// the remaining CollectorConfiguration flags flow untouched to DefaultComponentFactory.
nameof(AgentConfigResponse.Collectors),
// Merge reads one Analyzers knob (EnableRealmJoinWatcher → diagnostics RealmJoin sections);
// the remaining flags flow untouched to DefaultComponentFactory + AgentAnalyzerManager.
nameof(AgentConfigResponse.Analyzers),
};
/// <summary>
@@ -70,8 +73,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
nameof(AgentConfigResponse.ImeLogPatterns),
// Passed by AgentRuntimeHost into IComponentFactory.CreateCollectorHosts → ImeLogHost sealing re-emit.
nameof(AgentConfigResponse.WhiteGloveSealingPatternIds),
// Consumed by DefaultComponentFactory (RealmJoin/KeepAwake/ConsoleBypass host gates) + AgentAnalyzerManager.
nameof(AgentConfigResponse.Analyzers),
// Self-update ZIP integrity second trust channel — BinaryIntegrityVerifier / Program update path.
nameof(AgentConfigResponse.LatestAgentSha256),
// Running-EXE integrity verification — BinaryIntegrityVerifier post-config check.
@@ -368,6 +368,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.Equal(30, agent.HelloWaitTimeoutSeconds);
}
[Fact]
public void Merge_mirrors_analyzers_realmjoin_watcher_onto_agent_config()
{
// The diagnostics package gates its RealmJoin sections on this flag but never
// sees the remote response — the merger carries the one Analyzers knob over.
var agent = NewAgentConfig();
var remote = FullRemote();
remote.Analyzers = new AnalyzerConfiguration { EnableRealmJoinWatcher = true };
RemoteConfigMerger.Merge(agent, remote);
Assert.True(agent.EnableRealmJoinWatcher);
}
[Fact]
public void Merge_clears_realmjoin_watcher_when_remote_analyzers_missing()
{
var agent = NewAgentConfig();
agent.EnableRealmJoinWatcher = true; // stale cached true must not survive
var remote = FullRemote();
remote.Analyzers = null!;
RemoteConfigMerger.Merge(agent, remote);
Assert.False(agent.EnableRealmJoinWatcher);
}
[Fact]
public void Merge_replaces_diagnostics_log_paths_with_empty_list_when_remote_null()
{
@@ -128,6 +128,94 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
Assert.Equal(4, svc.LastFetchAttempts);
}
// ── Network-link gate (edu.kosh audit 2026-08-24) ───────────────────
[Fact]
public async Task FetchConfig_InitialFetch_WaitsForNetworkLink()
{
// The initial startup fetch must gate on the link-level signal so the retry
// budget isn't burned into a NIC that is provably still down (agent relaunched
// at boot, Wi-Fi still associating).
var svc = NewService(BackendBehaviour.SuccessImmediately);
await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(1, svc.NetworkLinkWaits);
}
[Fact]
public async Task FetchConfig_RotatePath_SkipsNetworkLinkGate()
{
// rotate_config runs mid-session on a proven link and must stay latency-bounded.
var svc = NewService(BackendBehaviour.SuccessImmediately);
await svc.FetchConfigAsync(retryOnTransientErrors: false);
Assert.Equal(0, svc.NetworkLinkWaits);
}
// ── Post-registration recovery ──────────────────────────────────────
[Fact]
public async Task TryRecover_AfterFailedFetch_AppliesLiveConfig_KeepsFailureRecord()
{
// The core self-heal contract: registration proved the backend reachable, the
// recovery swaps in the live config — but the LastFetch* observability keeps
// describing the failed startup fetch so the wire event can report both.
var svc = NewService(BackendBehaviour.FailAllThenSucceed);
await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
var recovered = await svc.TryRecoverConfigAsync();
Assert.NotNull(recovered);
Assert.Equal(42, recovered.ConfigVersion);
Assert.Equal(42, svc.CurrentConfig.ConfigVersion);
Assert.True(svc.RecoveredAfterFetchFailure);
// Startup-failure record intact:
Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
Assert.Equal(1 + RemoteConfigService.InitialFetchRetryBackoff.Length, svc.LastFetchAttempts);
Assert.Equal(nameof(HttpRequestException), svc.LastFetchFailureType);
}
[Fact]
public async Task TryRecover_AfterSuccessfulFetch_IsNoOp()
{
var svc = NewService(BackendBehaviour.SuccessImmediately);
await svc.FetchConfigAsync(retryOnTransientErrors: true);
var recovered = await svc.TryRecoverConfigAsync();
Assert.Null(recovered);
Assert.False(svc.RecoveredAfterFetchFailure);
}
[Fact]
public async Task TryRecover_BeforeAnyFetch_IsNoOp()
{
var svc = NewService(BackendBehaviour.SuccessImmediately);
var recovered = await svc.TryRecoverConfigAsync();
Assert.Null(recovered);
Assert.False(svc.RecoveredAfterFetchFailure);
}
[Fact]
public async Task TryRecover_WhenBackendStillFailing_KeepsFallback_NeverThrows()
{
var svc = NewService(BackendBehaviour.AlwaysTransientFailure);
var fallback = await svc.FetchConfigAsync(retryOnTransientErrors: true);
Assert.Equal(0, fallback.ConfigVersion);
var recovered = await svc.TryRecoverConfigAsync();
Assert.Null(recovered);
Assert.False(svc.RecoveredAfterFetchFailure);
Assert.Equal(0, svc.CurrentConfig.ConfigVersion); // fallback config untouched
Assert.Equal(RemoteConfigFetchOutcome.UsedDefaults, svc.LastFetchOutcome);
}
// ── Outcome state defaults ──────────────────────────────────────────
[Fact]
@@ -145,6 +233,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
SuccessImmediately,
AlwaysTransientFailure,
FailTwiceThenSucceed,
/// <summary>All 4 initial attempts fail; the NEXT call (the recovery fetch) succeeds.</summary>
FailAllThenSucceed,
AlwaysAuthFailure401,
AlwaysReturnNull,
}
@@ -177,6 +267,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
_behaviour = behaviour;
}
/// <summary>Number of times the network-link gate seam was awaited.</summary>
public int NetworkLinkWaits { get; private set; }
protected override Task<AgentConfigResponse> CallBackendAsync()
{
_calls++;
@@ -188,6 +281,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
if (_calls < 3)
throw new HttpRequestException("simulated cold-start timeout");
return Task.FromResult(new AgentConfigResponse { ConfigVersion = 42 });
case BackendBehaviour.FailAllThenSucceed:
if (_calls <= 1 + RemoteConfigService.InitialFetchRetryBackoff.Length)
throw new HttpRequestException("simulated dead network link");
return Task.FromResult(new AgentConfigResponse { ConfigVersion = 42 });
case BackendBehaviour.AlwaysTransientFailure:
throw new HttpRequestException("simulated network failure");
case BackendBehaviour.AlwaysAuthFailure401:
@@ -202,6 +299,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// Zero-delay so the suite finishes in ms instead of the live 10/30/60 s schedule.
protected override Task DelayBetweenAttemptsAsync(TimeSpan delay) => Task.CompletedTask;
// Record + skip the real link poll so the suite is deterministic on offline CI.
protected override Task WaitForNetworkLinkAsync()
{
NetworkLinkWaits++;
return Task.CompletedTask;
}
// In-memory + per-instance cache so a prior test's successful fetch can't
// bleed into a later test that expects UsedDefaults.
private AgentConfigResponse? _cached;
@@ -1,9 +1,11 @@
#nullable enable
using System.Collections.Generic;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Text;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Shared.Models;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
@@ -46,7 +48,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
Directory.CreateDirectory(SpoolFolder);
}
public DiagnosticsPackageService Build(System.Action<AgentConfiguration>? mutateConfig = null)
public DiagnosticsPackageService Build(
System.Action<AgentConfiguration>? mutateConfig = null,
IReadOnlyDictionary<string, string>? sectionFolderOverrides = null,
System.Func<bool>? devicePreparationProbe = null)
{
// BackendApiClient is required by the public ctor but BuildArchiveBytes
// never touches it — construct with a throwaway HttpClient. Tests only
@@ -71,7 +76,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
imeLogFolderOverride: ImeTmp.Path,
agentStateFolderOverride: StateFolder,
agentSpoolFolderOverride: SpoolFolder,
agentDataFolderOverride: DataFolder);
agentDataFolderOverride: DataFolder,
sectionFolderOverrides: sectionFolderOverrides,
devicePreparationProbe: devicePreparationProbe);
}
public void Dispose()
@@ -287,6 +294,185 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
Assert.DoesNotContain(entries, e => e.StartsWith("AgentSpool/"));
}
// ===================================================== built-in catalog gates ====
// The RealmJoin sections ride behind the tenant's RealmJoin Watcher toggle, the
// Device Preparation bootstrapper event log behind the deterministic WDP marker.
// Every conditional section is skipped WITH a manifest line, never silently.
/// <summary>RealmJoin system-side folders redirected to temp (user-side goes via UserProfileResolver).</summary>
private sealed class RealmJoinRig : System.IDisposable
{
public TempDirectory WindowsLogs { get; } = new TempDirectory();
public TempDirectory Choco { get; } = new TempDirectory();
public TempDirectory UserProfile { get; } = new TempDirectory();
public string PackagesFolder => Path.Combine(WindowsLogs.Path, "RealmJoin");
public RealmJoinRig()
{
File.WriteAllText(Path.Combine(WindowsLogs.Path, "realmjoin.log"), "rj");
File.WriteAllText(Path.Combine(WindowsLogs.Path, "realmjoin2.log"), "rj2");
var pkg = Path.Combine(PackagesFolder, "Packages", "generic-7zip");
Directory.CreateDirectory(pkg);
File.WriteAllText(Path.Combine(pkg, "install.log"), "pkg");
var choco = Path.Combine(Choco.Path, "generic-7zip");
Directory.CreateDirectory(choco);
File.WriteAllText(Path.Combine(choco, "2026-06-30_install.log"), "choco");
var userRj = Path.Combine(UserProfile.Path, "AppData", "Local", "RealmJoin");
Directory.CreateDirectory(Path.Combine(userRj, "Logs", "generic-office"));
File.WriteAllText(Path.Combine(userRj, "tray.log"), "tray");
File.WriteAllText(Path.Combine(userRj, "config.pjson"), "not a log");
File.WriteAllText(Path.Combine(userRj, "Logs", "RjImeHost.log"), "host");
File.WriteAllText(Path.Combine(userRj, "Logs", "generic-office", "usersettings.log"), "pkg-user");
}
public IReadOnlyDictionary<string, string> Overrides => new Dictionary<string, string>
{
["RealmJoinWindows"] = WindowsLogs.Path,
["RealmJoinPackages"] = PackagesFolder,
["RealmJoinChoco"] = Choco.Path,
};
public void Dispose()
{
WindowsLogs.Dispose();
Choco.Dispose();
UserProfile.Dispose();
}
}
[Fact]
public void BuildArchiveBytes_skips_realmjoin_sections_when_watcher_disabled()
{
using var rig = new Rig();
using var rj = new RealmJoinRig();
UserProfileResolver.SetForTesting(rj.UserProfile.Path);
try
{
var bytes = rig.Build(sectionFolderOverrides: rj.Overrides).BuildArchiveBytes(enrollmentSucceeded: true);
var entries = ZipEntryNames(bytes);
Assert.DoesNotContain(entries, e => e.StartsWith("RealmJoinLogs/"));
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
Assert.Contains("SCENARIO: devicePreparation=False realmJoinWatcher=False", manifest);
Assert.Contains("BUILT-IN SKIPPED (RealmJoin Watcher disabled): RealmJoinWindows", manifest);
Assert.Contains("BUILT-IN SKIPPED (RealmJoin Watcher disabled): RealmJoinUserLogs", manifest);
}
finally
{
UserProfileResolver.Reset();
}
}
[Fact]
public void BuildArchiveBytes_includes_realmjoin_sections_when_watcher_enabled()
{
using var rig = new Rig();
using var rj = new RealmJoinRig();
UserProfileResolver.SetForTesting(rj.UserProfile.Path);
try
{
var bytes = rig.Build(cfg => cfg.EnableRealmJoinWatcher = true, rj.Overrides)
.BuildArchiveBytes(enrollmentSucceeded: false);
var entries = ZipEntryNames(bytes);
// ZIP layout mirrors the disk layout: flat client logs + package tree under
// Windows/, tray logs + per-user Logs tree under User/.
Assert.Contains("RealmJoinLogs/Windows/realmjoin.log", entries);
Assert.Contains("RealmJoinLogs/Windows/realmjoin2.log", entries);
Assert.Contains("RealmJoinLogs/Windows/RealmJoin/Packages/generic-7zip/install.log", entries);
Assert.Contains("RealmJoinLogs/Choco/generic-7zip/2026-06-30_install.log", entries);
Assert.Contains("RealmJoinLogs/User/tray.log", entries);
Assert.Contains("RealmJoinLogs/User/Logs/RjImeHost.log", entries);
Assert.Contains("RealmJoinLogs/User/Logs/generic-office/usersettings.log", entries);
// Only tray*.log is collected from the user RealmJoin root — config stays out.
Assert.DoesNotContain(entries, e => e.EndsWith("config.pjson"));
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
Assert.Contains("BUILT-IN: RealmJoinUserLogs -> folder=", manifest);
Assert.DoesNotContain("BUILT-IN SKIPPED (RealmJoin Watcher disabled)", manifest);
}
finally
{
UserProfileResolver.Reset();
}
}
[Fact]
public void BuildArchiveBytes_skips_realmjoin_user_sections_without_user_session()
{
using var rig = new Rig();
using var rj = new RealmJoinRig();
// Token present, no interactive user detected → user-profile sections skip with a
// manifest line; the system-side sections are unaffected.
UserProfileResolver.SetForTesting(null!);
try
{
var bytes = rig.Build(cfg => cfg.EnableRealmJoinWatcher = true, rj.Overrides)
.BuildArchiveBytes(enrollmentSucceeded: true);
var entries = ZipEntryNames(bytes);
Assert.Contains("RealmJoinLogs/Windows/realmjoin.log", entries);
Assert.DoesNotContain(entries, e => e.StartsWith("RealmJoinLogs/User"));
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
Assert.Contains("BUILT-IN SKIPPED (no user session for token): RealmJoinUserTray", manifest);
Assert.Contains("BUILT-IN SKIPPED (no user session for token): RealmJoinUserLogs", manifest);
}
finally
{
UserProfileResolver.Reset();
}
}
[Fact]
public void BuildArchiveBytes_collects_bootstrapper_event_log_only_on_device_preparation()
{
using var rig = new Rig();
using var winevt = new TempDirectory();
// Synthetic evtx: no registered channel claims the file, so the export falls
// back to a raw copy — the point here is the scenario gate, not wevtutil.
File.WriteAllBytes(Path.Combine(winevt.Path, "BootstrapperAgentServiceLogProvider.evtx"), new byte[] { 1, 2, 3 });
var overrides = new Dictionary<string, string> { ["ImeBootstrapperEventLog"] = winevt.Path };
var classic = rig.Build(sectionFolderOverrides: overrides, devicePreparationProbe: () => false)
.BuildArchiveBytes(enrollmentSucceeded: true);
Assert.DoesNotContain("ImeLogs/BootstrapperAgentServiceLogProvider.evtx", ZipEntryNames(classic));
Assert.Contains("BUILT-IN SKIPPED (not a Device Preparation enrollment): ImeBootstrapperEventLog",
ReadZipEntryText(classic, "package-manifest.txt"));
var wdp = rig.Build(sectionFolderOverrides: overrides, devicePreparationProbe: () => true)
.BuildArchiveBytes(enrollmentSucceeded: true);
Assert.Contains("ImeLogs/BootstrapperAgentServiceLogProvider.evtx", ZipEntryNames(wdp));
Assert.Contains("SCENARIO: devicePreparation=True", ReadZipEntryText(wdp, "package-manifest.txt"));
}
[Fact]
public void BuildArchiveBytes_treats_failing_device_preparation_probe_as_classic()
{
using var rig = new Rig();
var bytes = rig.Build(devicePreparationProbe: () => throw new System.InvalidOperationException("registry boom"))
.BuildArchiveBytes(enrollmentSucceeded: true);
var manifest = ReadZipEntryText(bytes, "package-manifest.txt")!;
Assert.Contains("SCENARIO PROBE FAILED (treated as Classic): registry boom", manifest);
Assert.Contains("BUILT-IN SKIPPED (not a Device Preparation enrollment): ImeBootstrapperEventLog", manifest);
Assert.Contains("sessioninfo.txt", ZipEntryNames(bytes)); // archive still built
}
[Theory]
[InlineData(DiagnosticsSectionCondition.Always, false, false, true)]
[InlineData(DiagnosticsSectionCondition.RealmJoinWatcher, false, true, false)]
[InlineData(DiagnosticsSectionCondition.RealmJoinWatcher, true, false, true)]
[InlineData(DiagnosticsSectionCondition.DevicePreparation, true, false, false)]
[InlineData(DiagnosticsSectionCondition.DevicePreparation, false, true, true)]
[InlineData((DiagnosticsSectionCondition)999, true, true, false)]
public void IsSectionActive_gates_on_condition(DiagnosticsSectionCondition condition, bool watcher, bool wdp, bool expected)
{
var section = new DiagnosticsBuiltInSection("X", "X", @"C:\X", new[] { "*.log" }, false, "x", condition);
var cfg = Cfg();
cfg.EnableRealmJoinWatcher = watcher;
Assert.Equal(expected, DiagnosticsPackageService.IsSectionActive(section, cfg, wdp));
}
private static string[] ZipEntryNames(byte[] zipBytes)
{
using var ms = new MemoryStream(zipBytes);
@@ -1,5 +1,6 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.DecisionCore.Signals;
@@ -11,8 +12,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
internal sealed class FakeSignalIngressSink : ISignalIngressSink
{
private readonly List<PostedSignal> _posted = new List<PostedSignal>();
private readonly object _gate = new object();
public IReadOnlyList<PostedSignal> Posted => _posted;
/// <summary>
/// Snapshot, not the live list: producers post from timer / worker threads (e.g. the
/// parked-dwell tripwire) while tests poll this in a SpinWait loop. Enumerating the live
/// list across an Add throws "Collection was modified" — a rare CI-only failure
/// (run 32644425162) with no assertion text.
/// </summary>
public IReadOnlyList<PostedSignal> Posted
{
get { lock (_gate) return _posted.ToArray(); }
}
/// <summary>When non-null, <see cref="Post"/> throws instead of capturing.</summary>
public Exception? ThrowOnPost { get; set; }
@@ -27,7 +38,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
object? typedPayload = null)
{
if (ThrowOnPost != null) throw ThrowOnPost;
_posted.Add(new PostedSignal(kind, occurredAtUtc, sourceOrigin, evidence, payload, kindSchemaVersion, typedPayload));
lock (_gate)
_posted.Add(new PostedSignal(kind, occurredAtUtc, sourceOrigin, evidence, payload, kindSchemaVersion, typedPayload));
}
internal sealed class PostedSignal
@@ -182,6 +182,73 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
Assert.Equal("FromCache", data["outcome"]);
}
// ── Post-registration recovery (Phase 6.5, edu.kosh audit 2026-08-24) ─
[Fact]
public void EmitFetchFailedIfAny_AfterRecovery_MarksRecoveredAndDemotesToInfo()
{
using var rig = new EmitterRig();
// Startup fetch failed (network link not up yet), then the post-registration
// recovery fetched ConfigVersion 39. The event must keep the original failure
// details AND mark the heal.
rig.RemoteConfig.SetFakeOutcome(
RemoteConfigFetchOutcome.UsedDefaults,
configVersion: 39, // CurrentConfig now holds the recovered config
attempts: 4,
failureType: "HttpRequestException",
failureMessage: "simulated dead network link",
recovered: true);
LifecycleEmitters.EmitRemoteConfigFetchFailedIfAny(
rig.Post, rig.AgentConfig, rig.RemoteConfig, rig.Logger);
var data = AssertSingleEmit(rig.Sink, expectedEventType: "remote_config_fetch_failed");
Assert.Equal("UsedDefaults", data["outcome"]); // startup failure record intact
Assert.Equal(4, data["attempts"]);
Assert.Equal(true, data["recovered"]);
Assert.Equal(39, data["recoveredConfigVersion"]);
Assert.Equal(nameof(EventSeverity.Info), rig.Sink.Posted[0].Payload!["severity"]);
}
[Fact]
public void EmitFetchFailedIfAny_WithoutRecovery_StaysWarning()
{
using var rig = new EmitterRig();
rig.RemoteConfig.SetFakeOutcome(
RemoteConfigFetchOutcome.UsedDefaults,
configVersion: 0,
attempts: 4,
failureType: "HttpRequestException",
failureMessage: "still down");
LifecycleEmitters.EmitRemoteConfigFetchFailedIfAny(
rig.Post, rig.AgentConfig, rig.RemoteConfig, rig.Logger);
var data = AssertSingleEmit(rig.Sink, expectedEventType: "remote_config_fetch_failed");
Assert.False(data.ContainsKey("recovered"));
Assert.False(data.ContainsKey("recoveredConfigVersion"));
Assert.Equal(nameof(EventSeverity.Warning), rig.Sink.Posted[0].Payload!["severity"]);
}
[Fact]
public void EmitAgentStarted_AfterRecovery_FlagsLiveConfig()
{
using var rig = new EmitterRig();
rig.RemoteConfig.SetFakeOutcome(
RemoteConfigFetchOutcome.UsedDefaults,
configVersion: 39,
recovered: true);
LifecycleEmitters.EmitAgentStarted(
rig.Post, rig.AgentConfig, rig.PreviousExit, "2.0.806", rig.RemoteConfig, rig.Logger);
var data = AssertSingleEmit(rig.Sink, expectedEventType: "agent_started");
Assert.Equal(39, data["configVersion"]);
Assert.Equal(true, data["remoteConfigFetched"]); // running on a live-fetched config
Assert.Equal("UsedDefaults", data["remoteConfigOutcome"]); // startup outcome stays honest
Assert.Equal(true, data["remoteConfigRecovered"]);
}
// ── Helpers ─────────────────────────────────────────────────────────
private static Dictionary<string, object> AssertSingleEmit(
@@ -258,8 +325,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
int attempts = 1,
string? failureType = null,
string? failureMessage = null,
int? authStatusCode = null)
int? authStatusCode = null,
bool recovered = false)
{
typeof(RemoteConfigService).GetProperty(nameof(RecoveredAfterFetchFailure))?.SetValue(this, recovered);
// CurrentConfig is set via the protected SetConfig path — easiest:
// a real FetchConfigAsync replacement that just stashes a canned config.
typeof(RemoteConfigService).GetMethod("SetConfig",
@@ -97,7 +97,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
}
[Fact]
public async Task Fails_after_five_exceptions_without_auth_error()
public async Task Fails_after_six_exceptions_without_auth_error()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
@@ -111,8 +111,50 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
backoffDelay: _ => Task.CompletedTask);
Assert.Equal(SessionRegistrationOutcome.Failed, result.Outcome);
Assert.Equal(5, apiClient.CallCount);
Assert.Contains("transient attempt 5", result.ErrorMessage);
Assert.Equal(6, apiClient.CallCount);
Assert.Contains("transient attempt 6", result.ErrorMessage);
}
[Fact]
public async Task Waits_for_network_link_before_first_attempt()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
var config = NewConfig();
var order = new System.Collections.Generic.List<string>();
var apiClient = new FakeApiClient(_ =>
{
order.Add("register");
return new RegisterSessionResponse { Success = true, SessionId = config.SessionId };
});
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
apiClient, config, agentVersion: "0.0.0", logger: logger,
backoffDelay: _ => Task.CompletedTask,
networkLinkWait: _ => { order.Add("link-wait"); return Task.CompletedTask; });
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
Assert.Equal(new[] { "link-wait", "register" }, order);
}
[Fact]
public async Task Network_link_wait_failure_does_not_block_registration()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
var config = NewConfig();
var apiClient = new FakeApiClient(_ =>
new RegisterSessionResponse { Success = true, SessionId = config.SessionId });
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
apiClient, config, agentVersion: "0.0.0", logger: logger,
backoffDelay: _ => Task.CompletedTask,
networkLinkWait: _ => throw new InvalidOperationException("probe broken"));
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
Assert.Equal(1, apiClient.CallCount);
}
[Fact]
@@ -262,6 +262,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
/// </summary>
public bool UnrestrictedMode { get; set; } = false;
/// <summary>
/// Mirror of <c>AnalyzerConfiguration.EnableRealmJoinWatcher</c> (remote, per tenant).
/// Gates the RealmJoin built-in sections of the diagnostics package — the package
/// service only sees this configuration, not the remote response. The RealmJoin host
/// itself keeps reading the remote Analyzers block. Default: false.
/// </summary>
public bool EnableRealmJoinWatcher { get; set; } = false;
/// <summary>
/// Sanitized command-line arguments for the agent_started event.
/// Secrets (e.g. bootstrap token) are redacted before assignment.
@@ -101,6 +101,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
agentConfig.DiagnosticsUploadEnabled = remoteConfig.DiagnosticsUploadEnabled;
agentConfig.DiagnosticsUploadMode = remoteConfig.DiagnosticsUploadMode;
agentConfig.DiagnosticsLogPaths = remoteConfig.DiagnosticsLogPaths ?? new List<DiagnosticsLogPath>();
// The diagnostics package gates its RealmJoin sections on the tenant's watcher
// toggle but never sees the remote response — mirror the one Analyzers knob here.
// The remaining Analyzers flags flow untouched to DefaultComponentFactory.
agentConfig.EnableRealmJoinWatcher = remoteConfig.Analyzers?.EnableRealmJoinWatcher ?? false;
agentConfig.SendTraceEvents = remoteConfig.SendTraceEvents;
@@ -42,6 +42,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
TimeSpan.FromSeconds(60),
};
/// <summary>
/// Upper bound for the pre-fetch network-link wait on the initial startup fetch.
/// Deliberately longer than the registration gate's 15s: the fetch runs FIRST after a
/// boot-time relaunch, when Wi-Fi may still be associating (802.1X, OOBE), and a fetch
/// burned into a dead link strands the whole session on built-in defaults — edu.kosh
/// audit 2026-08-24: ~50% of sessions lost their tenant config exactly this way.
/// </summary>
internal static readonly TimeSpan InitialFetchNetworkLinkWaitMax = TimeSpan.FromSeconds(60);
/// <summary>
/// Gets the current configuration (thread-safe)
/// </summary>
@@ -76,6 +85,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
/// <summary>HTTP status for auth failures (401/403), null otherwise.</summary>
public int? LastFetchAuthStatusCode { get; private set; }
/// <summary>
/// True once <see cref="TryRecoverConfigAsync"/> replaced a failed startup fetch with a
/// live config. The <c>LastFetch*</c> properties intentionally keep describing the failed
/// startup fetch so the wire event can report both the original failure and the recovery.
/// </summary>
public bool RecoveredAfterFetchFailure { get; private set; }
public RemoteConfigService(BackendApiClient apiClient, string tenantId, AgentLogger logger, EmergencyReporter emergencyReporter = null, DistressReporter distressReporter = null, AuthFailureTracker authFailureTracker = null)
{
_apiClient = apiClient ?? throw new ArgumentNullException(nameof(apiClient));
@@ -116,6 +132,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
LastFetchFailureMessage = null;
LastFetchAuthStatusCode = null;
// Initial startup fetch only: give a still-associating NIC a chance to come up
// before the first attempt. Without this the retry budget (10/30/60s) is burned
// into a provably dead link — the agent starts at boot, Wi-Fi associates minutes
// later, and the session runs on defaults although the backend was never the
// problem. The rotate_config path (retryOnTransientErrors:false) runs mid-session
// on a proven link and must stay latency-bounded, so it skips the gate.
if (retryOnTransientErrors)
{
try { await WaitForNetworkLinkAsync().ConfigureAwait(false); }
catch (Exception ex) { _logger.Debug($"Pre-fetch network wait failed ({ex.Message}) — fetching anyway."); }
}
var maxAttempts = retryOnTransientErrors ? InitialFetchRetryBackoff.Length + 1 : 1;
Exception lastException = null;
@@ -211,6 +239,52 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
return defaults;
}
/// <summary>
/// Single-attempt recovery fetch for a session whose startup fetch failed. Called right
/// after session registration succeeded — registration proves the backend is reachable,
/// so the config that Phase 4 could not get is one cheap call away instead of the whole
/// session being stranded on cache/defaults (only a manual <c>rotate_config</c> action
/// could rescue it before this existed).
/// <para>
/// Unlike <see cref="FetchConfigAsync"/> this does NOT reset the <c>LastFetch*</c>
/// observability state — those keep describing the failed startup fetch so
/// <c>remote_config_fetch_failed</c> can report the original failure AND the recovery.
/// Returns the recovered config, or null when there is nothing to recover (startup fetch
/// succeeded / never ran) or the recovery attempt itself failed. Never throws.
/// </para>
/// </summary>
public async Task<AgentConfigResponse> TryRecoverConfigAsync()
{
if (LastFetchOutcome != RemoteConfigFetchOutcome.FromCache &&
LastFetchOutcome != RemoteConfigFetchOutcome.UsedDefaults)
return null;
try
{
_logger.Info("Retrying remote config fetch after successful session registration...");
var config = await CallBackendAsync().ConfigureAwait(false);
if (config == null)
{
_logger.Warning("Config recovery fetch returned empty response — keeping fallback config.");
return null;
}
_logger.Info($"Remote config recovered: ConfigVersion={config.ConfigVersion}, " +
$"GatherRules={config.GatherRules?.Count ?? 0}");
LogCollectorSettings(config.Collectors);
SetConfig(config);
CacheConfig(config);
_authFailureTracker?.RecordSuccess();
RecoveredAfterFetchFailure = true;
return config;
}
catch (Exception ex)
{
_logger.Warning($"Config recovery fetch failed ({ex.Message}) — keeping fallback config.");
return null;
}
}
private void SetConfig(AgentConfigResponse config)
{
lock (_configLock)
@@ -358,6 +432,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
protected virtual Task DelayBetweenAttemptsAsync(TimeSpan delay)
=> Task.Delay(delay);
/// <summary>
/// Awaits the pre-fetch network-link gate (initial startup fetch only). Production
/// polls the link-level signal via <see cref="NetworkLinkWait"/> bounded by
/// <see cref="InitialFetchNetworkLinkWaitMax"/>; tests subclass to record the call
/// and return immediately.
/// </summary>
protected virtual Task WaitForNetworkLinkAsync()
=> NetworkLinkWait.WaitAsync(_logger, InitialFetchNetworkLinkWaitMax, "Remote config fetch");
public void Dispose()
{
// Nothing to dispose
@@ -10,7 +10,9 @@ using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.DeviceInfo;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Gather;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
using AutopilotMonitor.Agent.V2.Core.Security;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
@@ -124,62 +126,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
}
}
private readonly string _imeLogFolder;
private readonly string _agentLogFolder;
private readonly string _agentStateFolder;
private readonly string _agentSpoolFolder;
private readonly string _agentDataFolder;
// Built-in sections come from the Shared catalog (DiagnosticsBuiltInSections.All) — the
// same list the backend serves to the portal, so what administrators see is exactly
// what this service collects. Test seam: section id → folder; production leaves the
// map empty and every section resolves from the catalog's (unexpanded) source folder.
private readonly IReadOnlyDictionary<string, string> _sectionFolderOverrides;
private static readonly string DefaultImeLogFolder =
Environment.ExpandEnvironmentVariables(@"%ProgramData%\Microsoft\IntuneManagementExtension\Logs");
private static readonly string DefaultAgentLogFolder =
Environment.ExpandEnvironmentVariables(Constants.LogDirectory);
private static readonly string DefaultAgentStateFolder =
Environment.ExpandEnvironmentVariables(Constants.StateDirectory);
private static readonly string DefaultAgentSpoolFolder =
Environment.ExpandEnvironmentVariables(Constants.SpoolDirectory);
private static readonly string DefaultAgentDataFolder =
Environment.ExpandEnvironmentVariables(@"%ProgramData%\AutopilotMonitor");
/// <summary>
/// File extensions collected from built-in log folders (Agent + IME).
/// Covers standard logs, structured data, traces, event logs, and diagnostics archives.
/// </summary>
private static readonly string[] LogFilePatterns = new[]
{
"*.log", "*.txt", "*.json", "*.jsonl",
"*.etl", "*.evtx", "*.xml", "*.csv", "*.cab"
};
/// <summary>
/// Patterns collected from the agent state folder. Adds completion markers on top of
/// the standard log patterns so files like <c>enrollment-complete.marker</c> and
/// <c>whiteglove-backfill-state.json</c> reach the diagnostics archive.
/// </summary>
private static readonly string[] StateFilePatterns = new[]
{
"*.log", "*.txt", "*.json", "*.jsonl",
"*.etl", "*.evtx", "*.xml", "*.csv", "*.cab",
"*.complete", "*.marker"
};
/// <summary>
/// Marker patterns from the top-level data directory. Kept tight on purpose: only
/// completion/exit markers, no session.id / bootstrap.json (those are config, not
/// forensic state).
/// </summary>
private static readonly string[] RootMarkerPatterns = new[] { "*.complete", "*.marker" };
/// <summary>
/// Telemetry spool: only the JSON-shaped files. The spool folder may also contain
/// quarantine subfolders, so subfolders are excluded to keep the archive focused
/// on what is actually pending upload.
/// </summary>
private static readonly string[] SpoolFilePatterns = new[] { "*.jsonl", "*.json" };
// Scenario oracle for DiagnosticsSectionCondition.DevicePreparation. Production reads
// the deterministic WDP registry marker (the same gate every other WDP switch uses);
// tests inject a constant so they never touch the registry.
private readonly Func<bool> _devicePreparationProbe;
public DiagnosticsPackageService(AgentConfiguration configuration, AgentLogger logger, BackendApiClient apiClient)
: this(configuration, logger, apiClient, null, null, null, null, null)
@@ -188,7 +144,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
// Test seam: allows xUnit fixtures to redirect log/state/spool/data folders to a
// temp dir without touching real %ProgramData% paths. Production callers always go
// through the public ctor.
// through the public ctor. The five named overrides map onto the catalog's section
// ids; sectionFolderOverrides covers every other section (RealmJoin*, ImeBootstrapper…).
internal DiagnosticsPackageService(
AgentConfiguration configuration,
AgentLogger logger,
@@ -197,18 +154,38 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
string imeLogFolderOverride,
string agentStateFolderOverride,
string agentSpoolFolderOverride,
string agentDataFolderOverride)
string agentDataFolderOverride,
IReadOnlyDictionary<string, string> sectionFolderOverrides = null,
Func<bool> devicePreparationProbe = null)
{
_configuration = configuration;
_logger = logger;
_apiClient = apiClient;
_httpClient = new HttpClient { Timeout = TimeSpan.FromMinutes(5) };
_agentLogFolder = agentLogFolderOverride ?? DefaultAgentLogFolder;
_imeLogFolder = imeLogFolderOverride ?? DefaultImeLogFolder;
_agentStateFolder = agentStateFolderOverride ?? DefaultAgentStateFolder;
_agentSpoolFolder = agentSpoolFolderOverride ?? DefaultAgentSpoolFolder;
_agentDataFolder = agentDataFolderOverride ?? DefaultAgentDataFolder;
var overrides = new Dictionary<string, string>(StringComparer.Ordinal);
void Map(string sectionId, string folder)
{
if (folder != null) overrides[sectionId] = folder;
}
Map("AgentLogs", agentLogFolderOverride);
Map("ImeLogs", imeLogFolderOverride);
Map("AgentState", agentStateFolderOverride);
Map("AgentSpool", agentSpoolFolderOverride);
Map("AgentMarkers", agentDataFolderOverride);
if (sectionFolderOverrides != null)
{
foreach (var kv in sectionFolderOverrides)
overrides[kv.Key] = kv.Value;
}
_sectionFolderOverrides = overrides;
// Tests that pass folder overrides but no probe must stay registry-free: a seam
// caller gets "not WDP" by default, the production ctor gets the real marker.
_devicePreparationProbe = devicePreparationProbe
?? (overrides.Count == 0
? (Func<bool>)EnrollmentRegistryDetector.IsDeterministicDevicePreparation
: () => false);
}
/// <summary>
@@ -370,33 +347,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
// 1. sessioninfo.txt — informational metadata, not subject to caps.
AddSessionInfo(archive, enrollmentSucceeded);
// 2. Agent logs
foreach (var pattern in LogFilePatterns)
AddLogFiles(archive, _agentLogFolder, "AgentLogs", pattern, tracker);
// 2. Built-in sections — ONE catalog shared with the backend (the portal
// shows exactly this list): agent logs, IME logs (+ the Device Preparation
// bootstrapper event log on WDP), agent state (recursive: `.quarantine`
// and `.part1-<utc>/` buckets ride along), telemetry spool, top-level
// markers, and the RealmJoin logs behind the tenant's watcher toggle.
// The scenario probe is a registry read — evaluate it once per build.
var isDevicePreparation = ProbeDevicePreparation();
ManifestLine($"SCENARIO: devicePreparation={isDevicePreparation} realmJoinWatcher={_configuration.EnableRealmJoinWatcher}");
foreach (var section in DiagnosticsBuiltInSections.All)
AddBuiltInSection(archive, section, tracker, isDevicePreparation);
// 3. IME logs (all relevant files in the IME log folder)
foreach (var pattern in LogFilePatterns)
AddLogFiles(archive, _imeLogFolder, "ImeLogs", pattern, tracker);
// 4. Agent state: snapshot, journal, signal-log, ime-tracker-state,
// enrollment-complete.marker, final-status.json, whiteglove-backfill-state.json.
// Recursive to include `.quarantine` subfolders AND `.part1-<utc>/` buckets
// that StateArchiver creates on WG Part-2 boot — both ride along under
// AgentState/ via the same recursive walk.
foreach (var pattern in StateFilePatterns)
AddLogFiles(archive, _agentStateFolder, "AgentState", pattern, tracker, includeSubfolders: true);
// 5. Telemetry spool — pending uploads + upload cursor (forensic: what didn't ship?)
foreach (var pattern in SpoolFilePatterns)
AddLogFiles(archive, _agentSpoolFolder, "AgentSpool", pattern, tracker, includeSubfolders: false);
// 6. Top-level markers (whiteglove.complete, clean-exit marker) — Part-1/Part-2
// bridge + ghost-restart guard. Top-directory only to avoid re-zipping
// State/Spool/Logs/Agent which already have their own sections.
foreach (var pattern in RootMarkerPatterns)
AddLogFiles(archive, _agentDataFolder, "AgentMarkers", pattern, tracker, includeSubfolders: false);
// 7. Configured additional log paths (global + tenant, validated by guards)
// 3. Configured additional log paths (global + tenant, validated by guards)
foreach (var entry in _configuration.DiagnosticsLogPaths ?? new System.Collections.Generic.List<Shared.Models.DiagnosticsLogPath>())
{
// Resolve %LOGGED_ON_USER_PROFILE% token and get profile path for guard exception
@@ -447,6 +409,81 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
}
}
private bool ProbeDevicePreparation()
{
try
{
return _devicePreparationProbe();
}
catch (Exception ex)
{
// Fail toward Classic like every other WDP gate: the evtx export is skipped,
// never the whole package.
_logger.Warning($"Device Preparation probe failed — treating as Classic: {ex.Message}");
ManifestLine($"SCENARIO PROBE FAILED (treated as Classic): {ex.Message}");
return false;
}
}
/// <summary>
/// Pure gate: is this built-in section collected under the given configuration and
/// enrollment scenario? Unknown conditions (a newer Shared catalog than this agent)
/// fail closed.
/// </summary>
internal static bool IsSectionActive(DiagnosticsBuiltInSection section, AgentConfiguration configuration, bool isDevicePreparation)
{
switch (section.Condition)
{
case DiagnosticsSectionCondition.Always:
return true;
case DiagnosticsSectionCondition.RealmJoinWatcher:
return configuration.EnableRealmJoinWatcher;
case DiagnosticsSectionCondition.DevicePreparation:
return isDevicePreparation;
default:
return false;
}
}
private static string DescribeInactiveCondition(DiagnosticsSectionCondition condition)
{
switch (condition)
{
case DiagnosticsSectionCondition.RealmJoinWatcher:
return "RealmJoin Watcher disabled";
case DiagnosticsSectionCondition.DevicePreparation:
return "not a Device Preparation enrollment";
default:
return $"unknown condition {condition}";
}
}
private void AddBuiltInSection(ZipArchive archive, DiagnosticsBuiltInSection section, BudgetTracker tracker, bool isDevicePreparation)
{
if (!IsSectionActive(section, _configuration, isDevicePreparation))
{
ManifestLine($"BUILT-IN SKIPPED ({DescribeInactiveCondition(section.Condition)}): {section.Id}");
return;
}
// Override (test seam) or catalog source folder with %ProgramData% /
// %LOGGED_ON_USER_PROFILE% expanded; null means the token is present but no
// interactive user has been detected yet — same skip as configured paths.
var folder = _sectionFolderOverrides.TryGetValue(section.Id, out var overrideFolder)
? overrideFolder
: UserProfileResolver.ExpandCustomTokens(section.SourceFolder);
if (folder == null)
{
_logger.Warning($"Built-in diagnostics section skipped (no user session for token): {section.Id}");
ManifestLine($"BUILT-IN SKIPPED (no user session for token): {section.Id} '{section.SourceFolder}'");
return;
}
ManifestLine($"BUILT-IN: {section.Id} -> folder='{folder}' zip='{section.ZipFolder}' recursive={section.IncludeSubfolders}");
foreach (var pattern in section.Patterns)
AddLogFiles(archive, folder, section.ZipFolder, pattern, tracker, section.IncludeSubfolders);
}
private void AddSessionInfo(ZipArchive archive, bool? enrollmentSucceeded)
{
var sb = new StringBuilder();
@@ -20,7 +20,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
/// </summary>
public static class UserProfileResolver
{
public const string Token = "%LOGGED_ON_USER_PROFILE%";
// Spelled once, in Shared: the built-in diagnostics catalog uses the same token.
public const string Token = Shared.Models.DiagnosticsBuiltInSections.UserProfileToken;
private static readonly object Lock = new object();
private static bool _resolved;
@@ -0,0 +1,38 @@
using System;
using System.Net.NetworkInformation;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Logging;
namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
{
/// <summary>
/// Boot-time NIC grace shared by every first-contact backend call (remote-config fetch,
/// session registration, emergency-break reporter pattern): polls the cheap link-level
/// signal once per second for at most <paramref name="maxWait"/>. It cannot prove backend
/// reachability — the callers' retry loops handle that — it only keeps retry budgets from
/// being burned into a link that is provably still down (BootTrigger relaunch after a
/// mid-enrollment reboot, Wi-Fi still associating; tenant aebdce78 audits 2026-08-23/24).
/// Free on the normal path — a live link returns immediately. Probe errors end the wait,
/// never the caller's operation.
/// </summary>
internal static class NetworkLinkWait
{
internal static async Task WaitAsync(AgentLogger logger, TimeSpan maxWait, string context)
{
if (NetworkInterface.GetIsNetworkAvailable()) return;
logger?.Info($"{context}: no network link yet — waiting up to {maxWait.TotalSeconds:F0}s.");
var deadline = DateTime.UtcNow + maxWait;
while (DateTime.UtcNow < deadline)
{
await Task.Delay(1000).ConfigureAwait(false);
if (NetworkInterface.GetIsNetworkAvailable())
{
logger?.Info($"{context}: network link is up.");
return;
}
}
logger?.Warning($"{context}: still no network link after {maxWait.TotalSeconds:F0}s — attempting anyway.");
}
}
}
@@ -19,7 +19,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <b>Retry contract (V1 <c>MonitoringService.RegisterSessionAsync</c>):</b>
/// </para>
/// <list type="bullet">
/// <item>5 attempts total; between attempts <c>2^attempt</c> seconds (2s, 4s, 8s, 16s).</item>
/// <item>Boot-time NIC grace before the first attempt: when no network link is up yet
/// (BootTrigger relaunch after a mid-enrollment reboot, Wi-Fi still associating) wait
/// up to <see cref="NetworkLinkWaitMax"/> for the link-level signal. Best-effort and
/// free on the normal path — a live link returns immediately.</item>
/// <item>6 attempts total; between attempts <c>2^attempt</c> seconds (2s, 4s, 8s, 16s, 32s —
/// ~62 s of backoff). The original V1 budget of 5 attempts / ~30 s was too short for the
/// reboot-relaunch on Wi-Fi kiosks: the relaunched agent gave up and exited 7 before
/// the network came back, leaving the session silent forever (tenant aebdce78,
/// 2026-08-23 audit). A relaunch that misses here is lost — nothing retries later.</item>
/// <item>On <c>response.Success == true</c> → stop immediately.</item>
/// <item>On <see cref="BackendAuthException"/> (401/403) → report to the
/// <see cref="AuthFailureTracker"/> (which fires the first-failure distress + may trip
@@ -37,16 +45,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// </summary>
public static class SessionRegistrationHelper
{
private const int MaxAttempts = 5;
private const int MaxAttempts = 6;
/// <summary>Upper bound for the pre-registration link wait (see class remarks).</summary>
internal static readonly TimeSpan NetworkLinkWaitMax = TimeSpan.FromSeconds(15);
/// <summary>
/// Executes the 5-retry register-session handshake against the backend. Returns the
/// Executes the retrying register-session handshake against the backend. Returns the
/// response + classified outcome. Never throws.
/// </summary>
/// <param name="backoffDelay">
/// Optional delay-provider used by tests to avoid real-time waits. Production callers
/// pass <c>null</c> → V1 parity <c>Task.Delay(2^attempt * 1000)</c> between attempts.
/// </param>
/// <param name="networkLinkWait">
/// Optional pre-registration link wait used by tests. Production callers pass
/// <c>null</c> → <see cref="WaitForNetworkLinkAsync"/> (bounded by <see cref="NetworkLinkWaitMax"/>).
/// </param>
public static async Task<SessionRegistrationResult> RegisterWithRetryAsync(
BackendApiClient apiClient,
AgentConfiguration agentConfig,
@@ -56,14 +71,19 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
EmergencyReporter emergencyReporter = null,
Func<int, Task> backoffDelay = null,
Func<Exception, Task> onTerminalTransportFailure = null,
(string Manufacturer, string Model, string SerialNumber)? deviceHardware = null)
(string Manufacturer, string Model, string SerialNumber)? deviceHardware = null,
Func<AgentLogger, Task> networkLinkWait = null)
{
if (apiClient == null) throw new ArgumentNullException(nameof(apiClient));
if (agentConfig == null) throw new ArgumentNullException(nameof(agentConfig));
if (logger == null) throw new ArgumentNullException(nameof(logger));
backoffDelay ??= DefaultBackoffDelay;
networkLinkWait ??= WaitForNetworkLinkAsync;
var registration = BuildRegistration(agentConfig, agentVersion, deviceHardware);
try { await networkLinkWait(logger).ConfigureAwait(false); }
catch (Exception ex) { logger.Debug($"Pre-registration network wait failed ({ex.Message}) — registering anyway."); }
string lastError = null;
Exception lastException = null;
@@ -113,7 +133,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
if (attempt < MaxAttempts)
{
var delaySeconds = (int)Math.Pow(2, attempt); // 2, 4, 8, 16
var delaySeconds = (int)Math.Pow(2, attempt); // 2, 4, 8, 16, 32
logger.Info($"Retrying session registration in {delaySeconds}s");
await backoffDelay(attempt).ConfigureAwait(false);
}
@@ -143,6 +163,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
private static Task DefaultBackoffDelay(int attempt)
=> Task.Delay(((int)Math.Pow(2, attempt)) * 1000);
/// <summary>
/// Boot-time NIC grace before the first registration attempt — shared polling loop in
/// <see cref="NetworkLinkWait"/>, bounded by <see cref="NetworkLinkWaitMax"/>. Probe
/// errors end the wait, never the registration.
/// </summary>
internal static Task WaitForNetworkLinkAsync(AgentLogger logger)
=> NetworkLinkWait.WaitAsync(logger, NetworkLinkWaitMax, "Session registration");
private static SessionRegistration BuildRegistration(
AgentConfiguration agentConfig,
string agentVersion,
@@ -194,7 +222,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <summary>401/403 from the backend — cert/token rejected. No retry was attempted.</summary>
AuthFailed = 1,
/// <summary>All 5 attempts failed with non-auth errors (network, 5xx, malformed response).</summary>
/// <summary>All attempts failed with non-auth errors (network, 5xx, malformed response).</summary>
Failed = 2,
}
@@ -274,6 +274,15 @@ namespace AutopilotMonitor.Agent.V2
{
return registration.ExitCode;
}
// Phase 6.5 (config self-heal): when the Phase 4 fetch ran before the network link
// was up, registration has just proven the backend reachable — retry the fetch once
// so the session runs on the live tenant config instead of cache/defaults.
// See AgentRuntimeConfig.TryRecoverAfterRegistration for what is deliberately
// NOT re-run here (kill-signal check, endpoint migration).
Runtime.AgentRuntimeConfig.TryRecoverAfterRegistration(
bootstrap.AgentConfig, auth, runtimeConfig, GetAgentVersion(), consoleMode, logger);
// Phase 7+8 (orchestrator + shutdown lifecycle + onIngressReady wiring +
// initial signal posts + analyzer/probes startup + shutdown.Wait + finally)
// is encapsulated in AgentRuntimeHost. See Runtime/AgentRuntimeHost.cs.
@@ -69,6 +69,63 @@ namespace AutopilotMonitor.Agent.V2.Runtime
}
}
var configMergeResult = ApplyFetchedConfig(agentConfig, remoteConfig, auth, agentVersion, consoleMode, logger);
return new RuntimeConfigBundle(remoteConfigService, remoteConfig, configMergeResult, pendingMigrationTarget: null);
}
/// <summary>
/// Phase 6.5 — config self-heal. When the Phase 4 fetch ran before the network link was
/// up (agent relaunched at boot, Wi-Fi still associating — edu.kosh audit 2026-08-24:
/// ~50% of sessions ran on ConfigVersion=0 exactly this way), a successful session
/// registration has just proven the backend reachable. Retry the fetch once and apply
/// the live tenant config instead of stranding the whole session on cache/defaults.
/// <para>
/// Deliberately NOT re-run here: the kill-signal check (post-registration kills arrive
/// via the telemetry-channel control signals on every upload response) and endpoint
/// migration (registration already succeeded against the current base URL; a migration
/// signal is honoured at the next agent start). Never throws — recovery is strictly
/// best-effort on top of an already-degraded-but-working fallback.
/// </para>
/// </summary>
public static void TryRecoverAfterRegistration(
AgentConfiguration agentConfig,
BackendAuthBundle auth,
RuntimeConfigBundle bundle,
string agentVersion,
bool consoleMode,
AgentLogger logger)
{
try
{
var service = bundle?.RemoteConfigService;
if (service == null) return;
var recovered = service.TryRecoverConfigAsync().GetAwaiter().GetResult();
if (recovered == null) return;
var mergeResult = ApplyFetchedConfig(agentConfig, recovered, auth, agentVersion, consoleMode, logger);
bundle.ApplyRecovered(recovered, mergeResult);
}
catch (Exception ex)
{
logger.Warning($"Post-registration config recovery failed: {ex.Message}");
}
}
/// <summary>
/// Applies a live-fetched config to the runtime: merge, tracker ceilings, log level,
/// integrity SHA + check, bootstrap-config cleanup. Shared by the Phase 4 startup path
/// and the post-registration recovery path so both apply identical semantics.
/// </summary>
private static RemoteConfigMergeResult ApplyFetchedConfig(
AgentConfiguration agentConfig,
AgentConfigResponse remoteConfig,
BackendAuthBundle auth,
string agentVersion,
bool consoleMode,
AgentLogger logger)
{
// Project remote tenant-controlled knobs onto the runtime AgentConfiguration so that
// downstream consumers (CleanupService, SummaryDialogLauncher, StartupEnvironmentProbes,
// DiagnosticsPackageService, EnrollmentTerminationHandler, the logger, the watchdog)
@@ -91,7 +148,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
TryCleanupBootstrapConfig(agentConfig, agentVersion, logger);
return new RuntimeConfigBundle(remoteConfigService, remoteConfig, configMergeResult, pendingMigrationTarget: null);
return configMergeResult;
}
/// <summary>
@@ -175,10 +232,21 @@ namespace AutopilotMonitor.Agent.V2.Runtime
internal sealed class RuntimeConfigBundle
{
public RemoteConfigService RemoteConfigService { get; }
public AgentConfigResponse RemoteConfig { get; }
public RemoteConfigMergeResult MergeResult { get; }
public AgentConfigResponse RemoteConfig { get; private set; }
public RemoteConfigMergeResult MergeResult { get; private set; }
public string PendingMigrationTarget { get; }
/// <summary>
/// Swaps in the config recovered by <see cref="AgentRuntimeConfig.TryRecoverAfterRegistration"/>
/// so Phase 7+8 (orchestrator, collectors, gather rules, IME patterns, the
/// unrestricted-mode audit) run on the live tenant config instead of the fallback.
/// </summary>
public void ApplyRecovered(AgentConfigResponse recoveredConfig, RemoteConfigMergeResult mergeResult)
{
RemoteConfig = recoveredConfig;
MergeResult = mergeResult;
}
public RuntimeConfigBundle(
RemoteConfigService remoteConfigService,
AgentConfigResponse remoteConfig,
@@ -11,7 +11,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{
/// <summary>
/// Phase 6 of <see cref="Program"/>'s <c>RunAgent</c>: blocking
/// <c>POST /api/agent/register-session</c> with 5-retry exponential backoff before the
/// <c>POST /api/agent/register-session</c> with link-wait + 6-retry exponential backoff before the
/// orchestrator starts. V1 parity (MonitoringService.RegisterSessionAsync) — without
/// this call the backend's Sessions table never gets a row for this session, so
/// <c>IncrementSessionEventCountAsync</c> / <c>UpdateSessionStatusAsync</c> silently
@@ -52,6 +52,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
// prior to this, ConfigVersion=0 was only deducible by inferring from
// the absence of tenant-controlled knobs in the DataJson.
var outcome = remoteConfigService?.LastFetchOutcome ?? RemoteConfigFetchOutcome.NotAttempted;
var recovered = remoteConfigService?.RecoveredAfterFetchFailure == true;
var configVersion = remoteConfigService?.CurrentConfig?.ConfigVersion ?? 0;
var data = new Dictionary<string, object>
@@ -66,8 +67,12 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ "diagnosticsUploadMode", agentConfig.DiagnosticsUploadMode ?? "Off" },
{ "previousExitType", previousExit?.ExitType ?? "unknown" },
{ "unrestrictedMode", agentConfig.UnrestrictedMode },
// remoteConfigFetched answers "is the agent running on a live-fetched
// config" — true also when the post-registration recovery replaced a
// failed startup fetch (remoteConfigOutcome then still names the startup
// failure; remoteConfigRecovered marks the heal).
{ "configVersion", configVersion },
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded },
{ "remoteConfigFetched", outcome == RemoteConfigFetchOutcome.Succeeded || recovered },
{ "remoteConfigOutcome", outcome.ToString() },
// WinRT OOBE state sampled at this start. Together with previousExitType
// this makes late starts provable: "completed" on a first_run means OOBE
@@ -75,6 +80,9 @@ namespace AutopilotMonitor.Agent.V2.Runtime
{ "oobeStateAtAgentStart", OobeStateReader.Read() },
};
if (recovered)
data["remoteConfigRecovered"] = true;
if (!string.IsNullOrEmpty(previousExit?.CrashExceptionType))
data["previousCrashException"] = previousExit.CrashExceptionType;
@@ -135,12 +143,25 @@ namespace AutopilotMonitor.Agent.V2.Runtime
? $"Remote config fetch failed after {remoteConfigService.LastFetchAttempts} attempt(s); using cached config."
: $"Remote config fetch failed after {remoteConfigService.LastFetchAttempts} attempt(s); running with built-in defaults (ConfigVersion=0).";
// Post-registration recovery (Phase 6.5) replaced the fallback with a live
// fetch before this emitter ran: keep the failure details (the network-late
// condition is worth surfacing) but mark the heal and demote to Info — the
// session is NOT degraded.
var recovered = remoteConfigService.RecoveredAfterFetchFailure;
if (recovered)
{
var recoveredVersion = remoteConfigService.CurrentConfig?.ConfigVersion ?? 0;
data["recovered"] = true;
data["recoveredConfigVersion"] = recoveredVersion;
msg += $" Recovered after session registration (ConfigVersion={recoveredVersion}).";
}
post.Emit(new EnrollmentEvent
{
SessionId = agentConfig.SessionId,
TenantId = agentConfig.TenantId,
EventType = SharedConstants.EventTypes.RemoteConfigFetchFailed,
Severity = EventSeverity.Warning,
Severity = recovered ? EventSeverity.Info : EventSeverity.Warning,
Source = "Agent",
Phase = EnrollmentPhase.Unknown,
Message = msg,
@@ -0,0 +1,99 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Functions.Services;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// <see cref="TableStorageService.BuildCveIndexEntities"/> collapses report findings to one
/// CveIndex entity per CVE. Production reports routinely list the same CVE under many
/// findings (one RD CVE across 14 installed RD components), which previously produced
/// 14 concurrent upserts racing on the same PK/RK.
/// </summary>
public class CveIndexEntityMergeTests
{
private const string Tenant = "11111111-1111-1111-1111-111111111111";
private const string Session = "22222222-2222-2222-2222-222222222222";
private static Dictionary<string, object> Finding(string software, string risk, params Dictionary<string, object>[] vulns)
=> new()
{
["softwareName"] = software,
["riskLevel"] = risk,
["vulnerabilities"] = vulns.ToList(),
};
private static Dictionary<string, object> Vuln(string cve, double score, string severity, bool kev = false)
=> new()
{
["cveId"] = cve,
["cvssScore"] = score,
["cvssSeverity"] = severity,
["isKev"] = kev,
};
[Fact]
public void SameCveAcrossManyFindings_CollapsesToOneEntity()
{
var findings = Enumerable.Range(0, 14)
.Select(i => Finding($"RD Component {i}", "high", Vuln("CVE-2025-29966", 8.8, "HIGH")))
.ToList();
var entities = TableStorageService.BuildCveIndexEntities(Tenant, Session, findings);
var e = Assert.Single(entities);
Assert.Equal($"{Tenant}_CVE-2025-29966", e.PartitionKey);
Assert.Equal(Session, e.RowKey);
Assert.Equal("RD Component 0", e.GetString("SoftwareName")); // tie keeps report order
}
[Fact]
public void Merge_HighestScoreWins_KevOrs_RiskMaxes()
{
var findings = new List<Dictionary<string, object>>
{
Finding("App A", "medium", Vuln("CVE-1", 5.0, "MEDIUM", kev: true)),
Finding("App B", "critical", Vuln("CVE-1", 9.1, "CRITICAL")),
Finding("App C", "low", Vuln("CVE-1", 3.0, "LOW")),
};
var e = Assert.Single(TableStorageService.BuildCveIndexEntities(Tenant, Session, findings));
Assert.Equal(9.1, e.GetDouble("CvssScore"));
Assert.Equal("CRITICAL", e.GetString("CvssSeverity"));
Assert.Equal("App B", e.GetString("SoftwareName"));
Assert.True(e.GetBoolean("IsKev"));
Assert.Equal("critical", e.GetString("OverallRisk"));
}
[Fact]
public void DistinctCves_StayDistinct_AndInvalidEntriesAreSkipped()
{
var findings = new List<Dictionary<string, object>>
{
Finding("App A", "high", Vuln("CVE-1", 7.0, "HIGH"), Vuln("CVE-2", 6.0, "MEDIUM")),
Finding("App B", "high", new Dictionary<string, object> { ["cvssScore"] = 9.0 }), // no cveId
new() { ["softwareName"] = "No vulns" },
};
var entities = TableStorageService.BuildCveIndexEntities(Tenant, Session, findings);
Assert.Equal(2, entities.Count);
Assert.Contains(entities, x => x.GetString("CveId") == "CVE-1");
Assert.Contains(entities, x => x.GetString("CveId") == "CVE-2");
}
[Fact]
public void CveIdComparison_IsCaseInsensitive()
{
var findings = new List<Dictionary<string, object>>
{
Finding("A", "high", Vuln("CVE-2024-1", 7.0, "HIGH")),
Finding("B", "high", Vuln("cve-2024-1", 7.0, "HIGH")),
};
Assert.Single(TableStorageService.BuildCveIndexEntities(Tenant, Session, findings));
}
}
@@ -0,0 +1,96 @@
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Invariants of the built-in diagnostics catalog (<see cref="DiagnosticsBuiltInSections"/>).
/// Shared has no test project, so the catalog is pinned here: the agent iterates it to build the
/// archive and the backend serves it to the portal, so a malformed entry would either break
/// collection on every device or mislead every administrator reading the list.
/// </summary>
public class DiagnosticsBuiltInSectionsTests
{
[Fact]
public void SectionIds_AreUnique()
{
var ids = DiagnosticsBuiltInSections.All.Select(s => s.Id).ToArray();
Assert.Equal(ids.Length, ids.Distinct(StringComparer.Ordinal).Count());
}
[Fact]
public void ZipFolderAndSourceFolderPairs_AreUnique()
{
// Two sections may share a ZIP folder (ImeLogs + the bootstrapper evtx) only when they
// read from different source folders — the same (zip, source) pair would double-add
// every file under duplicate entry names.
var pairs = DiagnosticsBuiltInSections.All
.Select(s => $"{s.ZipFolder}|{s.SourceFolder}".ToLowerInvariant())
.ToArray();
Assert.Equal(pairs.Length, pairs.Distinct().Count());
}
[Fact]
public void AlwaysSections_KeepHistoricalArchiveLayoutAndOrder()
{
// Archive-layout lock: these five sections (and their order) are what every diagnostics
// ZIP has carried since the PR1-B forensics work; tooling and support habits rely on it.
var always = DiagnosticsBuiltInSections.All
.Where(s => s.Condition == DiagnosticsSectionCondition.Always)
.Select(s => s.Id)
.ToArray();
Assert.Equal(new[] { "AgentLogs", "ImeLogs", "AgentState", "AgentSpool", "AgentMarkers" }, always);
}
[Fact]
public void ConditionalSections_CarryTheExpectedGate()
{
foreach (var section in DiagnosticsBuiltInSections.All)
{
if (section.Id.StartsWith("RealmJoin", StringComparison.Ordinal))
Assert.Equal(DiagnosticsSectionCondition.RealmJoinWatcher, section.Condition);
if (section.Id == "ImeBootstrapperEventLog")
Assert.Equal(DiagnosticsSectionCondition.DevicePreparation, section.Condition);
}
Assert.Contains(DiagnosticsBuiltInSections.All, s => s.Id == "ImeBootstrapperEventLog");
Assert.Equal(5, DiagnosticsBuiltInSections.All.Count(s => s.Condition == DiagnosticsSectionCondition.RealmJoinWatcher));
}
[Fact]
public void UserProfileTokenSections_AreNeverAlwaysOn()
{
// The token resolves only once an interactive user exists; an Always section reading
// from the user profile would be skipped on every pre-desktop package and mislead the
// portal's "collected by every agent" wording.
var tokenSections = DiagnosticsBuiltInSections.All
.Where(s => s.SourceFolder.Contains(DiagnosticsBuiltInSections.UserProfileToken, StringComparison.OrdinalIgnoreCase))
.ToArray();
Assert.NotEmpty(tokenSections);
Assert.All(tokenSections, s => Assert.NotEqual(DiagnosticsSectionCondition.Always, s.Condition));
}
[Fact]
public void Patterns_AreFileNamesWithoutPathSeparators()
{
Assert.All(DiagnosticsBuiltInSections.All, s =>
{
Assert.NotEmpty(s.Patterns);
Assert.All(s.Patterns, p =>
{
Assert.False(string.IsNullOrWhiteSpace(p));
Assert.DoesNotContain('\\', p);
Assert.DoesNotContain('/', p);
});
Assert.False(string.IsNullOrWhiteSpace(s.Description));
Assert.False(s.ZipFolder.StartsWith("/") || s.ZipFolder.EndsWith("/"));
Assert.DoesNotContain('\\', s.ZipFolder);
});
}
[Fact]
public void NoBuiltInSection_UsesTheConfiguredPathsZipFolder()
{
// AdditionalLogs/<folder> is reserved for admin-configured entries.
Assert.All(DiagnosticsBuiltInSections.All,
s => Assert.False(s.ZipFolder.StartsWith("AdditionalLogs", StringComparison.OrdinalIgnoreCase)));
}
}
@@ -0,0 +1,124 @@
using System.Text.Json;
using AutopilotMonitor.Functions.Services.Notifications;
using AutopilotMonitor.Shared.Models.Notifications;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Discord embed rendering: payload shape, color mapping, and the Discord embed limits
/// (25 fields, 256/1024 name/value, 4096 description, 6000 total).
/// </summary>
public class DiscordRendererTests
{
private static JsonElement RenderEmbed(NotificationAlert alert)
{
var json = new DiscordRenderer().RenderToJson(alert);
var doc = JsonDocument.Parse(json);
var embeds = doc.RootElement.GetProperty("embeds");
Assert.Equal(1, embeds.GetArrayLength());
return embeds[0];
}
[Fact]
public void RendersSingleEmbed_WithTitleColorFieldsAndLinks()
{
var alert = new NotificationAlert
{
Title = "Enrollment Succeeded",
Summary = "Enrollment Succeeded: DESK-001",
Severity = NotificationSeverity.Success,
ThemeColor = "00B050",
Facts = { new NotificationFact { Name = "Device", Value = "DESK-001" } },
Sections = { new NotificationSection { Title = "Details", Text = "All apps installed." } },
Actions = { new NotificationAction { Type = "openUrl", Title = "Open session", Url = "https://portal.example.com/sessions/abc" } },
};
var embed = RenderEmbed(alert);
Assert.Equal("✅ Enrollment Succeeded", embed.GetProperty("title").GetString());
Assert.Equal(0x00B050, embed.GetProperty("color").GetInt32());
var description = embed.GetProperty("description").GetString()!;
Assert.StartsWith("Enrollment Succeeded: DESK-001", description);
Assert.Contains("**Details**\nAll apps installed.", description);
// Webhooks cannot post buttons — actions become markdown links.
Assert.Contains("[Open session](https://portal.example.com/sessions/abc)", description);
var fields = embed.GetProperty("fields");
Assert.Equal(1, fields.GetArrayLength());
Assert.Equal("Device", fields[0].GetProperty("name").GetString());
Assert.Equal("DESK-001", fields[0].GetProperty("value").GetString());
Assert.True(fields[0].GetProperty("inline").GetBoolean());
}
[Theory]
[InlineData(NotificationSeverity.Success, 0x2ECC71)]
[InlineData(NotificationSeverity.Error, 0xE74C3C)]
[InlineData(NotificationSeverity.Warning, 0xF1C40F)]
[InlineData(NotificationSeverity.Info, 0x3498DB)]
public void Color_FallsBackToSeverity_WhenThemeColorMissingOrInvalid(NotificationSeverity severity, int expected)
{
foreach (var themeColor in new[] { "", "not-hex" })
{
var embed = RenderEmbed(new NotificationAlert { Title = "T", Severity = severity, ThemeColor = themeColor });
Assert.Equal(expected, embed.GetProperty("color").GetInt32());
}
}
[Fact]
public void Fields_CappedAt25_AndEmptyFactsSkipped()
{
var alert = new NotificationAlert { Title = "T", Summary = "S" };
alert.Facts.Add(new NotificationFact { Name = "", Value = "dropped" });
alert.Facts.Add(new NotificationFact { Name = "dropped", Value = " " });
for (var i = 0; i < 30; i++)
alert.Facts.Add(new NotificationFact { Name = $"f{i}", Value = "v" });
var fields = RenderEmbed(alert).GetProperty("fields");
Assert.Equal(25, fields.GetArrayLength());
Assert.Equal("f0", fields[0].GetProperty("name").GetString()); // empty facts skipped, not counted
}
[Fact]
public void FieldValue_TruncatedTo1024_WithEllipsis()
{
var alert = new NotificationAlert { Title = "T" };
alert.Facts.Add(new NotificationFact { Name = "long", Value = new string('x', 2000) });
var value = RenderEmbed(alert).GetProperty("fields")[0].GetProperty("value").GetString()!;
Assert.Equal(1024, value.Length);
Assert.EndsWith("…", value);
}
[Fact]
public void TotalEmbedLength_CappedAt6000_ByDroppingTrailingFields()
{
var alert = new NotificationAlert { Title = "T", Summary = "S" };
for (var i = 0; i < 25; i++)
alert.Facts.Add(new NotificationFact { Name = $"f{i}", Value = new string('v', 1000) });
var embed = RenderEmbed(alert);
var fields = embed.GetProperty("fields");
var total = embed.GetProperty("title").GetString()!.Length
+ embed.GetProperty("description").GetString()!.Length;
for (var i = 0; i < fields.GetArrayLength(); i++)
total += fields[i].GetProperty("name").GetString()!.Length + fields[i].GetProperty("value").GetString()!.Length;
Assert.True(fields.GetArrayLength() < 25);
Assert.True(total <= 6000);
Assert.Equal("f0", fields[0].GetProperty("name").GetString()); // leading fields kept
}
[Fact]
public void EmptyDescription_Omitted_AndProviderTypeIsDiscord()
{
var embed = RenderEmbed(new NotificationAlert { Title = "T", Severity = NotificationSeverity.Info });
Assert.False(embed.TryGetProperty("description", out _));
Assert.False(embed.TryGetProperty("fields", out _));
Assert.Equal(WebhookProviderType.Discord, new DiscordRenderer().ProviderType);
}
}
@@ -0,0 +1,315 @@
using System.Collections.Generic;
using System.Net;
using System.Net.Http;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Coverage for the provider-neutral <see cref="EmailService"/> (welcome + farewell mails).
/// The provider HTTP call is driven through a stub handler, so the Mandrill wire format,
/// the result-array interpretation and the fail-soft contract are pinned without network.
/// <list type="bullet">
/// <item>Neither send method ever throws — not on a missing key, missing recipient,
/// HTTP 500, a rejected recipient, nor a malformed body.</item>
/// <item>Gate ordering: missing-key short-circuits before missing-recipient.</item>
/// <item>Tracking is off on the wire (trust-page claim: provider receives only the
/// recipient address and tenant domain).</item>
/// <item>The farewell template is final: no [DRAFT]/placeholder residue, domain is
/// interpolated, empty domain falls back to "your organization".</item>
/// </list>
/// </summary>
public sealed class EmailServiceTests
{
private const string TenantId = "88888888-8888-8888-8888-888888888888";
// ----- configuration gates -----
[Fact]
public async Task SendAsync_EmptyApiKey_NoOps_AndLogsMissingKeyDebugLine()
{
var (sut, handler, logger) = Build(apiKey: "");
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId, CancellationToken.None);
Assert.Equal(0, handler.CallCount);
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Debug && e.Message.Contains("Email:ApiKey not configured"));
}
[Fact]
public async Task SendAsync_EmptyRecipient_NoOps_AndLogsMissingEmailDebugLine()
{
var (sut, handler, logger) = Build(apiKey: "any-non-empty-key");
await sut.SendAsync("", "contoso.invalid", TenantId);
Assert.Equal(0, handler.CallCount);
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Debug && e.Message.Contains("No notification email captured"));
}
[Fact]
public async Task SendAsync_EmptyApiKey_ShortCircuitsBeforeRecipientCheck()
{
// Pins the gate ordering: missing-key > missing-email. Both are empty here; only
// the key line must show up.
var (sut, _, logger) = Build(apiKey: "");
await sut.SendAsync("", "contoso.invalid", TenantId);
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Debug && e.Message.Contains("Email:ApiKey not configured"));
Assert.DoesNotContain(logger.Entries,
e => e.Message.Contains("No notification email captured"));
}
[Fact]
public async Task SendPreviewApprovedEmailAsync_EmptyApiKey_NoOps()
{
var (sut, handler, logger) = Build(apiKey: "");
await sut.SendPreviewApprovedEmailAsync("it@contoso.invalid", "contoso.invalid");
Assert.Equal(0, handler.CallCount);
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Debug && e.Message.Contains("Email:ApiKey not configured"));
}
// ----- wire format -----
[Fact]
public async Task SendPreviewApprovedEmailAsync_PostsMandrillPayload_WithTrackingOff()
{
var (sut, handler, logger) = Build(apiKey: "md-test-key");
await sut.SendPreviewApprovedEmailAsync("it@contoso.invalid", "contoso.invalid");
Assert.Equal(1, handler.CallCount);
Assert.Equal(HttpMethod.Post, handler.LastRequest!.Method);
Assert.Equal(EmailService.DefaultEndpoint, handler.LastRequest.RequestUri!.ToString());
using var doc = JsonDocument.Parse(handler.LastBody!);
var root = doc.RootElement;
Assert.Equal("md-test-key", root.GetProperty("key").GetString());
var message = root.GetProperty("message");
Assert.Equal(EmailService.DefaultFromAddress, message.GetProperty("from_email").GetString());
Assert.Equal(EmailService.DefaultFromName, message.GetProperty("from_name").GetString());
Assert.Equal("it@contoso.invalid", message.GetProperty("to")[0].GetProperty("email").GetString());
Assert.Equal("to", message.GetProperty("to")[0].GetProperty("type").GetString());
Assert.Equal(EmailTemplates.PreviewApprovedSubject, message.GetProperty("subject").GetString());
Assert.Contains("contoso.invalid", message.GetProperty("html").GetString());
Assert.True(message.GetProperty("auto_text").GetBoolean());
Assert.False(message.GetProperty("track_opens").GetBoolean());
Assert.False(message.GetProperty("track_clicks").GetBoolean());
Assert.Equal("welcome", message.GetProperty("tags")[0].GetString());
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Information && e.Message.Contains("Preview approval email sent"));
}
[Fact]
public async Task SendAsync_UsesFarewellTemplate_AndFarewellTag()
{
var (sut, handler, logger) = Build(apiKey: "md-test-key");
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId);
using var doc = JsonDocument.Parse(handler.LastBody!);
var message = doc.RootElement.GetProperty("message");
Assert.Equal(EmailTemplates.OffboardingFarewellSubject, message.GetProperty("subject").GetString());
Assert.Equal("offboarding-farewell", message.GetProperty("tags")[0].GetString());
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Information && e.Message.Contains("Offboard farewell email sent"));
}
[Fact]
public async Task ConfiguredEndpointAndSender_OverrideDefaults()
{
var (sut, handler, _) = Build(apiKey: "md-test-key", extra: new Dictionary<string, string?>
{
["Email:Endpoint"] = "https://mail.example.invalid/send",
["Email:FromAddress"] = "hello@example.invalid",
["Email:FromName"] = "Example Sender",
});
await sut.SendPreviewApprovedEmailAsync("it@contoso.invalid", "contoso.invalid");
Assert.Equal("https://mail.example.invalid/send", handler.LastRequest!.RequestUri!.ToString());
using var doc = JsonDocument.Parse(handler.LastBody!);
var message = doc.RootElement.GetProperty("message");
Assert.Equal("hello@example.invalid", message.GetProperty("from_email").GetString());
Assert.Equal("Example Sender", message.GetProperty("from_name").GetString());
}
// ----- provider responses / fail-soft -----
[Fact]
public async Task ProviderHttpError_DoesNotThrow_AndLogsWarning_WithoutSuccessLine()
{
var (sut, _, logger) = Build(apiKey: "md-test-key",
responder: _ => Json(HttpStatusCode.InternalServerError,
"{\"status\":\"error\",\"code\":-1,\"name\":\"Invalid_Key\",\"message\":\"Invalid API key\"}"));
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId);
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Warning && e.Message.Contains("500") && e.Message.Contains("Invalid_Key"));
Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Information);
}
[Fact]
public async Task ProviderRejectsRecipient_LogsWarningWithReason_WithoutSuccessLine()
{
var (sut, _, logger) = Build(apiKey: "md-test-key",
responder: _ => Json(HttpStatusCode.OK,
"[{\"email\":\"ops@contoso.invalid\",\"status\":\"rejected\",\"reject_reason\":\"hard-bounce\",\"_id\":\"abc\"}]"));
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId);
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Warning && e.Message.Contains("rejected") && e.Message.Contains("hard-bounce"));
Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Information);
}
[Theory]
[InlineData("sent")]
[InlineData("queued")]
[InlineData("scheduled")]
public async Task ProviderAcceptedStatuses_CountAsSuccess(string status)
{
var (sut, _, logger) = Build(apiKey: "md-test-key",
responder: _ => Json(HttpStatusCode.OK,
$"[{{\"email\":\"ops@contoso.invalid\",\"status\":\"{status}\",\"_id\":\"abc\"}}]"));
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId);
Assert.Contains(logger.Entries, e => e.Level == LogLevel.Information);
Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Warning);
}
[Fact]
public async Task MalformedProviderBody_DoesNotThrow_AndLogsWarning()
{
var (sut, _, logger) = Build(apiKey: "md-test-key",
responder: _ => Json(HttpStatusCode.OK, "not json"));
await sut.SendAsync("ops@contoso.invalid", "contoso.invalid", TenantId);
Assert.Contains(logger.Entries, e => e.Level == LogLevel.Warning);
Assert.DoesNotContain(logger.Entries, e => e.Level == LogLevel.Information);
}
[Fact]
public async Task TransportException_DoesNotThrow_AndLogsWarning()
{
var (sut, _, logger) = Build(apiKey: "md-test-key",
responder: _ => throw new HttpRequestException("simulated provider outage"));
await sut.SendPreviewApprovedEmailAsync("it@contoso.invalid", "contoso.invalid");
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Warning && e.Message.Contains("Failed to send welcome mail"));
}
// ----- templates -----
[Fact]
public void FarewellTemplate_IsFinal_NoDraftResidue()
{
Assert.DoesNotContain("DRAFT", EmailTemplates.OffboardingFarewellSubject);
var html = EmailTemplates.GetOffboardingFarewellHtml("contoso.invalid");
Assert.DoesNotContain("DRAFT", html);
Assert.DoesNotContain("TEMPLATE NOT FINALISED", html);
Assert.DoesNotContain("TODO", html);
}
[Fact]
public void FarewellTemplate_InterpolatesDomain_AndLinksFeedbackChannels()
{
var html = EmailTemplates.GetOffboardingFarewellHtml("contoso.invalid");
Assert.Contains("contoso.invalid", html);
// The mail is sent from a noreply address after portal access is gone, so the
// feedback pointers must be external channels the recipient can actually reach.
Assert.Contains("https://github.com/okieselbach/AutopilotMonitor/issues", html);
Assert.Contains("linkedin.com", html);
}
[Fact]
public void FarewellTemplate_EmptyDomain_FallsBackToGenericLabel()
{
var html = EmailTemplates.GetOffboardingFarewellHtml("");
Assert.Contains("your organization", html);
}
// ----- harness -----
private static HttpResponseMessage Json(HttpStatusCode status, string body)
=> new(status) { Content = new StringContent(body, System.Text.Encoding.UTF8, "application/json") };
private static (EmailService sut, StubHandler handler, CapturingLogger<EmailService> logger) Build(
string apiKey,
Dictionary<string, string?>? extra = null,
System.Func<HttpRequestMessage, HttpResponseMessage>? responder = null)
{
var settings = new Dictionary<string, string?> { ["Email:ApiKey"] = apiKey };
if (extra is not null)
foreach (var kv in extra) settings[kv.Key] = kv.Value;
var config = new ConfigurationBuilder().AddInMemoryCollection(settings).Build();
var handler = new StubHandler();
if (responder is not null) handler.Responder = responder;
var logger = new CapturingLogger<EmailService>();
var templates = new EmailTemplateService(
Moq.Mock.Of<AutopilotMonitor.Shared.DataAccess.IConfigRepository>(),
new Microsoft.Extensions.Caching.Memory.MemoryCache(new Microsoft.Extensions.Caching.Memory.MemoryCacheOptions()),
Microsoft.Extensions.Logging.Abstractions.NullLogger<EmailTemplateService>.Instance);
var sut = new EmailService(new HttpClient(handler, disposeHandler: false), config, templates, logger);
return (sut, handler, logger);
}
private sealed class StubHandler : HttpMessageHandler
{
public int CallCount { get; private set; }
public HttpRequestMessage? LastRequest { get; private set; }
public string? LastBody { get; private set; }
public System.Func<HttpRequestMessage, HttpResponseMessage> Responder { get; set; } = _ =>
Json(HttpStatusCode.OK, "[{\"email\":\"x@y.invalid\",\"status\":\"sent\",\"_id\":\"abc\"}]");
protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
CallCount++;
LastRequest = request;
LastBody = request.Content is null ? null : await request.Content.ReadAsStringAsync(cancellationToken);
return Responder(request);
}
}
private sealed class CapturingLogger<T> : ILogger<T>
{
public readonly List<(LogLevel Level, string Message)> Entries = new();
public IDisposable BeginScope<TState>(TState state) where TState : notnull => NullScope.Instance;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(
LogLevel logLevel, EventId eventId, TState state, System.Exception? exception,
System.Func<TState, System.Exception?, string> formatter)
{
Entries.Add((logLevel, formatter(state, exception)));
}
private sealed class NullScope : IDisposable
{
public static readonly NullScope Instance = new();
public void Dispose() { }
}
}
}
@@ -0,0 +1,131 @@
using System.Collections.Generic;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins the operator-override layer over the built-in transactional templates:
/// override wins when stored, built-in otherwise; placeholder rendering matches the
/// built-ins' empty-domain fallback; the send path is fail-soft on storage errors while the
/// admin path is not; save validates and invalidates the cache; reset is idempotent.
/// </summary>
public sealed class EmailTemplateServiceTests
{
[Fact]
public async Task NoOverride_ReturnsBuiltIn_Rendered()
{
var (sut, _) = Build();
var html = await sut.GetHtmlAsync(EmailTemplateKind.Welcome, "contoso.invalid");
Assert.Equal(EmailTemplates.GetPreviewApprovedHtml("contoso.invalid"), html);
Assert.DoesNotContain(EmailTemplateService.DomainPlaceholder, html);
}
[Fact]
public async Task Override_WinsAndRendersPlaceholder()
{
var (sut, repo) = Build();
repo.Setup(r => r.GetEmailTemplateOverrideAsync("farewell"))
.ReturnsAsync(new EmailTemplateOverride { Kind = "farewell", Html = "<p>Bye {{domainName}}</p>" });
var html = await sut.GetHtmlAsync(EmailTemplateKind.Farewell, "contoso.invalid");
Assert.Equal("<p>Bye contoso.invalid</p>", html);
}
[Fact]
public async Task Override_EmptyDomain_FallsBackToGenericLabel_LikeBuiltIn()
{
var (sut, repo) = Build();
repo.Setup(r => r.GetEmailTemplateOverrideAsync("welcome"))
.ReturnsAsync(new EmailTemplateOverride { Kind = "welcome", Html = "Hi {{domainName}}" });
Assert.Equal("Hi your organization", await sut.GetHtmlAsync(EmailTemplateKind.Welcome, ""));
}
[Fact]
public async Task StorageError_SendPath_FallsBackToBuiltIn_AdminPath_Throws()
{
var (sut, repo) = Build();
repo.Setup(r => r.GetEmailTemplateOverrideAsync(It.IsAny<string>()))
.ThrowsAsync(new System.InvalidOperationException("storage down"));
var html = await sut.GetHtmlAsync(EmailTemplateKind.Welcome, "contoso.invalid");
Assert.Contains("contoso.invalid", html);
await Assert.ThrowsAsync<System.InvalidOperationException>(() => sut.GetOverrideAsync(EmailTemplateKind.Welcome));
}
[Fact]
public async Task Save_Validates_ThenInvalidatesCache()
{
var (sut, repo) = Build();
repo.Setup(r => r.GetEmailTemplateOverrideAsync("welcome")).ReturnsAsync((EmailTemplateOverride?)null);
// warm the cache with "no override"
Assert.Null(await sut.GetOverrideAsync(EmailTemplateKind.Welcome));
await Assert.ThrowsAsync<System.ArgumentException>(() => sut.SaveOverrideAsync(EmailTemplateKind.Welcome, " ", "ga@contoso.invalid"));
await Assert.ThrowsAsync<System.ArgumentException>(() =>
sut.SaveOverrideAsync(EmailTemplateKind.Welcome, new string('x', EmailTemplateService.MaxHtmlLength + 1), "ga@contoso.invalid"));
repo.Verify(r => r.SaveEmailTemplateOverrideAsync(It.IsAny<EmailTemplateOverride>()), Times.Never);
EmailTemplateOverride? stored = null;
repo.Setup(r => r.SaveEmailTemplateOverrideAsync(It.IsAny<EmailTemplateOverride>()))
.Callback<EmailTemplateOverride>(e => stored = e).Returns(Task.CompletedTask);
repo.Setup(r => r.GetEmailTemplateOverrideAsync("welcome")).ReturnsAsync(() => stored);
var saved = await sut.SaveOverrideAsync(EmailTemplateKind.Welcome, "<p>{{domainName}}</p>", "ga@contoso.invalid");
Assert.Equal("welcome", saved.Kind);
Assert.Equal("ga@contoso.invalid", saved.UpdatedBy);
// cache was invalidated: the next read sees the stored override, not the cached null
Assert.NotNull(await sut.GetOverrideAsync(EmailTemplateKind.Welcome));
Assert.Equal("<p>contoso.invalid</p>", await sut.GetHtmlAsync(EmailTemplateKind.Welcome, "contoso.invalid"));
}
[Fact]
public async Task Delete_ClearsCache_ReturnsToBuiltIn()
{
var (sut, repo) = Build();
var stored = new EmailTemplateOverride { Kind = "farewell", Html = "custom {{domainName}}" };
repo.Setup(r => r.GetEmailTemplateOverrideAsync("farewell")).ReturnsAsync(() => stored);
Assert.Equal("custom contoso.invalid", await sut.GetHtmlAsync(EmailTemplateKind.Farewell, "contoso.invalid"));
repo.Setup(r => r.DeleteEmailTemplateOverrideAsync("farewell")).Callback(() => stored = null!).Returns(Task.CompletedTask);
await sut.DeleteOverrideAsync(EmailTemplateKind.Farewell, "ga@contoso.invalid");
Assert.Equal(EmailTemplates.GetOffboardingFarewellHtml("contoso.invalid"),
await sut.GetHtmlAsync(EmailTemplateKind.Farewell, "contoso.invalid"));
}
[Theory]
[InlineData("welcome", true)]
[InlineData("Farewell", true)]
[InlineData("invoice", false)]
[InlineData(null, false)]
public void TryParseKind_AcceptsOnlyKnownKinds(string? value, bool expected)
=> Assert.Equal(expected, EmailTemplateService.TryParseKind(value, out _));
[Fact]
public void BuiltInRaw_ContainsPlaceholder_ForBothKinds()
{
Assert.Contains(EmailTemplateService.DomainPlaceholder, EmailTemplateService.BuiltInRaw(EmailTemplateKind.Welcome));
Assert.Contains(EmailTemplateService.DomainPlaceholder, EmailTemplateService.BuiltInRaw(EmailTemplateKind.Farewell));
}
private static (EmailTemplateService sut, Mock<IConfigRepository> repo) Build()
{
var repo = new Mock<IConfigRepository>(MockBehavior.Loose);
var sut = new EmailTemplateService(repo.Object, new MemoryCache(new MemoryCacheOptions()), NullLogger<EmailTemplateService>.Instance);
return (sut, repo);
}
}
@@ -191,6 +191,21 @@ public class EndpointPolicyCatalogCompletenessTests
Assert.Equal(EndpointPolicy.GlobalAdminOnly, reseed.Policy);
}
/// <summary>
/// The built-in diagnostics catalog + global paths are platform-wide, secret-free data that
/// every settings viewer needs (what is collected from the tenant's devices). Member tier,
/// JWT-scoped — an upgrade to a global tier would hide it from tenant admins again.
/// </summary>
[Fact]
public void DiagnosticsPaths_IsMemberReadJwtScoped()
{
var entry = EndpointAccessPolicyCatalog.FindPolicy("GET", "/api/diagnostics/paths");
Assert.NotNull(entry);
Assert.Equal(EndpointPolicy.MemberRead, entry.Policy);
Assert.Equal(TenantScoping.Jwt, entry.TenantScoping);
}
/// <summary>
/// Unregistered routes must return null (fail-closed).
/// </summary>
@@ -163,6 +163,12 @@ public class EnrollmentTimeoutClassifierTests
// -------- ClassifyTimedOutSession --------
private static (SessionStatus, string) Classify(IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 6, int grace = 72)
{
var (status, reason, _) = ClassifyWithRule(events, hoursSinceStart, grace);
return (status, reason);
}
private static (SessionStatus, string, string) ClassifyWithRule(IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 6, int grace = 72)
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var now = Start.AddHours(hoursSinceStart);
@@ -220,7 +226,7 @@ public class EnrollmentTimeoutClassifierTests
Esp(DeviceSetup44), Esp(AccountSetup15),
Evt("desktop_arrived"), Evt("hello_provisioning_completed"), Evt("realmjoin_detected"),
});
var (status, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
var (status, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, now, graceHours: 72, lastEventAtUtc: lastEvent);
Assert.Equal(SessionStatus.Succeeded, status);
@@ -471,4 +477,230 @@ public class EnrollmentTimeoutClassifierTests
Assert.NotEqual(SessionStatus.Failed, status);
}
}
// -------- WhiteGlove Part-2 awaiting-user gate (fairstone.ca analysis 2026-08-21) --------
private static readonly DateTime Resumed = Start.AddMinutes(20);
private static (SessionStatus, string) ClassifyWhiteGlove(
IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 6, int grace = 51)
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var now = Start.AddHours(hoursSinceStart);
var (status, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, now, grace, isPreProvisioned: true, resumedAt: Resumed);
return (status, reason);
}
/// <summary>The fairstone event shape: Part 1 sealed, Part 2 resumed, nobody signed in.</summary>
private static EnrollmentEvent[] WgParkedEvents() => new[]
{
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup05),
};
[Fact]
public void ExtractRollup_separates_enrollment_complete_from_whiteglove_complete()
{
var wgOnly = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("whiteglove_complete") });
Assert.True(wgOnly.HasTerminalComplete);
Assert.False(wgOnly.HasEnrollmentComplete);
var real = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("enrollment_complete") });
Assert.True(real.HasTerminalComplete);
Assert.True(real.HasEnrollmentComplete);
}
[Fact]
public void WhiteGloveGate_requires_preprovisioned_and_resume_and_device_setup()
{
var parked = EnrollmentTimeoutClassifier.ExtractRollup(WgParkedEvents());
Assert.True(EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(parked, true, Resumed));
Assert.False(EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(parked, false, Resumed));
Assert.False(EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(parked, true, null));
// Powered off before Device Setup ever provisioned → not the parking state.
var early = EnrollmentTimeoutClassifier.ExtractRollup(new[]
{
Esp("ESP provisioning status: DeviceSetup — 3 of 4 subcategories completed"),
});
Assert.False(EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(early, true, Resumed));
}
[Fact]
public void WhiteGloveGate_any_user_evidence_or_terminal_disarms_it()
{
foreach (var evidence in new[]
{
Esp(AccountSetup15), // Account Setup progressed → user was there
Evt("desktop_arrived"), // real-user desktop
Evt("hello_provisioning_completed"), // positive Hello terminal
Evt("enrollment_complete"), // real completion
Evt("enrollment_failed"), // explicit failure
})
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(new[]
{
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup05), evidence,
});
Assert.False(EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(rollup, true, Resumed));
}
}
[Fact]
public void Classify_whiteglove_part2_parked_within_grace_is_AwaitingUser()
{
// The fairstone shape: technician powers the device off at the logon screen minutes
// after the reseal-reboot. Part-1 whiteglove_complete must NOT reconcile to Succeeded.
var (status, reason) = ClassifyWhiteGlove(WgParkedEvents(), hoursSinceStart: 6);
Assert.Equal(SessionStatus.AwaitingUser, status);
Assert.Contains("sealed or powered off awaiting the end user", reason);
Assert.Contains("(Account Setup 0/5)", reason);
}
[Fact]
public void Classify_whiteglove_part2_parked_past_grace_reconciles_with_honest_reason()
{
var (status, reason) = ClassifyWhiteGlove(WgParkedEvents(), hoursSinceStart: 60, grace: 51);
Assert.Equal(SessionStatus.Succeeded, status);
Assert.Contains("pre-provisioning (WhiteGlove Part 1) completed", reason);
Assert.DoesNotContain("Account Setup completed", reason);
}
[Fact]
public void Classify_whiteglove_part2_with_partial_user_progress_is_not_reconciled_on_part1_evidence()
{
// User signed in and Account Setup progressed to 1/5, then silence. The Part-1
// whiteglove_complete previously reconciled this to Succeeded claiming "Account Setup
// completed" — on a resumed session only a real enrollment_complete proves the outcome,
// so this honest shape falls through to the AwaitingUser/Incomplete rules.
var events = new[]
{
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup15),
};
var (withinGrace, _) = ClassifyWhiteGlove(events, hoursSinceStart: 6);
Assert.Equal(SessionStatus.AwaitingUser, withinGrace);
var (pastGrace, reason) = ClassifyWhiteGlove(events, hoursSinceStart: 60, grace: 51);
Assert.Equal(SessionStatus.Incomplete, pastGrace);
Assert.Contains("1/5", reason);
}
[Fact]
public void Classify_whiteglove_part2_with_desktop_and_hello_still_reconciles_via_user_evidence()
{
// Both Classic completion prerequisites in → the user provably finished; the
// user-completed reconcile (rule 4) owns this, not the parking gate.
var (status, reason) = ClassifyWhiteGlove(new[]
{
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup15),
Evt("desktop_arrived"), Evt("hello_provisioning_completed"),
}, hoursSinceStart: 6);
Assert.Equal(SessionStatus.Succeeded, status);
Assert.Contains("user completed setup", reason);
}
[Fact]
public void Classify_whiteglove_part2_real_enrollment_complete_still_reconciles_to_Succeeded()
{
var (status, _) = ClassifyWhiteGlove(new[]
{
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup05), Evt("enrollment_complete"),
}, hoursSinceStart: 6);
Assert.Equal(SessionStatus.Succeeded, status);
}
[Fact]
public void Classify_unresumed_session_keeps_terminal_complete_reconcile()
{
// No resumedAt (plain sessions, or a sealed Part 1 whose Pending write was lost):
// whiteglove_complete keeps reconciling to Succeeded exactly as before.
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(new[]
{
Esp(DeviceSetup44), Evt("whiteglove_complete"), Esp(AccountSetup05),
});
var (status, _, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, Start.AddHours(6), 51, isPreProvisioned: true, resumedAt: null);
Assert.Equal(SessionStatus.Succeeded, status);
}
// -------- Self-deploying profile gate (kiosk tenant aebdce78, audit 2026-08-23) --------
private static (SessionStatus, string) ClassifySelfDeploying(
IReadOnlyList<EnrollmentEvent> events, double hoursSinceStart = 3, int grace = 51)
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var (status, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, Start.AddHours(hoursSinceStart), grace,
lastEventAtUtc: Start.AddMinutes(4), isSelfDeployingProfile: true);
return (status, reason);
}
// Session 195593e2 replay: DeviceSetup 4/4 (fallback-confirmed), AccountSetup registry
// at 0/5 (the IME false positive — user ESP never runs on this profile), Hello disabled,
// SkipUser=True, then silence after the post-ESP reboot. No user will ever sign in.
private static readonly EnrollmentEvent[] KioskSilentStream =
{
Esp(DeviceSetup44), Esp(AccountSetup05), Esp(DeviceSetupFallback),
HelloPolicy(enabled: false), EspConfig(skipUser: true),
};
[Fact]
public void IsSelfDeployingProvisioned_requires_flag_device_setup_and_no_failure()
{
var provisioned = EnrollmentTimeoutClassifier.ExtractRollup(KioskSilentStream);
Assert.True(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(provisioned, isSelfDeployingProfile: true));
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(provisioned, isSelfDeployingProfile: false));
var beforeDeviceSetup = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(AccountSetup05) });
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(beforeDeviceSetup, isSelfDeployingProfile: true));
var failed = EnrollmentTimeoutClassifier.ExtractRollup(new[] { Esp(DeviceSetup44), Evt("enrollment_failed") });
Assert.False(EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned(failed, isSelfDeployingProfile: true));
}
[Fact]
public void Classify_self_deploying_silent_after_device_setup_reconciles_to_Succeeded_within_grace()
{
var (status, reason) = ClassifySelfDeploying(KioskSilentStream, hoursSinceStart: 3);
Assert.Equal(SessionStatus.Succeeded, status);
Assert.Contains("self-deploying profile", reason);
Assert.Contains("no user / Account Setup phase", reason);
Assert.Contains("Agent last reported 2026-07-06 15:04 UTC", reason);
Assert.DoesNotContain("Account Setup completed", reason);
}
[Fact]
public void Classify_self_deploying_silent_after_device_setup_reconciles_to_Succeeded_past_grace()
{
// Grace is a user-phase concept; it does not apply when there is no user phase.
var (status, _) = ClassifySelfDeploying(KioskSilentStream, hoursSinceStart: 60, grace: 51);
Assert.Equal(SessionStatus.Succeeded, status);
}
[Fact]
public void Classify_same_stream_without_self_deploying_flag_keeps_user_phase_verdicts()
{
// Guard against the gate leaking into user-driven sessions: the identical event shape
// on a user-driven profile is still "awaiting user" within grace / Incomplete past it.
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(KioskSilentStream);
var (within, _, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(3), 51);
var (past, _, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(rollup, Start, Start.AddHours(60), 51);
Assert.Equal(SessionStatus.AwaitingUser, within);
Assert.Equal(SessionStatus.Incomplete, past);
}
[Fact]
public void Classify_self_deploying_explicit_failure_stays_Failed()
{
var (status, _) = ClassifySelfDeploying(new[] { Esp(DeviceSetup44), Evt("enrollment_failed") });
Assert.Equal(SessionStatus.Failed, status);
}
[Fact]
public void Classify_self_deploying_silent_before_device_setup_is_Incomplete()
{
// The profile flag alone proves nothing — Device ESP must have finished.
var (status, _) = ClassifySelfDeploying(new[] { Esp(AccountSetup05), Evt("agent_started") });
Assert.Equal(SessionStatus.Incomplete, status);
}
}
@@ -1,4 +1,5 @@
using AutopilotMonitor.Functions.Functions.Config;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
@@ -32,6 +33,44 @@ public class GetAgentConfigFunctionTests
Assert.Equal(expectedMajor, actual);
}
// ── MergeDiagnosticsLogPaths — global ∪ tenant without duplicate ZIP entries ──────────
private static DiagnosticsLogPath P(string path, string description = "") =>
new() { Path = path, Description = description };
[Fact]
public void MergeDiagnosticsLogPaths_GlobalFirst_OrderPreserved_DuplicatesDroppedCaseInsensitively()
{
var global = new[] { P(@"C:\Windows\Panther\setupact.log", "global"), P(@"C:\Windows\Logs\DISM\dism.log") };
var tenant = new[]
{
P(@"c:\windows\panther\SETUPACT.LOG", "tenant copy"), // same path, different case → dropped
P(@" C:\Windows\Logs\DISM\dism.log "), // trimmed duplicate → dropped
P(@"C:\Windows\SetupDiag\*.log", "tenant"),
};
var merged = GetAgentConfigFunction.MergeDiagnosticsLogPaths(global, tenant);
Assert.Equal(new[]
{
@"C:\Windows\Panther\setupact.log",
@"C:\Windows\Logs\DISM\dism.log",
@"C:\Windows\SetupDiag\*.log",
}, merged.Select(p => p.Path));
Assert.Equal("global", merged[0].Description); // first occurrence (global) wins
}
[Fact]
public void MergeDiagnosticsLogPaths_DropsBlankEntries()
{
var merged = GetAgentConfigFunction.MergeDiagnosticsLogPaths(
new[] { P(""), P(" "), P(@"C:\Install\Log\*.log") },
new DiagnosticsLogPath[] { null!, P(@"C:\Install\Log\*.log") });
Assert.Single(merged);
Assert.Equal(@"C:\Install\Log\*.log", merged[0].Path);
}
[Theory]
// CustomerSas: gated on the per-tenant SAS URL being present.
[InlineData("https://acct.blob.core.windows.net/c?sv=...", "CustomerSas", true)]
@@ -0,0 +1,70 @@
using System.Text.Json;
using AutopilotMonitor.Functions.Functions.Diagnostics;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Wire shape of <c>GET /api/diagnostics/paths</c> (<see cref="GetDiagnosticsPathsFunction.BuildPayload"/>).
/// The route is MemberRead, so the field set is locked: the payload must stay "what is
/// collected" — never grow admin-only config.
/// </summary>
public class GetDiagnosticsPathsPayloadTests
{
// Mirrors the worker's serializer (Program.cs: PropertyNamingPolicy = CamelCase) so the
// nested DiagnosticsLogPath class serializes the way the portal actually receives it.
private static readonly JsonSerializerOptions WorkerJson = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase };
private static JsonElement Serialize(AdminConfiguration config)
{
var json = JsonSerializer.Serialize(GetDiagnosticsPathsFunction.BuildPayload(config), WorkerJson);
return JsonDocument.Parse(json).RootElement;
}
[Fact]
public void Payload_HasExactlyBuiltInAndGlobalPaths()
{
var element = Serialize(new AdminConfiguration());
var fieldNames = element.EnumerateObject().Select(p => p.Name).OrderBy(n => n).ToArray();
Assert.Equal(new[] { "builtIn", "globalPaths" }, fieldNames);
Assert.Equal(JsonValueKind.Array, element.GetProperty("globalPaths").ValueKind);
Assert.Equal(0, element.GetProperty("globalPaths").GetArrayLength());
}
[Fact]
public void BuiltIn_MirrorsTheCatalog_WithConditionAsName()
{
var builtIn = Serialize(new AdminConfiguration()).GetProperty("builtIn");
Assert.Equal(DiagnosticsBuiltInSections.All.Count, builtIn.GetArrayLength());
var first = builtIn[0];
Assert.Equal("AgentLogs", first.GetProperty("id").GetString());
Assert.Equal("AgentLogs", first.GetProperty("zipFolder").GetString());
Assert.Equal(AutopilotMonitor.Shared.Constants.LogDirectory, first.GetProperty("sourceFolder").GetString());
Assert.Equal(DiagnosticsBuiltInSections.LogFilePatterns.Length, first.GetProperty("patterns").GetArrayLength());
Assert.False(first.GetProperty("includeSubfolders").GetBoolean());
Assert.Equal("Always", first.GetProperty("condition").GetString());
var conditions = builtIn.EnumerateArray().Select(s => s.GetProperty("condition").GetString()).ToArray();
Assert.Contains("RealmJoinWatcher", conditions);
Assert.Contains("DevicePreparation", conditions);
Assert.All(conditions, c => Assert.False(int.TryParse(c, out _), "condition must serialize as the enum name"));
}
[Fact]
public void GlobalPaths_RoundTripTheStoredJson()
{
var config = new AdminConfiguration
{
DiagnosticsGlobalLogPathsJson =
"[{\"path\":\"C:\\\\Windows\\\\Panther\\\\setupact.log\",\"description\":\"Setup\",\"isBuiltIn\":true,\"includeSubfolders\":false}]",
};
var globalPaths = Serialize(config).GetProperty("globalPaths");
Assert.Equal(1, globalPaths.GetArrayLength());
Assert.Equal(@"C:\Windows\Panther\setupact.log", globalPaths[0].GetProperty("path").GetString());
Assert.Equal("Setup", globalPaths[0].GetProperty("description").GetString());
}
}
@@ -0,0 +1,166 @@
using System;
using System.Collections.Generic;
using System.Linq;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Pins that the ONE projected window scan the maintenance tick shares across its rolling
/// sweeps (<c>GetMaintenanceWindowSessionsAsync</c> / <c>MaintenanceSweepSessionProjection</c>)
/// carries every field the time-attribution, device-journey and verdict-calibration sweeps
/// read. In Azure Table Storage a <c>$select</c> returns ONLY the listed properties, so the
/// projected row here is built from the production array — dropping a column there fails the
/// sweep that consumes it. Mirrors <see cref="UsageMetricsProjectionEquivalenceTests"/>.
/// </summary>
public class MaintenanceSweepProjectionEquivalenceTests
{
private const string TenantId = "00000000-0000-0000-0000-000000000abc";
private static readonly DateTime Started = new(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
private static readonly TableStorageService Sut =
new(new Mock<TableServiceClient>().Object, NullLogger<TableStorageService>.Instance);
/// <summary>Full wide Sessions row with every column the sweeps read set to a distinct non-default value, plus noise.</summary>
private static TableEntity FullRow() => new(TenantId, "11111111-1111-1111-1111-111111111111")
{
["Status"] = "Succeeded",
["StartedAt"] = new DateTimeOffset(Started),
["CompletedAt"] = new DateTimeOffset(Started.AddHours(1)),
["LastEventAt"] = new DateTimeOffset(Started.AddHours(2)),
["DurationSeconds"] = 3600,
["EventCount"] = 321,
["DeletionState"] = "None",
["SerialNumber"] = "SN-1234",
["Manufacturer"] = "Contoso",
["Model"] = "Model-X",
["EnrollmentType"] = "v2",
["IsPreProvisioned"] = true,
["IsSelfDeployingProfile"] = true,
["IsUserDriven"] = false,
["ResumedAt"] = new DateTimeOffset(Started.AddMinutes(30)),
["AdminMarkedAction"] = "Succeeded",
["VerdictPath"] = "manual:succeeded",
["PriorStatus"] = "Incomplete",
["PriorVerdictPath"] = "sweep:r6",
["FailureReason"] = "No Device Setup completion or explicit failure signal observed before timeout",
["ReconcileReason"] = "Reconciled at timeout: user completed setup (desktop + Windows Hello)",
["FailureSource"] = "manual",
["EspSoftFailure"] = true,
// Noise the sweeps never read — present on the wide row, dropped by the projection.
["DeviceName"] = "PC-FULL",
["OsName"] = "Windows 11",
["OsBuild"] = "26200.1234",
["GeoCountry"] = "DE",
["AgentVersion"] = "2.9.0",
["DiagnosticsBlobName"] = "diag.zip",
["FailureSnapshotJson"] = "{\"big\":\"" + new string('x', 2000) + "\"}",
};
private static TableEntity Project(TableEntity full)
{
var keep = new HashSet<string>(TableStorageService.MaintenanceSweepSessionProjection, StringComparer.Ordinal);
var projected = new TableEntity(full.PartitionKey, full.RowKey);
foreach (var kv in full)
{
if (keep.Contains(kv.Key)) projected[kv.Key] = kv.Value;
}
return projected;
}
[Fact]
public void Projection_drops_the_wide_columns()
{
var projected = Project(FullRow());
Assert.False(projected.ContainsKey("FailureSnapshotJson"));
Assert.False(projected.ContainsKey("GeoCountry"));
Assert.False(projected.ContainsKey("OsBuild"));
Assert.True(projected.Count < FullRow().Count);
}
[Fact]
public void Every_field_the_sweeps_read_survives_the_projection()
{
var full = Sut.MapToSessionSummary(FullRow());
var proj = Sut.MapToSessionSummary(Project(FullRow()));
// Shared / structural
Assert.Equal(full.TenantId, proj.TenantId);
Assert.Equal(full.SessionId, proj.SessionId);
Assert.Equal(full.Status, proj.Status);
Assert.Equal(full.StartedAt, proj.StartedAt);
Assert.Equal(full.CompletedAt, proj.CompletedAt);
Assert.Equal(full.DeletionState, proj.DeletionState);
// Time attribution: Status/DurationSeconds/DeletionState/EventCount + enrollment class
Assert.Equal(full.DurationSeconds, proj.DurationSeconds);
Assert.Equal(full.EventCount, proj.EventCount);
Assert.Equal(TimeAttributionCalculator.GetEnrollmentClass(full), TimeAttributionCalculator.GetEnrollmentClass(proj));
// Device journeys: BuildSessionRef + BuildDeviceHistoryRow display fields
Assert.Equal(full.SerialNumber, proj.SerialNumber);
Assert.Equal(full.Manufacturer, proj.Manufacturer);
Assert.Equal(full.Model, proj.Model);
Assert.Equal(full.EnrollmentType, proj.EnrollmentType);
Assert.Equal(full.IsPreProvisioned, proj.IsPreProvisioned);
Assert.Equal(full.AdminMarkedAction, proj.AdminMarkedAction);
var refFull = DeviceJourneyCalculator.BuildSessionRef(full)!;
var refProj = DeviceJourneyCalculator.BuildSessionRef(proj)!;
Assert.Equal(System.Text.Json.JsonSerializer.Serialize(refFull), System.Text.Json.JsonSerializer.Serialize(refProj));
// Verdict calibration: attribution + legacy derivation inputs + proxy timestamps
Assert.Equal(full.LastEventAt, proj.LastEventAt);
Assert.Equal(full.VerdictPath, proj.VerdictPath);
Assert.Equal(full.PriorStatus, proj.PriorStatus);
Assert.Equal(full.PriorVerdictPath, proj.PriorVerdictPath);
Assert.Equal(full.FailureReason, proj.FailureReason);
Assert.Equal(full.ReconcileReason, proj.ReconcileReason);
Assert.Equal(full.FailureSource, proj.FailureSource);
Assert.Equal(full.EspSoftFailure, proj.EspSoftFailure);
Assert.Equal(VerdictPathDerivation.Derive(full), VerdictPathDerivation.Derive(proj));
// Unstamped variant exercises the derivation on the projected inputs.
var unstampedFull = FullRow(); unstampedFull.Remove("VerdictPath"); unstampedFull.Remove("AdminMarkedAction"); unstampedFull.Remove("FailureSource");
Assert.Equal(VerdictPathDerivation.Derive(Sut.MapToSessionSummary(unstampedFull)),
VerdictPathDerivation.Derive(Sut.MapToSessionSummary(Project(unstampedFull))));
}
[Fact]
public void Aggregation_cores_produce_identical_rows_from_the_projected_scan()
{
var now = new DateTime(2026, 8, 23, 12, 0, 0, DateTimeKind.Utc);
var fullList = new List<SessionSummary> { Sut.MapToSessionSummary(FullRow()) };
var projList = new List<SessionSummary> { Sut.MapToSessionSummary(Project(FullRow())) };
var histories = new Dictionary<string, List<DeviceHistory>>();
var a = MaintenanceService.BuildVerdictCalibrationAggregates(fullList, histories, now);
var b = MaintenanceService.BuildVerdictCalibrationAggregates(projList, histories, now);
Assert.Equal(System.Text.Json.JsonSerializer.Serialize(a), System.Text.Json.JsonSerializer.Serialize(b));
}
[Fact]
public void SliceSweepWindow_keeps_sessions_from_the_window_start()
{
var window = new List<SessionSummary>
{
new() { SessionId = "old", StartedAt = Started.AddDays(-40) },
new() { SessionId = "edge", StartedAt = Started },
new() { SessionId = "new", StartedAt = Started.AddDays(1) },
};
var sliced = MaintenanceService.SliceSweepWindow(window, Started);
Assert.Equal(new[] { "edge", "new" }, sliced.Select(s => s.SessionId));
}
[Fact]
public void Shared_window_covers_every_sweeps_horizon()
{
// 7d window + 28d baseline anchored on yesterday = 35 days back ≥ the 30-day rolling sweeps.
Assert.Equal(35, MaintenanceService.SweepWindowDays);
Assert.True(MaintenanceService.SweepWindowDays >= 30);
}
}
@@ -1,4 +1,5 @@
using AutopilotMonitor.Functions.Functions.Config;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models;
using AutopilotMonitor.Shared.Models.Notifications;
@@ -64,6 +65,17 @@ public class NotificationChannelsTests
Assert.Empty(slack.GetCustomHeaders());
}
[Fact]
public void GetSigningSecret_OnlyForGenericJson()
{
var secret = "0123456789abcdef";
Assert.Equal(secret, new NotificationChannel { Id = "g", ProviderType = 20, SigningSecret = secret }.GetSigningSecret());
Assert.Null(new NotificationChannel { Id = "s", ProviderType = 10, SigningSecret = secret }.GetSigningSecret());
Assert.Null(new NotificationChannel { Id = "d", ProviderType = 30, SigningSecret = secret }.GetSigningSecret());
Assert.Null(new NotificationChannel { Id = "w", ProviderType = 20, SigningSecret = " " }.GetSigningSecret());
Assert.Null(new NotificationChannel { Id = "n", ProviderType = 20 }.GetSigningSecret());
}
// ── Legacy synthesis (GetNotificationChannels) ────────────────────────
[Fact]
@@ -141,7 +153,7 @@ public class NotificationChannelsTests
var cfg = new TenantConfiguration
{
NotificationChannelsJson = "[" +
ChannelJson("ch-1", extra: ",\"customHeadersJson\":\"{\\\"X-Api-Key\\\":\\\"k\\\"}\"") + "," +
ChannelJson("ch-1", extra: ",\"customHeadersJson\":\"{\\\"X-Api-Key\\\":\\\"k\\\"}\",\"signingSecret\":\"0123456789abcdef\"") + "," +
ChannelJson("ch-2", providerType: 10, url: "https://slack.example/h") + "]",
};
@@ -151,8 +163,10 @@ public class NotificationChannelsTests
Assert.Equal("ch-1", redacted[0].Id); // structure visible
Assert.Equal(Constants.RedactedSecretPlaceholder, redacted[0].Url); // secrets masked
Assert.Equal(Constants.RedactedSecretPlaceholder, redacted[0].CustomHeadersJson);
Assert.Equal(Constants.RedactedSecretPlaceholder, redacted[0].SigningSecret);
Assert.Equal(Constants.RedactedSecretPlaceholder, redacted[1].Url);
Assert.Null(redacted[1].CustomHeadersJson); // unset stays unset
Assert.Null(redacted[1].SigningSecret);
}
[Fact]
@@ -169,7 +183,7 @@ public class NotificationChannelsTests
{
NotificationChannelsJson = "[" +
ChannelJson("ch-1", url: "https://real.example/hook",
extra: ",\"customHeadersJson\":\"{\\\"X-Api-Key\\\":\\\"real\\\"}\"") + "]",
extra: ",\"customHeadersJson\":\"{\\\"X-Api-Key\\\":\\\"real\\\"}\",\"signingSecret\":\"real-secret-0123\"") + "]",
};
// Round-trip the redacted view back onto a save (defense-in-depth path).
@@ -179,6 +193,7 @@ public class NotificationChannelsTests
var ch = Assert.Single(NotificationChannel.ParseList(incoming.NotificationChannelsJson));
Assert.Equal("https://real.example/hook", ch.Url);
Assert.Equal("{\"X-Api-Key\":\"real\"}", ch.CustomHeadersJson);
Assert.Equal("real-secret-0123", ch.SigningSecret);
}
[Fact]
@@ -244,4 +259,40 @@ public class NotificationChannelsTests
Assert.NotNull(error);
Assert.Contains("headers", error);
}
// ── Signing-secret validation ─────────────────────────────────────────
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
[InlineData("0123456789abcdef")] // 16 chars (minimum)
public void ValidateWebhookSigningSecret_AcceptsAbsentOrValid(string? secret)
{
Assert.Null(TenantConfigValidation.ValidateWebhookSigningSecret(secret));
// 128 chars (maximum) and a typical generated 64-char hex secret
Assert.Null(TenantConfigValidation.ValidateWebhookSigningSecret(new string('a', 128)));
Assert.Null(TenantConfigValidation.ValidateWebhookSigningSecret(new string('f', 64)));
}
[Theory]
[InlineData("short-15-chars.")] // below minimum
[InlineData("has a space in the middle!")] // whitespace
[InlineData("line\nbreak-0123456789")] // control char
[InlineData("umlauts-äöü-0123456789")] // non-ASCII
[InlineData("***REDACTED***")] // unrestored sentinel (too short anyway)
public void ValidateWebhookSigningSecret_RejectsInvalid(string secret)
{
Assert.NotNull(TenantConfigValidation.ValidateWebhookSigningSecret(secret));
Assert.NotNull(TenantConfigValidation.ValidateWebhookSigningSecret(new string('a', 129)));
}
[Fact]
public void ValidateNotificationChannels_RejectsBadPerChannelSigningSecret()
{
var json = "[" + ChannelJson("ch-1", extra: ",\"signingSecret\":\"too-short\"") + "]";
var error = UpdateTenantConfigurationFunction.ValidateNotificationChannels(json);
Assert.NotNull(error);
Assert.Contains("signing secret", error);
}
}
@@ -7,7 +7,7 @@ namespace AutopilotMonitor.Functions.Tests.Offboarding;
/// <summary>
/// Test double for <see cref="IOffboardFarewellEmailSender"/>. Records every invocation so
/// the handler-side tests can assert "did the farewell send fire after History → Completed?"
/// without booting the real Resend client. Default = silent noop; set <see cref="ThrowOnSend"/>
/// without booting the real mail provider client. Default = silent noop; set <see cref="ThrowOnSend"/>
/// to simulate downstream failures and prove the handler's fail-soft contract.
/// </summary>
internal sealed class FakeOffboardFarewellEmailSender : IOffboardFarewellEmailSender
@@ -221,7 +221,7 @@ public class TenantOffboardingHandlerOrderingTests
public async Task PostDrain_FarewellEmail_Skipped_WhenNotificationEmailNull()
{
// Tenants that never set a preview-notification email get null capture; the handler
// skip-gates the send so no Resend call is even attempted.
// skip-gates the send so no provider call is even attempted.
var harness = Harness.New();
Assert.Null(harness.Repo.History[HistoryRowKey].NotificationEmail);
@@ -235,7 +235,7 @@ public class TenantOffboardingHandlerOrderingTests
public async Task PostDrain_FarewellEmail_Skipped_WhenNotificationEmailWhitespace()
{
// Defensive: legacy History rows may have a non-null empty/whitespace value. Treat
// these as "no email captured" so we never call Resend with a useless string.
// these as "no email captured" so we never call the mail provider with a useless string.
var harness = Harness.New();
harness.Repo.History[HistoryRowKey].NotificationEmail = " ";
@@ -248,11 +248,11 @@ public class TenantOffboardingHandlerOrderingTests
[Fact]
public async Task PostDrain_FarewellEmail_SenderThrows_HandlerSurvives_HistoryStillCompleted()
{
// The send is best-effort. A Resend outage / template bug / serializer crash must
// The send is best-effort. A mail-provider outage / template bug / serializer crash must
// NOT propagate to the queue worker — that would re-poison a Completed offboard.
var harness = Harness.New();
harness.Repo.History[HistoryRowKey].NotificationEmail = "ops@contoso.invalid";
harness.FarewellEmail.ThrowOnSend = new InvalidOperationException("simulated Resend outage");
harness.FarewellEmail.ThrowOnSend = new InvalidOperationException("simulated mail-provider outage");
await harness.Sut.HandleAsync(harness.Envelope());
@@ -112,9 +112,11 @@ public class OffboardingSessionEnumeratorTests
public Task<List<SessionSummary>> GetSessionsOlderThanAsync(string tenantId, DateTime cutoffDate, int maxResults = int.MaxValue, bool excludeInFlightDeletions = false) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetSessionsByDateRangeAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetUsageWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetMaintenanceWindowSessionsAsync(DateTime startDate, DateTime endDate) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetStalledSessionsAsync(string tenantId, DateTime cutoffTime) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff) => throw new NotImplementedException();
public Task<List<string>> GetAllTenantIdsAsync() => throw new NotImplementedException();
@@ -0,0 +1,145 @@
using AutopilotMonitor.Functions.Functions.Config;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Tests for <see cref="PlanManagementFunction.ApplyPlanChanges"/> — the pure mutation core
/// of the PATCH plan endpoint. Pins the retention-grace anchor lifecycle: an EFFECTIVE
/// Pro→Community transition stamps <see cref="TenantConfiguration.ProDowngradedUtc"/>, any
/// effectively-Pro outcome clears it, and a planTier downgrade under an active trial does
/// NOT stamp (the edition is still Pro — the later trial expiry is the anchor then).
/// </summary>
public class PlanManagementTransitionTests
{
private static readonly DateTime Now = new(2026, 8, 22, 12, 0, 0, DateTimeKind.Utc);
private const string Caller = "ga@operator.example";
private static (TenantEdition Before, TenantEdition After, Dictionary<string, string> Changes) Apply(
TenantConfiguration config, string? planTier = null, bool trialProvided = false, DateTime? trialExpiresUtc = null)
{
var changes = new Dictionary<string, string>();
var (before, after) = PlanManagementFunction.ApplyPlanChanges(
config, planTier, trialProvided, trialExpiresUtc, Caller, Now, changes);
return (before, after, changes);
}
[Fact]
public void Downgrade_ProToCommunity_StampsAnchorAndRecordsChange()
{
var config = new TenantConfiguration { TenantId = "t1", PlanTier = "pro" };
var (before, after, changes) = Apply(config, planTier: "community");
Assert.Equal(TenantEdition.Pro, before);
Assert.Equal(TenantEdition.Community, after);
Assert.Equal(Now, config.ProDowngradedUtc);
Assert.True(changes.ContainsKey("ProDowngradedUtc"));
Assert.True(changes.ContainsKey("PlanTier"));
}
[Fact]
public void Upgrade_CommunityToPro_ClearsAnchor()
{
var config = new TenantConfiguration
{
TenantId = "t1",
PlanTier = "community",
ProDowngradedUtc = Now.AddDays(-10)
};
var (_, after, changes) = Apply(config, planTier: "pro");
Assert.Equal(TenantEdition.Pro, after);
Assert.Null(config.ProDowngradedUtc);
Assert.True(changes.ContainsKey("ProDowngradedUtc"));
}
[Fact]
public void Downgrade_WithActiveTrial_DoesNotStamp_EditionStaysPro()
{
// planTier drops but the trial keeps the EFFECTIVE edition Pro: no anchor now —
// the trial's own expiry timestamp anchors the grace later, read-time.
var config = new TenantConfiguration
{
TenantId = "t1",
PlanTier = "pro",
TrialExpiresUtc = Now.AddDays(5)
};
var (before, after, changes) = Apply(config, planTier: "community");
Assert.Equal(TenantEdition.Pro, before);
Assert.Equal(TenantEdition.Pro, after);
Assert.Null(config.ProDowngradedUtc);
Assert.False(changes.ContainsKey("ProDowngradedUtc"));
}
[Fact]
public void EndingTrialExplicitly_StampsAnchor()
{
// GA sets trialExpiresUtc: null on a trial-Pro tenant — TrialExpiresUtc is gone, so
// WITHOUT the explicit stamp the grace would have no anchor at all.
var config = new TenantConfiguration
{
TenantId = "t1",
PlanTier = "free",
TrialExpiresUtc = Now.AddDays(5)
};
var (before, after, _) = Apply(config, trialProvided: true, trialExpiresUtc: null);
Assert.Equal(TenantEdition.Pro, before);
Assert.Equal(TenantEdition.Community, after);
Assert.Equal(Now, config.ProDowngradedUtc);
Assert.Null(config.TrialExpiresUtc);
}
[Fact]
public void GrantingTrial_ToDowngradedTenant_ClearsAnchor()
{
var config = new TenantConfiguration
{
TenantId = "t1",
PlanTier = "community",
ProDowngradedUtc = Now.AddDays(-10)
};
var (_, after, _) = Apply(config, trialProvided: true, trialExpiresUtc: Now.AddDays(30));
Assert.Equal(TenantEdition.Pro, after);
Assert.Null(config.ProDowngradedUtc);
Assert.Equal(Caller, config.TrialGrantedBy);
}
[Fact]
public void NoOpPatch_SameTier_RecordsNothing()
{
var config = new TenantConfiguration { TenantId = "t1", PlanTier = "community" };
var (before, after, changes) = Apply(config, planTier: "community");
Assert.Equal(TenantEdition.Community, before);
Assert.Equal(TenantEdition.Community, after);
Assert.Empty(changes);
Assert.Null(config.ProDowngradedUtc);
}
[Fact]
public void RepeatedDowngradeAfterReUpgrade_RestampsWithNewTimestamp()
{
var config = new TenantConfiguration { TenantId = "t1", PlanTier = "pro" };
Apply(config, planTier: "community");
Assert.Equal(Now, config.ProDowngradedUtc);
Apply(config, planTier: "pro");
Assert.Null(config.ProDowngradedUtc);
var changes = new Dictionary<string, string>();
var later = Now.AddDays(40);
PlanManagementFunction.ApplyPlanChanges(config, "community", false, null, Caller, later, changes);
Assert.Equal(later, config.ProDowngradedUtc);
}
}
@@ -0,0 +1,76 @@
using System.Collections.Concurrent;
using AutopilotMonitor.Functions.Services;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Both sliding-window limiters keep the request history and its lock in one cache entry.
/// Previously the entry was re-Set without its eviction callback, which (a) leaked per-key lock
/// objects and (b) let two threads mutate the same List under different locks. These tests hammer
/// one key from many threads: the count of allowed requests must be exactly the limit and nothing
/// may throw (a List mutated concurrently throws InvalidOperationException or corrupts Count).
/// </summary>
public class RateLimitServiceConcurrencyTests
{
private const int Threads = 16;
private const int RequestsPerThread = 500;
[Fact]
public void CheckRateLimit_ManyThreadsSameKey_AllowsExactlyTheLimit()
{
var sut = new RateLimitService(new MemoryCache(new MemoryCacheOptions()), NullLogger<RateLimitService>.Instance);
const int limit = 1000;
var allowed = 0;
var errors = new ConcurrentBag<Exception>();
RunInParallel(() =>
{
try
{
if (sut.CheckRateLimit("thumb-shared", limit).IsAllowed)
Interlocked.Increment(ref allowed);
}
catch (Exception ex) { errors.Add(ex); }
});
Assert.Empty(errors);
Assert.Equal(limit, allowed);
Assert.False(sut.CheckRateLimit("thumb-shared", limit).IsAllowed);
}
[Fact]
public void DistressCheck_ManyThreadsSameIp_AllowsExactlyTheIpLimit()
{
var sut = new DistressRateLimitService(new MemoryCache(new MemoryCacheOptions()), NullLogger<DistressRateLimitService>.Instance);
var allowed = 0;
var errors = new ConcurrentBag<Exception>();
RunInParallel(() =>
{
try
{
if (sut.Check("10.0.0.9", "a1b2c3d4-e5f6-7890-abcd-ef1234567890").IsAllowed)
Interlocked.Increment(ref allowed);
}
catch (Exception ex) { errors.Add(ex); }
});
Assert.Empty(errors);
Assert.Equal(5, allowed); // MaxPerIp
}
private static void RunInParallel(Action body)
{
using var gate = new ManualResetEventSlim(false);
var tasks = Enumerable.Range(0, Threads).Select(_ => Task.Run(() =>
{
gate.Wait();
for (var i = 0; i < RequestsPerThread; i++) body();
})).ToArray();
gate.Set();
Task.WaitAll(tasks);
}
}
@@ -1,134 +0,0 @@
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Coverage for the post-offboarding farewell-email send path. The actual Resend HTTP
/// call is not exercised here — every test drives one of the no-op gates. Pins:
/// <list type="bullet">
/// <item><see cref="ResendEmailService.SendAsync"/> never throws — not when the API key
/// is empty and not when the recipient is missing. The handler's fail-soft
/// try/catch is a belt; this is the suspenders.</item>
/// <item>Gate ordering: missing-key short-circuits before missing-recipient.</item>
/// <item>The farewell template is final: no [DRAFT]/placeholder residue, domain is
/// interpolated, empty domain falls back to "your organization".</item>
/// </list>
/// </summary>
public sealed class ResendEmailServiceTests
{
[Fact]
public async Task SendAsync_EmptyApiKey_NoOps_AndLogsMissingKeyDebugLine()
{
var logger = new CapturingLogger<ResendEmailService>();
var sut = Build(apiKey: "", logger);
await sut.SendAsync(
toEmail: "ops@contoso.invalid",
domainName: "contoso.invalid",
tenantId: "88888888-8888-8888-8888-888888888888",
ct: CancellationToken.None);
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Debug && e.Message.Contains("RESEND_API_KEY not configured"));
}
[Fact]
public async Task SendAsync_EmptyRecipient_NoOps_AndLogsMissingEmailDebugLine()
{
var logger = new CapturingLogger<ResendEmailService>();
var sut = Build(apiKey: "any-non-empty-key", logger);
await sut.SendAsync(
toEmail: "",
domainName: "contoso.invalid",
tenantId: "88888888-8888-8888-8888-888888888888");
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Debug && e.Message.Contains("No notification email captured"));
}
[Fact]
public async Task SendAsync_EmptyApiKey_ShortCircuitsBeforeRecipientCheck()
{
// Pins the gate ordering: missing-key > missing-email. Both are empty here; only
// the key line must show up.
var logger = new CapturingLogger<ResendEmailService>();
var sut = Build(apiKey: "", logger);
await sut.SendAsync(
toEmail: "",
domainName: "contoso.invalid",
tenantId: "88888888-8888-8888-8888-888888888888");
Assert.Contains(logger.Entries,
e => e.Level == LogLevel.Debug && e.Message.Contains("RESEND_API_KEY not configured"));
Assert.DoesNotContain(logger.Entries,
e => e.Message.Contains("No notification email captured"));
}
[Fact]
public void FarewellTemplate_IsFinal_NoDraftResidue()
{
Assert.DoesNotContain("DRAFT", EmailTemplates.OffboardingFarewellSubject);
var html = EmailTemplates.GetOffboardingFarewellHtml("contoso.invalid");
Assert.DoesNotContain("DRAFT", html);
Assert.DoesNotContain("TEMPLATE NOT FINALISED", html);
Assert.DoesNotContain("TODO", html);
}
[Fact]
public void FarewellTemplate_InterpolatesDomain_AndLinksFeedbackChannels()
{
var html = EmailTemplates.GetOffboardingFarewellHtml("contoso.invalid");
Assert.Contains("contoso.invalid", html);
// The mail is sent from a noreply address after portal access is gone, so the
// feedback pointers must be external channels the recipient can actually reach.
Assert.Contains("https://github.com/okieselbach/AutopilotMonitor/issues", html);
Assert.Contains("linkedin.com", html);
}
[Fact]
public void FarewellTemplate_EmptyDomain_FallsBackToGenericLabel()
{
var html = EmailTemplates.GetOffboardingFarewellHtml("");
Assert.Contains("your organization", html);
}
private static ResendEmailService Build(string apiKey, ILogger<ResendEmailService> logger)
{
var config = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?>
{
["RESEND_API_KEY"] = apiKey,
})
.Build();
return new ResendEmailService(config, logger);
}
private sealed class CapturingLogger<T> : ILogger<T>
{
public readonly List<(LogLevel Level, string Message)> Entries = new();
public IDisposable BeginScope<TState>(TState state) where TState : notnull => NullScope.Instance;
public bool IsEnabled(LogLevel logLevel) => true;
public void Log<TState>(
LogLevel logLevel, EventId eventId, TState state, System.Exception? exception,
System.Func<TState, System.Exception?, string> formatter)
{
Entries.Add((logLevel, formatter(state, exception)));
}
private sealed class NullScope : IDisposable
{
public static readonly NullScope Instance = new();
public void Dispose() { }
}
}
}
@@ -86,6 +86,9 @@ public class SessionIndexFieldManifestTests
["ValidatedBy"] = "AutopilotV1",
["FailureSnapshotJson"] = "{\"x\":1}",
["AdminMarkedAction"] = "failed",
["VerdictPath"] = "sweep:r5_incomplete",
["PriorStatus"] = "Incomplete",
["PriorVerdictPath"] = "maxlife:r6",
["DurationSeconds"] = 360,
["DiagnosticsBlobName"] = "diag-0001.zip",
["DiagnosticsBlobDestination"] = "Hosted",
@@ -331,8 +331,9 @@ public class SessionRetentionFanoutServiceTests
var baseTime = new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
var deadline = baseTime.AddMinutes(50);
var clockCalls = 0;
// Call sequence: tenant-A boundary (1), session s0 (2), session s1 (3), session s2 (4 → past deadline).
var harness = new Harness(utcNow: () => ++clockCalls >= 4 ? deadline.AddMinutes(1) : baseTime);
// Call sequence: tenant-A boundary (1), retention resolution (2), session s0 (3),
// session s1 (4), session s2 (5 → past deadline).
var harness = new Harness(utcNow: () => ++clockCalls >= 5 ? deadline.AddMinutes(1) : baseTime);
var sessions = new List<SessionSummary>();
for (int i = 0; i < 4; i++) sessions.Add(Summary(TenantA, $"s{i}"));
@@ -388,6 +389,38 @@ public class SessionRetentionFanoutServiceTests
};
}
// =========================================== Retention downgrade grace ====
[Fact]
public async Task Tenant_InRetentionGrace_KeepsProCutoff()
{
// Downgraded 10 days ago, stored retention 365: the grace keeps the Pro cap, so the
// cutoff must be now-365d — NOT now-90d (which would enqueue the 90–365d band).
var now = new DateTime(2026, 8, 22, 12, 0, 0, DateTimeKind.Utc);
var harness = new Harness(utcNow: () => now);
harness.WithTenant("t1", retentionDays: 365, sessions: Array.Empty<SessionSummary>(),
planTier: "community", proDowngradedUtc: now.AddDays(-10));
await harness.RunAsync();
harness.MaintenanceRepo.Verify(m => m.GetSessionsOlderThanAsync(
"t1", now.AddDays(-365), It.IsAny<int>(), It.IsAny<bool>()), Times.Once);
}
[Fact]
public async Task Tenant_AfterRetentionGrace_CommunityCutoffApplies()
{
var now = new DateTime(2026, 8, 22, 12, 0, 0, DateTimeKind.Utc);
var harness = new Harness(utcNow: () => now);
harness.WithTenant("t1", retentionDays: 365, sessions: Array.Empty<SessionSummary>(),
planTier: "community", proDowngradedUtc: now.AddDays(-31));
await harness.RunAsync();
harness.MaintenanceRepo.Verify(m => m.GetSessionsOlderThanAsync(
"t1", now.AddDays(-90), It.IsAny<int>(), It.IsAny<bool>()), Times.Once);
}
// ============================================================ Harness ====
private sealed class Harness
@@ -450,11 +483,18 @@ public class SessionRetentionFanoutServiceTests
return result;
}
public void WithTenant(string tenantId, int retentionDays, SessionSummary[] sessions, string planTier = "free")
public void WithTenant(string tenantId, int retentionDays, SessionSummary[] sessions, string planTier = "free",
DateTime? proDowngradedUtc = null)
{
_tenantIds.Add(tenantId);
TenantConfig.Setup(t => t.GetConfigurationAsync(tenantId))
.ReturnsAsync(new TenantConfiguration { TenantId = tenantId, DataRetentionDays = retentionDays, PlanTier = planTier });
.ReturnsAsync(new TenantConfiguration
{
TenantId = tenantId,
DataRetentionDays = retentionDays,
PlanTier = planTier,
ProDowngradedUtc = proDowngradedUtc,
});
MaintenanceRepo.Setup(m => m.GetSessionsOlderThanAsync(tenantId, It.IsAny<DateTime>(), It.IsAny<int>(), It.IsAny<bool>()))
.ReturnsAsync(new List<SessionSummary>(WithTenantId(tenantId, sessions)));
}
@@ -74,7 +74,7 @@ public class SessionStartedAtBackwardShiftGuardTests
var harness = new Harness(SessionRow(), earliestStoredEvent: poisoned);
var ok = await harness.Sut.UpdateSessionStatusAsync(
TenantId, SessionId, SessionStatus.Pending,
TenantId, SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
earliestEventTimestamp: poisoned, latestEventTimestamp: lastEvent);
Assert.True(ok);
@@ -93,7 +93,7 @@ public class SessionStartedAtBackwardShiftGuardTests
var harness = new Harness(SessionRow(), earliestStoredEvent: earliest);
var ok = await harness.Sut.UpdateSessionStatusAsync(
TenantId, SessionId, SessionStatus.Pending,
TenantId, SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
latestEventTimestamp: lastEvent);
Assert.True(ok);
@@ -101,6 +101,52 @@ public class SessionStartedAtBackwardShiftGuardTests
Assert.Equal((int)(lastEvent - earliest).TotalSeconds, harness.Written!.GetInt32("DurationSeconds"));
}
// =========================== system-timeline rows are not session anchors ====
[Fact]
public void IsSessionAnchorEligible_excludes_system_timeline_watcher_rows()
{
Assert.False(TableStorageService.IsSessionAnchorEligible(
new TableEntity("p", "r") { ["Source"] = "SystemTimelineWatcher" }));
Assert.True(TableStorageService.IsSessionAnchorEligible(
new TableEntity("p", "r") { ["Source"] = "ImeLogTracker" }));
Assert.True(TableStorageService.IsSessionAnchorEligible(
new TableEntity("p", "r"))); // legacy row without Source projection
}
[Fact]
public async Task Pending_duration_skips_backfilled_system_timeline_row_and_anchors_on_activity()
{
// Field case d0c5b672: a pre-agent system_clock_changed event (backfilled from the
// Windows event log, original timestamp ~14min before the first real activity) was
// the partition's earliest row and re-anchored StartedAt. The probe must skip
// SystemTimelineWatcher rows and anchor on the first activity event instead —
// even when the watcher row sits well inside the 2h plausibility window.
var clockEventAt = StartedAt.AddMinutes(-90);
var firstActivity = StartedAt.AddMinutes(-5);
var lastEvent = StartedAt.AddMinutes(70);
var harness = new Harness(SessionRow(), eventRows: new[]
{
EventRow(clockEventAt, sequence: 18, source: "SystemTimelineWatcher"),
EventRow(firstActivity, sequence: 2, source: "ImeLogTracker"),
});
var ok = await harness.Sut.UpdateSessionStatusAsync(
TenantId, SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
latestEventTimestamp: lastEvent);
Assert.True(ok);
Assert.NotNull(harness.Written);
Assert.Equal((int)(lastEvent - firstActivity).TotalSeconds, harness.Written!.GetInt32("DurationSeconds"));
}
private static TableEntity EventRow(DateTime occurredUtc, int sequence, string source) =>
new($"{TenantId}_{SessionId}", $"{occurredUtc:yyyyMMddHHmmssfff}_{sequence:D10}")
{
["OccurredUtc"] = new DateTimeOffset(occurredUtc),
["Source"] = source,
};
// ============================================================ Harness ====
private static TableEntity SessionRow()
@@ -126,7 +172,7 @@ public class SessionStartedAtBackwardShiftGuardTests
public TableStorageService Sut { get; }
public TableEntity? Written { get; private set; }
public Harness(TableEntity existing, DateTime? earliestStoredEvent = null)
public Harness(TableEntity existing, DateTime? earliestStoredEvent = null, TableEntity[]? eventRows = null)
{
var sessions = new Mock<TableClient>();
@@ -150,7 +196,7 @@ public class SessionStartedAtBackwardShiftGuardTests
});
var events = new Mock<TableClient>();
var eventRows = earliestStoredEvent.HasValue
var servedRows = eventRows ?? (earliestStoredEvent.HasValue
? new[]
{
new TableEntity($"{TenantId}_{SessionId}", $"{earliestStoredEvent.Value:yyyyMMddHHmmssfff}_0000000001")
@@ -158,12 +204,12 @@ public class SessionStartedAtBackwardShiftGuardTests
["OccurredUtc"] = new DateTimeOffset(earliestStoredEvent.Value),
},
}
: Array.Empty<TableEntity>();
: Array.Empty<TableEntity>());
events.Setup(t => t.QueryAsync<TableEntity>(
It.IsAny<string>(), It.IsAny<int?>(), It.IsAny<IEnumerable<string>>(), It.IsAny<CancellationToken>()))
.Returns(AsyncPageable<TableEntity>.FromPages(new[]
{
Page<TableEntity>.FromValues(eventRows, null, new Mock<Response>().Object),
Page<TableEntity>.FromValues(servedRows, null, new Mock<Response>().Object),
}));
var serviceClient = new Mock<TableServiceClient>();
@@ -0,0 +1,80 @@
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using Azure;
using Azure.Data.Tables;
using Azure.Data.Tables.Models;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.ApplicationInsights.Extensibility;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests
{
public class StartupTelemetryServiceTests
{
private sealed class CapturingChannel : ITelemetryChannel
{
public List<ITelemetry> Items { get; } = new();
public bool? DeveloperMode { get; set; }
public string EndpointAddress { get; set; } = "";
public void Send(ITelemetry item) => Items.Add(item);
public void Flush() { }
public void Dispose() { }
}
private sealed class Lifetime : IHostApplicationLifetime
{
private readonly CancellationTokenSource _started = new();
public CancellationToken ApplicationStarted => _started.Token;
public CancellationToken ApplicationStopping => CancellationToken.None;
public CancellationToken ApplicationStopped => CancellationToken.None;
public void StopApplication() { }
public void FireStarted() => _started.Cancel();
}
[Fact]
public async Task ApplicationStarted_EmitsStartupAndTableInitMetrics()
{
var channel = new CapturingChannel();
var telemetry = new TelemetryClient(new TelemetryConfiguration { TelemetryChannel = channel, ConnectionString = "InstrumentationKey=00000000-0000-0000-0000-000000000000" });
// Table storage: unreadable sentinel → full pass → FullPassRan = true.
var service = new Mock<TableServiceClient>();
var admin = new Mock<TableClient>();
service.Setup(s => s.GetTableClient(It.IsAny<string>())).Returns(admin.Object);
admin.Setup(c => c.GetEntityIfExistsAsync<TableEntity>(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<string>>(), It.IsAny<CancellationToken>()))
.ThrowsAsync(new RequestFailedException(404, "TableNotFound"));
admin.Setup(c => c.UpsertEntityAsync(It.IsAny<TableEntity>(), It.IsAny<TableUpdateMode>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(Mock.Of<Response>());
service.Setup(s => s.CreateTableIfNotExistsAsync(It.IsAny<string>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(Response.FromValue(new TableItem("x"), Mock.Of<Response>()));
var storage = new TableStorageService(service.Object, NullLogger<TableStorageService>.Instance);
await storage.InitializeTablesAsync();
var lifetime = new Lifetime();
var sut = new StartupTelemetryService(lifetime, telemetry, storage, new BackendBuildInfo(), NullLogger<StartupTelemetryService>.Instance);
await sut.StartAsync(CancellationToken.None);
Assert.Empty(channel.Items);
lifetime.FireStarted();
var metrics = channel.Items.ConvertAll(i => (MetricTelemetry)i);
Assert.Equal(2, metrics.Count);
var startup = metrics.Find(m => m.Name == StartupTelemetryService.StartupMetricName)!;
var tableInit = metrics.Find(m => m.Name == StartupTelemetryService.TableInitMetricName)!;
Assert.True(startup.Sum > 0);
Assert.True(tableInit.Sum >= 0);
Assert.Equal("True", startup.Properties["tableInitFullPass"]);
Assert.False(string.IsNullOrEmpty(startup.Properties["version"]));
Assert.True(startup.Sum >= tableInit.Sum);
}
}
}
@@ -48,11 +48,74 @@ public class StorageDependencyFilterProcessorTests
Assert.Empty(next.Received);
}
[Fact]
public void FailedStorageDependency_IsKept()
[Theory]
[InlineData("429")]
[InlineData("500")]
[InlineData("503")]
[InlineData("403")]
[InlineData("")] // unknown outcome (timeout/network) — keep
public void FailedStorageDependency_IsKept(string resultCode)
{
var (processor, next) = Build();
var dep = Dep("myacct.queue.core.windows.net", success: false);
dep.ResultCode = resultCode;
processor.Process(dep);
Assert.Same(dep, Assert.Single(next.Received));
}
// Expected outcomes (point-read miss, ETag precondition, insert conflict) are normal control
// flow — live-verified as 100% of the remaining "failed" storage rows. Dropped.
[Theory]
[InlineData("404")]
[InlineData("412")]
[InlineData("409")]
public void ExpectedStorageOutcome_HttpShape_IsDropped(string resultCode)
{
var (processor, next) = Build();
var dep = Dep("myacct.table.core.windows.net", success: false);
dep.ResultCode = resultCode;
processor.Process(dep);
Assert.Empty(next.Received);
}
[Theory]
[InlineData("Azure.RequestFailedException: The specified resource does not exist.\nRequestId:x\nStatus: 404 (Not Found)\nErrorCode: ResourceNotFound")]
[InlineData("Azure.RequestFailedException: The update condition specified in the request was not satisfied.\nStatus: 412 (Precondition Failed)\nErrorCode: UpdateConditionNotSatisfied")]
[InlineData("Azure.RequestFailedException: The specified entity already exists.\nStatus: 409 (Conflict)\nErrorCode: EntityAlreadyExists")]
public void ExpectedStorageOutcome_InProcShape_IsDropped(string error)
{
// InProc spans have no ResultCode — only the exception text in Properties["Error"].
var (processor, next) = Build();
var dep = Dep(target: "TableClient.GetEntity", success: false, type: "InProc | Microsoft.Tables");
dep.Properties["Error"] = error;
processor.Process(dep);
Assert.Empty(next.Received);
}
[Fact]
public void ThrottledInProcStorageDependency_IsKept()
{
var (processor, next) = Build();
var dep = Dep(target: "TableClient.SubmitTransaction", success: false, type: "InProc | Microsoft.Tables");
dep.Properties["Error"] = "Azure.RequestFailedException: Too many requests.\nStatus: 429 (Too Many Requests)\nErrorCode: ServerBusy";
processor.Process(dep);
Assert.Same(dep, Assert.Single(next.Received));
}
[Fact]
public void SuccessfulSignalRRestCall_IsDropped()
{
var (processor, next) = Build();
processor.Process(Dep("autopilotmonitor-eu.service.signalr.net", success: true, data: "POST /api/hubs/autopilotmonitor/groups/t-x/connections/y", type: "HTTP"));
Assert.Empty(next.Received);
}
[Fact]
public void FailedSignalRRestCall_IsKept()
{
var (processor, next) = Build();
var dep = Dep("autopilotmonitor-eu.service.signalr.net", success: false, type: "HTTP");
dep.ResultCode = "500";
processor.Process(dep);
Assert.Same(dep, Assert.Single(next.Received));
}
@@ -0,0 +1,180 @@
using System;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using Azure;
using Azure.Data.Tables;
using Azure.Data.Tables.Models;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests
{
/// <summary>
/// Startup table initialization is gated by a schema sentinel (hash over TableNames.All)
/// stored in AdminConfiguration: a matching sentinel means zero CreateTableIfNotExists
/// calls on a cold start; anything else runs the full idempotent pass and rewrites it.
/// </summary>
public class TableSchemaSentinelTests
{
private sealed class Harness
{
public Mock<TableServiceClient> Service { get; } = new();
public Mock<TableClient> AdminConfig { get; } = new();
public TableEntity? UpsertedSentinel { get; private set; }
public int CreateCalls;
public Harness()
{
Service.Setup(s => s.GetTableClient(Constants.TableNames.AdminConfiguration)).Returns(AdminConfig.Object);
Service.Setup(s => s.CreateTableIfNotExistsAsync(It.IsAny<string>(), It.IsAny<CancellationToken>()))
.Returns<string, CancellationToken>((_, _) =>
{
Interlocked.Increment(ref CreateCalls);
return Task.FromResult(Response.FromValue(new TableItem("x"), Mock.Of<Response>()));
});
AdminConfig.Setup(c => c.UpsertEntityAsync(It.IsAny<TableEntity>(), It.IsAny<TableUpdateMode>(), It.IsAny<CancellationToken>()))
.Callback<ITableEntity, TableUpdateMode, CancellationToken>((e, _, _) => UpsertedSentinel = (TableEntity)e)
.ReturnsAsync(Mock.Of<Response>());
}
public void SentinelReturns(string? hash)
{
var response = new Mock<NullableResponse<TableEntity>>();
response.SetupGet(r => r.HasValue).Returns(hash != null);
if (hash != null)
{
response.SetupGet(r => r.Value).Returns(new TableEntity(
TableStorageService.SchemaSentinelPartitionKey, TableStorageService.SchemaSentinelRowKey)
{
[TableStorageService.SchemaSentinelHashProperty] = hash
});
}
AdminConfig.Setup(c => c.GetEntityIfExistsAsync<TableEntity>(
TableStorageService.SchemaSentinelPartitionKey, TableStorageService.SchemaSentinelRowKey,
It.IsAny<System.Collections.Generic.IEnumerable<string>>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(response.Object);
}
public void SentinelThrows()
{
AdminConfig.Setup(c => c.GetEntityIfExistsAsync<TableEntity>(
It.IsAny<string>(), It.IsAny<string>(),
It.IsAny<System.Collections.Generic.IEnumerable<string>>(), It.IsAny<CancellationToken>()))
.ThrowsAsync(new RequestFailedException(404, "TableNotFound"));
}
public TableStorageService Build() => new(Service.Object, NullLogger<TableStorageService>.Instance);
}
private static string CurrentHash => TableStorageService.ComputeTableSchemaHash(Constants.TableNames.All);
[Fact]
public void Hash_IsOrderIndependent_AndChangesWhenTableAdded()
{
var a = TableStorageService.ComputeTableSchemaHash(new[] { "B", "A" });
var b = TableStorageService.ComputeTableSchemaHash(new[] { "A", "B" });
var c = TableStorageService.ComputeTableSchemaHash(new[] { "A", "B", "C" });
Assert.Equal(a, b);
Assert.NotEqual(a, c);
Assert.Equal(64, a.Length);
}
[Fact]
public async Task MatchingSentinel_SkipsTableCreation()
{
var h = new Harness();
h.SentinelReturns(CurrentHash);
var fullPass = await h.Build().InitializeTablesAsync();
Assert.False(fullPass);
Assert.Equal(0, h.CreateCalls);
Assert.Null(h.UpsertedSentinel);
}
[Fact]
public async Task StaleSentinel_RunsFullPass_AndRewritesSentinel()
{
var h = new Harness();
h.SentinelReturns("STALE");
var fullPass = await h.Build().InitializeTablesAsync();
Assert.True(fullPass);
Assert.Equal(Constants.TableNames.All.Length, h.CreateCalls);
Assert.NotNull(h.UpsertedSentinel);
Assert.Equal(CurrentHash, h.UpsertedSentinel!.GetString(TableStorageService.SchemaSentinelHashProperty));
}
[Fact]
public async Task MissingSentinel_RunsFullPass()
{
var h = new Harness();
h.SentinelReturns(null);
Assert.True(await h.Build().InitializeTablesAsync());
Assert.Equal(Constants.TableNames.All.Length, h.CreateCalls);
}
[Fact]
public async Task UnreadableSentinel_FreshStorage_RunsFullPass()
{
var h = new Harness();
h.SentinelThrows();
Assert.True(await h.Build().InitializeTablesAsync());
Assert.Equal(Constants.TableNames.All.Length, h.CreateCalls);
Assert.NotNull(h.UpsertedSentinel);
}
[Fact]
public async Task FailedTableCreate_DoesNotWriteSentinel_AndRetriesNextCall()
{
var h = new Harness();
h.SentinelReturns(null);
var failing = Constants.TableNames.All.First();
h.Service.Setup(s => s.CreateTableIfNotExistsAsync(failing, It.IsAny<CancellationToken>()))
.ThrowsAsync(new RequestFailedException(500, "boom"));
var svc = h.Build();
Assert.True(await svc.InitializeTablesAsync());
Assert.Null(h.UpsertedSentinel);
// Not marked initialized → a second call runs the pass again instead of short-circuiting.
var before = h.CreateCalls;
Assert.True(await svc.InitializeTablesAsync());
Assert.True(h.CreateCalls > before);
}
[Fact]
public async Task SecondCallAfterSuccess_IsNoOp()
{
var h = new Harness();
h.SentinelReturns(null);
var svc = h.Build();
await svc.InitializeTablesAsync();
var calls = h.CreateCalls;
Assert.False(await svc.InitializeTablesAsync());
Assert.Equal(calls, h.CreateCalls);
}
[Fact]
public async Task BackfillClaim_WinnerTrue_LoserFalse()
{
var h = new Harness();
h.AdminConfig.SetupSequence(c => c.AddEntityAsync(It.IsAny<TableEntity>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(Mock.Of<Response>())
.ThrowsAsync(new RequestFailedException(409, "EntityAlreadyExists"));
var svc = h.Build();
Assert.True(await svc.TryClaimSessionIndexBackfillAsync());
Assert.False(await svc.TryClaimSessionIndexBackfillAsync());
}
}
}
@@ -21,7 +21,7 @@ public class TenantApprovalServiceTests
private readonly Mock<PreviewWhitelistService> _previewMock;
private readonly Mock<TenantConfigurationService> _tenantConfigMock;
private readonly Mock<ResendEmailService> _resendMock;
private readonly Mock<IEmailService> _emailMock;
private readonly TenantApprovalService _sut;
public TenantApprovalServiceTests()
@@ -37,9 +37,10 @@ public class TenantApprovalServiceTests
configRepo, cache, NullLogger<PreviewWhitelistService>.Instance, _tenantConfigMock.Object)
{ CallBase = false };
_resendMock = new Mock<ResendEmailService>(
Mock.Of<Microsoft.Extensions.Configuration.IConfiguration>(),
NullLogger<ResendEmailService>.Instance);
_emailMock = new Mock<IEmailService>();
_emailMock.Setup(x => x.SendPreviewApprovedEmailAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()))
.Returns(Task.CompletedTask);
_sut = new TenantApprovalService(
NullLogger<TenantApprovalService>.Instance,
@@ -47,7 +48,7 @@ public class TenantApprovalServiceTests
_tenantConfigMock.Object,
new Mock<TenantAdminsService>(
Mock.Of<IAdminRepository>(), cache, NullLogger<TenantAdminsService>.Instance).Object,
_resendMock.Object);
_emailMock.Object);
}
[Fact]
@@ -94,7 +95,7 @@ public class TenantApprovalServiceTests
Assert.False(sent);
_previewMock.Verify(x => x.TryMarkWelcomeEmailSentAsync(It.IsAny<string>()), Times.Never);
_resendMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
_emailMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()), Times.Never);
}
[Fact]
@@ -109,7 +110,7 @@ public class TenantApprovalServiceTests
var sent = await _sut.TrySendWelcomeEmailAsync(TenantId);
Assert.True(sent);
_resendMock.Verify(x => x.SendPreviewApprovedEmailAsync("it@contoso.com", "contoso.com"), Times.Once);
_emailMock.Verify(x => x.SendPreviewApprovedEmailAsync("it@contoso.com", "contoso.com", It.IsAny<CancellationToken>()), Times.Once);
}
[Fact]
@@ -121,7 +122,7 @@ public class TenantApprovalServiceTests
var sent = await _sut.TrySendWelcomeEmailAsync(TenantId);
Assert.False(sent);
_resendMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
_emailMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()), Times.Never);
}
[Fact]
@@ -134,7 +135,7 @@ public class TenantApprovalServiceTests
var sent = await _sut.TrySendWelcomeEmailAsync(TenantId);
Assert.False(sent);
_resendMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
_emailMock.Verify(x => x.SendPreviewApprovedEmailAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<CancellationToken>()), Times.Never);
}
[Fact]
@@ -151,6 +152,6 @@ public class TenantApprovalServiceTests
var newlyApproved = await _sut.ApproveWithSideEffectsAsync(TenantId, ApprovedBy);
Assert.True(newlyApproved);
_resendMock.Verify(x => x.SendPreviewApprovedEmailAsync("it@contoso.com", It.IsAny<string>()), Times.Once);
_emailMock.Verify(x => x.SendPreviewApprovedEmailAsync("it@contoso.com", It.IsAny<string>(), It.IsAny<CancellationToken>()), Times.Once);
}
}
@@ -50,9 +50,7 @@ public class TenantAutoApproveHandlerTests
_tenantConfigMock.Object,
new Mock<TenantAdminsService>(
Mock.Of<IAdminRepository>(), cache, NullLogger<TenantAdminsService>.Instance).Object,
new Mock<ResendEmailService>(
Mock.Of<Microsoft.Extensions.Configuration.IConfiguration>(),
NullLogger<ResendEmailService>.Instance).Object)
Mock.Of<IEmailService>())
{ CallBase = false };
var opsRepo = new Mock<IOpsEventRepository>();
@@ -20,6 +20,7 @@ public class TenantConfigTableSerializationTests
{
var expires = new DateTime(2026, 8, 6, 10, 30, 0, DateTimeKind.Utc);
var started = new DateTime(2026, 7, 7, 10, 30, 0, DateTimeKind.Utc);
var downgraded = new DateTime(2026, 8, 10, 8, 0, 0, DateTimeKind.Utc);
var config = new TenantConfiguration
{
TenantId = TenantId,
@@ -29,7 +30,8 @@ public class TenantConfigTableSerializationTests
TrialExpiresUtc = expires,
TrialStartedUtc = started,
TrialConsumed = true,
TrialGrantedBy = "alice@contoso.com"
TrialGrantedBy = "alice@contoso.com",
ProDowngradedUtc = downgraded
};
var entity = TableConfigRepository.ConvertToTenantTableEntity(config);
@@ -40,6 +42,7 @@ public class TenantConfigTableSerializationTests
Assert.Equal(started, mapped.TrialStartedUtc);
Assert.True(mapped.TrialConsumed);
Assert.Equal("alice@contoso.com", mapped.TrialGrantedBy);
Assert.Equal(downgraded, mapped.ProDowngradedUtc);
}
[Fact]
@@ -59,6 +62,7 @@ public class TenantConfigTableSerializationTests
Assert.Null(mapped.TrialStartedUtc);
Assert.False(mapped.TrialConsumed);
Assert.Null(mapped.TrialGrantedBy);
Assert.Null(mapped.ProDowngradedUtc);
}
[Fact]
@@ -77,6 +81,7 @@ public class TenantConfigTableSerializationTests
Assert.Null(mapped.TrialExpiresUtc);
Assert.False(mapped.TrialConsumed);
Assert.Null(mapped.TrialGrantedBy);
Assert.Null(mapped.ProDowngradedUtc); // pre-grace row: no anchor, no grace
}
[Fact]
@@ -194,8 +194,10 @@ public class TenantEntitlementServiceTests
}
[Fact]
public void GetEffectiveRetentionDays_TrialExpiry_DegradesCapFrom365To90()
public void GetEffectiveRetentionDays_TrialExpiry_ProCapHoldsThroughGrace_ThenDegradesTo90()
{
// Trial expiry is a downgrade: the Pro cap keeps applying for the 30-day retention
// grace (anchor = TrialExpiresUtc, no write involved), THEN the Community cap bites.
var config = new TenantConfiguration
{
TenantId = TenantId,
@@ -204,6 +206,93 @@ public class TenantEntitlementServiceTests
TrialExpiresUtc = Now.AddDays(1)
};
Assert.Equal(365, TenantEntitlementService.GetEffectiveRetentionDays(config, Now));
Assert.Equal(90, TenantEntitlementService.GetEffectiveRetentionDays(config, Now.AddDays(2)));
// 1 day after expiry: inside the grace — still the Pro cap.
Assert.Equal(365, TenantEntitlementService.GetEffectiveRetentionDays(config, Now.AddDays(2)));
// 31 days after expiry: grace over — Community cap enforced.
Assert.Equal(90, TenantEntitlementService.GetEffectiveRetentionDays(config, Now.AddDays(32)));
}
// ── Retention downgrade grace (GetRetentionGraceEndUtc + cap interplay) ──────
[Fact]
public void GetRetentionGraceEndUtc_ProTenant_ReturnsNull()
{
var config = new TenantConfiguration
{
TenantId = TenantId,
PlanTier = "pro",
ProDowngradedUtc = Now.AddDays(-5) // stale leftover must not matter while Pro
};
Assert.Null(TenantEntitlementService.GetRetentionGraceEndUtc(config, Now));
}
[Fact]
public void GetRetentionGraceEndUtc_NeverLostPro_ReturnsNull()
{
var config = new TenantConfiguration { TenantId = TenantId, PlanTier = "free" };
Assert.Null(TenantEntitlementService.GetRetentionGraceEndUtc(config, Now));
}
[Fact]
public void GetRetentionGraceEndUtc_ExplicitDowngrade_AnchorPlusGraceDays()
{
var downgraded = Now.AddDays(-10);
var config = new TenantConfiguration
{
TenantId = TenantId,
PlanTier = "community",
ProDowngradedUtc = downgraded
};
Assert.Equal(
downgraded.AddDays(FeatureEntitlementCatalog.RetentionDowngradeGraceDays),
TenantEntitlementService.GetRetentionGraceEndUtc(config, Now));
}
[Fact]
public void GetRetentionGraceEndUtc_LatestAnchorWins()
{
// Downgrade, re-granted trial, trial expired again: the LATER trial expiry anchors.
var config = new TenantConfiguration
{
TenantId = TenantId,
PlanTier = "community",
ProDowngradedUtc = Now.AddDays(-20),
TrialExpiresUtc = Now.AddDays(-3)
};
Assert.Equal(
Now.AddDays(-3).AddDays(FeatureEntitlementCatalog.RetentionDowngradeGraceDays),
TenantEntitlementService.GetRetentionGraceEndUtc(config, Now));
}
[Fact]
public void GetEffectiveRetentionDays_ExplicitDowngrade_ProCapDuringGrace_CommunityCapAfter()
{
var config = new TenantConfiguration
{
TenantId = TenantId,
PlanTier = "community",
DataRetentionDays = 365,
ProDowngradedUtc = Now
};
Assert.Equal(365, TenantEntitlementService.GetEffectiveRetentionDays(config, Now.AddDays(29)));
Assert.Equal(90, TenantEntitlementService.GetEffectiveRetentionDays(config, Now.AddDays(31)));
}
[Fact]
public void GetEffectiveRetentionDays_GraceDoesNotRaiseAnythingElse()
{
// 0 (infinite) passes through untouched, and a stored value below the Community cap
// is unaffected — the grace only swaps which CAP clamps.
var zero = new TenantConfiguration
{
TenantId = TenantId, PlanTier = "community", DataRetentionDays = 0, ProDowngradedUtc = Now
};
Assert.Equal(0, TenantEntitlementService.GetEffectiveRetentionDays(zero, Now.AddDays(5)));
var below = new TenantConfiguration
{
TenantId = TenantId, PlanTier = "community", DataRetentionDays = 60, ProDowngradedUtc = Now
};
Assert.Equal(60, TenantEntitlementService.GetEffectiveRetentionDays(below, Now.AddDays(5)));
}
}
@@ -20,7 +20,8 @@ public class TrialExpirySweepFunctionTests
private static readonly DateTime Now = new(2026, 7, 7, 3, 30, 0, DateTimeKind.Utc);
private static TenantConfiguration Tenant(
string id, DateTime? trialExpiresUtc, string planTier = "free", string domain = "contoso.com") => new()
string id, DateTime? trialExpiresUtc, string planTier = "free", string domain = "contoso.com",
int retentionDays = 90, DateTime? proDowngradedUtc = null) => new()
{
TenantId = id,
DomainName = domain,
@@ -28,6 +29,8 @@ public class TrialExpirySweepFunctionTests
PlanTier = planTier,
TrialExpiresUtc = trialExpiresUtc,
TrialConsumed = trialExpiresUtc.HasValue,
DataRetentionDays = retentionDays,
ProDowngradedUtc = proDowngradedUtc,
};
private static (TrialExpirySweepFunction Sut, List<OpsEventEntry> Events) Build(params TenantConfiguration[] configs)
@@ -150,6 +153,88 @@ public class TrialExpirySweepFunctionTests
Assert.Equal(2, events.Count);
}
// ── Retention downgrade grace events ─────────────────────────────────────────
[Fact]
public async Task GraceEndingSoon_WithDataAtRisk_EmitsGraceExpiring()
{
// Downgraded 28 days ago with stored retention 365: grace ends in 2 days.
var (sut, events) = Build(Tenant("t1", null, planTier: "community",
retentionDays: 365, proDowngradedUtc: Now.AddDays(-28)));
var result = await sut.RunCoreAsync(CancellationToken.None);
Assert.Equal(1, result.GraceExpiringEmitted);
var e = Assert.Single(events);
Assert.Equal("TenantRetentionGraceExpiring", e.EventType);
Assert.Equal(OpsEventSeverity.Warning, e.Severity);
Assert.Contains("2 day", e.Message);
}
[Fact]
public async Task GraceEndedWithinLookBack_EmitsGraceEnded_OlderStaysSilent()
{
var (sut, events) = Build(
// Grace ended 12h ago → emit once.
Tenant("just-ended", null, planTier: "community",
retentionDays: 365, proDowngradedUtc: Now.AddDays(-30).AddHours(-12)),
// Grace ended 3 days ago → reported by a previous run, silent.
Tenant("long-ended", null, planTier: "community",
retentionDays: 365, proDowngradedUtc: Now.AddDays(-33)));
var result = await sut.RunCoreAsync(CancellationToken.None);
Assert.Equal(1, result.GraceEndedEmitted);
var e = Assert.Single(events);
Assert.Equal("TenantRetentionGraceEnded", e.EventType);
Assert.Equal("just-ended", e.TenantId);
}
[Fact]
public async Task Grace_NoDataAtRisk_StaysSilent()
{
var (sut, events) = Build(
// Stored retention at the Community cap — nothing to lose.
Tenant("at-cap", null, planTier: "community",
retentionDays: 90, proDowngradedUtc: Now.AddDays(-29)),
// 0 = infinite escape hatch — the fanout never deletes, no warning needed.
Tenant("infinite", null, planTier: "community",
retentionDays: 0, proDowngradedUtc: Now.AddDays(-29)));
var result = await sut.RunCoreAsync(CancellationToken.None);
Assert.Equal(0, result.GraceExpiringEmitted);
Assert.Equal(0, result.GraceEndedEmitted);
Assert.Empty(events);
}
[Fact]
public async Task Grace_ReUpgradedTenant_StaysSilent()
{
// Back on Pro: GetRetentionGraceEndUtc returns null even with a stale anchor.
var (sut, events) = Build(Tenant("t1", null, planTier: "pro",
retentionDays: 365, proDowngradedUtc: Now.AddDays(-29)));
var result = await sut.RunCoreAsync(CancellationToken.None);
Assert.Equal(0, result.GraceExpiringEmitted);
Assert.Empty(events);
}
[Fact]
public async Task ExpiredTrial_WithDataAtRisk_GetsBothTrialExpiredAndLaterGraceEvents()
{
// The trial-expiry anchor drives the grace window too: expired 2h ago fires
// TenantTrialExpired now; 28 days later the same tenant enters the grace heads-up.
var (sut, events) = Build(Tenant("t1", Now.AddHours(-2), retentionDays: 365));
var result = await sut.RunCoreAsync(CancellationToken.None);
Assert.Equal(1, result.ExpiredEmitted);
Assert.Equal(0, result.GraceExpiringEmitted); // grace end is ~30d out — beyond heads-up
Assert.Single(events);
}
[Fact]
public async Task ConfigLoadFailure_ReturnsEmptyResult_NeverThrows()
{
@@ -0,0 +1,206 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Verdict-calibration drift radar (docs/backend/verdict-calibration.md): the three finding
/// kinds, the small-n / lift / Wilson gates, re-arm, and the tracker round-trip.
/// </summary>
public class VerdictCalibrationRadarTests
{
private static readonly DateTime Target = new(2026, 8, 22, 0, 0, 0, DateTimeKind.Utc);
/// <summary>35 daily rows ending on Target; per-day buckets chosen by window/baseline membership.</summary>
private static List<VerdictCalibrationDailyAggregate> Horizon(
Func<bool, (string Path, string Status, int Count)[]> bucketsFor, int sessionsPerDay = 20)
{
var rows = new List<VerdictCalibrationDailyAggregate>();
for (var i = 34; i >= 0; i--)
{
var inWindow = i < 7;
var date = Target.AddDays(-i);
rows.Add(new VerdictCalibrationDailyAggregate
{
TenantId = "t", Date = date.ToString("yyyy-MM-dd"), SessionCount = sessionsPerDay, TerminalSessionCount = sessionsPerDay,
Buckets = bucketsFor(inWindow).Select(b => new VerdictCalibrationBucket { VerdictPath = b.Path, Status = b.Status, Count = b.Count }).ToList(),
});
}
return rows;
}
[Fact]
public void Share_regression_fires_on_a_doubled_separated_share()
{
// baseline: 1/20 per day (5%), window: 4/20 per day (20%) → lift 4, clearly separated
var rows = Horizon(w => new[] { ("sweep:r5_incomplete", "Incomplete", w ? 4 : 1), ("agent:complete", "Succeeded", w ? 16 : 19) });
var findings = VerdictCalibrationRadar.Evaluate(rows, Target);
var f = Assert.Single(findings, x => x.Kind == VerdictCalibrationAlertKinds.ShareRegression);
Assert.Equal("sweep:r5_incomplete", f.VerdictPath);
Assert.Equal("Incomplete", f.Status);
Assert.Equal(28, f.WindowHitCount);
Assert.Equal(140, f.WindowSessionCount);
Assert.Equal(28, f.BaselineHitCount);
Assert.Equal(560, f.BaselineSessionCount);
Assert.Equal(20.0, f.WindowRatePct);
Assert.Equal(5.0, f.BaselineRatePct);
Assert.Equal(4.0, f.Lift);
Assert.Equal("2026-08-16", f.WindowStartDate);
Assert.Equal("2026-08-22", f.WindowEndDate);
// sweep:* is also the silence group → silence share doubled as well
Assert.Contains(findings, x => x.Kind == VerdictCalibrationAlertKinds.SilenceShareRegression && x.VerdictPath == VerdictCalibrationRadar.SilenceGroupPath);
}
[Fact]
public void Share_regression_is_suppressed_below_lift_two_or_without_separation()
{
// 5% → 8%: lift 1.6 — under the gate
var mild = Horizon(w => new[] { ("agent:failed", "Failed", w ? 2 : 1), ("agent:complete", "Succeeded", w ? 18 : 19) }, sessionsPerDay: 20);
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(mild, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
// tiny n: 1/2 per day in window vs 0/2 baseline — lift infinite but Wilson intervals overlap / hits < 5
var tiny = Horizon(w => new[] { ("agent:failed", "Failed", w ? 1 : 0), ("agent:complete", "Succeeded", w ? 1 : 2) }, sessionsPerDay: 2);
Assert.Empty(VerdictCalibrationRadar.Evaluate(tiny, Target));
}
[Fact]
public void Agent_and_registration_paths_never_share_alert_but_still_feed_the_silence_group()
{
// Workflow mix, not classifier drift: a WhiteGlove rollout week doubles the pending share.
var wg = Horizon(w => new[] { ("agent:whiteglove_pending", "Pending", w ? 6 : 1), ("agent:complete", "Succeeded", w ? 14 : 19) });
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(wg, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
var reg = Horizon(w => new[] { ("register:superseded", "Incomplete", w ? 6 : 1), ("agent:complete", "Succeeded", w ? 14 : 19) });
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(reg, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
// Backend-decided paths stay eligible.
Assert.True(VerdictCalibrationRadar.IsShareRegressionEligible("sweep:r6"));
Assert.True(VerdictCalibrationRadar.IsShareRegressionEligible("rule:ANALYZE-ESP-001"));
Assert.True(VerdictCalibrationRadar.IsShareRegressionEligible("manual:failed"));
Assert.False(VerdictCalibrationRadar.IsShareRegressionEligible("agent:complete_soft"));
// An episode opened by the earlier, broader gate re-arms on the next pass regardless of numbers.
var stale = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "agent:whiteglove_pending", Status = "Pending" };
Assert.True(VerdictCalibrationRadar.ShouldReArm(stale, wg, Target));
}
[Fact]
public void Derived_legacy_paths_never_alert()
{
var rows = Horizon(w => new[] { ("legacy:r6", "Incomplete", w ? 10 : 1), ("agent:complete", "Succeeded", w ? 10 : 19) });
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(rows, Target), f => f.Kind == VerdictCalibrationAlertKinds.ShareRegression);
}
[Fact]
public void Evidence_gap_fires_on_r6_share_and_ignores_non_rule_sweep_paths()
{
// classifier verdicts in window: r6 5/day + r5 15/day = 140; r6 share 25% → fires. sweep:stalled is not a rule verdict.
var rows = Horizon(w => new[]
{
("sweep:r6", "Incomplete", 5), ("sweep:r5_incomplete", "Incomplete", 15), ("sweep:stalled", "Stalled", 30),
}, sessionsPerDay: 50);
var f = Assert.Single(VerdictCalibrationRadar.Evaluate(rows, Target), x => x.Kind == VerdictCalibrationAlertKinds.EvidenceGap);
Assert.Equal(VerdictCalibrationRadar.EvidenceGapPath, f.VerdictPath);
Assert.Equal(35, f.WindowHitCount);
Assert.Equal(140, f.WindowSessionCount);
Assert.Equal(25.0, f.WindowRatePct);
Assert.Null(f.Lift);
}
[Fact]
public void Evidence_gap_needs_twenty_classifier_verdicts()
{
var rows = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 2 : 0), ("agent:complete", "Succeeded", 18) });
Assert.DoesNotContain(VerdictCalibrationRadar.Evaluate(rows, Target), f => f.Kind == VerdictCalibrationAlertKinds.EvidenceGap);
}
[Fact]
public void ReArm_when_share_falls_under_one_point_five_or_path_stops()
{
var alert = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "sweep:r6", Status = "Incomplete" };
var stillHigh = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 4 : 1), ("agent:complete", "Succeeded", w ? 16 : 19) });
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, stillHigh, Target));
var elevatedButUnderGate = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 2 : 1), ("agent:complete", "Succeeded", w ? 18 : 19) }, sessionsPerDay: 20);
// 10% vs 5% = 2.0× — still at/above 1.5× → keep
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, elevatedButUnderGate, Target));
var back = Horizon(w => new[] { ("sweep:r6", "Incomplete", 1), ("agent:complete", "Succeeded", 19) });
Assert.True(VerdictCalibrationRadar.ShouldReArm(alert, back, Target));
var gone = Horizon(w => new[] { ("sweep:r6", "Incomplete", w ? 0 : 1), ("agent:complete", "Succeeded", w ? 20 : 19) });
Assert.True(VerdictCalibrationRadar.ShouldReArm(alert, gone, Target));
}
[Fact]
public void Evidence_gap_rearms_under_fifteen_percent()
{
var alert = new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.EvidenceGap, VerdictPath = "r6", Status = "*" };
var still = Horizon(w => new[] { ("sweep:r6", "Incomplete", 4), ("sweep:r5_incomplete", "Incomplete", 16) });
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, still, Target)); // 20%
var between = Horizon(w => new[] { ("sweep:r6", "Incomplete", 17), ("sweep:r5_incomplete", "Incomplete", 83) }, sessionsPerDay: 100);
Assert.False(VerdictCalibrationRadar.ShouldReArm(alert, between, Target)); // 17%
var low = Horizon(w => new[] { ("sweep:r6", "Incomplete", 1), ("sweep:r5_incomplete", "Incomplete", 19) });
Assert.True(VerdictCalibrationRadar.ShouldReArm(alert, low, Target)); // 5%
}
[Fact]
public void CurrentSums_follow_the_alert_kind()
{
var rows = Horizon(w => new[] { ("sweep:r6", "Incomplete", 2), ("maxlife:r3", "Incomplete", 1), ("agent:complete", "Succeeded", 17) });
var share = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.ShareRegression, VerdictPath = "sweep:r6", Status = "Incomplete" }, rows, Target);
Assert.Equal((14, 140, 56, 560), share);
var silence = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.SilenceShareRegression }, rows, Target);
Assert.Equal((21, 140, 84, 560), silence);
var gap = VerdictCalibrationRadar.CurrentSums(new VerdictCalibrationAlert { Kind = VerdictCalibrationAlertKinds.EvidenceGap }, rows, Target);
Assert.Equal((14, 21, 56, 84), gap);
}
[Fact]
public void Rows_outside_the_horizon_are_ignored()
{
var rows = Horizon(w => new[] { ("agent:complete", "Succeeded", 20) });
rows.Add(new VerdictCalibrationDailyAggregate { TenantId = "t", Date = "2026-08-23", SessionCount = 1000, Buckets = { new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = 1000 } } });
rows.Add(new VerdictCalibrationDailyAggregate { TenantId = "t", Date = "2026-07-01", SessionCount = 1000, Buckets = { new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = 1000 } } });
Assert.Empty(VerdictCalibrationRadar.Evaluate(rows, Target));
}
// ---- tracker round-trip ----
[Fact]
public void Alert_round_trips_through_the_tracker_entity()
{
var alert = new VerdictCalibrationAlert
{
TenantId = "11111111-1111-1111-1111-111111111111", Kind = VerdictCalibrationAlertKinds.ShareRegression,
VerdictPath = "rule:ANALYZE-ESP-001", Status = "Failed",
WindowHitCount = 7, WindowSessionCount = 40, BaselineHitCount = 3, BaselineSessionCount = 160,
WindowRatePct = 17.5, BaselineRatePct = 1.9, Lift = 9.3, WindowStartDate = "2026-08-16", WindowEndDate = "2026-08-22",
Dimension = new RuleRegressionDimension { Dimension = "osBuild", Value = "26200.1234", HitCount = 6, HitSharePct = 85.7, AllSharePct = 20.0, Lift = 4.3 },
FirstNotifiedAt = new DateTime(2026, 8, 20, 6, 0, 0, DateTimeKind.Utc), LastEvaluatedAt = new DateTime(2026, 8, 23, 6, 0, 0, DateTimeKind.Utc),
};
var entity = TableHardwareRejectionNotificationTracker.BuildVerdictCalibrationEntity(alert.TenantId, alert);
Assert.Equal("verdictcalibration|share_regression|rule:analyze-esp-001|failed", entity.RowKey);
Assert.Equal(alert.TenantId, entity.PartitionKey);
var back = TableHardwareRejectionNotificationTracker.MapToVerdictCalibrationAlert(entity);
Assert.Equal(JsonSerializer.Serialize(alert), JsonSerializer.Serialize(back));
// Tri-states: no lift / no dimension → absent columns → null on read, never 0 / a guessed dimension.
var bare = new VerdictCalibrationAlert
{
TenantId = alert.TenantId, Kind = VerdictCalibrationAlertKinds.EvidenceGap, VerdictPath = "r6", Status = "*",
FirstNotifiedAt = alert.FirstNotifiedAt, LastEvaluatedAt = alert.LastEvaluatedAt,
};
var bareBack = TableHardwareRejectionNotificationTracker.MapToVerdictCalibrationAlert(
TableHardwareRejectionNotificationTracker.BuildVerdictCalibrationEntity(bare.TenantId, bare));
Assert.Null(bareBack.Lift);
Assert.Null(bareBack.Dimension);
Assert.Equal("verdictcalibration|evidence_gap|r6|*", TableHardwareRejectionNotificationTracker.BuildVerdictCalibrationRowKey(bare.Kind, bare.VerdictPath, bare.Status));
}
}
@@ -0,0 +1,301 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Text.Json;
using AutopilotMonitor.Functions.Functions.Metrics;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Verdict calibration aggregate (docs/backend/verdict-calibration.md): the pure bucketing core,
/// the persistence round-trip and the matrix arithmetic of the read endpoint.
/// </summary>
public class VerdictCalibrationTests
{
private const string TenantA = "11111111-1111-1111-1111-111111111111";
private const string TenantB = "22222222-2222-2222-2222-222222222222";
private static readonly DateTime Now = new(2026, 8, 23, 12, 0, 0, DateTimeKind.Utc);
private static SessionSummary Session(string tenant, string id, SessionStatus status, DateTime started,
string? path = null, string? priorPath = null, string? priorStatus = null, string? adminMarked = null,
DateTime? completed = null, string serial = "SN-1", string? failureReason = null, string? reconcileReason = null) => new()
{
TenantId = tenant, SessionId = id, SerialNumber = serial, Status = status, StartedAt = started,
CompletedAt = completed, VerdictPath = path, PriorVerdictPath = priorPath, PriorStatus = priorStatus,
AdminMarkedAction = adminMarked, FailureReason = failureReason ?? string.Empty,
ReconcileReason = reconcileReason ?? string.Empty, FailureSource = string.Empty,
};
private static DeviceHistory History(string tenant, string serial, params (string Id, DateTime Started, DateTime? Completed, string Status)[] refs) => new()
{
TenantId = tenant, SerialKey = serial.ToLowerInvariant(), SerialNumber = serial,
Chain = refs.Select(r => new DeviceSessionRef { SessionId = r.Id, StartedAt = r.Started, CompletedAt = r.Completed, Status = r.Status }).ToList(),
};
private static VerdictCalibrationBucket Bucket(VerdictCalibrationDailyAggregate row, string path, string status)
=> Assert.Single(row.Buckets, b => b.VerdictPath == path && b.Status == status);
// ---- BuildVerdictCalibrationAggregates ----
[Fact]
public void Buckets_by_tenant_date_path_and_status_with_global_mirror()
{
var d1 = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
var sessions = new List<SessionSummary>
{
Session(TenantA, "a1", SessionStatus.Succeeded, d1, path: "agent:complete", completed: d1.AddHours(1)),
Session(TenantA, "a2", SessionStatus.Succeeded, d1.AddHours(2), path: "agent:complete", completed: d1.AddHours(3)),
Session(TenantA, "a3", SessionStatus.Incomplete, d1, path: "sweep:r6", completed: d1.AddHours(6)),
Session(TenantA, "a4", SessionStatus.Stalled, d1.AddDays(1), path: "sweep:stalled"),
Session(TenantB, "b1", SessionStatus.Failed, d1, path: "agent:failed", completed: d1.AddHours(1)),
};
var rows = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, new Dictionary<string, List<DeviceHistory>>(), Now);
var a10 = Assert.Single(rows, r => r.TenantId == TenantA && r.Date == "2026-08-10");
Assert.Equal(3, a10.SessionCount);
Assert.Equal(3, a10.TerminalSessionCount);
Assert.Equal(2, Bucket(a10, "agent:complete", "Succeeded").Count);
Assert.Equal(1, Bucket(a10, "sweep:r6", "Incomplete").Count);
var a11 = Assert.Single(rows, r => r.TenantId == TenantA && r.Date == "2026-08-11");
Assert.Equal(1, a11.SessionCount);
Assert.Equal(0, a11.TerminalSessionCount); // Stalled is not terminal
Assert.Equal(1, Bucket(a11, "sweep:stalled", "Stalled").Count);
var g10 = Assert.Single(rows, r => r.TenantId == "global" && r.Date == "2026-08-10");
Assert.Equal(4, g10.SessionCount);
Assert.Equal(1, Bucket(g10, "agent:failed", "Failed").Count);
Assert.Equal(VerdictCalibrationDailyAggregate.CurrentVersion, g10.Version);
Assert.Equal(Now, g10.ComputedAt);
// Deterministic ordering: rows by tenant then date, buckets by path then status.
Assert.Equal(rows.Select(r => (r.TenantId, r.Date)).OrderBy(k => k.TenantId, StringComparer.Ordinal).ThenBy(k => k.Date, StringComparer.Ordinal), rows.Select(r => (r.TenantId, r.Date)));
Assert.Equal(a10.Buckets.Select(b => b.VerdictPath).OrderBy(p => p, StringComparer.Ordinal), a10.Buckets.Select(b => b.VerdictPath));
}
[Fact]
public void Unstamped_rows_are_derived_and_counted_as_derived()
{
var d = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
var sessions = new List<SessionSummary>
{
Session(TenantA, "a1", SessionStatus.Succeeded, d, path: null, completed: d.AddHours(1)), // legacy agent completion
Session(TenantA, "a2", SessionStatus.Incomplete, d, path: null, completed: d.AddHours(5),
failureReason: "No Device Setup completion or explicit failure signal observed before timeout"),
Session(TenantA, "a3", SessionStatus.Succeeded, d, path: "agent:complete", completed: d.AddHours(1)),
};
var row = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, new Dictionary<string, List<DeviceHistory>>(), Now)
.Single(r => r.TenantId == TenantA);
var complete = Bucket(row, "agent:complete", "Succeeded");
Assert.Equal(2, complete.Count);
Assert.Equal(1, complete.DerivedCount);
var r6 = Bucket(row, "legacy:r6", "Incomplete");
Assert.Equal(1, r6.Count);
Assert.Equal(1, r6.DerivedCount);
}
[Fact]
public void Overrides_are_attributed_to_the_prior_path()
{
var d = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
var sessions = new List<SessionSummary>
{
// sweep said Incomplete, agent later completed → late completion against sweep:r5_incomplete
Session(TenantA, "a1", SessionStatus.Succeeded, d, path: "agent:complete", priorPath: "sweep:r5_incomplete", priorStatus: "Incomplete", completed: d.AddHours(1)),
// admin flipped an agent success to Failed
Session(TenantA, "a2", SessionStatus.Failed, d, path: "manual:failed", priorPath: "agent:complete", priorStatus: "Succeeded", adminMarked: "Failed", completed: d.AddHours(1)),
// retro reclassification Failed → Incomplete
Session(TenantA, "a3", SessionStatus.Incomplete, d, path: "retro:r6", priorPath: "agent:failed", priorStatus: "Failed", completed: d.AddHours(1)),
// admin mark to Succeeded counts as admin, not late completion
Session(TenantA, "a4", SessionStatus.Succeeded, d, path: "manual:succeeded", priorPath: "sweep:r6", priorStatus: "Incomplete", adminMarked: "Succeeded", completed: d.AddHours(1)),
};
var row = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, new Dictionary<string, List<DeviceHistory>>(), Now)
.Single(r => r.TenantId == TenantA);
var priorSweep = Bucket(row, "sweep:r5_incomplete", "Incomplete");
Assert.Equal(0, priorSweep.Count); // nobody is on it any more
Assert.Equal(1, priorSweep.OverriddenByLateCompletion);
var priorAgent = Bucket(row, "agent:complete", "Succeeded");
Assert.Equal(1, priorAgent.Count); // a1 is on it now
Assert.Equal(1, priorAgent.OverriddenByAdmin); // a2 used to be
Assert.Equal(1, Bucket(row, "agent:failed", "Failed").OverriddenOther);
Assert.Equal(1, Bucket(row, "sweep:r6", "Incomplete").OverriddenByAdmin);
Assert.Equal(0, Bucket(row, "manual:succeeded", "Succeeded").OverriddenByAdmin);
}
[Fact]
public void ReEnrollment_proxy_counts_only_eligible_terminal_sessions()
{
var old = Now.AddDays(-20);
var young = Now.AddDays(-2);
var sessions = new List<SessionSummary>
{
// old, re-enrolled 3 days later → eligible + re-enrolled
Session(TenantA, "a1", SessionStatus.Incomplete, old, path: "sweep:r6", completed: old.AddHours(5), serial: "SN-A"),
// old, next attempt 10 days later → eligible, not re-enrolled
Session(TenantA, "a2", SessionStatus.Failed, old, path: "agent:failed", completed: old.AddHours(1), serial: "SN-B"),
// old, no next attempt → eligible, not re-enrolled
Session(TenantA, "a3", SessionStatus.Succeeded, old, path: "agent:complete", completed: old.AddHours(1), serial: "SN-C"),
// too young to judge → not eligible even though the chain shows a follow-up
Session(TenantA, "a4", SessionStatus.Failed, young, path: "agent:failed", completed: young.AddHours(1), serial: "SN-D"),
// non-terminal → never eligible
Session(TenantA, "a5", SessionStatus.AwaitingUser, old, path: "sweep:r5_awaiting", serial: "SN-E"),
};
var histories = new Dictionary<string, List<DeviceHistory>>
{
[TenantA] = new()
{
History(TenantA, "SN-A", ("a1", old, old.AddHours(5), "Incomplete"), ("x1", old.AddDays(3), old.AddDays(3).AddHours(1), "Succeeded")),
History(TenantA, "SN-B", ("a2", old, old.AddHours(1), "Failed"), ("x2", old.AddDays(10), null, "Succeeded")),
History(TenantA, "SN-C", ("a3", old, old.AddHours(1), "Succeeded")),
History(TenantA, "SN-D", ("a4", young, young.AddHours(1), "Failed"), ("x4", young.AddHours(2), null, "Succeeded")),
},
};
var rows = MaintenanceService.BuildVerdictCalibrationAggregates(sessions, histories, Now);
var oldRow = rows.Single(r => r.TenantId == TenantA && r.Date == old.ToString("yyyy-MM-dd"));
var r6 = Bucket(oldRow, "sweep:r6", "Incomplete");
Assert.Equal((1, 1), (r6.Eligible7d, r6.ReEnrolled7d));
var failed = Bucket(oldRow, "agent:failed", "Failed");
Assert.Equal((1, 0), (failed.Eligible7d, failed.ReEnrolled7d));
var complete = Bucket(oldRow, "agent:complete", "Succeeded");
Assert.Equal((1, 0), (complete.Eligible7d, complete.ReEnrolled7d));
var awaiting = Bucket(oldRow, "sweep:r5_awaiting", "AwaitingUser");
Assert.Equal((0, 0), (awaiting.Eligible7d, awaiting.ReEnrolled7d));
var youngRow = rows.Single(r => r.TenantId == TenantA && r.Date == young.ToString("yyyy-MM-dd"));
var youngFailed = Bucket(youngRow, "agent:failed", "Failed");
Assert.Equal((0, 0), (youngFailed.Eligible7d, youngFailed.ReEnrolled7d));
}
[Fact]
public void Sessions_inside_a_deletion_cascade_are_skipped()
{
var d = new DateTime(2026, 8, 10, 9, 0, 0, DateTimeKind.Utc);
var deleting = Session(TenantA, "a1", SessionStatus.Succeeded, d, path: "agent:complete");
deleting.DeletionState = "Pending";
var rows = MaintenanceService.BuildVerdictCalibrationAggregates(new[] { deleting }, new Dictionary<string, List<DeviceHistory>>(), Now);
Assert.Empty(rows);
}
// ---- Persistence round-trip ----
[Fact]
public void Aggregate_round_trips_through_the_table_entity()
{
var aggregate = new VerdictCalibrationDailyAggregate
{
TenantId = TenantA, Date = "2026-08-10", Version = 1, SessionCount = 5, TerminalSessionCount = 4, ComputedAt = Now,
Buckets =
{
new VerdictCalibrationBucket { VerdictPath = "agent:complete", Status = "Succeeded", Count = 3, DerivedCount = 1, Eligible7d = 3, ReEnrolled7d = 1, OverriddenByAdmin = 1, OverriddenByLateCompletion = 0, OverriddenOther = 2 },
new VerdictCalibrationBucket { VerdictPath = "sweep:r6", Status = "Incomplete", Count = 1 },
},
};
var entity = TableStorageService.BuildVerdictCalibrationAggregateEntity(aggregate);
Assert.Equal(TenantA, entity.PartitionKey);
Assert.Equal("2026-08-10", entity.RowKey);
var back = TableStorageService.MapToVerdictCalibrationAggregate(entity);
Assert.Equal(JsonSerializer.Serialize(aggregate), JsonSerializer.Serialize(back));
}
// ---- Response builder ----
private static VerdictCalibrationDailyAggregate Day(string tenant, DateTime date, int sessions, params (string Path, string Status, int Count)[] buckets) => new()
{
TenantId = tenant, Date = date.ToString("yyyy-MM-dd"), SessionCount = sessions, TerminalSessionCount = sessions, Version = 1, ComputedAt = Now,
Buckets = buckets.Select(b => new VerdictCalibrationBucket { VerdictPath = b.Path, Status = b.Status, Count = b.Count, Eligible7d = b.Count, ReEnrolled7d = b.Count / 5 }).ToList(),
};
// Same naming policy as the worker host (Program.cs: camelCase) so the assertions pin the wire shape.
private static readonly JsonSerializerOptions Wire = new() { PropertyNamingPolicy = JsonNamingPolicy.CamelCase };
private static JsonElement BuildJson(IReadOnlyList<VerdictCalibrationDailyAggregate> daily, DateTime today, int days)
=> JsonSerializer.SerializeToElement(VerdictCalibrationResponse.Build(daily, "global", today, days), Wire);
[Fact]
public void Response_sums_the_window_and_computes_share_rate_and_trend()
{
var today = new DateTime(2026, 8, 23, 0, 0, 0, DateTimeKind.Utc);
var daily = new List<VerdictCalibrationDailyAggregate>();
// 35 days: baseline days carry 10 sessions (1 sweep:r6), the last 7 days carry 10 sessions (3 sweep:r6) → share 10% → 30%, lift 3.0
for (var i = 34; i >= 0; i--)
{
var date = today.AddDays(-i);
var r6 = i < 7 ? 3 : 1;
daily.Add(Day("global", date, 10, ("agent:complete", "Succeeded", 10 - r6), ("sweep:r6", "Incomplete", r6)));
}
var json = BuildJson(daily, today, days: 30);
Assert.True(json.GetProperty("success").GetBoolean());
Assert.Equal(30, json.GetProperty("windowDays").GetInt32());
Assert.Equal(300, json.GetProperty("totals").GetProperty("sessions").GetInt32());
Assert.Equal(30, json.GetProperty("totals").GetProperty("days").GetInt32());
var paths = json.GetProperty("paths").EnumerateArray().ToList();
var r6Row = paths.Single(p => p.GetProperty("verdictPath").GetString() == "sweep:r6");
Assert.Equal(7 * 3 + 23 * 1, r6Row.GetProperty("count").GetInt32());
Assert.Equal(Math.Round(100.0 * 44 / 300, 1), r6Row.GetProperty("sharePct").GetDouble());
Assert.Equal(30.0, r6Row.GetProperty("window7").GetProperty("sharePct").GetDouble());
Assert.Equal(10.0, r6Row.GetProperty("baseline28").GetProperty("sharePct").GetDouble());
Assert.Equal(3.0, r6Row.GetProperty("lift").GetDouble());
// Eligible 44 ≥ 20 → rate stated; ReEnrolled7d = count/5 per day → 7*0 + 23*0 = 0 → 0%
Assert.Equal(0.0, r6Row.GetProperty("reEnrollRatePct").GetDouble());
// Sorted by count desc → agent:complete first.
Assert.Equal("agent:complete", paths[0].GetProperty("verdictPath").GetString());
}
[Fact]
public void Response_withholds_rate_below_twenty_eligible_and_lift_without_baseline()
{
var today = new DateTime(2026, 8, 23, 0, 0, 0, DateTimeKind.Utc);
var daily = new List<VerdictCalibrationDailyAggregate>
{
Day("global", today, 5, ("maxlife:r3", "Incomplete", 5)),
};
var json = BuildJson(daily, today, days: 7);
var row = json.GetProperty("paths").EnumerateArray().Single();
Assert.Equal(JsonValueKind.Null, row.GetProperty("reEnrollRatePct").ValueKind);
Assert.Equal(JsonValueKind.Null, row.GetProperty("lift").ValueKind); // no baseline rows → never invented
Assert.Equal(100.0, row.GetProperty("sharePct").GetDouble());
}
[Fact]
public void Response_keeps_a_zero_count_bucket_that_carries_overrides()
{
var today = new DateTime(2026, 8, 23, 0, 0, 0, DateTimeKind.Utc);
var day = Day("global", today, 1, ("agent:complete", "Succeeded", 1));
day.Buckets.Add(new VerdictCalibrationBucket { VerdictPath = "sweep:r5_incomplete", Status = "Incomplete", Count = 0, OverriddenByLateCompletion = 1 });
var json = BuildJson(new[] { day }, today, days: 7);
var rows = json.GetProperty("paths").EnumerateArray().ToList();
var prior = rows.Single(p => p.GetProperty("verdictPath").GetString() == "sweep:r5_incomplete");
Assert.Equal(0, prior.GetProperty("count").GetInt32());
Assert.Equal(1, prior.GetProperty("overriddenByLateCompletion").GetInt32());
}
[Fact]
public void Response_is_empty_but_well_formed_before_the_first_sweep()
{
var json = BuildJson(Array.Empty<VerdictCalibrationDailyAggregate>(), new DateTime(2026, 8, 23), 30);
Assert.True(json.GetProperty("success").GetBoolean());
Assert.Empty(json.GetProperty("paths").EnumerateArray());
Assert.Equal(0, json.GetProperty("totals").GetProperty("sessions").GetInt32());
Assert.Equal(JsonValueKind.Null, json.GetProperty("computedAt").ValueKind);
}
[Theory]
[InlineData(null, 30)]
[InlineData("abc", 30)]
[InlineData("0", 1)]
[InlineData("500", 180)]
[InlineData("14", 14)]
public void ClampDays_bounds_the_window(string? raw, int expected)
=> Assert.Equal(expected, VerdictCalibrationResponse.ClampDays(raw));
}
@@ -0,0 +1,201 @@
using System;
using System.Collections.Generic;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Verdict calibration instrumentation (docs/backend/verdict-calibration.md): every status write
/// declares its origin, overriding a prior verdict preserves it, the classifier names the rule
/// behind each verdict, and pre-instrumentation rows derive an honest path from their literals.
/// </summary>
public class VerdictPathTests
{
private static readonly DateTime Start = new(2026, 8, 20, 8, 0, 0, DateTimeKind.Utc);
// ---- ComputePriorVerdict: the correction stream ----
[Theory]
[InlineData("Succeeded", "agent:complete", "Failed", true)] // admin mark over an agent success
[InlineData("Incomplete", "sweep:r5_incomplete", "Succeeded", true)] // late agent completion upgrades a sweep verdict
[InlineData("AwaitingUser", "sweep:r5_awaiting", "Incomplete", true)] // grace expired
[InlineData("Failed", "agent:failed", "Incomplete", true)] // retro reclassification
[InlineData("Stalled", "sweep:stalled", "InProgress", false)] // stall heal is not a correction
[InlineData("Stalled", "sweep:stalled", "Succeeded", false)]
[InlineData("InProgress", "register:new", "Succeeded", false)]
[InlineData("Pending", "agent:whiteglove_pending", "InProgress", false)]
[InlineData("AwaitingUser", "sweep:wg_awaiting", "AwaitingUser", false)] // same-status refresh keeps the original prior
public void Prior_is_preserved_only_when_a_verdict_is_overridden(string existing, string existingPath, string incoming, bool expectPrior)
{
var prior = TableStorageService.ComputePriorVerdict(existing, existingPath, Enum.Parse<SessionStatus>(incoming));
if (expectPrior)
{
Assert.NotNull(prior);
Assert.Equal(existing, prior!.Value.PriorStatus);
Assert.Equal(existingPath, prior.Value.PriorVerdictPath);
}
else
{
Assert.Null(prior);
}
}
[Fact]
public void Prior_needs_a_stamped_path_on_the_existing_row()
{
// Pre-instrumentation rows carry no path — there is nothing trustworthy to preserve.
Assert.Null(TableStorageService.ComputePriorVerdict("Incomplete", null, SessionStatus.Succeeded));
Assert.Null(TableStorageService.ComputePriorVerdict("Incomplete", "", SessionStatus.Succeeded));
Assert.Null(TableStorageService.ComputePriorVerdict(null, "sweep:r6", SessionStatus.Succeeded));
}
// ---- VerdictPaths vocabulary ----
[Fact]
public void Compose_and_Origin_round_trip()
{
var path = VerdictPaths.Compose(VerdictPaths.OriginSweep, ClassifierRules.R5DeviceSetupIncomplete);
Assert.Equal("sweep:r5_incomplete", path);
Assert.Equal("sweep", VerdictPaths.Origin(path));
Assert.True(VerdictPaths.IsClassifierPath(path));
Assert.True(VerdictPaths.IsClassifierPath("maxlife:r1"));
Assert.True(VerdictPaths.IsClassifierPath("late:r4"));
Assert.True(VerdictPaths.IsClassifierPath("retro:r6"));
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.AgentComplete));
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.SweepStalled));
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.RetroSuperseded));
Assert.False(VerdictPaths.IsClassifierPath("legacy:r6"));
Assert.False(VerdictPaths.IsClassifierPath(VerdictPaths.ManualFailed));
Assert.Throws<ArgumentException>(() => VerdictPaths.Compose("", "r1"));
Assert.Throws<ArgumentException>(() => VerdictPaths.Compose("sweep", " "));
}
// ---- Classifier: every return names its rule ----
private static EnrollmentEvent Evt(string type, string? message = null) => new()
{
EventType = type, Timestamp = Start, Source = "test", Message = message ?? type,
};
private static EnrollmentEvent Esp(string message) => Evt("esp_provisioning_status", message);
private const string DeviceSetup44 = "ESP provisioning status: DeviceSetup — 4 of 4 subcategories completed";
private const string AccountSetup55 = "ESP provisioning status: AccountSetup — 5 of 5 subcategories completed";
private const string AccountSetup15 = "ESP provisioning status: AccountSetup — 1 of 5 subcategories completed";
private static string RuleOf(IReadOnlyList<EnrollmentEvent> events, double hours = 6, int grace = 72,
bool preProvisioned = false, DateTime? resumedAt = null, bool selfDeploying = false)
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var (_, _, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, Start.AddHours(hours), grace, isPreProvisioned: preProvisioned,
resumedAt: resumedAt, isSelfDeployingProfile: selfDeploying);
return rule;
}
[Fact]
public void Classifier_returns_the_rule_id_behind_each_verdict()
{
Assert.Equal(ClassifierRules.R1ExplicitFailure, RuleOf(new[] { Evt("enrollment_failed") }));
Assert.Equal(ClassifierRules.R1bWhiteGloveAwaiting, RuleOf(
new[] { Esp(DeviceSetup44), Evt("whiteglove_complete") }, hours: 6, grace: 51, preProvisioned: true, resumedAt: Start.AddHours(1)));
Assert.Equal(ClassifierRules.R1bWhiteGloveSucceeded, RuleOf(
new[] { Esp(DeviceSetup44), Evt("whiteglove_complete") }, hours: 60, grace: 51, preProvisioned: true, resumedAt: Start.AddHours(1)));
Assert.Equal(ClassifierRules.R1cSelfDeploying, RuleOf(new[] { Esp(DeviceSetup44) }, selfDeploying: true));
Assert.Equal(ClassifierRules.R2AccountSetupComplete, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup55) }));
Assert.Equal(ClassifierRules.R3EmergencyBreak, RuleOf(new[] { Esp(DeviceSetup44), Evt("agent_emergency_break") }));
Assert.Equal(ClassifierRules.R4DesktopHello, RuleOf(new[] { Esp(DeviceSetup44), Evt("desktop_arrived"), Evt("hello_provisioning_completed") }));
Assert.Equal(ClassifierRules.R5DeviceSetupAwaiting, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 6, grace: 72));
Assert.Equal(ClassifierRules.R5DeviceSetupIncomplete, RuleOf(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 80, grace: 72));
Assert.Equal(ClassifierRules.R6Fallthrough, RuleOf(Array.Empty<EnrollmentEvent>()));
}
// ---- Derivation for pre-instrumentation rows ----
private static SessionSummary Row(SessionStatus status, string? failureReason = null, string? reconcileReason = null,
string? failureSource = null, string? adminMarked = null, bool soft = false, string? verdictPath = null) => new()
{
SessionId = "s", TenantId = "t", Status = status,
FailureReason = failureReason ?? string.Empty, ReconcileReason = reconcileReason ?? string.Empty,
FailureSource = failureSource ?? string.Empty, AdminMarkedAction = adminMarked, EspSoftFailure = soft,
VerdictPath = verdictPath,
};
private static string Classified(IReadOnlyList<EnrollmentEvent> events, double hours = 80, int grace = 72, bool selfDeploying = false)
{
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var (_, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, Start, Start.AddHours(hours), grace, isSelfDeployingProfile: selfDeploying);
return reason;
}
[Fact]
public void Stamped_rows_are_returned_verbatim()
{
var (path, derived) = VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete, failureReason: "anything", verdictPath: "sweep:r6"));
Assert.Equal("sweep:r6", path);
Assert.False(derived);
}
[Fact]
public void Derivation_uses_the_real_classifier_literals()
{
// Reasons come straight from the classifier so a wording change there surfaces here.
Assert.Equal(("legacy:r6", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
failureReason: Classified(Array.Empty<EnrollmentEvent>()))));
Assert.Equal(("legacy:r5_incomplete", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
failureReason: Classified(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }))));
Assert.Equal(("legacy:r5_awaiting", true), VerdictPathDerivation.Derive(Row(SessionStatus.AwaitingUser,
failureReason: Classified(new[] { Esp(DeviceSetup44), Esp(AccountSetup15) }, hours: 6))));
Assert.Equal(("legacy:r3", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
failureReason: Classified(new[] { Esp(DeviceSetup44), Evt("agent_emergency_break") }))));
Assert.Equal(("legacy:r2", true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
reconcileReason: Classified(new[] { Esp(DeviceSetup44), Esp(AccountSetup55) }))));
Assert.Equal(("legacy:r4", true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
reconcileReason: Classified(new[] { Esp(DeviceSetup44), Evt("desktop_arrived"), Evt("hello_provisioning_completed") }))));
Assert.Equal(("legacy:r1c", true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
reconcileReason: Classified(new[] { Esp(DeviceSetup44) }, selfDeploying: true))));
Assert.Equal(("legacy:r1", true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed,
failureReason: Classified(new[] { Evt("enrollment_failed") }))));
}
[Fact]
public void Derivation_recovers_the_origin_from_writer_suffixes()
{
var r6 = Classified(Array.Empty<EnrollmentEvent>());
Assert.Equal(("maxlife:r6", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
failureReason: r6 + " Verdict triggered by the agent's max-lifetime watchdog shutdown.")));
Assert.Equal(("retro:r6", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete,
failureReason: r6 + " Retro-reclassified from the legacy blanket timeout verdict.")));
}
[Fact]
public void Derivation_attributes_unambiguous_non_classifier_paths()
{
Assert.Equal((VerdictPaths.ManualFailed, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed,
failureReason: "Manually marked as failed by administrator", failureSource: "manual", adminMarked: "Failed")));
Assert.Equal((VerdictPaths.ManualSucceeded, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded, adminMarked: "Succeeded")));
Assert.Equal(("rule:ANALYZE-ESP-001", true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed,
failureReason: "Rule: ESP timeout", failureSource: "rule:ANALYZE-ESP-001")));
Assert.Equal((VerdictPaths.AgentComplete, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded)));
Assert.Equal((VerdictPaths.AgentComplete, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded,
reconcileReason: "Late completion report received — upgraded prior 'Incomplete' verdict")));
Assert.Equal((VerdictPaths.AgentCompleteSoft, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded, soft: true)));
Assert.Equal((VerdictPaths.AgentFailed, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed, failureReason: "ESP timeout (0x800705B4)")));
Assert.Equal((VerdictPaths.AgentEspFailureFallback, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed, failureReason: "ESP failure (backend fallback)")));
Assert.Equal((VerdictPaths.SweepStalled, true), VerdictPathDerivation.Derive(Row(SessionStatus.Stalled, failureReason: "Agent silent for 130min (detected by maintenance sweep)")));
Assert.Equal((VerdictPaths.AgentStallProbe, true), VerdictPathDerivation.Derive(Row(SessionStatus.Stalled, failureReason: "Agent reported stall after 60min without progress (stall_probe)")));
Assert.Equal(("legacy:superseded", true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete, failureReason: "Superseded by session abc: re-registered")));
Assert.Equal((VerdictPaths.AgentWhiteGlovePending, true), VerdictPathDerivation.Derive(Row(SessionStatus.Pending)));
}
[Fact]
public void Derivation_never_invents_a_path()
{
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.Failed, failureReason: "Session timed out after 5 hours (started at 2026-01-01 UTC)")));
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.Incomplete)));
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.InProgress)));
Assert.Equal((VerdictPaths.LegacyUnknown, true), VerdictPathDerivation.Derive(Row(SessionStatus.Succeeded, reconcileReason: "Some future wording")));
}
}
@@ -0,0 +1,39 @@
using AutopilotMonitor.Functions.Services.Notifications;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// HMAC request-signature scheme for generic JSON webhooks: known-vector stability (the
/// scheme is a published contract receivers implement against), determinism, and
/// secret/message sensitivity.
/// </summary>
public class WebhookSignatureTests
{
[Fact]
public void ComputeSignature_MatchesKnownVector()
{
// Independently computed: HMACSHA256("whsec_test_0123456789abcdef", "1700000000.{"a":1}")
var signature = WebhookSignatureCalculator.ComputeSignature(
"whsec_test_0123456789abcdef", "1700000000", "{\"a\":1}");
Assert.Equal("sha256=07728d7e7b1736bac3de36a46c9bc0b660d59aa4a6f251544124fede1dc42a6c", signature);
}
[Fact]
public void ComputeSignature_IsDeterministic_AndSensitiveToEveryInput()
{
var baseline = WebhookSignatureCalculator.ComputeSignature("secret-0123456789", "1700000000", "{}");
Assert.Equal(baseline, WebhookSignatureCalculator.ComputeSignature("secret-0123456789", "1700000000", "{}"));
Assert.NotEqual(baseline, WebhookSignatureCalculator.ComputeSignature("secret-0123456789X", "1700000000", "{}"));
Assert.NotEqual(baseline, WebhookSignatureCalculator.ComputeSignature("secret-0123456789", "1700000001", "{}"));
Assert.NotEqual(baseline, WebhookSignatureCalculator.ComputeSignature("secret-0123456789", "1700000000", "{ }"));
}
[Fact]
public void HeaderNames_AreTheDocumentedContract()
{
Assert.Equal("X-AutopilotMonitor-Timestamp", WebhookSignatureCalculator.TimestampHeader);
Assert.Equal("X-AutopilotMonitor-Signature", WebhookSignatureCalculator.SignatureHeader);
}
}
@@ -56,7 +56,6 @@
<PackageReference Include="Microsoft.Identity.Web" Version="4.3.0" />
<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.10" />
<PackageReference Include="Microsoft.Extensions.Http.Polly" Version="8.0.25" />
<PackageReference Include="Resend" Version="0.2.1" />
</ItemGroup>
<ItemGroup>
@@ -494,6 +494,54 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
}
}
// --- Email Template Overrides ---
private const string EmailTemplatesPartition = "EmailTemplates";
public async Task<EmailTemplateOverride?> GetEmailTemplateOverrideAsync(string kind)
{
try
{
var entity = await _previewConfigTableClient.GetEntityAsync<TableEntity>(EmailTemplatesPartition, kind);
var html = entity.Value.GetString("Html");
if (string.IsNullOrEmpty(html)) return null;
return new EmailTemplateOverride
{
Kind = kind,
Html = html,
UpdatedBy = entity.Value.GetString("UpdatedBy") ?? string.Empty,
UpdatedUtc = entity.Value.GetDateTime("UpdatedUtc") ?? DateTime.MinValue,
};
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
return null;
}
}
public async Task SaveEmailTemplateOverrideAsync(EmailTemplateOverride overrideEntry)
{
var entity = new TableEntity(EmailTemplatesPartition, overrideEntry.Kind)
{
["Html"] = overrideEntry.Html,
["UpdatedBy"] = overrideEntry.UpdatedBy,
["UpdatedUtc"] = overrideEntry.UpdatedUtc == default ? DateTime.UtcNow : overrideEntry.UpdatedUtc,
};
await _previewConfigTableClient.UpsertEntityAsync(entity, TableUpdateMode.Replace);
}
public async Task DeleteEmailTemplateOverrideAsync(string kind)
{
try
{
await _previewConfigTableClient.DeleteEntityAsync(EmailTemplatesPartition, kind);
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
// already gone — reset is idempotent
}
}
// --- Preview Notification Email ---
public async Task<string?> GetNotificationEmailAsync(string tenantId)
@@ -671,6 +719,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
{ "TrialStartedUtc", config.TrialStartedUtc },
{ "TrialConsumed", config.TrialConsumed },
{ "TrialGrantedBy", config.TrialGrantedBy },
{ "ProDowngradedUtc", config.ProDowngradedUtc },
// SLA targets
{ "SlaTargetSuccessRate", config.SlaTargetSuccessRate.HasValue ? (double)config.SlaTargetSuccessRate.Value : (double?)null },
{ "SlaTargetMaxDurationMinutes", config.SlaTargetMaxDurationMinutes },
@@ -783,6 +832,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
TrialStartedUtc = entity.GetDateTime("TrialStartedUtc"),
TrialConsumed = entity.GetBoolean("TrialConsumed") ?? false,
TrialGrantedBy = entity.GetString("TrialGrantedBy"),
ProDowngradedUtc = entity.GetDateTime("ProDowngradedUtc"),
// SLA targets
SlaTargetSuccessRate = entity.GetDouble("SlaTargetSuccessRate") != null ? (decimal)entity.GetDouble("SlaTargetSuccessRate")! : null,
SlaTargetMaxDurationMinutes = entity.GetInt32("SlaTargetMaxDurationMinutes"),
@@ -422,6 +422,152 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
};
}
// ===== VERDICT CALIBRATION DRIFT (docs/backend/verdict-calibration.md) =====
internal const string VerdictCalibrationRowKeyPrefix = "verdictcalibration|";
internal static string BuildVerdictCalibrationRowKey(string kind, string verdictPath, string status)
{
// Paths are a closed vocabulary plus rule ids (ANALYZE-...), statuses are enum names and
// the group marker "*" - none contain table-key-hostile characters, but sanitize anyway.
var k = SanitizeTableKey((kind ?? string.Empty).Trim().ToLowerInvariant());
var p = SanitizeTableKey((verdictPath ?? string.Empty).Trim().ToLowerInvariant());
var s = SanitizeTableKey((status ?? string.Empty).Trim().ToLowerInvariant());
return $"{VerdictCalibrationRowKeyPrefix}{k}|{p}|{s}";
}
public async Task<bool> TryRegisterVerdictCalibrationAlertAsync(string tenantId, VerdictCalibrationAlert alert)
{
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(alert?.Kind) || string.IsNullOrWhiteSpace(alert?.VerdictPath))
return false;
try
{
await _table.AddEntityAsync(BuildVerdictCalibrationEntity(tenantId, alert!));
_logger.LogInformation(
"VerdictCalibration tracker registered: tenant={TenantId} kind={Kind} path={Path}", tenantId, alert!.Kind, alert.VerdictPath);
return true;
}
catch (RequestFailedException ex) when (ex.Status == 409)
{
return false; // episode already active - one ops event per episode
}
catch (Exception ex)
{
_logger.LogWarning(ex, "VerdictCalibration tracker failed: tenant={TenantId} kind={Kind} path={Path}", tenantId, alert!.Kind, alert.VerdictPath);
return false; // fail closed - never double-fire
}
}
public async Task RefreshVerdictCalibrationAlertAsync(string tenantId, VerdictCalibrationAlert alert)
{
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(alert?.Kind) || string.IsNullOrWhiteSpace(alert?.VerdictPath))
return;
try
{
await _table.UpsertEntityAsync(BuildVerdictCalibrationEntity(tenantId, alert!), TableUpdateMode.Replace);
}
catch (Exception ex)
{
_logger.LogWarning(ex, "VerdictCalibration tracker refresh failed: tenant={TenantId} kind={Kind} path={Path}", tenantId, alert!.Kind, alert.VerdictPath);
}
}
public async Task DeleteVerdictCalibrationAlertAsync(string tenantId, string kind, string verdictPath, string status)
{
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(kind) || string.IsNullOrWhiteSpace(verdictPath))
return;
try
{
await _table.DeleteEntityAsync(tenantId.ToLowerInvariant(), BuildVerdictCalibrationRowKey(kind, verdictPath, status));
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
// already gone - idempotent
}
catch (Exception ex)
{
_logger.LogWarning(ex, "VerdictCalibration tracker delete failed: tenant={TenantId} kind={Kind} path={Path}", tenantId, kind, verdictPath);
}
}
public async Task<List<VerdictCalibrationAlert>> GetVerdictCalibrationAlertsAsync(string tenantId)
{
var results = new List<VerdictCalibrationAlert>();
if (string.IsNullOrWhiteSpace(tenantId)) return results;
try
{
var partitionKey = tenantId.ToLowerInvariant();
// '}' sorts directly after '|' - prefix range over "verdictcalibration|...".
var filter = $"PartitionKey eq '{partitionKey}' and RowKey ge '{VerdictCalibrationRowKeyPrefix}' and RowKey lt 'verdictcalibration}}'";
await foreach (var entity in _table.QueryAsync<TableEntity>(filter: filter))
{
results.Add(MapToVerdictCalibrationAlert(entity));
}
}
catch (Exception ex)
{
_logger.LogWarning(ex, "VerdictCalibration tracker list failed for tenant {TenantId}", tenantId);
}
return results;
}
// internal static: pinned by round-trip unit tests (table-storage-serialization rule).
internal static TableEntity BuildVerdictCalibrationEntity(string tenantId, VerdictCalibrationAlert alert)
{
var entity = new TableEntity(tenantId.ToLowerInvariant(), BuildVerdictCalibrationRowKey(alert.Kind, alert.VerdictPath, alert.Status))
{
["TenantId"] = tenantId,
["Kind"] = alert.Kind,
["VerdictPath"] = alert.VerdictPath,
["Status"] = alert.Status ?? string.Empty,
["WindowHitCount"] = alert.WindowHitCount,
["WindowSessionCount"] = alert.WindowSessionCount,
["BaselineHitCount"] = alert.BaselineHitCount,
["BaselineSessionCount"] = alert.BaselineSessionCount,
["WindowRatePct"] = alert.WindowRatePct,
["BaselineRatePct"] = alert.BaselineRatePct,
["WindowStartDate"] = alert.WindowStartDate ?? string.Empty,
["WindowEndDate"] = alert.WindowEndDate ?? string.Empty,
["FirstNotifiedAt"] = alert.FirstNotifiedAt,
["LastEvaluatedAt"] = alert.LastEvaluatedAt,
};
if (alert.Lift.HasValue)
entity["Lift"] = alert.Lift.Value;
if (alert.Dimension != null)
entity["DimensionJson"] = JsonSerializer.Serialize(alert.Dimension);
return entity;
}
internal static VerdictCalibrationAlert MapToVerdictCalibrationAlert(TableEntity entity)
{
RuleRegressionDimension? dimension = null;
var dimensionJson = entity.GetString("DimensionJson");
if (!string.IsNullOrEmpty(dimensionJson))
{
try { dimension = JsonSerializer.Deserialize<RuleRegressionDimension>(dimensionJson!); }
catch (JsonException) { dimension = null; }
}
return new VerdictCalibrationAlert
{
TenantId = entity.GetString("TenantId") ?? entity.PartitionKey,
Kind = entity.GetString("Kind") ?? string.Empty,
VerdictPath = entity.GetString("VerdictPath") ?? string.Empty,
Status = entity.GetString("Status") ?? string.Empty,
WindowHitCount = entity.GetInt32("WindowHitCount") ?? 0,
WindowSessionCount = entity.GetInt32("WindowSessionCount") ?? 0,
BaselineHitCount = entity.GetInt32("BaselineHitCount") ?? 0,
BaselineSessionCount = entity.GetInt32("BaselineSessionCount") ?? 0,
WindowRatePct = entity.GetDouble("WindowRatePct") ?? 0,
BaselineRatePct = entity.GetDouble("BaselineRatePct") ?? 0,
Lift = entity.GetDouble("Lift"),
WindowStartDate = entity.GetString("WindowStartDate") ?? string.Empty,
WindowEndDate = entity.GetString("WindowEndDate") ?? string.Empty,
Dimension = dimension,
FirstNotifiedAt = entity.GetDateTimeOffset("FirstNotifiedAt")?.UtcDateTime ?? DateTime.MinValue,
LastEvaluatedAt = entity.GetDateTimeOffset("LastEvaluatedAt")?.UtcDateTime ?? DateTime.MinValue,
};
}
internal const string AppVersionRegressionRowKeyPrefix = "appversionregression|";
/// <summary>
@@ -60,6 +60,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<List<SessionSummary>> GetUsageWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null)
=> _storage.GetUsageWindowSessionsAsync(startDate, endDate, tenantId);
public Task<List<SessionSummary>> GetMaintenanceWindowSessionsAsync(DateTime startDate, DateTime endDate)
=> _storage.GetMaintenanceWindowSessionsAsync(startDate, endDate);
public Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null)
=> _storage.GetGeoWindowSessionsAsync(startDate, endDate, tenantId);
@@ -72,6 +75,9 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults)
=> _storage.GetLegacyTimeoutFailedSessionsAsync(tenantId, maxResults);
public Task<List<SessionSummary>> GetSelfDeployingSilentSessionsAsync(string tenantId, int maxResults)
=> _storage.GetSelfDeployingSilentSessionsAsync(tenantId, maxResults);
public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId)
=> _storage.GetSessionsLeanAsync(tenantId);
@@ -159,5 +159,17 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<int> DeleteDeviceJourneyAggregatesOlderThanAsync(DateTime cutoffDate)
=> _storage.DeleteDeviceJourneyAggregatesOlderThanAsync(cutoffDate);
public Task<bool> SaveVerdictCalibrationAggregateAsync(VerdictCalibrationDailyAggregate aggregate)
=> _storage.SaveVerdictCalibrationAggregateAsync(aggregate);
public Task<List<VerdictCalibrationDailyAggregate>> GetVerdictCalibrationAggregatesAsync(string tenantId, DateTime startDate, DateTime endDate)
=> _storage.GetVerdictCalibrationAggregatesAsync(tenantId, startDate, endDate);
public Task DeleteVerdictCalibrationAggregateAsync(string tenantId, string dateKey)
=> _storage.DeleteVerdictCalibrationAggregateAsync(tenantId, dateKey);
public Task<int> DeleteVerdictCalibrationAggregatesOlderThanAsync(DateTime cutoffDate)
=> _storage.DeleteVerdictCalibrationAggregatesOlderThanAsync(cutoffDate);
}
}
@@ -58,7 +58,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
=> _storage.GetAllSessionStatsAsync(tenantIdFilter, days, allowedTenantIds);
public async Task<bool> UpdateSessionStatusAsync(
string tenantId, string sessionId, SessionStatus status,
string tenantId, string sessionId, SessionStatus status, string verdictPath,
EnrollmentPhase? currentPhase = null, string? failureReason = null,
DateTime? completedAt = null, DateTime? earliestEventTimestamp = null,
DateTime? latestEventTimestamp = null, bool? isPreProvisioned = null,
@@ -68,7 +68,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
string? failureSnapshotJson = null, bool allowTerminalReclassification = false,
bool espSoftFailure = false, string? completionSource = null)
{
var transitioned = await _storage.UpdateSessionStatusAsync(tenantId, sessionId, status,
var transitioned = await _storage.UpdateSessionStatusAsync(tenantId, sessionId, status, verdictPath,
currentPhase, failureReason, completedAt, earliestEventTimestamp,
latestEventTimestamp, isPreProvisioned, isUserDriven, resumedAt,
stalledAt, clearStalledAt, clearFailureReason, failureSource, adminMarkedAction,
@@ -142,8 +142,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
=> _storage.UpdateSessionDiagnosticsBlobAsync(tenantId, sessionId, blobName, destination);
public Task SetSessionPreProvisionedAsync(string tenantId, string sessionId, bool isPreProvisioned,
SessionStatus? status = null, bool? isUserDriven = null)
=> _storage.SetSessionPreProvisionedAsync(tenantId, sessionId, isPreProvisioned, status, isUserDriven);
SessionStatus? status = null, bool? isUserDriven = null, string? verdictPath = null)
=> _storage.SetSessionPreProvisionedAsync(tenantId, sessionId, isPreProvisioned, status, isUserDriven, verdictPath);
public Task UpdateSessionGeoAsync(string tenantId, string sessionId,
string? country, string? region, string? city, string? loc)
@@ -18,5 +18,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task InitializeAsync()
=> _storage.InitializeTablesAsync();
public Task EnsureAllAsync()
=> _storage.EnsureAllTablesAsync();
}
}
@@ -0,0 +1,163 @@
using System.Net;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Admin;
/// <summary>
/// Global-Admin management of the two transactional email templates (welcome, farewell):
/// read the effective template, store/reset an HTML override, and send a test mail.
/// All routes are GlobalAdminOnly via <c>EndpointAccessPolicyCatalog</c>.
/// </summary>
public class EmailTemplatesFunction
{
private readonly ILogger<EmailTemplatesFunction> _logger;
private readonly EmailTemplateService _templates;
private readonly IEmailService _email;
private readonly PreviewWhitelistService _previewWhitelistService;
private readonly TenantConfigurationService _tenantConfigurationService;
public EmailTemplatesFunction(
ILogger<EmailTemplatesFunction> logger,
EmailTemplateService templates,
IEmailService email,
PreviewWhitelistService previewWhitelistService,
TenantConfigurationService tenantConfigurationService)
{
_logger = logger;
_templates = templates;
_email = email;
_previewWhitelistService = previewWhitelistService;
_tenantConfigurationService = tenantConfigurationService;
}
/// <summary>
/// GET /api/global/email-templates/{kind}
/// Effective template: the override when stored, otherwise the built-in (raw, with the
/// {{domainName}} placeholder so it can be edited as-is).
/// </summary>
[Function("GetEmailTemplate")]
public async Task<HttpResponseData> Get(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "global/email-templates/{kind}")] HttpRequestData req,
string kind)
{
if (!EmailTemplateService.TryParseKind(kind, out var templateKind))
return await BadRequest(req, "Unknown template kind. Use 'welcome' or 'farewell'.");
var overrideEntry = await _templates.GetOverrideAsync(templateKind);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new
{
kind = EmailTemplateService.KindKey(templateKind),
subject = EmailTemplateService.Subject(templateKind),
isOverridden = overrideEntry is not null,
html = overrideEntry?.Html ?? EmailTemplateService.BuiltInRaw(templateKind),
builtInHtml = EmailTemplateService.BuiltInRaw(templateKind),
updatedBy = overrideEntry?.UpdatedBy,
updatedUtc = overrideEntry?.UpdatedUtc,
placeholder = EmailTemplateService.DomainPlaceholder,
maxLength = EmailTemplateService.MaxHtmlLength,
});
return response;
}
/// <summary>PUT /api/global/email-templates/{kind} — body { html } stores an override.</summary>
[Function("SaveEmailTemplate")]
public async Task<HttpResponseData> Save(
[HttpTrigger(AuthorizationLevel.Anonymous, "put", Route = "global/email-templates/{kind}")] HttpRequestData req,
string kind)
{
if (!EmailTemplateService.TryParseKind(kind, out var templateKind))
return await BadRequest(req, "Unknown template kind. Use 'welcome' or 'farewell'.");
var body = await req.ReadFromJsonAsync<EmailTemplateRequest>();
var error = EmailTemplateService.Validate(body?.Html);
if (error is not null)
return await BadRequest(req, error);
var upn = TenantHelper.GetUserIdentifier(req);
var saved = await _templates.SaveOverrideAsync(templateKind, body!.Html!, upn);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { kind = saved.Kind, isOverridden = true, updatedBy = saved.UpdatedBy, updatedUtc = saved.UpdatedUtc });
return response;
}
/// <summary>DELETE /api/global/email-templates/{kind} — resets to the built-in template.</summary>
[Function("ResetEmailTemplate")]
public async Task<HttpResponseData> Reset(
[HttpTrigger(AuthorizationLevel.Anonymous, "delete", Route = "global/email-templates/{kind}")] HttpRequestData req,
string kind)
{
if (!EmailTemplateService.TryParseKind(kind, out var templateKind))
return await BadRequest(req, "Unknown template kind. Use 'welcome' or 'farewell'.");
await _templates.DeleteOverrideAsync(templateKind, TenantHelper.GetUserIdentifier(req));
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { kind = EmailTemplateService.KindKey(templateKind), isOverridden = false });
return response;
}
/// <summary>
/// POST /api/global/email-templates/{kind}/test — body { html? } sends the effective template
/// (or the unsaved draft) to the caller's tenant contact address: the activation
/// notification email, then TenantConfiguration.ContactEmail, then the caller's UPN.
/// Domain = the caller's tenant domain.
/// </summary>
[Function("SendEmailTemplateTest")]
public async Task<HttpResponseData> SendTest(
[HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "global/email-templates/{kind}/test")] HttpRequestData req,
string kind)
{
if (!EmailTemplateService.TryParseKind(kind, out var templateKind))
return await BadRequest(req, "Unknown template kind. Use 'welcome' or 'farewell'.");
var body = await req.ReadFromJsonAsync<EmailTemplateRequest>();
var draft = string.IsNullOrWhiteSpace(body?.Html) ? null : body!.Html;
if (draft is not null)
{
var error = EmailTemplateService.Validate(draft);
if (error is not null) return await BadRequest(req, error);
}
var tenantId = TenantHelper.GetTenantId(req);
var upn = TenantHelper.GetUserIdentifier(req);
var tenantConfig = await _tenantConfigurationService.GetConfigurationAsync(tenantId);
var toEmail = await _previewWhitelistService.GetNotificationEmailAsync(tenantId);
if (string.IsNullOrWhiteSpace(toEmail)) toEmail = tenantConfig.ContactEmail;
if (string.IsNullOrWhiteSpace(toEmail)) toEmail = upn.Contains('@') ? upn : null;
if (string.IsNullOrWhiteSpace(toEmail))
return await BadRequest(req, "No contact email found for your tenant — set a notification email or ContactEmail first.");
var sent = await _email.SendTestAsync(templateKind, toEmail, tenantConfig.DomainName, draft);
_logger.LogInformation("Email template test {Kind} requested by {Upn} → {ToEmail}: {Result}", templateKind, upn, toEmail, sent ? "accepted" : "failed");
if (!sent)
{
var failed = req.CreateResponse(HttpStatusCode.BadGateway);
await failed.WriteAsJsonAsync(new { error = "The email provider did not accept the message. Check the backend log for the provider response.", sentTo = toEmail });
return failed;
}
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new { sentTo = toEmail, domainName = tenantConfig.DomainName, draft = draft is not null });
return response;
}
private static async Task<HttpResponseData> BadRequest(HttpRequestData req, string error)
{
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
await bad.WriteAsJsonAsync(new { error });
return bad;
}
}
public class EmailTemplateRequest
{
public string? Html { get; set; }
}
@@ -35,6 +35,8 @@ namespace AutopilotMonitor.Functions.Functions.Admin
/// Query parameters:
/// - mode: "legacy_timeouts" (default) — re-classify pre-classifier "Session timed out" Failed rows;
/// "pending_orphans" — resolve Pending rows superseded by a later session of the same device.
/// "self_deploying_silent" — heal self-deploying (kiosk) rows parked Incomplete/AwaitingUser/Stalled
/// after Device ESP provisioned to Succeeded (classifier rule 1c, 2026-08-23).
/// - dryRun: "false" to actually write; anything else (or absent) reports only.
/// - tenantId: optional single-tenant scope (GUID); absent = all tenants.
/// - maxSessions: per-invocation examination cap (default 200, max 2000). Re-run to continue.
@@ -82,6 +84,8 @@ namespace AutopilotMonitor.Functions.Functions.Admin
await _service.ReclassifyLegacyTimeoutsAsync(tenantIdScope, dryRun, maxSessions, userEmail),
LegacyReclassificationService.ModePendingOrphans =>
await _service.ResolvePendingOrphansAsync(tenantIdScope, dryRun, maxSessions, userEmail),
LegacyReclassificationService.ModeSelfDeployingSilent =>
await _service.ReconcileSelfDeployingSilentAsync(tenantIdScope, dryRun, maxSessions, userEmail),
_ => null,
};
@@ -90,7 +94,7 @@ namespace AutopilotMonitor.Functions.Functions.Admin
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
await bad.WriteAsJsonAsync(new
{
error = $"Unknown mode '{mode}'. Valid: {LegacyReclassificationService.ModeLegacyTimeouts}, {LegacyReclassificationService.ModePendingOrphans}"
error = $"Unknown mode '{mode}'. Valid: {LegacyReclassificationService.ModeLegacyTimeouts}, {LegacyReclassificationService.ModePendingOrphans}, {LegacyReclassificationService.ModeSelfDeployingSilent}"
});
return bad;
}
@@ -142,6 +142,23 @@ namespace AutopilotMonitor.Functions.Functions.Config
|| string.Equals(destination, "Hosted", StringComparison.OrdinalIgnoreCase);
}
/// <summary>
/// Global ∪ tenant diagnostics paths: global first, order preserved, blank paths dropped,
/// duplicates (trimmed, case-insensitive) collapsed onto the first occurrence. The agent
/// keys ZIP entries on the path, so a path present in both lists would otherwise produce
/// duplicate entry names inside the archive.
/// </summary>
internal static List<DiagnosticsLogPath> MergeDiagnosticsLogPaths(
IEnumerable<DiagnosticsLogPath> global,
IEnumerable<DiagnosticsLogPath> tenant)
{
return global.Concat(tenant)
.Where(p => !string.IsNullOrWhiteSpace(p?.Path))
.GroupBy(p => p.Path.Trim(), StringComparer.OrdinalIgnoreCase)
.Select(g => g.First())
.ToList();
}
/// <summary>
/// Returns the (ZipSha256, ExeSha256) pair appropriate for the calling agent.
/// Reads the X-Agent-Version header, parses the major, and dispatches to the
@@ -242,10 +259,10 @@ namespace AutopilotMonitor.Functions.Functions.Config
// Get active IME log patterns for this tenant (from Table Storage)
var imeLogPatterns = await _imeLogPatternService.GetActivePatternsForTenantAsync(tenantId);
// Merge global + tenant-specific diagnostics log paths
var globalDiagPaths = adminConfig.GetDiagnosticsGlobalLogPaths();
var tenantDiagPaths = tenantConfig.GetDiagnosticsLogPaths();
var diagLogPaths = globalDiagPaths.Concat(tenantDiagPaths).ToList();
// Merge global + tenant-specific diagnostics log paths (the built-in sections are
// compiled into the agent — DiagnosticsBuiltInSections — and never travel here)
var diagLogPaths = MergeDiagnosticsLogPaths(
adminConfig.GetDiagnosticsGlobalLogPaths(), tenantConfig.GetDiagnosticsLogPaths());
// Select per-line hash oracle from the X-Agent-Version header (parametric per major).
// The wire response keeps generic field names (LatestAgentSha256 / LatestAgentExeSha256)
@@ -27,14 +27,16 @@ namespace AutopilotMonitor.Functions.Functions.Config
private readonly TenantConfigurationService _configService;
private readonly AdminConfigurationService _adminConfigService;
private readonly IMaintenanceRepository _maintenanceRepo;
private readonly OpsEventService _opsEvents;
private readonly TimeProvider _time;
public PlanManagementFunction(
ILogger<PlanManagementFunction> logger,
TenantConfigurationService configService,
AdminConfigurationService adminConfigService,
IMaintenanceRepository maintenanceRepo)
: this(logger, configService, adminConfigService, maintenanceRepo, TimeProvider.System)
IMaintenanceRepository maintenanceRepo,
OpsEventService opsEvents)
: this(logger, configService, adminConfigService, maintenanceRepo, opsEvents, TimeProvider.System)
{
}
@@ -44,12 +46,14 @@ namespace AutopilotMonitor.Functions.Functions.Config
TenantConfigurationService configService,
AdminConfigurationService adminConfigService,
IMaintenanceRepository maintenanceRepo,
OpsEventService opsEvents,
TimeProvider time)
{
_logger = logger;
_configService = configService;
_adminConfigService = adminConfigService;
_maintenanceRepo = maintenanceRepo;
_opsEvents = opsEvents;
_time = time;
}
@@ -139,22 +143,8 @@ namespace AutopilotMonitor.Functions.Functions.Config
var nowUtc = _time.GetUtcNow().UtcDateTime;
var changes = new Dictionary<string, string>();
if (newPlanTier != null && !string.Equals(config.PlanTier, newPlanTier, StringComparison.Ordinal))
{
changes["PlanTier"] = $"{config.PlanTier} -> {newPlanTier}";
config.PlanTier = newPlanTier;
}
if (trialProvided && config.TrialExpiresUtc != newTrialExpiresUtc)
{
changes["TrialExpiresUtc"] = $"{FormatUtc(config.TrialExpiresUtc)} -> {FormatUtc(newTrialExpiresUtc)}";
config.TrialExpiresUtc = newTrialExpiresUtc;
if (newTrialExpiresUtc.HasValue)
{
config.TrialStartedUtc ??= nowUtc;
config.TrialGrantedBy = caller;
}
}
var (editionBefore, editionAfter) =
ApplyPlanChanges(config, newPlanTier, trialProvided, newTrialExpiresUtc, caller, nowUtc, changes);
if (changes.Count > 0)
{
@@ -171,9 +161,17 @@ namespace AutopilotMonitor.Functions.Functions.Config
requestCtx.TargetTenantId,
caller,
changes);
}
var effectiveEdition = FeatureEntitlementCatalog.ResolveEdition(config.PlanTier, config.TrialExpiresUtc, nowUtc);
if (editionBefore == TenantEdition.Pro && editionAfter == TenantEdition.Community)
{
await _opsEvents.RecordTenantPlanDowngradedAsync(
requestCtx.TargetTenantId,
config.DomainName,
caller,
TenantEntitlementService.GetRetentionGraceEndUtc(config, nowUtc),
config.DataRetentionDays);
}
}
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new
@@ -182,7 +180,8 @@ namespace AutopilotMonitor.Functions.Functions.Config
planTier = config.PlanTier,
trialExpiresUtc = config.TrialExpiresUtc,
trialConsumed = config.TrialConsumed,
effectiveEdition = effectiveEdition.ToString().ToLowerInvariant()
effectiveEdition = editionAfter.ToString().ToLowerInvariant(),
retentionGraceEndsUtc = TenantEntitlementService.GetRetentionGraceEndUtc(config, nowUtc)
});
return response;
}
@@ -195,6 +194,61 @@ namespace AutopilotMonitor.Functions.Functions.Config
}
}
/// <summary>
/// Pure mutation core of the PATCH plan endpoint — no I/O, unit-testable. Applies the
/// plan/trial changes to <paramref name="config"/>, records human-readable entries in
/// <paramref name="changes"/>, and returns the EFFECTIVE edition before/after (same
/// <paramref name="nowUtc"/> for both, so only the mutation itself can flip it).
///
/// Maintains the retention downgrade grace anchor: an effective Pro→Community
/// transition stamps <see cref="TenantConfiguration.ProDowngradedUtc"/> (covers both a
/// planTier downgrade and an explicitly ended trial); any state that is effectively Pro
/// afterwards clears the anchor, so a re-upgrade also resets the grace clock.
/// </summary>
internal static (TenantEdition Before, TenantEdition After) ApplyPlanChanges(
TenantConfiguration config,
string? newPlanTier,
bool trialProvided,
DateTime? newTrialExpiresUtc,
string caller,
DateTime nowUtc,
Dictionary<string, string> changes)
{
var before = FeatureEntitlementCatalog.ResolveEdition(config.PlanTier, config.TrialExpiresUtc, nowUtc);
if (newPlanTier != null && !string.Equals(config.PlanTier, newPlanTier, StringComparison.Ordinal))
{
changes["PlanTier"] = $"{config.PlanTier} -> {newPlanTier}";
config.PlanTier = newPlanTier;
}
if (trialProvided && config.TrialExpiresUtc != newTrialExpiresUtc)
{
changes["TrialExpiresUtc"] = $"{FormatUtc(config.TrialExpiresUtc)} -> {FormatUtc(newTrialExpiresUtc)}";
config.TrialExpiresUtc = newTrialExpiresUtc;
if (newTrialExpiresUtc.HasValue)
{
config.TrialStartedUtc ??= nowUtc;
config.TrialGrantedBy = caller;
}
}
var after = FeatureEntitlementCatalog.ResolveEdition(config.PlanTier, config.TrialExpiresUtc, nowUtc);
if (before == TenantEdition.Pro && after == TenantEdition.Community)
{
changes["ProDowngradedUtc"] = $"{FormatUtc(config.ProDowngradedUtc)} -> {FormatUtc(nowUtc)}";
config.ProDowngradedUtc = nowUtc;
}
else if (after == TenantEdition.Pro && config.ProDowngradedUtc.HasValue)
{
changes["ProDowngradedUtc"] = $"{FormatUtc(config.ProDowngradedUtc)} -> (none)";
config.ProDowngradedUtc = null;
}
return (before, after);
}
/// <summary>
/// POST /api/config/{tenantId}/trial — TenantAdminOrGA (catalog-enforced). Self-service
/// 30-day Pro trial, exactly once per tenant. 409 when the trial was already
@@ -262,6 +316,8 @@ namespace AutopilotMonitor.Functions.Functions.Config
config.TrialExpiresUtc = nowUtc.AddDays(SelfServiceTrialDays);
config.TrialConsumed = true;
config.TrialGrantedBy = caller;
// Effectively Pro again — a leftover retention grace anchor is obsolete.
config.ProDowngradedUtc = null;
config.UpdatedBy = caller;
// Fail-loud: throws when the write did not persist (cache invalidated in finally).
@@ -82,7 +82,7 @@ namespace AutopilotMonitor.Functions.Functions.Config
var testAlert = NotificationAlertBuilder.BuildTestAlert();
var result = await _webhookNotificationService.SendNotificationWithResultAsync(
channel.Url, (WebhookProviderType)channel.ProviderType, testAlert, channel.GetCustomHeaders());
channel.Url, (WebhookProviderType)channel.ProviderType, testAlert, channel.GetCustomHeaders(), channel.GetSigningSecret());
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(new
@@ -0,0 +1,77 @@
using System;
using System.Linq;
using System.Net;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Diagnostics
{
public class GetDiagnosticsPathsFunction
{
private readonly ILogger<GetDiagnosticsPathsFunction> _logger;
private readonly AdminConfigurationService _adminConfigService;
public GetDiagnosticsPathsFunction(
ILogger<GetDiagnosticsPathsFunction> logger,
AdminConfigurationService adminConfigService)
{
_logger = logger;
_adminConfigService = adminConfigService;
}
/// <summary>
/// GET /api/diagnostics/paths
/// What every diagnostics package collects before a tenant's own entries: the built-in
/// section catalog (compiled into the agent, shared through
/// <see cref="DiagnosticsBuiltInSections"/>) and the platform-wide global paths set by
/// Global Admins. Member-readable by design — the catalog is code, the global list is
/// non-secret admin data, and every settings viewer needs to see what is collected from
/// the tenant's devices. Tenant-less: the payload is platform-wide, not tenant data.
/// </summary>
[Function("GetDiagnosticsPaths")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "diagnostics/paths")] HttpRequestData req)
{
try
{
// Authentication + MemberRead authorization enforced by PolicyEnforcementMiddleware
var adminConfig = await _adminConfigService.GetConfigurationAsync();
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(BuildPayload(adminConfig));
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error getting diagnostics paths");
var response = req.CreateResponse(HttpStatusCode.InternalServerError);
await response.WriteAsJsonAsync(new { error = "Internal server error" });
return response;
}
}
/// <summary>
/// Static projection so unit tests pin the wire shape without an HttpRequestData mock.
/// <c>condition</c> travels as the enum NAME ("Always" | "RealmJoinWatcher" |
/// "DevicePreparation"), never the integer — the web switches on the string.
/// </summary>
internal static object BuildPayload(AdminConfiguration adminConfig) => new
{
builtIn = DiagnosticsBuiltInSections.All.Select(s => new
{
id = s.Id,
zipFolder = s.ZipFolder,
sourceFolder = s.SourceFolder,
patterns = s.Patterns,
includeSubfolders = s.IncludeSubfolders,
description = s.Description,
condition = s.Condition.ToString(),
}).ToArray(),
globalPaths = adminConfig.GetDiagnosticsGlobalLogPaths(),
};
}
}
@@ -10,15 +10,21 @@ using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Maintenance;
/// <summary>
/// Daily informational sweep over tenant Pro trials. NOT load-bearing — trial expiry
/// degrades a tenant to Community at READ time (FeatureEntitlementCatalog.ResolveEdition);
/// this timer only surfaces the transitions as ops events so the operator has visibility:
/// Daily informational sweep over the tenant plan lifecycle. NOT load-bearing — trial expiry
/// degrades a tenant to Community at READ time (FeatureEntitlementCatalog.ResolveEdition),
/// and the retention grace is likewise resolved read-time by the retention fanout; this timer
/// only surfaces the transitions as ops events so the operator has visibility:
/// <list type="bullet">
/// <item><c>TenantTrialExpired</c> (Warning) — the trial ended within the last sweep window
/// (24h look-back matching the daily cadence).</item>
/// <item><c>TenantTrialExpiring</c> (Info) — the trial ends within the next
/// <see cref="ExpiringHeadsUpDays"/> days (re-emitted each daily run until expiry —
/// acceptable for an Info-tier heads-up, no dedupe state to maintain).</item>
/// <item><c>TenantRetentionGraceExpiring</c> / <c>TenantRetentionGraceEnded</c> (Warning) —
/// same window mechanics for the retention downgrade grace
/// (TenantEntitlementService.GetRetentionGraceEndUtc). Emitted only when data is
/// actually at risk: stored retention above the Community cap (0 = infinite ⇒ never
/// at risk, the fanout skips those tenants).</item>
/// </list>
/// Tenants whose stored <c>PlanTier</c> is already a permanent Pro tier ("pro"/legacy "enterprise") are skipped —
/// their trial timestamps are inert leftovers and expiry changes nothing.
@@ -90,45 +96,93 @@ public class TrialExpirySweepFunction
{
ct.ThrowIfCancellationRequested();
if (config.TrialExpiresUtc is not DateTime expiry)
continue;
// Permanent Pro: the trial timestamps are inert leftovers — expiry changes nothing.
if (FeatureEntitlementCatalog.IsPermanentProTier(config.PlanTier))
continue;
result.TrialsSeen++;
if (expiry <= now)
{
if (expiry > now - ExpiredLookBack)
{
await _opsEvents.RecordTenantTrialExpiredAsync(config.TenantId, config.DomainName, expiry)
.ConfigureAwait(false);
result.ExpiredEmitted++;
}
// Older expiries were reported by a previous run — stay silent.
}
else if (expiry <= now + ExpiringHeadsUp)
{
var daysLeft = Math.Max(1, (int)Math.Ceiling((expiry - now).TotalDays));
await _opsEvents.RecordTenantTrialExpiringAsync(config.TenantId, config.DomainName, expiry, daysLeft)
.ConfigureAwait(false);
result.ExpiringEmitted++;
}
await SweepTrialAsync(config, now, result).ConfigureAwait(false);
await SweepRetentionGraceAsync(config, now, result).ConfigureAwait(false);
}
_logger.LogInformation(
"TrialExpirySweep completed: trialsSeen={TrialsSeen} expiredEmitted={ExpiredEmitted} expiringEmitted={ExpiringEmitted}",
result.TrialsSeen, result.ExpiredEmitted, result.ExpiringEmitted);
"TrialExpirySweep completed: trialsSeen={TrialsSeen} expiredEmitted={ExpiredEmitted} expiringEmitted={ExpiringEmitted} " +
"graceExpiringEmitted={GraceExpiringEmitted} graceEndedEmitted={GraceEndedEmitted}",
result.TrialsSeen, result.ExpiredEmitted, result.ExpiringEmitted,
result.GraceExpiringEmitted, result.GraceEndedEmitted);
return result;
}
private async Task SweepTrialAsync(
AutopilotMonitor.Shared.Models.TenantConfiguration config, DateTime now, SweepResult result)
{
if (config.TrialExpiresUtc is not DateTime expiry)
return;
// Permanent Pro: the trial timestamps are inert leftovers — expiry changes nothing.
if (FeatureEntitlementCatalog.IsPermanentProTier(config.PlanTier))
return;
result.TrialsSeen++;
if (expiry <= now)
{
if (expiry > now - ExpiredLookBack)
{
await _opsEvents.RecordTenantTrialExpiredAsync(config.TenantId, config.DomainName, expiry)
.ConfigureAwait(false);
result.ExpiredEmitted++;
}
// Older expiries were reported by a previous run — stay silent.
}
else if (expiry <= now + ExpiringHeadsUp)
{
var daysLeft = Math.Max(1, (int)Math.Ceiling((expiry - now).TotalDays));
await _opsEvents.RecordTenantTrialExpiringAsync(config.TenantId, config.DomainName, expiry, daysLeft)
.ConfigureAwait(false);
result.ExpiringEmitted++;
}
}
/// <summary>
/// Retention downgrade grace visibility: warn shortly before and once after the grace ends,
/// but only when data is actually at risk — stored retention above the Community cap.
/// (0 = the GA-only infinite escape hatch: the fanout skips those tenants, nothing is ever
/// deleted, so the comparison excludes it naturally.) GetRetentionGraceEndUtc returns null
/// for effectively-Pro tenants, so re-upgraded tenants go silent automatically.
/// </summary>
private async Task SweepRetentionGraceAsync(
AutopilotMonitor.Shared.Models.TenantConfiguration config, DateTime now, SweepResult result)
{
var communityCap = FeatureEntitlementCatalog.Get(TenantEdition.Community).RetentionCapDays;
if (config.DataRetentionDays <= communityCap)
return;
if (TenantEntitlementService.GetRetentionGraceEndUtc(config, now) is not DateTime graceEnd)
return;
if (graceEnd <= now)
{
if (graceEnd > now - ExpiredLookBack)
{
await _opsEvents.RecordTenantRetentionGraceEndedAsync(
config.TenantId, config.DomainName, graceEnd, config.DataRetentionDays)
.ConfigureAwait(false);
result.GraceEndedEmitted++;
}
}
else if (graceEnd <= now + ExpiringHeadsUp)
{
var daysLeft = Math.Max(1, (int)Math.Ceiling((graceEnd - now).TotalDays));
await _opsEvents.RecordTenantRetentionGraceExpiringAsync(
config.TenantId, config.DomainName, graceEnd, daysLeft, config.DataRetentionDays)
.ConfigureAwait(false);
result.GraceExpiringEmitted++;
}
}
/// <summary>Summary counters returned from <see cref="RunCoreAsync"/> so tests can assert outcomes.</summary>
internal sealed class SweepResult
{
public int TrialsSeen { get; set; }
public int ExpiredEmitted { get; set; }
public int ExpiringEmitted { get; set; }
public int GraceExpiringEmitted { get; set; }
public int GraceEndedEmitted { get; set; }
}
}
@@ -0,0 +1,232 @@
using System.Globalization;
using System.Net;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Functions.Metrics
{
/// <summary>
/// Operator-only verdict calibration matrix (docs/backend/verdict-calibration.md): per verdict
/// path, how many sessions it produced in the window, its share, how often it was overridden
/// (admin / late completion / other), the 7-day re-enrollment proxy, and a 7d-vs-28d trend.
/// Platform scope ("global" partition) or one tenant via ?tenantId=. Serves the admin
/// metrics page and the <c>get_verdict_calibration</c> MCP tool. Never customer-facing —
/// these are platform-internal classifier diagnostics, not product features.
/// </summary>
public class GetVerdictCalibrationFunction
{
private readonly ILogger<GetVerdictCalibrationFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
private readonly IHardwareRejectionNotificationTracker _tracker;
public GetVerdictCalibrationFunction(
ILogger<GetVerdictCalibrationFunction> logger,
IMetricsRepository metricsRepo,
IHardwareRejectionNotificationTracker tracker)
{
_logger = logger;
_metricsRepo = metricsRepo;
_tracker = tracker;
}
[Function("GetVerdictCalibration")]
public async Task<HttpResponseData> Run(
[HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "global/metrics/verdict-calibration")] HttpRequestData req)
{
try
{
// Authentication + GlobalReadOrAdmin authorization enforced by PolicyEnforcementMiddleware
var query = System.Web.HttpUtility.ParseQueryString(req.Url.Query);
var tenantIdFilter = query["tenantId"];
var days = VerdictCalibrationResponse.ClampDays(query["days"]);
var partition = "global";
if (!string.IsNullOrWhiteSpace(tenantIdFilter))
{
if (!Guid.TryParse(tenantIdFilter, out var parsed))
{
var bad = req.CreateResponse(HttpStatusCode.BadRequest);
await bad.WriteAsJsonAsync(new { success = false, message = "tenantId must be a GUID" });
return bad;
}
partition = parsed.ToString("D");
}
var today = DateTime.UtcNow.Date;
// One read covers both the requested window and the trend horizon (7d window + 28d baseline).
var readDays = Math.Max(days, VerdictCalibrationResponse.TrendHorizonDays);
var daily = await _metricsRepo.GetVerdictCalibrationAggregatesAsync(
partition, VerdictCalibrationResponse.InclusiveWindowStart(today, readDays), today);
// Active drift episodes of this partition (tracker keyspace; fail-soft empty).
var alerts = await _tracker.GetVerdictCalibrationAlertsAsync(partition);
var response = req.CreateResponse(HttpStatusCode.OK);
await response.WriteAsJsonAsync(VerdictCalibrationResponse.Build(daily, partition, today, days, alerts));
return response;
}
catch (Exception ex)
{
_logger.LogError(ex, "Error fetching verdict calibration");
var errorResponse = req.CreateResponse(HttpStatusCode.InternalServerError);
await errorResponse.WriteAsJsonAsync(new { success = false, message = "Internal server error" });
return errorResponse;
}
}
}
/// <summary>Pure response builder — internal static so the matrix arithmetic is pinned by tests.</summary>
internal static class VerdictCalibrationResponse
{
internal const int DefaultWindowDays = 30;
internal const int MaxWindowDays = 180; // aggregate retention
internal const int TrendWindowDays = 7;
internal const int TrendBaselineDays = 28;
internal const int TrendHorizonDays = TrendWindowDays + TrendBaselineDays;
internal static int ClampDays(string? raw)
{
if (!int.TryParse(raw, out var days)) return DefaultWindowDays;
return Math.Clamp(days, 1, MaxWindowDays);
}
/// <summary>"Last N days" = exactly N calendar day keys including today (both range ends inclusive).</summary>
internal static DateTime InclusiveWindowStart(DateTime today, int days) => today.AddDays(-(days - 1));
internal sealed class PathRow
{
public string VerdictPath { get; set; } = string.Empty;
public string Status { get; set; } = string.Empty;
public int Count { get; set; }
public double SharePct { get; set; }
public int DerivedCount { get; set; }
public int Eligible7d { get; set; }
public int ReEnrolled7d { get; set; }
/// <summary>Null below the minimum eligible count — never a rate on a handful of sessions.</summary>
public double? ReEnrollRatePct { get; set; }
public int OverriddenByAdmin { get; set; }
public int OverriddenByLateCompletion { get; set; }
public int OverriddenOther { get; set; }
public TrendWindow Window7 { get; set; } = new();
public TrendWindow Baseline28 { get; set; } = new();
/// <summary>Window share ÷ baseline share; null when the baseline share is 0 (a new path has no finite lift — never invented).</summary>
public double? Lift { get; set; }
}
internal sealed class TrendWindow
{
public int Count { get; set; }
public int Sessions { get; set; }
public double SharePct { get; set; }
}
/// <summary>Minimum eligible sessions before a re-enrollment rate is stated (truthfulness: no rate on n&lt;20).</summary>
internal const int MinEligibleForRate = 20;
internal static object Build(
IReadOnlyList<VerdictCalibrationDailyAggregate> daily, string partition, DateTime today, int days,
IReadOnlyList<VerdictCalibrationAlert>? alerts = null)
{
var windowStart = Key(InclusiveWindowStart(today, days));
var trendWindowStart = Key(InclusiveWindowStart(today, TrendWindowDays));
var baselineStart = Key(InclusiveWindowStart(today, TrendHorizonDays));
var rows = new Dictionary<(string Path, string Status), PathRow>();
int sessions = 0, terminal = 0, derived = 0;
int trendSessions = 0, baselineSessions = 0;
DateTime? computedAt = null;
var versions = new HashSet<int>();
PathRow Row(VerdictCalibrationBucket b)
{
var key = (b.VerdictPath, b.Status);
if (!rows.TryGetValue(key, out var row))
{
row = new PathRow { VerdictPath = b.VerdictPath, Status = b.Status };
rows[key] = row;
}
return row;
}
foreach (var day in daily)
{
var inWindow = string.CompareOrdinal(day.Date, windowStart) >= 0;
var inTrendWindow = string.CompareOrdinal(day.Date, trendWindowStart) >= 0;
var inBaseline = !inTrendWindow && string.CompareOrdinal(day.Date, baselineStart) >= 0;
if (inWindow)
{
sessions += day.SessionCount;
terminal += day.TerminalSessionCount;
versions.Add(day.Version);
if (computedAt == null || day.ComputedAt > computedAt) computedAt = day.ComputedAt;
}
if (inTrendWindow) trendSessions += day.SessionCount;
if (inBaseline) baselineSessions += day.SessionCount;
foreach (var b in day.Buckets)
{
// Overrides are attributed to the prior path and can exist on a bucket with Count=0.
var row = Row(b);
if (inWindow)
{
row.Count += b.Count;
row.DerivedCount += b.DerivedCount;
row.Eligible7d += b.Eligible7d;
row.ReEnrolled7d += b.ReEnrolled7d;
row.OverriddenByAdmin += b.OverriddenByAdmin;
row.OverriddenByLateCompletion += b.OverriddenByLateCompletion;
row.OverriddenOther += b.OverriddenOther;
derived += b.DerivedCount;
}
if (inTrendWindow) row.Window7.Count += b.Count;
if (inBaseline) row.Baseline28.Count += b.Count;
}
}
var list = rows.Values
.Where(r => r.Count > 0 || r.OverriddenByAdmin + r.OverriddenByLateCompletion + r.OverriddenOther > 0)
.OrderByDescending(r => r.Count)
.ThenBy(r => r.VerdictPath, StringComparer.Ordinal)
.ThenBy(r => r.Status, StringComparer.Ordinal)
.ToList();
foreach (var r in list)
{
r.SharePct = Pct(r.Count, sessions);
r.ReEnrollRatePct = r.Eligible7d >= MinEligibleForRate ? Pct(r.ReEnrolled7d, r.Eligible7d) : null;
r.Window7.Sessions = trendSessions;
r.Window7.SharePct = Pct(r.Window7.Count, trendSessions);
r.Baseline28.Sessions = baselineSessions;
r.Baseline28.SharePct = Pct(r.Baseline28.Count, baselineSessions);
r.Lift = r.Baseline28.SharePct > 0 && trendSessions > 0
? Math.Round(r.Window7.SharePct / r.Baseline28.SharePct, 2)
: null;
}
return new
{
success = true,
tenantId = partition,
windowDays = days,
windowStart,
windowEnd = Key(today),
computedAt,
versions = versions.OrderBy(v => v).ToArray(),
totals = new { sessions, terminal, derived, days = daily.Count(d => string.CompareOrdinal(d.Date, windowStart) >= 0) },
trend = new { windowDays = TrendWindowDays, baselineDays = TrendBaselineDays, windowSessions = trendSessions, baselineSessions },
paths = list,
// Wording contract shared with the rule radar: a dimension is correlated, not causal.
alerts = (alerts ?? Array.Empty<VerdictCalibrationAlert>())
.OrderByDescending(a => a.FirstNotifiedAt)
.ToList(),
};
}
private static string Key(DateTime d) => d.ToString("yyyy-MM-dd", CultureInfo.InvariantCulture);
private static double Pct(int part, int whole) => whole <= 0 ? 0 : Math.Round(100.0 * part / whole, 1);
}
}
@@ -19,20 +19,20 @@ public class PreviewWhitelistFunction
private readonly PreviewWhitelistService _previewWhitelistService;
private readonly TenantApprovalService _tenantApprovalService;
private readonly TenantConfigurationService _tenantConfigurationService;
private readonly ResendEmailService _resendEmailService;
private readonly IEmailService _emailService;
public PreviewWhitelistFunction(
ILogger<PreviewWhitelistFunction> logger,
PreviewWhitelistService previewWhitelistService,
TenantApprovalService tenantApprovalService,
TenantConfigurationService tenantConfigurationService,
ResendEmailService resendEmailService)
IEmailService emailService)
{
_logger = logger;
_previewWhitelistService = previewWhitelistService;
_tenantApprovalService = tenantApprovalService;
_tenantConfigurationService = tenantConfigurationService;
_resendEmailService = resendEmailService;
_emailService = emailService;
}
/// <summary>
@@ -221,7 +221,7 @@ public class PreviewWhitelistFunction
}
var tenantConfig = await _tenantConfigurationService.GetConfigurationAsync(tenantId);
await _resendEmailService.SendPreviewApprovedEmailAsync(email, tenantConfig.DomainName);
await _emailService.SendPreviewApprovedEmailAsync(email, tenantConfig.DomainName);
// Explicit GA send always sends; consume the once-only marker (best-effort) so
// the automatic paths won't produce a duplicate afterwards.
@@ -49,6 +49,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
tenantId,
sessionId,
SessionStatus.Failed,
VerdictPaths.ManualFailed,
currentPhase: null, // Keep current phase
failureReason: "Manually marked as failed by administrator",
failureSource: "manual",
@@ -49,6 +49,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
tenantId,
sessionId,
SessionStatus.Succeeded,
VerdictPaths.ManualSucceeded,
currentPhase: null, // Keep current phase
failureReason: "Manually marked as succeeded by administrator",
adminMarkedAction: "Succeeded"
@@ -305,7 +305,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
$"enrollment session at {registration.StartedAt:yyyy-MM-dd HH:mm} UTC before this one " +
"reached a terminal state.";
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
registration.TenantId, stale.SessionId, SessionStatus.Incomplete,
registration.TenantId, stale.SessionId, SessionStatus.Incomplete, VerdictPaths.RegisterSuperseded,
failureReason: reason, failureSource: "superseded_by_reregistration");
if (!transitioned)
continue;
@@ -182,6 +182,35 @@ namespace AutopilotMonitor.Functions.Helpers
var headerError = ValidateWebhookCustomHeaders(channel.CustomHeadersJson);
if (headerError != null)
return $"channel \"{label}\" headers: {headerError}";
var secretError = ValidateWebhookSigningSecret(channel.SigningSecret);
if (secretError != null)
return $"channel \"{label}\" signing secret: {secretError}";
}
return null;
}
/// <summary>
/// Validates a generic-webhook HMAC signing secret. Returns an error message, or null when
/// valid/absent. A short key is brute-forceable as an HMAC key; the printable-ASCII rule
/// keeps the secret copy-paste safe and (as a side effect) rejects a stray unrestored
/// redaction sentinel, which is shorter than the minimum. No provider gate here — a secret
/// on a non-generic channel is stored but inert at dispatch (GetSigningSecret), matching
/// how CustomHeadersJson behaves.
/// </summary>
internal static string? ValidateWebhookSigningSecret(string? secret)
{
if (string.IsNullOrWhiteSpace(secret))
return null;
if (secret.Length < 16 || secret.Length > 128)
return "must be 16-128 characters.";
foreach (var ch in secret)
{
if (ch < 0x21 || ch > 0x7E)
return "must be printable ASCII without spaces or line breaks.";
}
return null;
@@ -0,0 +1,281 @@
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Helpers;
/// <summary>
/// One drift finding of the verdict-calibration radar (pure output; the maintenance partial
/// turns it into a tracker episode + ops event).
/// </summary>
public class VerdictCalibrationFinding
{
public string Kind { get; set; } = string.Empty;
public string VerdictPath { get; set; } = string.Empty;
public string Status { get; set; } = string.Empty;
public int WindowHitCount { get; set; }
public int WindowSessionCount { get; set; }
public int BaselineHitCount { get; set; }
public int BaselineSessionCount { get; set; }
public double WindowRatePct { get; set; }
public double BaselineRatePct { get; set; }
public double? Lift { get; set; }
public string WindowStartDate { get; set; } = string.Empty;
public string WindowEndDate { get; set; } = string.Empty;
}
/// <summary>
/// Pure drift detection over one tenant's verdict-calibration daily rows
/// (docs/backend/verdict-calibration.md) — the same statistics as the rule-regression radar
/// (7-day window vs the prior 28 days, lift ≥2×, Wilson-separated via
/// <see cref="MetricsMath.RateIncreaseSeparated"/>, small-n gates) applied to three questions:
/// <list type="bullet">
/// <item><b>share_regression</b> — one verdict path's share of all sessions doubled. Points at a rule
/// (or an agent path) whose conditions suddenly match much more often.</item>
/// <item><b>silence_share_regression</b> — the share of sessions the backend had to decide because
/// the agent went silent (sweep:* + maxlife:*) doubled. An agent-liveness signal, not a classifier one.</item>
/// <item><b>evidence_gap</b> — absolute gate: the pure fallthrough rule (r6) decides ≥20 % of the
/// window's classifier verdicts. The evidence the other rules need is not arriving.</item>
/// </list>
/// Derived (<c>legacy:*</c>) paths are excluded from all three — pre-instrumentation attribution
/// is too weak to alert on. Agent-declared paths (<c>agent:*</c>, <c>register:*</c>) are excluded
/// from the per-path share regression: their share tracks the customer's workflow mix (a
/// WhiteGlove rollout week doubles <c>agent:whiteglove_pending</c>; a Continue-Anyway tenant
/// lives on <c>agent:complete_soft</c>) — real signals, but not about the BACKEND classifier
/// this radar calibrates. First production pass (2026-08-23) fired exactly those three. Re-arm
/// mirrors the rule radar: share back under 1.5× baseline or the path stops occurring; the
/// evidence gap re-arms under 15 %.
/// </summary>
public static class VerdictCalibrationRadar
{
public const int WindowDays = RuleRegressionRadar.WindowDays;
public const int BaselineDays = RuleRegressionRadar.BaselineDays;
public const int MinWindowHits = RuleRegressionRadar.MinWindowHitSessions;
public const int MinWindowSessions = RuleRegressionRadar.MinWindowSessions;
public const double MinLift = RuleRegressionRadar.MinLift;
public const double ReArmLiftFactor = RuleRegressionRadar.ReArmLiftFactor;
/// <summary>Evidence gap fires at ≥20 % r6 share of the window's classifier verdicts (≥20 verdicts) and re-arms under 15 %.</summary>
public const double EvidenceGapFirePct = 20.0;
public const double EvidenceGapReArmPct = 15.0;
public const int EvidenceGapMinVerdicts = 20;
public const string SilenceGroupPath = "sweep+maxlife";
public const string EvidenceGapPath = "r6";
public const string GroupStatus = "*";
private const string LegacyOrigin = "legacy";
/// <summary>
/// True when a path is eligible for the per-path share regression: backend-decided paths only
/// (sweep / maxlife / late / retro / rule / manual / ingest). Agent-declared and registration
/// paths mirror customer workflow, legacy paths are derived.
/// </summary>
public static bool IsShareRegressionEligible(string path)
{
var origin = VerdictPaths.Origin(path);
return origin != LegacyOrigin
&& origin != "agent"
&& origin != "register";
}
public static List<VerdictCalibrationFinding> Evaluate(IReadOnlyList<VerdictCalibrationDailyAggregate> tenantRows, DateTime targetDateUtc)
{
var findings = new List<VerdictCalibrationFinding>();
var horizon = Summarize(tenantRows, targetDateUtc);
var windowStart = targetDateUtc.Date.AddDays(-(WindowDays - 1)).ToString("yyyy-MM-dd");
var windowEnd = targetDateUtc.Date.ToString("yyyy-MM-dd");
// 1. Per-path share regression.
foreach (var pair in horizon.Paths)
{
if (!IsShareRegressionEligible(pair.Key.Path)) continue;
var f = TryShareRegression(VerdictCalibrationAlertKinds.ShareRegression, pair.Key.Path, pair.Key.Status,
pair.Value.WindowHits, horizon.WindowSessions, pair.Value.BaselineHits, horizon.BaselineSessions, windowStart, windowEnd);
if (f != null) findings.Add(f);
}
// 2. Silence-share regression (group).
{
var f = TryShareRegression(VerdictCalibrationAlertKinds.SilenceShareRegression, SilenceGroupPath, GroupStatus,
horizon.SilenceWindowHits, horizon.WindowSessions, horizon.SilenceBaselineHits, horizon.BaselineSessions, windowStart, windowEnd);
if (f != null) findings.Add(f);
}
// 3. Evidence gap (absolute).
if (horizon.ClassifierWindowVerdicts >= EvidenceGapMinVerdicts)
{
var pct = 100.0 * horizon.R6WindowHits / horizon.ClassifierWindowVerdicts;
if (pct >= EvidenceGapFirePct)
{
findings.Add(new VerdictCalibrationFinding
{
Kind = VerdictCalibrationAlertKinds.EvidenceGap,
VerdictPath = EvidenceGapPath,
Status = GroupStatus,
WindowHitCount = horizon.R6WindowHits,
WindowSessionCount = horizon.ClassifierWindowVerdicts,
BaselineHitCount = horizon.R6BaselineHits,
BaselineSessionCount = horizon.ClassifierBaselineVerdicts,
WindowRatePct = Math.Round(pct, 1),
BaselineRatePct = horizon.ClassifierBaselineVerdicts > 0
? Math.Round(100.0 * horizon.R6BaselineHits / horizon.ClassifierBaselineVerdicts, 1) : 0,
Lift = null,
WindowStartDate = windowStart,
WindowEndDate = windowEnd,
});
}
}
return findings
.OrderByDescending(f => f.WindowRatePct)
.ThenBy(f => f.Kind, StringComparer.Ordinal)
.ThenBy(f => f.VerdictPath, StringComparer.Ordinal)
.ToList();
}
/// <summary>True when an ACTIVE alert may re-arm given the current horizon sums (or when its path is no longer eligible — episodes from an earlier, broader gate clear themselves).</summary>
public static bool ShouldReArm(VerdictCalibrationAlert alert, IReadOnlyList<VerdictCalibrationDailyAggregate> tenantRows, DateTime targetDateUtc)
{
if (alert.Kind == VerdictCalibrationAlertKinds.ShareRegression && !IsShareRegressionEligible(alert.VerdictPath))
return true;
var horizon = Summarize(tenantRows, targetDateUtc);
switch (alert.Kind)
{
case VerdictCalibrationAlertKinds.EvidenceGap:
if (horizon.ClassifierWindowVerdicts < EvidenceGapMinVerdicts) return true;
return 100.0 * horizon.R6WindowHits / horizon.ClassifierWindowVerdicts < EvidenceGapReArmPct;
case VerdictCalibrationAlertKinds.SilenceShareRegression:
return ReArmByRate(horizon.SilenceWindowHits, horizon.WindowSessions, horizon.SilenceBaselineHits, horizon.BaselineSessions);
default:
horizon.Paths.TryGetValue((alert.VerdictPath, alert.Status), out var sums);
return ReArmByRate(sums.WindowHits, horizon.WindowSessions, sums.BaselineHits, horizon.BaselineSessions);
}
}
/// <summary>Current numbers for an ongoing episode (refresh without re-firing).</summary>
public static (int WindowHits, int WindowSessions, int BaselineHits, int BaselineSessions) CurrentSums(
VerdictCalibrationAlert alert, IReadOnlyList<VerdictCalibrationDailyAggregate> tenantRows, DateTime targetDateUtc)
{
var horizon = Summarize(tenantRows, targetDateUtc);
switch (alert.Kind)
{
case VerdictCalibrationAlertKinds.EvidenceGap:
return (horizon.R6WindowHits, horizon.ClassifierWindowVerdicts, horizon.R6BaselineHits, horizon.ClassifierBaselineVerdicts);
case VerdictCalibrationAlertKinds.SilenceShareRegression:
return (horizon.SilenceWindowHits, horizon.WindowSessions, horizon.SilenceBaselineHits, horizon.BaselineSessions);
default:
horizon.Paths.TryGetValue((alert.VerdictPath, alert.Status), out var sums);
return (sums.WindowHits, horizon.WindowSessions, sums.BaselineHits, horizon.BaselineSessions);
}
}
private static bool ReArmByRate(int windowHits, int windowSessions, int baselineHits, int baselineSessions)
{
if (windowHits == 0 || windowSessions == 0) return true;
if (baselineSessions == 0 || baselineHits == 0) return false; // zero-baseline episode re-arms only via hits-stopped
var windowRate = (double)windowHits / windowSessions;
var baselineRate = (double)baselineHits / baselineSessions;
return windowRate < ReArmLiftFactor * baselineRate;
}
private static VerdictCalibrationFinding? TryShareRegression(string kind, string path, string status,
int windowHits, int windowSessions, int baselineHits, int baselineSessions, string windowStart, string windowEnd)
{
if (windowHits < MinWindowHits) return null;
if (windowSessions < MinWindowSessions) return null;
if (baselineSessions <= 0) return null;
var windowRate = (double)windowHits / windowSessions;
var baselineRate = (double)baselineHits / baselineSessions;
if (windowRate < MinLift * baselineRate) return null;
if (!MetricsMath.RateIncreaseSeparated(windowHits, windowSessions, baselineHits, baselineSessions)) return null;
return new VerdictCalibrationFinding
{
Kind = kind,
VerdictPath = path,
Status = status,
WindowHitCount = windowHits,
WindowSessionCount = windowSessions,
BaselineHitCount = baselineHits,
BaselineSessionCount = baselineSessions,
WindowRatePct = Math.Round(windowRate * 100, 1),
BaselineRatePct = Math.Round(baselineRate * 100, 1),
Lift = baselineRate > 0 ? Math.Round(windowRate / baselineRate, 1) : null,
WindowStartDate = windowStart,
WindowEndDate = windowEnd,
};
}
internal readonly record struct PathSums(int WindowHits, int BaselineHits);
internal sealed class Horizon
{
public int WindowSessions;
public int BaselineSessions;
public int SilenceWindowHits;
public int SilenceBaselineHits;
public int R6WindowHits;
public int R6BaselineHits;
public int ClassifierWindowVerdicts;
public int ClassifierBaselineVerdicts;
public readonly Dictionary<(string Path, string Status), PathSums> Paths = new();
}
/// <summary>
/// Sums the daily rows into the trailing window [target−6d, target] and the prior baseline
/// [target−34d, target−7d] (ISO-string date compare, rows outside the horizon ignored).
/// Silence group = sweep:* + maxlife:* (incl. non-rule sweep paths like sweep:stalled — the
/// question is "how often did the backend have to decide", not which rule did). Classifier
/// verdicts = stamped rule paths (<see cref="VerdictPaths.IsClassifierPath"/>); r6 = their
/// fallthrough subset.
/// </summary>
internal static Horizon Summarize(IReadOnlyList<VerdictCalibrationDailyAggregate> rows, DateTime targetDateUtc)
{
var windowStart = targetDateUtc.Date.AddDays(-(WindowDays - 1)).ToString("yyyy-MM-dd");
var baselineStart = targetDateUtc.Date.AddDays(-(WindowDays - 1 + BaselineDays)).ToString("yyyy-MM-dd");
var target = targetDateUtc.Date.ToString("yyyy-MM-dd");
var h = new Horizon();
foreach (var row in rows)
{
if (string.CompareOrdinal(row.Date, target) > 0) continue;
bool inWindow;
if (string.CompareOrdinal(row.Date, windowStart) >= 0) inWindow = true;
else if (string.CompareOrdinal(row.Date, baselineStart) >= 0) inWindow = false;
else continue;
if (inWindow) h.WindowSessions += row.SessionCount; else h.BaselineSessions += row.SessionCount;
foreach (var b in row.Buckets)
{
if (b.Count == 0) continue;
var key = (b.VerdictPath, b.Status);
h.Paths.TryGetValue(key, out var sums);
h.Paths[key] = inWindow
? new PathSums(sums.WindowHits + b.Count, sums.BaselineHits)
: new PathSums(sums.WindowHits, sums.BaselineHits + b.Count);
var origin = VerdictPaths.Origin(b.VerdictPath);
if (origin == VerdictPaths.OriginSweep || origin == VerdictPaths.OriginMaxLifetime)
{
if (inWindow) h.SilenceWindowHits += b.Count; else h.SilenceBaselineHits += b.Count;
}
if (VerdictPaths.IsClassifierPath(b.VerdictPath))
{
if (inWindow) h.ClassifierWindowVerdicts += b.Count; else h.ClassifierBaselineVerdicts += b.Count;
if (b.VerdictPath.EndsWith(":" + ClassifierRules.R6Fallthrough, StringComparison.Ordinal))
{
if (inWindow) h.R6WindowHits += b.Count; else h.R6BaselineHits += b.Count;
}
}
}
}
return h;
}
}
@@ -0,0 +1,127 @@
using System;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Helpers
{
/// <summary>
/// Read-side attribution for sessions written BEFORE <see cref="SessionSummary.VerdictPath"/>
/// was stamped. No mass backfill: the calibration aggregate calls <see cref="Derive"/> and
/// counts the result with <c>Derived=true</c>, so pre-instrumentation history still shows up
/// in the matrix and is visibly weaker. The derivation reads the fixed reason literals the
/// writers emit (classifier rule prefixes, the max-lifetime / retro suffixes, the manual and
/// rule attributions) — when the origin is ambiguous but the rule is known the path is
/// <c>legacy:{rule}</c>; when nothing is attributable it is <see cref="VerdictPaths.LegacyUnknown"/>.
/// A stamped row is returned verbatim with <c>Derived=false</c>.
/// </summary>
public static class VerdictPathDerivation
{
private const string OriginLegacy = "legacy";
// Literals are owned by EnrollmentTimeoutClassifier / the writers; pinned by
// VerdictPathDerivationTests so a wording change there fails here, not silently.
private const string R1Prefix = "Enrollment reported an explicit failure before timeout";
private const string R1bAwaitingPrefix = "Pre-provisioning (WhiteGlove) completed; no user sign-in since Part 2 resumed";
private const string R1bSucceededPrefix = "Reconciled at timeout: pre-provisioning (WhiteGlove Part 1) completed";
private const string R1cPrefix = "Reconciled: self-deploying profile";
private const string R2Prefix = "Reconciled at timeout: Account Setup completed";
private const string R3Prefix = "Agent emergency break fired";
private const string R4Prefix = "Reconciled at timeout: user completed setup";
private const string R5AwaitingPrefix = "Device Setup completed; awaiting user / Account Setup phase";
private const string R5IncompletePrefix = "No completion signal within";
private const string R6Prefix = "No Device Setup completion or explicit failure signal observed before timeout";
private const string SupersededPrefix = "Superseded by session";
private const string SweepStalledPrefix = "Agent silent for";
private const string AgentStallProbePrefix = "Agent reported stall after";
private const string EspFallbackLiteral = "ESP failure (backend fallback)";
/// <summary>Pre-classifier blanket timeout verdict (LegacyReclassificationService) — never agent-reported.</summary>
private const string LegacyBlanketTimeoutPrefix = "Session timed out";
private const string MaxLifetimeSuffix = "max-lifetime watchdog";
private const string RetroSuffix = "Retro-reclassified";
public static (string Path, bool Derived) Derive(SessionSummary s)
{
if (s == null) throw new ArgumentNullException(nameof(s));
if (!string.IsNullOrWhiteSpace(s.VerdictPath))
return (s.VerdictPath!, false);
return (DeriveLegacy(s), true);
}
private static string DeriveLegacy(SessionSummary s)
{
var failureReason = s.FailureReason ?? string.Empty;
var reconcileReason = s.ReconcileReason ?? string.Empty;
var failureSource = s.FailureSource ?? string.Empty;
// Unambiguous attributions first — independent of status text.
if (string.Equals(s.AdminMarkedAction, "Succeeded", StringComparison.OrdinalIgnoreCase))
return VerdictPaths.ManualSucceeded;
if (string.Equals(s.AdminMarkedAction, "Failed", StringComparison.OrdinalIgnoreCase))
return VerdictPaths.ManualFailed;
if (failureSource.StartsWith("rule:", StringComparison.OrdinalIgnoreCase))
return VerdictPaths.Compose(VerdictPaths.OriginRule, failureSource.Substring(5));
switch (s.Status)
{
case SessionStatus.Succeeded:
if (reconcileReason.Length == 0
|| reconcileReason.StartsWith("Late completion report received", StringComparison.Ordinal))
return s.EspSoftFailure ? VerdictPaths.AgentCompleteSoft : VerdictPaths.AgentComplete;
if (Starts(reconcileReason, R1bSucceededPrefix)) return Classifier(reconcileReason, ClassifierRules.R1bWhiteGloveSucceeded);
if (Starts(reconcileReason, R1cPrefix)) return Classifier(reconcileReason, ClassifierRules.R1cSelfDeploying);
if (Starts(reconcileReason, R2Prefix)) return Classifier(reconcileReason, ClassifierRules.R2AccountSetupComplete);
if (Starts(reconcileReason, R4Prefix)) return Classifier(reconcileReason, ClassifierRules.R4DesktopHello);
return VerdictPaths.LegacyUnknown;
case SessionStatus.Failed:
if (Starts(failureReason, R1Prefix)) return Classifier(failureReason, ClassifierRules.R1ExplicitFailure);
if (string.Equals(failureSource, "max_lifetime_watchdog", StringComparison.OrdinalIgnoreCase))
return VerdictPaths.Compose(VerdictPaths.OriginMaxLifetime, ClassifierRules.R1ExplicitFailure);
if (failureReason.Contains(EspFallbackLiteral, StringComparison.Ordinal)) return VerdictPaths.AgentEspFailureFallback;
if (Starts(failureReason, LegacyBlanketTimeoutPrefix)) return VerdictPaths.LegacyUnknown;
if (failureReason.Length > 0 && failureSource.Length == 0) return VerdictPaths.AgentFailed;
return VerdictPaths.LegacyUnknown;
case SessionStatus.Incomplete:
if (Starts(failureReason, R3Prefix)) return Classifier(failureReason, ClassifierRules.R3EmergencyBreak);
if (Starts(failureReason, R5IncompletePrefix)) return Classifier(failureReason, ClassifierRules.R5DeviceSetupIncomplete);
if (Starts(failureReason, R6Prefix)) return Classifier(failureReason, ClassifierRules.R6Fallthrough);
if (Starts(failureReason, SupersededPrefix)) return VerdictPaths.Compose(OriginLegacy, "superseded");
return VerdictPaths.LegacyUnknown;
case SessionStatus.AwaitingUser:
if (Starts(failureReason, R1bAwaitingPrefix)) return VerdictPaths.Compose(OriginLegacy, "wg_awaiting");
if (Starts(failureReason, R5AwaitingPrefix)) return Classifier(failureReason, ClassifierRules.R5DeviceSetupAwaiting);
return VerdictPaths.LegacyUnknown;
case SessionStatus.Stalled:
if (Starts(failureReason, SweepStalledPrefix)) return VerdictPaths.SweepStalled;
if (Starts(failureReason, AgentStallProbePrefix)) return VerdictPaths.AgentStallProbe;
return VerdictPaths.LegacyUnknown;
case SessionStatus.Pending:
return VerdictPaths.AgentWhiteGlovePending;
default:
return VerdictPaths.LegacyUnknown;
}
}
/// <summary>
/// Classifier verdict with a known rule: the origin is recoverable only from the suffixes
/// the max-lifetime and retro writers append; sweep / late-reconcile share the bare text
/// and stay <c>legacy:{rule}</c>.
/// </summary>
private static string Classifier(string reason, string rule)
{
if (reason.Contains(MaxLifetimeSuffix, StringComparison.Ordinal))
return VerdictPaths.Compose(VerdictPaths.OriginMaxLifetime, rule);
if (reason.Contains(RetroSuffix, StringComparison.Ordinal))
return VerdictPaths.Compose(VerdictPaths.OriginRetro, rule);
return VerdictPaths.Compose(OriginLegacy, rule);
}
private static bool Starts(string text, string prefix)
=> text.StartsWith(prefix, StringComparison.Ordinal);
}
}
@@ -1,5 +1,6 @@
using System.Diagnostics;
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.DataContracts;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Middleware;
@@ -140,6 +141,15 @@ public class RequestTelemetryMiddleware : IFunctionsWorkerMiddleware
if (caughtException != null)
requestTelemetry.Properties["ExceptionType"] = caughtException.GetType().Name;
// Bypass the worker's adaptive sampling for this item. The SDK sampling processor
// passes through any item whose SamplingPercentage is already set, so this is the
// per-item opt-out that host.json "excludedTypes" (host process only) cannot provide.
// Live-verified 2026-08-23 BEFORE this change: worker request rows carried
// ItemCount 2-5 on weekdays (~30% of individual requests missing), so count()-based
// queries undercounted. This copy is the single canonical request record — the host
// duplicate is dropped by a workspace DCR transformation on AppRequests.
((ISupportSampling)requestTelemetry).SamplingPercentage = 100;
try
{
_telemetryClient.TrackRequest(requestTelemetry);
@@ -71,6 +71,10 @@ builder.Services
// unchanged host duplicate). The host request item is emitted by the HOST process and cannot be
// dropped from worker code or by host.json "Host.Results" level. Re-attempt only with a mechanism
// verified against the isolated-worker telemetry pipeline — do not re-add the manual chain.
// 2026-08-23: solved differently — RequestTelemetryMiddleware sets SamplingPercentage=100 on its
// own item (per-item sampling bypass), and the host duplicate is removed by a workspace DCR
// transformation on AppRequests (keeps rows with Properties.Source=='WorkerMiddleware' plus
// non-HTTP host rows with empty Url, i.e. timer/queue invocations). See docs/backend/telemetry-ingest-shaping.md.
// Drop successful Azure Storage dependencies (Table/Queue/Blob) from the worker telemetry
// pipeline to curb AppDependencies ingestion cost — this backend is storage-I/O heavy and those
@@ -226,6 +230,7 @@ builder.Services.AddHostedService<AutopilotMonitor.Functions.Services.Offboardin
// no AddHostedService call needed. DI for its dependencies comes from the singletons above.
builder.Services.AddHostedService<TableInitializerService>(); // Initialize all tables at startup
builder.Services.AddHostedService<StartupTelemetryService>(); // BackendStartupMs / BackendTableInitMs metrics on ApplicationStarted
// Data Access Layer — repository interfaces backed by Table Storage.
// To switch to Cosmos DB: replace AddTableStorageDataAccess() with AddCosmosDataAccess().
@@ -437,8 +442,15 @@ builder.Services.AddHttpClient<AutopilotMonitor.Functions.Services.Notifications
.AddPolicyHandler((sp, _) => sp.GetRequiredService<ResiliencePolicies>().Notification);
builder.Services.AddHttpClient<TelegramNotificationService>()
.AddPolicyHandler((sp, _) => sp.GetRequiredService<ResiliencePolicies>().Notification);
builder.Services.AddSingleton<ResendEmailService>();
builder.Services.AddSingleton<IOffboardFarewellEmailSender>(sp => sp.GetRequiredService<ResendEmailService>());
// Transactional email (welcome / farewell). Typed client like the other outbound notifiers;
// the provider is an implementation detail of EmailService + the Email:* settings.
builder.Services.AddSingleton<EmailTemplateService>();
builder.Services.AddSingleton<IEmailTemplateProvider>(sp => sp.GetRequiredService<EmailTemplateService>());
builder.Services.AddHttpClient<EmailService>()
.ConfigureHttpClient(client => client.Timeout = TimeSpan.FromSeconds(15))
.AddPolicyHandler((sp, _) => sp.GetRequiredService<ResiliencePolicies>().Notification);
builder.Services.AddTransient<IEmailService>(sp => sp.GetRequiredService<EmailService>());
builder.Services.AddTransient<IOffboardFarewellEmailSender>(sp => sp.GetRequiredService<EmailService>());
builder.Services.AddSingleton<GlobalNotificationService>();
builder.Services.AddSingleton<TenantNotificationService>();
@@ -282,6 +282,9 @@ public static class EndpointAccessPolicyCatalog
new("GET", "audit/tpm-pss-unsupported", EndpointPolicy.MemberRead),
new("GET", "diagnostics/download-url", EndpointPolicy.MemberRead, TenantScoping.QueryParam),
new("POST", "diagnostics/download-ticket", EndpointPolicy.MemberRead, TenantScoping.QueryParam),
// Platform-defined diagnostics collection (built-in catalog + global paths): read-only,
// no secrets, needed by every settings viewer — member tier, JWT-scoped (no tenant parameter).
new("GET", "diagnostics/paths", EndpointPolicy.MemberRead, TenantScoping.Jwt),
new("GET", "rules/gather", EndpointPolicy.MemberRead),
new("GET", "rules/analyze", EndpointPolicy.MemberRead),
new("GET", "rules/ime-log-patterns", EndpointPolicy.MemberRead),
@@ -461,6 +464,7 @@ public static class EndpointAccessPolicyCatalog
new("GET", "global/metrics/fleet-health", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/time-attribution", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/device-journeys", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/metrics/verdict-calibration", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/apps/list", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/apps/{appName}/analytics", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
new("GET", "global/apps/{appName}/sessions", EndpointPolicy.GlobalReadOrAdmin, TenantScoping.QueryParam),
@@ -497,6 +501,12 @@ public static class EndpointAccessPolicyCatalog
new("POST", "versions/block", EndpointPolicy.GlobalAdminOnly),
new("DELETE", "versions/block/{encodedPattern}", EndpointPolicy.GlobalAdminOnly),
new("POST", "maintenance/trigger", EndpointPolicy.GlobalAdminOnly),
// Transactional email templates (welcome / farewell): read, override, reset, test send.
// Read stays GA-only too — the override HTML is operator content, not tenant data.
new("GET", "global/email-templates/{kind}", EndpointPolicy.GlobalAdminOnly),
new("PUT", "global/email-templates/{kind}", EndpointPolicy.GlobalAdminOnly),
new("DELETE", "global/email-templates/{kind}", EndpointPolicy.GlobalAdminOnly),
new("POST", "global/email-templates/{kind}/test", EndpointPolicy.GlobalAdminOnly),
// One-time retro-reconcile of misclassified historical sessions (audit 2026-07-16) —
// mutating, dry-run by default, GA only.
new("POST", "maintenance/reclassify-legacy", EndpointPolicy.GlobalAdminOnly),
@@ -95,6 +95,16 @@ namespace AutopilotMonitor.Functions.Security
/// </summary>
public const string LegacyEnterpriseTierName = "enterprise";
/// <summary>
/// Retention downgrade grace period: for this many days after a tenant loses Pro
/// (explicit downgrade or trial expiry) the retention sweep keeps enforcing the PRO
/// cap, so the downgrade does not immediately hard-delete data older than the
/// Community cap. Retention is the only entitlement with a grace — everything else
/// gates read-time immediately because it is reversible without data loss.
/// Resolved via <see cref="Services.TenantEntitlementService.GetRetentionGraceEndUtc"/>.
/// </summary>
public const int RetentionDowngradeGraceDays = 30;
private static readonly EditionEntitlements Community = new()
{
Edition = TenantEdition.Community,
@@ -162,16 +162,17 @@ namespace AutopilotMonitor.Functions.Services.Deletion
// trial expired, or whose stored value predates the cap, is enforced here even though
// the stored DataRetentionDays is left untouched. days <= 0 stays the GA-only
// "infinite" escape hatch (skipped below, never clamped).
var nowUtc = _utcNow();
var storedDays = config?.DataRetentionDays ?? 90;
var retentionDays = config == null
? 90
: TenantEntitlementService.GetEffectiveRetentionDays(config, DateTime.UtcNow);
: TenantEntitlementService.GetEffectiveRetentionDays(config, nowUtc);
if (retentionDays > 0 && retentionDays < storedDays)
{
_logger.LogWarning(
"Tenant {TenantId}: DataRetentionDays={Stored} exceeds the {Edition}-edition cap — enforcing {Effective} days",
tenantId, storedDays,
TenantEntitlementService.ResolveEdition(config!, DateTime.UtcNow), retentionDays);
TenantEntitlementService.ResolveEdition(config!, nowUtc), retentionDays);
}
if (retentionDays <= 0)
@@ -180,7 +181,7 @@ namespace AutopilotMonitor.Functions.Services.Deletion
return;
}
var cutoffUtc = DateTime.UtcNow.AddDays(-retentionDays);
var cutoffUtc = nowUtc.AddDays(-retentionDays);
// Server-bounded read: fetch one more than the per-run dispatch cap. The loop below
// only ever advances MaxEnqueuesPerTenantPerRun sessions, so loading the whole backlog
@@ -1,5 +1,4 @@
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using Microsoft.Extensions.Caching.Memory;
@@ -21,8 +20,15 @@ namespace AutopilotMonitor.Functions.Services
private readonly IMemoryCache _cache;
private readonly ILogger<DistressRateLimitService> _logger;
// Per-key locks (same proven pattern as RateLimitService)
private readonly ConcurrentDictionary<string, object> _locks = new();
// History + lock in one cache entry (same pattern as RateLimitService): same list ⇒ same lock.
private sealed class RequestBucket
{
public readonly List<DateTime> History = new();
public readonly object Sync = new();
}
// Serializes bucket creation (GetOrCreate is not atomic); hot path is a lock-free TryGetValue.
private readonly object _createLock = new();
// Layer 1: Per-IP
private const int MaxPerIp = 5;
@@ -91,17 +97,12 @@ namespace AutopilotMonitor.Functions.Services
/// </summary>
private bool CheckSlidingWindow(string cacheKey, int maxRequests, TimeSpan window, DateTime now)
{
var lockObj = _locks.GetOrAdd(cacheKey, _ => new object());
var history = _cache.GetOrCreate(cacheKey, entry =>
{
entry.SlidingExpiration = window;
entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(2);
entry.RegisterPostEvictionCallback((key, _, _, _) => _locks.TryRemove((string)key, out _));
return new List<DateTime>();
})!;
var bucket = GetOrCreateBucket(cacheKey, window);
lock (lockObj)
lock (bucket.Sync)
{
var history = bucket.History;
// Remove entries outside the window
var windowStart = now.Subtract(window);
history.RemoveAll(ts => ts < windowStart);
@@ -113,16 +114,26 @@ namespace AutopilotMonitor.Functions.Services
history.Add(now);
_cache.Set(cacheKey, history, new MemoryCacheEntryOptions
{
SlidingExpiration = window,
AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(2)
});
return true;
}
}
private RequestBucket GetOrCreateBucket(string cacheKey, TimeSpan window)
{
if (_cache.TryGetValue(cacheKey, out RequestBucket? existing) && existing != null)
return existing;
lock (_createLock)
{
return _cache.GetOrCreate(cacheKey, entry =>
{
entry.SlidingExpiration = window;
entry.AbsoluteExpirationRelativeToNow = TimeSpan.FromHours(2);
return new RequestBucket();
})!;
}
}
private bool IsCircuitOpen(DateTime now)
{
lock (_globalLock)
@@ -0,0 +1,259 @@
using System.Net.Http.Json;
using System.Text.Json;
using System.Text.Json.Serialization;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services;
/// <summary>
/// Sends the product's transactional emails (tenant-activation welcome, post-offboarding
/// farewell). Best-effort: failures are logged as warnings and never propagated.
/// <para>
/// The class and its callers are provider-neutral. Everything provider-specific is confined
/// to the <c>Email:*</c> configuration section and <see cref="SendViaMandrillAsync"/>, so a
/// future provider swap touches the transport method and the app settings only.
/// Current provider: Mailchimp Transactional (Mandrill) — <c>POST {Email:Endpoint}</c> with the
/// API key in the JSON body. Open/click tracking is explicitly disabled so the provider
/// receives nothing beyond the recipient address and the tenant domain (trust-page claim).
/// </para>
/// <para>
/// Configuration (App Settings use <c>Email__Key</c>):
/// <c>Email:ApiKey</c> (required; empty ⇒ every send is a logged no-op),
/// <c>Email:Endpoint</c> (default <see cref="DefaultEndpoint"/>),
/// <c>Email:FromAddress</c> (default <see cref="DefaultFromAddress"/>),
/// <c>Email:FromName</c> (default <see cref="DefaultFromName"/>).
/// </para>
/// </summary>
public class EmailService : IEmailService, IOffboardFarewellEmailSender
{
public const string ApiKeyConfigKey = "Email:ApiKey";
public const string DefaultEndpoint = "https://mandrillapp.com/api/1.0/messages/send";
public const string DefaultFromAddress = "noreply@autopilotmonitor.com";
public const string DefaultFromName = "Autopilot Monitor";
private static readonly JsonSerializerOptions JsonOptions = new()
{
DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull,
};
private readonly HttpClient _http;
private readonly IEmailTemplateProvider _templates;
private readonly ILogger<EmailService> _logger;
private readonly string _apiKey;
private readonly string _endpoint;
private readonly string _fromAddress;
private readonly string _fromName;
public EmailService(
HttpClient http,
IConfiguration configuration,
IEmailTemplateProvider templates,
ILogger<EmailService> logger)
{
_http = http;
_templates = templates;
_logger = logger;
_apiKey = configuration[ApiKeyConfigKey] ?? string.Empty;
_endpoint = FirstNonBlank(configuration["Email:Endpoint"], DefaultEndpoint);
_fromAddress = FirstNonBlank(configuration["Email:FromAddress"], DefaultFromAddress);
_fromName = FirstNonBlank(configuration["Email:FromName"], DefaultFromName);
}
/// <inheritdoc />
public async Task SendPreviewApprovedEmailAsync(string toEmail, string domainName, CancellationToken ct = default)
{
if (string.IsNullOrWhiteSpace(_apiKey))
{
_logger.LogDebug("{ConfigKey} not configured — skipping preview approval email", ApiKeyConfigKey);
return;
}
if (string.IsNullOrWhiteSpace(toEmail))
{
_logger.LogDebug("No notification email set — skipping preview approval email for {Domain}", domainName);
return;
}
var sent = await SendViaMandrillAsync(
toEmail,
EmailTemplates.PreviewApprovedSubject,
await _templates.GetHtmlAsync(EmailTemplateKind.Welcome, domainName, ct),
tag: "welcome",
ct);
if (sent)
{
_logger.LogInformation(
"Preview approval email sent to {ToEmail} for domain {Domain}",
toEmail, domainName);
}
}
/// <summary>
/// Sends the post-offboarding "sorry to see you go" farewell email.
/// No-op if the API key or recipient email is not configured. Best-effort: failures
/// are logged as warnings and never propagated (the offboarding correctness contract
/// does not depend on email delivery).
/// </summary>
public async Task SendAsync(string toEmail, string domainName, string tenantId, CancellationToken ct = default)
{
if (string.IsNullOrWhiteSpace(_apiKey))
{
_logger.LogDebug(
"{ConfigKey} not configured — skipping offboard farewell email for tenant {TenantId}",
ApiKeyConfigKey, tenantId);
return;
}
if (string.IsNullOrWhiteSpace(toEmail))
{
_logger.LogDebug(
"No notification email captured — skipping offboard farewell email for tenant {TenantId} ({Domain})",
tenantId, domainName);
return;
}
var sent = await SendViaMandrillAsync(
toEmail,
EmailTemplates.OffboardingFarewellSubject,
await _templates.GetHtmlAsync(EmailTemplateKind.Farewell, domainName, ct),
tag: "offboarding-farewell",
ct);
if (sent)
{
_logger.LogInformation(
"Offboard farewell email sent to {ToEmail} for tenant {TenantId} ({Domain})",
toEmail, tenantId, domainName);
}
}
/// <inheritdoc />
public async Task<bool> SendTestAsync(EmailTemplateKind kind, string toEmail, string domainName, string? draftHtml, CancellationToken ct = default)
{
if (string.IsNullOrWhiteSpace(_apiKey))
{
_logger.LogWarning("{ConfigKey} not configured — cannot send {Kind} test email", ApiKeyConfigKey, kind);
return false;
}
var html = draftHtml is null
? await _templates.GetHtmlAsync(kind, domainName, ct)
: EmailTemplateService.Render(draftHtml, domainName);
var sent = await SendViaMandrillAsync(toEmail, EmailTemplateService.Subject(kind), html, tag: "test", ct);
if (sent)
_logger.LogInformation("{Kind} test email sent to {ToEmail} (draft={IsDraft})", kind, toEmail, draftHtml is not null);
return sent;
}
/// <summary>
/// The only provider-specific code path. Posts a Mandrill <c>messages/send</c> request and
/// interprets the per-recipient result array: <c>sent</c>/<c>queued</c>/<c>scheduled</c>
/// count as success, <c>rejected</c>/<c>invalid</c> as failure (with the provider's
/// <c>reject_reason</c> in the warning). Never throws.
/// </summary>
private async Task<bool> SendViaMandrillAsync(string toEmail, string subject, string html, string tag, CancellationToken ct)
{
try
{
var request = new MandrillSendRequest
{
Key = _apiKey,
Message = new MandrillMessage
{
FromEmail = _fromAddress,
FromName = _fromName,
To = new[] { new MandrillRecipient { Email = toEmail, Type = "to" } },
Subject = subject,
Html = html,
AutoText = true,
TrackOpens = false,
TrackClicks = false,
Tags = new[] { tag },
},
};
using var response = await _http.PostAsJsonAsync(_endpoint, request, JsonOptions, ct);
var body = await response.Content.ReadAsStringAsync(ct);
if (!response.IsSuccessStatusCode)
{
_logger.LogWarning(
"Email provider returned {StatusCode} for {Tag} mail to {ToEmail}: {Body}",
(int)response.StatusCode, tag, toEmail, Truncate(body));
return false;
}
var results = JsonSerializer.Deserialize<MandrillSendResult[]>(body, JsonOptions);
var result = results?.FirstOrDefault();
if (result is null)
{
_logger.LogWarning(
"Email provider returned an empty result for {Tag} mail to {ToEmail}: {Body}",
tag, toEmail, Truncate(body));
return false;
}
if (!IsAcceptedStatus(result.Status))
{
_logger.LogWarning(
"Email provider did not accept {Tag} mail to {ToEmail}: status={Status} reason={Reason}",
tag, toEmail, result.Status, result.RejectReason ?? "n/a");
return false;
}
return true;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to send {Tag} mail to {ToEmail}", tag, toEmail);
return false;
}
}
private static bool IsAcceptedStatus(string? status)
=> status is "sent" or "queued" or "scheduled";
private static string FirstNonBlank(string? value, string fallback)
=> string.IsNullOrWhiteSpace(value) ? fallback : value.Trim();
private static string Truncate(string value)
=> value.Length <= 500 ? value : value[..500] + "…";
// --- Mandrill wire format (https://mailchimp.com/developer/transactional/api/messages/send-new-message/) ---
private sealed class MandrillSendRequest
{
[JsonPropertyName("key")] public string Key { get; set; } = string.Empty;
[JsonPropertyName("message")] public MandrillMessage Message { get; set; } = new();
}
private sealed class MandrillMessage
{
[JsonPropertyName("from_email")] public string FromEmail { get; set; } = string.Empty;
[JsonPropertyName("from_name")] public string FromName { get; set; } = string.Empty;
[JsonPropertyName("to")] public MandrillRecipient[] To { get; set; } = Array.Empty<MandrillRecipient>();
[JsonPropertyName("subject")] public string Subject { get; set; } = string.Empty;
[JsonPropertyName("html")] public string Html { get; set; } = string.Empty;
[JsonPropertyName("auto_text")] public bool AutoText { get; set; }
[JsonPropertyName("track_opens")] public bool TrackOpens { get; set; }
[JsonPropertyName("track_clicks")] public bool TrackClicks { get; set; }
[JsonPropertyName("tags")] public string[] Tags { get; set; } = Array.Empty<string>();
}
private sealed class MandrillRecipient
{
[JsonPropertyName("email")] public string Email { get; set; } = string.Empty;
[JsonPropertyName("type")] public string Type { get; set; } = "to";
}
private sealed class MandrillSendResult
{
[JsonPropertyName("email")] public string? Email { get; set; }
[JsonPropertyName("status")] public string? Status { get; set; }
[JsonPropertyName("reject_reason")] public string? RejectReason { get; set; }
[JsonPropertyName("_id")] public string? Id { get; set; }
}
}
@@ -0,0 +1,151 @@
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Services;
/// <summary>The two transactional mails the product sends.</summary>
public enum EmailTemplateKind
{
Welcome,
Farewell,
}
/// <summary>
/// Resolves the HTML body for a transactional mail: the operator override when one is
/// stored, otherwise the built-in template. Consumed by <see cref="EmailService"/>.
/// </summary>
public interface IEmailTemplateProvider
{
Task<string> GetHtmlAsync(EmailTemplateKind kind, string domainName, CancellationToken ct = default);
}
/// <summary>
/// Operator-editable email templates. Overrides live in the PreviewConfig table (partition
/// "EmailTemplates", row key = kind) and are cached for 5 minutes; the built-in templates in
/// <see cref="EmailTemplates"/> remain the fallback. Subjects are not editable — they are pinned
/// constants.
/// <para>
/// Rendering contract: the only placeholder is <c>{{domainName}}</c>; an empty domain renders as
/// "your organization", exactly like the built-ins. Resolution is fail-soft — a storage error
/// falls back to the built-in template so activation/offboarding mails keep going out.
/// </para>
/// </summary>
public class EmailTemplateService : IEmailTemplateProvider
{
public const string DomainPlaceholder = "{{domainName}}";
public const string EmptyDomainLabel = "your organization";
/// <summary>Azure Table string properties cap at 32K UTF-16 chars; keep headroom.</summary>
public const int MaxHtmlLength = 30_000;
private static readonly TimeSpan CacheDuration = TimeSpan.FromMinutes(5);
private readonly IConfigRepository _configRepo;
private readonly IMemoryCache _cache;
private readonly ILogger<EmailTemplateService> _logger;
public EmailTemplateService(IConfigRepository configRepo, IMemoryCache cache, ILogger<EmailTemplateService> logger)
{
_configRepo = configRepo;
_cache = cache;
_logger = logger;
}
public static bool TryParseKind(string? value, out EmailTemplateKind kind)
{
switch (value?.Trim().ToLowerInvariant())
{
case "welcome": kind = EmailTemplateKind.Welcome; return true;
case "farewell": kind = EmailTemplateKind.Farewell; return true;
default: kind = default; return false;
}
}
public static string KindKey(EmailTemplateKind kind) => kind == EmailTemplateKind.Welcome ? "welcome" : "farewell";
public static string Subject(EmailTemplateKind kind) => kind == EmailTemplateKind.Welcome
? EmailTemplates.PreviewApprovedSubject
: EmailTemplates.OffboardingFarewellSubject;
/// <summary>Built-in template with the placeholder left in place (for the editor).</summary>
public static string BuiltInRaw(EmailTemplateKind kind) => kind == EmailTemplateKind.Welcome
? EmailTemplates.GetPreviewApprovedHtml(DomainPlaceholder)
: EmailTemplates.GetOffboardingFarewellHtml(DomainPlaceholder);
/// <summary>Substitutes the placeholder; an empty domain becomes "your organization".</summary>
public static string Render(string rawHtml, string? domainName)
{
var display = string.IsNullOrWhiteSpace(domainName) ? EmptyDomainLabel : domainName.Trim();
return rawHtml.Replace(DomainPlaceholder, display, StringComparison.Ordinal);
}
/// <summary>Returns null when the HTML is acceptable, otherwise the validation message.</summary>
public static string? Validate(string? html)
{
if (string.IsNullOrWhiteSpace(html)) return "Template HTML must not be empty.";
if (html.Length > MaxHtmlLength) return $"Template HTML exceeds {MaxHtmlLength:N0} characters ({html.Length:N0}).";
return null;
}
public async Task<string> GetHtmlAsync(EmailTemplateKind kind, string domainName, CancellationToken ct = default)
{
var overrideEntry = await GetOverrideAsync(kind, failSoft: true);
return overrideEntry is null
? Render(BuiltInRaw(kind), domainName)
: Render(overrideEntry.Html, domainName);
}
/// <summary>
/// The stored override, or null for built-in. <paramref name="failSoft"/> swallows storage
/// errors (send path must never depend on the override store); the admin endpoints pass
/// false so a broken store surfaces as an error instead of silently showing the built-in.
/// </summary>
public virtual async Task<EmailTemplateOverride?> GetOverrideAsync(EmailTemplateKind kind, bool failSoft = false)
{
var key = CacheKey(kind);
if (_cache.TryGetValue(key, out EmailTemplateOverride? cached))
return cached;
try
{
var entry = await _configRepo.GetEmailTemplateOverrideAsync(KindKey(kind));
// null is a valid (and the common) value — cache it too so the send path
// doesn't hit storage on every mail.
_cache.Set(key, entry, CacheDuration);
return entry;
}
catch (Exception ex) when (failSoft)
{
_logger.LogWarning(ex, "Failed to read {Kind} email template override — using built-in template", kind);
return null;
}
}
public virtual async Task<EmailTemplateOverride> SaveOverrideAsync(EmailTemplateKind kind, string html, string updatedBy)
{
var error = Validate(html);
if (error is not null) throw new ArgumentException(error, nameof(html));
var entry = new EmailTemplateOverride
{
Kind = KindKey(kind),
Html = html,
UpdatedBy = updatedBy,
UpdatedUtc = DateTime.UtcNow,
};
await _configRepo.SaveEmailTemplateOverrideAsync(entry);
_cache.Remove(CacheKey(kind));
_logger.LogInformation("Email template override saved: {Kind} by {UpdatedBy} ({Length} chars)", kind, updatedBy, html.Length);
return entry;
}
public virtual async Task DeleteOverrideAsync(EmailTemplateKind kind, string deletedBy)
{
await _configRepo.DeleteEmailTemplateOverrideAsync(KindKey(kind));
_cache.Remove(CacheKey(kind));
_logger.LogInformation("Email template override reset to built-in: {Kind} by {DeletedBy}", kind, deletedBy);
}
private static string CacheKey(EmailTemplateKind kind) => $"email-template-override:{KindKey(kind)}";
}
@@ -3,7 +3,7 @@ using AutopilotMonitor.Shared;
namespace AutopilotMonitor.Functions.Services;
/// <summary>
/// Email templates for Resend notifications.
/// Templates for the product's transactional emails (sent via <see cref="EmailService"/>).
/// Keep all email content here for easy maintenance.
/// </summary>
public static class EmailTemplates
@@ -37,12 +37,33 @@ public static class EmailTemplates
<tr><td align=""center"">
<table width=""600"" cellpadding=""0"" cellspacing=""0"" style=""background-color:#ffffff; border-radius:12px; overflow:hidden; box-shadow:0 4px 6px rgba(0,0,0,0.07);"">
<!-- Header -->
<!-- Header: image-free. Base for every client: solid bgcolor band, text wordmark, brand mark
drawn with CSS borders (trapezoid + a green wedge for the slanted right edge; must stay
<div>s — Outlook paints transparent <td> borders white). Progressive enhancement where
supported: ONE linear-gradient (hex colours, % stops — the only form Outlook web/iOS
renders; multiple layers or rgba stops are dropped) forms the facets, and clip-path cuts
the slanted bottom edge (Apple Mail only). No bgcolor inside the mark: Outlook dark mode
inverts cell backgrounds but leaves border colours alone.
Geometry is the approved reference (scratch header-render.html): facets
light right of (40%,0)->(70%,100%), dark right of (55%,0)->(85%,100%) and left of
(78%,0)->(100%,60%) — the last one is dropped for the single-gradient constraint.
Change only against a side-by-side render. -->
<tr>
<td style=""background:linear-gradient(135deg,#1e8a4c,#33b161); padding:32px 40px; text-align:center;"">
<h1 style=""color:#ffffff; margin:0; font-size:24px; font-weight:700;"">Autopilot Monitor</h1>
<p style=""color:#c9efd8; margin:8px 0 0; font-size:14px;"">Windows Autopilot monitoring</p>
</td>
<td bgcolor=""#33b161"" style=""background-color:#33b161; background-image:linear-gradient(39.8deg, #33b161 0%, #33b161 53.8%, #3cbb6a 53.8%, #3cbb6a 65.4%, #2ea85a 65.4%, #2ea85a 100%); clip-path:polygon(0 0, 100% 0, 100% 100%, 0 82%); padding:28px 32px 52px; line-height:0; font-size:0;"">
<table cellpadding=""0"" cellspacing=""0"" border=""0""><tr>
<td valign=""middle"" style=""padding:0 12px 0 0; line-height:0; font-size:0;"">
<!-- brand mark, 34x33, three non-overlapping rows (Outlook ignores negative margins and
paints transparent td borders white, so: div triangles + one white td only) -->
<div style=""width:0; height:0; border-left:31px solid #ffffff; border-top:13px solid transparent; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div>
<table cellpadding=""0"" cellspacing=""0"" border=""0"" style=""border-collapse:collapse;""><tr>
<td style=""padding:0; font-size:0; line-height:0;""><div style=""width:0; height:18px; border-left:31px solid #ffffff; border-top:0; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div></td>
<td style=""padding:0; font-size:0; line-height:0;""><div style=""width:0; height:0; border-left:3px solid #ffffff; border-top:18px solid transparent; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div></td>
</tr></table>
<div style=""width:0; height:0; border-right:34px solid #ffffff; border-bottom:2px solid transparent; border-left:0; border-top:0; display:block; font-size:0; line-height:0;""></div>
</td>
<td valign=""middle"" style=""color:#ffffff; font-size:22px; font-weight:600; letter-spacing:-0.2px; line-height:1; white-space:nowrap; font-family:-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;"">Autopilot Monitor</td>
</tr></table>
</td>
</tr>
<!-- Body -->
@@ -56,14 +77,14 @@ public static class EmailTemplates
<p style=""color:#374151; font-size:15px; line-height:1.6; margin:0 0 24px;"">
Thank you for taking the time to try Autopilot Monitor. Every organization that
puts it to work in a real environment helps shape what it becomes &mdash; we're
puts it to work in a real environment helps shape what it becomes &ndash; we're
genuinely grateful you gave it a chance.
</p>
<!-- Gentle feedback ask -->
<p style=""color:#374151; font-size:15px; line-height:1.6; margin:0 0 12px;"">
If something didn't fit &mdash; a missing feature, a bug that got in the way, or
simply a better alternative &mdash; we'd love to hear about it. A couple of honest
If something didn't fit &ndash; a missing feature, a bug that got in the way, or
simply a better alternative &ndash; we'd love to hear about it. A couple of honest
sentences help more than you'd think:
</p>
@@ -78,7 +99,7 @@ public static class EmailTemplates
<p style=""color:#6b7280; font-size:13px; line-height:1.6; margin:0;"">
This is the last email you'll receive from us. This mailbox doesn't accept
replies &mdash; please use the links above if you'd like to get in touch.
replies &ndash; please use the links above if you'd like to get in touch.
</p>
</td>
</tr>
@@ -115,12 +136,33 @@ public static class EmailTemplates
<tr><td align=""center"">
<table width=""600"" cellpadding=""0"" cellspacing=""0"" style=""background-color:#ffffff; border-radius:12px; overflow:hidden; box-shadow:0 4px 6px rgba(0,0,0,0.07);"">
<!-- Header -->
<!-- Header: image-free. Base for every client: solid bgcolor band, text wordmark, brand mark
drawn with CSS borders (trapezoid + a green wedge for the slanted right edge; must stay
<div>s — Outlook paints transparent <td> borders white). Progressive enhancement where
supported: ONE linear-gradient (hex colours, % stops — the only form Outlook web/iOS
renders; multiple layers or rgba stops are dropped) forms the facets, and clip-path cuts
the slanted bottom edge (Apple Mail only). No bgcolor inside the mark: Outlook dark mode
inverts cell backgrounds but leaves border colours alone.
Geometry is the approved reference (scratch header-render.html): facets
light right of (40%,0)->(70%,100%), dark right of (55%,0)->(85%,100%) and left of
(78%,0)->(100%,60%) — the last one is dropped for the single-gradient constraint.
Change only against a side-by-side render. -->
<tr>
<td style=""background:linear-gradient(135deg,#1e8a4c,#33b161); padding:32px 40px; text-align:center;"">
<h1 style=""color:#ffffff; margin:0; font-size:24px; font-weight:700;"">Autopilot Monitor</h1>
<p style=""color:#c9efd8; margin:8px 0 0; font-size:14px;"">Windows Autopilot monitoring</p>
</td>
<td bgcolor=""#33b161"" style=""background-color:#33b161; background-image:linear-gradient(39.8deg, #33b161 0%, #33b161 53.8%, #3cbb6a 53.8%, #3cbb6a 65.4%, #2ea85a 65.4%, #2ea85a 100%); clip-path:polygon(0 0, 100% 0, 100% 100%, 0 82%); padding:28px 32px 52px; line-height:0; font-size:0;"">
<table cellpadding=""0"" cellspacing=""0"" border=""0""><tr>
<td valign=""middle"" style=""padding:0 12px 0 0; line-height:0; font-size:0;"">
<!-- brand mark, 34x33, three non-overlapping rows (Outlook ignores negative margins and
paints transparent td borders white, so: div triangles + one white td only) -->
<div style=""width:0; height:0; border-left:31px solid #ffffff; border-top:13px solid transparent; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div>
<table cellpadding=""0"" cellspacing=""0"" border=""0"" style=""border-collapse:collapse;""><tr>
<td style=""padding:0; font-size:0; line-height:0;""><div style=""width:0; height:18px; border-left:31px solid #ffffff; border-top:0; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div></td>
<td style=""padding:0; font-size:0; line-height:0;""><div style=""width:0; height:0; border-left:3px solid #ffffff; border-top:18px solid transparent; border-right:0; border-bottom:0; display:block; font-size:0; line-height:0;""></div></td>
</tr></table>
<div style=""width:0; height:0; border-right:34px solid #ffffff; border-bottom:2px solid transparent; border-left:0; border-top:0; display:block; font-size:0; line-height:0;""></div>
</td>
<td valign=""middle"" style=""color:#ffffff; font-size:22px; font-weight:600; letter-spacing:-0.2px; line-height:1; white-space:nowrap; font-family:-apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;"">Autopilot Monitor</td>
</tr></table>
</td>
</tr>
<!-- Body -->
@@ -129,8 +171,8 @@ public static class EmailTemplates
<h2 style=""color:#111827; margin:0 0 16px; font-size:20px;"">Welcome to Autopilot Monitor!</h2>
<p style=""color:#374151; font-size:15px; line-height:1.6; margin:0 0 16px;"">
Great news &mdash; access for <strong>{displayDomain}</strong> has been activated and is ready to use.
You can now sign in and start monitoring your Windows Autopilot enrollments in real time.
Great news &ndash; access for <strong>{displayDomain}</strong> has been activated and is ready to use.
You can now <a href=""{Constants.PortalBaseUrl}"" target=""_blank"" style=""color:#1e8a4c; text-decoration:underline;"">sign in</a> and start monitoring your Windows Autopilot enrollments in real time.
</p>
<p style=""color:#374151; font-size:15px; line-height:1.6; margin:0 0 24px;"">
@@ -149,13 +191,6 @@ public static class EmailTemplates
</tr>
</table>
<!-- Active development note -->
<div style=""background-color:#fef3c7; border:1px solid #fde68a; border-radius:8px; padding:16px 20px; margin:0 0 24px;"">
<p style=""color:#92400e; font-size:14px; line-height:1.5; margin:0;"">
<strong>Please note:</strong> Autopilot Monitor is in active development. Some features are still being built
and things may occasionally not work as expected. Your patience and understanding are greatly appreciated!
</p>
</div>
<!-- Feedback -->
<p style=""color:#374151; font-size:15px; line-height:1.6; margin:0 0 12px;"">
@@ -169,7 +204,7 @@ public static class EmailTemplates
</ul>
<p style=""color:#6b7280; font-size:14px; line-height:1.6; margin:0;"">
Thanks for being part of the journey &mdash; enjoy Autopilot Monitor!
Thanks for being part of the journey &ndash; enjoy Autopilot Monitor!
</p>
</td>
</tr>
@@ -92,6 +92,7 @@ namespace AutopilotMonitor.Functions.Services
bool AccountSetupAllSucceeded,
bool HasExplicitFailure,
bool HasTerminalComplete,
bool HasEnrollmentComplete,
bool HasAgentEmergencyBreak,
bool DesktopArrived,
bool HelloResolved,
@@ -113,7 +114,7 @@ namespace AutopilotMonitor.Functions.Services
bool deviceAll = false;
int acctBestN = 0, acctBestM = 0;
bool acctFallbackAll = false;
bool hasFailure = false, hasComplete = false, hasEmergencyBreak = false;
bool hasFailure = false, hasComplete = false, hasEnrollmentComplete = false, hasEmergencyBreak = false;
bool desktopArrived = false, helloResolved = false;
bool realmJoinDetected = false, realmJoinResolved = false;
bool sawHelloPolicyEnabled = false, sawHelloPolicyDisabled = false;
@@ -135,7 +136,12 @@ namespace AutopilotMonitor.Functions.Services
hasFailure = true;
}
else if (Eq(type, "enrollment_complete") || Eq(type, "whiteglove_complete"))
{
hasComplete = true;
// whiteglove_complete only proves Part 1 (technician flow) — the WhiteGlove
// awaiting-user gate needs to know whether a REAL enrollment completion exists.
if (Eq(type, "enrollment_complete")) hasEnrollmentComplete = true;
}
else if (Eq(type, "agent_emergency_break"))
hasEmergencyBreak = true;
else if (Eq(type, "desktop_arrived"))
@@ -217,6 +223,7 @@ namespace AutopilotMonitor.Functions.Services
AccountSetupAllSucceeded: acctAll,
HasExplicitFailure: hasFailure,
HasTerminalComplete: hasComplete,
HasEnrollmentComplete: hasEnrollmentComplete,
HasAgentEmergencyBreak: hasEmergencyBreak,
DesktopArrived: desktopArrived,
HelloResolved: helloResolved,
@@ -226,6 +233,74 @@ namespace AutopilotMonitor.Functions.Services
RealmJoinResolved: realmJoinResolved);
}
/// <summary>
/// WhiteGlove Part-2 awaiting-user gate (fairstone.ca analysis 2026-08-21). After the
/// reseal-reboot the session is BY DEFINITION in the user-driven final phase: the agent
/// resumed (<c>whiteglove_resumed</c> → ResumedAt, the only writer of that column), the
/// Part-2 Device ESP re-confirms in minutes, and the device sits at the logon screen.
/// Technicians routinely power the machine off there to box it — well before the 10-min
/// hybrid-login evidence window — so agent silence in this state means "sealed / powered
/// off awaiting the end user", not a stall. The gate demands the full evidence set:
/// pre-provisioned + resumed + Device Setup provisioned, and NO trace of a real user
/// since the resume (no Account Setup progress, no real-user desktop, no Hello terminal,
/// no enrollment completion) and no explicit failure. Any user evidence keeps the honest
/// Stalled/timeout handling — a stall after the user showed up is a real stall.
/// Shared by the maintenance sweep stage 1, the <c>session_stalled</c> ingest mapping and
/// <see cref="ClassifyTimedOutSession"/> so all paths apply one evidence rule.
/// </summary>
public static bool IsWhiteGloveAwaitingUser(
in EspProvisioningRollup rollup, bool isPreProvisioned, DateTime? resumedAt)
=> isPreProvisioned
&& resumedAt.HasValue
&& rollup.DeviceSetupAllSucceeded
&& !rollup.HasExplicitFailure
&& !rollup.HasEnrollmentComplete
&& rollup.AccountSetupSucceededCount == 0
&& !rollup.DesktopArrived
&& !rollup.HelloResolved;
/// <summary>
/// Customer-facing reason for a WhiteGlove Part-2 session parked as AwaitingUser by the
/// silence/stall paths. One authoring spot so the sweep, the ingest mapping and the
/// timeout classifier show the identical text.
/// </summary>
public static string WhiteGloveAwaitingUserReason(in EspProvisioningRollup rollup)
{
var acct = rollup.AccountSetupTotal > 0
? $" (Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal})"
: " (Account Setup not yet started)";
return "Pre-provisioning (WhiteGlove) completed; no user sign-in since Part 2 resumed — " +
$"device presumably sealed or powered off awaiting the end user{acct}";
}
/// <summary>
/// Self-deploying (kiosk / shared-device) profile gate. Such a profile has NO user phase:
/// Device ESP reaching all-succeeded IS the enrollment's end, and the agent's own
/// SelfDeploying terminal only adds a 5-min confirmation window on top of it. When the
/// agent goes silent after that point (reboot into the kiosk autologon, powered off and
/// boxed for the school term, WiFi never re-associating for the relaunch — tenant
/// aebdce78, 836 sessions 2026-08-17..21), "awaiting user / Account Setup" is a factual
/// error and the later Incomplete("Account Setup 0/5") is worse: nobody will ever sign
/// in, the devices are finished and in service. Demands the registration-time
/// registry-confirmed profile flag (CloudAssignedOobeConfig 0x20|0x40), Device Setup
/// all-succeeded and no explicit failure. Shared by the sweep (stage 1 + 2), the
/// max-lifetime ingest verdict and the retro-reclassification so one evidence rule applies.
/// </summary>
public static bool IsSelfDeployingProvisioned(in EspProvisioningRollup rollup, bool isSelfDeployingProfile)
=> isSelfDeployingProfile
&& rollup.DeviceSetupAllSucceeded
&& !rollup.HasExplicitFailure;
/// <summary>
/// Customer-facing Succeeded-reconcile reason for <see cref="IsSelfDeployingProvisioned"/>.
/// Carries the silence-transparency clause like every other reconcile reason.
/// </summary>
public static string SelfDeployingReconcileReason(DateTime startedAtUtc, DateTime nowUtc, DateTime? lastEventAtUtc)
=> AppendReconcileTiming(
"Reconciled: self-deploying profile — Device Setup fully provisioned (this profile has no user / Account Setup phase); " +
"agent went silent before confirming completion",
startedAtUtc, nowUtc, lastEventAtUtc);
/// <summary>
/// Decide the honest terminal (or non-terminal) state for a timed-out session. See the
/// decision table in the design note. Returns the target status and a human-readable
@@ -242,27 +317,66 @@ namespace AutopilotMonitor.Functions.Services
/// mechanism declared success too early" — session efbc17ff. Falls back to
/// <paramref name="startedAtUtc"/> when unknown (same fallback the stalled-marker uses).
/// </param>
public static (SessionStatus Status, string Reason) ClassifyTimedOutSession(
/// <param name="isPreProvisioned">Session's IsPreProvisioned flag (WhiteGlove Part-2 gate input).</param>
/// <param name="resumedAt">Session's ResumedAt — set exclusively by the whiteglove_resumed ingest.</param>
/// <param name="isSelfDeployingProfile">Session's IsSelfDeployingProfile flag (rule 1c gate input).</param>
public static (SessionStatus Status, string Reason, string Rule) ClassifyTimedOutSession(
EspProvisioningRollup rollup, DateTime startedAtUtc, DateTime nowUtc, int graceHours,
DateTime? lastEventAtUtc = null)
DateTime? lastEventAtUtc = null, bool isPreProvisioned = false, DateTime? resumedAt = null,
bool isSelfDeployingProfile = false)
{
// 1. An explicit terminal failure event is a real failure (defensive — such a session
// would normally already be terminal via ingest and not reach the sweep).
if (rollup.HasExplicitFailure)
return (SessionStatus.Failed, "Enrollment reported an explicit failure before timeout");
return (SessionStatus.Failed, "Enrollment reported an explicit failure before timeout", ClassifierRules.R1ExplicitFailure);
// 1b. WhiteGlove Part 2 with zero user evidence since the resume: the device is parked
// between technician and end user (sealed box / powered off at the logon screen).
// Must run BEFORE rule 2 — the Part-1 whiteglove_complete in the same event stream
// sets HasTerminalComplete and would otherwise reconcile the session to Succeeded
// with a reason claiming Account Setup completed, which never happened. Within
// grace the honest label is AwaitingUser; past grace the Part-1 success stands as
// the session outcome (the user phase is tracked by the session the device
// registers when it is finally unboxed), stated with an honest WhiteGlove reason.
if (IsWhiteGloveAwaitingUser(rollup, isPreProvisioned, resumedAt))
{
if ((nowUtc - startedAtUtc).TotalHours < graceHours)
return (SessionStatus.AwaitingUser, WhiteGloveAwaitingUserReason(rollup), ClassifierRules.R1bWhiteGloveAwaiting);
return (SessionStatus.Succeeded, AppendReconcileTiming(
$"Reconciled at timeout: pre-provisioning (WhiteGlove Part 1) completed — no user sign-in observed within {graceHours}h grace",
startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R1bWhiteGloveSucceeded);
}
// 1c. Self-deploying profile with Device Setup fully provisioned: there is no user phase
// to wait for, so neither AwaitingUser nor the grace window apply — the device is
// done (see IsSelfDeployingProvisioned). Runs before rule 2 so the reason names the
// profile instead of claiming "Account Setup completed"; real user evidence cannot
// contradict it (a kiosk autologon is not a user) and an explicit failure already
// returned in rule 1.
if (IsSelfDeployingProvisioned(rollup, isSelfDeployingProfile))
return (SessionStatus.Succeeded, SelfDeployingReconcileReason(startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R1cSelfDeploying);
// 2. Account Setup fully succeeded (or a terminal completion event) → reconcile to success.
if (rollup.AccountSetupAllSucceeded || rollup.HasTerminalComplete)
// Once Part 2 has resumed, the Part-1 whiteglove_complete in the same stream no longer
// proves the session's outcome — only a real enrollment_complete does. Without this,
// a resumed session with partial user progress (e.g. Account Setup 1/5, then silence)
// reconciled to Succeeded on Part-1 evidence, with a reason claiming Account Setup
// completed; rules 4-6 below hold the honest verdicts for that shape.
var terminalCompleteProvesOutcome = rollup.HasEnrollmentComplete
|| (rollup.HasTerminalComplete && !resumedAt.HasValue);
if (rollup.AccountSetupAllSucceeded || terminalCompleteProvesOutcome)
return (SessionStatus.Succeeded, AppendReconcileTiming(
"Reconciled at timeout: Account Setup completed (all subcategories succeeded / enrollment_complete observed)",
startedAtUtc, nowUtc, lastEventAtUtc));
startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R2AccountSetupComplete);
// 3. The agent's absolute-age emergency break fired — the agent has cleaned up and exited, so
// nothing more will ever arrive for this session. Terminalize NOW (skip the AwaitingUser grace):
// it did not complete (2) and did not explicitly fail (1), so the honest verdict is Incomplete.
if (rollup.HasAgentEmergencyBreak)
return (SessionStatus.Incomplete,
"Agent emergency break fired (absolute session-age cap) — agent gone without completion");
"Agent emergency break fired (absolute session-age cap) — agent gone without completion",
ClassifierRules.R3EmergencyBreak);
// 4. The user demonstrably finished setup: a real-user desktop was observed AND
// Windows Hello reached a positive terminal (provisioned or skipped). Those are
@@ -297,7 +411,7 @@ namespace AutopilotMonitor.Functions.Services
: "agent went silent before reporting completion";
return (SessionStatus.Succeeded, AppendReconcileTiming(
$"Reconciled at timeout: user completed setup ({evidence}) — {detail}",
startedAtUtc, nowUtc, lastEventAtUtc));
startedAtUtc, nowUtc, lastEventAtUtc), ClassifierRules.R4DesktopHello);
}
// 5. Device Setup fully provisioned (device is AADJ + MDM-enrolled), user phase pending.
@@ -310,7 +424,8 @@ namespace AutopilotMonitor.Functions.Services
? $" (Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal})"
: " (Account Setup not yet started)";
return (SessionStatus.AwaitingUser,
$"Device Setup completed; awaiting user / Account Setup phase — agent silent, within {graceHours}h grace{acct}");
$"Device Setup completed; awaiting user / Account Setup phase — agent silent, within {graceHours}h grace{acct}",
ClassifierRules.R5DeviceSetupAwaiting);
}
// "0/0" would suggest a parsed rollup that never existed — when no Account Setup
@@ -319,12 +434,14 @@ namespace AutopilotMonitor.Functions.Services
? $"last Account Setup {rollup.AccountSetupSucceededCount}/{rollup.AccountSetupTotal}"
: "Account Setup progress never observed";
return (SessionStatus.Incomplete,
$"No completion signal within {graceHours}h grace after Device Setup completed ({acctDetail})");
$"No completion signal within {graceHours}h grace after Device Setup completed ({acctDetail})",
ClassifierRules.R5DeviceSetupIncomplete);
}
// 6. Silent before Device Setup completed, with no explicit failure → unknown, not a failure.
return (SessionStatus.Incomplete,
"No Device Setup completion or explicit failure signal observed before timeout");
"No Device Setup completion or explicit failure signal observed before timeout",
ClassifierRules.R6Fallthrough);
}
/// <summary>
@@ -14,7 +14,15 @@ namespace AutopilotMonitor.Functions.Services
foreach (var evt in storedEvents)
{
if (!classification.EarliestEventTimestamp.HasValue || evt.Timestamp < classification.EarliestEventTimestamp.Value)
// System-timeline-watcher rows (clock steps, sleep episodes) are backfilled
// from the Windows event log with original timestamps up to 24h before the
// agent started — environment observation, not enrollment activity. They must
// not pull the session anchor toward pre-enrollment OOBE idle time (mirrors
// TableStorageService.IsSessionAnchorEligible on the Events-table probe;
// field case d0c5b672).
var anchorEligible = evt.Source != "SystemTimelineWatcher";
if (anchorEligible &&
(!classification.EarliestEventTimestamp.HasValue || evt.Timestamp < classification.EarliestEventTimestamp.Value))
classification.EarliestEventTimestamp = evt.Timestamp;
if (!classification.LatestEventTimestamp.HasValue || evt.Timestamp > classification.LatestEventTimestamp.Value)
classification.LatestEventTimestamp = evt.Timestamp;
@@ -278,8 +286,10 @@ namespace AutopilotMonitor.Functions.Services
var effectiveStart = session?.ResumedAt ?? session?.StartedAt ?? triggerEvent.Timestamp;
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session?.LastEventAt ?? c.LatestEventTimestamp);
var (targetStatus, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session?.LastEventAt ?? c.LatestEventTimestamp,
isPreProvisioned: session?.IsPreProvisioned == true, resumedAt: session?.ResumedAt,
isSelfDeployingProfile: session?.IsSelfDeployingProfile == true);
// Keep the max-lifetime trigger visible: the Succeeded reasons already carry their own
// silence-transparency clause, everything else gets the watchdog context appended.
@@ -305,7 +315,8 @@ namespace AutopilotMonitor.Functions.Services
}
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, targetStatus, triggerEvent.Phase, reason,
request.TenantId, request.SessionId, targetStatus, VerdictPaths.Compose(VerdictPaths.OriginMaxLifetime, rule),
triggerEvent.Phase, reason,
completedAt: isTerminalNonSuccess ? triggerEvent.Timestamp : (DateTime?)null,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
failureSource: isTerminalNonSuccess ? "max_lifetime_watchdog" : null,
@@ -347,14 +358,17 @@ namespace AutopilotMonitor.Functions.Services
var now = DateTime.UtcNow;
var effectiveStart = session.ResumedAt ?? session.StartedAt;
var (targetStatus, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt ?? c.LatestEventTimestamp);
var (targetStatus, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt ?? c.LatestEventTimestamp,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
if (targetStatus != SessionStatus.Succeeded)
return;
var healed = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Succeeded,
VerdictPaths.Compose(VerdictPaths.OriginLateReconcile, rule),
failureReason: reason,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
if (healed)
@@ -368,6 +382,49 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// WhiteGlove Part-2 awaiting-user gate for the agent-reported stall
/// (<c>session_stalled</c>). Pre-provisioned sessions with a ResumedAt whose event
/// stream shows no user evidence since the resume are parked as AwaitingUser instead of
/// Stalled — same evidence rule as the maintenance sweep
/// (<see cref="EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser"/>). Returns true
/// when the AwaitingUser write was applied; false (also on any error, fail-soft) lets
/// the caller run the normal Stalled mapping.
/// </summary>
private async Task<bool> TryMapStallToWhiteGloveAwaitingUserAsync(
IngestEventsRequest request, string sessionPrefix, EventClassification c)
{
try
{
var session = await _sessionRepo.GetSessionAsync(request.TenantId, request.SessionId);
if (session?.IsPreProvisioned != true || !session.ResumedAt.HasValue)
return false;
var sessionEvents = await _sessionRepo.GetSessionEventsAsync(
request.TenantId, request.SessionId, maxResults: 1000);
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(sessionEvents);
if (!EnrollmentTimeoutClassifier.IsWhiteGloveAwaitingUser(
rollup, isPreProvisioned: true, session.ResumedAt))
return false;
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.AwaitingUser, VerdictPaths.IngestWhiteGloveAwaiting,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
failureReason: EnrollmentTimeoutClassifier.WhiteGloveAwaitingUserReason(rollup));
if (transitioned)
_logger.LogInformation(
"{SessionPrefix} Status: AwaitingUser (agent stall probe on WhiteGlove Part 2 with no user evidence since resume)",
sessionPrefix);
return true;
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"{SessionPrefix} WhiteGlove awaiting-user gate failed for session_stalled; falling back to Stalled", sessionPrefix);
return false;
}
}
private static bool IsPeriodicOrStallEvent(string? eventType) => eventType switch
{
"performance_snapshot" => true,
@@ -410,7 +467,9 @@ namespace AutopilotMonitor.Functions.Services
? c.CompletionEvent.Data["completionSource"]?.ToString() : null;
statusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Succeeded, c.CompletionEvent.Phase,
request.TenantId, request.SessionId, SessionStatus.Succeeded,
espSoftFailure ? VerdictPaths.AgentCompleteSoft : VerdictPaths.AgentComplete,
c.CompletionEvent.Phase,
completedAt: c.CompletionEvent.Timestamp,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
espSoftFailure: espSoftFailure, completionSource: completionSource);
@@ -437,7 +496,8 @@ namespace AutopilotMonitor.Functions.Services
: c.FailureEvent.Message;
statusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Failed, c.FailureEvent.Phase, failureReason,
request.TenantId, request.SessionId, SessionStatus.Failed, VerdictPaths.AgentFailed,
c.FailureEvent.Phase, failureReason,
completedAt: c.FailureEvent.Timestamp,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
_logger.LogWarning("{SessionPrefix} Status: Failed - {FailureReason} (transitioned={Transitioned})", sessionPrefix, failureReason, statusTransitioned);
@@ -447,7 +507,8 @@ namespace AutopilotMonitor.Functions.Services
{
failureReason = c.EspFailureEvent.Message ?? "ESP failure (backend fallback)";
statusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Failed, c.EspFailureEvent.Phase, failureReason,
request.TenantId, request.SessionId, SessionStatus.Failed, VerdictPaths.AgentEspFailureFallback,
c.EspFailureEvent.Phase, failureReason,
completedAt: c.EspFailureEvent.Timestamp,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
_logger.LogWarning("{SessionPrefix} Status: Failed via esp_failure fallback - {FailureReason} (transitioned={Transitioned})",
@@ -456,7 +517,8 @@ namespace AutopilotMonitor.Functions.Services
else if (c.GatherCompletionEvent != null)
{
await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Succeeded, c.GatherCompletionEvent.Phase,
request.TenantId, request.SessionId, SessionStatus.Succeeded, VerdictPaths.AgentGatherComplete,
c.GatherCompletionEvent.Phase,
completedAt: c.GatherCompletionEvent.Timestamp,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp);
_logger.LogInformation("{SessionPrefix} Status: Succeeded (gather_rules)", sessionPrefix);
@@ -464,7 +526,8 @@ namespace AutopilotMonitor.Functions.Services
else if (c.WhiteGloveEvent != null)
{
whiteGloveStatusTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Pending, EnrollmentPhase.AppsDevice,
request.TenantId, request.SessionId, SessionStatus.Pending, VerdictPaths.AgentWhiteGlovePending,
EnrollmentPhase.AppsDevice,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
isPreProvisioned: true, isUserDriven: false);
@@ -485,7 +548,7 @@ namespace AutopilotMonitor.Functions.Services
_logger.LogWarning("{SessionPrefix} WhiteGlove UpdateSessionStatusAsync failed, attempting unconditional fallback for IsPreProvisioned + Status", sessionPrefix);
try
{
await _sessionRepo.SetSessionPreProvisionedAsync(request.TenantId, request.SessionId, true, SessionStatus.Pending, isUserDriven: false);
await _sessionRepo.SetSessionPreProvisionedAsync(request.TenantId, request.SessionId, true, SessionStatus.Pending, isUserDriven: false, verdictPath: VerdictPaths.AgentWhiteGlovePending);
whiteGloveStatusTransitioned = true;
_logger.LogInformation("{SessionPrefix} WhiteGlove fallback succeeded: IsPreProvisioned + Status=Pending set via unconditional merge", sessionPrefix);
}
@@ -525,7 +588,8 @@ namespace AutopilotMonitor.Functions.Services
if (currentSession?.Status == SessionStatus.Pending)
{
await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.InProgress, c.WhiteGloveResumedEvent.Phase,
request.TenantId, request.SessionId, SessionStatus.InProgress, VerdictPaths.AgentWhiteGloveResumed,
c.WhiteGloveResumedEvent.Phase,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
isUserDriven: true, resumedAt: c.WhiteGloveResumedEvent.Timestamp);
_logger.LogInformation("{SessionPrefix} Status: InProgress (WhiteGlove Part 2 resumed, IsUserDriven=true)", sessionPrefix);
@@ -559,9 +623,19 @@ namespace AutopilotMonitor.Functions.Services
if (c.SessionStalledEvent != null)
{
// WhiteGlove Part-2 awaiting-user gate (fairstone.ca analysis 2026-08-21): the
// agent-side stall probe also fires on a resumed Part-2 device that simply sits
// at the logon screen with nobody signing in. With zero user evidence since the
// resume that is the expected parking state between technician and end user, so
// the honest label is AwaitingUser, not Stalled. Rare path (one probe batch per
// stalling session), so the session + events reads are acceptable; fail-soft —
// any error falls back to the plain Stalled mapping.
if (await TryMapStallToWhiteGloveAwaitingUserAsync(request, sessionPrefix, c))
return (statusTransitioned, whiteGloveStatusTransitioned, failureReason);
var stalledReason = "Agent reported stall after 60min without progress (stall_probe)";
var stalledTransitioned = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.Stalled,
request.TenantId, request.SessionId, SessionStatus.Stalled, VerdictPaths.AgentStallProbe,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
stalledAt: c.SessionStalledEvent.Timestamp, failureReason: stalledReason);
if (stalledTransitioned)
@@ -584,7 +658,7 @@ namespace AutopilotMonitor.Functions.Services
if (currentStatus == SessionStatus.Stalled)
{
var healed = await _sessionRepo.UpdateSessionStatusAsync(
request.TenantId, request.SessionId, SessionStatus.InProgress,
request.TenantId, request.SessionId, SessionStatus.InProgress, VerdictPaths.AgentStallHeal,
earliestEventTimestamp: c.EarliestEventTimestamp, latestEventTimestamp: c.LatestEventTimestamp,
clearStalledAt: true, clearFailureReason: true);
if (healed)
@@ -0,0 +1,26 @@
namespace AutopilotMonitor.Functions.Services;
/// <summary>
/// Provider-neutral transactional email contract for the tenant-activation welcome mail.
/// <para>
/// Always fail-soft — implementations must NOT throw; activation correctness never depends
/// on email delivery (the Global Admin "Send welcome email" button is the manual fallback).
/// The farewell mail after offboarding lives on the sibling
/// <see cref="IOffboardFarewellEmailSender"/> contract; both are served by <see cref="EmailService"/>.
/// </para>
/// </summary>
public interface IEmailService
{
/// <summary>
/// Sends the tenant-activation welcome email. No-op when the provider is not configured
/// or the recipient is empty.
/// </summary>
Task SendPreviewApprovedEmailAsync(string toEmail, string domainName, CancellationToken ct = default);
/// <summary>
/// Operator test send: the effective template of <paramref name="kind"/> (or an unsaved
/// <paramref name="draftHtml"/>, placeholder-rendered) to <paramref name="toEmail"/>.
/// Returns true when the provider accepted the message. Never throws.
/// </summary>
Task<bool> SendTestAsync(EmailTemplateKind kind, string toEmail, string domainName, string? draftHtml, CancellationToken ct = default);
}
@@ -18,6 +18,11 @@ namespace AutopilotMonitor.Functions.Services
/// registered a LATER session (Part 2 under a fresh session id, or re-provisioning) are
/// resolved as Incomplete("Superseded by ..."). Forward-looking coverage lives in the
/// registration supersede pass; this mode clears the existing backlog.</item>
/// <item><b>self_deploying_silent</b> — self-deploying-profile rows parked as
/// Incomplete / AwaitingUser / Stalled by the pre-2026-08-23 classifier (which had no
/// notion of a profile without a user phase) are re-run through the classifier; only a
/// Succeeded verdict (<see cref="EnrollmentTimeoutClassifier.IsSelfDeployingProvisioned"/>)
/// is applied, everything else keeps its verdict.</item>
/// </list>
/// Hand-marked sessions (AdminMarkedAction) are never touched.
/// </summary>
@@ -30,6 +35,7 @@ namespace AutopilotMonitor.Functions.Services
public const string ModeLegacyTimeouts = "legacy_timeouts";
public const string ModePendingOrphans = "pending_orphans";
public const string ModeSelfDeployingSilent = "self_deploying_silent";
private const int MaxSamplesInResponse = 100;
public LegacyReclassificationService(
@@ -108,8 +114,10 @@ namespace AutopilotMonitor.Functions.Services
var events = await _sessionRepo.GetSessionEventsAsync(tenantId, session.SessionId, maxResults: 1000);
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var effectiveStart = session.ResumedAt ?? session.StartedAt;
var (target, reason) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt);
var (target, reason, rule) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
if (target == SessionStatus.Failed)
{
@@ -137,7 +145,7 @@ namespace AutopilotMonitor.Functions.Services
}
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
tenantId, session.SessionId, target,
tenantId, session.SessionId, target, VerdictPaths.Compose(VerdictPaths.OriginRetro, rule),
failureReason: reason,
failureSource: target == SessionStatus.Incomplete ? "legacy_timeout_retro_reconcile" : null,
allowTerminalReclassification: true);
@@ -164,6 +172,99 @@ namespace AutopilotMonitor.Functions.Services
return result;
}
public async Task<ReclassificationResult> ReconcileSelfDeployingSilentAsync(
string? tenantIdScope, bool dryRun, int maxSessions, string triggeredBy)
{
var result = new ReclassificationResult { Mode = ModeSelfDeployingSilent, DryRun = dryRun };
var now = DateTime.UtcNow;
foreach (var tenantId in await ResolveTenantsAsync(tenantIdScope))
{
if (result.SessionsExamined >= maxSessions) { result.CapReached = true; break; }
result.TenantsExamined++;
int? tenantGraceHours = null, absoluteMaxHours = null;
try
{
var config = await _configService.GetConfigurationAsync(tenantId);
tenantGraceHours = config?.SessionGraceHours;
absoluteMaxHours = config?.AbsoluteMaxSessionHours;
}
catch { /* defaults */ }
var graceHours = EnrollmentTimeoutClassifier.ResolveGraceHours(tenantGraceHours, absoluteMaxHours);
var candidates = await _maintenanceRepo.GetSelfDeployingSilentSessionsAsync(
tenantId, maxSessions - result.SessionsExamined + 1);
if (candidates.Count > maxSessions - result.SessionsExamined)
{
result.CapReached = true;
candidates = candidates.Take(maxSessions - result.SessionsExamined).ToList();
}
foreach (var session in candidates)
{
result.SessionsExamined++;
try
{
if (!string.IsNullOrEmpty(session.AdminMarkedAction))
{
result.Skipped++;
continue;
}
var events = await _sessionRepo.GetSessionEventsAsync(tenantId, session.SessionId, maxResults: 1000);
var rollup = EnrollmentTimeoutClassifier.ExtractRollup(events);
var effectiveStart = session.ResumedAt ?? session.StartedAt;
var (target, reason, _) = EnrollmentTimeoutClassifier.ClassifyTimedOutSession(
rollup, effectiveStart, now, graceHours, session.LastEventAt,
isPreProvisioned: session.IsPreProvisioned, resumedAt: session.ResumedAt,
isSelfDeployingProfile: session.IsSelfDeployingProfile);
// Heal-only: a verdict other than Succeeded means the device never reached
// Device ESP all-succeeded (or failed explicitly) — the existing label stands.
if (target != SessionStatus.Succeeded)
{
result.Skipped++;
continue;
}
RecordSample(result, tenantId, session, target, reason, oldStatus: session.Status.ToString());
if (dryRun)
{
result.WouldChange++;
continue;
}
// Succeeded may upgrade Incomplete / AwaitingUser / Stalled through the normal
// reconcile gate (no terminal-reclassification override needed); the storage
// layer additionally refuses admin-marked rows.
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
tenantId, session.SessionId, SessionStatus.Succeeded, VerdictPaths.RetroSelfDeployingReconcile,
failureReason: reason);
if (transitioned)
{
result.Changed++;
result.ToSucceeded++;
}
else
{
result.Skipped++;
}
}
catch (Exception ex)
{
result.Errors++;
_logger.LogWarning(ex, "Self-deploying retro-reconcile failed for session {SessionId} (tenant {TenantId})",
session.SessionId, tenantId);
}
}
}
await WriteAuditAsync(result, tenantIdScope, triggeredBy);
return result;
}
public async Task<ReclassificationResult> ResolvePendingOrphansAsync(
string? tenantIdScope, bool dryRun, int maxSessions, string triggeredBy)
{
@@ -213,7 +314,7 @@ namespace AutopilotMonitor.Functions.Services
// Pending is non-terminal — the standard transition guard covers this write.
var transitioned = await _sessionRepo.UpdateSessionStatusAsync(
tenantId, pending.SessionId, SessionStatus.Incomplete,
tenantId, pending.SessionId, SessionStatus.Incomplete, VerdictPaths.RetroSuperseded,
failureReason: reason, failureSource: "superseded_by_reregistration");
if (transitioned) { result.Changed++; result.ToIncomplete++; }
else result.Skipped++;

Some files were not shown because too many files have changed in this diff Show More