Fresh cross-tenant usage computes took 20-43s server-side but the web app's
default 30s fetch timeout aborted them client-side (HTTP 499), and every
retry stacked another concurrent full compute.
Backend:
- UsageMetricsService: cache 5→15min, single-flight per days-window (retries
join the in-flight compute), the four source scans run concurrently via
Task.WhenAll, statics converted to instance fields (singleton, testable)
- New column-projected queries for the usage compute only:
GetUsageWindowSessionsAsync ($select 13 consumed columns, drops
FailureSnapshotJson & co.) and GetAppInstallRefsAsync ($select
SessionId+AppName, lean SessionAppRef DTO); GetSessionsByDateRangeAsync
is untouched for its other consumers
- Projection equivalence pinned by UsageMetricsProjectionEquivalenceTests
(incl. the WhiteGlove Part-2 duration branch via IsPreProvisioned/
ResumedAt); single-flight/cache/failure-eviction pinned by
UsageMetricsServiceSingleFlightTests
Web (SectionPlatformUsage):
- 180s AbortSignal for this call (backend keeps computing and caches)
- Calculating card with elapsed time and a progress bar sized from the last
fresh-compute duration (localStorage, cache hits excluded, capped at 95%)
- Friendly timeout error explaining the retry will hit the server cache
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"Total Sessions" on the usage-metrics page is window-bound (90d default)
and capped by retention. Add a retention-independent per-tenant counter
surfaced as "Total Enrollments (since signup)".
- TenantStats row in the PlatformStats table (PK=tenantId, RK="current";
the platform row's "global" PK cannot collide with a tenant GUID)
- Incremented at session registration, fire-and-forget, fresh
registrations only (agent restarts / WG Part 2 resumes re-register the
same session and must not inflate the counter)
- ETag CAS with retries instead of last-writer-wins: lost increments are
permanent (no recompute can recover pruned sessions), and rollout
waves register concurrently
- Nightly platform-stats recompute seeds/self-heals the counter to at
least the live session count (raise-only floor, analog TotalUsers clamp)
- Tenant usage metrics expose SessionMetrics.TotalAllTime =
max(counter, window total); web page shows a "Total Enrollments" card
plus clarifying window note ("Total Sessions" is not a lifetime total)
Backend tests 2876/2876, tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reorganize docs/ tree:
- docs/design/mcp-oauth-flow.md -> docs/mcp-oauth-flow.md (no customer data)
- enrollment-status-reclassification{,-plan}.md -> tasks/ (git-ignored work
files; they contained a customer domain that must not ship in the public repo)
- remove now-empty docs/design/ folder
Repoint the 22 source files whose comments referenced
docs/design/enrollment-status-reclassification.md to the new tasks/ path.
Comment/docstring text only; no functional code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Session 7443317c proved a WU-tracker blind spot: an OOBE quality update
installed (OsBuild 26200.8037 -> 26200.8655 across the mid-enrollment reboot)
yet the deployed WindowsUpdateTracker emitted ZERO events - the watcher armed
fine (no collector_degradation), the spool is durable, and two 60-min backfills
covered the install window, so the WindowsUpdateClient channel simply carried
none of the targeted EventIDs 19/20/43/44. The tenant has diagnostics upload
disabled, so the fix makes the feature self-evidencing instead of guessing IDs:
- OsBuildChangeHost: persists CurrentBuild.UBR (Registry64 view) to
AgentState/os-build-state.json and emits os_build_changed (Info, immediate)
when it differs across a restart - deterministic update corroboration that
cannot be lost to timing, watermarks, or unexpected servicing paths. First
run seeds silently; corrupt state re-seeds.
- WindowsUpdateTracker channel census: when the build provably changed AND zero
targeted WU events were emitted this run, emit a one-shot
windows_update_channel_census (Debug) with an unfiltered EventID histogram of
the WindowsUpdateClient AND UpdateOrchestrator channels over the lookback
window (capped at 5000 records, cap surfaced as censusTruncated). The
histogram arrives via normal event upload and tells us which IDs to add to
WindowsUpdateTargetedEventIds via remote config - no redeploy, no diag ZIP.
Gated by CollectorConfiguration.WindowsUpdateChannelCensusEnabled (default
on, model-default pattern, no ConfigVersion bump).
- No new hardcoded EventIDs (per the device-validation rule) - census data
decides what to enable remotely.
- ANALYZE-DEV-006 "OS Build Changed During Enrollment" (info, not KO):
interpolates {{previousBuild}}/{{currentBuild}}, counts the census as
corroboration. Catalogs: Constants.cs + MCP resource-catalog.ts + web
eventTypes.ts (drift-test enforced).
Tests: 6 OsBuildChangeHost (seed/same/changed/unreadable/corrupt/no-dir),
7 census (gating matrix, truncation surfaced, scanner-throw swallowed,
histogram format), 3 ANALYZE-DEV-006. Agent.V2.Core 1754/1754, Backend
2874/2874, MCP 379 (drift green), web tsc + vitest 367/367, sln 0 errors.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 9a107a19 reboot-cancel guard correctly stopped false
esp_exit_without_completion_evidence failures, but a pure cancel left the
post-AccountSetup dead-end zone with no resolution-capable deadline: session
7443317c (same tenant/shape as 1924092e, plus a mid-OOBE Windows-Update reboot
chain) tripped the session_parked_without_deadline liveness tripwire on a live
enrollment that later completed normally.
RebaseEspExitVariantAdvisoryWindowOnReboot now replaces the stale window with a
fresh 30-min one whose enforcement baselines (armSignalOrdinal /
armAppTerminalCount) anchor at the reboot signal, from both
HandleSystemRebootObservedV1 and HandleEspResumedV1:
- A live enrollment keeps disproving the dead-end at each fire via
HasEnforcementProgressSinceArming (re-arm, never fail).
- A true dead-end where the user rebooted inside the window now resolves one
window later (30-60 min) instead of idling to the 6-h max-lifetime watchdog -
the 9a107a19 trade-off gap is closed.
- The parked tripwire needs no change: a resolution-capable deadline is armed
throughout, so IsParkedWithoutDeadline stays false and the tripwire keeps its
"every fire is a bug" charter.
- The rebase emits a completion_waiting with the new resolutionDeadlineDueAtUtc
(trigger <Signal>:AdvisoryRebase), deliberately bypassing the fingerprint
dedupe like the EnforcementActive re-arm site - bounded to one per reboot.
Tests: rebase with reboot-anchored baselines (both call sites), advisory
variant exempt, no-window no-op, true-dead-end-after-reboot fails at the
rebased due-time, 7443317c double-reboot live-install replay, fire-once keeps
the rebased window, 1924092e full replay updated (stray fire re-arms on
progress), engine-driven parked-tripwire regression.
DecisionCore 445/445, Agent.V2.Core 1754/1754.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session a4537c36: Windows closed the User-ESP page normally (Shell-Core 62407,
errorCode=0) while the AccountSetup registry never wrote categorySucceeded -
the strong gate (arm A) was unsatisfiable by construction and the session only
completed because the Hello policy happened to flip to enabled after a reboot.
ShouldTransitionToAwaitingHello gains arm C: AccountSetup entered + normal ESP
final exit (post-AccountSetup by INGEST ordinal - backdated CMTrace timestamps
are irrelevant, sequence is canonical) + genuine IME user-session completion
(defaultuser0 guard: at-or-after the AccountSetup anchor, extracted as shared
IsImeUserSessionGenuine and reused by the AdvisoryCompletion backstop) + DAD-
validated real-user desktop. ALL FOUR mandatory; Hello gating stays at the
call sites - arm C never bypasses an enabled-unresolved Hello.
Wiring per signal ordering: desktop last via the fast-path (desktopArrived-
InFlight), exit last via HandleEspExitingV1 (espFinalExitInFlight + deferred-
completion parity mirroring the caa6cf50 branch), IME last via new completion/
promote wiring in HandleImeUserSessionCompletedV1. completion_waiting stops
reporting account_setup_provisioning_complete once the evidence set holds.
AdvisoryCompletion stays as the 30-min backstop for incomplete evidence.
The 08c99638 regression guard (3-of-4, no IME fact -> parked) stays green
unmodified; the L5 backdated-exit test now asserts the eager completion.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 1ec8f4c6 dead-end window (guard-blocked post-AccountSetup esp_exiting arms a
30-min advisory_completion resolution deadline) misfired on the Device->Account
handoff exit whenever the IME logged an AccountSetup phase line pre-sign-in
(defaultuser0/AutoLogon frame). The recovered deadline then failed sessions
mid-AccountSetup while apps were actively installing:
enrollment_failed: esp_exit_without_completion_evidence on a live enrollment.
41 of the 65 affected sessions since 2026-06-15 show reboots or app activity
after the arming exit - false positives.
Two guards, esp-exit variant only (the advisory variant keeps its real-failure
un-defang semantics):
- Reboot cancel: SystemRebootObserved / EspResumed cancel the armed window -
an arming exit that predates a reboot was a pre-reboot page close, not the
"page closed, nothing happens" dead-end. A later genuine guard-blocked exit
re-arms a fresh window.
- Enforcement-progress re-arm: arming baselines (armSignalOrdinal,
armAppTerminalCount) ride on the deadline's FiresPayload (replay-deterministic,
snapshot-round-trips, no DecisionState schema change). At fire time, apps
reaching terminal states or an ESP user-phase re-assert since arming re-arms
the window instead of failing. Convergent: each re-arm demands new progress;
legacy deadlines without baselines resolve with pre-fix semantics.
11 new tests incl. a full session-1924092e replay (handoff exit -> reboot ->
active installs -> stray recovered-timer fire dead-ends -> Completed) and pins
for the true dead-end verdict, the advisory variant, and baseline serialization.
DecisionCore 435/435, Agent.V2.Core 1725/1725.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An ESP subcategory failure can be retracted inside the 30s settle window
(user clicks "Try again" / IME re-sync flips failed -> inProgress). The
expiry handler fired the latched EspFailureDetectedEventArgs
unconditionally, terminating the agent on a failure Windows no longer
reported and losing the retry outcome (session c071e92b: recovery at
+12s, terminal esp_terminal_failure at +30s anyway).
- OnProvisioningFailureSettleExpired re-checks current registry-derived
state under _stateLock: recovered = categorySucceeded != false AND no
subcategory currently "failed". On recovery: emit new
esp_failure_settle_recovered event, suppress EspFailureDetected, clear
the fire-once gate (+resolved set) so a re-failure arms a FRESH settle
window and is then terminal.
- esp_failure_settle_started now names tracked apps not completed
(trackedAppsNotCompleted/-Count) on Apps-subcategory failures via a new
packageStatesProbe (ImeLogHost.AllKnownPackageStates through
DefaultComponentFactory -> EspAndHelloHost -> EspAndHelloTracker).
Covers the Store/WinGet blind spot (Company Portal pending with zero
IME events) that both app_install_starved (IME-phase + intent gates)
and esp_apps_failure_correlation (IsActive device apps only) miss.
- New event type registered in Constants.EventTypes + MCP resource
catalog (drift test green); 5 new settle-window tests, suite 1711/1711.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
netstandard2.0's string.IsNullOrEmpty lacks [NotNullWhen(false)], so the
compiler cannot prove the else branch is non-null. Null-forgiving assert.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Session 2bc884b6: ESP failed AccountSetup/Apps with 0x80073cf9 while all 8
tracked Win32/WinGet apps succeeded — the failing app was a MSIX/Store app
that installs outside the IME Win32 pipeline and is invisible to
ImeLogTracker. The session ended as a blackbox failure.
When an Apps-subcategory failure arms the 30s settle window, the tracker now
queues a one-shot scan of Microsoft-Windows-AppXDeploymentServer/Operational
(Critical+Error, session window, 200-record/10s cap), scores candidate
packages (exact-HRESULT match > AppX-family error > time proximity, recency
bonus), and emits esp_appx_failure_analysis with flat RuleEngine fields plus
a capped candidates[] detail list — before enrollment_failed lands in the
timeline. Negative results are emitted too (no_appx_candidates points back
at Win32). Fire-once per session; scan runs lock-free on the threadpool;
settle expiry never waits; failures degrade to collector_degraded.
- New: AppxDeploymentFailureScanner (one-shot reader, injectable seam),
AppxFailureAnalyzer (pure scoring/verdicts, incl. negative-decimal
HRESULT + PackageFullName extraction from message/properties)
- ProvisioningStatusTracker: optional scanner/dispatcher ctor seams,
MaybeStartAppxFailureScan at settle-arm time
- Rule ANALYZE-APP-014 (MSIX/Store app failure during ESP) keyed on
verdict=appx_candidate_identified; dist regenerated
- MCP resource-catalog + web HRESULT badge + gather-rules eventTypes
- Tests: 16 analyzer + 7 trigger; settle-window fixture now injects a
fake scanner + sync dispatcher (no real event log in tests)
Inert until agent redeploy (rule needs backend redeploy). Post-deploy
checks in tasks/todo.md.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New one-shot network_bandwidth_estimate event derived passively from the
Delivery Optimization byte counters the DeliveryOptimizationCollector
already polls every 3s - zero synthetic traffic, zero extra load, no
config change.
- BandwidthEstimator: per-FileId delta windows (a DO job that drops off
the list and returns is averaged over its own absence window), p90/max
reduction, WAN (HTTP + internet peers) vs LAN (LAN/group/link-local
peers + Connected Cache) split so peer-fed downloads cannot inflate
the WAN figure; bucket (<10/10-50/50-100/100-250/250+) + confidence.
- Emitted at most twice per session (snapshotTrigger): interim snapshot
on first AccountSetup sighting (device_setup_end - survives sessions
that starve in the account phase) and authoritative final at collector
stop (collector_stop, lands before diagnostics_collecting).
- All DO jobs feed it (IME apps, Office CDN, WU/Store/Defender).
- Registered in Constants.EventTypes, MCP resource-catalog
(network_events) and web gather-rules autocomplete.
- Drive-by: add missing agent_emergency_break to the MCP catalog
(pre-existing drift, event-types-drift test was red).
13 new estimator unit tests; agent suite 1655/1655, MCP drift 6/6,
web tsc + eventTypes vitest green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Phase 1 - named notification channels (1..N per tenant):
- NotificationChannel model (provider/url/headers/enabled + per-event
opt-ins incl. single NotifyOnSlaEvents); stored as JSON column
NotificationChannelsJson, cap 10, fail-soft ParseList
- No data migration: GetNotificationChannels() synthesizes one channel
from the legacy single-webhook fields under stable id "legacy";
web materializes on first save and clears legacy fields
- All dispatch sites (ingest success/failure/white-glove/ESP, SLA breach/
resolved/consecutive, enrollment-start, hardware-rejected, test) route
via SendToChannelsAsync with per-channel toggle filtering
- Per-channel secret redaction (structure stays visible to readers) with
restore-by-id; save-time validation (dup ids, provider, https, headers)
- Settings UI: channel card list with add/remove, event pills, per-channel
test button ({channelId} body, legacy-compatible without body)
Phase 2 - analyze-rule channel notifications:
- NotifyDefault (definition, false for all built-ins, test-guarded) +
Notify/NotifyChannelIds tenant override in RuleStates (two-tier mirror
of MarkSessionAsFailed); custom rules fold into the tenant-owned row
- Dispatch in AnalyzeOnEnrollmentEndHandler only: outcome.Results is
new-findings-only (engine dedup) so no re-notification; manual
"Analyze Now" reanalyze never dispatches
- BuildRuleFiredAlert (eventType analyze_rule_fired): device, serial,
rule, severity, category, confidence, explanation + session deep link
- Rule card: notify toggle + channel target pills, bell header badge
2858/2858 Functions tests (19 new), web tsc + vitest clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The 2026-07-08 timeout reclassification (Incomplete = terminal, non-failure)
migrated SessionStats, FleetHealthStats and SessionStatusBuckets, but two
reporting surfaces were overlooked: the live UsageMetricsService.SessionMetrics
(both platform + tenant paths, feeding get_usage_metrics and the usage
dashboards) and the persisted daily UsageMetricsSnapshot. Terminal Incomplete
sessions were counted in Total but in no breakdown bucket, so
Succeeded+Failed+InProgress no longer reconciled to Total and operators lost
sight of those devices.
- SessionMetrics + UsageMetricsSnapshot gain an Incomplete / SessionsIncomplete
count, mirroring the migrated peers (Incomplete only, not AwaitingUser).
- Snapshot column stored + mapped in TableStorageService.Metrics; legacy rows
deserialize to 0 (no seed bump needed).
- Success rate unchanged everywhere: denominator was already Succeeded+Failed,
correctly excluding Incomplete/AwaitingUser. SLA paths verified already-correct
(BuildSnapshot + breach evaluator receive terminal-filtered sessions).
- Web: Incomplete card added to both usage views (slate, matches the badge).
- MCP: query_raw_sessions status enum learns AwaitingUser/Incomplete so callers
can filter by the new states.
- Test: UsageMetricsServiceWindowTests pins the Incomplete bucket + 50% rate on
the crcins 1/1/8 shape.
Backend build clean, full Functions suite 2825 pass, web + MCP tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Addresses four review rounds on the enrollment-status reclassification
(timeout != failure; new AwaitingUser/Incomplete states).
Backend correctness:
- UpdateSessionStatusAsync ETag force-fallback now mirrors the normal path:
uses IsTerminalTransitionAllowed + admin-marked guard, stamps CompletedAt
for Incomplete, and persists FailureReason/FailureSnapshotJson for
Incomplete/AwaitingUser (not just Failed) so a contended write cannot
regress a terminal or drop the classification reason/snapshot.
- StoreSessionAsync CAS loop re-reads the terminal tuple in the fresh-read
window and preserves a just-landed terminal (Succeeded/Failed/Incomplete)
verbatim, so re-registration can no longer revert it via the Replace.
- SessionGraceHours + AbsoluteMaxSessionHours now roundtrip through the
tenant-config Store/Map pair (overrides were silently lost).
- ResolveGraceHours clamps the assumed agent cap to >= the real runtime
default (48h) until the agent honors the override, so a low override can
never terminalize a still-running session early.
- Incomplete now funnels through the terminal counter-reconcile seam, and
the maintenance sweep re-reconciles after the synthetic session_timeout
event (also fixes the pre-existing off-by-one on the Failed path).
Web:
- SessionInfoCard delegates its status pill to the shared SessionStatusBadge
(removes a divergent local map; picks up AwaitingUser/Incomplete).
- SessionTable search color map + status filter chips learn the new states.
- useSessionEvents uses isTerminalStatus() instead of a hard-coded
Succeeded/Failed check (no needless refetch on Incomplete).
- DataManagementSection copy no longer says "Failed - Timed Out".
Docs/comments:
- Plan doc + TenantConfiguration comments corrected (+3h/51h buffer, and the
timeout now reclassifies to AwaitingUser/Incomplete, not "Failed").
Tests: +ResolveGraceHours clamp case, +grace/abs-cap config roundtrip +
legacy-row defaults. Full backend suite green (2712), web tsc green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Windows Update activity during Autopilot OOBE/ESP is a blind spot no other tool
(Intune console included) surfaces well — a cumulative update installing mid-OOBE
can break enrollment (r/Intune KB5095189). This makes it visible with secured
evidence.
Agent (net48):
- WindowsUpdateTracker + WindowsUpdateWatcherHost: watch
Microsoft-Windows-WindowsUpdateClient/Operational (EventIDs 19/20/43/44 =
success/failure/install-started/download-started), startup backfill (OOBE updates
run before the agent starts), RecordId watermark dedup (restart boundary vs
intra-run seen-set so a high-id live event can't hide older backfill events),
HRESULT->symbol decode. Emits windows_update_succeeded/_failed/_started with the
real event Timestamp. Default-on config knobs; uncertain EventIDs configurable
without redeploy.
- RegistryCollector: emitOnlyIfExists (suppress positive-assertion events when the
key is absent) + Registry64 base view by default (bitness-independent, matches
TenantIdResolver) with optional registryView override.
Rules:
- ANALYZE-DEV-004 (high: WU failed during enrollment, high-insight not KO) +
ANALYZE-DEV-005 (info: WU installed). reboot_pending corroboration matches
event_data exists==true.
- GATHER-DEVICE-005 -> windows_update_reboot_pending (emitOnlyIfExists),
GATHER-DEVICE-004 -> windows_update_history (Get-HotFix, opt-in).
- 5 EventTypes synced across Constants + MCP resource-catalog + web eventTypes.
Self-maintaining rule lifecycle (analyze + gather parity):
- Gather rules gain the sunset/orphan-GC that only analyze had
(ProcessSunsetGatherRuleAsync + runtime filter + seed/reseed/github paths).
- RuleProvenance field (embedded/github, null=embedded back-compat): the embedded
catalog only sunsets its own rules; GitHub-reseeded rules ahead of the deployed
binary (new ids AND content updates to existing ids) are exempt and reclaimed to
embedded once the binary catches up. No orphans, no operator knowledge required.
Verified: full solution builds; backend suite 2770 pass, agent suite 1638 pass;
rules AJV-valid + combined.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- SessionStatusBuckets: explicit AwaitingUser + Incomplete buckets (out of Other),
still reconciling to Total.
- Metrics summary: expose awaitingUser + incomplete; failure rate is now
Failed/(Succeeded+Failed) — Incomplete and non-terminal states are out of the
denominator (crcins drops from failed/total to the real terminal rate).
- SessionStats.IncompleteLastNDays; dashboard SuccessRatePct was already terminal-only.
- Daily aggregation already used Succeeded+Failed denominator — unchanged.
- Tests: SessionStatusBucketsTests. Suite: 2705 pass. Fleet Health Incomplete surfacing
deferred to PR5 (web) with its documented over-all-sessions semantics.
Groundwork for making the agent's silent 48h emergency break transparent
(docs/design/enrollment-status-reclassification.md):
- Constants.EventTypes.AgentEmergencyBreak + AgentErrorType.SessionAgeEmergencyBreak.
- EnrollmentTimeoutClassifier: ExtractRollup.HasAgentEmergencyBreak; classify a
session carrying the break as Incomplete NOW (agent is gone) instead of holding it
as AwaitingUser through the grace — a real completion still wins. Tests added (22 total).
Next: agent best-effort emit at the break + backend materializes the timeline event.
The 5h→terminal grace was a magic 72h. Anchor it to the agent instead: the 6h
per-run brake emits enrollment_failed (→ Failed at ~6h), but the 48h absolute
emergency break (AbsoluteMaxSessionHours) is SILENT to the backend — it writes a
local marker, cleans up and exits with no terminal event. So a session only stays
non-terminal past 48h when the agent hit that silent cap and is provably gone.
Grace must therefore be >= the agent absolute cap (never mark Incomplete while the
agent could still legitimately enroll) and only slightly beyond it:
effectiveGrace = max(AbsoluteMaxSessionHours + 12h buffer, SessionGraceHours) = 60h default.
- EnrollmentTimeoutClassifier.ResolveGraceHours (pure, tested): floors grace at
agentCap + buffer; a SessionGraceHours override can only raise it.
- TenantConfiguration.SessionGraceHours default 0 (= auto-derive); mirror
AbsoluteMaxSessionHours so the floor follows a tenant override of the agent cap.
- Sweep uses ResolveGraceHours instead of the hard-coded 72h.
Follow-up noted: wire AbsoluteMaxSessionHours down into AgentConfigResponse so the
agent and backend share one value. Functions.Tests: 2678 pass.
Wire EnrollmentTimeoutClassifier into MarkStalledSessionsAsTimedOutAsync so the
5h inactivity sweep stops blindly failing silent-but-provisioned sessions
(docs/design/enrollment-status-reclassification.md). Forward-only.
- TenantConfiguration.SessionGraceHours (default 72h) + default config.
- Sweep Stage 2: classify from the ESP subcategory rollup instead of hard Failed.
Fast-path skips within-grace AwaitingUser sessions with NO event read (grace is
time-based; late completions are caught by ingest). Per-outcome counters; audit
+ ops event only when the pass actually terminalized/transitioned something.
- session_timeout event + auto-analyze kept ONLY for Failed/Incomplete (parity with
the agent max-lifetime path); AwaitingUser is non-terminal, Succeeded is a reconcile.
- GetStalledSessionsAsync includes AwaitingUser so it graduates to Incomplete once
grace elapses; agent-silent (Stage 1) stays InProgress-only.
- UpdateSessionStatusAsync: Incomplete is terminal (idempotency); AwaitingUser can't
regress a terminal; CompletedAt stamped for Incomplete (no skewing duration); reason
+ snapshot persisted for Incomplete/AwaitingUser.
Full Functions.Tests suite: 2670 pass.
Foundation for reclassifying 5h-timeout sessions (docs/design/
enrollment-status-reclassification.md). No behaviour change to the sweep yet —
this only adds the pieces PR2 wires in.
- SessionStatus: append AwaitingUser (non-terminal) + Incomplete (terminal,
non-failure). Members appended so persisted ordinals never shift.
- EnrollmentTimeoutClassifier: pure ExtractRollup (parses the esp_provisioning_status
"N of M subcategories completed" message + the 30s all-succeeded fallback line —
categorySucceeded is ignored, Windows never sets it) and ClassifyTimedOutSession
implementing the decision table (explicit failure→Failed; AccountSetup all-succeeded
/ enrollment_complete→Succeeded; DeviceSetup done + within grace→AwaitingUser, else
Incomplete; silent pre-DeviceSetup→Incomplete). Never returns Failed without an
explicit failure signal.
- FailureSnapshotBuilder: embed the same rollup (schema v2) so the snapshot records the
evidence behind the verdict.
- Tests: EnrollmentTimeoutClassifierTests (12). Full Functions.Tests suite 2670 pass.
- High: TenantConfigurationService.SaveConfigurationAsync now THROWS when the repository reports a failed persist (it swallows storage exceptions and returns false) - plan/trial/config writes can no longer be audited and returned as 200 without being persisted. Cache is invalidated on every attempted save (finally), covering in-place mutations of the cached instance; the now-redundant invalidate-on-failure wrapper in PlanManagementFunction is removed. AuthFunction login side-effect saves (first-login domain write, auto-re-enable) are wrapped best-effort so a storage blip cannot 500 auth/me - they self-gate and retry on next login.
- Medium: MCP usage increment moved from AuthenticationMiddleware (fire-and-forget BEFORE the quota decision) into McpQuotaEnforcementMiddleware: strictly check-then-increment and only for requests actually served (GA tracked but quota-exempt; denied 403/429 requests no longer inflate counters; the exact-limit request is now deterministic - 1..limit pass, limit+1 blocks).
- Low: fixed both new nullable warnings (PlanTierDefinitionParser guarded json, McpQuotaEnforcementMiddleware principal null-flow).
Tests: +3 save-semantics regression tests (throw on false, cache invalidation on failed/successful save). Backend 2656/2656 green, build clean.
Fail-closed entitlement layer (FeatureEntitlementCatalog + TenantEntitlementService): edition resolves at read time (PlanTier == 'enterprise' OR active trial; legacy free/pro => Community, no migration).
- Plan/trial endpoints: PATCH config/{tid}/plan (GA: tier + grant/extend/end trial, audited, cache-safe) and POST config/{tid}/trial (tenant-admin self-service 30d, once per tenant, 409 after). Generic PUT preserves plan/trial fields for all callers. Fixes SetPlanTier writing past the config cache without audit/UpdatedBy.
- Rate limits: Enterprise floor 150/min for BOTH user (portal/JWT) and device (agent/cert) paths - floor raises the global default, an explicit per-tenant override still wins; MSP users follow their home tenant.
- Retention cap: Community 90d / Enterprise 365d, clamped in the retention fanout (warn log) + non-GA write validation 7..cap (only on changed values; 0=infinite stays GA-only).
- Delegated admin (MSP): Enterprise gate on the delegated admin's HOME tenant (JWT tid, fail-closed on unknown) - managed target tenants may be any edition. Grant-time target check removed; inert grants are suppressed at resolve time (single choke point in GetScopeAsync, applied after the UPN-keyed cache).
- MCP quota: McpQuotaService + McpQuotaEnforcementMiddleware (X-Client-Source: mcp only, GA exempt) enforce daily/monthly budgets - per-user plan override > tenant edition plan; limits from SectionUsagePlans with catalog fallbacks (community 100/3000, enterprise 1000/20000); 429 with X-MCP-Quota-* headers + Retry-After; fail-open on counter reads, fail-closed on plan resolution. /api/metrics/mcp-usage/me reports effective plan + counters.
- Trial visibility: TrialExpirySweepFunction (daily 03:30 UTC, informational) emits TenantTrialExpired (Warning, 24h look-back) + TenantTrialExpiring (Info, <=3d heads-up); dual-registered in web OPS_EVENT_TYPES.
- Web: EditionBadge + one-time trial CTA in settings header, Plan & Trial card in the GA tenant edit modal (own PATCH save), retention cap hints from entitlements, pure lib/edition.ts (vitest), feature-flags surface edition/trial/entitlements; MCP TENANT_SAFE_FIELDS + trial fields.
Verified: solution build clean; backend 2653/2653 tests; web tsc + 516/516 vitest; MCP server tsc + tests green.
Replace the denormalized per-tenant RateLimitRequestsPerMinute mirror (and its
SaveConfigurationAsync sync job) with compute-at-read resolution: effective limit =
per-tenant override ?? global default. Fixes the clobbering bug where a per-tenant
edit to the base field was overwritten on the next global config save.
- Model: drop TenantConfiguration.RateLimitRequestsPerMinute; add nullable
CustomUserRateLimitRequestsPerMinute (device override CustomRateLimit... kept).
- RateLimitResolver centralizes precedence (device + user); both paths call it.
- SecurityValidator reads the global default (AdminConfigurationService threaded
through ValidateSecurityAsync + all device-auth callers); 5-min cached.
- UserRateLimitMiddleware: per-tenant user override for tenant-scoped callers only
(skipped for HasGlobalScope = GlobalAdmin|GlobalReader); GA -> GA budget, others ->
user default. Uses GetConfigurationIfExistsAsync so an unregistered-tenant JWT can
no longer auto-create a config row via read endpoints.
- Validation: reject rate limits < 1 at the write functions; RateLimitService clamps
< 1 to 1 as a safety net (avoids Min()-on-empty / throttle-everything).
- Web: Global Settings exposes User + GlobalAdmin limits; tenant modal writes the
override fields (blank = inherit); report + usage-plans updated.
Legacy RateLimitRequestsPerMinute Table column is ignored on read and dropped on the
next Replace save (self-cleaning; no migration).
Backend 2577 xUnit green, tsc clean, web 506 vitest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Retire ANALYZE-ESP-003 (duplicate of ESP-001) and ANALYZE-CORR-002 (duplicate
of APP-004) via the established sunset path; ESP-001 gains a >60min factor,
APP-004 absorbs the co-occurrence story with working confidence factors
- CORR-003: download evidence (errorPatternId) is now a required condition —
no more firing on proxy + any unrelated app failure
- DEV-002 v2: 'sustained' semantics — new event_count value filter
(filterField/filterOperator/filterValue in schema, Shared model, engine, web
types) requires >=3 snapshots above 90% memory instead of one transient spike
- DEV-003 v2: build-cutoff-only condition (<=22621) now also covers Windows 10;
LTSC caveat + maintenance note (raise to 22631 after 2026-11-10)
- SEC-001 v2: rewritten for the post-June-2026 reality — certificates HAVE
expired, active remediation required (OEM firmware hint added)
- SEC-005/006 v2.1: allow-list extended with OEM factory-preload families
(Intel PPM, AMD Power, Lenovo ThinkPad power/battery, Surface/carrier),
telemetry-curated from 1780 fired sessions; imaging leftovers (USMT.PPKG,
InstallOrder.ini) and generic names deliberately still alert
- APP-001: suppressByEvent on app_install_completed; APP-002: exact MSI exit
codes instead of bare 16xx; APP-006 reframed as 'App Install Retry Loop';
ID-001: title/logic mismatch fixed + GATHER-ID-002 dependency documented
- Fix all 13 dead confidence factors (signal-name mismatches / unsupported
syntax) by rewiring or removal; version bumps everywhere so seeding
propagates the updates
- New RuleEngineEventCountFilterTests + SEC-005 OEM allow-list coverage
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kill/Block rules were only served on the telemetry response, which cannot
reach an agent whose upload loop is paused (indefinite block pins the drain
until DateTime.MaxValue), whose spool is empty, or whose binary is kill-blind.
Config is fetched at every process start (boot via Scheduled Task), making it
the reliable control channel - no heartbeat added by design (48h emergency
break + 6h max lifetime bound the remaining cases).
Backend (effective on deploy):
- KillSwitchEvaluator: shared device+version check for both channels; ingest
wire response and check order unchanged (device first, short-circuits)
- Every served Kill emits ops event KillSignalDelivered (Security/Warning),
throttled 24h per tenant+serial+pattern; Blocks stay event-free
- GET agent/config (incl. bootstrap wrapper) now returns DeviceBlocked/
DeviceKillSignal/UnblockAt alongside the full config; old binaries ignore
the new fields
- Web: KillSignalDelivered dual-registered in OPS_EVENT_TYPES
Agent V2 (effective with next agent release):
- CheckConfigKillSignal guard after the Phase-4 fetch: forced self-destruct
(overrides SelfDestructOnComplete), enrollment-complete marker, session
cleared, exit before mTLS/registration
- Kill honoured ONLY from a live fetch: config disk cache strips the flags on
write and read (UnrestrictedMode pattern), guard re-checks LastFetchOutcome
Tests: 9 KillSwitchEvaluatorTests + 7 ProgramConfigKillSignalTests;
Functions 2557/2557, Agent V2 1578/1578, web tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Delta-review 2026-07-02 findings:
- M1: session_parked_without_deadline evaluated the parked predicate on the
very step that produced the state, so the healthy Classic window
(HelloResolved -> AwaitingDesktop, no deadline, desktop seconds away) fired
a spurious Warning AND consumed the one-shot, masking a real variant-4 park
later in the run. The tripwire now only arms a 10-min dwell timer; it fires
(and consumes the one-shot) only if the session is still parked when the
dwell elapses. Steps that clear the parked condition cancel the pending
arm; parked pass-throughs do not re-base it. Processor is now IDisposable
(timer), disposed by the orchestrator.
- L8 (rides along): the tripwire post moved off the ingress worker onto the
timer thread — a full channel can no longer self-deadlock on it.
- M2: ImeUserSessionCompletedUtc was set-once, so an early defaultuser0
completion permanently failed the AdvisoryCompletion conjunction and
resolved genuinely-completed sessions as Failed. New rule: first
post-AccountSetup observation wins (one upgrade of a pre-anchor stamp,
then frozen). Pure function of (state, signal) — replay-safe.
- L5: HandleAdvisoryCompletionDeadlineFired`s esp-exit anchor is presence-only
now; the old >= AccountSetupEntered comparison rejected fires whose arming
exit carried a backfilled source timestamp, re-opening the dead-end.
deadlineStillArmed remains the stray-fire guard.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Persists the agent-sent self-deploying/kiosk profile marker (template:
IsHybridJoin, write-once): Sessions column + SessionsIndex full-mirror
projection + SessionSummary mapping. Re-registration preserve is a
sticky-true OR - a boot with an unreadable Autopilot policy cache must
not pin the flag to false once it was observed true.
Filterable end-to-end: SessionSearchFilter + continuation-token
fingerprint + search/raw query params + index/basic scan filters, plus
MCP search_sessions / query_raw_sessions inputs and the get_session
overview field.
Display/analytics only - the completion fix itself ships in the agent
(DecisionCore gates on the EnrollmentFactsObserved seed, not this flag).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kiosk devices (self-deploying profile) never completed: the IME logs an
AccountSetup phase line for the kioskUser0 autologon session although user
ESP never runs, which cancelled/blocked the DeviceOnlyEspDetection deadline
- the sole SelfDeploying terminal path - and the session idled to the 5h
backend timeout (reference session 320b3bf7; 436 false Failed platform-wide,
90% of all self-deploying failures).
Platform sweep (8983 sessions / 63 tenants): CloudAssignedOobeConfig bits
0x20|0x40 are exclusive to self-deploying/kiosk/shared profiles (0 of 6785
user-driven sessions carry them, bits always co-occur). The fix seeds
Mode=SelfDeploying@High from that registry marker via EnrollmentFactsObserved
and waives the AccountSetup veto at the four existing decision points
(DSPC short-circuit, deadline race guard B, EspPhaseChanged cancel,
RealmJoin deferred-release re-check) - only when the seeded profile holds
AND user ESP made no real progress. Bits absent = byte-identical behaviour.
Also registers IsSelfDeployingProfile on SessionRegistration (display
metadata; backend persistence follows separately).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
M6: DeleteByFilterInBatchesAsync is fail-loud and counts only confirmed
batch submits; the orphan cleanup keeps the EventSessionIndex entry when
the event delete fails, so surviving rows stay discoverable and retry.
M7: GetSessionsOlderThanAsync gains excludeInFlightDeletions - sessions
locked in deletion states no longer occupy the capped read head, so a
stuck head (Poisoned / stranded Queued) cannot starve the retention
fanout tail. Warnings surface a fully-locked backlog and zero-progress
runs.
M8: ingest logs a per-batch warning with per-kind counts and the first
TelemetryItemId when unparseable telemetry items are dropped (agent
clears its spool on success, so drops were silent data loss).
M9: ReconcileSessionRebootCountAsync -> ReconcileSessionCountersAsync;
one partition scan reconciles EventCount and RebootCount at terminal
transitions, self-correcting at-least-once retry double-counts. Script
counters stay increment-only with the drift documented.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The V1 agent line is retired: no new V1 sessions since 2026-05-09, only
orphaned straggler traffic remained on /api/agent/ingest (last seen
2026-06-29, events for long-cleaned-up sessions). Bootstrap/Intune/CI
have deployed V2-only since the v2 cutover.
Agent:
- Delete AutopilotMonitor.Agent, .Agent.Core, .Agent.Core.Tests
(projects + sln entries). V2 trio, SummaryDialog, DecisionCore.Tests
and V2Isolation.targets stay.
Backend:
- Relocate the shared statics/DTOs V2 reused from the V1 partials
(StampServerFields, SanitizeEventTimestamps, IsMaxLifetimeAgentShutdown,
RecalculateAppDurations, EventClassification, AppInstallAggregationState)
into EventIngestProcessor; IngestEventsOutput moves next to
IngestTelemetryFunction (same namespace, zero ripple).
- Delete IngestEventsFunction (route agent/ingest), BootstrapIngestEvents
(route bootstrap/ingest), NdjsonParser and their DI registration.
EventDataNormalizer stays (used by TelemetryPayloadParser).
- Drop both routes from EndpointAccessPolicyCatalog and
RequestTelemetryMiddleware.TenantHeaderTrustedPaths.
- Retire the V1 update-hash oracle: LatestAgentV1*/LatestBootstrapV1*
fields + GetAgentLine case 1 (default arm returns empty hashes, so
stragglers gracefully skip the integrity check; ParseAgentMajor keeps
its default of 1 on purpose — defaulting to 2 would feed V2 hashes to
V1 binaries). TableConfigRepository Store+Map updated in lockstep.
- Remove ApiEndpoints.IngestEvents/.BootstrapIngestEvents constants.
Tests: NdjsonParserTests deleted; timestamp/critical-path/max-lifetime
tests repointed to EventIngestProcessor; the two NDJSON pipeline tests
removed (coverage retained by direct-list siblings); catalog/no-store
InlineData rows updated.
Docs: architecture.md (endpoint tables, ingest flow diagram, solution
layout, test matrix), web SectionAgentInternals endpoint tables,
build.ps1 help text.
Azure: no config change needed — /api/agent is not in the client-cert
exclusion paths (agent/ingest was always cert-enforced) and the
/api/bootstrap prefix exclusion remains for the surviving bootstrap
routes (verified via ARM on autopilotmonitor-api).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Completes the Tenant Templates -> Tenant Groups rename down to identifiers,
tables, routes and DTOs (the earlier commit renamed only the UI labels). Done
now while the table is effectively empty; the new TenantGroups /
TenantGroupAssignments tables are auto-created on deploy, the operator re-enters
their group, and the old TenantTemplates table is dropped in Azure manually.
- Tables: TenantTemplates -> TenantGroups, TenantTemplateAssignments ->
TenantGroupAssignments (Constants + All).
- Entities/DTOs: TenantTemplate(Entity)/TenantTemplateAssignment(Entity) ->
TenantGroup*, property TemplateId -> GroupId.
- Repo/service/API: all *Template* methods -> *Group*; routes
/api/global/tenant-templates -> /api/global/tenant-groups, {templateId} ->
{groupId}; function class + file renamed; audit entity DelegatedTemplateAccess
-> DelegatedGroupAccess; NormalizeTemplateId -> NormalizeGroupId.
- Offboarding wipe list + tests updated.
- Web: api.tenantGroups, SectionTenantGroups (file renamed), response key groups.
Scoped strictly to the feature: unrelated 'template' usages (analyze-rule
templates, EmailTemplates, interpolate-rule-template) left untouched. Backend
2538 tests green; web tsc clean + vitest 494 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Let a GlobalAdmin define a named Tenant Template (a bundle of tenants) and
assign a delegated-admin UPN to the template instead of to each tenant. A UPN's
effective scope = union of direct grants + every tenant in every assigned
template, resolved in DelegatedAdminService.GetScopeAsync — so middleware
enforcement, /auth/me delegatedTenantIds and MCP all inherit it unchanged.
Adding a tenant to a template grants it to all assignees at once (membership-
driven MSP management). Fully app-internal: no Entra group claim / Graph.
- Storage: TenantTemplates (PK=templateId; meta + membership rows) and
TenantTemplateAssignments (PK=upn) tables; meta-backed existence invariant in
both management reads AND the auth hot path (no ghost templates, no scope from
stray rows).
- Offboarding hygiene: property-only wipe now purges TenantTemplates membership
AND DelegatedAdmins rows for the offboarded tenant (closes a pre-existing
orphaned-grant gap).
- Management API: GlobalAdminOnly /api/global/tenant-templates/* (8 routes);
per-affected-tenant audit; all mutations route through the service so the
delegated-scope cache is invalidated in lockstep; existence/membership
invariant owned by the service (no ghost create, no false revoke audits);
templateId normalized case-insensitively at the service boundary.
- Web: SectionTenantTemplates management UI (create/rename/delete, add/remove
tenants, assign/unassign), triple-registered + api.ts block.
Backend 2538 tests green; web tsc clean + vitest 488 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Make a delegated ("MSP") admin a "scoped Global Reader": the same pages a Global
Admin sees, but every tenant switcher/filter is bounded to the managed allow-list.
Reader path only (readWrite deferred). MCP stays read-only (Phase 6 already covers
delegated reads).
Backend (bounded aggregate — reuses the GA per-tenant fan-out):
- global/sessions + global/stats/sessions: GlobalReadOrAdmin -> GlobalReadOrDelegatedSubset
(keep QueryParam). A delegated caller WITHOUT ?tenantId= is admitted as a bounded
aggregate; the handler restricts the cross-tenant fan-out to RequestContext.AllowedTenantIds.
With ?tenantId=X it drills into one managed tenant (middleware validates X in scope).
- TableStorageService: GetAllSessions{,Page,Stats}Async gain an optional allowedTenantIds
set; FetchAllSessionsPageInternalAsync swaps its Step-1 tenant source to that subset
(everything downstream — per-tenant query, inverted-tick merge, cursor — is reused).
- IsDelegatedReadTier += GlobalReadOrDelegatedSubset so the single-tenant drill rescue fires.
- SignalR AddToGroup/RemoveFromGroup: admit a delegated caller to a managed tenant-{id}
group (in scope); global-admins stays platform-scope-only.
Web:
- useAggregatedAdminScope / useGlobalAdminScope: cross-tenant mode is now on for a delegated
admin too (isGlobalAdmin = crossTenant; new isDelegatedScope). Tenant list bounded to the
managed subset; always a concrete tenant (no "All tenants" aggregate on analytics pages).
- TenantScopeSelector hides the aggregate option for delegated; GlobalAdminBanner gains a
blue "Delegated Admin View" variant (wired on all analytics pages + dashboard).
- Nav: Fleet group exposes Dashboard + Fleet Health + Software + Geographic + SLA + Usage;
showDashboard now includes delegated.
- Dashboard delegated-ready: crossTenant drives the /global endpoints + filter UI; empty
filter = bounded aggregate over managed tenants, a selected tenant = drill-in; global-admins
SignalR join stays real-GA-only; read-only (mutations already GA-gated). `?tenant=` deep-link
seeds the filter.
- Fleet is now a pure stats card grid; the per-tenant session browser /fleet/[tenantId] is
removed and cards link to /dashboard?tenant=<id>.
Deferred follow-ups: Audit Log + Geographic session drill-in for delegated (both diverge from
the scope-hook pattern / need tenantId threading).
Tests: backend 2484 green (catalog/middleware contract updated for the new subset tier),
web 481 green, MCP 292 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Close the write path for MSP mode: a GlobalAdmin can now grant/revoke/enable/
disable delegated cross-tenant read assignments from the portal, instead of
hand-editing the DelegatedAdmins table (no more Boolean/lowercase/5-min-cache
foot-guns). Mirrors the MCP-user management surface.
Backend:
- DelegatedAdminManagementFunction at /api/global/delegated-admins: GET list
(GlobalReadOrAdmin), POST grant, DELETE {upn}/{tenantId} revoke,
PATCH .../enable|disable (all GlobalAdminOnly). Grant defaults role
DelegatedReader, stamps status=Active + source=OperatorGranted + grantedBy;
rejects an unrecognized role (fail-closed). Service.UpsertAsync invalidates
the caller's scope cache, so a grant takes effect on the next request.
- IAdminRepository.GetAllDelegatedAdminsAsync + TableAdminRepository impl
(small full-table scan, off the hot path) + DelegatedAdminService.GetAllAsync.
- Catalog: 5 routes. The {tenantId}-bearing mutation routes use
TenantScoping.RouteParam (fail-closed convention; sets TargetTenantId to the
managed tenant). Policy stays GlobalAdminOnly so a delegated caller cannot
manage delegations (GlobalAdminOnly is not a read tier).
- Mutations are audited under the TARGET managed tenant's trail (CREATE/DELETE/
UPDATE, entityType DelegatedAdmin), so the customer sees who was granted
access to their tenant.
Frontend:
- SectionDelegatedAdmins settings section: grant form (UPN + tenant dropdown +
role), assignments grouped by admin, enable/disable/revoke. Triple-registered
(globalNavConfig ga-settings + settingsNavSections + [section] map).
- api.delegatedAdmins family (list/grant/revoke/enable/disable).
Backend build clean; 2473/2473 backend tests; web tsc clean; 481/481 web tests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Foundation for "MSP mode" — a scoped-global authorization tier between a
single-tenant member and a platform GlobalAdmin. A delegated admin may access a
SUBSET of tenants (its assignment rows), resolved UPN-based and tid-agnostic,
exactly like GlobalAdmins.
Phase 0 is purely additive: the service is registered but no auth path consults
it yet (the cross-tenant gate change lands in Phase 1), so behavior is unchanged.
- Constants: DelegatedRoles/DelegatedStatus/DelegatedSource; new DelegatedAdmins
table added to TableNames.All (auto-create), CriticalBackupTables.All and
AuthTablesFullRestoreForbidden (identity table — full-table restore blocked).
- IAdminRepository + TableAdminRepository: DelegatedAdminEntry DTO + CRUD
(PK=UPN, RK=TenantId). Hot-path GetDelegatedTenantsAsync (partition scan) and
reverse GetDelegatedAssigneesAsync. Queries use the typed predicate overload
(no string-interpolated OData filter — avoids filter injection).
- DelegatedAdminService: resolves DelegatedScope (Covers/RoleFor/CanWrite) with a
5-min cache + write-through invalidation. Fail-closed: only Active+enabled rows
with a recognized role confer scope; empty role -> DelegatedReader; unknown -> dropped.
- 16 unit tests (fake repo); full Functions suite 2434/2434 green. Backup
catalog-count assertion bumped 15 -> 16.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The app dashboard and metrics endpoints loaded entire AppInstallSummaries
partitions (tenant scope) or the full table (global scope) and filtered
StartedAt >= cutoff in memory, even for days=30. The device-model join then
issued one serial GetSessionAsync point-read per distinct session.
- Add optional DateTime? sinceUtc to GetAppInstallSummariesByTenantAsync /
GetAllAppInstallSummariesAsync: when set, a server-side StartedAt ge filter
scopes the scan so only in-window rows are deserialized/transferred. Default
null preserves the legacy full scan for callers that need all history.
- AppsAnalyticsHelper.LoadSummariesAsync derives sinceUtc from days; all six
/api/apps/* functions pass it. The four metrics/app + metrics/geographic
functions pass their existing cutoff. The in-memory Where stays as the exact
sub-second trim (the OData filter is second-granular). MCP get_app_install_metrics
benefits automatically (days already flows through; response shape unchanged).
- Replace the serial session point-read loop with a bounded-concurrency
(SemaphoreSlim=10) LoadSessionLookupAsync, shared by the analytics and
sessions builders.
AppInstallSummaries is already bounded by the session-retention cascade
(DeletionManifestBuilder) + tenant offboarding, so no separate table retention
is added. Full Backend suite: 2406 passed / 0 failed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The retention fanout materialized the entire eligible-session backlog every
run (GetSessionsOlderThanAsync returned a full List<SessionSummary>) but only
advanced MaxEnqueuesPerTenantPerRun=100 per tenant. A large backlog was reread
and rematerialized each run, so maintenance got slower as the backlog grew
while only 100 sessions progressed.
Cap the read at the per-run dispatch limit: GetSessionsOlderThanAsync gains a
maxResults parameter that caps maxPerPage and stops enumerating once the cap is
reached. The fanout fetches MaxEnqueuesPerTenantPerRun+1 (the +1 is a cheap
"more remaining" probe) instead of the whole backlog. Read cost is now
proportional to the cap, not the backlog; per-run progress is unchanged.
The exact eligible count is no longer known without a full scan, so the audit
now records EligibleThisRun + a MoreRemaining flag instead of the exact total.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Several append-only tables had no purge mechanism and grew unbounded. Add
age-based retention to the 2h MaintenanceFunction (and the manual path) via a
new CleanupUnboundedTablesAsync orchestrator (per-table try/catch, fixed const
windows, mirroring the DistressReports pattern):
- GlobalNotifications + TenantNotifications: hybrid retention — dismissed rows
pruned 30d after DismissedAt, any row pruned 180d after CreatedAt. Unread
(still-actionable) admin warnings survive the dismiss window so they are never
silently lost; the 180d catch-all bounds the table. Shared pure predicate
NotificationRetentionFilter.BuildPredicate (unit-tested).
- HardwareRejectionNotificationTracker: 30d (FirstNotifiedAt). Pruning re-arms
the lifetime dedup, so a model rejected again >30d later re-bells (intended).
- AuditLogs: 180d (system Timestamp, cross-partition).
- UsageMetrics: 180d (PartitionKey date-string range).
- BackupJobs: 365d (QueuedAtUtc).
PlatformStats intentionally excluded — it is a single upserted row, not
append-only. Bootstrap sessions left as-is (feature currently unused).
Each repo delete mirrors DeleteUserActivityOlderThanAsync: server-side
"X lt cutoff" filter + select PK/RK + delete loop. Deletes are resumable across
runs, so a partial run self-heals on the next cycle.
Observability: end-of-run MaintenanceLongRunning Warning OpsEvent if RunAllAsync
exceeds 10min, dual-registered in OpsAlertRulesSection.tsx, to surface a run
climbing toward the 60min functionTimeout before a future hard-abort (which
emits nothing).
Inert until backend redeploy; first post-deploy run bulk-prunes the backlog.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Global-Admin cross-tenant view of web users active "now" (made an authenticated
request within a window, default 5min), backed by a new self-maintaining
UserPresence table. Not App Insights (sampling) / not SignalR.
Backend:
- UserPresence table (one row/user, PK=tenantId, RK=SHA-256(lowercase UPN));
upserted fire-and-forget by UserPresenceMiddleware (after PolicyEnforcement,
60s per-process throttle, UPN-gated so agent cert calls are skipped).
- GET /api/global/presence (GlobalReadOrAdmin) via GetActiveUsersFunction; read
path propagates storage errors (5xx) instead of masking them as "0 active".
- 1-day presence retention + 90-day UserActivity retention in maintenance;
PlatformStats.TotalUsers clamped monotonic so the public "Users Seen" figure
cannot regress after a prune.
Web:
- /admin/presence page (live counter + table), nav entry under GA > Metrics.
- "Total Users" relabeled "Unique Users (90d)" to match the windowed metric.
Backend+web only, no ConfigVersion bump, inert until redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Born from /session-debug analysis of f148976f: a customer platform script
(c3e0124c) hung for the full ~30-min IME script-execution timeout, blocking
IME's serial platform-script queue, so the Autopilot-Monitor bootstrap +
agent only started ~34 min after boot (right as ESP hit its own 30-min
SyncFailureTimeout). The agent observed only the already-decided end-state
and lived ~30s — a session that looks like a normal multi-minute failure but
had near-zero coverage of the actual failure window.
Two detectors classify this automatically:
- agent_late_start (Warning, EnrollmentTerminationHandler): fires when the
agent started >=10 min after boot AND lived <=5 min before a terminal
outcome. Device boot via GetTickCount64 P/Invoke (net48 has no
Environment.TickCount64), monotonic / clock-jump-resistant. Shared gate in
Termination/ObservationCoverage.cs (DRY with the builder). final-status.json
gains bootToAgentStartSeconds + lowObservationCoverage (FinalStatusBuilder
stays pure via an optional deviceBootUtc param).
- script_timeout_suspected (Warning, ImeLogTrackerAdapter): a platform script
that ran >=25 min AND ended Failed -> one-shot per policyId, naming it as the
likely enrollment-pipeline blocker. Works on replay (StartedAtUtc from the
CMTrace source timestamp), so it fires even when the script ran before the
agent launched.
Plus per-script run duration for all platform scripts: new
ScriptExecutionState.StartedAtUtc -> durationSeconds on every
script_completed/script_failed -> Web ScriptExecutions renders "Duration"
(amber past 10 min) so slow/inefficient scripts surface generally.
Scope: platform scripts only (remediation/health-script duration deferred --
the multi-source HS reducer makes clean attribution fragile). No ConfigVersion
bump, no backend code (ingest is event-agnostic). New event types wired through
the mandatory dual-sync: Constants.EventTypes -> MCP resource-catalog ->
Web eventTypes (drift tests enforce).
Tests: V2.Core 1536 (+88), MCP drift 6, MCP/Web tsc 0, Web 84. The f148976f
case is reproduced in unit tests (ObservationCoverage 34min/30s; adapter
c3e0124c 30min/Failed). Inert until redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lets tenant-admins and GAs run end-to-end session debugging over MCP: backend
telemetry plus the agent diagnostics ZIP, downloaded and analyzed locally by the
AI client. The backend never unzips/parses the archive (no parse CPU); it only
mints a short-lived signed download ticket and proxy-streams bytes.
Backend (no ConfigVersion bump, no agent change):
- DiagnosticsDownloadTicket: HMAC codec (reuses PaginationTokenSigningKey,
purpose-tagged diag-dl-v1, 10-min TTL) so a no-JWT client can download.
- DiagnosticsBlobStreamer: shared proxy-stream engine; DiagnosticsDownloadFunction
(web/JWT) refactored onto it. ClassifyBlobName/BlobDestination stay put (xUnit).
- POST diagnostics/download-ticket (MemberRead, QueryParam, GA-bypass) +
GET diagnostics/download?t= (anonymous; tenantId/blobName read only from the
signed ticket -> no retargeting). Both added to NoStoreCacheMiddleware.
MCP:
- get_session_diagnostics tool (all roles) -> absolute downloadUrl + static zipMap
+ instructions; unwraps the {success,session} envelope.
- diag_zip_layout catalog (get_resource + protocol resource).
- debug-session prompt (tenant + GA variants).
- Tool descriptions flag the client requirement (needs local file/shell tools;
pure chat clients cannot unzip).
Tests: backend 2380/2380 (ticket codec, NoStore, classify, catalog completeness);
MCP 261 passed (incl. envelope-unwrap regression). Inert until redeploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two V2 IME fixes, verified against client logs of sessions fc48c71a / 6b4993e5.
Bug 1 — on an ESP terminal failure of the DeviceSetup Apps subcategory the only
device app was stuck mid-install (RealmJoin stuck Downloading 0%) and IME logged
no per-app error, so app_tracking_summary showed 0 failed and the failure was
visible only via the opaque "Apps (Error)" registry string. New
esp_apps_failure_correlation event names the in-flight (IsActive: InProgress/
Downloading/Installing) device app(s) as the likely cause WITHOUT mutating state
(no fabricated app_install_failed) and only on the Apps subcategory.
Bug 2 — AgentExecutor.exe hosts both platform and proactive-remediation scripts
interleaved in the shared AgentExecutor.log; the policyId-less PS-AGENT-EXITCODE /
PS-AGENT-OUTPUT lines from a remediation invocation bled into the last-started
platform script via the sticky pointer (c3e0124c emitted result=Failed but with
exit 0 + the Teams remediation's "[Compliant] No Classic Teams found" stdout).
New PS-AGENT-INVOCATION pattern (resetPlatformScriptContext) clears the platform
line-capture pointer at every "ExecutorLog AgentExecutor gets invoked" boundary,
so a remediation invocation cannot capture into a platform slot; the platform
result stays keyed by policyId via PS-SCRIPT-RESULT.
Agent-only, V2, inert until redeploy; no ConfigVersion bump. New event type added
to the MCP resource catalog (dual-sync). Green: V2.Core 1522, DecisionCore 400,
Backend BuiltInRules 8, MCP vitest 252.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GlobalReader = cross-tenant READ visibility like a Global Admin, but no mutating
operations and no platform settings. Provisioned exactly like a GA: a GlobalAdmins
table row with Role=GlobalReader (empty/legacy Role = GlobalAdmin, back-compat).
Semantics are ADDITIVE — a user who is also their own tenant's Admin keeps edit
rights on that tenant; a pure GlobalReader is read-only everywhere.
Backend:
- GlobalAdmins.Role column + GlobalAdminService.GetGlobalRoleAsync; IsGlobalAdminAsync
(incl. TableAdminRepository) delegates so a GlobalReader row never reads back as admin
- RequestContext.IsGlobalReader + HasGlobalScope
- Split GlobalAdminOnly into GlobalReadOrAdmin (cross-tenant reads) vs GlobalAdminOnly
(writes/settings) + TenantAdminOrGlobalReader for config-read
- PolicyEnforcementMiddleware: extracted testable DecideAsync seam; cross-tenant gate keys
off HasGlobalScope; additive own-tenant role carried so IsTenantAdmin stays correct
- Config secret redaction for readers (RedactedCopyForReader) + save-time placeholder
restore (RestoreRedactedSecretsFrom) so a redacted view can't blank real secrets
- raw table/log tools stay GlobalAdminOnly; SignalR global-admins read group admits HasGlobalScope
- /auth/me emits isGlobalReader (+ preview-gate bypass, no auto-admin); /auth/mcp emits globalRole
MCP: hasGlobalScope (GA or Reader) drives routing + tool catalog; raw secret-bearing tools
(query_table/list_tables/query_backend_logs) gated strict-GA only.
Web: AuthContext isGlobalReader + hasGlobalScope; reader reaches /dashboard + cross-tenant
views; platform Settings/Ops/Backups/Software GA-only (nav-hidden + requireGlobalAdmin route
layouts); mutating controls on mixed data pages disabled via useCanMutatePlatform; rules pages
read-only on foreign tenants; additive own-tenant edit preserved.
Tests: backend 2369, MCP 252, web 469 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds WebhookProviderType.GenericJson (20) so a tenant can POST a stable,
channel-agnostic JSON payload to any HTTPS endpoint - ticketing systems,
automation, or an SMTP gateway like Postal - instead of only Teams/Slack.
Backend:
- GenericJsonRenderer emits schemaVersion 1.0 with eventType, severity,
title/summary/themeColor, primaryUrl (first openUrl action), facts[],
sections[], actions[]; registered in WebhookNotificationService.
- NotificationAlert.EventType set on all 8 builders (enrollment_succeeded/
_failed/_started, preprovisioning_*, hardware_rejected, sla_breach/_resolved,
consecutive_failures, test) for consumer-side routing.
- Per-tenant WebhookCustomHeadersJson + GetGenericWebhookHeaders() (generic-only,
restricted-header denylist, fail-soft parse) for API-key auth; dispatcher
gained optional customHeaders applied via HttpRequestMessage; threaded into
all 12 tenant call sites (Ops/admin-config sites unchanged).
- TableConfigRepository persists the field; UpdateTenantConfigurationFunction
validates it (object, string values, RFC-7230 token names, CRLF-injection
guard, 8 KB / 25-header caps).
Web:
- Generic provider option + key->value WebhookHeaderEditor (rendered only for
provider 20); leaving the generic provider clears local headers so a later
switch back cannot revive stale auth headers; context/types wiring sends
headers only when provider===20; report shows Generic JSON provider name.
ConfigVersion unchanged (backend-only, no agent payload). No new
Constants.EventTypes, so no MCP catalog sync. Still fire-and-forget (no SMTP
client / retry by design). New GenericWebhookTests cover renderer, header
parsing/denylist, and save-time validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Surface WMI watcher arm failures: new ConsoleBypassWatcher.WatcherArmFailed
event -> host emits collector_degraded (reason=watcher_arm_failed) via
CollectorDegradationReporter, so the backend can tell "no console seen" from
"live detector never started" instead of both looking like a clean run.
- Close a startup blind gap: arm the live Win32_ProcessStartTrace FIRST, then
run the startup snapshot. A cmd spawned between snapshot and arm was previously
missed; the _seen set dedups the now-possible overlap.
- Fix command-line classification for combined cmd switches: strip the executable
token, then match /c|/k anywhere in the arguments (was anchored to whitespace),
so e.g. `cmd /d/q/c exit 9` is correctly scripted (was false high-confidence).
- Docs: replace stale parent-based wording (AgentConfigResponse, the Web analyzer
toggle help text, the prefetch coverageNote) with the interactive-session model.
Tests: combined-switch + StripExecutable cases added. V2.Core 1505/1505.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Field session f240a555 (Win11 24H2) disproved the parent==winlogon assumption:
the Shift+F10 cmd's parent (pid 9016) was a short-lived non-winlogon launcher
that exited within ms ("parent resolve failed ... is not running"), and the cmd
ran as defaultuser0, not SYSTEM. The conservative "can't confirm winlogon -> don't
flag" path therefore dropped a real Shift+F10 console.
ConsoleBypassWatcher rewritten to be parent-free:
- Gate race-free on SessionID != 0 (interactive session) from the trace — IME/
Intune install `cmd /c` run as SYSTEM in session 0 and are dropped without a probe.
- Confirm on the cmd's OWN command line: bare (no /c|/k) = interactive shell ->
confidence=high; a scripted `cmd /c` is ignored.
- Instant-close race (command line unreadable before the probe) -> still surfaced
as confidence=low (interactive_session_unclassified) rather than missed.
- Event data now: sessionId, owner, commandLine, confidence, classification,
parentProcessId (dropped parentProcessName/parentMatchesWinlogon).
Lifecycle gate: ConsoleBypassHost.StopForDesktopArrival() wired into the
DesktopArrival onRealUserOwnerObserved callback — once the real-user desktop
arrives (excludes SYSTEM + defaultuser0) Shift+F10 is no longer possible, so the
watcher stops and cannot flag ordinary post-enrollment user consoles. AccountSetup
phase alone is too early (the portal shows it before the real desktop, while
Shift+F10 still works), so desktop arrival is the binding stop signal.
Tests rewritten: Classify + HasScriptArgument (pure), high/low-confidence,
session-0 ignore, scripted ignore, dedup, owner enrichment. V2.Core 1499/1499.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Detects a SYSTEM console opened during enrollment — the classic Shift+F10
OOBE bypass (winlogon.exe spawning cmd.exe as SYSTEM). Two complementary
Warning signals, one per-tenant toggle EnableConsoleBypassDetection
(default ON / opt-out), ConfigVersion 31 -> 32:
- oobe_console_spawned (live): ConsoleBypassWatcher + ConsoleBypassHost.
WMI Win32_ProcessStartTrace on cmd.exe + startup probe; flags only a
cmd.exe whose parent is winlogon.exe. PID dedup happens ONLY after a
confirmed winlogon match, so an ignored cmd cannot suppress a later real
one via PID reuse.
- console_prefetch_detected (startup forensic): ConsolePrefetchScanner
(IAgentAnalyzer). A CMD.EXE-*.pf prefetch artifact whose last-run is
after boot, covering the pre-agent OOBE window the live watcher misses.
stat-only; corroborating artifacts sorted for a stable gate fingerprint.
Both events carry coverageComplete:false + an honest coverageNote (and the
prefetch event attribution:"coarse") — detection is best-effort, not gapless,
and only reports, never blocks.
Plumbing: Constants (2 event types) + MCP resource-catalog.ts + Web
eventTypes.ts; AnalyzerConfiguration/TenantConfiguration + GetAgentConfig
(default true) + TableConfigRepository; DefaultComponentFactory host gate +
AgentAnalyzerManager registration; full Web opt-out toggle. Tests: watcher
classification (9) + prefetch scanner (7) + analyzer-manager on/off.
Also fix(web): unrestricted-mode toggle persisted the stale pre-toggle value
because onSave() read async React state. saveConfiguration now takes an
explicit unrestrictedMode override that the toggle passes through, so the
saved value matches the UI state.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In larger tenants the multi-tenant app is often pre-approved by someone
with consent rights, while the admin operating our UI lacks them. The
/adminconsent redirect then fails, so ValidateAutopilotDevice/
ValidateCorporateIdentifier never persist and the tenant stays "Disabled"
even though the app + DeviceManagementServiceConfig.Read.All effectively
exist. That bool is also the hard gate in SecurityValidator, so a UI-only
badge would leave agent traffic 403'd.
Add a read-only access-check probe that verifies the concrete role in the
SP token's roles claim (via GraphFeatureDetector, not mere token
acquirability) and, when present, persists the gate bool through the
normal config PUT — opening the UI badge AND the agent gate together.
Backend:
- GraphAppPermissions.DeviceManagementServiceConfigReadAll constant.
- GET /api/config/{tenantId}/autopilot-device-validation/access-check
(TenantAdminOrGA): InvalidateTenant + fresh GetSnapshotAsync (4s budget),
pure ResolveAccessPresent (transient => never present), emits
ConsentAccessDetected telemetry. Always-fresh so a propagation-time
empty-roles snapshot can't wedge the admin out for the token TTL.
Web:
- lib/accessCheck.ts classifyAccessCheck pure fn (transient/absent never
reconcile) + tests.
- Reconcile on consent-redirect failure (silently enables) and a per-toggle
"Detect existing access" button for Autopilot, Corporate, and DevPrep
Device Association (all share the same role).
- Successful consent callback now verifies the concrete role too: persists
ONLY on authoritative "reconciled"; transient => retry/propagation
message, absent => "permission not granted" — neither opens the gate.
- saveConfiguration returns boolean so success is shown only after a
confirmed persist.
No ConfigVersion bump (backend + web only). Backend 2326/2326, web 464/464.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oliver Kieselbach