Deploy-verified on f4d51ea4: the worker-side adaptive-sampling exclusion
(EnableAdaptiveSampling=false + manual Configure<TelemetryConfiguration>
UseAdaptiveSampling(excludedTypes:"Request;Event")) does NOT take effect in the
Functions isolated worker. Live data showed the enriched worker request copy
(Source=WorkerMiddleware) sampled to ~63% while the redundant host request item
(Source="") stayed fully billed — strictly worse (lossy enriched copy, host
duplicate unchanged). host.json "Host.Results": "Error" likewise did not drop
the host request item (emitted by the host process, not governed by that level).
Restores known-good: default worker sampling (enriched copy complete at normal
traffic, burst-capped) + host.json Host.Results back to Information. L1 storage
dependency filter and L2 Function-trace suppression are unaffected and still
verified live. L4 deferred pending a mechanism verified against the isolated-
worker pipeline. H2/H3/O1/O2 from f4d51ea4 verified healthy (0 exceptions).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Telemetry cost (L4): worker-side adaptive sampling now excludes Request;Event
so the enriched RequestTelemetryMiddleware copy is never thinned; host.json
Host.Results=Error drops the duplicate unenriched host request item. One
complete request row per call instead of two (one unsampled+unenriched, one
sampled+enriched).
Ingest hot path (H2/H3): BlockedDeviceService caches negative answers (30s
revalidation, symmetric with positive entries) so healthy devices no longer
pay a BlockedDevices point-read per request; SessionDeletionGuard returns its
already-loaded row and IncrementSessionEventCountAsync returns the post-merge
snapshot, collapsing ~4 Sessions reads/batch to ~2.
Queue-worker correctness (O1/O2): new GetSessionEventsStrictAsync propagates
storage failures on the analyze + vulnerability-correlate paths so a transient
read fault retries/poisons instead of silently deleting the message after
"analyzing" an empty stream. Fail-soft variant kept for UI/metrics surfaces.
Review fixes: parse prefetched Status as string (the actual wire format, was
int -> optimization never fired); session-aware block over a cached negative
entry keeps its session scope instead of widening to whole-device.
2274/2274 backend tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Setting requestTelemetry.Context.Location.Ip never reached the requests
table: App Insights uses the IP only for geo-lookup then masks the stored
client_IP to 0.0.0.0 (verified — every request in the resource shows
0.0.0.0). On top of that, the only IP reaching the isolated worker is an
Azure-infra hop (West Europe / Amsterdam), not the device's real egress IP
(device geo = Falkenstein). A usable device IP would need a non-masked
custom property AND the correct forwarded header (Front-Door-topology
dependent) — not worth it: TenantId + CorrelationId already isolate a
device (they dissolved the IngestTelemetry "multiplication" illusion).
Remove the dead Context.Location.Ip block + now-unused Security using,
replace with a NOTE documenting why client_IP is intentionally absent so it
is not re-added. ClientIpExtractor.GetClientEgressIp stays (still used by
ReportDistressFunction's stored SourceIp, which is not App-Insights-masked).
Full backend suite 2249/2249 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Makes the requests table actually usable for diagnosing agent traffic and rate
limits, without adding any new telemetry items — only properties on the already-
emitted, sampling-excluded Request telemetry (host.json excludedTypes "Request;
Event"), consistent with StorageDependencyFilterProcessor.
RequestTelemetryMiddleware:
- client_IP: the isolated worker behind Front Door/App Service left it 0.0.0.0.
Set Context.Location.Ip from X-Azure-ClientIP (Front Door's true client egress
IP) else trusted rightmost X-Forwarded-For. Header values are joined via
StringValues.ToString() so the rightmost trusted hop wins (not a spoofable
left entry). Diagnostic only — the rate limiter independently uses the trusted
hop, so a spoofed value here cannot affect throttling.
- TenantId: agent ingest skips PolicyEnforcementMiddleware so reqCtx.TenantId was
empty. Fall back to X-Tenant-Id, but ONLY on an exact allowlist of device
endpoints that actually validate it (agent telemetry/ingest/error/distress,
bootstrap ingest/error) and only when it is a well-formed GUID — a prefix match
would let anonymous/management routes (bootstrap/validate, bootstrap/sessions,
agent/config, …) be tagged with a spoofed tenant.
RateLimitService: 429 warning is now structured ({RequestsInWindow}/{MaxRequests}
/{WindowSeconds}/{RateLimitKeyPrefix}) so it's queryable; privacy-safe (8-char
prefix only — the key may be a cert thumbprint, bootstrap token, or UPN).
ReportDistressFunction: store the real device egress IP in SourceIp (table entry
+ AgentDistressReport event) via new ClientIpExtractor.GetClientEgressIp; the
rate-limit key keeps the un-spoofable trusted hop.
Full backend suite 2249/2249 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a tenant-scoped, informational overview of devices that failed enrollment
with HTTP 403 because they were not in the tenant's Autopilot / Corporate
Identifier registry. Mirrors the Hardware Whitelist rejection insights but
aggregates per device (serial number), matching how Autopilot registers
individual devices rather than models. Read-only, no webhook.
Backend
- GetDeviceNotRegisteredFunction: route audit/device-not-registered, MemberRead,
filters distress reports on ErrorType == "DeviceNotRegistered", aggregates by
serial (case-insensitive; empty serial -> single bucket); 14-day horizon is
the existing distress-report retention.
- Register route in EndpointAccessPolicyCatalog (fail-closed).
- 8 unit tests for BuildAggregatedResult.
Frontend
- api.distress.deviceNotRegistered().
- NotRegisteredDevicesInsights component (serial/mfr/model/attempts/first/last
seen, green empty-state, unverified-data notice).
- Wired below AutopilotValidationSection.
DeviceNotRegistered (403) covers both Autopilot and Corporate Identifier
validation; the agent cannot distinguish which failed, so copy is neutral.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ANALYZE-SEC-002 fired Warning (confidence 90) on every normal user-driven
Entra-join enrollment. During ESP, Windows writes AutoAdminLogon=1 +
DefaultUserName + AutoLogonSID + no DefaultPassword for the enrolling user —
the exact fingerprint the sysinternals_autologon_suspected heuristic targeted
(enabled && user && !password). AutoLogon-enabled is registry-indistinguishable
from a Sysinternals/kiosk setup during Autopilot, so the heuristic and SEC-002
could never tell a benign enrollment from a real one.
Declare AutoLogon only when a plaintext password is actually present:
- delete ANALYZE-SEC-002 (broad AutoLogon-active warning)
- remove the non-discriminating sysinternals heuristic from AutoLogonAnalyzer
(computation + finding label + checks.sysinternals_autologon_suspected)
- strip the sysinternals condition+factor from SEC-004 (kiosk allow-list, still
a disabled opt-in template) and refresh its now-stale SEC-002 guidance
- ANALYZE-SEC-003 (plaintext DefaultPassword -> high) is now the sole default-on
AutoLogon grade; the autologon_analysis event still carries all raw facts at Info
Tests: assert SEC-002 absent + no rule references the sysinternals signal; add
analyzer regression guard for enabled+user+no-password. V2-only, no ConfigVersion
bump. Build 0 err, AutoLogon 19/19, BuiltInRulesTests 8/8, web tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaces the real Microsoft 365 Apps Click-to-Run install lifecycle that the
Intune "integrated" app hides (IME reports done in 1-2 min while C2R keeps
streaming in the background). V2-only.
Event-driven, no idle polling:
- OfficeProcessWatcher: WMI Win32_ProcessStartTrace push on OfficeC2RClient.exe
(+ startup probe) for start; Process.Exited for stop, debounced by a settle
window (OfficeInstallSettleSeconds, default 10s) so short worker gaps in one
install don't trigger a premature office_install_failed.
- RegistryChangeWatcher: RegNotifyChangeKeyValue push on ...\Office\ClickToRun
(Configuration value + Scenario subkey churn); bootstrap fallback watches the
nearest existing ancestor non-recursive + name-only until ClickToRun appears.
- OfficeInstallDetector: pure state machine (Idle->Active->Terminal), emits
office_install_started/progress/completed/failed (progress on real change only,
no heartbeat). Error code only from non-zero numeric/hex Scenario values.
Office Delivery Optimization:
- DeliveryOptimizationCollector extended with a 2nd wake source (Office active),
classifies non-IME Office-CDN jobs and folds aggregated DO bytes/peers/real
download-% into the office_install_* events (not emitted as download_progress
to avoid a phantom app in the backend AppInstallSummary).
Plumbing: office_install_* event-type constants, EnableOfficeInstallDetector +
OfficeInstallSettleSeconds config (ConfigVersion 28->29), MCP + web event-type
catalogs, ANALYZE-OFFICE-001 (office_install_failed -> Warning). 22 detector unit
tests; full V2 suite green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A telemetry batch containing two rows with the same RowKey made Azure Table
Storage reject the entire entity-group-transaction with HTTP 400 /
InvalidDuplicateRow, surfacing as HTTP 500 on POST IngestTelemetry. Observed
as a 23-minute spike (36x 500) from a single session replaying an overlapping
SessionSignalOrdinal; self-resolved when the agent stopped retrying.
The three V2 ingest write paths (Signals, DecisionTransitions, Index) built
the transaction without deduping duplicate RowKeys and had no fallback (unlike
the Events path). Add a shared TableBatchDedup.ByRowKey helper that collapses
duplicate (PartitionKey, RowKey) entities last-write-wins (idempotent
UpsertReplace) and logs a WARN with the dropped count, so bad agents become
visible in logs instead of failing the batch. SubmitTransaction stays
fail-loud for any other batch error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fleet Health previously drained up to 200k raw sessions into the browser
(MAX_EAGER_PAGES=200 x 1000) over as many HTTP round-trips, then computed
7 aggregates in useMemo on the main thread. Move that aggregation to the
backend, mirroring the dashboard-stats server-aggregation path.
- MetricsMath.BuildFleetHealthPayload: single pass over the windowed
session list producing stats, daily timeline, top failure reasons, and
model health/slowest/top-failing breakdowns. Semantics ported verbatim
from the client (success rate over ALL sessions; avg duration over
non-in-progress sessions incl. failures), so displayed numbers are
unchanged. Typed FleetHealthMetrics DTO in Shared for a typed contract.
- GET /api/metrics/fleet-health (MemberRead) and
GET /api/global/metrics/fleet-health (GlobalAdminOnly, optional
?tenantId=) reuse the existing ISessionRepository window readers.
- Registered both routes in EndpointAccessPolicyCatalog (fail-closed).
- 9 xUnit tests covering the aggregation semantics.
Frontend swap to these endpoints lands in a follow-up commit. App-metrics
(slowest/failing apps) is already backend-aggregated and untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surface "how many reboots did this enrollment have" as a stored,
filterable per-session metric so admins can ask e.g. "machines with
many reboots" via MCP search_sessions.
No agent change: V2 already emits one system_reboot_detected per reboot
(EventLog 6009 boot-time vs session start). The backend counts and stores
it; V1 stays 0 (never emits it).
Hybrid counting:
- Incremental per ingest batch in ClassifyEvents (V2 + legacy V1, shared
EventClassification) -> rebootIncrement on IncrementSessionEventCountAsync,
for a live in-flight value (passed 0 on terminal batches).
- Authoritative reconcile (ReconcileSessionRebootCountAsync): distinct count
of system_reboot_detected from the Events table, idempotent + fail-soft.
Runs centrally in TableSessionRepository.UpdateSessionStatusAsync after any
successful terminal transition (ingest, admin mark, maintenance timeout,
rule-engine fail), and again post-increment in the ingest path to cover
already-terminal batch replays. Self-corrects at-least-once batch double-counts.
Storage/correctness:
- RebootCount preserved across re-registration in StoreSessionAsync (a reboot
IS the restart trigger) and projected in BuildSessionIndexEntity (superset rule).
Search/MCP:
- rebootCountMin/Max pushed to OData in BuildSearchScanFilter (sparse-subset),
decoded in SearchSessionsFunction + QueryRawSessionsFunction, added to the
continuation fingerprint and the fields= projection. Client-side
MatchesRebootCountBounds backstops the device-snapshot + legacy scan paths.
- search_sessions MCP tool gets rebootCountMin/Max + rebootCount projection.
Tests: SessionIndexProjection (superset), StoreSessionReregistrationPreserve,
RebootCountFilterBounds. Suite 2228/2228 green; MCP 220 green.
Inert until backend redeploy; legacy index rows lack the column until next
upsert (excluded by min-filter, intended). Web UI badge deferred.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The apps/list endpoints returned the full per-app list in one shot, so there was
no way to page a large fleet's apps (e.g. for a future MCP consumer).
Backend (apps/list + global/apps/list):
- AppsAnalyticsHelper.BuildAppsListResponse gains opt-in pagination: no pageSize
keeps the legacy full-array body (the web UI pages client-side); pageSize
returns an offset-paginated envelope with count/offset/pageSize/nextLink.
- The list is aggregated in-memory from a deterministic sort, so a plain integer
offset is a stable cursor — no HMAC continuation token needed (unlike the
Azure-Table-backed config/all surface). Added an appName tiebreaker so the sort
is fully deterministic across pages.
- New ParseAppsPaging validates pageSize (1..1000) / skip; both functions 400 on
bad input and build a route-specific nextLink (global echoes tenantId).
Web (Installs tab): client-side pagination (pageSize 20) over the filtered/sorted
apps, matching the Inventory tab — sort/search stay across the full set.
Tests: AppsAnalyticsHelperPagingTests (legacy vs paginated, deterministic order,
last-page nextLink null). Backend 2206/2206, web 446, tsc clean, next build OK.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- DO rollup no longer drops legacy telemetry. MetricsMath.BuildAppMetricsPayload
now delegates to DoAggregator.Compute (the existing single source), which falls
back to peers + http when DoTotalBytesDownloaded is 0 and filters to rows that
actually carry DO telemetry (DoDownloadMode >= 0). Removes the duplicate
summation that used DoTotalBytesDownloaded directly as the denominator.
- InstallsTab DO card now renders whenever any DO source bytes exist (effective
total = max(total, peers+MCC+http)) instead of hiding on totalBytesDownloaded <= 0.
- get_software_inventory: guard the GA inventory path — a Global Admin call with
scope="inventory" and no tenantId now returns actionable guidance instead of a
raw backend 400.
Tests: new MetricsMath fallback test; DO test rows set DoDownloadMode so they pass
the telemetry filter. Backend 2203/2203, MCP 220, web 446, all tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Backend:
- Extract MetricsMath.BuildAppMetricsPayload as the single source for the
app-metrics aggregation (was duplicated verbatim across the tenant and
global functions) and add a Delivery Optimization rollup: per-app peer/MCC
offload bytes + overall deliveryOptimization summary (peers and MCC counted
separately, peerOffloadPercent credits both as not-from-CDN).
- New tenant-scoped GetTenantSoftwareInventoryFunction (GET metrics/software-
inventory, MemberRead, JWT tenant) + fail-closed policy-catalog entry.
MCP server:
- get_app_install_metrics: role-aware routing (global/metrics/app vs
metrics/app), surfaces the DO rollup.
- get_software_inventory: one role-aware tool; non-GA sees only its own tenant
(no tenantId/scope in schema), GA additionally gets tenantId + scope=unmatched.
New exported paginateInventory client-side pager.
Tests: MetricsMathAppPayloadTests (DO rollup/offload/sample-floor) + vitest
paginateInventory. Backend 2202/2202, MCP 220 passed, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reworks PPKG detection to one aggregate event the rules parse as an array,
adds a default + customizable allow-list, and makes the event storage-safe.
Rule engine:
- New condition source event_data_array + ItemField: iterates an array field
(any-element match), with dot-path dataField support. (RuleEngine.ConditionEvaluators)
Agent:
- Drop per-package provisioning_package_detected events (noise: a clean Win11 VM
ships ~22 OS-inbox .ppkg). One provisioning_package_scan carries an artifacts[]
array; anyPpkgFound derived from it (consistent, incl. recovery residue).
- Size-based chunking: split artifacts across multiple scan events so each event's
Newtonsoft DataJson stays < 24k (well under the 30k Table-Storage truncation that
would blind the rule); never drops a tail artifact. Per-artifact string fields
(identity/dir/rank/installTime) hard-capped at 256; scanErrors capped.
Rules:
- ANALYZE-SEC-005 (default on) + SEC-006 (template) fire on artifacts.identity
not matching an anchored allow-list (^(?:...)\b) that covers OS-inbox families.
- ANALYZE-SEC-007 alarms when a per-source collection cap was hit (scanTruncated).
Web/MCP/docs: event_data_array in the rule form (types + fields), analyze-rules
docs section + example, event-type catalogs.
Tests: rule-engine array/anchor/dot-path/SEC-007; agent artifacts/chunking/field-cap
/worst-case-size. Build green; V2.Core 1236; backend rule + web tsc green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
V2-only AutoLogonAnalyzer that reads the Winlogon AutoLogon configuration and
emits an autologon_analysis event with raw facts at Info severity. AutoLogon can
be a legitimate kiosk/shared-device setup or a way to automate enrollment/OOBE;
the agent reports facts only and never reads the DefaultPassword value (presence
detected via value-name enumeration). Backend analyze-rules grade the facts.
- AutoLogonAnalyzer (always-on, no tenant toggle): startup no-op, runs at
DeviceSetup-phase completion (new AutoLogonDeviceSetupTrigger on
EspAndHelloHost.DeviceSetupProvisioningComplete) and at final shutdown.
- Sysinternals Autologon heuristic (AutoLogon enabled + default user + no
registry password = LSA-secret credential); never reads the LSA secret.
- Analyze rules: ANALYZE-SEC-002 (AutoLogon enabled -> warning, default-on),
ANALYZE-SEC-003 (plaintext DefaultPassword -> high), ANALYZE-SEC-004 (kiosk
allow-list template). Sysinternals graded via a real confidence condition
inside SEC-002/004 so the allow-list still suppresses approved kiosks.
- autologon_analysis added to all event-type catalogs; rule-builder picker also
gains the previously-missing integrity_bypass_analysis + local_admin_analysis.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
StoreSessionAsync rewrites the Sessions row with UpsertEntity (Replace mode)
on every agent restart (same session id re-registration). The preserve-list
carried EventCount/CurrentPhase/durationSeconds/geo but had forgotten
PlatformScriptCount and RemediationScriptCount (added later), so Replace
zeroed them on each restart. Combined with the subsequent full index upsert
and the increment-merge only writing a counter when its increment > 0, this
undercounted scripts and drifted Sessions vs. SessionsIndex (e.g. get_session
platformScriptCount=0 while the index-backed list showed 3).
Preserve both counters from the existing row, mirroring the EventCount path.
Add StoreSessionReregistrationPreserveTests covering preservation on
re-registration and zero-default on first registration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MCP-side mirror of the web interpolateRuleTemplate: resolve
{{reason}}/{{appName}}/{{errorCode}}/... from each rule result's
matchedConditions before get_session_details and get_session_summary
ship the explanation + remediation text, so MCP tool output never leaks
raw {{...}} placeholders. Resolution order is kept in lock-step with the
web copy (byField -> whitelisted auto-field -> signal name).
Also rewords the ANALYZE-ENRL-001 footnote to explain that a remaining
literal {{...}} means the field simply was not recorded for this session
(version 1.1.0 -> 1.1.1; compiled dist and the rule-version assertion in
RuleEngineDetectionFailureTests updated to match).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Three data-hygiene fixes for the MCP metric tools, all driven by
incomplete/abandoned sessions distorting aggregates:
- get_usage_metrics: clamp each session duration to the shared 7-day
ceiling before avg/median/p95/p99, so stuck non-terminal sessions
(unclamped now-StartedAt wall-clock) no longer push p99 into weeks.
Surface SampleCount + ClampedSessionCount so the skew stays visible.
- get_metrics: tally Pending/Stalled into their own buckets and route
any other status into `other`, so the per-tenant buckets reconcile to
total by construction (closes the silent 214/134 gap). Extracted the
fold into SessionStatusBuckets.
- slowestApps: require >=3 successful installs via the shared
MetricsMath.SelectSlowestApps helper, so a single N=1 (often
unfinished, legacy pre-clamp) install cannot dominate the ranking;
also de-dupes the copy-pasted block across the tenant + global
app-metrics functions.
Adds xUnit regression cover for all three.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The get_audit_logs MCP tool could only narrow by tenant + date window, even
though the audit entity already stores Action/PerformedBy/EntityType/EntityId.
Add exact-match server-side filtering across all three layers so a filtered
query never falls back to in-memory scanning.
- Shared: AuditLogQueryFilters carrier + extended IMaintenanceRepository
- Storage: shared AppendAuditFieldFilters (OData-escaped eq clauses) in both
the tenant and global fan-out filter builders; threaded through page +
legacy methods
- Functions: new shared AuditLogFilterRequest parser folds filters into the
pagination fingerprint AND echoes them on nextLink (cross-filter token
replay rejected, follow-up pages round-trip), identical for tenant + global
- MCP: four optional params on get_audit_logs + pass-through
- Tests: AuditLogFieldFilterTests (eq clauses, escaping, coexistence with
performer suppression + deletion exclusion, extras ordering)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
refactor(backend): extract shared MetricsMath.Percentile helper
CalculatePercentile was duplicated verbatim across four services
(UsageMetricsService, SlaMetricsService, SlaBreachEvaluationService,
MaintenanceService.Aggregation) with only cosmetic differences. Hoist
the nearest-rank percentile into Helpers/MetricsMath and route all call
sites through it. No behavior change; callers already pre-sort.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@
The effective consecutive-failure threshold was derived two different ways:
the timer path used Math.Max(cfg, 2) while the inline path used (< 2 ? 5).
For a sub-minimum config value (0/1) this desynced the raise window (inline,
5) from the resolve window (timer, 2), so a breach requiring 5-in-a-row could
be resolved as soon as the top 2 weren't both failed.
Introduce a single canonical EffectiveConsecutiveFailureThreshold helper with
(< 2 ? 5) semantics (5 = the default used by model, DB read, and web UI) and
use it at all three sites. Extract BuildInlineService/FailedPage test helpers
and add a sub-minimum-threshold fallback test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
ResetAutoResolveFailedCache() was documented as "called when seed/community
mappings are reloaded" but had no callers, so the 7-day NVD-miss cache
survived every reseed. Software that previously missed stayed blacklisted
against the warm Function instance and never got a fresh resolution attempt
against the reloaded mappings.
Fold the negative-cache flush into ResetMappingsCache() so all 7 reload
sites (reseed seed/community, save/delete/bulk-auto-resolve custom mappings)
clear it — no call site can forget it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two hardening fixes surfaced by 48h backend log review:
- AzureMonitorMetricsReader.GetTotalAsync: skip the round-trip when the query range is shorter than 1 minute. The SignalR-quota watcher and the health check both anchor the start of the range to start-of-day, so a call landing on the 00:00 UTC boundary sent from==to and absorbed a daily 400 BadRequest.
- SignalRNotificationService: cap every imperative send (hub-context resolution + SendCoreAsync) at 5s via CancellationTokenSource. Without it a slow/unreachable SignalR endpoint blocks for the SDK HttpClient default of 100s, pinning the calling queue-worker/Task.Run thread. These pushes are observability nudges, never correctness, so fail fast and swallow as before.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "All (excl. deletions)" audit view filtered deletions client-side AFTER a
fixed 20-row backend page. During/after a cleanup sweep the table fills with
deletion_started/deletion_completed rows (written with the operator UPN, so the
existing System.Maintenance exclusion misses them); a 20-row page came back
all-deletions and the client stripped it to an empty list while real entries
sat on later pages.
Push the exclusion into the OData query (mirroring the System.Maintenance
clause) and back-fill the page: a server-side `Action ne ...` exclusion can make
a single Azure page return far fewer than pageSize matches, so follow the opaque
continuation until pageSize matches or the window is exhausted (accumulate whole
Azure pages to keep the token gap-free; cap at 20 round-trips). The flag is
bound into the continuation fingerprint scope (audit:tenant:nodel) so a token
cannot be replayed across views. Threaded through interface -> repo -> storage
with a defaulted param; global GA fan-out path covered too.
Frontend sends excludeDeletions=true when the filter is DEFAULT and refetches
page 1 on toggle.
Tests: 2172/2172 backend green + 4 new AuditLogDeletionExclusionTests.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Audit findings #4 (SessionSummary mapper x3) and #3 dual-write Phase 1.
- Collapse the two ~99%-identical SessionSummary mappers into one core mapper
taking the resolved sessionId (index vs primary differ only in that source).
- BuildSessionIndexEntity is now the single SessionsIndex projection manifest and
a SUPERSET of every field the merge call sites write, so a StartedAt-shift full
upsert can no longer drop a merged field (the recurring drift class, cf ab90423b).
- Fix two real bugs surfaced while grounding the design: ImeAgentVersion was never
written to the index although search/MCP push an OData filter against it (filter
matched nothing); UpdateSessionImeAgentVersionAsync now mirrors it. AdminMarkedAction
(dashboard list badge) was merged but missing from the projection -> now included.
- SyncSessionIndexAsync funnels the upsert-vs-merge + IndexRowKey-resolution decision
that was duplicated across the 3 StartedAt-shift branch sites.
- Correct the DeletionState doc-comment (it is primary-only, not index-surfaced).
The SessionsIndex is a full denormalized mirror of Sessions (different RowKey sort)
serving list/search/stats/raw directly; lean-drop would break /api/raw/sessions.
Backend build 0 warnings; full xUnit suite 2172/2172 (incl. new index-projection +
ImeAgentVersion-mirror tests). BuildSessionIndexEntity made internal for the test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
UpdateSessionStatusAsync force-write fallback (after ETag retries are
exhausted) mirrored most fields but silently dropped FailureSource and
AdminMarkedAction. Under ETag contention at the terminal transition
(e.g. WhiteGlove drain racing an admin Mark Succeeded/Failed), this
neutralized the admin-driven AdminAction trigger sent to agents and
lost the failure attribution. Mirror both fields analogous to the
existing FailureSnapshotJson mirror.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Eliminates the copy-paste queue scaffolding flagged as finding D2: identical
Managed-Identity/connection-string QueueClient bootstrap hand-rolled across 13
sites, plus a poll-loop/poison/safe-delete skeleton duplicated in 5 workers.
- QueueClientFactory: resolves the MI/connstring model once; consumed by all 5
producers, the offboarding enqueuer, the backup producer and the poison probe.
- QueuePollingWorkerBase: shared bootstrap + poll loop + poison-move + safe-delete
+ queue-create (ExecuteAsync/ProcessOneAsync sealed to prevent drift).
- QueuePollingWorker<TEnvelope>: deserialize -> validate -> handler flow with an
opt-in in-flight heartbeat.
- 3 vanilla workers (Index/Analyze/Vuln) collapse to ctor + HandleAsync.
- SessionDeletion/Offboarding use BatchSize/UseHeartbeat/ShouldPauseAsync +
Before/AfterPoisonMoveAsync hooks (kill-switch, CAS state transition, OpsEvent
enrichment, Failed-state-before-poison all preserved bit-for-bit).
- Backup derives from the non-generic base and overrides ProcessOneAsync (its
lease + CAS state machine is intentionally bespoke).
Producers get the factory only -- no shared base, their business logic is
genuinely divergent. Test seams unchanged; the static SessionDeletionWorker.
MaxDequeueCount reference moves to QueuePollingWorkerBase.DefaultMaxDequeueCount.
2168/2168 backend tests green, 0 warnings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The maintenance sweep graduated a stalled session to terminal Failed via the repo
(MaintenanceService Stage 2, SessionTimeoutHours=5h default) but wrote no event into
the session stream and never enqueued analyze. The analyze pipeline fires on stream
events, so a server-timed-out session analyzed to totalIssues:0 — unlike the agent
max-lifetime path, which emits a real enrollment_failed and is correctly analyzed.
- New server-authored event type Constants.EventTypes.SessionTimeout = "session_timeout"
(+ MCP termination_events catalog; drift test green).
- MaintenanceService Stage 2 now captures the UpdateSessionStatusAsync result and, only
on the real first transition (idempotent — the sweep never re-collects a Failed
session), emits a session_timeout event (new pure static BuildSessionTimeoutEvent:
Sequence = max(session events)+1 so it sorts last, Phase=Unknown, Source=System.
Maintenance, Severity=Error) and enqueues auto-analyze with ReasonEnrollmentFailed.
Best-effort try/catch — never blocks the timeout transition. Producer injected.
- RuleEngine: EvaluatePrecondition gains a pure event-type-presence gate (no-dataField
exists/not_exists) — additive (no rule used dataField-less preconditions), enables a
session-level suppression with no shared join field.
- New rule ANALYZE-ENRL-002 "Session Timed Out": fires on session_timeout, suppressed
when enrollment_complete OR enrollment_failed exists (5h<6h late-completion race +
no double-report with ANALYZE-ENRL-001). dist regenerated via combine.js.
- Web: session_timeout treated as terminal for stale-status refetch (the sweep sends
no SignalR delta).
Tests: +3 RuleEngine pure-type precondition, SessionTimeoutRuleTests (real shipped rule
fire/suppress), MaintenanceServiceSessionTimeoutEventTests (shape/sequence). Backend
2168 pass, web 434 pass, MCP 177 pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MCP server health probe previously sat inside PerformAllChecksAsync's
Task.WhenAll, so /health/detailed blocked on the MCP timeout before returning
any card, and a scaled-to-zero cold start was reported as an immediate warning.
Backend:
- Move the MCP probe out of the blocking all-checks batch onto its own
GET /api/health/mcp endpoint (AuthenticatedUser; visible to all, not GA-only).
- CheckMcpServerAsync now uses a wake-capable timeout (McpServerHealthTimeoutSeconds,
default 30s). Azure Container Apps holds the request during cold-start activation,
so the probe doubles as the wake signal: a successful wake -> healthy ("woke from
idle" when >2.5s), only a failed wake -> warning, a reachable-but-erroring server
(non-2xx) -> unhealthy.
- Register health/mcp in EndpointAccessPolicyCatalog (fail-closed).
Frontend:
- Fetch the MCP check on its own track, concurrently with and independent of the
fast checks. The card renders a blue "checking..." state immediately and updates
in place when the probe resolves; the other cards never wait on it.
- The MCP result folds into the overall banner/counts only once resolved, so a slow
cold start no longer flips the page to "Warnings detected".
Test: warning-path assertion updated for the new wording (10 HealthCheckService
tests green; 61 policy/endpoint tests green).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a "MCP Server" check to the System Health dashboard, visible to all
authenticated users (not GA-only). Probes the MCP server's public /health
endpoint (in front of the /mcp access guard, so auth-free), reports
reachability + latency, and surfaces the deployed MCP build version.
- MCP /health now returns { status, version } (SERVER_VERSION from package.json;
same value already in the MCP handshake, leaks nothing new).
- New Constants.McpServerBaseUrl (prod singleton, like ApiBaseUrl/AgentBlobBaseUrl),
overridable via McpServerUrl app setting.
- Cold-start handling: the Container App scales to zero and only serves /health
after boot, so timeout/connection failure is classified as a warning (with a
re-check hint) rather than a false "unhealthy" outage; the probe itself wakes
the scaler. A real non-2xx status is unhealthy.
- Server URL is never exposed in the card message or details.
- 4 service-level tests (healthy+version, healthy-no-version, non-2xx, timeout).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
get_platform_metrics: expose maxSessions (->&limit=), echo sessionLimit +
truncated. Backend already honored days+limit; the tool only sent days, so
results were silently capped at the newest 100 sessions.
get_geographic_sessions: backend (tenant+global) now matches structured
country/region/city against actual Geo* fields via FilterSessionsByFields
(case-insensitive, optional narrowing); 400 relaxed to require locationKey OR
country. MCP forwards structured filters and infers groupBy on the raw
locationKey path so country/region keys resolve (previously matched nothing).
locationKey path retained for the web UI.
get_geographic_metrics: blank Region/City for coarser groupings so a country
row is not mislabeled with an arbitrary sample city; keep Loc for map pins.
Tests: +GeographicDrilldownFilterTests (backend), +geo-platform-helpers (MCP).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GetAllBlockedDevices never parsed the query string and always called the
cross-tenant GetAllBlockedDevicesAsync(), so a supplied tenantId was
silently ignored (cross-tenant widening) and a malformed tenantId was
not rejected. Parse tenantId, scope to GetBlockedDevicesAsync() when a
valid GUID is given, and return 400 for a non-GUID value. Classification
extracted to a testable ParseTenantFilter helper with unit coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "raw" tools were oversold: query_raw_sessions returned curated
SessionSummary DTOs and its fields= ran through a hard-coded whitelist
(ProjectFields) that silently dropped ~20 real SessionsIndex columns
(OsEdition, ImeAgentVersion, GeoRegion/City/Loc, FailureSource,
CurrentPhaseDetail, LastEventAt, ResumedAt, StalledAt, DeletionState, ...),
and it accepted fewer filters than search_sessions -> a strict subset.
The raw endpoints are the deliberate "fall back to the real stored bytes"
escape hatch; the enriched views already live on search_sessions/get_session
and search_events/get_session_events. So both raw tools now return the
LITERAL stored rows: every column verbatim, PascalCase, incl.
PartitionKey/RowKey/Timestamp, with fields= a pure pass-through (narrows,
never silently drops a real column; always keeps PartitionKey/RowKey).
- New RawEntityProjection helper: ToDictionary(TableEntity) + Project(rows, fields)
- 3 new ISessionRepository methods returning IReadOnlyDictionary<string,object?>
(no Azure types leak into Shared); reuse BuildSearchScanFilter +
MatchesScanClientFilters for the predicate, emit the raw entity
- query_raw_sessions: delete ProjectFields; add filter parity (manufacturer,
model, enrollmentType, deviceName, osBuild, geoCountry, isPreProvisioned,
isHybridJoin) -- already covered by the continuation fingerprint
- query_raw_events: literal rows, drop ErrorCodeEnricher + EventFieldProjection
(DataJson raw string, Severity/Phase raw ints); severity/source/date filters
reimplemented on raw dicts preserving prior match semantics
- EventFieldProjection kept for the enriched get_session_events; doc-comment fixed
- MCP admin.ts: rewrite both tool descriptions, add the new query_raw_sessions
inputs, PascalCase pass-through fields= docs; prompts.ts hint updated
Note: contract change for these two tools (camelCase curated keys -> PascalCase
literal columns). deviceProperties join stays out of raw (owned by
search_sessions); query_table remains the generic literal escape hatch.
Tests: new RawEntityProjectionTests (8); full backend suite 2134 passed;
MCP tsc clean + vitest 149 passed/41 skipped; tools-raw.test.ts updated to
the PascalCase contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two contract gaps in the raw-event MCP readers, surfaced by an MCP-server review:
1. No fields= projection. query_raw_events/get_session_events returned full
EnrollmentEvent rows including the heavy Data dict (a single app_install_failed
event can be tens of KB), so lean counting/aggregation was impossible. Add a
shared EventFieldProjection helper (mirrors query_raw_sessions fields=) wired
into both endpoints; Data is dropped unless explicitly requested, and
ErrorCodeEnricher is skipped when Data will be projected away.
2. tenantId contract inconsistency. query_raw_events hard-rejected a sessionId
query without tenantId, while get_session_events auto-resolves it for GA. Now
query_raw_events resolves tenantId via FindSessionTenantIdAsync on the global
scope only (tenant scope keeps its JWT-bound tenantId). The resolved tenantId
is NOT echoed into nextLink, keeping the filterTenantId=null continuation
fingerprint stable across pages.
Adds EventFieldProjectionTests; updates MCP tool schemas/descriptions and the
docs row. Backend 2127/2127, MCP 145 pass, web tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Architecture rides the existing hardware_spec telemetry event (alongside
CPU/RAM/GPU), so backend storage needs zero changes - it lands in the generic
Props_hardware_spec JSON blob and flows through the dynamic prop.* filter.
- Agent V2: CollectCpuInfo() adds Architecture to the existing Win32_Processor
SELECT and emits cpuArchitecture via new MapProcessorArchitecture helper
(0->x86, 5->ARM, 6->ia64, 9->x64, 12->ARM64). 64-bit WMI host reports true
HW arch even under ARM64 emulation, unlike PROCESSOR_ARCHITECTURE.
- Backend: trailing-* prefix wildcard in MatchesDevicePropertyFilter so
cpuArchitecture=ARM* matches ARM + ARM64 in one query (generic, exact/numeric
behavior unchanged).
- Web: Architecture DetailRow in DeviceDetailsCard hardware section.
- MCP: resource-catalog entry + wildcard hint in search_sessions description.
- Tests: ProcessorArchitectureMappingTests (9), DevicePropertyFilterWildcardTests (15).
V2-only; old V1 agents and pre-existing sessions simply lack the key (no backfill).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After the MCP search tools moved to the paginated /api/raw/events fan-out
(prev commit), /api/raw/events/search had zero callers anywhere in the repo
(no MCP tool, no web). Remove the now-dead surface end to end:
- Delete SearchEventsFunction (raw/events/search + global/raw/events/search).
- Drop both routes from EndpointAccessPolicyCatalog and the no-store cache
middleware allowlist (+ its test InlineData). EndpointPolicyCatalog
completeness test stays green (route and entry removed together).
- Remove the now-orphaned repo method SearchEventsByTypesAsync (its only
caller was the deleted endpoint) from ISessionRepository + TableSessionRepository
delegation + TableStorageService impl.
- Remove the non-paginated SearchSessionsByEventAsync: its only runtime caller
was SearchEventsByTypesAsync, and it is strictly redundant with the resumable
SearchSessionsByEventPageAsync (used by /api/raw/events + search/sessions-by-event).
Reword the surviving paged method's xmldoc/comment that referenced it.
No capability lost: multi-type cross-session event search still runs via the
MCP fan-out over the paginated single-type endpoint; single-type primitives
remain exposed via query_raw_events + search_sessions_by_event. Full solution
builds clean (0 errors); catalog-completeness + cache-middleware tests 77/77.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds a second, higher threshold to the runaway-session detector that
auto-blocks or auto-kills the device (admin-selectable mode), so the
existing 2000-event warn keeps observing while the harder action runs
unattended at e.g. 2500 events. Both tiers stay independently
idempotent per session and the maintenance batch picks them up in a
single query pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review (Low/Medium): even after the safe-state-first sunset sequence
landed in 4667033f, a narrow race remained — between a partial GC failure
(global row kept for retry, by design) and the next successful retry cycle,
a surviving per-tenant RuleState{Enabled=true} would resurrect the rule via
the override in AnalyzeRuleService.GetAllRulesForTenantAsync line 142
(rule.Enabled = state.Enabled overwrites the safe-state Enabled=false).
For the ANALYZE-CORR-001 / ANALYZE-APP-009 case the impact is small (only
two known orphan RuleState rows on a single partition, GC is essentially
always one-shot), but as a generic mechanism the window can be material
for future sunsets with many opt-in tenants where cross-partition GC is
more failure-prone (rate limits, transients).
Close the window with a runtime filter in GetAllRulesForTenantAsync:
built-in / community rules whose code definition is no longer in
BuiltInAnalyzeRules.GetAll() are excluded from the merged result regardless
of any surviving tenant override. The global DB row is preserved on
purpose so the next seed cycle still sees the rule in the sunset diff
and retries the GC + tombstone. Semantic becomes immediate instead of
eventual: "rule.json removed from source + deployed" = "rule cannot fire
this request".
Implementation:
- LiveCatalogIds: lazy-cached HashSet<string> on the service instance.
BuiltInAnalyzeRules.GetAll() re-deserialises the embedded JSON on every
call, so caching the projected set materially helps the hot read path.
Source of truth is the embedded resource (immutable per deployed
binary), so the cache never needs invalidation.
- Filter: `(rule.IsBuiltIn || rule.IsCommunity) && !liveCatalog.Contains(...)`
Custom tenant rules (IsBuiltIn=false, IsCommunity=false) are
deliberately exempt — they live entirely on the tenant partition and
aren't part of the built-in catalog.
Tests (+2):
- runtime_filter_hides_sunset_rule_even_with_enabled_tenant_override:
pins the exact scenario Codex flagged. GC fails (rule stays in DB),
tenant has RuleState{Enabled=true}, but the merged result still
excludes the sunset rule.
- runtime_filter_does_not_hide_custom_tenant_rules: defense — custom
rules whose RowKey doesn't match any built-in must survive.
Backend 2084/2084.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When an ExcessiveSessionEvents (or other session-bound) ops alert fires,
the operator only has a sessionId in hand. Looking up tenant + serial to
fill the manual Block form was a friction point during incident response.
- Backend: POST /api/devices/block now accepts optional blockedSessionId
and forwards it to the existing BlockedDeviceService session-aware path;
audit log + warn log carry the session id.
- Web (Device Block form): new "Session ID" shortcut input + Resolve
button fills Tenant + Serial via GET /api/sessions/{id} (GA cross-tenant
lookup already supported); resolved-device confirmation pill; URL
deep-linking via ?sessionId=, ?action=, ?reason= for auto-fill on mount;
blockedSessionId only forwarded when the resolved device still matches
Serial+Tenant so manual edits don't bind to a stale session.
- Web (Ops Events modal): when the event details JSON carries a sessionId,
surface "Block this device" / "Kill this device" deep-link buttons with
an auto-generated reason scoped to the event type.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review (Medium): even with the GC-first ordering from b855b2b9, a
race remained — after the orphan-GC deleted tenant RuleState rows but
BEFORE the global tombstone delete completed, a transient Azure failure
on the tombstone would leave the global rule alive while tenant overrides
were already gone. For a future default-ENABLED sunset rule this would
silently flip RuleState{Enabled=false} (explicit tenant opt-out) into
"now using the global default (enabled)" — exactly the regression class
the orphan-GC was meant to prevent.
Fix: a strict three-step sequence with mutual dependencies. Extracted
into a single ProcessSunsetRuleAsync helper so EnsureBuiltInRulesSeededAsync,
ReseedBuiltInRulesAsync, and ReseedFromGitHubFunction.ReseedAnalyzeAsync
all use the same primitive:
1. Safe-state: write the global rule with Enabled=false +
MarkSessionAsFailedDefault=false. Neutralises the default BEFORE
touching tenant state. If a default-enabled sunset rule's
RuleState{Enabled=false} (opt-out) survives every later step
failing, the override now matches the global default — same
effective result, no silent flip.
2. Orphan GC across tenants. Only attempted if safe-state succeeded.
3. Tombstone (DeleteAnalyzeRuleAsync global). Only attempted if GC
reported zero failures.
Returns a SunsetOutcome enum with three explicit failure values
(SkippedOnSafeStateFailure / SkippedOnGcFailure / SkippedOnGlobalDeleteFailure)
so callers and tests can pin which step bailed.
Additionally fixes the same-instance retry gap Codex called out:
EnsureBuiltInRulesSeededAsync only sets _seeded=true when EVERY sunset
rule reached Completed. Any partial outcome leaves the seed flag false
so the next GetAllRulesForTenantAsync call retries without waiting for
a process restart. The Function-App can recycle on its own schedule
(hours on a warm instance) and we don't want sunset cleanup to wait
that long.
Behaviour for the original CORR-001/APP-009 case: both shipped
Enabled=false globally and the tenant overrides were Enabled=true
(opt-IN). Safe-state is a no-op (already Enabled=false), GC removes
the opt-ins, tombstone drops the global row. Same end state as before,
just retry-safe across transient Azure errors.
Tests (10 new sunset cases on top of the previous 6):
- Full-path happy case asserts safe-state PRECEDES GC PRECEDES
tombstone via a shared CallOrder log.
- Safe-state failure path: NO GC + NO tombstone (tenant state preserved).
- Per-step failure paths for SkippedOnGcFailure / SkippedOnGlobalDeleteFailure.
- _seeded flag stays false on partial sunset, flips to true only on
a clean cycle; the "next call retries" scenario is exercised by
reconfiguring the mock between two GetAllRulesForTenantAsync calls.
Backend 2082/2082.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review of the sunset orphan-GC introduced in dff5ba65 surfaced three
issues; all three are valid and now fixed:
(Medium) The previous sequence was "delete global rule first, then GC
per-tenant RuleStates". If the GC failed partially (per-row error) or
entirely (cross-partition enumeration error), the global rule was already
gone — the sunset diff went empty on the next seed cycle and the surviving
orphan RuleStates would linger indefinitely. Worst case for a future
default-ENABLED sunset rule: tenants that had explicitly disabled it would
lose the disable, silently re-enabling something they didn't want.
Three coordinated fixes:
1. Storage methods made idempotent: DeleteAnalyzeRuleAsync and
DeleteRuleStateAsync now treat Azure 404 (row already gone) as success
instead of returning false. The 404-on-individual-row case inside the
cross-tenant GC enumeration is also counted as success — a row that
was raced-deleted between the query and our delete is a clean state,
not a failure.
2. Cross-tenant GC return shape changed from `int` to `(int deleted,
int failed)`:
- `deleted` counts fresh deletes AND 404s (idempotent retries)
- `failed > 0` signals at least one per-row delete threw a non-404
- `failed == -1` signals the cross-partition enumeration itself
failed — we genuinely don't know how many orphans are still out
there
Callers must check `failed != 0` before proceeding to the global
delete. The integer-only return shape couldn't distinguish "GC found
nothing" from "GC failed silently".
3. Caller sequence inverted to GC-FIRST in all three sunset code paths:
- AnalyzeRuleService.EnsureBuiltInRulesSeededAsync (on-startup seed)
- AnalyzeRuleService.ReseedBuiltInRulesAsync (explicit reseed)
- ReseedFromGitHubFunction.ReseedAnalyzeAsync (admin UI button)
GC is attempted first. The global rule is deleted ONLY when GC returns
`failed == 0`. On partial GC failure the rule stays in the global
catalog so the next seed/reseed cycle sees it in the sunset diff
again and re-attempts the cleanup. End state is eventually consistent
even across transient Azure errors.
(Low) The GitHub-reseed HTTP response shape was extended from
`{ deleted, written }` to `{ deleted, written, orphanStatesGcd,
sunsetSkipped }` so operators can see the cleanup count + any rules
that got skipped for the retry loop.
Tests:
- 3 new sunset-mechanism tests covering partial GC failure, enumeration
failure (-1 sentinel), and the still-living-rule no-touch invariant.
- Existing ReseedBuiltIn_returns_orphanGc_count_in_tuple updated to the
new helper shape.
- ReseedBuiltIn_skips_global_delete_on_partial_GC_failure pins the
retry-safe semantics in the explicit reseed path.
Coverage: backend 2076/2076 (+3 sunset edge-case tests).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Session 080edee9 follow-up — the customer screenshots showed ANALYZE-CORR-001
("Slow Network") and ANALYZE-APP-009 ("App Download Stalled") still firing
on a clean re-analyze, despite my previous fix shipping them as
enabled:false in the JSON. Investigation via direct DB query:
- AnalyzeRules/global rows for both: Enabled=false ✓ (version-bump landed)
- RuleStates/{tenant} rows for both: Enabled=true ✗ (orphan overrides
from the user toggling them in the portal UI on 2026-03-09)
[AnalyzeRuleService.cs:60](src/Backend/AutopilotMonitor.Functions/Services/AnalyzeRuleService.cs#L60)
overlays per-tenant RuleState on top of the global Enabled, so the orphans
kept resurrecting the rules. No reseed path was cleaning them up.
The right semantic is "sunset = remove from catalog", not "ship as
enabled:false and hope nobody overrides". Three coordinated changes:
1. Both sunset rules deleted from the rules/analyze/ source tree (they
never had a working condition shape — CORR-001 didn't actually check
download speeds, APP-009 matched the V2 termination handler's
post-mortem-synthetic download_progress:failed event).
2. New IRuleRepository.DeleteRuleStatesForRuleIdAcrossTenantsAsync
cross-partition cleanup primitive — scans RuleStates by RowKey, deletes
matching rows across every tenant partition.
3. Both reseed paths now do orphan-state GC:
- AnalyzeRuleService.EnsureBuiltInRulesSeededAsync (on-startup):
detects rules that exist in DB as built-in but aren't in
BuiltInAnalyzeRules.GetAll() anymore → deletes the global row +
fans out the per-tenant RuleState GC.
- AnalyzeRuleService.ReseedBuiltInRulesAsync (delete+reimport): same
diff-based detection on the pre-delete snapshot.
- ReseedFromGitHubFunction.ReseedAnalyzeAsync (admin "Reseed from
GitHub" button — the path the user actually clicks): mirrored
identical logic so every reseed flavor leaves the same DB invariant.
4. ReseedBuiltInRulesAsync return shape extended to
(deleted, written, orphanStatesGcd) so the admin UI can surface the
cleanup count to operators.
Coverage:
- RuleEngineDetectionFailureTests: the two version-bump assertions for
CORR-001/APP-009 are replaced with `DoesNotContain` against the
built-in catalog (sunset semantic).
- AnalyzeRuleServiceSunsetTests (new, 3 tests): pins EnsureSeed orphan-GC
invariant, asserts live-rule states are NOT touched, and pins the
ReseedBuiltIn (deleted, written, orphanStatesGcd) tuple shape.
After deploy + a single GetAllRulesForTenantAsync call (any analyze run
triggers it), the orphan RuleStates for sunset rules are automatically
cleaned. The user's tenant 5ca2b350-… will see CORR-001 + APP-009
disappear from the Rules UI on the next analyze cycle.
Backend 2073/2073.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Session 080edee9 follow-up — the analyze cards the user saw on re-analysis
were neither accurate (Slow Network / App Download Stalled both fired as
false positives) nor specific (Enrollment Failed never named the offending
app; ANALYZE-APP-013 never fired). Five coordinated changes, all
version-bumped so EnsureBuiltInRulesSeededAsync re-imports them on the next
backend boot — the title/severity/version-check filter does not pick up
condition-only diffs:
ANALYZE-APP-013 v1.1.0 — fire on legacy + new agent shape:
Previous rule keyed on `failureType == esp_apps_detection_failure`,
which is only emitted by agents that shipped AFTER 4dd050e0. Historical
sessions (and any agent on the old build) won't match.
Replace with a precondition + condition pair that works for BOTH:
* precondition: esp_failure_settle_started.errorCode == 0x87d1041c
(V2 ProvisioningStatusTracker emits this on every Apps subcategory
failure with that HRESULT, regardless of agent version)
* required: app_install_failed.detectionResult == NotDetected
(ImeLogTracker sets this from the IME's DetectionActionHandler line
across all V2 agents)
* optional: esp_failure_settle_started.errorCode (for {{errorCode}}
interpolation; appName comes via auto-field from the required match)
ANALYZE-ENRL-001 v1.1.0 — restore appName + cross-link to APP-013:
My previous trim dropped the failed_app condition on the theory that the
auto-field path would carry it. That was wrong — auto-fields come from
the FIRST matched condition's evidence, which here is `enrollment_failed`
(no appName). Add back the optional failed_app condition with
dataField=appName so byField resolves {{appName}}. Explanation now leads
with the failed app, lists reason/subcategory/errorCode, and cross-links
to the actionable ANALYZE-APP-013 instead of "see other rule results".
ANALYZE-CORR-001 v1.1.0 ("Slow Network") — version bump for DB sync:
Was already enabled:false in source but the DB still had it enabled (the
seed mechanism only resyncs on title/severity/trigger/version diff, so
the source `enabled` flip never propagated). Bump version → next deploy
resyncs. Also adds a suppressByEvent on the app_failed condition so a
later app_install_completed for the same appId suppresses the match (a
retry that succeeded shouldn't count as failure).
ANALYZE-APP-009 v1.1.0 ("App Download Stalled") — same version bump for
DB sync + documents why a status-aware suppression isn't expressible yet:
The V2 EnrollmentTerminationHandler synthesizes
`download_progress: failed` events for promoted-stuck apps AFTER the
real `download_progress: 100%` event already landed. The engine's
suppressByEvent can't filter on a third event's status field, so
status-aware suppression has to wait for either an engine extension or
a dedicated `download_stalled` event type.
RuleEngineDetectionFailureTests — 6 new tests:
* APP-013 fires on the historical 080edee9 event shape
* APP-013 does NOT fire for non-Apps subcategory HRESULTs
* APP-013 does NOT fire without esp_failure_settle_started (precondition)
* ENRL-001 MatchedConditions carries appName + errorCode + subcategory
+ reason from three distinct event types (interpolator material)
* CORR-001 + APP-009 shipped as disabled
Coverage: 2064/2064 backend, 413/413 web.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Walks each EnrollmentEvent.Data for known error-code keys (errorCode,
exitCode, hresult, hresultFromWin32, failureCode, code, lastError) and
injects a sibling *Info entry { description, confidence, source } via
the shared ErrorCodeCatalog. Applied in GetSessionEvents (legacy +
paginated), SearchEvents (used by search_events_semantic +
deep_search_events MCP tools) and QueryRawEvents (both session-scoped
and cross-session paths).
Idempotent + read-only on storage — alte gespeicherte Events erhalten
die Anreicherung on-the-fly. MCP-Konsumenten (Claude/Copilot) bekommen
ab jetzt automatisch die kuratierten Descriptions mit Confidence-Level,
nicht nur den rohen HRESULT. 14 enricher tests; 2058/2058 backend tests
green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Lifts the Windows/MSI/Intune HRESULT catalog out of the web app into
src/Shared/AutopilotMonitor.Shared/Resources/error-codes.json so backend
and web load from the same authoritative file. Web syncs the JSON at
prebuild time. Adds ErrorCodeCatalog C# service with normalised-key
lookup (hex case, signed-decimal HRESULT, leading-zero padding) and a
parity test on both sides. No behavior change yet — this is groundwork
for backend read-time enrichment so MCP/API responses can carry
errorCodeInfo (description + confidence + source). 11 C# + 14 web tests
green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Normal (non-Global-Admin) MCP users no longer see any Global-Admin / cross-tenant
hints — reducing both confusion and attack surface / curiosity. The per-request
stateless server already runs inside the access-guard's runWithCaller context, so
the caller role is known at server-creation time.
- index.ts: createMcpServer(ga) reads isGlobalAdmin() and tailors instructions
(non-GA gets a tenant-scoped scope line, no cross-tenant mention).
- tools.ts / sessions.ts / search.ts / admin.ts: thread `ga` into every registrar.
* GA-only tools (list_tenants, list_tables, query_table, query_backend_logs,
get_platform_metrics, get_ops_events, list_session_reports, list_blocked_devices,
get_api_usage) are registered only when ga — absent from tools/list for non-GA.
* Dual-role tools: GA/cross-tenant wording in descriptions and tenantId param
text is gated behind `ga ? ... : ...`; non-GA sees clean tenant-scoped text.
- prompts.ts: compare-agent-versions (uses GA-only get_platform_metrics) hidden for
non-GA; cve-exposure-audit cross-tenant wording scrubbed for non-GA.
- error-handler.ts: 403 message no longer names "Global Admin".
- backend McpUserFunction (/api/auth/mcp): omit isGlobalAdmin from the response when
false, so a normal user gets no hint that a GA tier exists. The MCP guard reads
`=== true` (absent → non-GA), so behavior is unchanged.
Verified: GA catalog 29 tools / non-GA 20 (9 GA-only hidden), zero GA/cross-tenant
wording in any non-GA description, prompts split 3/2. tsc clean; backend targeted
suite 135 green; mcp unit 52 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Second batch of pre-release MCP tool audit fixes:
- mcp: get_session_summary phase labels were desynced from the backend
EnrollmentPhase enum — phases 3-7 were mislabeled (e.g. 7 showed
"Pre-Provisioning" but is "Complete"; 4 showed "Device ESP" but is
"Account Setup"). Replaced with the canonical taxonomy (matches the web
phaseConstants.ts), enrollment-type-aware (phase 3 = Apps (Device) on V1,
App Installation on V2), plus -1=Unknown / 99=Failed.
- backend: get_usage_metrics reported PlatformStats.lastFullCompute /
lastUpdated as 0001-01-01 even though the persisted row was fresh -
UsageMetricsService projected a new PlatformStats and dropped the two
timestamp fields (defaulted to DateTime.MinValue). Now carried through.
- backend: query_table (TableQueryFunction) now returns 400 with the
actionable Azure message for an invalid OData filter / bad query input
(RequestFailedException 400) instead of a generic 500.
Tests: backend targeted suite 31 green; mcp tsc clean + unit suite 52 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Oliver Kieselbach