New anonymous POST ops/alert-webhook receives Common Alert Schema
notifications from an Action Group webhook receiver and re-emits them
as ops events (category Platform, type AzureMonitorAlert), so Azure
Monitor alerts flow through the existing OpsEventService ->
OpsAlertDispatchService fan-out (Telegram/Teams/Slack + portal
ops-events list) like every internally raised operational event. The
delivery path is independent of the portal: a SWA outage cannot
suppress the notification; the action group's e-mail receiver stays as
the backend-independent fallback.
Security: action-group webhooks cannot send custom headers, so the
shared secret travels as ?secret= and is compared in constant time
(FixedTimeEquals) against the OpsAlertWebhookSecret app setting -
fail-closed while unset. 64 KB payload cap, defensive essentials-only
parsing (only alertRule is mandatory), severity mapped Sev0->Critical,
Sev1->Error, Sev2->Warning, Sev3/4->Info with unknown values falling
back to Info so schema drift can never escalate.
Dual-registered per convention: EndpointAccessPolicyCatalog entry
(PublicAnonymous), OPS_EVENT_TYPES in OpsAlertRulesSection (rule
wiring) and Platform category in OpsEventsSection (badge + filter).
Backend suite 3511 green (incl. new helper tests + catalog
completeness), web tsc + vitest 651 green.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Navbar + GlobalSidebar render the full navigation on every page, so the
App Router's default viewport prefetch fired one ?_rsc= request per
visible link against the SWA runtime (~10.4k requests / 14 days). That
origin serves document, chunks and RSC payloads over HTTP/1.1 (6
connections per origin); when the runtime queued the prefetches (p95
896ms, tail up to 234s), they held every socket and the portal froze
until the queue drained. Measured on the 2026-07-27 17:25 incident:
/admin/presence?_rsc= at 151s while the backend API answered in <2.2s.
All 16 nav links now set prefetch={false}. Every target route is a
client component behind ProtectedRoute that fetches its own data after
mount, so the prefetched RSC payload carried no benefit.
New OKF doc docs/web/portal-navigation-prefetch.md records the full
evidence chain and where browser telemetry lives (separate App Insights
resource, not reachable via MCP query_backend_logs).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A GA with a persisted tenant override opening /usage-metrics (or /sla) saw
their OWN tenant's stats until a manual refresh. useGlobalAdminScope seeded
the persisted selection in a post-mount effect, so the page effect first saw
effectiveTenantId = own tenant and fired an own-tenant fetch immediately
followed by the override fetch; with raw authenticatedFetch (no latest-wins
guard, unlike useAuthenticatedFetch since 82934a37) the last-RESOLVED
response won.
- useGlobalAdminScope: seed the GA selection render-time (same pattern as
the delegated seed) so consumers never observe a transient own-tenant
effectiveTenantId - the wrong fetch no longer starts at all.
- usage-metrics/sla pages: latest-wins sequence guard in fetchMetrics so a
superseded request can neither write data/notifications nor clear the
newer request's loading state (covers rapid selector switches).
Verified: tsc 0 errors, vitest 651/651.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three defects found by measuring production after the 1.5.1012 deploy:
- AppMetricsProjection dropped TerminalState, so the PR0 skip
classification never reached metrics/app (totalSkipped stuck at 0).
Column added; equivalence fixture now populates TerminalState/AppId/
EspBlocking/AppIdCollision and asserts them (proven: test fails
without the projection fix).
- BuildTimeAttributionAggregates excluded breakdowns on ANY quality
flag; 903/1000 carry BlockingSetUnknown, starving fleet stats
(Clean=12 vs 331 excluded on the largest tenant). New
TimeAttributionFlagQuality.DurationCriticalFlags mask - only
ClockSkew/PartialObservation/WGAnchors exclude; blocking-set flags
keep their measured spans (positive-evidence semantics unchanged).
- HasMeasuredDuration gets an upper plausibility bound (6h = default
AgentMaxLifetimeMinutes): 490 rows carried back-stamped completions
(up to 117 days; 60175s app rows in a 2697s session) that dominated
slowestApps with six-figure averages. Read-side, heals old + new rows.
Backend 3484/3484 green; solution build clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A tab still running the previous deployment's bundle requests lazily-loaded
chunks (next/dynamic charts, route chunks) whose content-hashed filenames no
longer exist after the SWA swap - the user is stuck on "Loading chart..." or a
dead navigation until a manual refresh. Standard remedy: reload once so the
fresh document references the new bundle.
- utils/chunkReloadRecovery.ts: chunk-error detection (webpack ChunkLoadError +
Chromium/Firefox/Safari dynamic-import variants) and a sessionStorage guard -
at most one auto-reload per tab per 5 minutes; without working storage no
auto-reload at all, so a reload loop is structurally impossible. Emits a
ChunkReloadRecovery App Insights event before reloading.
- components/ChunkReloadRecovery.tsx (root layout): catches rejected dynamic
imports via unhandledrejection and failed /_next/ script/link tags via a
capture-phase error listener (third-party scripts never trigger a reload).
- app/error.tsx: a chunk error reaching the boundary reloads once instead of
showing the error card; inside the guard window it falls through to the card.
Verified: tsc clean, vitest 651/651 (8 new tests).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GA override on gather-rules called GET /api/global/config/{tenantId}, a route
that does not exist in the backend; the swallowed 404 left the unrestricted
badge showing the GA's HOME tenant's mode for every foreign tenant. Both paths
now read the member-readable config/{tenantId}/feature-flags (MemberRead has a
global-scope bypass and the route is RouteParam-scoped), with a reset-on-switch
so a failed fetch never carries the previous tenant's mode over. The dead
globalConfig.tenant entry is removed from the URL register.
useAuthenticatedFetch now enforces latest-wins: the scope hook seeds a
persisted foreign tenant only after mount, so an own-tenant fetch races the
override fetch and the last-RESOLVED response used to win, leaving the rules
list on the home tenant while the selector showed the override. Stale requests
no longer write data/error/loading; the spinner is owned by the last-started
non-silent request.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customers who additionally deploy Microsoft 365 Apps as their own Win32 app saw two
rows in Install Progress (the IME view and the Office C2R lifecycle) with nothing
telling them apart, which read as a duplicate.
- New lib/installProgress.ts: buildInstallItems() extracted from the component so the
event-folding rules are unit-testable; component is now render-only.
- InstallItem gains `source` (ime | office-c2r | realmjoin) and a source-scoped `key`.
Rows were keyed by app name alone, so a Win32 app named exactly "Microsoft 365 Apps"
(= OfficeInstallDetector.AppName) collided with the C2R row and overwrote its state.
- Pill only for non-default sources: "Click-to-Run" / "RealmJoin", neutral grey so it
does not compete with the state badge, with a tooltip explaining what the row measures.
Plain Intune app rows stay unlabelled.
- Drops the "RJ: " name prefix, which broke sorting and truncation.
Verified: tsc clean, vitest 643/643 (16 new).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Codex review (P2): the ESP-config refresh triggers each run on their own
Task.Run (as do CollectAll's startup sweep and CollectAtEnrollmentStart),
so two concurrent collects could interleave read-before-emit - the LATER
emission then carried the OLDER (smaller) registry snapshot. The backend
uses the latest list-bearing esp_config_detected emission as positive
blocking evidence, so an out-of-order emission would silently shrink it.
CollectEspConfiguration now runs its read->emit sequence inside a
dedicated lock covering ALL callers; a later emission always carries an
equal-or-newer snapshot. The race predates the Q2 refresh triggers (they
only raised its frequency).
Deterministic concurrency test: 12 parallel refreshes against a probe
whose list grows per read - the emission sequence must be exactly 1..12.
Verified: Agent V2 Core 2059/2059.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All six review findings verified against code and real; five fixed here,
the agent-side race lands in the companion agent commit.
[P1] WhiteGlove pause counted as time: ClampToWindows clipped intervals
against the HULL of both WG windows, but the excluded pause lies inside
the hull - a reboot gap or app interval straddling the pause counted the
entire pause (a ~2-day "reboot" in a 30-minute enrollment was possible).
Seconds are now the per-window intersection sum (chronological endpoints
kept); an interval wholly inside the pause makes no claim and no longer
raises ClockSkewDropped. What-if savings are additionally capped at the
session's WallClockSeconds - endpoint deltas can still straddle the
pause, and no removal can save more than the enrollment took.
[P1] Late-arriving events never updated breakdowns: the terminal-seam
compute is one-shot and the sweep only healed missing/version-stale rows.
SessionTimeBreakdown now stores EventCountAtCompute (0 = pre-fix row,
recomputed once); the sweep recomputes on EventCount drift and then also
re-runs ResolveEspBlockingForSessionAsync (newly exposed on
IMetricsRepository) so late esp_config_detected re-emits and app batches
reach both the breakdown and the blocking evidence.
[P2] Stale aggregate buckets: the sweeps now reconcile every partition
they wrote - buckets that existed but were not regenerated (sessions
deleted, class left the window) are deleted via new targeted deletes;
rolling30 rows additionally get a 48h ComputedAt freshness filter at
read time, so a fully quiet tenant stops serving year-old numbers as
"last 30 days".
[P2] Date windows included an extra calendar day: both inclusive range
ends meant days=30 returned 31 keys and days=1 summed yesterday AND
today. InclusiveWindowStart(today, days) = today-(days-1) in both
response builders; the sweep's rolling30 source is filtered to the same
30 calendar days (its query window stays one day wider for backfill).
[P3] Device-history banner counted later sessions as "previous
enrollments" when viewing an older session: selectPriorSessions
(deviceHistoryPrior.ts) anchors on the session's own chain entry (live
sessions: new sessionStartedAt prop), sessionId tie-break mirroring the
server chain sort.
Verified: solution build clean, Backend 3480/3480 (+6: three WG-pause
golden fixtures, window-start facts, RK builder/delete consistency,
EventCountAtCompute roundtrip), Web tsc + vitest 633/633 (+6).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Q1 (AppInstallTiming): a terminal->active transition (IME retry path,
e.g. Error->Downloading or Skipped->Downloading) re-arms CompletedAtUtc
so the final terminal payload covers first-start -> last-terminal instead
of freezing at the first terminal state. Plus a negative-duration guard
in AppInstallTiming.DurationSeconds (inverted endpoints -> null, field
omitted) - observed in production as durationSeconds=-2342 on a
Skipped-then-retried user-scope app (session 66f1c5e1).
Q2 (esp_config_detected): the ESP blocking lists are read too early -
device lists grow progressively, user-scope S-<SID> lists appear only
after sign-in (88 of 89 prod sessions had an empty user list at last
emission). DeviceInfoHost now re-collects the ESP configuration as
phase-driven one-shots: apps sub-phase declaration (AppsDevice/AppsUser
phase_transition = first app activity) and AccountSetup detection.
The StartupEventGate keeps unchanged payloads silent (fingerprint covers
list contents), so only grown lists actually re-emit. F1 EspBlocking
resolution (latest list-bearing emission) benefits without backend change.
Q5 (HardwareInfo): WMI values (serial, manufacturer, model) trimmed at
the source via NormalizeWmiValue; whitespace-only collapses to the
"Unknown" sentinel the retry logic already treats as unresolved.
Attach-window investigation (11a3b67a): NO agent fix - hypothesis
disproved against two real agent logs (diag bundles db5ce007/2.0.1337,
4a9e9273/2.0.1379). "completed without started" first apps were already
installed (WinGet detection-only, zero install lines in AppWorkload.log);
pre-attach replay provably works (pattern-match log). PR0's
HasMeasuredDuration already handles the class server-side.
Duplicate app_install_completed (66f1c5e1): mechanism narrowed to an
in-process re-read of exactly one AppWorkload rotation window (<24h-old
lines, historic suppression not applicable) - documented as follow-up,
no fix without a bundle-bearing repro per lessons.md.
Verified: solution build clean, Agent V2 Core 2058/2058 (14 new tests),
DecisionCore 493/493. No ConfigVersion bump, no new event types,
decision rail untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Analyze rules only (gather stats stay per-batch-deduped until Q6 is
fixed - never silently included). Per (tenant, rule) the radar compares
the trailing 7-day hit rate (FireCount/SessionsEvaluated from the daily
RuleStats rows) against the prior 28-day baseline and fires only when
the window has >=5 hit sessions and >=20 evaluated sessions, the rate
is >=2x baseline AND the Wilson 95% intervals are disjoint
(MetricsMath.WilsonInterval/RateIncreaseSeparated, hand-computed test
vectors). Suppressed for deleted rules, rules inside the CreatedAt
grace period (<35d), rules edited inside the window, empty baselines,
and the RuleRegressionRadarDisabled app-setting kill switch.
Alert episodes live as ruleregression|{ruleId} rows in the notification
tracker (third keyspace): AddEntity-409 dedup = one bell + one
RuleFrequencyRegression ops event (category Tenant, Warning, full
numbers, dual-registered in the web OPS_EVENT_TYPES) per episode;
numbers refresh while active without moving FirstNotifiedAt; episodes
re-arm under 1.5x baseline, on fire-stop, or via the tracker's 30d
retention prune (spec-intended reminder). On-fire dimension correlation
(osBuild/model/agentVersion/imeVersion; hit set from a RuleResults
range scan intersected with the window sessions) reports both shares
with the "correlated, not necessarily causal" wording contract or
honestly "no clear dimension concentration". Radar runs in the daily
maintenance pass after the rule-stats aggregation, anchored on
yesterday (whole days only).
Surfaces: regressions[] in the rule-stats response (MCP get_rule_stats
inherits it), tenant bell deep-linking to the rule card, and the
analyze-rules page gains a densified 30-day fire sparkline plus a
regression badge while an episode is active. OKF doc
backend/rule-regression-radar.md + index/log.
Backend 3474/3474 (27 new: Wilson vectors, every suppression branch,
window boundary dates, re-arm matrix, dimension gates, tracker
round-trips incl. tri-states, keyspace dedup/prune, wording pins),
web tsc + vitest 627/627, MCP tsc + 455 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three read routes over the PR4 data: metrics/device-history
(MemberRead + tenant query param - one route serves the web banner AND
MCP, GA cross-tenant via ?tenantId= with a sessionId-based owner
fallback; history:null is a normal outcome), metrics/device-journeys
and global/metrics/device-journeys (days selector honored because FTR
counts are additive - window rate = ratio of summed daily rows; the
tenant-less global variant serves the "global" partition and discloses
repeatDevices:null). Attempt numbers are server-computed
(DeviceJourneyCalculator.ComputeAttemptNumber: chain position within
the journey; live sessions via a virtual non-successful attempt so the
redeploy and 30d-gap rules apply) - no consumer re-derives journey
semantics. Repeat-devices violator list: last journey >=2 attempts with
newest terminal ref in-window, attempts desc then recency, cap 10,
failure reason from the newest failed attempt (bounded point-reads).
Web: DeviceHistoryBanner on session detail ("Attempt N for this device
- View history", expandable terminal chain, durations strictly the
authoritative durationSeconds, WG/DevPrep/admin-marked badges) and the
Fleet Health "First-time-right" section (rate gated at >=20 completed
journeys with disclosed n, junk-serial exclusion note, Monday-UTC
weekly trend, attempt histogram, repeat devices) following the 7/30/90
selector. MCP: get_device_history (serial mode / fleet FTR mode,
role-aware routing, all roles - GA_FULL privilege snapshot extended
deliberately). Docs: OKF backend/device-journeys.md + index/log.
Backend 3447/3447 (14 new: attempt-number vectors incl. gap boundaries
and live sessions, additive sums, days clamping, repeat-device
gate/sort/cap), web tsc + vitest 627/627, MCP tsc + 455 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Device-key normalization (trim+casefold, junk-serial exclusion incl. the
agent's "Unknown" WMI sentinel), pure journey grouping (terminal set
Succeeded/Failed/Incomplete; journey ends at first success; >30d gap from
the previous attempt's CompletedAt starts a new journey; WG Pending stays
an open session, never an attempt), and the DeviceHistories table
(PK tenant, RK encoded normalized serial): terminal-only session-ref
chains capped at 20 with derived journey counts, written inline at the
session-terminal seam (all three terminal statuses, DeletionState-guarded)
and healed by a rolling 30d maintenance sweep. The sweep prunes chain refs
of deleted sessions tombstone-driven (SessionTombstones cover cascade AND
retention deletions), deletes emptied rows, and recomputes daily FTR
aggregates (DeviceJourneyAggregates, PK tenant/"global", RK date):
completed journeys, first-time-right count, attempt histogram and
junk-serial disclosure — additive counts, bucketed on the completing
success session's StartedAt date. Chain refs carry the authoritative
DurationSeconds verbatim (never CompletedAt-StartedAt) plus an
adminMarked flag. Both tables registered in Constants.All + offboarding
TenantPartitionTables; aggregates under the 180d retention block;
deliberately NO per-session deletion-manifest step for DeviceHistories.
Backend 3433/3433 (51 new tests: junk/normalization vectors, terminal
set, exact 30d gap boundaries, WG one-row-one-attempt, redeploy split,
deletion pruning, entity round-trips incl. row-key escaping, FTR
aggregation core), Agent 2044+493 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Insights expansion PR3 (tasks/insights-expansion-spec.md §F1 "Surfaces").
Completes F1: schema (PR1) -> persistence/aggregation (PR2) -> surfaces.
- Rolling range rows: the sweep now also writes "rolling30|{class}"
aggregate rows — range statistics over the window's actual sessions. A
median of per-day medians is not the range median, so the fleet panel
reads these and is deliberately fixed to "last 30 days"; the daily rows
stay in the payload for trends/MCP.
- Routes (all registered in EndpointAccessPolicyCatalog):
GET sessions/{sessionId}/time-attribution (MemberRead + tenant query
param, GA fallback via FindSessionTenantId; breakdown=null is SUCCESS —
a missing lane is a normal state, not an error), GET
metrics/time-attribution (MemberRead) and GET
global/metrics/time-attribution (GlobalReadOrAdmin + tenantId filter),
both returning { windowDays, classes (rolling), daily }.
- Web: TimeAttributionLane under the session-detail phase timeline —
proportional stacked bar of segment totals incl. the explicit
unattributed remainder (bar always sums to the session's authoritative
duration), ESP-blocking apps with critical-path occupancy, reboot badge
(cross-cutting, never a bar slice), quality-flag chips; fetched
fail-soft for terminal sessions only. Fleet Health gains the "Time
attribution" section: per-class stacked medians gated at n>=20
("insufficient data (n=...)" below), flagged/missing exclusions
disclosed, and a top-blocking-apps table with class-scoped rows and
"up to X (p75 Y)" what-if wording. Deliberate deviation: the spec's
per-app trend column is omitted — per-day app rows are too sparse under
the daily n>=5 gate; revisit once rolling history exists.
- MCP: get_time_attribution (one tool, session + fleet mode, role-aware
routing like get_app_install_metrics, registered for all roles) — the
privilege-leak-guard catalog snapshot extended deliberately.
- Docs: OKF concept doc backend/time-attribution.md (invariant, windows,
segment mapping, positive-evidence blocking, Q7 reboot spans,
persistence + surfaces) registered in index.md + log.md.
Backend 3382/3382 (+1 rolling-aggregate test), web tsc clean + vitest
627/627, MCP tsc clean + 455 passed. Post-deploy: web deploy +
MCP redeploy (new tool + docs corpus).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Insights expansion PR2 (tasks/insights-expansion-spec.md §F1 "Data & compute
changes"):
- SessionTimeBreakdowns table (PK=TenantId, RK=SessionId): span/reboot/
blocking-app JSON columns, QualityFlags, AttributionVersion,
EspAppsOccupancySeconds kept tri-state (absent column = unknown, never 0).
Written once at the terminal single-writer seam (TableSessionRepository,
next to the counter reconcile + EspBlocking resolution; Succeeded/Failed
only — Incomplete deliberately stores no duration). Deleted with its
session: new deletion-manifest step 16 (PkRkExact; tombstone moves to 19)
plus tenant-offboarding partition wipe — no separate age sweep.
- Self-healing 30d maintenance sweep instead of a one-off backfill (rides
the existing 2h tick + manual parity): backfills pre-deploy sessions,
heals inline-compute misses, folds late-terminating sessions into their
StartedAt-date aggregates, and enforces AttributionVersion purity — a
stale-version breakdown whose events aged out counts as missing rather
than mixing algorithm semantics (truthfulness rule 8). Skips sessions
with a deletion cascade in flight.
- TimeAttributionAggregates table (PK=TenantId or "global",
RK="{date}|{class}"): per enrollment class (whiteglove /
device_preparation / self_deploying / user_driven — never mixed),
median/p75/p90 per segment over the fixed six-segment stack (incl.
unattributed; absent segment = honest 0), clean/flagged/missing counts
disclosed (rule 7), rows written below the n>=20 UI gate on purpose
(the UI needs the n — rule 4). 180d retention alongside UsageMetrics.
- What-if bound: WhatIfSavingSeconds implements the spec formula
max(0, cpEnd - cpEndWithoutX) including the idle gap before X (the ESP
waits for X to start, too); sole-interval fallback = own start. Per-app
aggregate rows gate at >=5 sessions, top 20 by median interval, carry
median/p75 of interval and saving ("up to" wording is the consumer
contract).
Backend 3381/3381 (13 new tests: entity round-trips incl. tri-state,
aggregation bucketing/disclosure/gates, what-if vectors, class precedence,
manifest step-count pins 17/19), solution build clean, web untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Insights expansion PR1 (tasks/insights-expansion-spec.md §F1, audit §0.5):
- AppInstallSummary + AppId / EspBlocking (bool?, tri-state) / AppIdCollision:
sentinel-gated columns + mapper + both projections. appId adopted from app
events (first-seen wins); a second appId under the same display name flags
the name-keyed row as a collision (audit Q3) — in-batch and cross-batch
(sticky in the reconcile). Collision rows leave every per-app fleet
aggregate (app metrics payload, apps list, app analytics) with a disclosed
exclusion count; the fleet DO rollup and per-app session drill-down keep
them.
- EspBlocking resolution at the session-terminal single-writer seam
(TableSessionRepository.UpdateSessionStatusAsync, next to the counter
reconcile — covers ingest, admin marks, maintenance sweep, rule engine):
joins the session's app rows against the LATEST list-carrying
esp_config_detected emission as a positive-evidence set (audit Q2) —
listed => true, absent => stays unknown, never false. EspBlockingSets
parser handles truncation via the uncapped espTracked*Count fields.
- TimeAttributionCalculator (pure, AttributionVersion 1) + SessionTimeBreakdown
models: exact partition of the session's authoritative DurationSeconds into
device_prep / esp_apps / identity_hello / user_esp / desktop_handoff spans
plus an explicit unattributed remainder — invariant sum(spans)+unattributed
== wall clock enforced in every test. WhiteGlove = two observation windows
in ONE session (part1 anchored on whiteglove_part1_complete, pause never
attributed). Per-app blocking intervals from EVENT timestamps (first start
-> LAST terminal, audit Q1 retry freeze), occupancy = merged union clipped
to esp_apps. Reboot spans from event gaps around lastBootUtc (audit Q7),
deduped per boot. QualityFlags: ClockSkewDropped, PartialObservation,
BlockingSetUnknown, BlockingSetTruncated, WhiteGloveAnchorsIncomplete.
Persistence (SessionTimeBreakdowns) and surfaces follow in PR2/PR3.
Backend 3368/3368 (35 new tests: 19 calculator golden fixtures + 16
identity/blocking/aggregate-exclusion), solution build clean, web untouched
(new response fields are additive).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Source-data audit (tasks/insights-expansion-spec.md §0.5) verified against
production: 17.9% of Succeeded AppInstallSummaries rows carry no duration —
78% are genuine skips (app_install_completed with payload state=Skipped,
"Update for X" WinGet policies not applicable) whose state the ingest
discarded, the rest are real installs whose start the agent never observed
(attach window). Averaging them as 0s understated avg install duration by
18% and distorted the slowest-apps ranking.
- AppInstallSummary.TerminalState (Installed/Skipped/Postponed/Error) from
the closing event's payload state; sentinel-gated column + mapper +
AppsDashboardProjection. Status stays "Succeeded" for compatibility.
- MetricsMath.IsSkipTerminalState/HasMeasuredDuration: durations computed
over measured rows only; skips leave the failure/success rate (decision
2026-07-26 — a skip is not an install attempt); per-app skipped/unmeasured
counts + totals exposed; slowest ranking gates on measured installs.
Applied consistently: app metrics payload, apps dashboard (list/analytics/
time series/trend/versions/device models), SLA weekly/snapshot/breach.
- StoreAppInstallSummaryAsync: reconcile extracted to internal static
ReconcileAppInstallSummaryWithExisting (test seam). Fixes Q4 (terminal
batch before started batch left duration permanently unset) and hardens
terminal status against late/replayed InProgress batches.
- Web: measured-only hints on app detail cards, skip suffix in apps list,
fleet-health "measured installs" caption.
Backend 3333/3333 (20 new tests), web tsc clean, vitest 627/627.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The "Collect Logs" feature (b382e567) added a second opt-in path into hosted
diagnostics upload — the quick-config dialog on a session — and a new collection
trigger, neither of which the customer-facing pages described. The trust FAQ still
said hosted upload is enabled "in the settings UI", and the dialog itself named
hosted storage without stating that packages leave the tenant boundary, so the
"clearly marked disclosure" promise was not met on that path.
- CollectLogsButton: marked data-boundary disclosure in the quick-config dialog
(leaves your Azure tenant, per-tenant isolation, blob-scoped write-only token,
removal per Data Retention Days)
- privacy + about pages: both opt-in paths named; on-demand collection described
(Admin/Operator of the own tenant, same content as the automatic package,
delivered with the agent's next check-in, visible on the session timeline)
- docs repo counterpart in trust/security-faq.md (separate commit)
Distress-bell dedup: the TpmPssUnsupported one-shot bell had key-shape tests only.
New HardwareRejectionNotificationTrackerDedupTests pins the 409 dedup branch, the
fail-soft branch, the blank-input short-circuit, the disjoint hardware/tpmpss key
spaces, and the retention sweep — which also prunes tpmpss rows and thereby re-arms
that device's bell, now stated in the interface and implementation docs.
Backend tests 3313/3313, tsc clean, web vitest app/sessions 60/60.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Rule list cards (analyze rules, gather rules, IME log patterns) used a
single-line flex header where every badge was flex-shrink-0 and only the
title could shrink — on phone widths the title truncated to a few
letters. The header now wraps: on mobile the title drops onto its own
full-width row (order-last + w-full) under the badge row and may wrap;
on >=sm the layout is unchanged (title inline, truncated).
PhaseTimeline squeezed up to 8 flex-1 columns with 48px circles and
whitespace-nowrap labels into the viewport, so circles and label text
overlapped completely on mobile. The timeline now has a minimum width
of 80px per phase inside an overflow-x-auto container (swipe on phone,
no scrollbar on desktop) and labels wrap inside their own column
instead of overlaying neighbours.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dark mode: globals.css was missing the entire slate family (the
"Incomplete" status filter chip rendered white-on-white) and all light
hover tints outside gray/indigo/cyan (hover:bg-purple-50 in the tenant
suggestion dropdown flashed near-white under light text). Added slate
bg/text/border mappings mirroring gray, plus dark equivalents for every
hover:bg-*-50/100/200 class in use (blue, purple, red, amber, yellow,
green, orange, sky, teal, emerald, slate, gray-200/300).
Mobile: ScriptExecutions rows were single-line flex with flex-shrink-0
right clusters, so badges and status text overlapped/overflowed on
narrow screens. Header counters, cycle card headers and item rows now
flex-wrap with gap spacing instead of space-x.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A Global Admin editing a gather/analyze rule while viewing a FOREIGN
tenant (override) got a green "saved" banner, but the write landed in
the GA's OWN home tenant: the JWT-scoped rules/{type}/{ruleId} routes
resolve the tenant from the caller's token, and no cross-tenant write
route existed (reads had /api/global/rules/*, writes did not). The
reload re-read the foreign tenant and showed the unchanged rule.
Fix: symmetric cross-tenant WRITE routes mirroring the global reads.
Backend:
- GlobalRulesFunction: PUT/DELETE global/rules/gather/{ruleId} and
global/rules/analyze/{ruleId}; tenant from ?tenantId= (400 if
missing), same 1 MB guard + ValidateScopeAndEmitMode as the JWT
variants, same service calls (toggle-partial/built-in-state/full-edit
semantics are tenant-parameterized and identical).
- Catalog: 4 entries GlobalAdminOnly + TenantScoping.QueryParam — a
write tier, so the delegated-read rescue never fires; only a platform
admin may mutate another tenant's rules. No global CREATE route:
create stays JWT-only and the portal hides Create in override.
Web:
- api.rules.globalGatherRule/globalAnalyzeRule helpers; both rule pages
route save/toggle/mark-failed/notify/delete through a
*RuleWriteUrl(ruleId) switch on isGlobalOverride.
- Analyze template-enable (a create) is blocked in override with a
clear message instead of silently writing to the wrong tenant.
Tests: CrossTenantAccessTests guards the 4 routes are GA-only +
QueryParam and that a GA reaches the named target tenant. Backend
3295/3295, web vitest 627/627, tsc clean.
Note: the old bug planted orphan copies of foreign rules into the GA's
home tenant on each attempted cross-tenant edit; those are not
auto-cleaned.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TpmPssUnsupported distress reports (TPM cannot RSA-PSS-sign, Schannel drops
the client cert on Win11 25H2+, agent can never authenticate) were operator-
only, yet the remediation is tenant-side: TPM firmware update or device
replacement.
- New GET audit/tpm-pss-unsupported (MemberRead), aggregated per serial,
mirroring audit/device-not-registered
- Bell notification "tpm_pss_unsupported" (Admin audience) once per
(tenant, serial); dedup via the existing rejection tracker table with a
disjoint "tpmpss|{serial}" row-key space; no webhook by design
- Web: TpmIncompatibleDevicesInsights panel in the Hardware Whitelist
section; renders only when a device is affected (no permanent card)
No agent change, no ConfigVersion bump. Backend tests 3287/3287, tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session 8e4cc4ae: two "Collect Logs" round trips completed on the agent (both
ZIPs landed in hosted storage) but the portal never flipped to Download and
the button hung at "Collecting..." — the ServerActionDispatcher confirms an
on-demand upload only via server_action_executed(actionType=request_diagnostics,
blobName=...) and emits NO diagnostics_uploaded (only the terminal upload path
does), so UpdateSessionDiagnosticsBlobAsync never ran.
The ingest classification now treats such an executed event as the upload
confirmation (a real diagnostics_uploaded in the same batch still wins — it
carries destination). Because the executed event has no destination field,
InferDiagnosticsDestination derives "Hosted" from the backend-built
"{tenantId}/{filename}" blob-name shape (CustomerSas blobs are bare
filenames); without it the download route would look in the customer's
container for a hosted blob.
Backend-only — works with the already-deployed agent fleet, no agent release.
Backend suite 3274/3274.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Operators previously saw editable settings forms bound to hardcoded defaults
whose Save always 403'd. Now they see the tenant's real configuration
read-only, with no save affordances at all.
Backend:
- GET config/{tenantId}: TenantAdminOrGlobalReader -> MemberRead. Safe because
the handler redacts by default (CanViewSecrets: only GA or own-tenant Admin
see secrets); Operators/Viewers receive RedactedCopyForReader().
- POST diagnostics/files: TenantAdminOrGA -> TenantAdminOrOperator (submitting
logs is a troubleshooting action; Viewer and read-only GlobalReader excluded).
- Tests: policy pins updated; new middleware tests (own-tenant Operator allowed
+ redacted, cross-tenant Operator 403); CanViewSecrets theory for
Operator/Viewer.
Web:
- TenantConfigContext loads the full config for all settings users (403
fallback to feature-flags minimal config for deploy-order safety) and
exposes canEditConfig; saveConfiguration guards on it.
- New ReadOnlyFieldset wraps section content (native fieldset disable, values
stay copyable); SaveResetBar is not rendered at all for read-only viewers.
Applied to Contact, SLA Targets, Hardware Whitelist (+ rejection-insights
toggle), Notifications, Agent Settings, Agent Analyzers, Diagnostics, Data
Management.
- Hidden entirely from Operators (in-page admin-only card + sidebar sub-item
gate): Autopilot Validation, Access Management, Offboarding, Unrestricted
Mode. GlobalReader sidebar parity kept via hasGlobalScope.
Tests: backend 3257/3257, web vitest 627/627, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers the queue-on-session/deliver-on-ingest channel, the
TenantAdminOrOperator tier with in-function per-type re-gating, the portal
Collect Logs flow (quick-config orders rotate_config before
request_diagnostics), sequence-based progress matching, and the hosted-prefix
sweep in the offboarding cascade. Registered in index.md, logged in log.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Queues a request_diagnostics ServerAction (delivered with the agent's next
ingest response) and tracks progress live: server_action_received flips the
button to "Collecting…", the DiagnosticsBlobName SignalR delta completes it
(Download Diagnostics appears), server_action_failed or a 3-min quiet timeout
surface as toasts.
- Button is ALWAYS rendered — disabled states carry tooltips (inactive
session, cross-tenant view, missing role, diagnostics not configured) as a
deliberate nudge towards enabling diagnostics upload.
- Tenant Admin on an unconfigured tenant gets a quick-config dialog: switches
the tenant to Hosted + OnFailure (GET config -> patch two fields -> PUT
verbatim), then queues rotate_config BEFORE request_diagnostics so the agent
collects against the fresh config. Operators/Viewers never see the dialog.
- Operator may collect when configured (new TenantAdminOrOperator backend tier).
- useSessionTenantConfig picks up diagnosticsUploadConfigured from the
member-readable feature-flags endpoint.
- Progress matching is sequence-based (backend-assigned) so agent clock skew
and at-least-once action redelivery cannot produce false matches.
Pure state logic in collectLogsLogic.ts with vitest matrix (21 tests).
tsc clean; web suite 627/627.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New EndpointPolicy.TenantAdminOrOperator (Admin/Operator/GA, no Viewer) with
middleware evaluator; POST sessions/{sessionId}/actions moves to this tier so
troubleshooting operators can queue request_diagnostics ("Collect Logs").
- QueueSessionActionFunction re-gates per type: terminate_session and
rotate_config stay Admin/GA-only (403 for operators).
- GET config/{tenantId}/feature-flags exposes diagnosticsUploadConfigured so
members below Admin can see whether an on-demand upload would work.
- Offboarding cascade: hosted diagnostics deletion now sweeps the session
prefix {tenantId}/AgentDiagnostics-{sessionId}- (plus the manifest-named
blob) so on-demand packages overwritten on the Sessions row by the terminal
package no longer survive deletion. CustomerSas branch unchanged.
- Notification dismiss routes deliberately NOT migrated: dismissal is still
tenant-shared; operators would clear admin-tier notifications they cannot see.
Tests: middleware role matrix on the actions route, per-type gate matrix,
feature-flag matrix, prefix-sweep + real hosted-branch cascade tests.
Backend suite 3253/3253.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Dependabot alert #74: PostCSS <= 8.5.11 allows arbitrary file read /
information disclosure via an attacker-controlled sourceMappingURL in CSS
comments. Raise the floor (devDependency + override) from ^8.5.10 to
^8.5.12; the lockfile now resolves postcss 8.5.22 (its nanoid dep moves
3.3.11 -> 3.3.16 with it). npm audit reports 0 vulnerabilities.
Next.js pins postcss 8.4.31 transitively, but the existing root override
collapses the tree to a single postcss node, so the bump covers it too.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Customer follow-up corrected the field-case mechanics: TrackingPoliciesCreated=1
WAS set under the foreign ConfigMgr provider - the ESP ignores a foreign
provider's completion values and waits on the Intune-registered "Sidecar"
provider by name (renaming the key unblocks instantly without creating any
tracking entries). The completeness-keyed detector would have missed exactly
the incident it was built for.
- Arm 2: >=1 Setup\Apps provider registered without Sidecar, continuously for
the 15-min dwell -> reason=sidecar_provider_missing; clock resets when
Sidecar registers later (legitimate startup ordering); DevicePreparation-only
snapshots don't activate it
- Arm 1 (continuously-incomplete providers) unchanged -> reason=provider_incomplete;
sidecar-missing wins when both arms cross in one round (single event)
- Cross-restart fingerprint gains a "sidecarMissing" sentinel; payload adds
reason + sidecarMissingForMinutes
- Regression test: ConfigMgr complete + Sidecar absent -> fires (previously
classified healthy); probe/doc headers and OKF doc corrected, web event-type
description updated
Evidence (research session c7e3db08): Intune's ESP-policy WAP payload always
registers Sidecar (ConfigMgr only via co-management settings policy, per
Niehaus' documented payload); Microsoft's ESP troubleshooting models app
tracking exclusively through the Sidecar subkey.
Agent tests 2043/2043, web tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Azure/static-web-apps-deploy@v1 has no `skip_deploy_on_missing_secrets`
input (confirmed against its action.yml), so passing it did nothing but
emit an "Unexpected input(s)" warning — the intended "skip green on
Dependabot PRs" behaviour was never actually in effect.
Replace it with the documented mechanism: surface the deploy token as a
job-level env var and gate the deploy/close step with `if: env.DEPLOY_TOKEN
!= ''` (secrets cannot be referenced directly in an `if:`). On Dependabot
PRs the token is absent, so the step skips and the job stays green; real
deploys on main and internal PRs still run. Applied to both jobs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BuildFleetFailureReasons truncated each reason to a 50-char prefix + "..."
and used that as both the grouping key AND the display value. The full text
never reached the UI, so its tooltip/tap-to-expand had nothing to reveal and
near-identical rows were indistinguishable past the cut.
Group by the 50-char prefix as before (variants still collapse into one row),
but carry the longest full reason as the display value. The UI truncates and
expands it on demand, so the complete text is now available end-to-end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The prior ">=22" made Oryx build with the highest available Node (24.13),
which the Static Web Apps runtime rejects: "Node version 24.13.0 is not
supported. Please use one of the following versions 18, 20, 22." Pin to
22.x so Oryx selects the SWA-supported major. CI setup-node '22' already
resolves to latest 22.x, so build and deploy now agree.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node 20 reached EOL (2026-04-30); Node 22 is the current maintenance LTS
and already the runtime the MCP container ships (node:22-slim). Straighten
the version skew so per-component CI major == deploy runtime == engines floor:
- ci.yml: web, mcp and guardrails setup-node 20 -> 22
- combine-rules.yml: setup-node 20 -> 22
- mcp package.json: add engines.node ">=22" (matches Dockerfile)
- web package.json: add engines.node ">=22" (first explicit pin; SWA/Oryx
no longer picks a Node version implicitly)
engine-strict stays off (no .npmrc), so the pins are advisory and cannot
hard-break local installs on other Node versions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The expanded state used line-clamp-2, which re-truncated long values with an
ellipsis after two lines — unreadable on touch where the hover tooltip is
unavailable. Expand now wraps the entire string over as many lines as needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add reusable TruncatedLabel component: a single-line label that truncates
with an ellipsis, shows the full text as a native hover tooltip, and on
click/Enter/Space expands to two wrapped lines (touch-friendly). An
interactive={false} mode provides tooltip-only behaviour for use inside
already-clickable rows/links where an inner onClick would hijack navigation.
Apply it consistently to the long, dynamic label columns across:
- fleet-health: failure reasons, model + app ranked lists
- sla: top failing apps, violators failure reason
- geographic-performance: fastest/slowest location cards
- vulnerability exposure panel: CVE sample software
- agent metrics: top exception types
- progress + download/install: app-name labels
- admin software tabs: software names
- session reports (comment/adminNote) + ops events (message): tooltip-only
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second half of the terminal-only rate convention (follows 1cfc32c9, which
converted the enrollment success rates): app-install failureRate divided
by ALL install rows including "InProgress" ones — installs still running,
or orphaned when a session died mid-install — diluting every failure rate
customers see on the Apps dashboard and Fleet Health.
- New MetricsMath.TerminalFailureRatePct (Failed / (Failed + Succeeded),
0 when nothing finished) shared by both aggregation sites
- MetricsMath.BuildAppMetricsPayload: per-app failureRate terminal-based
(Fleet Health "Top Failing Apps"; subtitle now "% of N finished")
- AppsAnalyticsHelper: list, analytics summary, versionBreakdown,
timeSeries and deviceModelBreakdown rates terminal-based; the
duplicated trend block is deduplicated into ComputeFailureTrend, whose
rates and >= 5-sample gate now run on finished installs per half; the
device-model >= 5 sample floor likewise counts finished installs
- Web: InstallsTab "Avg Failure Rate" card computed over summed finished
installs instead of totalInstalls
Tests: new AppsAnalyticsFailureRateTests (list/trend/analytics/model
cases incl. in-flight dilution and gate regressions) and a terminal-rate
case in MetricsMathAppPayloadTests. Backend 3223/3223, tsc clean,
vitest 602/602.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer feedback: the geographic performance pane showed a poor success
rate for a site mid-rollout because in-progress/pending sessions counted
against the rate. Success rate is an outcome quota — align Geo and Fleet
Health with the SLA convention already used by dashboard stats, usage
snapshots and the fleet rollup: Succeeded / (Succeeded + Failed).
Incomplete (terminal, non-failure) stays excluded.
- Geo: per-location SuccessRate over finished sessions; table badge and
map tooltip render an em dash instead of a red 0% when nothing has
finished yet
- Fleet Health: Stats.SuccessRate terminal-based; card shows "X of Y
finished enrollments" and a neutral em dash state; per-model health
gets a Failed count (additive wire field) so the client rate excludes
in-flight devices; TopFailingModels FailureRate now failed/finished
- Tests: FleetHealthPayloadTests flipped to the new convention plus
zero-finished, model-failed and in-flight-dilution cases; new
GeographicMetricsSuccessRateTests pin the geo semantics
Backend 3216/3216, web tsc clean, vitest 602/602.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI run 30047512186 failed Worker_drops_envelope_missing_required_fields_
without_invoking_handler: RunForAsync starts the BackgroundService poll loop
and waits a fixed 500ms wall-clock window — on a loaded CI runner (xUnit runs
test classes in parallel) the loop may not have processed the message before
the positive assertions (DeleteMessageAsync at least once) run. All ten
positive tests in the file shared that latent race; this run it hit the
malformed-envelope drop test.
Fix mirrors the watchdog event-gate de-flake: new Harness.RunUntilAsync
(condition, 10s ceiling, 25ms poll) runs the worker until the awaited side
effect is observable, then stops. Gates: ops-event recorded (last step of the
poison path, so send+delete+CAS are already done), main-queue delete for the
drop tests, >=3 visibility heartbeats for the heartbeat test. The kill-switch
test keeps the fixed window — a Times.Never expectation has no event to wait
for. OpsEventCalls capture is now lock-guarded so the condition poll can read
it while the worker thread appends.
16/16 worker tests green across 4 consecutive runs; full backend suite
3211/3211.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part 2 of the eaf3d8c4 fix (follows 088c3b04). In that session 147 of 157 app
events were replayed from a previous enrollment's IME log — all 10 "installed
apps" were phantoms, polluting the timeline, AppInstallSummaries, MCP counts,
the app_tracking_summary header, culprit lists and final-status.json.
Rename (nothing deployed yet): historic_script_replay_detected →
historic_ime_replay_detected; SuppressIfHistoricReplay / IsHistoricImeReplay.
Agent:
- EmitAppState suppresses stale-replay transitions BEFORE the sub-phase
declaration, timing bookkeeping and terminal-signal dedup — covers the
informational event, the AppInstallCompleted/Failed DecisionSignal, the
shadow download_progress and the summary snapshot. Behavior note: phantom
terminal signals can no longer spuriously re-arm the AdvisoryCompletion
window (app signals are otherwise observation-only — verified no completion
gate reads them). EmitDoTelemetry gets the same guard.
- Tracker-level guard at the HandlePatternMatch choke point skips the 23
app-mutating actions (incl. espphasedetected — a stale AccountSetup line
would make fresh DeviceSetup lines bounce as "backward") for source lines
>24h stale, so replayed apps never enter PackageStates, persistence,
app_tracking_summary, culprit lists or final-status. New internal
UtcNowProvider seam (no ctor change), threshold single-sourced as
ImeLogTracker.HistoricReplayThreshold.
- SimulationMode now bypasses all replay suppression — 088c3b04 had broken
the --replay-log-dir dev tool, which feeds historic lines on purpose.
Backend: IsHistoricImeReplay guard in AggregateAppInstallEvent keeps replayed
events out of AppInstallSummaries (SLA/dashboards/metrics inherit).
Web: generic lib/historicReplay.ts partition (scriptExecutions becomes a
wrapper); InstallProgress hides replayed installs with a muted note;
DownloadProgress filters silently.
MCP: get_session_summary filters replayed events — cleans appInstalls stats,
errorCount (replayed app_install_failed are Error-severity) and keyEvents.
Tests: +13 agent (app suppression, tracker guard), +1 backend, +7 web,
+6 MCP. Agent 2039, backend 3211, web 602, MCP 455 all green; drift test
validates the rename.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session eaf3d8c4: a previous enrollment's IME logs survived on disk (dated
2026-07-16, 7 days old) and the agent replayed them at startup — 156 phantom
script_completed events in a 7-minute session, each showing ~170h durations
(clock-clamped completion minus RAW stale start), plus an immediateUpload
flood (3x ingress_backpressure) and RemediationScriptCount inflated to 156.
Agent:
- Script events whose source line is >24h stale are suppressed entirely; a
one-shot historic_script_replay_detected Info event (new EventType) dates
the replayed window via earliestRejectedSourceTimestamp. Future-skew
rejections and clock-only fallbacks still emit.
- durationSeconds now uses a timeline-consistent pair: on rejected-timestamp
emissions the raw source completion pairs with the raw start (true runtime
instead of cross-timeline garbage). 24h plausibility backstop omits
implausible values; same consistency for script_timeout_suspected.
- Stale pending platform-script slots (>24h older than the current start
line) are discarded so an old orphaned start can't pair with a fresh run
(extracted HandlePlatformScriptStarted as internal test seam).
Web: partitionHistoricScriptEvents hides legacy-agent replay events (muted
"N historic executions hidden" note) and the reducer blanks >24h durations —
covers sessions recorded before the agent fix rolls out.
Backend: IsHistoricScriptReplay guard keeps replayed events from inflating
Platform-/RemediationScriptCount during the rollout window.
Registered historic_script_replay_detected in Constants.EventTypes, MCP
resource-catalog (script_events) + KEY_EVENT_TYPES, gather-rules eventTypes.
Tests: 8 agent (suppression/one-shot/raw-pair/backstop/slot-hardening),
8 web (partition/blanking), 10 backend (predicate). Agent 2026, backend
3210, web 595, MCP 449 all green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The straggler integration test copied powershell.exe under a synthetic
agent name and launched it — textbook MITRE T1036.003 ("Rename Legitimate
Utilities"). Defender for Endpoint flagged the run ("System executable
renamed and launched", initiating process testhost.net48.exe) and ISOLATED
the dev machine.
Replaced with function shadowing: a harness script defines Get-Process /
Stop-Process stub functions (PowerShell resolves functions before cmdlets)
and dot-sources the cleanup script. The stubs report a live agent-named
straggler on every probe and record the kill, pinning the identical
contract — poll by NAME until the deadline, force-kill by NAME, then
delete — with zero process spawns, zero PE copies, nothing EDR-visible.
The production routine never renames or copies executables; the flagged
technique existed only in this test. Lesson recorded in tasks/lessons.md.
22/22 cleanup tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DeviceSetup/Apps failed, the user pressed "Try again" ~23 min later, apps
recovered to 10/10 and DeviceSetup resolved - yet the advisory_completion
window un-defanged the stale failure and failed the live enrollment,
because the advisory variant was exempt from all re-arm/rebase guards.
Engine fix (DecisionCore):
- New facts EspAdvisoryFailureResolvedUtc + EspAdvisoryFailureCategory
(state-borne; deadline payload would be lost on re-arm/rebase).
- TryResolveAdvisoryOnCategoryRecovery in both ProvisioningComplete
handlers: category match sets the fact + emits
esp_failure_advisory_resolved.
- Guards now (!hasAdvisoryAnchor || advisoryResolved): enforcement-progress
re-arm, hello-never-observed promote and the reboot rebase apply again
once the failure demonstrably un-happened; unresolved advisories keep
the legacy un-defang exactly.
- Truthful terminal reason esp_recovered_without_completion_evidence
(+ failureRecovered=true, LastFailureTrigger=DeadlineFired so the
likely-stuck app promotion stays off).
Story events (agent tracker):
- esp_failure_retry_detected: failed subcategory leaves "failed" after the
terminal fire - consistent with the user pressing "Try again".
- esp_failure_recovered: previously failed category reaches success.
- EspFailureDetectedEventArgs.LikelyCulpritApps (never-started first,
cap 3) rides the EspTerminalFailure payload so esp_failure_advisory /
enrollment_failed name the culprit app directly.
Surfacing: MCP resource-catalog, web timeline badges (Recovered /
User retry pills, culprit chip on advisory), gather-rule autocomplete,
ANALYZE-ESP-006 "ESP Failure Recovered After User Retry".
Tests: AdvisoryRecoveryTests (13, incl. the 527/611/635/671 field-ordinal
replay that no longer fails the session), EspFailureRecoveryStoryTests (5),
snapshot-builder guards 18 -> 20. All suites green (DecisionCore 493,
V2.Core 2018, Backend 3201, web vitest 589, MCP 449 incl. drift).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer field case: pre-provisioned devices with the ConfigMgr client as a
blocking app occasionally park the user ESP at "Apps (Identifying)" for days.
Root cause (verified against the EnrollmentStatusTracking CSP doc): only a
"ConfigMgr" provider is registered under Device\Setup\Apps\PolicyProviders
(no "Sidecar" = IME) and TrackingPoliciesCreated never becomes 1 - a wait the
CSP defines WITHOUT any timeout. Renaming the key to Sidecar unblocks
immediately. Microsoft documents that ESP providers are not aware of each
other and that pre-provisioning is unsupported with co-management.
New always-on esp_policy_provider_stalled tripwire, generalized to any
provider in both kinds (Setup/Apps TrackingPoliciesCreated != 1,
DevicePreparation InstallationState not in {2,3}):
- EspPolicyProviderProbe: fail-soft registry read of the
EnrollmentStatusTracking mirror (device + user-SID scopes), defensive
value parsing (missing/unparseable = incomplete), ScopedOverride test seam.
Research and CSP contract quotes captured in the XML doc header.
- EspPolicyProviderStallDetector: pure EvaluateStallTransition on a 15-min
continuous-incompleteness dwell (resets on completion/disappearance;
absent key = normal early enrollment), one-shot per provider,
StartupEventGate fingerprint over the sorted stalled key set for
cross-restart dedup. Warning + ImmediateUpload via InformationalEventPost;
purely observational, no decision-engine coupling, no config toggle,
no ConfigVersion bump (disk_space_low precedent).
- EspPolicyProviderStallHost: 60-s wall-clock tick, deliberately outside the
StallProbe idle gate (idle clock resets on session activity) and the
idle-stopped periodic collectors.
- Registered in Constants.EventTypes, MCP resource-catalog (health_events),
web gather-rule autocomplete (stall category).
Guardrail companion: SOFTWARE\Microsoft\Windows\Autopilot joins the
gather-rule registry allowlist ("Autopilot / OOBE / Setup") so tenants can
build custom rules on the subtree (e.g. not_exists on the Sidecar provider
key); generated web mirror regenerated via combine.js; positive + sibling-
spoof guard tests added.
Docs: OKF concept doc agent/esp-policy-provider-stall.md (+ index/log).
Tests: 22 new detector tests (pure transition matrix + emission surface incl.
the ConfigMgr-only customer scenario and gate-across-restart dedup); agent
suite 2013/2013, web vitest 589/589 + tsc clean, MCP 449 passed incl.
event-types-drift.
Both the detector and the embedded allowlist take effect with the next agent
release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer report (session fcb44595): dashboard said 115 min, session detail
said 5h 2m for the same pre-provisioned enrollment. The detail's per-block
duration calc took min/max over ALL event timestamps inside each
sequence-block, which two real-world timestamp shapes break:
- an app_install_started re-emitted by the ImeLogTracker after the Part-2
resume legitimately carries its historical Part-1 event time -> the User
Enrollment block stretched back across the sealed pause (56m -> 3h18m)
- a do_telemetry with a +1h source timestamp (IME log line with a wrong
timezone bias, just under the agent's 1h future-skew clamp) stretched
Pre-Provisioning (58m -> 1h44m)
Sum: exactly the "5h 2m" the customer saw.
Block spans are now anchored on the first/last event BY SEQUENCE (sequence
is the canonical order), with Part 1 starting at session.startedAt for
backend DurationSeconds parity. Extracted as pure helper
computeWhiteGloveDurations with tests modeled on the real session; replay
against the customer's 880 events yields 1h 54m == backend 6872s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer report: ANALYZE-SEC-001 warned on devices whose firmware already
carried the Windows UEFI CA 2023 certificate — the agent only read the
SecureBoot\Servicing registry value, which mirrors the Windows Update
rollout and lags behind reality (OEM factory image, WinCS, WU not run yet).
Agent: new UefiSecureBootCertReader reads the UEFI db/KEK variables
natively (GetFirmwareEnvironmentVariableExW; SeSystemEnvironmentPrivilege
enabled on the process token and restored — first privilege-adjustment
code in the repo). Fail-soft, never throws; legacy BIOS maps to not_uefi.
secureboot_status gains per-certificate booleans, uefiFirmwareReadStatus,
and the one-sided marker uefiCa2023FirmwareConfirmed (emitted only on
positive confirmation, never as false).
Rule v3.0.0: not_exists precondition on the marker suppresses the rule
when the firmware confirms the certificate; the marker's absence keeps
old agents on exact v2 behavior, so the rule is safe to seed before the
agent rollout. Firmware-verified-missing raises confidence (+20 db,
+10 KEK 2K CA 2023).
Verified: full solution build; backend 3201, agent 1988, web 582 tests
green; tsc/ajv clean; elevated cross-check on real firmware matches
Get-SecureBootUEFI on all seven certificate probes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sample Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState
(build >= 17763, SYSTEM-callable, CLR-native WinRT projection — no NuGet)
as a purely observational signal, mirroring the bootstrap guard marker:
- oobeStateAtAgentStart on agent_started and agent_late_start payloads:
makes late starts provable instead of heuristic ("completed" on a
first_run = OOBE was already over when the agent arrived) and validates
the bootstrap OOBE guard fleet-wide.
- One-shot oobe_state_completed trace event on the in_progress->completed
flip, piggybacked on the DesktopArrivalDetector 30s poll: an
owner-independent desktop corroboration for DAD-starved sessions
(4d5a0b78 class — WTS+WMI owner resolution dead, desktop_arrived never
fires). Often absent in healthy sessions since the DAD timer is
disposed at desktop arrival, which is when the flip lands.
OobeStateReader is fail-soft (never throws, "unavailable" on missing
contract); the flip check has its own try/catch so it cannot pollute the
DAD wmiErrorCount liveness diagnostics, "unavailable" never overwrites
the baseline, and the one-shot flag survives ResetForRealUserSwitch.
No decision-engine involvement, no config/backend changes. Event type
registered in Constants.EventTypes, MCP resource-catalog (phase_events)
and web gather-rule autocomplete.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six integration tests execute the generated cleanup script with a real
powershell.exe against GUID-unique scratch trees — proving what the string
pins cannot:
- locked tree ⇒ complete abort, marker + binaries survive (the 2026-07-22
regression: the old script force-deleted around the locks)
- full incident replay: a renamed second process holds a file in the tree,
is found by NAME, force-killed after the wait deadline, cleanup then
completes with nothing left behind
- fresh foreign lock ⇒ loser backs off without touching the tree; stale
lock (>10min) is taken over and cleanup proceeds
- happy path + keepLogs variant
BuildCleanupScript gains test-only timing parameters (defaults unchanged
for production) so the replays run in seconds. Agent suite 1969/1969.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Oliver Kieselbach