IME never writes its "Completed user session" line when the user has no
assigned Win32 apps - its zero-app branch returns before that statement
(ApplicationPoller.cs:713-721, identical in 1.80/1.83/1.97/1.104). So
ImeUserSessionCompletedUtc stayed null, every arm of
ShouldTransitionToAwaitingHello was unsatisfiable, and the 30-min
advisory_completion backstop resolved healthy enrollments as
esp_exit_without_completion_evidence. One tenant: 10 of 30 sessions in a week,
all false positives - DAD-validated desktop, Hello provisioned, RealmJoin still
installing. Every Succeeded session there carries ime_user_session_completed;
not one of the failed ones does.
IME-USER-SESSION-ZERO-APPS reads the line IME does write there and routes it
through the same OnUserSessionCompleted callback. That is IME's own verdict
rather than a weakened gate: the zero-app branch locks the ESP in with an empty
application list, and the sibling emit site returns ProvisioningComplete right
after logging it. Guarded to the AccountSetup phase so it cannot burn the
adapter fire-once flag on a pre-sign-in timestamp, and replayed on the phase
transition when the observation predates the phase marker. AreUserEspAppsSettled
accepts an empty app list only with that explicit observation, never on
emptiness alone.
Second, independent defect: advisory_completion (30 min) overran the RealmJoin
completion gate, which deliberately holds the session and carries its own
bounded resolution (60 min from detection, activity-extended, 4 h cap). One
session installed 25 RealmJoin packages inside the window, the last 2 min before
the failure, while completion_waiting still read "waiting on:
realmjoin_resolution". The handler now re-arms while that gate is closed and its
own deadline is still armed; convergence comes from the gate itself, which
always ends with an Outcome set.
ime_user_session_completed gains an evidence field (zero_user_apps_enumerated /
user_session_completed) - no new event type, so the Constants ->
resource-catalog -> shared-manifests triple stays untouched. The pattern ships
via the config API and reaches existing agents, where an unknown action is a
single Debug line; the behaviour needs the agent release.
Build 0 warnings. DecisionCore 558/558, V2.Core 2530/2530, Backend 4634/4634,
MCP event-types-drift 6/6.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Same gate the cert-device-binding shadow got in 35a3ed0c, applied to its
sibling. Measured on the live enrollment: 263 identical shadow lines for one
device off a handful of Graph calls, because the cache answered nearly every
request and the log sat outside it.
DeviceAssociationResult.AsCacheHit() copies rather than marking the shared
cached instance, so the first real lookup is never mistaken for a cache hit
and dropped. Transient outcomes are not cached and keep logging every time.
Backend 4633/4633, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured on the first live enrollment: 136 agent requests produced exactly
ONE Graph call, because the 30-minute positive cache absorbed the rest. So
the check is not a throttling risk (the per-request Autopilot serial
validator made 25 calls in the same window) and the positive TTL needs no
change — raising it would only delay noticing a wiped or removed device.
What the 136 requests did cost was 136 log lines all saying the same thing.
Gate the shadow log on ServedFromCache so it emits one line per real lookup.
Cached repeats carry no new information: the reported age just counts up
mechanically from the same enrolledDateTime. Outcomes that are never cached
(Transient, NoDeviceIdInCert) keep logging every time, which is what an
anomaly warrants.
AsCacheHit() copies rather than marking the shared cached instance, so the
first real lookup is never mistaken for a cache hit and dropped.
Backend 4631/4631, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tenant binding (stage 1) proves the client certificate was issued to this
tenant. It does not prove the certificate belongs to a device the tenant
still has: a cert lifted from a decommissioned machine keeps working.
IntuneDeviceBindingValidator resolves the Intune managedDevice id from the
certificate's Subject CN against the tenant's own managedDevices inventory.
Point-GET on managedDevices/{id} rather than an OData filter, so there is no
filter literal to inject into and a 404 is unambiguous. Cache/retry/transient
contract mirrors CloudPcDeviceValidator.
Shadow only, and visible only to Global Admins (same gate as the DevPrep
association preview) because the open question is a race, not an attack: a
device object can appear after the agent's first call. Every result carries
enrolledDateTime and the log line records enrolledAgeSeconds, so a device
created seconds before the request is distinguishable from a foreign cert.
Grep marker: CERT-DEVICE-BINDING-SHADOW.
The lookup is fire-and-forget like the DevPrep shadow so a cold Graph
round-trip never sits in the agent request path. That rules out stage 1's
request-row dimension (the row is written before the task finishes), hence
the Warning log; widening beyond preview should move to inline + dimension.
Needs DeviceManagementManagedDevices.Read.All, granted per tenant via the
customer script as feature IntuneDeviceBinding. The script's catalog,
ValidateSet and help block stay pinned to GraphFeatureCatalog by
GraphFeatureCatalogSyncTests (extended with the help-block case).
Build 0 warnings, backend 4629/4629, web 874/874, tsc clean, eslint 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Post-deploy verification of 249e39bc found zero AgentCertTenantBinding rows
across 32k agent requests. Cause: the Match outcome was emitted with
LogInformation, and worker-side ILogger below Warning never reaches App
Insights (provider default rule is Warning+, host forwarding of worker logs
is off). The bulk outcome was therefore invisible, which is indistinguishable
from the code never running — the shadow telemetry had no denominator.
Stamp every outcome onto the request row instead (FunctionContext.Items ->
RequestTelemetryMiddleware -> CertTenantBinding dimension). That row already
exists per request and is unsampled, so the denominator costs no extra
telemetry. Non-Match outcomes keep their Warning trace with full context.
FromCache existed only to sample the dead Match line and is removed with it.
Backend 4606/4606, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A chain-valid Intune MDM certificate only proves "issued to some tenant" —
the pinned roots are shared by every Intune tenant. Tenant scoping rested
entirely on the device serial in the downstream validators, which is not a
secret. Bind the tenant stamped into the auth cert to the requested tenant.
The Entra tenant id sits in OID 1.2.840.113556.5.14 of the cert the agent
already presents for mTLS, so this needs no Graph call, no consent and no
agent release. Not 5.6 (Intune Account id, a different GUID) and not 5.4
(device id, stored as 16 bare bytes) — the decoder handles both encodings
and now lives once in Shared, used by agent and backend.
Stage 1 observes only: ObserveCertTenantBinding returns void and swallows
its own exceptions, so it cannot fail an otherwise valid request. Match is
logged only on cert-cache miss, keeping the denominator without per-request
volume. Grep marker for stage 2: CERT-TENANT-BINDING-SHADOW.
Backend 4604/4604, Agent V2.Core 2519/2519, 0 warnings.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One CVE commonly appears under many findings of the same report (e.g. one
RD CVE across 14 installed RD components). Each occurrence produced its own
concurrent upsert on the same PK/RK, so the stored SoftwareName/OverallRisk
was race-dependent. Collapse to one entity per CVE (highest CVSS wins,
IsKev OR-ed, OverallRisk max) and write with Parallel.ForEachAsync(8) and
per-entity fail-soft logging instead of Task.WhenAll + one outer catch.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both sliding-window limiters re-Set the cache entry without its eviction callback.
The replaced entry fired the callback asynchronously, removing the per-key lock while
a new one was already handed out, so two threads could mutate the same List under
different locks; the replacement entry never cleaned its lock, leaking one object per
key for the process lifetime.
The history now lives in a RequestBucket together with its own lock: same list ⇒ same
lock, no callback, no dictionary. Bucket creation is serialized (GetOrCreate is not
atomic) behind a lock-free TryGetValue fast path. The redundant re-Set is gone — the
lookup already refreshes the sliding expiration.
Concurrency tests (16 threads × 500 requests, one key) fail 3/3 on the old code and
pass on the new one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
StartupTelemetryService emits BackendStartupMs (process start ->
ApplicationStarted) and BackendTableInitMs as App Insights custom
metrics with version/commit/tableInitFullPass/instance dimensions;
TableStorageService exposes LastInitialization. MCP server writes one
{"type":"boot","readyMs"} stderr line per start next to the
tool_call lines so cold-start time is queryable as a trend.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cold start point-reads a SHA-256 over TableNames.All stored in
AdminConfiguration/SchemaSentinel/tables instead of issuing 66
CreateTableIfNotExists calls. Missing/stale sentinel runs the full
idempotent pass once and rewrites it. SessionsIndex startup backfill
only after a full pass, behind a conditional-insert claim (one instance
under scale-out); IsSessionIndexEmptyAsync no longer treats errors as
empty. Daily maintenance runs IStorageInitializer.EnsureAllAsync to
repair out-of-band table deletions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First production pass alerted on agent:whiteglove_pending (two tenants in a
WhiteGlove rollout week) and agent:complete_soft (a Continue-Anyway tenant) —
customer workflow mix, not classifier drift. agent:* and register:* are now
ineligible for share_regression (still in the matrix and the silence group);
episodes opened by the broader gate re-arm on the next pass.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cross-tenant StartedAt-only date-range query is a full-table drain in
Table Storage; time attribution, device journeys, verdict calibration and the
calibration radar's tenant discovery each ran their own — four identical reads
per 2h tick. RunAllAsync / RunManualAsync now load the window once through
GetMaintenanceWindowSessionsAsync (MaintenanceSweepSessionProjection: the union
of the sweeps' columns, drops FailureSnapshotJson/geo/OS/diagnostics) and the
sweeps slice it by StartedAt. MaintenanceSweepProjectionEquivalenceTests pins
that every consumed field survives the projection.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Applies the rule-regression statistics (7d vs 28d, lift >=2, Wilson-separated,
small-n gates) to the verdict-calibration rows: per-path share regression,
sweep+maxlife silence-share regression (agent liveness signal) and an
absolute r6 evidence-gap gate. Episodes live in the notification tracker
(verdictcalibration|kind|path|status), fire one VerdictCalibrationDrift ops
event (operator-only, no tenant bell), re-arm under 1.5x / 15%, and surface as
alerts[] + a Drift pill in the calibration matrix.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Daily (tenant x StartedAt date) buckets per verdict path: count, derived
attribution, 7-day re-enrollment proxy (eligible/re-enrolled), and the
correction stream attributed to the PRIOR path (admin / late completion /
other). Written by the 2h maintenance sweep after the device-journey pass,
180d retention, regenerable. Read via GET global/metrics/verdict-calibration
(GlobalReadOrAdmin, 7d-vs-28d share trend, rates withheld below n=20),
the get_verdict_calibration MCP tool, and the admin metrics section.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every status writer now declares the code path that produced the status
(VerdictPaths vocabulary, stamped for all statuses — unlike FailureSource
which is Failed-only), the timeout classifier returns its rule id, and a
write that overrides a prior verdict preserves it as PriorStatus/
PriorVerdictPath (admin mark, late completion, retro). Pre-instrumentation
rows derive a path read-side (VerdictPathDerivation, legacy:* when
ambiguous). Foundation for the verdict calibration aggregate + radar.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Kiosk tenant audit 2026-08-23: 836 of ~970 self-deploying sessions in one week
were parked AwaitingUser ("Account Setup 0/5") and graduated to Incomplete 51h
later although the devices were finished — the profile has no user phase, so the
classifier's rule 5 was asking for evidence that can never arrive.
- EnrollmentTimeoutClassifier: new rule 1c IsSelfDeployingProvisioned (profile flag +
Device Setup all-succeeded + no explicit failure) => Succeeded reconcile with an
honest reason; isSelfDeployingProfile passed by every caller (sweep stage 1+2,
max-lifetime verdict, late-telemetry heal, legacy retro mode).
- MaintenanceService stage 1: TryReconcileSelfDeployingAsync gate before the Stalled
marker (mirrors the WhiteGlove Part-2 gate), audit counter SessionsSelfDeployingReconciled.
- LegacyReclassificationService: new admin mode self_deploying_silent (heal-only,
dry-run by default) + GetSelfDeployingSilentSessionsAsync for the existing backlog.
- Agent SessionRegistrationHelper: wait up to 15s for a network link and retry 6x
(~62s) instead of 5x (~30s) so the BootTrigger relaunch on Wi-Fi no longer exits 7
before the link is back (the mechanism that left these sessions silent).
- Tests: 6 classifier cases (gate, within/past grace, user-driven guard, failure,
pre-Device-Setup), 2 registration tests; docs/backend/silent-session-classification.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The diagnostics package's built-in collection is now ONE catalog in Shared
(DiagnosticsBuiltInSections.All): the agent iterates it when building the ZIP and
the backend serves the same list to the portal (GET /api/diagnostics/paths,
MemberRead), so what administrators see is exactly what the agent collects.
Agent
- New conditional sections: RealmJoin client/package/Chocolatey/per-user logs
behind the tenant's RealmJoin Watcher toggle (mirrored onto
AgentConfiguration.EnableRealmJoinWatcher by RemoteConfigMerger) and the Device
Preparation bootstrapper event log behind the deterministic WDP marker (probe
once per build, fail toward Classic). ZIP layout mirrors the disk
(RealmJoinLogs/Windows|Choco|User). Every skip is a package-manifest line.
- Built-ins bypass DiagnosticsPathGuards by design (reviewed code); caps,
reparse-point skip and evtx export still apply. No ConfigVersion bump.
Backend
- GET diagnostics/paths (MemberRead, JWT-scoped) with the catalog + global paths.
- MergeDiagnosticsLogPaths: global ∪ tenant dedup (trimmed, case-insensitive) —
duplicates used to produce duplicate ZIP entry names.
Web
- Collapsed read-only built-in list (RealmJoin on/off in the tenant header,
neutral in the Global-Admin view), one-line DiagnosticsPathRow for every entry,
add-row above the lists; tenant context no longer fetches /api/global/config.
Tests: agent RealmJoin on/off, no user session, bootstrapper evtx, probe failure,
merger mirror; backend catalog invariants, payload shape, dedup, policy; web
builtInSectionDisplay. Docs: docs/agent/diagnostics-package.md (+index/log).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- EmailTemplateService: override-or-built-in resolution per send (PreviewConfig
table, partition EmailTemplates, 5-min cache, {{domainName}} placeholder,
30k limit); send path fail-soft, admin read path strict
- GA endpoints global/email-templates/{kind} (GET/PUT/DELETE) and /test, which
mails the effective template or an unsaved draft to the caller's tenant
contact address
- portal: Settings → Email Templates with preview frame, test send, HTML editor
- mail header: brand mark drawn with CSS borders, no images left in the mails
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- EmailService + IEmailService: Mandrill messages/send via typed HttpClient with the
Notification resilience policy; provider confined to one transport method and the
Email:* settings (ApiKey, Endpoint, FromAddress, FromName). Resend NuGet dropped.
- open/click tracking explicitly off; sent/queued/scheduled = ok, rejected/invalid = warn
- callers (TenantApprovalService, PreviewWhitelistFunction) depend on IEmailService
- EmailServiceTests pin wire format, result semantics and fail-soft contract
- privacy page + OKF welcome-mail doc updated for the new sub-processor
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Discord (provider 30) renders alerts as a single embed with Discord's
field/length caps; actions become markdown links since webhooks cannot
post buttons. Generic JSON channels gain an optional per-channel signing
secret: requests carry X-AutopilotMonitor-Timestamp and
X-AutopilotMonitor-Signature (sha256 hex HMAC over "{timestamp}.{body}"),
computed over the exact posted JSON with same-named custom headers
stripped first. Secret is redacted for readers and restored by channel
id on save; validation enforces 16-128 printable ASCII characters.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Retention was the only enforcement point that destroyed state on a
Pro -> Community downgrade: the 12-hourly sweep clamps to the Community
cap at read time and would immediately start hard-deleting the
90-365-day band (trial expiry identical, silently).
New policy: for 30 days after losing Pro
(FeatureEntitlementCatalog.RetentionDowngradeGraceDays) the Pro cap
keeps applying. Anchor model: ProDowngradedUtc (backend-only field,
patch-denied + revert-protected) stamped/cleared by the plan endpoint's
new pure ApplyPlanChanges core on EFFECTIVE-edition transitions (covers
explicitly ended trials; a planTier downgrade under an active trial
deliberately does not stamp); trial expiry anchors read-time via
TrialExpiresUtc. days<=0 (infinite) stays untouched.
Ops visibility: TenantPlanDowngraded (PATCH handler, new OpsEventService
dep) plus TenantRetentionGraceExpiring/Ended from the daily
TrialExpirySweep with the same stateless window mechanics, emitted only
when stored retention exceeds the Community cap; all dual-registered in
the web OPS_EVENT_TYPES catalog. Fanout now uses its _utcNow seam
instead of inline DateTime.UtcNow. OKF doc
backend/plan-downgrade-retention-grace.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fairstone.ca analysis 2026-08-21: after the reseal-reboot technicians routinely
power devices off at the logon screen to box them. Both silence paths (2h
maintenance sweep, agent stall probe) then labeled the expected parking state
"Stalled - Agent silent", and stage 2 later reconciled it to Succeeded on the
Part-1 whiteglove_complete with a reason claiming Account Setup completed.
- EnrollmentTimeoutClassifier: shared pure gate IsWhiteGloveAwaitingUser
(pre-provisioned + ResumedAt + Device Setup provisioned + zero user evidence
since resume) with one canonical customer-facing reason; new rollup field
HasEnrollmentComplete separates real completions from Part-1 whiteglove_complete
- ClassifyTimedOutSession: gated sessions are AwaitingUser within grace and
reconcile past grace with an honest WhiteGlove reason; on resumed sessions
whiteglove_complete no longer counts as terminal-complete evidence (rule 2),
so partial user progress falls through to the honest AwaitingUser/Incomplete rules
- MaintenanceService stage 1 + session_stalled ingest mapping apply the same
gate (fail-soft to Stalled); sweep audit now carries SessionsAwaitingUser
- Callsites (stage 2, max-lifetime verdict, late-telemetry reconcile, legacy
reclassification) pass IsPreProvisioned/ResumedAt through
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Field case d0c5b672: a pre-agent system_clock_changed event (backfilled from
the Windows event log with its original timestamp, ~14min before the first
real activity) became the partition's earliest row and re-anchored the
session's StartedAt, inflating the duration toward pre-enrollment OOBE idle
time. With the watcher's 24h backfill lookback, the 2h backward-shift guard
only caps this effect, it does not prevent it.
SystemTimelineWatcher rows (clock steps, sleep episodes) are environment
observation, not enrollment activity, and are now excluded from all three
anchor derivations: the Events-table earliest-row probe (new
IsSessionAnchorEligible, probe scans past ineligible rows), the ingest
batch's EarliestEventTimestamp fold, and the web's client-side
enrollment-duration derivation. The events themselves stay on the timeline
unchanged.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two enrollment blind spots become timeline ground truth from the System event
log via one EventLogWatcher (WindowsUpdateTracker pattern, 24h backfill with
RecordId watermark recovering pre-agent records):
- system_clock_changed (Kernel-General 1): oldTime/newTime/signed timeDeltaMs,
reason and the setting process. 2s floor suppresses w32time 1ms micro-slews;
steps >= 5min are Warning + immediate upload. Payload instants are the
authoritative source (event timestamps can be clamped server-side).
- system_sleep_episode (Power-Troubleshooter 1 classic sleep/hibernate,
Kernel-Power 507 Modern Standby; 506 deliberately unconsumed): kind,
enteredAt/exitedAt, durationSeconds, wake source / AC state. 60s real-sleep
floor filters Modern Standby screen-off scenarios.
ConfigVersion 39 (Collectors.EnableSystemTimelineWatcher +
SystemEventBackfillLookbackMinutes). Both types are non-activity on agent and
backend (idle/stall clocks unaffected; a sleeping device stays Stalled).
Consumers: TimeJumpBadge gains the ground-truth cause "clock-set"
(classifyTimeJump matches backward display steps against recorded clock sets);
EventTimeline badge blocks for both events plus the previously bare
power_state_change; SessionInfoCard keeps the wall-clock duration and adds a
standby note; TimeAttributionCalculator builds SleepSpans (RebootSpan clone,
AttributionVersion 3) persisted as SleepSeconds/SleepSpansJson and rendered in
TimeAttributionLane; ANALYZE-DEV-012 (info, single) fires once per session for
episodes >= 5min - deliberately the only new rule, clock findings stay with
DEV-008 to avoid duplicate-finding flood.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All is a script-side convenience that expands to every GraphFeatureCatalog
feature (works with -Revoke too). The catalog sync test now pins All in the
ValidateSet while keeping it out of the catalog comparison, and the admin UI
mentions the shorthand next to the generated grant command.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Event-driven AC/battery observation during enrollment (no polling): a WMI
Win32_PowerManagementEvent (EventCode 10) push subscription re-probes
GetSystemPowerStatus and diffs snapshots through a pure tracker into
power_state_change events - transition ac_to_battery (Warning, immediate) /
battery_to_ac (Info) / threshold_crossed at 50/30/15% (per-level lifetime
latch, lowest-crossed-only on jumps, arm-time baseline covers sessions
starting low, 5s trailing-edge debounce, emission cap 20). Never armed on
devices without a battery; arm failure surfaces as collector_degraded.
Kill-switch Collectors.EnablePowerStateWatcher (ConfigVersion 38).
power_state_change is classified non-activity on both sides
(SignalActivityClassifier + IsPeriodicOrStallEvent) so battery drain can
neither keep idle collectors alive nor heal a Stalled session.
New analyze rules, both with on_event:power_state_change interim triggers
and deliberately single-required-condition (event_data conditions scan
events independently - no same-instance join):
- ANALYZE-DEV-009 (high): thresholdPercent equals 15
- ANALYZE-DEV-010 (warning): transition equals ac_to_battery
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Custom rules are tenant-partitioned and functionally collision-free, but three
surfaces still keyed on the bare non-unique ruleId string:
- RuleStats global aggregate (global_{ruleId}) summed same-ID custom rules of
unrelated tenants (title/severity last-writer-wins). The global row is now
catalog-only: analyze gates on IsBuiltIn/IsCommunity, gather gates on the
reserved-namespace pattern (agent events carry no flag; custom rules can
never occupy the reserved pattern).
- The 'never ship a built-in in the CUSTOM namespace' commitment from
RuleIdPolicy was documented but unenforced: combine.js now fails on
duplicate or non-reserved built-in IDs, BuiltInRuleCatalogPolicyTests pins
the same invariants on the embedded catalog, and the GitHub reseed rejects
non-reserved IDs before writing to the global partition.
- DeleteRuleStatesForRuleIdAcrossTenantsAsync (cross-partition delete by bare
RowKey) now refuses IDs outside the reserved namespace with failed=-1 so
callers abort the follow-up global catalog delete.
Verified: prod global partitions (45 analyze + 11 gather) are fully inside the
reserved namespace, so the guards touch no existing data. New OKF doc
docs/rules/rule-id-namespace.md captures the identity/collision model.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The async queue correlation stored the event with Sequence 0, sorting it to
the top of the timeline (and tripping the time-jump badge) even though it
arrives last. Follow the BuildSessionTimeoutEvent/BuildAgentEmergencyBreakEvent
pattern: max+1 so it sorts last; WhiteGlove Part 1 anchors at the Part-1
terminal markers so a late device_setup report can never cross the web
timeline's sequence-based Part-1/Part-2 split. Also upsert the EventType
index the queue path never wrote, making the event visible to cross-session
search.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The IME processes the ESP app list in multiple passes (device-ESP evaluation
ending Skipped, real install up to an hour later), and AppInstallSummary
folded first observation -> last terminal across them: a 23-second app late
in the queue reported 4002 s (session 44862cd3, "Set TimeZone").
Backend ingest: new max-folded LastAttemptStartedAt anchors DurationSeconds
(CompletedAt - last attempt start; legacy rows without the anchor keep the
span fallback), replay-guarded additive InstallPassCount, attempt-scoped
DownloadDurationSeconds pairing, and a weaker-terminal guard (completed with
state Skipped/Postponed never overrides Installed/Error, in-batch and in the
cross-batch reconcile). StartedAt stays the earliest-wins window/version-
ordering key. AppsDashboardProjection carries the new columns.
AppVersionRegressionRadar: cutover gate — only attempt-anchored rows enter
the duration population on BOTH comparison sides, so mixed-semantics medians
can neither mask real regressions nor falsely re-arm active episodes.
Time attribution: per-pass segment pairing replaces the first-start ->
last-terminal hull. An app's seconds = sum of its ACTIVE segments (failed
attempts and evaluation passes count; inter-pass idle gaps don't), occupancy
merges the segments, hull endpoints remain for display and what-if bounds.
Agent: a terminal->active transition re-arms BOTH timing endpoints, so event
payload durationSeconds also describes the attempt.
Web/PDF/MCP: labels and tooltips say "install time (final attempt)", the
apps sessions table gets a xN multi-pass chip, the time-series chart carries
a cutover footnote, MCP tool descriptions document the semantics split
between get_app_install_metrics and get_time_attribution.
No backfill: pre-cutover rows keep span values (documented in the public
changelog). OKF doc: docs/backend/app-install-duration-attempt.md.
Suites: Functions 4383, Agent 2407, MCP 564, Web 814 - all green, build 0/0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Enrollment durations are heavily right-skewed; a handful of overnight/WhiteGlove
outliers made the Fleet Health and Dashboard average cards read like hours
(541 min) and answer no real question. Both cards now lead with the median
(robust 'typical enrollment' answer) and keep the tail visible via a P90
subtext. AvgDurationMinutes stays in both DTOs for API compatibility.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
bbc7a1d8 stamped LastIngestAt into IncrementSessionEventCountAsync's merge
entity intending it as Sessions-only ("deliberately not mirrored"), but that
same entity flows through SyncSessionIndexAsync into the index merge — the
field was already landing in SessionsIndex on every ingest batch, just
undeclared. The fail-soft drift guard has been warning ~1,200x/h in prod
since the 08-20 deploy, and a StartedAt-shift full rebuild would silently
drop the column, making index-served session reads lose lastIngestAt that
Sessions-served reads carry.
Declare it properly instead of suppressing: ConditionallyProjected in
SessionIndexFieldManifest, projected in BuildSessionIndexEntity (read side
MapToSessionSummary already handled it), sample value pinned in the
bidirectional manifest tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Agent stamps X-Send-Time-Utc (device clock, ISO-8601 round-trip) per upload
ATTEMPT; ingest parses it (year>=2000 guard, deliberately no upper clamp) and
server-stamps SentAt onto every event of the request. SentAt - OccurredUtc is
pure spool delay, ReceivedAt - SentAt is network + device-vs-server clock
offset - previously indistinguishable and only separable via median statistics.
clock_skew condition: batches carrying SentAt measure the frame directly
(spool-immune by construction - no spread cap, IME/clamped exclusions
irrelevant, IME-only sessions become measurable; evidence sentAtBatchCount);
legacy batches keep the per-event median path unchanged. CmTraceSkewTripwire
deliberately untouched: its differential median math cancels clock offset as
common mode.
Full persistence chain: Store x2 + Map, fields= projection (sentAt), MCP
get_session_events field list + rule-authoring-guide. New OKF doc
backend/send-time-frame-separation.md.
Suites: Functions 4370 (+16), Agent 2407 (+1), MCP vitest 564 + tsc 0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Operator side: CmTraceSkewTripwire piggybacks on the terminal counter
reconcile's partition scan (zero extra reads) and compares median
ReceivedAt-OccurredUtc deltas of IME-derived vs other events. A clean
15-min-grid divergence (residual < 2 min, >=20 samples / >=3 distinct
batches per side) emits the CmTraceTimeSkewRegression ops event
(Agent/Warning, goal: never fires). Bias-dominated sessions are
suppressed; measuredWriterOffsetMinutes is never consulted (sticky
after era flip-backs). Kill switch: CmTraceSkewTripwireDisabled.
Customer side: new clock_skew analyze condition source (skewMetric
clock_jump | sustained_offset, threshold seconds, gt/gte magnitude)
behind built-in ANALYZE-DEV-008 (300 s = the Kerberos/Entra 5-min skew
boundary). Batch-median frames with spool-spread filter, plateau step
detection (a gradual backlog-drain ramp cannot fake a jump) and
end-state persistence; IME-derived events are excluded so a CMTrace
anchoring regression can never fire the customer rule. Evidence carries
field=skewSummary/value for {{skewSummary}} interpolation.
Full mirroring chain: schema enum + skewMetric property, shared
manifest + combine.js regeneration, MCP authoring guide + semantic
lint, web SOURCES/form/card, ops-event dual-register, docs. P5 is
subsumed by the tripwire. PercentileMath is now the single nearest-rank
implementation (Apps/PlatformMetrics delegate).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
IntegrityCheckFailed emergency reports used to land in App Insights
only — session e9753578's agent reported a hash mismatch and no product
surface showed it, which cost a day of mis-attributed root-causing.
They now materialize as an AgentBinaryIntegrityMismatch ops event
(Warning, Agent category, dual-registered in the web alert-rule
catalog per feedback_ops_event_types_dual_register).
With suppression for the KNOWN, accepted release race the operator
correctly called out: around every release, agents compare against
hash snapshots that do not belong to their own binary — an older agent
sees the new release's hash on its next config fetch, and a freshly
installed latest agent can receive the PREVIOUS release's hashes from
the 5-minute AdminConfiguration cache (e9753578: genuine 1410 exe
checked against the cached 1409 hash — proven by the 1409 archive
logging "verified OK" with exactly that hash). Both self-heal via the
forced self-update and are suppressed at materialization time against
the CURRENT config: reported running hash == published exe, or
reported version != latest. What remains — an agent claiming the
latest version whose binary is NOT the published one — is the real
signal: tamper, stale blob, or a build outside the release pipeline.
Missing config fails open to recording; visibility beats suppression.
Ops event only, deliberately no timeline event: the report is about
the binary, not the enrollment, and may precede session registration.
Backend suite 4320 green, web tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LastEventAt is the maximum AGENT-supplied event timestamp and therefore lives in
the device clock frame. The silence sweep compared it against a server-derived
cutoff, mixing two frames.
Measured over 11,068 sessions: 26 carry a genuine CMTrace timezone skew on their
IME-derived events, from -17 h to +1 h (a further 27 carry plain device-clock
offsets). At -17 h a live agent looks silent the moment it starts and is marked
Stalled with a customer-visible "Agent silent for 1020min"; at +1 h the sweep is
blind for an hour.
Adds SessionSummary.LastIngestAt, stamped from the server clock in
IncrementSessionEventCountAsync — reaching that method IS contact from the agent,
so it is written unconditionally rather than gated on latestEventTimestamp.
The sweep's server-side filter now spans both frames (LastIngestAt lt X or
LastEventAt lt X): filtering on LastIngestAt alone would silently drop every row
written before the field existed, because Table Storage does not match a filter
against a missing property. The authoritative decision moves into
ServerFrameLastContact() (LastIngestAt ?? LastEventAt ?? StartedAt) and is applied
in both stages — before the Stalled marker and in the stage-2 alive guard.
Deliberately not mirrored into SessionsIndex: the sweep reads Sessions, and the
index manifest is pinned bidirectionally by tests.
shared-manifests.json + .generated.ts regenerated — SharedManifestParityTests is
the drift brake and correctly went red on the new field.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IME 1.104 removed the [DO TEL] log line, leaving the agent's DO collector as
the only do_telemetry source — and its gate only fires when a 3s poll itself
observes the completed download. Fast, interrupted or between-poll-completed
downloads never emit one, so DoDownloadMode stayed -1 and DoAggregator dropped
the whole row: DO bytes read as "practically null" on 1.104 fleets even though
download_progress events carried the full do* set on every poll.
- Shared ApplyDoFields for both writers: download_progress folds byte counters
and file size via Math.Max (monotonic, replay-idempotent) and sets the mode
only when >= 0; do_telemetry stays authoritative (last-write-wins) but never
regresses DoDownloadMode to the -1 unset sentinel.
- DownloadBytes now prefers actually-transferred bytes over doFileSize (the
file size used to inflate the transfer measure).
Intended effect after deploy: DO aggregates count previously-invisible
incomplete downloads, so app counts/bytes rise. No backfill of historic rows.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New GET /api/global/metrics/agent-efficiency (GlobalReadOrAdmin, ?tenantId=):
per-agent-version percentiles (CPU, working set, private bytes, threads,
handles, spool, API latency/requests), crash rates per version, top offender
sessions per dimension, 5-min cache keyed (days, limit, tenant). First
backend consumption of agent_thread_count/agent_handle_count.
- PlatformAgentMetricsResponse.SessionsScanned: sessions the scan actually
covered — the honest basis for the caller's truncated flag (Sessions.Count
only counts sessions WITH snapshots).
- Per-session fetch no longer drains the full Events partition: new
GetSessionEventsByTypesAsync (multi-type OData filter + column projection +
real maxResults enforcement); the 20 delivery-latency sample sessions keep
the full fetch so latency deltas stay computed over all events.
- Shared AgentMetricsAggregation (crash tally, payload getters, bounded
fan-out, metrics event types) so the two services cannot drift.
- GetAllSessionsPageAsync gains an optional select projection (default null,
existing callers unchanged), pinned by an equivalence test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session f475e697: a device re-enrolled without a wipe (IME log residue from a
previous enrollment) completes its pre-installed apps as instant detections,
pulling the phase anchors ~6.5 min ahead of the real ESP page — the breakdown
was mechanically exact but semantically wrong, and entered fleet stats as clean.
- New duration-critical flag PriorEnrollmentResidue, set on
historic_ime_replay_detected; excluded from fleet segment stats via the
existing DurationCriticalFlags mask. registry_app_baseline successes are
deliberately NOT a trigger (by-design normal for WDP Batch-1) — pinned by test.
- AttributionVersion 1 -> 2 so the maintenance sweep recomputes in-retention
breakdowns and flags old residue sessions retroactively (rule 8).
- Web: flag label in TimeAttributionLane; docs: quality-flags section.
- Stale comment fix: SaveWhiteGloveSuccessResult is NOT a WhiteGlove indicator
on Win11 25H2+ (validated 2026-08-18 against user-driven sessions) —
observability only, consumed nowhere for classification.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Daily-review finding: the MSI upload is write-once (If-None-Match:*), so a
first attempt that died between the MSI and the provenance upload funneled
every retry into the 409 short-circuit — which returned "Archived" without
ever writing provenance.json (permanently lost). The 409 path now checks for
the sidecar and, when missing, rehashes the ARCHIVED bytes and writes it;
heal failures ride the normal retryable ladder.
Also closes the review's test gaps: ImeMsiArchiveQueueWorker + producer
worker-level tests (internal QueueClient test ctors, SessionDeletionWorker
pattern), CorporateIdentifierValidator null/whitespace + OData special-char
+ page-cap-transient tests, registry observer helper tests
(TerminalOutcome/ChangedFields/ClassifyStatusServiceStatus).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
P0b/P1 of the 2026-08-17 IME code audit (agent release required):
- Tail AppActionProcessor.log (+ rollover glob) in ImeLogTracker and the
--run-ime-matching CLI. IME routes every [Win32App][ActionProcessor]
line there (enforcement decision engine), so IME-NO-ACTION and
IME-NOT-TARGETED-1/2 could never fire before, and GRS skips were
invisible. Live session 0fe2c562 carried 16 no-action and 64
classification lines in that file.
- New patterns (wordings verified in decompiled IME 1.97 AND 1.104):
IME-GRS-SKIP ("is in GRS. The app will not be enforced." -> Postponed;
THE canonical silent non-install reason), IME-GRS-CLEARED (early
subgraph termination, retry next check-in -> Postponed),
IME-TOKEN-FAILURE ("Failed to get AAD token ... errorCode = N").
- New action imeTokenFailure -> ime_token_failure Warning event (new
EventType: Constants + MCP resource-catalog + shared manifests
regenerated). Tracker dedups per distinct errorCode; included in
the historic-replay guard so stale lines from a previous enrollment
cannot emit a misleading warning. Complements IME-TOKEN-SUCCESS:
an auth outage previously left check-ins silently doing nothing.
Tests: adapter emission + unknown-code + tracker dedup; known-actions
catalog extended; all suites green (Functions 4262, Agent 2252, web 762).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
When RecordImeVersionAsync reports a genuinely new IME version (~monthly),
EventIngestProcessor now also enqueues an ime-msi-archive job. The worker
(QueuePollingWorker, heartbeat, pause on ImeMsiArchivingEnabled=false)
downloads the installer and stores it write-once (If-None-Match:*) in the
permanent ime-archive container ({version}/IntuneWindowsAgent.msi +
provenance.json), then merges MsiArchiveStatus/BlobPath/Sha256/Bytes/SourceUrl
onto the ImeVersionHistory row - so every fleet-observed IME build stays
available for decompilation/diffing after Microsoft's versionless CDN moves on.
The download URL comes from agent telemetry (msiDownloadUrl enrichment) and is
treated as untrusted: https-only *.manage.microsoft.com allowlist with exact
filename match, canonical versionless fallback, no redirects, size cap
(MaxImeMsiDownloadSizeMB, default 250, preflight + mid-stream) enforced by a
SHA-256-hashing pass-through stream. Archiver never throws; the worker retries
only transient statuses (download/timeout/error), permanent rejections
complete with their Failed:* status persisted.
Admin settings ImeMsiArchivingEnabled (default true) + MaxImeMsiDownloadSizeMB
via the full 4-file web chain; archive fields on the version-history API are
GA-only (non-GA projection unchanged, redaction tests extended). Shared
manifests regenerated. Tests: archiver status branches via recording subclass
(allowlist, caps, 409 idempotency, cancellation propagation), config defaults.
Backend suite 4262 green, web tsc + vitest 762 green, solution 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PUT rules/ime-log-patterns/{patternId} sat on the TenantAdminOrGA tier
while the handler only checks ?global=true and writes the GLOBAL pattern
with no GA re-gate - any customer tenant admin could mutate platform-wide
patterns via direct API call. The UI gate (isGlobalAdmin && globalAdminMode)
was cosmetic only.
Moved the entry to GlobalAdminOnly next to the reseed route and added a
regression test pinning both IME pattern write routes to that tier.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Field-verified against the portal list: Intune normalizes corporate
identifiers on upload - uppercased, all non-alphanumerics stripped per
component ("Microsoft Corporation,Virtual Machine,7801-5131-..." is stored
as "MICROSOFTCORPORATION,VIRTUALMACHINE,78015131...") and the device
matched + enrolled against that normalized entry. Our exact-match against
the raw device values (and the contains() filter using the dashed serial)
could therefore never hit.
Both sides now normalize before comparison, and the contains() narrowing
uses the normalized serial. Raw-form entries created directly via Graph
still match. The not-found message shows raw AND normalized form so
rejection logs line up with the portal list.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The searchExistingIdentities ACTION requires
DeviceManagementServiceConfig.ReadWrite.All, which the app deliberately does
not hold - the feature was broken in every tenant. Listing the same data via
GET works with Read.All, which the app already has for
windowsAutopilotDeviceIdentities. No new permission, no customer re-consent.
- contains(importedDeviceIdentifier,serial) filter for server-side
narrowing (mirrors AutopilotDeviceValidator), exact case-insensitive
match client-side on manufacturerModelSerial identities only
- 400 filter rejection falls back to an unfiltered paged scan
(60 x 1000 covers the portal maximum of 10 CSVs x 5000 rows);
page-cap overrun is transient, never a cached false negative
- Graph 401/403 stays a definitive, negative-cached failure with an
actionable message (now naming Read.All)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two changes from the WDP live-test follow-up:
- Remove the serialNumber-type probe again (decision: only the documented
Windows manufacturerModelSerial identifier authorizes; a serial-type entry
is an Intune-side admin misconfiguration and must not count).
- Classify Graph 401/403 on searchExistingIdentities as DEFINITIVE with an
actionable message naming the missing permission
(DeviceManagementServiceConfig.ReadWrite.All / admin consent) and
negative-cache it. Field case: the missing consent was classified
transient and trapped the agent in an endless 503 Retry-After loop.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Docs research showed the serial-only corporate identifier type is NOT the
documented WDP path (Windows only supports manufacturer,model,serial; Intune
ignores serial-type for Windows ownership). The dual probe stays because the
serial type is the only one the portal offers for MANUAL entry, so admins
land there by mistake and an uploaded serial still expresses authorization
intent. Comments and test docs now state that rationale instead of the
incorrect "WDP guidance uses serial type" claim. No behavior change.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First live WDP enrollment 403d at device validation. Root causes:
- CorporateIdentifierValidator only searched the manufacturerModelSerial
identifier type, but the Device Preparation guidance has admins upload
SERIAL-type corporate identifiers - those never matched. The Graph
searchExistingIdentities call now probes both types in one round-trip.
- SecurityValidator overwrote deviceValidationError per validator, so the
403 Details only showed the LAST failure (CloudPc) and hid the corporate
identifier miss. Errors now aggregate in chain order and the transient
flag is sticky-OR (any transient failure yields 503 Retry-After instead
of a definitive 403).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two low-risk fixes ahead of the first live Device Preparation enrollment:
- StoreSessionAsync: EnrollmentType is now sticky-v2 across re-registration
(same contract as the sticky-true booleans) so a degraded AutopilotSettings
read after a reboot can no longer flip an observed v2 session back to v1.
- EnrollmentScenarioProfileUpdater: ImeUserSessionCompleted,
SelfDeployingDeadlineConfirmed and WhiteGloveSealingConfirmed no longer
overwrite an established DevicePreparation profile (seeded at Medium from
the registry-deterministic v2 fact).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Oliver Kieselbach