Customer follow-up corrected the field-case mechanics: TrackingPoliciesCreated=1
WAS set under the foreign ConfigMgr provider - the ESP ignores a foreign
provider's completion values and waits on the Intune-registered "Sidecar"
provider by name (renaming the key unblocks instantly without creating any
tracking entries). The completeness-keyed detector would have missed exactly
the incident it was built for.
- Arm 2: >=1 Setup\Apps provider registered without Sidecar, continuously for
the 15-min dwell -> reason=sidecar_provider_missing; clock resets when
Sidecar registers later (legitimate startup ordering); DevicePreparation-only
snapshots don't activate it
- Arm 1 (continuously-incomplete providers) unchanged -> reason=provider_incomplete;
sidecar-missing wins when both arms cross in one round (single event)
- Cross-restart fingerprint gains a "sidecarMissing" sentinel; payload adds
reason + sidecarMissingForMinutes
- Regression test: ConfigMgr complete + Sidecar absent -> fires (previously
classified healthy); probe/doc headers and OKF doc corrected, web event-type
description updated
Evidence (research session c7e3db08): Intune's ESP-policy WAP payload always
registers Sidecar (ConfigMgr only via co-management settings policy, per
Niehaus' documented payload); Microsoft's ESP troubleshooting models app
tracking exclusively through the Sidecar subkey.
Agent tests 2043/2043, web tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Azure/static-web-apps-deploy@v1 has no `skip_deploy_on_missing_secrets`
input (confirmed against its action.yml), so passing it did nothing but
emit an "Unexpected input(s)" warning — the intended "skip green on
Dependabot PRs" behaviour was never actually in effect.
Replace it with the documented mechanism: surface the deploy token as a
job-level env var and gate the deploy/close step with `if: env.DEPLOY_TOKEN
!= ''` (secrets cannot be referenced directly in an `if:`). On Dependabot
PRs the token is absent, so the step skips and the job stays green; real
deploys on main and internal PRs still run. Applied to both jobs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BuildFleetFailureReasons truncated each reason to a 50-char prefix + "..."
and used that as both the grouping key AND the display value. The full text
never reached the UI, so its tooltip/tap-to-expand had nothing to reveal and
near-identical rows were indistinguishable past the cut.
Group by the 50-char prefix as before (variants still collapse into one row),
but carry the longest full reason as the display value. The UI truncates and
expands it on demand, so the complete text is now available end-to-end.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The prior ">=22" made Oryx build with the highest available Node (24.13),
which the Static Web Apps runtime rejects: "Node version 24.13.0 is not
supported. Please use one of the following versions 18, 20, 22." Pin to
22.x so Oryx selects the SWA-supported major. CI setup-node '22' already
resolves to latest 22.x, so build and deploy now agree.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Node 20 reached EOL (2026-04-30); Node 22 is the current maintenance LTS
and already the runtime the MCP container ships (node:22-slim). Straighten
the version skew so per-component CI major == deploy runtime == engines floor:
- ci.yml: web, mcp and guardrails setup-node 20 -> 22
- combine-rules.yml: setup-node 20 -> 22
- mcp package.json: add engines.node ">=22" (matches Dockerfile)
- web package.json: add engines.node ">=22" (first explicit pin; SWA/Oryx
no longer picks a Node version implicitly)
engine-strict stays off (no .npmrc), so the pins are advisory and cannot
hard-break local installs on other Node versions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The expanded state used line-clamp-2, which re-truncated long values with an
ellipsis after two lines — unreadable on touch where the hover tooltip is
unavailable. Expand now wraps the entire string over as many lines as needed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add reusable TruncatedLabel component: a single-line label that truncates
with an ellipsis, shows the full text as a native hover tooltip, and on
click/Enter/Space expands to two wrapped lines (touch-friendly). An
interactive={false} mode provides tooltip-only behaviour for use inside
already-clickable rows/links where an inner onClick would hijack navigation.
Apply it consistently to the long, dynamic label columns across:
- fleet-health: failure reasons, model + app ranked lists
- sla: top failing apps, violators failure reason
- geographic-performance: fastest/slowest location cards
- vulnerability exposure panel: CVE sample software
- agent metrics: top exception types
- progress + download/install: app-name labels
- admin software tabs: software names
- session reports (comment/adminNote) + ops events (message): tooltip-only
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Second half of the terminal-only rate convention (follows 1cfc32c9, which
converted the enrollment success rates): app-install failureRate divided
by ALL install rows including "InProgress" ones — installs still running,
or orphaned when a session died mid-install — diluting every failure rate
customers see on the Apps dashboard and Fleet Health.
- New MetricsMath.TerminalFailureRatePct (Failed / (Failed + Succeeded),
0 when nothing finished) shared by both aggregation sites
- MetricsMath.BuildAppMetricsPayload: per-app failureRate terminal-based
(Fleet Health "Top Failing Apps"; subtitle now "% of N finished")
- AppsAnalyticsHelper: list, analytics summary, versionBreakdown,
timeSeries and deviceModelBreakdown rates terminal-based; the
duplicated trend block is deduplicated into ComputeFailureTrend, whose
rates and >= 5-sample gate now run on finished installs per half; the
device-model >= 5 sample floor likewise counts finished installs
- Web: InstallsTab "Avg Failure Rate" card computed over summed finished
installs instead of totalInstalls
Tests: new AppsAnalyticsFailureRateTests (list/trend/analytics/model
cases incl. in-flight dilution and gate regressions) and a terminal-rate
case in MetricsMathAppPayloadTests. Backend 3223/3223, tsc clean,
vitest 602/602.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer feedback: the geographic performance pane showed a poor success
rate for a site mid-rollout because in-progress/pending sessions counted
against the rate. Success rate is an outcome quota — align Geo and Fleet
Health with the SLA convention already used by dashboard stats, usage
snapshots and the fleet rollup: Succeeded / (Succeeded + Failed).
Incomplete (terminal, non-failure) stays excluded.
- Geo: per-location SuccessRate over finished sessions; table badge and
map tooltip render an em dash instead of a red 0% when nothing has
finished yet
- Fleet Health: Stats.SuccessRate terminal-based; card shows "X of Y
finished enrollments" and a neutral em dash state; per-model health
gets a Failed count (additive wire field) so the client rate excludes
in-flight devices; TopFailingModels FailureRate now failed/finished
- Tests: FleetHealthPayloadTests flipped to the new convention plus
zero-finished, model-failed and in-flight-dilution cases; new
GeographicMetricsSuccessRateTests pin the geo semantics
Backend 3216/3216, web tsc clean, vitest 602/602.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
CI run 30047512186 failed Worker_drops_envelope_missing_required_fields_
without_invoking_handler: RunForAsync starts the BackgroundService poll loop
and waits a fixed 500ms wall-clock window — on a loaded CI runner (xUnit runs
test classes in parallel) the loop may not have processed the message before
the positive assertions (DeleteMessageAsync at least once) run. All ten
positive tests in the file shared that latent race; this run it hit the
malformed-envelope drop test.
Fix mirrors the watchdog event-gate de-flake: new Harness.RunUntilAsync
(condition, 10s ceiling, 25ms poll) runs the worker until the awaited side
effect is observable, then stops. Gates: ops-event recorded (last step of the
poison path, so send+delete+CAS are already done), main-queue delete for the
drop tests, >=3 visibility heartbeats for the heartbeat test. The kill-switch
test keeps the fixed window — a Times.Never expectation has no event to wait
for. OpsEventCalls capture is now lock-guarded so the condition poll can read
it while the worker thread appends.
16/16 worker tests green across 4 consecutive runs; full backend suite
3211/3211.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Part 2 of the eaf3d8c4 fix (follows 088c3b04). In that session 147 of 157 app
events were replayed from a previous enrollment's IME log — all 10 "installed
apps" were phantoms, polluting the timeline, AppInstallSummaries, MCP counts,
the app_tracking_summary header, culprit lists and final-status.json.
Rename (nothing deployed yet): historic_script_replay_detected →
historic_ime_replay_detected; SuppressIfHistoricReplay / IsHistoricImeReplay.
Agent:
- EmitAppState suppresses stale-replay transitions BEFORE the sub-phase
declaration, timing bookkeeping and terminal-signal dedup — covers the
informational event, the AppInstallCompleted/Failed DecisionSignal, the
shadow download_progress and the summary snapshot. Behavior note: phantom
terminal signals can no longer spuriously re-arm the AdvisoryCompletion
window (app signals are otherwise observation-only — verified no completion
gate reads them). EmitDoTelemetry gets the same guard.
- Tracker-level guard at the HandlePatternMatch choke point skips the 23
app-mutating actions (incl. espphasedetected — a stale AccountSetup line
would make fresh DeviceSetup lines bounce as "backward") for source lines
>24h stale, so replayed apps never enter PackageStates, persistence,
app_tracking_summary, culprit lists or final-status. New internal
UtcNowProvider seam (no ctor change), threshold single-sourced as
ImeLogTracker.HistoricReplayThreshold.
- SimulationMode now bypasses all replay suppression — 088c3b04 had broken
the --replay-log-dir dev tool, which feeds historic lines on purpose.
Backend: IsHistoricImeReplay guard in AggregateAppInstallEvent keeps replayed
events out of AppInstallSummaries (SLA/dashboards/metrics inherit).
Web: generic lib/historicReplay.ts partition (scriptExecutions becomes a
wrapper); InstallProgress hides replayed installs with a muted note;
DownloadProgress filters silently.
MCP: get_session_summary filters replayed events — cleans appInstalls stats,
errorCount (replayed app_install_failed are Error-severity) and keyEvents.
Tests: +13 agent (app suppression, tracker guard), +1 backend, +7 web,
+6 MCP. Agent 2039, backend 3211, web 602, MCP 455 all green; drift test
validates the rename.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Session eaf3d8c4: a previous enrollment's IME logs survived on disk (dated
2026-07-16, 7 days old) and the agent replayed them at startup — 156 phantom
script_completed events in a 7-minute session, each showing ~170h durations
(clock-clamped completion minus RAW stale start), plus an immediateUpload
flood (3x ingress_backpressure) and RemediationScriptCount inflated to 156.
Agent:
- Script events whose source line is >24h stale are suppressed entirely; a
one-shot historic_script_replay_detected Info event (new EventType) dates
the replayed window via earliestRejectedSourceTimestamp. Future-skew
rejections and clock-only fallbacks still emit.
- durationSeconds now uses a timeline-consistent pair: on rejected-timestamp
emissions the raw source completion pairs with the raw start (true runtime
instead of cross-timeline garbage). 24h plausibility backstop omits
implausible values; same consistency for script_timeout_suspected.
- Stale pending platform-script slots (>24h older than the current start
line) are discarded so an old orphaned start can't pair with a fresh run
(extracted HandlePlatformScriptStarted as internal test seam).
Web: partitionHistoricScriptEvents hides legacy-agent replay events (muted
"N historic executions hidden" note) and the reducer blanks >24h durations —
covers sessions recorded before the agent fix rolls out.
Backend: IsHistoricScriptReplay guard keeps replayed events from inflating
Platform-/RemediationScriptCount during the rollout window.
Registered historic_script_replay_detected in Constants.EventTypes, MCP
resource-catalog (script_events) + KEY_EVENT_TYPES, gather-rules eventTypes.
Tests: 8 agent (suppression/one-shot/raw-pair/backstop/slot-hardening),
8 web (partition/blanking), 10 backend (predicate). Agent 2026, backend
3210, web 595, MCP 449 all green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The straggler integration test copied powershell.exe under a synthetic
agent name and launched it — textbook MITRE T1036.003 ("Rename Legitimate
Utilities"). Defender for Endpoint flagged the run ("System executable
renamed and launched", initiating process testhost.net48.exe) and ISOLATED
the dev machine.
Replaced with function shadowing: a harness script defines Get-Process /
Stop-Process stub functions (PowerShell resolves functions before cmdlets)
and dot-sources the cleanup script. The stubs report a live agent-named
straggler on every probe and record the kill, pinning the identical
contract — poll by NAME until the deadline, force-kill by NAME, then
delete — with zero process spawns, zero PE copies, nothing EDR-visible.
The production routine never renames or copies executables; the flagged
technique existed only in this test. Lesson recorded in tasks/lessons.md.
22/22 cleanup tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DeviceSetup/Apps failed, the user pressed "Try again" ~23 min later, apps
recovered to 10/10 and DeviceSetup resolved - yet the advisory_completion
window un-defanged the stale failure and failed the live enrollment,
because the advisory variant was exempt from all re-arm/rebase guards.
Engine fix (DecisionCore):
- New facts EspAdvisoryFailureResolvedUtc + EspAdvisoryFailureCategory
(state-borne; deadline payload would be lost on re-arm/rebase).
- TryResolveAdvisoryOnCategoryRecovery in both ProvisioningComplete
handlers: category match sets the fact + emits
esp_failure_advisory_resolved.
- Guards now (!hasAdvisoryAnchor || advisoryResolved): enforcement-progress
re-arm, hello-never-observed promote and the reboot rebase apply again
once the failure demonstrably un-happened; unresolved advisories keep
the legacy un-defang exactly.
- Truthful terminal reason esp_recovered_without_completion_evidence
(+ failureRecovered=true, LastFailureTrigger=DeadlineFired so the
likely-stuck app promotion stays off).
Story events (agent tracker):
- esp_failure_retry_detected: failed subcategory leaves "failed" after the
terminal fire - consistent with the user pressing "Try again".
- esp_failure_recovered: previously failed category reaches success.
- EspFailureDetectedEventArgs.LikelyCulpritApps (never-started first,
cap 3) rides the EspTerminalFailure payload so esp_failure_advisory /
enrollment_failed name the culprit app directly.
Surfacing: MCP resource-catalog, web timeline badges (Recovered /
User retry pills, culprit chip on advisory), gather-rule autocomplete,
ANALYZE-ESP-006 "ESP Failure Recovered After User Retry".
Tests: AdvisoryRecoveryTests (13, incl. the 527/611/635/671 field-ordinal
replay that no longer fails the session), EspFailureRecoveryStoryTests (5),
snapshot-builder guards 18 -> 20. All suites green (DecisionCore 493,
V2.Core 2018, Backend 3201, web vitest 589, MCP 449 incl. drift).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer field case: pre-provisioned devices with the ConfigMgr client as a
blocking app occasionally park the user ESP at "Apps (Identifying)" for days.
Root cause (verified against the EnrollmentStatusTracking CSP doc): only a
"ConfigMgr" provider is registered under Device\Setup\Apps\PolicyProviders
(no "Sidecar" = IME) and TrackingPoliciesCreated never becomes 1 - a wait the
CSP defines WITHOUT any timeout. Renaming the key to Sidecar unblocks
immediately. Microsoft documents that ESP providers are not aware of each
other and that pre-provisioning is unsupported with co-management.
New always-on esp_policy_provider_stalled tripwire, generalized to any
provider in both kinds (Setup/Apps TrackingPoliciesCreated != 1,
DevicePreparation InstallationState not in {2,3}):
- EspPolicyProviderProbe: fail-soft registry read of the
EnrollmentStatusTracking mirror (device + user-SID scopes), defensive
value parsing (missing/unparseable = incomplete), ScopedOverride test seam.
Research and CSP contract quotes captured in the XML doc header.
- EspPolicyProviderStallDetector: pure EvaluateStallTransition on a 15-min
continuous-incompleteness dwell (resets on completion/disappearance;
absent key = normal early enrollment), one-shot per provider,
StartupEventGate fingerprint over the sorted stalled key set for
cross-restart dedup. Warning + ImmediateUpload via InformationalEventPost;
purely observational, no decision-engine coupling, no config toggle,
no ConfigVersion bump (disk_space_low precedent).
- EspPolicyProviderStallHost: 60-s wall-clock tick, deliberately outside the
StallProbe idle gate (idle clock resets on session activity) and the
idle-stopped periodic collectors.
- Registered in Constants.EventTypes, MCP resource-catalog (health_events),
web gather-rule autocomplete (stall category).
Guardrail companion: SOFTWARE\Microsoft\Windows\Autopilot joins the
gather-rule registry allowlist ("Autopilot / OOBE / Setup") so tenants can
build custom rules on the subtree (e.g. not_exists on the Sidecar provider
key); generated web mirror regenerated via combine.js; positive + sibling-
spoof guard tests added.
Docs: OKF concept doc agent/esp-policy-provider-stall.md (+ index/log).
Tests: 22 new detector tests (pure transition matrix + emission surface incl.
the ConfigMgr-only customer scenario and gate-across-restart dedup); agent
suite 2013/2013, web vitest 589/589 + tsc clean, MCP 449 passed incl.
event-types-drift.
Both the detector and the embedded allowlist take effect with the next agent
release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer report (session fcb44595): dashboard said 115 min, session detail
said 5h 2m for the same pre-provisioned enrollment. The detail's per-block
duration calc took min/max over ALL event timestamps inside each
sequence-block, which two real-world timestamp shapes break:
- an app_install_started re-emitted by the ImeLogTracker after the Part-2
resume legitimately carries its historical Part-1 event time -> the User
Enrollment block stretched back across the sealed pause (56m -> 3h18m)
- a do_telemetry with a +1h source timestamp (IME log line with a wrong
timezone bias, just under the agent's 1h future-skew clamp) stretched
Pre-Provisioning (58m -> 1h44m)
Sum: exactly the "5h 2m" the customer saw.
Block spans are now anchored on the first/last event BY SEQUENCE (sequence
is the canonical order), with Part 1 starting at session.startedAt for
backend DurationSeconds parity. Extracted as pure helper
computeWhiteGloveDurations with tests modeled on the real session; replay
against the customer's 880 events yields 1h 54m == backend 6872s.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer report: ANALYZE-SEC-001 warned on devices whose firmware already
carried the Windows UEFI CA 2023 certificate — the agent only read the
SecureBoot\Servicing registry value, which mirrors the Windows Update
rollout and lags behind reality (OEM factory image, WinCS, WU not run yet).
Agent: new UefiSecureBootCertReader reads the UEFI db/KEK variables
natively (GetFirmwareEnvironmentVariableExW; SeSystemEnvironmentPrivilege
enabled on the process token and restored — first privilege-adjustment
code in the repo). Fail-soft, never throws; legacy BIOS maps to not_uefi.
secureboot_status gains per-certificate booleans, uefiFirmwareReadStatus,
and the one-sided marker uefiCa2023FirmwareConfirmed (emitted only on
positive confirmation, never as false).
Rule v3.0.0: not_exists precondition on the marker suppresses the rule
when the firmware confirms the certificate; the marker's absence keeps
old agents on exact v2 behavior, so the rule is safe to seed before the
agent rollout. Firmware-verified-missing raises confidence (+20 db,
+10 KEK 2K CA 2023).
Verified: full solution build; backend 3201, agent 1988, web 582 tests
green; tsc/ajv clean; elevated cross-check on real firmware matches
Get-SecureBootUEFI on all seven certificate probes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sample Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState
(build >= 17763, SYSTEM-callable, CLR-native WinRT projection — no NuGet)
as a purely observational signal, mirroring the bootstrap guard marker:
- oobeStateAtAgentStart on agent_started and agent_late_start payloads:
makes late starts provable instead of heuristic ("completed" on a
first_run = OOBE was already over when the agent arrived) and validates
the bootstrap OOBE guard fleet-wide.
- One-shot oobe_state_completed trace event on the in_progress->completed
flip, piggybacked on the DesktopArrivalDetector 30s poll: an
owner-independent desktop corroboration for DAD-starved sessions
(4d5a0b78 class — WTS+WMI owner resolution dead, desktop_arrived never
fires). Often absent in healthy sessions since the DAD timer is
disposed at desktop arrival, which is when the flip lands.
OobeStateReader is fail-soft (never throws, "unavailable" on missing
contract); the flip check has its own try/catch so it cannot pollute the
DAD wmiErrorCount liveness diagnostics, "unavailable" never overwrites
the baseline, and the one-shot flag survives ResetForRealUserSwitch.
No decision-engine involvement, no config/backend changes. Event type
registered in Constants.EventTypes, MCP resource-catalog (phase_events)
and web gather-rule autocomplete.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six integration tests execute the generated cleanup script with a real
powershell.exe against GUID-unique scratch trees — proving what the string
pins cannot:
- locked tree ⇒ complete abort, marker + binaries survive (the 2026-07-22
regression: the old script force-deleted around the locks)
- full incident replay: a renamed second process holds a file in the tree,
is found by NAME, force-killed after the wait deadline, cleanup then
completes with nothing left behind
- fresh foreign lock ⇒ loser backs off without touching the tree; stale
lock (>10min) is taken over and cleanup proceeds
- happy path + keepLogs variant
BuildCleanupScript gains test-only timing parameters (defaults unchanged
for production) so the replays run in seconds. Agent suite 1969/1969.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Incident 2026-07-22: the 48h emergency break left exe + loaded DLLs orphaned
forever — a second agent instance held the images while the cleanup script
(waiting on the launcher PID only) force-deleted marker + state around the
locks AFTER removing the Scheduled Task, destroying the retry path.
Agent — cleanup script (LIFE-F7):
- wait for ALL agent processes by name (PID-reuse-proof), then force-kill
stragglers; single-flight lock file serializes concurrent cleanup scripts
- probe-rename (GUID target) BEFORE task removal; any failure renames back
and aborts with task + marker + files intact for a next-boot retry; the
destructive Remove-Item-on-original fallback is gone
- script generation extracted to BuildCleanupScript, pinned by 13 tests;
all 4 variants parser-validated + E2E-smoked (happy path, locked tree)
Agent — break report delivery:
- EmergencyReporter counts DELIVERED reports: reservation rolls back on
failure (mark-before-send permanently suppressed a key after its first
transport failure); optional in-reservation retries
- break path: NIC wait <=15s, 3 attempts x 15s timeout (survives Functions
cold starts observed as >5s 503s), 75s overall bound
Backend:
- backend-materialized events (agent_emergency_break, session_timeout) now
upsert the cross-session EventType index — they were invisible to every
search-by-eventType surface (portal + MCP)
- new AgentEmergencyBreak ops event (Agent/Warning, once per session) so
operators see silently-lost agents; dual-registered in OPS_EVENT_TYPES
- materialization extracted to a static seam, covered by 4 new tests
Agent 1963/1963, Backend 3198/3198, tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Allowed Local Accounts entries may now contain glob wildcards (* / ?),
matching the hardware-whitelist semantics: Regex.Escape conversion,
case-insensitive, 100ms ReDoS timeout. Applies to both the account and
the profile-folder check. Entries without wildcard chars keep exact
matching; * and ? are invalid in Windows account names, so patterns and
literals cannot collide. UI documents the syntax and rejects patterns
with fewer than 2 literal characters (a bare * would disable the
analyzer). No schema change - old agents treat patterns as literals and
keep alerting (fail-safe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first release after the download-alias migration exposed Front Door
caching as a correctness hazard: the alias kept serving the previous
version.json (X-Cache: TCP_REMOTE_HIT) after the origin had rotated to
2.0.1370, pinning the portal's version badge to the old build. The dangerous
variant is worse than a stale badge -- manifest and ZIP are independent cache
entries, so a stale manifest paired with a fresh ZIP fails the bootstrap's
SHA-256 integrity check and the install dies.
Route caching on /agent/* has been disabled (infra side); this commit is the
defense-in-depth half: every upload now carries x-ms-blob-cache-control
no-cache -- the CI Send-Blob (eu and legacy mirror alike), the local build.ps1
counterpart (gitignored, changed in lockstep), and the versions/*.json
manifest publishes. Uniform no-cache rather than an immutable/mutable split
because every blob in the agent container rotates in place, including the
per-line AutopilotMonitor-Agent-v2.zip -- nothing there is safely cacheable.
Existing blobs get the header with the next release's re-upload.
Log/docs updated in url-registry.md; no code paths changed, YAML and ps1
parse-checked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An interactive logon during the RealmJoin first deployment (phase 100/101)
makes RJ reclassify the run as a secondary-user deployment, whose branch
writes 200/210 and never CompletedFirstDeployment (110) — no crash, no
reboot request. The V2 completion gate waited for exactly 110, so these
sessions hung until the 60-min hard timeout (session 224b2087; siblings
558bf067/3cc18724 delayed ~50 min the same way).
- promote realmjoin_phase_changed to a typed RealmJoinPhaseChanged signal
(dual-emitted by the adapter) so RealmJoinFacts.LastDeploymentPhase is
persisted across agent restarts
- abort rule: prev in {100,101} and current in {200,210} with no outcome
yet -> emit realmjoin_first_deployment_incomplete (Warning), cancel the
hard-timeout deadline, release the gate with outcome
FirstDeploymentIncomplete; also evaluated on the RealmJoinDetected
replay path (restart swallows the transition)
- deliberately NOT triggered by 101->0 (RJ service restart may still
complete the first deployment) nor by a first observation of 210
(session 6f1959c0: RJ already completed before agent boot)
- fix realmjoin_timeout always claiming "last phase: 0" - the persisted
phase now feeds the message
- backend EnrollmentTimeoutClassifier counts the new event as a gate
terminal; MCP resource catalog dual-synced
- CompleteIfDeferredOrBookkeep widened to a leading-effect list
Verified by replaying session 224b2087's signal log: gate releases at
15:05:17 (101->200) instead of hanging until the 15:54 deadline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every well-known host -- own subdomains, both blob accounts, the API origin,
Graph, Entra login -- now lives in exactly one registry per component:
Constants.cs (*BaseUrl) for C#, utils/config.ts for the web app, src/config.ts
for the MCP server. lib/hostRouting.ts derives its hostnames from the registry
and middleware.ts imports from it, retiring what turned out to be a fourth
independent copy of the www/portal host pair.
Three new guard tests (backend xUnit, web vitest, MCP vitest) scan non-test
source and fail on any enforced host outside its registry. This is the lesson
of the EU cutover: the registry existed then too, but LatestVersionsService
(feeds the portal's version badge) and ValidateBootstrapCodeFunction carried
their own literal copies of the legacy blob host and were missed -- kept
working only by the fail-soft legacy mirror in build-agent.yml, whose silent
failure would have pinned badge and self-update to stale bits with a green
release. Both guards proved themselves on their first run, flagging the
middleware host copy and a literal in an MCP tool description. Test files stay
exempt on purpose: literals there are independent oracles, and
ValidateBootstrapCodeResponseShapeTests now pins the full alias URL against
the Constants-built one so registry changes must be acknowledged consciously.
The two legacy readers plus the agent SelfUpdater now use
Constants.AgentDownloadBaseUrl (download.autopilotmonitor.com/agent); the
alias was verified live for version.json, version-v2.json and both ZIPs
before the switch. The portal's bootstrap allow-list accepts the alias AND
the legacy host (AGENT_DOWNLOAD_HOSTNAMES) for already-deployed customer
scripts, which imposes a deploy order: web before or with backend. The
legacy keepalive probes and the release mirror stay untouched -- they exist
for exactly this transition. Also: the bootstrap short-link now targets www
directly instead of riding the registrar's apex 301, and the welcome-mail
docs button points at docs. instead of bouncing through www./docs.
Suites: backend 3194, agent core 1925, DecisionCore 474, web 582, MCP 449 --
all green; next build verified (it, not tsc, proves the next.config.ts
registry import). Docs: docs/url-registry.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three components now report <major>.<minor>.<build> plus a 7-char commit
and a build timestamp. The backend had been pinned at a hand-written 1.0.0
since inception and starts at 1.5.1000; the MCP server's package.json version
was bumped by hand and starts at 1.5.300, and it now reports its own commit on
/health instead of only the documentation bundle's.
The build number keeps coming from a counter blob reserved before the build
with ETag-CAS, but there is now one implementation instead of two copies:
.github/scripts/Request-BuildNumber.ps1, called by all three workflows and by
the local build.ps1. It lives under .github/ because scripts/Deployment/ is
gitignored (those scripts carry SAS tokens) and would therefore be absent from
a CI checkout; scripts/Build/ and scripts/Release/ are shadowed by the
[Bb]uild/ and [Rr]elease/ artifact ignores.
The counter cannot be replaced by the published manifest, which is the obvious
simplification to reach for: version.json is written only on stable publish, so
two consecutive dev builds would derive the same number and the second would
overwrite the first's artifact. git rev-list --count fails the same way, two
builds of one commit collide.
Both deploy workflows now poll their health endpoint until it reports exactly
the version and commit just deployed, and publish versions/<component>.json
only then. A green deploy step alone means the package was accepted, not that
it is being served -- so a manifest that disagrees with the live endpoint now
means the running instance drifted (rollback, stale revision) rather than a
pipeline that stopped halfway. The agent manifest gains the same commit and
buildUtc fields additively; the bootstrap reads it by field name, so older
installers ignore them, and sha256/bootstrapVersion stay untouched.
Requires the versions container plus AZURE_BLOB_VERSIONS_SAS_TOKEN and the two
seeded counters (backend 999, mcp 299) before the first deploy run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GetExcessiveDataSendersAsync gated on LastEventAt - StartedAt alone. No event
count, no bytes, no rate — the name asserted something the code never checked.
What it actually selected was every enrollment session that spanned a night,
which for Autopilot is the ordinary case: device unboxed in the evening, ESP
runs, machine powered off, user signs in the next morning.
The per-device alerts from the previous commit made this visible. All three
devices blocked at the time of the analysis were false positives:
CND3320949 / 64ed21e8 23 events over 15.5h (1.5 events/hour)
5CG4303CPT / db5751bc 192 events over 16.8h (blocked mid Account Setup 1/5)
PF5YANFB / 2069de87 781 events over 31.5h
14 such blocks in 30 days across 9 customer tenants. Over the same window the
event-count detector fired 5 times and was right every time (2742, 2714, 2712,
2506 events). Each false block costs a live enrollment 8 hours of telemetry.
Repairing it rather than removing it does not pay: once a volume condition is
added, "many events AND long span" is a subset of "many events", which is what
the count detector already does. Volume is the quantity that costs anything; a
23-event session spanning 15 hours costs nothing. So the detector, its two admin
settings, the ExcessiveDataBlocked ops event and the portal section go, and the
event-count path is the only automatic block left.
MaxSessionWindowHours and MaintenanceBlockDurationHours are deleted rather than
deprecated: the config row is written with Replace, so the columns drop out on
the next portal save, and nothing reads them meanwhile. A comment on the
surviving detector records why there is no second one, so it does not get
rebuilt a year from now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two runaway detectors emitted alerts of unequal usefulness. The event-count
path fires per session and carries sessionId + serialNumber, so the Ops Events
detail modal can offer View session / Block / Kill. The time-window path
aggregated a whole tenant into one event with {devicesBlocked, windowHours} —
no device, no session, no shortcuts, on precisely the alert where an operator
wants to act. It now writes one event per blocked device with the same payload
shape; the aggregate count stays in the audit entry, where a sum is the useful
view.
The Active Blocks table showed the serial as plain text although the API already
returns BlockedSessionIds. Maintenance blocks now link the serial to the session
that triggered them; manual whole-device blocks have no session and stay text.
Two defects in the event-count path, found while comparing the detectors:
- No isBlocked guard, so the auto-action re-blocked an already-blocked device and
overwrote BlockedAt/UnblockAt/Reason of the existing block — usually the
time-window block on the same session, whose reason then vanished from the list.
- No status filter, so a session that finished days ago with a high EventCount
still cost its device a 24h block. The fix needed a layer below the obvious one:
the runaway query never selected Status, and SessionStatus.InProgress is
ordinal 0, so a guard on the unpopulated field would have been silently
always-true. ParseStatusForAutoAction is stricter than MapToSessionSummary --
Enum.TryParse maps "0" onto InProgress and ORs comma-separated names into an
unrelated member, both of which would open the gate.
Not addressed, and deliberately: a block is session-scoped and stops the upload,
so whichever detector fires first freezes the other for that session. With
MaxSessionWindowHours=12 the time detector wins in practice. That is correct
behaviour, not a defect; merging the detectors would be its own change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CodeQL analyzes C# and TypeScript, so a commit touching only docs or markdown
cannot change its result - it spent ~10 minutes (8:50 for C#, 1:32 for
TypeScript) re-reporting an unchanged outcome.
paths-ignore on push and pull_request only. The weekly cron stays unfiltered:
it exists to catch advisories published against code that did not change, and
it doubles as the backstop should the filter ever skip too much.
A commit touching docs AND code still runs - GitHub skips only when every
changed path matches the filter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The header claimed it "runs the full test suite" - it also typechecks the web
and MCP projects and fails on drift between rules/guardrails.json and the
generated rule files. It also called itself "the gate that decides whether a
change is allowed to be correct", which was both garbled and false: main is
deliberately unprotected, so a red run reports rather than blocks.
Comment only, no behaviour change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CI test job surfaced 10 annotations, but GitHub caps them at 10 per job —
a full rebuild showed 31 nullable warnings plus 26 xUnit analyzer warnings.
Production code:
- CertificateValidator: X509Certificate2(byte[]) is obsolete (SYSLIB0057) ->
X509CertificateLoader.LoadCertificate.
- AuthenticationMiddleware: IsValidTenantId now carries [NotNullWhen(true)], so
the guard above BuildTenantAuthority is visible to flow analysis.
- AgentEndpointMigrationRules: the out params are genuinely nullable on the
false path; netstandard2.0 has no usable [NotNullWhen], so the contract is
documented instead.
- Constants.ClassifyEspAppsFailure: errorCode is optional by design (the
"no HRESULT" branch is the timeout fallback) - signature says so now.
Tests: nullable annotations on helper signatures, Assert.Single/DoesNotContain
overloads instead of pre-filtering with Where, and [InlineData(null)] theories
declare nullable parameters. GatherRule helpers now seed Parameters with the
model default (empty) rather than null, matching a deserialized rule.
No behaviour change. 5616 tests pass; solution builds warning-free apart from
the two pre-existing NU1510 pruning notices on the pinned security packages.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The integrations list named start, success, failure and SLA alerts but omitted
hardware rejection, understating what tenants can actually subscribe to.
Verified before claiming it: WebhookNotifyOnHardwareRejection is wired
end-to-end - model, portal toggle in SectionHardwareWhitelist, and dispatch in
ReportDistressFunction. This list occurs nowhere else in the repo or in
autopilotmonitor-docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The guard functions had unit tests, but nothing proved that Registry, Wmi,
Json, Xml and File actually consult them before touching the registry, WMI or
disk. These drive the collectors themselves: each blocked target must come
back empty, emit exactly one security_warning naming the target, and produce
no rule output. Prefix-spoofing included, plus a positive case through
JsonCollector so it is pinned that the guard is a gate and not a wall.
Path traversal had no explicit coverage either. The sharpest case is
...\Logs\..\..\..\..\Secret.txt - the RAW string starts with an allowed
prefix, so a naive StartsWith would admit it; the test pins that
Path.GetFullPath canonicalizes first and the allowlist sees C:\Secret.txt.
Also traversal into the C:\Users privacy block and into System32\config, both
asserted under unrestrictedMode, which lifts the allowlist but never those.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sanitizer matched \p{Cc}, which covers the C0/C1 control ranges but not
U+2028 LINE SEPARATOR or U+2029 PARAGRAPH SEPARATOR - those are categories
Zl/Zp. A log viewer, a JSON-lines consumer or any JavaScript-side reader still
breaks a line on them, so they were a residual log-forging vector.
Matching \p{Zl} and \p{Zp} rather than listing escapes: the two categories
contain exactly those two characters, and it keeps the source free of
invisible literals.
Verified the test is not vacuous - the old pattern left "a<U+2028>b" as "a b",
the new one yields "ab".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tenant contact address had no server-side validation, so a direct API
caller could store any string in the field service notices are sent to.
UpdateTenantConfiguration now rejects non-addresses, recipient lists (a comma
would silently widen who receives those notices), display-name forms and
control characters (mail-header injection), and normalizes what it stores.
Empty stays valid - having no contact address is a legitimate state.
The seed path had a worse problem than missing validation. It was a
read-modify-write of the whole model through SaveTenantConfigurationAsync,
which upserts with TableUpdateMode.Replace and no ETag: a concurrent portal
save was clobbered by the seeder's stale snapshot of EVERY field, not just
ContactEmail. It also wrote straight through IConfigRepository, bypassing the
TenantConfigurationService cache, so a seeded address stayed invisible for the
full 5-minute TTL.
TrySeedTenantContactEmailAsync now writes ETag-conditionally in Merge mode -
one property, and a 412 means someone else wrote the row so the seed abandons
rather than overwriting the winner. A lost seed is recoverable on the next
maintenance run; a lost admin edit is not.
The same seed logic existed twice (preview notification-email save and the
maintenance backfill). Both now call a single owner on
TenantConfigurationService, which holds the never-overwrite invariant, the
cache eviction and the fail-soft guarantee. Fail-soft lives in the service
rather than the repository so it holds for any implementation.
Tests cover the validator, the service contract, and the repository against a
mocked TableClient: Merge mode, the ETag actually passed, only ContactEmail in
the patch, the 412 race, 404, storage faults. Plus a regression pin that
SaveTenantConfigurationAsync is never called from the seed path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to 479247cd. Publishing only a genuine UPN closed the fleet-wide bucket
collapse but opened a narrower hole: AuthenticationMiddleware validates issuer,
audience, lifetime and signature but requires neither upn nor scp, so an app-only
(client-credentials) token authenticates and reaches every AuthenticatedUser route
(feedback, realtime/negotiate, progress/*, health/detailed, auth/mcp, ...). Such a
token carries no upn, so it resolved to an empty identity and UserRateLimitMiddleware
skipped it entirely — previously it at least landed in the shared anonymous bucket.
Rather than reinstating a placeholder as a bucket key, separate the two concepts:
- RequestContext.UserPrincipalName keeps its meaning — a real UPN or nothing. Audit,
presence and admin flows continue to see only genuine user identities.
- RequestContext.CallerId is the throttle identity: UPN, else the token subject
(oid -> sub -> appid/azp), else a fail-closed shared bucket. Empty means exactly one
thing — no JWT at all (device/anonymous routes, which carry their own limits).
- UserRateLimitMiddleware gates and buckets on CallerId.
This matches how McpQuotaEnforcementMiddleware already keys its quota (oid), so the
subject-based identity is the existing house convention for enforcement.
10 new tests: the oid/sub/appid/azp fallback chain and its precedence, the fail-closed
bucket for a token with no identifying claim (asserting it is never "anonymous"), and
that empty CallerId occurs only on unauthenticated routes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Device and anonymous routes published the "anonymous" placeholder as
RequestContext.UserPrincipalName. UserRateLimitMiddleware skips only on an EMPTY UPN, so
every agent of every tenant was bucketed under user_ratelimit_anonymous and throttled
against the 120/min per-user limit. On 2026-07-21 21:00-21:30 UTC that rejected 1821
agent requests with 429 (/api/agent/telemetry, config, register-session, error); the
7-day trend shows it recurring and growing with fleet size.
- PolicyEnforcementMiddleware: RequestContext.UserPrincipalName carries a genuine JWT
UPN or nothing. decision.UserIdentifier keeps the placeholder for log lines.
- Device/bootstrap routes already hold their own limit in SecurityValidator (per cert
thumbprint / per bootstrap token), so they lose nothing.
- stats/platform and diagnostics/download had NO limiter of their own and were riding on
the accidental shared bucket. Both now limit per trusted client IP (60/min and 30/min)
via ClientIpExtractor, matching the HealthCheck pattern.
- 13 new tests pin the empty-UPN contract across all 12 device/anonymous routes, plus a
guard that an authenticated caller still surfaces its real UPN.
Solution suite 5552 tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Microsoft.Identity.Web 4.3.0 pulls Microsoft.AspNetCore.DataProtection and
System.Security.Cryptography.Xml transitively at 10.0.0, which carry known
advisories (GHSA-9mv3-2cwr-p262 critical on DataProtection; GHSA-37gx-xxp4-5rgx
and siblings, high, on Cryptography.Xml). Add explicit references pinning both
to the patched 10.0.10 servicing build (matching the existing JwtBearer pin) —
the NuGet equivalent of the npm overrides used for the web/MCP alerts.
Pinned in the production Functions project so the fix flows to the deployed
artifact and to Functions.Tests via project reference. `dotnet list package
--vulnerable` now reports 0 across every project; build clean; 3110 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bucket B (real fixes) + Bucket A (config) of the CodeQL alert triage.
Log-forging (cs/log-forging, 8 sinks): add Helpers/LogSanitizer that strips
CR/LF and other control characters, and apply it to the logged copy of every
user-controlled value in AuthenticationMiddleware, PolicyEnforcementMiddleware
(Invoke + DecideAsync), McpQuotaEnforcementMiddleware and TableUserUsageRepository.
Only the logged value is sanitized — routing, authorization and storage keys are
untouched, so tracing fidelity for legitimate values (paths, UPNs, GUIDs) is
preserved. Structured logging already isolated the fields; this closes the CRLF
vector as defense-in-depth and clears the query.
Three genuine JS/TS fixes:
- guardValidation.ts: replace the no-op `.replace(/%/g, "%")` with a real
escapeRegExp() — the author intended to escape; this also closes the latent
metacharacter gap (js/identity-replacement).
- access-guard.ts: use a `%s` format argument instead of interpolating the UPN
into the console.error format string (js/tainted-format-string).
- docs-corpus.ts: strip HTML tags with a loop-until-stable stripHtmlTags() so the
sanitizer is robust and recognized as complete (js/incomplete-multi-char-sanitization).
CodeQL config: filter the two queries that fire structurally on this codebase,
where the flagged behavior is the feature — cs/exposure-of-sensitive-information
(audit/notification logs legitimately record the actor UPN / recipient email) and
cs/cleartext-storage-of-sensitive-information (rate limiters key on the trusted
client IP). Default suites stay active; genuine one-off false positives from other
queries are dismissed individually so those queries stay sharp.
Tests: Backend xUnit 3110 (+9 LogSanitizer), MCP vitest 19 (+1 tag-strip),
Web vitest 49, all tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The three watchdog tests raced a fixed fanout-body sleep against sub-second
warn/severe thresholds (e.g. 150ms body vs 500ms severe). On a loaded CI
runner the whole RunCoreAsync body overran the severe threshold before the
finally-block cancelled the watchdog, so the severe OpsEvent fired and
"LongRunning past warning" flaked (3100/3101).
Replace the wall-clock race with event gates: the fanout body now blocks on
WaitForEventAsync(<watchdog OpsEvent>) instead of sleeping, so the target
watchdog is guaranteed to have fired before the body returns. The opposite
threshold is pushed to minutes (warn-only tests) or the body waits on the
severe event (both-fire test) so warn is necessarily first. OpsEvents
recording is now lock-guarded since watchdog tasks record concurrently.
No production change. Watchdog class ran 5x green (16/16); full suite 3110/3110.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dependabot PRs run without repository secrets, so AZURE_SWA_DEPLOY_TOKEN is
empty and Azure/static-web-apps-deploy exits with an error, painting a red
"Build and Deploy Job" check on every web-touching Dependabot PR. Setting
skip_deploy_on_missing_secrets makes the action skip the deploy (green) when
the token is missing; real pushes to main still carry the token and deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oliver Kieselbach