phase_change/phase_exit/on_event gather triggers (and the phase-scope
gates) were driven by the pre-reduce SignalIngress.SignalPosted stream,
so any raw signal carrying an EspPhase payload fired them before the
decision engine had applied its gates. In session 32312a32 (RealmJoin
tenant) a Phase Start rule on FinalizingSetup fired at the raw ESP-exit
EspPhaseChanged(FinalizingSetup) signal, 7 minutes before the engine's
RealmJoin-gated phase_transition(FinalizingSetup) - and read
HKLM\SOFTWARE\RealmJoin\Custom\BIOS before the RealmJoin package that
creates it had run (exists=false).
GatherRuleExecutorHost now subscribes to the new TimelineEventStream:
EventTimelineEmitter publishes every emitted event's (eventType, phase)
post-reduce, so phase triggers fire exactly when the UI timeline shows
the phase, and on_event sees engine-emitted event types
(enrollment_complete, phase_transition, realmjoin_resolved, ...) that
never existed on the raw signal stream - restoring V1 semantics and the
documented "On Event with enrollment_complete" pattern. Raw-signal
consumers with OS-state semantics (DeviceInfoHost, UserEspKeepAwakeHost,
ProvisioningPackageHost) stay on the raw feed deliberately.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
First production auto-approve (mueller-steinag.ch) activated 64s after signup,
5s before the user saved their notification address on the activation page -
the approval path found no address and silently skipped the welcome mail.
Both racing paths now write-then-read: the notification-email save path checks
approval fresh (bypassing the 30s negative cache) and triggers the send when
activation already happened. A conditional-insert welcome-email-sent marker in
PreviewWhitelist makes the send once-only across both paths; it is consumed
strictly after the address check so it can never be burned without an address.
Revoke clears the marker, the GA resend endpoint sends unconditionally and
consumes it best-effort.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- New "phase" stage: OnPhaseChanged writes a transition marker line
(phase change: Old -> New) so rule-level lines read against the current
phase; only on real changes, written outside _scopeLock.
- LogParser traces each regex match: line number, matched text (<=200 chars)
and capture groups (<=100 chars, same selection as event data), capped at
10 lines per file per run; the per-file summary keeps the full count.
- Invalid-regex error line now includes the offending pattern string.
Tests: 4 new (transition dedup, match trace, cap, pattern in error);
agent suite 2137/2137 green. OKF doc + docs/log.md updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
output:'export' + trailingSlash: the SWA now serves plain static files; the
managed SSR runtime — the portal's last cold-start source ("Backend call
failure" when a cold start exceeded SWA's 45s limit) — no longer exists.
- Detail routes are query-string based (/sessions?id=, /sessions/inspector?id=,
/diagnosis?id=, /apps/detail?name=, /admin/backups/detail?id=,
/admin/customs-archive/detail?tenantId=&rowKey=). Single-definition builders
in lib/routes.ts; all 21 link sites migrated, duplicate sessionHref helpers
consolidated. Suspense boundaries added everywhere useSearchParams drives
page identity (hard export-build errors otherwise).
- 9 [section] families prerender via server shells (generateStaticParams from
the *NavSections registries, dynamicParams=false, client bodies extracted to
SectionClient) — 37 pages; server redirect() index pages became
ClientRedirect shells. robots/sitemap/og-image routes force-static.
- Legacy PATH-shaped deep links are preserved FOREVER (sent Teams/Slack/
webhook notifications cannot be edited): SWA wildcard rewrites serve the
family page and the globally mounted LegacyPathRedirect rewrites to the
canonical query URL, preserving query + #event hashes (mapping pinned by
lib/__tests__/legacyPathRedirect.test.ts).
- middleware.ts deleted → client-side HostRoutingGuard (prefix list moved to
lib/hostRouting.ts). Legacy /go/[code] route + bootstrapValidation deleted —
BootstrapSessions table verified free of active pre-cutover sessions.
- staticwebapp.config.json now carries production routing: 19 docs/marketing
redirects (order-sensitive), legacy rewrites (exact before wildcard), 404
override, all security headers incl. literalized prod CSP (no unsafe-eval),
Content-Type routes for the extension-less og images. Guarded by the new
utils/__tests__/swaConfig.guard.test.ts. Maintenance mode is now a
first-routes-entry insert, never a whole-file swap (README rewritten).
- Backend notification deep links go through Constants.PortalSessionUrl
(query shape, escaped, pinned); 3 producers migrated.
- deploy-web.yml uploads out/ with is_static_export.
Export build: 79 static pages, no go/ output. Web: tsc clean, vitest 53
files green (618 incl. new guards; 2 deleted legacy suites). Backend:
solution build clean, 3622/3622 tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The portal's Next.js /go/[code] route was the web app's only request-time
server code and blocked the planned static export. The script generation
now lives in the Function App as GET /api/bootstrap/go/{code}
(GetBootstrapScriptFunction), reached by customers as
irm https://go.autopilotmonitor.com/{code} | iex via a Front Door custom
domain whose /* route rewrites onto the API path.
- New Constants.BootstrapGoBaseUrl; CreateBootstrapSessionFunction now
hands out the go URL (previously unguarded — CreateBootstrapSessionUrlShapeTests
pins it now). go host added to both URL-guard enforced lists.
- BootstrapScriptValueValidator ports the bootstrapValidation.ts injection
invariants (exact GUIDs, https + host allowlist + /agent/*.zip path,
expiry <= 14d) as defense-in-depth for in-process values.
- OobeBootstrapScriptGenerator ports the PS template as a Replace-token
raw string; golden test pins byte parity with the TS template
(TestData/oobe-bootstrap-golden.ps1, captured from route.ts).
- Always-HTTP-200 text/plain contract incl. rate limit (own bucket
bootstrap-script:{ip} 20/min); disabled tenant indistinguishable from
unknown code. Catalog: PublicAnonymous; NoStoreCacheMiddleware covers
/api/bootstrap/go/ (script inlines the bearer token).
- Web: BOOTSTRAP_GO_URL registry entry, sessions list shows the go host,
legacy route marked LEGACY (alive for pre-migration URLs <= 168h,
removal in the static-export PR).
- Docs: new OKF backend/oobe-bootstrap-go-flow.md, url-registry go
section, index/log entries.
Backend tests 3611/3611, web vitest 651/651, tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The HTTP/1.1 / six-connection-cap claim was wrong — it came from the
Schannel curl build that cannot speak HTTP/2. ALPN (openssl s_client)
and .NET HttpClient confirm the portal negotiates h2; the queue is
server-side in the SWA SSR runtime. Doc, index, log and the
GlobalSidebar comment now describe the real mechanism.
Adds the 2026-07-28 outcome section: Standard plan changed nothing
(stratified p50s match, Mann-Whitney p=0.81); prefetch={false} removed
the freezes (worst totalDuration 225s -> 4.9s) and also removed an
accidental keep-alive (~150ms added to cold navigations).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer feedback: backfilled modern_deployment_error events (pre-agent
Autopilot event-log errors replayed at agent start) rendered as plain
Error rows with one identical emission timestamp — indistinguishable
from a live failure.
Web (EventTimeline.tsx, effective for already-stored events):
- "⟲ Backfilled" badge (slate, explanatory tooltip) on data.backfilled
- row time now shows the original event-log timeCreated with a
"(reported …)" suffix; metadata block gains a "Recorded" line;
raw mode gets a compact ⟲ marker
- severity deliberately NOT downgraded — unresolved pre-agent errors
must stay visible in the Error filter
Agent (ModernDeploymentTracker):
- backfill scan buffers the capped record set, emits an Info summary
event first (modern_deployment_log, backfillSummary: true +
replayedCount/droppedOverCap/lookbackMinutes) announcing the block,
then replays oldest-first so timeline sequence follows event-log
chronology (cap semantics unchanged: newest 25 win)
No new EventType, no ConfigVersion bump. Tests: summary emission x2,
tracker subset 12/12; tsc + vitest 651/651 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customers cannot tell why a gather rule produces nothing in the timeline
(e.g. a logparser rule with emit_mode on_change that never emits). New
per-tenant boolean EnableGatherRuleDebugLog (EnableImeMatchLog-style
plumbing end to end): when enabled, the agent writes
%ProgramData%\AutopilotMonitor\Logs\gather_rules_debug.log with one
pipe-delimited line per evaluation decision across all previously silent
"produced nothing" exits — disabled rules, interval without
intervalSeconds, first-tick-after-one-full-interval, phase-scope skips
incl. currentPhase=Unknown, per-(rule, phase) dedup, empty collector
results (emitOnlyIfExists), on_change hash suppression (hash prefix +
streak), guard blocks, logparser per-file outcomes (position skips,
format=text hint when every line fails CMTrace parsing, maxLines and
wildcard caps), and full exception stacks the agent log drops.
- Backend: ConfigVersion 35 -> 36; TenantConfiguration/AgentConfigResponse
field + TableConfigRepository Store+Map
- Agent: GatherRuleDebugLog writer (10 MB cap, one .old generation,
failures swallowed, optional console echo), GatherRuleContext.DebugLog
(null-safe no-op), trace points in executor + collectors; never writes
under _scopeLock (reasons captured inside, written after release);
merger bool->path projection with --gather-debug-log CLI override
tolerance; deliberately NOT in the offline-cache strip lists;
--run-gather-rules always traces; zero-rules delivery leaves a one-line
file instead of nothing
- Side fix: GatherRuleExecutorHost now passes the logger into
InformationalEventPost so gather-event emissions hit the DEBUG log
- Web: tenant settings toggle (Settings -> Agent) + admin tenant-config
report rows
- Tests: merger on/off/override, writer format+rotation, executor debug
lines, logparser debug lines (agent 2131, backend 3561, tsc green)
- Docs: docs/agent/gather-rule-debug-log.md + index/log entries
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Navbar + GlobalSidebar render the full navigation on every page, so the
App Router's default viewport prefetch fired one ?_rsc= request per
visible link against the SWA runtime (~10.4k requests / 14 days). That
origin serves document, chunks and RSC payloads over HTTP/1.1 (6
connections per origin); when the runtime queued the prefetches (p95
896ms, tail up to 234s), they held every socket and the portal froze
until the queue drained. Measured on the 2026-07-27 17:25 incident:
/admin/presence?_rsc= at 151s while the backend API answered in <2.2s.
All 16 nav links now set prefetch={false}. Every target route is a
client component behind ProtectedRoute that fetches its own data after
mount, so the prefetched RSC payload carried no benefit.
New OKF doc docs/web/portal-navigation-prefetch.md records the full
evidence chain and where browser telemetry lives (separate App Insights
resource, not reachable via MCP query_backend_logs).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Analyze rules only (gather stats stay per-batch-deduped until Q6 is
fixed - never silently included). Per (tenant, rule) the radar compares
the trailing 7-day hit rate (FireCount/SessionsEvaluated from the daily
RuleStats rows) against the prior 28-day baseline and fires only when
the window has >=5 hit sessions and >=20 evaluated sessions, the rate
is >=2x baseline AND the Wilson 95% intervals are disjoint
(MetricsMath.WilsonInterval/RateIncreaseSeparated, hand-computed test
vectors). Suppressed for deleted rules, rules inside the CreatedAt
grace period (<35d), rules edited inside the window, empty baselines,
and the RuleRegressionRadarDisabled app-setting kill switch.
Alert episodes live as ruleregression|{ruleId} rows in the notification
tracker (third keyspace): AddEntity-409 dedup = one bell + one
RuleFrequencyRegression ops event (category Tenant, Warning, full
numbers, dual-registered in the web OPS_EVENT_TYPES) per episode;
numbers refresh while active without moving FirstNotifiedAt; episodes
re-arm under 1.5x baseline, on fire-stop, or via the tracker's 30d
retention prune (spec-intended reminder). On-fire dimension correlation
(osBuild/model/agentVersion/imeVersion; hit set from a RuleResults
range scan intersected with the window sessions) reports both shares
with the "correlated, not necessarily causal" wording contract or
honestly "no clear dimension concentration". Radar runs in the daily
maintenance pass after the rule-stats aggregation, anchored on
yesterday (whole days only).
Surfaces: regressions[] in the rule-stats response (MCP get_rule_stats
inherits it), tenant bell deep-linking to the rule card, and the
analyze-rules page gains a densified 30-day fire sparkline plus a
regression badge while an episode is active. OKF doc
backend/rule-regression-radar.md + index/log.
Backend 3474/3474 (27 new: Wilson vectors, every suppression branch,
window boundary dates, re-arm matrix, dimension gates, tracker
round-trips incl. tri-states, keyspace dedup/prune, wording pins),
web tsc + vitest 627/627, MCP tsc + 455 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three read routes over the PR4 data: metrics/device-history
(MemberRead + tenant query param - one route serves the web banner AND
MCP, GA cross-tenant via ?tenantId= with a sessionId-based owner
fallback; history:null is a normal outcome), metrics/device-journeys
and global/metrics/device-journeys (days selector honored because FTR
counts are additive - window rate = ratio of summed daily rows; the
tenant-less global variant serves the "global" partition and discloses
repeatDevices:null). Attempt numbers are server-computed
(DeviceJourneyCalculator.ComputeAttemptNumber: chain position within
the journey; live sessions via a virtual non-successful attempt so the
redeploy and 30d-gap rules apply) - no consumer re-derives journey
semantics. Repeat-devices violator list: last journey >=2 attempts with
newest terminal ref in-window, attempts desc then recency, cap 10,
failure reason from the newest failed attempt (bounded point-reads).
Web: DeviceHistoryBanner on session detail ("Attempt N for this device
- View history", expandable terminal chain, durations strictly the
authoritative durationSeconds, WG/DevPrep/admin-marked badges) and the
Fleet Health "First-time-right" section (rate gated at >=20 completed
journeys with disclosed n, junk-serial exclusion note, Monday-UTC
weekly trend, attempt histogram, repeat devices) following the 7/30/90
selector. MCP: get_device_history (serial mode / fleet FTR mode,
role-aware routing, all roles - GA_FULL privilege snapshot extended
deliberately). Docs: OKF backend/device-journeys.md + index/log.
Backend 3447/3447 (14 new: attempt-number vectors incl. gap boundaries
and live sessions, additive sums, days clamping, repeat-device
gate/sort/cap), web tsc + vitest 627/627, MCP tsc + 455 passed.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Insights expansion PR3 (tasks/insights-expansion-spec.md §F1 "Surfaces").
Completes F1: schema (PR1) -> persistence/aggregation (PR2) -> surfaces.
- Rolling range rows: the sweep now also writes "rolling30|{class}"
aggregate rows — range statistics over the window's actual sessions. A
median of per-day medians is not the range median, so the fleet panel
reads these and is deliberately fixed to "last 30 days"; the daily rows
stay in the payload for trends/MCP.
- Routes (all registered in EndpointAccessPolicyCatalog):
GET sessions/{sessionId}/time-attribution (MemberRead + tenant query
param, GA fallback via FindSessionTenantId; breakdown=null is SUCCESS —
a missing lane is a normal state, not an error), GET
metrics/time-attribution (MemberRead) and GET
global/metrics/time-attribution (GlobalReadOrAdmin + tenantId filter),
both returning { windowDays, classes (rolling), daily }.
- Web: TimeAttributionLane under the session-detail phase timeline —
proportional stacked bar of segment totals incl. the explicit
unattributed remainder (bar always sums to the session's authoritative
duration), ESP-blocking apps with critical-path occupancy, reboot badge
(cross-cutting, never a bar slice), quality-flag chips; fetched
fail-soft for terminal sessions only. Fleet Health gains the "Time
attribution" section: per-class stacked medians gated at n>=20
("insufficient data (n=...)" below), flagged/missing exclusions
disclosed, and a top-blocking-apps table with class-scoped rows and
"up to X (p75 Y)" what-if wording. Deliberate deviation: the spec's
per-app trend column is omitted — per-day app rows are too sparse under
the daily n>=5 gate; revisit once rolling history exists.
- MCP: get_time_attribution (one tool, session + fleet mode, role-aware
routing like get_app_install_metrics, registered for all roles) — the
privilege-leak-guard catalog snapshot extended deliberately.
- Docs: OKF concept doc backend/time-attribution.md (invariant, windows,
segment mapping, positive-evidence blocking, Q7 reboot spans,
persistence + surfaces) registered in index.md + log.md.
Backend 3382/3382 (+1 rolling-aggregate test), web tsc clean + vitest
627/627, MCP tsc clean + 455 passed. Post-deploy: web deploy +
MCP redeploy (new tool + docs corpus).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Covers the queue-on-session/deliver-on-ingest channel, the
TenantAdminOrOperator tier with in-function per-type re-gating, the portal
Collect Logs flow (quick-config orders rotate_config before
request_diagnostics), sequence-based progress matching, and the hosted-prefix
sweep in the offboarding cascade. Registered in index.md, logged in log.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer follow-up corrected the field-case mechanics: TrackingPoliciesCreated=1
WAS set under the foreign ConfigMgr provider - the ESP ignores a foreign
provider's completion values and waits on the Intune-registered "Sidecar"
provider by name (renaming the key unblocks instantly without creating any
tracking entries). The completeness-keyed detector would have missed exactly
the incident it was built for.
- Arm 2: >=1 Setup\Apps provider registered without Sidecar, continuously for
the 15-min dwell -> reason=sidecar_provider_missing; clock resets when
Sidecar registers later (legitimate startup ordering); DevicePreparation-only
snapshots don't activate it
- Arm 1 (continuously-incomplete providers) unchanged -> reason=provider_incomplete;
sidecar-missing wins when both arms cross in one round (single event)
- Cross-restart fingerprint gains a "sidecarMissing" sentinel; payload adds
reason + sidecarMissingForMinutes
- Regression test: ConfigMgr complete + Sidecar absent -> fires (previously
classified healthy); probe/doc headers and OKF doc corrected, web event-type
description updated
Evidence (research session c7e3db08): Intune's ESP-policy WAP payload always
registers Sidecar (ConfigMgr only via co-management settings policy, per
Niehaus' documented payload); Microsoft's ESP troubleshooting models app
tracking exclusively through the Sidecar subkey.
Agent tests 2043/2043, web tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Customer field case: pre-provisioned devices with the ConfigMgr client as a
blocking app occasionally park the user ESP at "Apps (Identifying)" for days.
Root cause (verified against the EnrollmentStatusTracking CSP doc): only a
"ConfigMgr" provider is registered under Device\Setup\Apps\PolicyProviders
(no "Sidecar" = IME) and TrackingPoliciesCreated never becomes 1 - a wait the
CSP defines WITHOUT any timeout. Renaming the key to Sidecar unblocks
immediately. Microsoft documents that ESP providers are not aware of each
other and that pre-provisioning is unsupported with co-management.
New always-on esp_policy_provider_stalled tripwire, generalized to any
provider in both kinds (Setup/Apps TrackingPoliciesCreated != 1,
DevicePreparation InstallationState not in {2,3}):
- EspPolicyProviderProbe: fail-soft registry read of the
EnrollmentStatusTracking mirror (device + user-SID scopes), defensive
value parsing (missing/unparseable = incomplete), ScopedOverride test seam.
Research and CSP contract quotes captured in the XML doc header.
- EspPolicyProviderStallDetector: pure EvaluateStallTransition on a 15-min
continuous-incompleteness dwell (resets on completion/disappearance;
absent key = normal early enrollment), one-shot per provider,
StartupEventGate fingerprint over the sorted stalled key set for
cross-restart dedup. Warning + ImmediateUpload via InformationalEventPost;
purely observational, no decision-engine coupling, no config toggle,
no ConfigVersion bump (disk_space_low precedent).
- EspPolicyProviderStallHost: 60-s wall-clock tick, deliberately outside the
StallProbe idle gate (idle clock resets on session activity) and the
idle-stopped periodic collectors.
- Registered in Constants.EventTypes, MCP resource-catalog (health_events),
web gather-rule autocomplete (stall category).
Guardrail companion: SOFTWARE\Microsoft\Windows\Autopilot joins the
gather-rule registry allowlist ("Autopilot / OOBE / Setup") so tenants can
build custom rules on the subtree (e.g. not_exists on the Sidecar provider
key); generated web mirror regenerated via combine.js; positive + sibling-
spoof guard tests added.
Docs: OKF concept doc agent/esp-policy-provider-stall.md (+ index/log).
Tests: 22 new detector tests (pure transition matrix + emission surface incl.
the ConfigMgr-only customer scenario and gate-across-restart dedup); agent
suite 2013/2013, web vitest 589/589 + tsc clean, MCP 449 passed incl.
event-types-drift.
Both the detector and the embedded allowlist take effect with the next agent
release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Customer report: ANALYZE-SEC-001 warned on devices whose firmware already
carried the Windows UEFI CA 2023 certificate — the agent only read the
SecureBoot\Servicing registry value, which mirrors the Windows Update
rollout and lags behind reality (OEM factory image, WinCS, WU not run yet).
Agent: new UefiSecureBootCertReader reads the UEFI db/KEK variables
natively (GetFirmwareEnvironmentVariableExW; SeSystemEnvironmentPrivilege
enabled on the process token and restored — first privilege-adjustment
code in the repo). Fail-soft, never throws; legacy BIOS maps to not_uefi.
secureboot_status gains per-certificate booleans, uefiFirmwareReadStatus,
and the one-sided marker uefiCa2023FirmwareConfirmed (emitted only on
positive confirmation, never as false).
Rule v3.0.0: not_exists precondition on the marker suppresses the rule
when the firmware confirms the certificate; the marker's absence keeps
old agents on exact v2 behavior, so the rule is safe to seed before the
agent rollout. Firmware-verified-missing raises confidence (+20 db,
+10 KEK 2K CA 2023).
Verified: full solution build; backend 3201, agent 1988, web 582 tests
green; tsc/ajv clean; elevated cross-check on real firmware matches
Get-SecureBootUEFI on all seven certificate probes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first release after the download-alias migration exposed Front Door
caching as a correctness hazard: the alias kept serving the previous
version.json (X-Cache: TCP_REMOTE_HIT) after the origin had rotated to
2.0.1370, pinning the portal's version badge to the old build. The dangerous
variant is worse than a stale badge -- manifest and ZIP are independent cache
entries, so a stale manifest paired with a fresh ZIP fails the bootstrap's
SHA-256 integrity check and the install dies.
Route caching on /agent/* has been disabled (infra side); this commit is the
defense-in-depth half: every upload now carries x-ms-blob-cache-control
no-cache -- the CI Send-Blob (eu and legacy mirror alike), the local build.ps1
counterpart (gitignored, changed in lockstep), and the versions/*.json
manifest publishes. Uniform no-cache rather than an immutable/mutable split
because every blob in the agent container rotates in place, including the
per-line AutopilotMonitor-Agent-v2.zip -- nothing there is safely cacheable.
Existing blobs get the header with the next release's re-upload.
Log/docs updated in url-registry.md; no code paths changed, YAML and ps1
parse-checked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An interactive logon during the RealmJoin first deployment (phase 100/101)
makes RJ reclassify the run as a secondary-user deployment, whose branch
writes 200/210 and never CompletedFirstDeployment (110) — no crash, no
reboot request. The V2 completion gate waited for exactly 110, so these
sessions hung until the 60-min hard timeout (session 224b2087; siblings
558bf067/3cc18724 delayed ~50 min the same way).
- promote realmjoin_phase_changed to a typed RealmJoinPhaseChanged signal
(dual-emitted by the adapter) so RealmJoinFacts.LastDeploymentPhase is
persisted across agent restarts
- abort rule: prev in {100,101} and current in {200,210} with no outcome
yet -> emit realmjoin_first_deployment_incomplete (Warning), cancel the
hard-timeout deadline, release the gate with outcome
FirstDeploymentIncomplete; also evaluated on the RealmJoinDetected
replay path (restart swallows the transition)
- deliberately NOT triggered by 101->0 (RJ service restart may still
complete the first deployment) nor by a first observation of 210
(session 6f1959c0: RJ already completed before agent boot)
- fix realmjoin_timeout always claiming "last phase: 0" - the persisted
phase now feeds the message
- backend EnrollmentTimeoutClassifier counts the new event as a gate
terminal; MCP resource catalog dual-synced
- CompleteIfDeferredOrBookkeep widened to a leading-effect list
Verified by replaying session 224b2087's signal log: gate releases at
15:05:17 (101->200) instead of hanging until the 15:54 deadline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every well-known host -- own subdomains, both blob accounts, the API origin,
Graph, Entra login -- now lives in exactly one registry per component:
Constants.cs (*BaseUrl) for C#, utils/config.ts for the web app, src/config.ts
for the MCP server. lib/hostRouting.ts derives its hostnames from the registry
and middleware.ts imports from it, retiring what turned out to be a fourth
independent copy of the www/portal host pair.
Three new guard tests (backend xUnit, web vitest, MCP vitest) scan non-test
source and fail on any enforced host outside its registry. This is the lesson
of the EU cutover: the registry existed then too, but LatestVersionsService
(feeds the portal's version badge) and ValidateBootstrapCodeFunction carried
their own literal copies of the legacy blob host and were missed -- kept
working only by the fail-soft legacy mirror in build-agent.yml, whose silent
failure would have pinned badge and self-update to stale bits with a green
release. Both guards proved themselves on their first run, flagging the
middleware host copy and a literal in an MCP tool description. Test files stay
exempt on purpose: literals there are independent oracles, and
ValidateBootstrapCodeResponseShapeTests now pins the full alias URL against
the Constants-built one so registry changes must be acknowledged consciously.
The two legacy readers plus the agent SelfUpdater now use
Constants.AgentDownloadBaseUrl (download.autopilotmonitor.com/agent); the
alias was verified live for version.json, version-v2.json and both ZIPs
before the switch. The portal's bootstrap allow-list accepts the alias AND
the legacy host (AGENT_DOWNLOAD_HOSTNAMES) for already-deployed customer
scripts, which imposes a deploy order: web before or with backend. The
legacy keepalive probes and the release mirror stay untouched -- they exist
for exactly this transition. Also: the bootstrap short-link now targets www
directly instead of riding the registrar's apex 301, and the welcome-mail
docs button points at docs. instead of bouncing through www./docs.
Suites: backend 3194, agent core 1925, DecisionCore 474, web 582, MCP 449 --
all green; next build verified (it, not tsc, proves the next.config.ts
registry import). Docs: docs/url-registry.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three components now report <major>.<minor>.<build> plus a 7-char commit
and a build timestamp. The backend had been pinned at a hand-written 1.0.0
since inception and starts at 1.5.1000; the MCP server's package.json version
was bumped by hand and starts at 1.5.300, and it now reports its own commit on
/health instead of only the documentation bundle's.
The build number keeps coming from a counter blob reserved before the build
with ETag-CAS, but there is now one implementation instead of two copies:
.github/scripts/Request-BuildNumber.ps1, called by all three workflows and by
the local build.ps1. It lives under .github/ because scripts/Deployment/ is
gitignored (those scripts carry SAS tokens) and would therefore be absent from
a CI checkout; scripts/Build/ and scripts/Release/ are shadowed by the
[Bb]uild/ and [Rr]elease/ artifact ignores.
The counter cannot be replaced by the published manifest, which is the obvious
simplification to reach for: version.json is written only on stable publish, so
two consecutive dev builds would derive the same number and the second would
overwrite the first's artifact. git rev-list --count fails the same way, two
builds of one commit collide.
Both deploy workflows now poll their health endpoint until it reports exactly
the version and commit just deployed, and publish versions/<component>.json
only then. A green deploy step alone means the package was accepted, not that
it is being served -- so a manifest that disagrees with the live endpoint now
means the running instance drifted (rollback, stale revision) rather than a
pipeline that stopped halfway. The agent manifest gains the same commit and
buildUtc fields additively; the bootstrap reads it by field name, so older
installers ignore them, and sha256/bootstrapVersion stay untouched.
Requires the versions container plus AZURE_BLOB_VERSIONS_SAS_TOKEN and the two
seeded counters (backend 999, mcp 299) before the first deploy run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LogParserCollector and EventLogCollector called no guard at all, while
File/Json/Xml did — and rules/gather-rules.md already promised customers
that log-parser paths were allowlisted. A tenant admin could author a
logparser rule against C:\Users via POST /api/rules/gather and have up to
1000 lines uploaded as telemetry, bypassing the hard C:\Users block the
same page guarantees; eventlog could read any channel, Security included.
The backend validates no collectorType, target, or parameters, so the
agent is the only place this can be enforced.
Both now guard. logparser cannot check rule.Target directly (GetFullPath
throws on a wildcard), so it checks the directory for wildcard targets and
re-checks each resolved path. Event log channels get an allowlist matched
on a "/" boundary. Hard-blocked channels live in a C# constant rather than
guardrails.json: a parse error drops the allowlists to empty, which is
fail-closed, but would have lifted a block sourced from the same file.
Also fixes an inert timeout in CommandCollector: ReadToEnd() blocks until
the pipe closes, so WaitForExit(30000) was only reached once the process
had already exited and a hung command pinned its worker indefinitely.
Adds the first real CI gate. No workflow ran tests on a pull request — the
deploy workflows build without testing and build-agent.yml covered only
DecisionCore.Tests on manual dispatch — while trust/security-faq.md
already told customers automated tests run in CI. ci.yml now runs all six
suites on every PR and merge, plus a job that fails on generated-guardrail
drift, alongside CodeQL and Dependabot. OfficeBinaryWatcherTests is marked
SerialThreading; its bounded re-probe races the shared ThreadPool and
failed once under full-suite load.
Verified: 3101 backend, 1911 agent V2, 474 decision core, 44 dialog, 572
web, 447 MCP. The timeout fix is verified by runtime, not assertion — a
Start-Sleep 120 command now returns after ~30s.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A German question returns zero results from search_docs, not poor ones: both the
documentation and the rules are English, and all-MiniLM-L6-v2 is an English-only
model, so the query vector lands nowhere near any chunk. For a German-speaking
admin driving an MCP client in their own language, the tool is effectively blind
and gives no hint why.
The caller is itself a language model, so the cheapest correct fix is to say so:
both tool descriptions now instruct it to translate the question to English before
searching and to answer the user in their own language. No infrastructure, no
model change, no ranking impact.
A multilingual embedding model would handle it natively and was considered, but it
is substantially larger — which matters on a scale-to-zero container — and it would
change English retrieval as well, so it would have to be measured against
scripts/eval-docs-search.ts rather than swapped in on the assumption that more
languages cannot hurt.
Verified: 447 tests pass, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The boot marks answered the question on the first production cold start, and the
answer overturns the reasoning behind an infrastructure change made earlier today.
+244ms node started (startup + module loading)
+431ms corpora read from disk (+187ms)
+491ms index read, validated, hydrated (+60ms for 3.12 MB)
+504ms listening
+677ms query embedder warm (background)
Against a 31.5s cold start, the application accounts for half a second. The other
~98% is Container Apps activation: KEDA reacting, pod scheduling, mounting the
236 MB image, starting the container. The container also boots faster than a
development laptop (504ms vs 1273ms), so nothing here is CPU-starved.
Consequences recorded rather than quietly dropped:
- Raising the container to 0.5 vCPU was justified by the embedding model load on
the cold-start path. Prebaked, that load is 173ms and runs in the background
where only the first search could wait on it. The justification does not survive
measurement. The setting stays — it plausibly helps concurrent requests, and at
scale-to-zero the cost difference is negligible — but the Bicep comment now says
what is true instead of what was assumed, so nobody re-derives the wrong reason.
- Index size is irrelevant to startup (60ms for 3.12 MB), so the corpus can grow
freely, and the embedding rounding earns its keep on file size alone.
- The only lever that would actually remove the cold start is minReplicas: 1, at
roughly 7x the monthly free consumption grant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A cold start after scale-to-zero measures 24.5s to first byte against 60ms warm,
and nothing outside the process could say why. Log Analytics stamps every console
line of one ingestion batch with the same time, so the boot sequence collapses to
a single instant however it is queried, and piping stdout locally buffers until
exit. So the server times itself: `[boot +Nms]` marks derived from
process.uptime(), which unlike an in-process timer also covers Node's own startup
and module loading.
Measuring the precomputed path locally already contradicts two assumptions worth
recording:
+860ms node started (startup + module loading)
+1199ms corpora read from disk (+339ms)
+1242ms index read, validated, hydrated (+43ms for 3.12 MB)
+1273ms listening (+31ms)
+1497ms query embedder warm (+224ms, background)
The index is 43ms, not seconds — its size is not the cold-start cost, and growing
the corpus is cheap. The prebaked embedder is 224ms and loads in the background,
so only the first search could ever wait on it. That was the stated reason for
raising the container to 0.5 vCPU, which now looks like a weaker justification
than it did; the production marks will settle it. If they show the app listening
after a few seconds, the rest of the 24.5s is platform activation that neither CPU
nor index changes can touch.
Verified: 447 tests pass, marks appear on both boot paths locally. Incidentally
the run also confirmed the corpus hash guard and the staleness signal both work —
the local docs repo has moved to 6333d56 while the deployed image reports d8cb7f4,
so the precomputed index was correctly rejected and recomputed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The form offered trigger, phase scope and emit mode unconditionally, which
produced combinations that cannot do anything plus one silently wrong state:
- A scope on a Phase Start / Phase End rule that names a concrete phase can only
ever suppress the single firing — the phase IS the moment. Same for emit mode:
a one-shot has no second result to dedup against.
- Selecting "From a phase onwards" (or "Only during specific phases") and not
filling the detail dropdown serialised to null, i.e. the rule ran everywhere
while the form still read "From a phase onwards".
supportsPhaseScope / supportsEmitMode now drive visibility; buildScopeFields
nulls hidden-control state so a value from an earlier trigger choice cannot leak
into the payload; validateScopeSelection rejects an unfilled scope mode on both
create and save. Startup is the deliberate exception — a scope there defers the
one-shot instead of suppressing it, so the scope control stays and only emit mode
is hidden. One-shot triggers now state in-form when they collect.
Sub-dropdown hints no longer claim "No selection = all phases" (they turn amber
and name the required action instead).
Tests: app/gather-rules/__tests__/gatherRuleFormLogic.test.ts (15) pins the
applicability matrix, the payload nulling incl. the trigger-switch leak, the
unfilled-scope rejection, and the JSON-mode scopeMode derivation.
vitest 572/572; tsc silent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live test found a third recall gap. The fallback only topped up when semantic
search returned FEWER than topK results — but the more common failure is a full
list of unconvincing ones. Querying "WOW6432Node" returned three chunks scoring
~0.29, none containing the word, while the single chunk that does was never
considered at all.
The fix is a ranking change, so it was measured rather than reasoned about. Adds
scripts/eval-docs-search.ts, which scores strategies over 2465 queries in three
deliberately different classes: every chunk heading (easy by construction, so a
regression detector), rare identifiers (the class this fallback serves), and
hand-written questions (the reality anchor — not derived from the corpus, so it
shares little vocabulary with it). Embeddings are computed once and each strategy
is a pure function over the same candidates, so sweeping a threshold costs no
extra inference.
known-item literal handwritten MRR fixed broken
append-only (before) 71.9% 66.2% 1.000 - -
weak < 0.30 71.9% 72.6% 1.000 142 0
weak < 0.35 73.7% 81.1% 1.000 335 0
weak < 0.40 76.3% 88.2% 0.944 502 3
weak < 0.45 77.2% 93.0% 0.889 613 5
0.35 is strictly dominant — better in every class, worse in none — so that is
what ships. Higher thresholds keep repairing identifier queries but start
demoting hand-written questions, the class that most resembles real use, so the
gain stops being free. When the head is weak, exact matches move to the front
while at least one semantic result is kept, so a weak-but-correct hit is never
pushed out entirely.
The measurement also resized the problem: identifier queries were succeeding only
66.2% of the time, so the reported case was one of roughly 750, not an outlier.
That is the part a single anecdote could not have shown, and the reason the
harness exists before the fix rather than after it.
Verified: 447 tests pass, including new cases for the weak path and for leaving a
confident ranking untouched. On the real corpus "WOW6432Node" now returns the
chunk that contains it at rank 1, while "how do I deploy the agent with Intune"
(0.819) and "what firewall URLs do I need to allow" (0.374) are unchanged and the
negative controls stay empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two measured recall gaps in search_docs, both silent.
1. Truncation. MiniLM's context is 256 tokens; 87 of 324 chunks exceeded it and
13% of all tokens never reached the ranking — the tokenizer drops the tail
without complaint. Concretely, in built-in-rules.md :: Apps (734 tokens) the
strings ANALYZE-APP-013, ANALYZE-OFFICE-001, ANALYZE-CORR-003 and Click-to-Run
all sit past the cutoff, so "proxy configuration blocking app downloads" did not
retrieve that section at all.
Shrinking chunks cannot fix this: the corpus ranges 2.43 to 5.81 characters per
token, so no character budget bounds tokens, and a budget safe for the worst case
(~620 chars) would shred prose and tables alike. So the chunk stays whole and the
embedding is split instead — index() cuts long text into 250-token windows with
40 overlap, a document carries embeddings[][], and search() scores it by its BEST
window. Max rather than mean: averaging would blur a long section's several
topics into one mediocre direction, the same dilution that makes large context
windows worse than chunking.
The tokenizer is a build-time dependency only (vocabulary, not ONNX weights,
~93 ms); production boots from precomputed vectors and never loads it.
2. No general fallback. Proper nouns embed poorly — "RealmJoin" occurs in 3
chunks yet scores under the threshold. Fuzzy matching was built, measured and
rejected: at a threshold low enough to surface RealmJoin (0.28), Fuse also
returns 5 hits for "SCEPman", a word this corpus contains zero times. Literal
containment has neither problem, and every provider already exposes it. So a
short result set is now topped up via lexicalMatch on distinctive query terms,
requiring two matching terms as corroboration — without that rule "how do I bake
sourdough bread" matched a page about LLM providers, because "bake" is a
substring of "baked into the image". Keyword hits are appended and flagged, never
sorted together with cosine scores.
Also sharpens the tool description with the measured reason to prefer this over
fetching the docs site (~1k tokens for topK=3 against ~9k raw / ~600k rendered,
version-matched, and matching on meaning), and replaces the growing tail of
positional docs parameters with a DocsSearchBundle.
Verified: token coverage 87% -> 100%; "Office Click-to-Run install never
finished" 0.318 -> 0.442; "proxy configuration blocking app downloads" absent ->
0.559 at rank 1; the ten reference queries keep byte-identical scores and rank 1
(short chunks are one window, so the same vector as before — no long-chunk bias).
The fallback is not a dead path: across 904 rare corpus tokens queried alone it
contributes hits for 431, while negative controls stay empty. Precompute output
is accepted by the boot validator and a mutated corpus still rejected; index
2.63 -> 3.12 MB for 324 chunks / 424 vectors. The rules corpus gains the same
treatment unplanned: 128 rules now carry 172 vectors. 445 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MCP server could answer telemetry questions ("why did this session fail")
but not product ones ("how do I deploy the agent", "where is my data stored").
Those answers already exist in the autopilotmonitor-docs bundle; search_docs
makes them semantically searchable.
Deliberately a separate index from search_knowledge: rules and prose answer
different questions, and sharing one topK would dilute both. Registered for
every role — published documentation carries no tenant data — and skipped
entirely when no corpus was baked in, so a doc-less build advertises no
broken tool.
Corpus handling:
- Chunked at h2/h3 and <details>/<summary>, each chunk prefixed with a
"Page > Section > Subsection" breadcrumb so a mid-page hit is self-describing.
- Newlines are normalized before any parsing. This is a correctness
precondition, not cosmetics: the bundle ships CRLF and JS `.` does not match
\r, so /^(#{2,3})\s+(.*)$/ matched zero headings — silently, degrading every
page to unlabelled paragraph splits.
- Two size exceptions: tables are never split (a split table leaves `| --- |`
rows with no header), and unbroken changelog lists split at line level (left
whole, their tails fall outside the model window and become unsearchable).
- Short sections are kept — a one-line answer is often the best hit, and the
breadcrumb supplies the context a short body lacks.
Build and delivery:
- Docs are embedded at image build time into an optional third
PrecomputedSection. Optional so a checkout without the docs repo does not
invalidate the rules and event-type vectors; the reverse (server has docs,
file does not) is staleness and is rejected.
- The deploy workflow checks the public docs repo into the build context and
passes its SHA as DOCS_COMMIT, which /health reports alongside the chunk
count — editing the docs does not update a running server, so staleness is
made checkable rather than memorable.
- Embeddings are rounded to 6 decimals on serialize: 55% smaller index (2.63
vs 5.88 MB) for a ~1e-5 norm error against a 1e-3 tolerance. indexPrecomputed
now asserts unit norm, since the hydrate path never checked it.
Infra: the Bicep template was two changes behind the live deployment (EU
cutover and ACR hardening, both applied directly in Azure). Reconciled against
the running resources — managed-identity image pull replaces the admin
credentials and the acr-password secret — and infra/ is no longer gitignored so
it stops drifting. Container raised to 0.5 vCPU / 1.0 GiB for the cold-start
query-embedder load. The template does not model the mcp.autopilotmonitor.com
custom domain, so a full deployment would remove it; the header warns about it.
Verified: 428 tests pass; 45 pages produce 324 chunks; sample queries hit the
right page top-1/2; all 8 derived URLs return 200 live; the precompute output
is accepted by the boot validator and a mutated corpus is rejected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to gather-rule phase scoping. One-shot collection at a phase START
already existed (phase_change dedups per (rule, phase)) and at enrollment END
(on_event + enrollment_complete); the END of a specific phase had no trigger.
New trigger value "phase_exit": fires once when TriggerPhase is LEFT (empty =
every phase transition). Together with phase_change it forms the two one-shot
bookends of a phase.
Agent (GatherRuleExecutor.OnPhaseChanged):
- Exit rules are evaluated BEFORE _currentPhase advances, so both the
TriggerPhase match and the phase-scope gate see the phase being LEFT. A rule
scoped to DeviceSetup therefore fires on its own exit even though the phase
being entered is out of scope.
- A transition into Failed fires the exit rules of the failing phase — the
snapshot at a failure boundary is the point of the trigger.
- Exit of Unknown never fires; a repeated identical phase signal is not a
transition.
- Dedup key "exit:{ruleId}|{phase}" shares _phaseRulesExecuted with the
phase_change keys but stays in a disjoint key space, so an enter-rule and an
exit-rule on the same phase cannot consume each other's slot.
Portal: trigger labels are now "Phase Start" / "Phase End"; the canonical phase
dropdown and the create/save payloads cover both via PHASE_TRIGGERS; the card
renders "entering X" / "leaving X".
No ConfigVersion bump: no new field is transported. Old agents do not know the
value and simply never fire the rule (silent no-op).
Tests: GatherRuleExecutorPhaseExitTests (11) — enter-vs-exit, no refire on
re-entry, empty-triggerPhase, exit-of-Unknown, same-phase-signal, into-Failed,
Complete, scope-against-left-phase, out-of-scope slot preservation, key-space
disjointness, on_change interplay. Build 0 Errors; V2.Core.Tests 1884/1884;
Functions.Tests 3099/3099; vitest 557/557; tsc silent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rework after the first real enrollment (session b2e890c1) exposed three
defects in a654c730:
- Shell-Core "RebootCoalescing" match was a false positive by construction:
the token appears in the ROUTINE SubcategoryProcessing_Started bootstrap
marker ("Starting subcategory DeviceSetup.RebootCoalescing...") on EVERY
enrollment. esp_reboot_coalescing removed entirely (branch, constant,
catalog, tests).
- 2800 reboot-required flags fire at every policy sync; only DeviceSetup-time
flags cause the coalesced reboot. The session's flags fired during
AccountSetup - no reboot - yet our Warning events claimed "device will
restart". Raw event is now a neutral Info observation; the reboot CLAIM
moved to ANALYZE-ESP-005 with a REQUIRED system_reboot_detected >= 1
condition (no observed reboot -> advisory silent).
- One event per 2800 record was spam: records are now buffered and debounced
(10 s) into ONE aggregated event per burst (rebootUris[]/uriCount/
firstRebootUri/recordCount/unparsedCount). Watermark is persisted at FLUSH,
not claim - a pre-emit reboot kill loses nothing, the post-restart backfill
re-emits the burst at its historical time.
Verified 2800 description shape recorded in code, tests, and concept doc:
"The following URI has triggered a reboot: (./Device/...)."; extraction
covered by a real-text regression test. Lesson added to tasks/lessons.md
(observation-level events vs outcome-gated claims; substring tokens appear
in framework bookkeeping records).
Tests: Agent.V2.Core 1867/1867, Functions 3099/3099, MCP 400 passed,
solution build 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Device-assigned MDM policies whose CSP URIs match the OS reboot-required
catalog force a coalesced reboot at the end of ESP DeviceSetup - the user
sees an unexpected restart and a second sign-in before AccountSetup.
Invisible in the Intune console; admins simply lived with it. Now attributed
and actionable (PMPC research).
- MdmRebootPolicyTracker (+host): watches DeviceManagement-Enterprise-
Diagnostics-Provider/Admin EventID 2800, emits one mdm_policy_reboot_required
per reboot-forcing URI (Warning, ImmediateUpload; tolerant EventData-first
URI extraction with regex fallback; fail-soft OMADM RebootRequired probe).
Cross-restart RecordId watermark (mdm-reboot-watermark.json) - mandatory,
the coalesced reboot restarts the agent and the backfill re-reads the log.
- ShellCoreTracker: RebootCoalescing branch on the already-watched 62407
stream -> esp_reboot_coalescing corroboration event; deliberate deviation:
emits on backfill too (live emit races the reboot kill).
- ANALYZE-ESP-005 advisory (warning): names the first URI via {{rebootUri}},
remediation = reassign profiles device -> user groups; +20 confidence on
coalescing corroboration, +10 on >=2 URIs.
- Config: Collectors.MdmRebootPolicyWatcherEnabled (default ON, internal
steering only - agent-native collector, NOT an IAgentAnalyzer, no admin UI),
MdmRebootPolicyBackfillLookbackMinutes=60; ConfigVersion 35.
- Event types deliberately distinct from system_reboot_detected so the
backend RebootCount (exact-string match, 2 sites) stays uninflated.
- Docs: docs/agent/mdm-reboot-coalescing.md (+index/log). Exact 2800/62407
message texts are unverified until E2E VM validation (tolerant matching).
Tests: Agent.V2.Core 1868/1868 (21 new), Functions 3099/3099, MCP 400 passed
(event-types drift), solution build 0 errors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The 2026-07-18 storage migration reset every row's system Timestamp,
exposing that AuditLogs, OpsEvents and Events displayed, filtered and
retention-swept on it (custom "Timestamp" writes were silent no-ops —
reserved system property). All migrated rows read as 2026-07-18.
- New Helpers/BusinessTimestamp: RowKey decoders (audit !revtick_guid,
ops bare revtick, event yyyyMMddHHmmssfff_seq), RowKey-range OData
clause builders (tick-exact, with the mandatory RowKey lt '"' guard
so no lower bound can ever match legacy bare-GUID audit rows), and
dual DateTimeOffset/DateTime column reads.
- Writes in all three tables now persist OccurredUtc (custom column,
copied verbatim by future migrations); no-op ["Timestamp"] removed.
- Read mappers resolve OccurredUtc -> RowKey decode -> system Timestamp;
GetEarliestSessionEventTimestampAsync uses the same resolution so
StartedAt/Duration reconciliation can no longer absorb the migration
date on migrated sessions.
- Date-window filters and retention moved off the system Timestamp onto
RowKey ranges; ops retention loses its full-table scan and the
post-migration mass-delete cliff; audit retention runs a temporary
second pass for legacy GUID rows (TODO: remove after 2027-03-01).
- New GlobalAdmin backfill endpoint POST /api/maintenance/
backfill-occurred-utc (dry-run by default, batched, idempotent MERGE
of only OccurredUtc, table=audit|ops) — Events rows are deliberately
never mutated; their read path decodes the RowKey instead.
- Tests: decoders, clause boundary proofs incl. GUID-safety pins,
mapper precedence, backfill row decisions (Functions.Tests 3078).
- Docs: docs/backend/business-timestamps.md (system Timestamp is never
authoritative; migration checklist rule).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New rules/ section in the knowledge bundle: semantics table, from-phase latch
behavior (never via Failed, sticky through it), on_change x emitOnlyIfExists
composition, restart re-emit, no-phase-signal caveat, old-agent degradation,
and the prerequisite toggle-wipe fix. Index + log updated.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-home agents to a new API base URL through the agent/config response
(backend move or per-tenant EU->US region move) - the lesson from the
2026-07-18 WEU->GWC cutover, where the compiled-in API hostname hard-cut
every in-flight agent (no DNS indirection possible: mTLS breaks behind a
proxy, Flex custom-domain TLS is broken).
Backend (ConfigVersion 33):
- AdminConfiguration.AgentMigrateApiBaseUrl (global) +
AgentMigrateTenantOverridesJson (tenantId->URL; empty value pins a
tenant against the global target; override wins)
- GetAgentConfigFunction.ResolveMigrateTarget serves the validated target;
AgentMigrateServed / AgentMigrateRejected warnings as delivery evidence
- TableConfigRepository Store+Map; admin portal Global Settings round-trip
Agent (kill-switch-mirrored hardening):
- Shared AgentEndpointMigrationRules: https-only, *.azurewebsites.net on a
label boundary, no path/port/userinfo - enforced on BOTH sides
- Live fetch only: RemoteConfigService cache strips the field on write AND
read; EndpointMigration.ResolveTarget re-checks the fetch outcome
- AgentRuntimeConfig.Resolve returns before merge/integrity/bootstrap
cleanup on a pending migration (no spurious IntegrityCheckFailed against
a stale migration stub), Program.RunAgent rebuilds Phase 3+4 against the
new URL - single hop, kill signal wins before and after the hop
Tests: AgentEndpointMigrationTests (backend rules + resolution),
EndpointMigrationTests + cache-hardening strip (agent). Full suites green
(Functions 2992, Agent Core 1822, vitest 557, tsc silent).
Doc: docs/agent/endpoint-migration.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
gh attestation verify only passes with the repo's current name
(okieselbach/AutopilotMonitor); the signed workflow identity carries it,
and the old dashed name fails identity matching even though git-remote
redirects still work. Verified against the first attested dry-run ZIP.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Build numbers now come from a public counter blob (buildcounter-v2.txt,
reserve-before-build with ETag-CAS) shared by local build.ps1 and CI, so the
two paths can never mint the same number. The CI step that committed
buildcounter.txt bumps to main is gone; the repo file is now a gitignored
offline fallback for -SkipUpload smoke builds only.
CI parity fixes vs the local build:
- ship SummaryDialog.exe.config in the ZIP (WPF AppContextSwitchOverrides;
without it the dialog dies with 0x8007045A when spawned from SYSTEM)
- verify EXE version == manifest == reserved build number before zipping
- update the AdminConfiguration hash-oracle only on publish_as_stable
CI-built release ZIPs now carry a GitHub Artifact Attestation (Sigstore
keyless provenance, actions/attest-build-provenance@v4).
Doc: docs/agent/build-counter-blob.md
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Probe enrollment e7dbac7a (24H2) showed Windows writes IsAutoPilotDisabled
(capital P) while the troubleshooting FAQ documents IsAutopilotDisabled —
the missing-event flat evidence lookup was case-sensitive and would have
missed it. Registry value names are case-insensitive by contract, so the
dump dictionary now uses OrdinalIgnoreCase; original casing is preserved
in the emitted payload. Doc notes the casing caveat.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
In the normal IME deployment model the agent installs only after MDM
enrollment, so TPM attestation and ZTD errors from earlier (retried)
attempts sat invisible in the event log. The tracker now replays
Autopilot-channel Level 1-2 records from the last 4h at startup through
the regular emission pipeline.
Spam containment (answering "spam or backfilled event?" — individual
events, but marked and bounded):
- errors only (Warning-level 100 heartbeat excluded by XPath)
- harmless-ID downgrade + burst rollup still applies to backfilled records
- cap of 25 records per scan, newest first (tail logged, not emitted)
- RecordId watermark persisted to autopilot-error-backfill.json, advanced
by both backfill and live watcher -> restarts never re-emit
- events carry backfilled=true + original timeCreated; the event's own
timestamp is emission time, so backend StartedAt alignment is not
dragged backwards by pre-enrollment records
Docs: Observability matrix updated (TPM row now visible-via-backfill) +
new "Startup error backfill" section. Tests: 5 new (XPath, watermark
roundtrip/monotonicity, backfilled/live markers, rollup-on-backfill);
full V2 suite 1804 green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The agent is normally installed by an IME platform script AFTER MDM
enrollment succeeded, so pre-enrollment failures (enrollment blockers,
TPM attestation) are invisible in the normal flow — no enrollment means
no agent and no authenticable session. Await-enrollment mode does not
help either: the binary only polls for the MDM certificate and cannot
authenticate without it. Only bootstrap-token deployments (one pilot
customer) can report pre-MDM.
- New Observability section in docs/agent/autopilot-ztd-diagnostics.md
with a per-rule visibility matrix (normal / bootstrap / await) and the
follow-up option of backfilling Autopilot-channel error records
- ANALYZE-ENRL-003/004 descriptions state the limitation explicitly so
portal users understand why these rules rarely fire
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A flip-flopping user-scoped WHfB CSP read "disabled" once, the arm-C
completion synthesized HelloOutcome="Skipped" and the session completed
5s later while the user sat inside the Hello wizard that had started
230ms earlier (Shell-Core 62404, previously InformationalEvent-only).
Layer 1 — engine (DecisionCore):
- New DecisionSignalKind.HelloWizardStarted + set-once fact
DecisionState.HelloWizardStartedUtc (additive-nullable, census +
snapshot wired).
- Prevention: centralized HelloSatisfiedForCompletion /
HelloPolicyDisabledWithoutWizard predicates veto the policy-disabled
stand-in at all five Skipped-synthesis sites and in the
completion_waiting prerequisites; the AdvisoryCompletion handler gets
a HelloWizardObservedPromote branch (its fallthrough is
re-arm-or-fail).
- Cure: HandleHelloWizardStartedV1 retracts an engine-synthesized
"Skipped" (exact-case discriminator vs the lowercase tracker
vocabulary; first fact-nulling reducer, stage-agnostic because the
RealmJoin gate-defer parks synthetic skips pre-Finalizing), cancels
finalizing_grace, returns to AwaitingHello and arms HelloSafety.
- Agent path: ShellCoreTracker raises HelloWizardStarted before
FinalizingSetupPhaseTriggered; 62404 included in the startup backfill
(fire-once); coordinator timestamp mirror; adapter emit (fire-once).
Layer 2 — HelloTracker confirmation read:
- ApplyPolicyRead state machine: "disabled" commits only after a second
confirming read (~10s poll); "enabled" commits immediately;
disabled→enabled flags flipFlopDetected=true; pending keeps the
policy-unknown wait cadence. Audit fields confirmedReads /
flipFlopDetected on hello_policy_detected.
Fixtures: signal-kinds/hello-wizard-started-v1.json +
enrollment-sessions/userdriven-hello-wizard-unskip-v1.jsonl (772fe502
ordering, ends Completed with HelloOutcome=completed).
Tests: DecisionCore 462/462, V2.Core 1775/1775.
Docs: OKF concept doc docs/agent/hello-wizard-unskip.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub resolves /-prefixed links against the repo root (missing /docs),
so cross-doc navigation broke. Switched all agent docs to standard
relative links and updated the bundle convention in index.md + CLAUDE.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adopt the Open Knowledge Format (GoogleCloudPlatform/knowledge-catalog)
for contributor/AI-facing technical docs: YAML frontmatter on the three
existing concept documents, new index.md (progressive disclosure, root
carries okf_version) and log.md, plus the maintenance convention in
.claude/CLAUDE.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reorganize docs/ tree:
- docs/design/mcp-oauth-flow.md -> docs/mcp-oauth-flow.md (no customer data)
- enrollment-status-reclassification{,-plan}.md -> tasks/ (git-ignored work
files; they contained a customer domain that must not ship in the public repo)
- remove now-empty docs/design/ folder
Repoint the 22 source files whose comments referenced
docs/design/enrollment-status-reclassification.md to the new tasks/ path.
Comment/docstring text only; no functional code changed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oliver Kieselbach