Sample Windows.System.Profile.SystemSetupInfo.OutOfBoxExperienceState
(build >= 17763, SYSTEM-callable, CLR-native WinRT projection — no NuGet)
as a purely observational signal, mirroring the bootstrap guard marker:
- oobeStateAtAgentStart on agent_started and agent_late_start payloads:
makes late starts provable instead of heuristic ("completed" on a
first_run = OOBE was already over when the agent arrived) and validates
the bootstrap OOBE guard fleet-wide.
- One-shot oobe_state_completed trace event on the in_progress->completed
flip, piggybacked on the DesktopArrivalDetector 30s poll: an
owner-independent desktop corroboration for DAD-starved sessions
(4d5a0b78 class — WTS+WMI owner resolution dead, desktop_arrived never
fires). Often absent in healthy sessions since the DAD timer is
disposed at desktop arrival, which is when the flip lands.
OobeStateReader is fail-soft (never throws, "unavailable" on missing
contract); the flip check has its own try/catch so it cannot pollute the
DAD wmiErrorCount liveness diagnostics, "unavailable" never overwrites
the baseline, and the one-shot flag survives ResetForRealUserSwitch.
No decision-engine involvement, no config/backend changes. Event type
registered in Constants.EventTypes, MCP resource-catalog (phase_events)
and web gather-rule autocomplete.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Six integration tests execute the generated cleanup script with a real
powershell.exe against GUID-unique scratch trees — proving what the string
pins cannot:
- locked tree ⇒ complete abort, marker + binaries survive (the 2026-07-22
regression: the old script force-deleted around the locks)
- full incident replay: a renamed second process holds a file in the tree,
is found by NAME, force-killed after the wait deadline, cleanup then
completes with nothing left behind
- fresh foreign lock ⇒ loser backs off without touching the tree; stale
lock (>10min) is taken over and cleanup proceeds
- happy path + keepLogs variant
BuildCleanupScript gains test-only timing parameters (defaults unchanged
for production) so the replays run in seconds. Agent suite 1969/1969.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Incident 2026-07-22: the 48h emergency break left exe + loaded DLLs orphaned
forever — a second agent instance held the images while the cleanup script
(waiting on the launcher PID only) force-deleted marker + state around the
locks AFTER removing the Scheduled Task, destroying the retry path.
Agent — cleanup script (LIFE-F7):
- wait for ALL agent processes by name (PID-reuse-proof), then force-kill
stragglers; single-flight lock file serializes concurrent cleanup scripts
- probe-rename (GUID target) BEFORE task removal; any failure renames back
and aborts with task + marker + files intact for a next-boot retry; the
destructive Remove-Item-on-original fallback is gone
- script generation extracted to BuildCleanupScript, pinned by 13 tests;
all 4 variants parser-validated + E2E-smoked (happy path, locked tree)
Agent — break report delivery:
- EmergencyReporter counts DELIVERED reports: reservation rolls back on
failure (mark-before-send permanently suppressed a key after its first
transport failure); optional in-reservation retries
- break path: NIC wait <=15s, 3 attempts x 15s timeout (survives Functions
cold starts observed as >5s 503s), 75s overall bound
Backend:
- backend-materialized events (agent_emergency_break, session_timeout) now
upsert the cross-session EventType index — they were invisible to every
search-by-eventType surface (portal + MCP)
- new AgentEmergencyBreak ops event (Agent/Warning, once per session) so
operators see silently-lost agents; dual-registered in OPS_EVENT_TYPES
- materialization extracted to a static seam, covered by 4 new tests
Agent 1963/1963, Backend 3198/3198, tsc clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Allowed Local Accounts entries may now contain glob wildcards (* / ?),
matching the hardware-whitelist semantics: Regex.Escape conversion,
case-insensitive, 100ms ReDoS timeout. Applies to both the account and
the profile-folder check. Entries without wildcard chars keep exact
matching; * and ? are invalid in Windows account names, so patterns and
literals cannot collide. UI documents the syntax and rejects patterns
with fewer than 2 literal characters (a bare * would disable the
analyzer). No schema change - old agents treat patterns as literals and
keep alerting (fail-safe).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The first release after the download-alias migration exposed Front Door
caching as a correctness hazard: the alias kept serving the previous
version.json (X-Cache: TCP_REMOTE_HIT) after the origin had rotated to
2.0.1370, pinning the portal's version badge to the old build. The dangerous
variant is worse than a stale badge -- manifest and ZIP are independent cache
entries, so a stale manifest paired with a fresh ZIP fails the bootstrap's
SHA-256 integrity check and the install dies.
Route caching on /agent/* has been disabled (infra side); this commit is the
defense-in-depth half: every upload now carries x-ms-blob-cache-control
no-cache -- the CI Send-Blob (eu and legacy mirror alike), the local build.ps1
counterpart (gitignored, changed in lockstep), and the versions/*.json
manifest publishes. Uniform no-cache rather than an immutable/mutable split
because every blob in the agent container rotates in place, including the
per-line AutopilotMonitor-Agent-v2.zip -- nothing there is safely cacheable.
Existing blobs get the header with the next release's re-upload.
Log/docs updated in url-registry.md; no code paths changed, YAML and ps1
parse-checked.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An interactive logon during the RealmJoin first deployment (phase 100/101)
makes RJ reclassify the run as a secondary-user deployment, whose branch
writes 200/210 and never CompletedFirstDeployment (110) — no crash, no
reboot request. The V2 completion gate waited for exactly 110, so these
sessions hung until the 60-min hard timeout (session 224b2087; siblings
558bf067/3cc18724 delayed ~50 min the same way).
- promote realmjoin_phase_changed to a typed RealmJoinPhaseChanged signal
(dual-emitted by the adapter) so RealmJoinFacts.LastDeploymentPhase is
persisted across agent restarts
- abort rule: prev in {100,101} and current in {200,210} with no outcome
yet -> emit realmjoin_first_deployment_incomplete (Warning), cancel the
hard-timeout deadline, release the gate with outcome
FirstDeploymentIncomplete; also evaluated on the RealmJoinDetected
replay path (restart swallows the transition)
- deliberately NOT triggered by 101->0 (RJ service restart may still
complete the first deployment) nor by a first observation of 210
(session 6f1959c0: RJ already completed before agent boot)
- fix realmjoin_timeout always claiming "last phase: 0" - the persisted
phase now feeds the message
- backend EnrollmentTimeoutClassifier counts the new event as a gate
terminal; MCP resource catalog dual-synced
- CompleteIfDeferredOrBookkeep widened to a leading-effect list
Verified by replaying session 224b2087's signal log: gate releases at
15:05:17 (101->200) instead of hanging until the 15:54 deadline.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every well-known host -- own subdomains, both blob accounts, the API origin,
Graph, Entra login -- now lives in exactly one registry per component:
Constants.cs (*BaseUrl) for C#, utils/config.ts for the web app, src/config.ts
for the MCP server. lib/hostRouting.ts derives its hostnames from the registry
and middleware.ts imports from it, retiring what turned out to be a fourth
independent copy of the www/portal host pair.
Three new guard tests (backend xUnit, web vitest, MCP vitest) scan non-test
source and fail on any enforced host outside its registry. This is the lesson
of the EU cutover: the registry existed then too, but LatestVersionsService
(feeds the portal's version badge) and ValidateBootstrapCodeFunction carried
their own literal copies of the legacy blob host and were missed -- kept
working only by the fail-soft legacy mirror in build-agent.yml, whose silent
failure would have pinned badge and self-update to stale bits with a green
release. Both guards proved themselves on their first run, flagging the
middleware host copy and a literal in an MCP tool description. Test files stay
exempt on purpose: literals there are independent oracles, and
ValidateBootstrapCodeResponseShapeTests now pins the full alias URL against
the Constants-built one so registry changes must be acknowledged consciously.
The two legacy readers plus the agent SelfUpdater now use
Constants.AgentDownloadBaseUrl (download.autopilotmonitor.com/agent); the
alias was verified live for version.json, version-v2.json and both ZIPs
before the switch. The portal's bootstrap allow-list accepts the alias AND
the legacy host (AGENT_DOWNLOAD_HOSTNAMES) for already-deployed customer
scripts, which imposes a deploy order: web before or with backend. The
legacy keepalive probes and the release mirror stay untouched -- they exist
for exactly this transition. Also: the bootstrap short-link now targets www
directly instead of riding the registrar's apex 301, and the welcome-mail
docs button points at docs. instead of bouncing through www./docs.
Suites: backend 3194, agent core 1925, DecisionCore 474, web 582, MCP 449 --
all green; next build verified (it, not tsc, proves the next.config.ts
registry import). Docs: docs/url-registry.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All three components now report <major>.<minor>.<build> plus a 7-char commit
and a build timestamp. The backend had been pinned at a hand-written 1.0.0
since inception and starts at 1.5.1000; the MCP server's package.json version
was bumped by hand and starts at 1.5.300, and it now reports its own commit on
/health instead of only the documentation bundle's.
The build number keeps coming from a counter blob reserved before the build
with ETag-CAS, but there is now one implementation instead of two copies:
.github/scripts/Request-BuildNumber.ps1, called by all three workflows and by
the local build.ps1. It lives under .github/ because scripts/Deployment/ is
gitignored (those scripts carry SAS tokens) and would therefore be absent from
a CI checkout; scripts/Build/ and scripts/Release/ are shadowed by the
[Bb]uild/ and [Rr]elease/ artifact ignores.
The counter cannot be replaced by the published manifest, which is the obvious
simplification to reach for: version.json is written only on stable publish, so
two consecutive dev builds would derive the same number and the second would
overwrite the first's artifact. git rev-list --count fails the same way, two
builds of one commit collide.
Both deploy workflows now poll their health endpoint until it reports exactly
the version and commit just deployed, and publish versions/<component>.json
only then. A green deploy step alone means the package was accepted, not that
it is being served -- so a manifest that disagrees with the live endpoint now
means the running instance drifted (rollback, stale revision) rather than a
pipeline that stopped halfway. The agent manifest gains the same commit and
buildUtc fields additively; the bootstrap reads it by field name, so older
installers ignore them, and sha256/bootstrapVersion stay untouched.
Requires the versions container plus AZURE_BLOB_VERSIONS_SAS_TOKEN and the two
seeded counters (backend 999, mcp 299) before the first deploy run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GetExcessiveDataSendersAsync gated on LastEventAt - StartedAt alone. No event
count, no bytes, no rate — the name asserted something the code never checked.
What it actually selected was every enrollment session that spanned a night,
which for Autopilot is the ordinary case: device unboxed in the evening, ESP
runs, machine powered off, user signs in the next morning.
The per-device alerts from the previous commit made this visible. All three
devices blocked at the time of the analysis were false positives:
CND3320949 / 64ed21e8 23 events over 15.5h (1.5 events/hour)
5CG4303CPT / db5751bc 192 events over 16.8h (blocked mid Account Setup 1/5)
PF5YANFB / 2069de87 781 events over 31.5h
14 such blocks in 30 days across 9 customer tenants. Over the same window the
event-count detector fired 5 times and was right every time (2742, 2714, 2712,
2506 events). Each false block costs a live enrollment 8 hours of telemetry.
Repairing it rather than removing it does not pay: once a volume condition is
added, "many events AND long span" is a subset of "many events", which is what
the count detector already does. Volume is the quantity that costs anything; a
23-event session spanning 15 hours costs nothing. So the detector, its two admin
settings, the ExcessiveDataBlocked ops event and the portal section go, and the
event-count path is the only automatic block left.
MaxSessionWindowHours and MaintenanceBlockDurationHours are deleted rather than
deprecated: the config row is written with Replace, so the columns drop out on
the next portal save, and nothing reads them meanwhile. A comment on the
surviving detector records why there is no second one, so it does not get
rebuilt a year from now.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The two runaway detectors emitted alerts of unequal usefulness. The event-count
path fires per session and carries sessionId + serialNumber, so the Ops Events
detail modal can offer View session / Block / Kill. The time-window path
aggregated a whole tenant into one event with {devicesBlocked, windowHours} —
no device, no session, no shortcuts, on precisely the alert where an operator
wants to act. It now writes one event per blocked device with the same payload
shape; the aggregate count stays in the audit entry, where a sum is the useful
view.
The Active Blocks table showed the serial as plain text although the API already
returns BlockedSessionIds. Maintenance blocks now link the serial to the session
that triggered them; manual whole-device blocks have no session and stay text.
Two defects in the event-count path, found while comparing the detectors:
- No isBlocked guard, so the auto-action re-blocked an already-blocked device and
overwrote BlockedAt/UnblockAt/Reason of the existing block — usually the
time-window block on the same session, whose reason then vanished from the list.
- No status filter, so a session that finished days ago with a high EventCount
still cost its device a 24h block. The fix needed a layer below the obvious one:
the runaway query never selected Status, and SessionStatus.InProgress is
ordinal 0, so a guard on the unpopulated field would have been silently
always-true. ParseStatusForAutoAction is stricter than MapToSessionSummary --
Enum.TryParse maps "0" onto InProgress and ORs comma-separated names into an
unrelated member, both of which would open the gate.
Not addressed, and deliberately: a block is session-scoped and stops the upload,
so whichever detector fires first freezes the other for that session. With
MaxSessionWindowHours=12 the time detector wins in practice. That is correct
behaviour, not a defect; merging the detectors would be its own change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CodeQL analyzes C# and TypeScript, so a commit touching only docs or markdown
cannot change its result - it spent ~10 minutes (8:50 for C#, 1:32 for
TypeScript) re-reporting an unchanged outcome.
paths-ignore on push and pull_request only. The weekly cron stays unfiltered:
it exists to catch advisories published against code that did not change, and
it doubles as the backstop should the filter ever skip too much.
A commit touching docs AND code still runs - GitHub skips only when every
changed path matches the filter.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The header claimed it "runs the full test suite" - it also typechecks the web
and MCP projects and fails on drift between rules/guardrails.json and the
generated rule files. It also called itself "the gate that decides whether a
change is allowed to be correct", which was both garbled and false: main is
deliberately unprotected, so a red run reports rather than blocks.
Comment only, no behaviour change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CI test job surfaced 10 annotations, but GitHub caps them at 10 per job —
a full rebuild showed 31 nullable warnings plus 26 xUnit analyzer warnings.
Production code:
- CertificateValidator: X509Certificate2(byte[]) is obsolete (SYSLIB0057) ->
X509CertificateLoader.LoadCertificate.
- AuthenticationMiddleware: IsValidTenantId now carries [NotNullWhen(true)], so
the guard above BuildTenantAuthority is visible to flow analysis.
- AgentEndpointMigrationRules: the out params are genuinely nullable on the
false path; netstandard2.0 has no usable [NotNullWhen], so the contract is
documented instead.
- Constants.ClassifyEspAppsFailure: errorCode is optional by design (the
"no HRESULT" branch is the timeout fallback) - signature says so now.
Tests: nullable annotations on helper signatures, Assert.Single/DoesNotContain
overloads instead of pre-filtering with Where, and [InlineData(null)] theories
declare nullable parameters. GatherRule helpers now seed Parameters with the
model default (empty) rather than null, matching a deserialized rule.
No behaviour change. 5616 tests pass; solution builds warning-free apart from
the two pre-existing NU1510 pruning notices on the pinned security packages.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The integrations list named start, success, failure and SLA alerts but omitted
hardware rejection, understating what tenants can actually subscribe to.
Verified before claiming it: WebhookNotifyOnHardwareRejection is wired
end-to-end - model, portal toggle in SectionHardwareWhitelist, and dispatch in
ReportDistressFunction. This list occurs nowhere else in the repo or in
autopilotmonitor-docs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The guard functions had unit tests, but nothing proved that Registry, Wmi,
Json, Xml and File actually consult them before touching the registry, WMI or
disk. These drive the collectors themselves: each blocked target must come
back empty, emit exactly one security_warning naming the target, and produce
no rule output. Prefix-spoofing included, plus a positive case through
JsonCollector so it is pinned that the guard is a gate and not a wall.
Path traversal had no explicit coverage either. The sharpest case is
...\Logs\..\..\..\..\Secret.txt - the RAW string starts with an allowed
prefix, so a naive StartsWith would admit it; the test pins that
Path.GetFullPath canonicalizes first and the allowlist sees C:\Secret.txt.
Also traversal into the C:\Users privacy block and into System32\config, both
asserted under unrestrictedMode, which lifts the allowlist but never those.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The sanitizer matched \p{Cc}, which covers the C0/C1 control ranges but not
U+2028 LINE SEPARATOR or U+2029 PARAGRAPH SEPARATOR - those are categories
Zl/Zp. A log viewer, a JSON-lines consumer or any JavaScript-side reader still
breaks a line on them, so they were a residual log-forging vector.
Matching \p{Zl} and \p{Zp} rather than listing escapes: the two categories
contain exactly those two characters, and it keeps the source free of
invisible literals.
Verified the test is not vacuous - the old pattern left "a<U+2028>b" as "a b",
the new one yields "ab".
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The tenant contact address had no server-side validation, so a direct API
caller could store any string in the field service notices are sent to.
UpdateTenantConfiguration now rejects non-addresses, recipient lists (a comma
would silently widen who receives those notices), display-name forms and
control characters (mail-header injection), and normalizes what it stores.
Empty stays valid - having no contact address is a legitimate state.
The seed path had a worse problem than missing validation. It was a
read-modify-write of the whole model through SaveTenantConfigurationAsync,
which upserts with TableUpdateMode.Replace and no ETag: a concurrent portal
save was clobbered by the seeder's stale snapshot of EVERY field, not just
ContactEmail. It also wrote straight through IConfigRepository, bypassing the
TenantConfigurationService cache, so a seeded address stayed invisible for the
full 5-minute TTL.
TrySeedTenantContactEmailAsync now writes ETag-conditionally in Merge mode -
one property, and a 412 means someone else wrote the row so the seed abandons
rather than overwriting the winner. A lost seed is recoverable on the next
maintenance run; a lost admin edit is not.
The same seed logic existed twice (preview notification-email save and the
maintenance backfill). Both now call a single owner on
TenantConfigurationService, which holds the never-overwrite invariant, the
cache eviction and the fail-soft guarantee. Fail-soft lives in the service
rather than the repository so it holds for any implementation.
Tests cover the validator, the service contract, and the repository against a
mocked TableClient: Merge mode, the ETag actually passed, only ContactEmail in
the patch, the 412 race, 404, storage faults. Plus a regression pin that
SaveTenantConfigurationAsync is never called from the seed path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow-up to 479247cd. Publishing only a genuine UPN closed the fleet-wide bucket
collapse but opened a narrower hole: AuthenticationMiddleware validates issuer,
audience, lifetime and signature but requires neither upn nor scp, so an app-only
(client-credentials) token authenticates and reaches every AuthenticatedUser route
(feedback, realtime/negotiate, progress/*, health/detailed, auth/mcp, ...). Such a
token carries no upn, so it resolved to an empty identity and UserRateLimitMiddleware
skipped it entirely — previously it at least landed in the shared anonymous bucket.
Rather than reinstating a placeholder as a bucket key, separate the two concepts:
- RequestContext.UserPrincipalName keeps its meaning — a real UPN or nothing. Audit,
presence and admin flows continue to see only genuine user identities.
- RequestContext.CallerId is the throttle identity: UPN, else the token subject
(oid -> sub -> appid/azp), else a fail-closed shared bucket. Empty means exactly one
thing — no JWT at all (device/anonymous routes, which carry their own limits).
- UserRateLimitMiddleware gates and buckets on CallerId.
This matches how McpQuotaEnforcementMiddleware already keys its quota (oid), so the
subject-based identity is the existing house convention for enforcement.
10 new tests: the oid/sub/appid/azp fallback chain and its precedence, the fail-closed
bucket for a token with no identifying claim (asserting it is never "anonymous"), and
that empty CallerId occurs only on unauthenticated routes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Device and anonymous routes published the "anonymous" placeholder as
RequestContext.UserPrincipalName. UserRateLimitMiddleware skips only on an EMPTY UPN, so
every agent of every tenant was bucketed under user_ratelimit_anonymous and throttled
against the 120/min per-user limit. On 2026-07-21 21:00-21:30 UTC that rejected 1821
agent requests with 429 (/api/agent/telemetry, config, register-session, error); the
7-day trend shows it recurring and growing with fleet size.
- PolicyEnforcementMiddleware: RequestContext.UserPrincipalName carries a genuine JWT
UPN or nothing. decision.UserIdentifier keeps the placeholder for log lines.
- Device/bootstrap routes already hold their own limit in SecurityValidator (per cert
thumbprint / per bootstrap token), so they lose nothing.
- stats/platform and diagnostics/download had NO limiter of their own and were riding on
the accidental shared bucket. Both now limit per trusted client IP (60/min and 30/min)
via ClientIpExtractor, matching the HealthCheck pattern.
- 13 new tests pin the empty-UPN contract across all 12 device/anonymous routes, plus a
guard that an authenticated caller still surfaces its real UPN.
Solution suite 5552 tests green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Microsoft.Identity.Web 4.3.0 pulls Microsoft.AspNetCore.DataProtection and
System.Security.Cryptography.Xml transitively at 10.0.0, which carry known
advisories (GHSA-9mv3-2cwr-p262 critical on DataProtection; GHSA-37gx-xxp4-5rgx
and siblings, high, on Cryptography.Xml). Add explicit references pinning both
to the patched 10.0.10 servicing build (matching the existing JwtBearer pin) —
the NuGet equivalent of the npm overrides used for the web/MCP alerts.
Pinned in the production Functions project so the fix flows to the deployed
artifact and to Functions.Tests via project reference. `dotnet list package
--vulnerable` now reports 0 across every project; build clean; 3110 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Bucket B (real fixes) + Bucket A (config) of the CodeQL alert triage.
Log-forging (cs/log-forging, 8 sinks): add Helpers/LogSanitizer that strips
CR/LF and other control characters, and apply it to the logged copy of every
user-controlled value in AuthenticationMiddleware, PolicyEnforcementMiddleware
(Invoke + DecideAsync), McpQuotaEnforcementMiddleware and TableUserUsageRepository.
Only the logged value is sanitized — routing, authorization and storage keys are
untouched, so tracing fidelity for legitimate values (paths, UPNs, GUIDs) is
preserved. Structured logging already isolated the fields; this closes the CRLF
vector as defense-in-depth and clears the query.
Three genuine JS/TS fixes:
- guardValidation.ts: replace the no-op `.replace(/%/g, "%")` with a real
escapeRegExp() — the author intended to escape; this also closes the latent
metacharacter gap (js/identity-replacement).
- access-guard.ts: use a `%s` format argument instead of interpolating the UPN
into the console.error format string (js/tainted-format-string).
- docs-corpus.ts: strip HTML tags with a loop-until-stable stripHtmlTags() so the
sanitizer is robust and recognized as complete (js/incomplete-multi-char-sanitization).
CodeQL config: filter the two queries that fire structurally on this codebase,
where the flagged behavior is the feature — cs/exposure-of-sensitive-information
(audit/notification logs legitimately record the actor UPN / recipient email) and
cs/cleartext-storage-of-sensitive-information (rate limiters key on the trusted
client IP). Default suites stay active; genuine one-off false positives from other
queries are dismissed individually so those queries stay sharp.
Tests: Backend xUnit 3110 (+9 LogSanitizer), MCP vitest 19 (+1 tag-strip),
Web vitest 49, all tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The three watchdog tests raced a fixed fanout-body sleep against sub-second
warn/severe thresholds (e.g. 150ms body vs 500ms severe). On a loaded CI
runner the whole RunCoreAsync body overran the severe threshold before the
finally-block cancelled the watchdog, so the severe OpsEvent fired and
"LongRunning past warning" flaked (3100/3101).
Replace the wall-clock race with event gates: the fanout body now blocks on
WaitForEventAsync(<watchdog OpsEvent>) instead of sleeping, so the target
watchdog is guaranteed to have fired before the body returns. The opposite
threshold is pushed to minutes (warn-only tests) or the body waits on the
severe event (both-fire test) so warn is necessarily first. OpsEvents
recording is now lock-guarded since watchdog tasks record concurrently.
No production change. Watchdog class ran 5x green (16/16); full suite 3110/3110.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dependabot PRs run without repository secrets, so AZURE_SWA_DEPLOY_TOKEN is
empty and Azure/static-web-apps-deploy exits with an error, painting a red
"Build and Deploy Job" check on every web-touching Dependabot PR. Setting
skip_deploy_on_missing_secrets makes the action skip the deploy (green) when
the token is missing; real pushes to main still carry the token and deploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Group config already batches minor/patch; majors fall outside the group and
each opens its own PR (zod 4, typescript 7, transformers 4, msal-react 5,
react-leaflet 5, react 19, @types/node 26, eslint-config-next 16 — all
breaking). Ignore version-update:semver-major for both npm ecosystems so
majors are reviewed deliberately instead of flooding the PR list. Security
updates are exempt from this filter, so vulnerability fixes still flow.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Force the security-patched versions of transitive dependencies without
downgrading their parents (next 15, @modelcontextprotocol/sdk 1.28), which
is what `npm audit fix` would otherwise do:
- sharp ^0.35.0 (web via next, mcp via @huggingface/transformers) — libvips CVEs
- fast-uri ^3.1.4 (mcp via ajv) — host confusion
- hono ^4.12.27 (mcp via sdk) — jsx context/XSS/header advisories
- @hono/node-server ^2.0.5 (mcp via sdk)
npm audit: 0 vulnerabilities in both projects. Builds clean; MCP 447 tests
and Web 572 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
LogParserCollector and EventLogCollector called no guard at all, while
File/Json/Xml did — and rules/gather-rules.md already promised customers
that log-parser paths were allowlisted. A tenant admin could author a
logparser rule against C:\Users via POST /api/rules/gather and have up to
1000 lines uploaded as telemetry, bypassing the hard C:\Users block the
same page guarantees; eventlog could read any channel, Security included.
The backend validates no collectorType, target, or parameters, so the
agent is the only place this can be enforced.
Both now guard. logparser cannot check rule.Target directly (GetFullPath
throws on a wildcard), so it checks the directory for wildcard targets and
re-checks each resolved path. Event log channels get an allowlist matched
on a "/" boundary. Hard-blocked channels live in a C# constant rather than
guardrails.json: a parse error drops the allowlists to empty, which is
fail-closed, but would have lifted a block sourced from the same file.
Also fixes an inert timeout in CommandCollector: ReadToEnd() blocks until
the pipe closes, so WaitForExit(30000) was only reached once the process
had already exited and a hung command pinned its worker indefinitely.
Adds the first real CI gate. No workflow ran tests on a pull request — the
deploy workflows build without testing and build-agent.yml covered only
DecisionCore.Tests on manual dispatch — while trust/security-faq.md
already told customers automated tests run in CI. ci.yml now runs all six
suites on every PR and merge, plus a job that fails on generated-guardrail
drift, alongside CodeQL and Dependabot. OfficeBinaryWatcherTests is marked
SerialThreading; its bounded re-probe races the shared ThreadPool and
failed once under full-suite load.
Verified: 3101 backend, 1911 agent V2, 474 decision core, 44 dialog, 572
web, 447 MCP. The timeout fix is verified by runtime, not assertion — a
Start-Sleep 120 command now returns after ~30s.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A German question returns zero results from search_docs, not poor ones: both the
documentation and the rules are English, and all-MiniLM-L6-v2 is an English-only
model, so the query vector lands nowhere near any chunk. For a German-speaking
admin driving an MCP client in their own language, the tool is effectively blind
and gives no hint why.
The caller is itself a language model, so the cheapest correct fix is to say so:
both tool descriptions now instruct it to translate the question to English before
searching and to answer the user in their own language. No infrastructure, no
model change, no ranking impact.
A multilingual embedding model would handle it natively and was considered, but it
is substantially larger — which matters on a scale-to-zero container — and it would
change English retrieval as well, so it would have to be measured against
scripts/eval-docs-search.ts rather than swapped in on the assumption that more
languages cannot hurt.
Verified: 447 tests pass, tsc clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The boot marks answered the question on the first production cold start, and the
answer overturns the reasoning behind an infrastructure change made earlier today.
+244ms node started (startup + module loading)
+431ms corpora read from disk (+187ms)
+491ms index read, validated, hydrated (+60ms for 3.12 MB)
+504ms listening
+677ms query embedder warm (background)
Against a 31.5s cold start, the application accounts for half a second. The other
~98% is Container Apps activation: KEDA reacting, pod scheduling, mounting the
236 MB image, starting the container. The container also boots faster than a
development laptop (504ms vs 1273ms), so nothing here is CPU-starved.
Consequences recorded rather than quietly dropped:
- Raising the container to 0.5 vCPU was justified by the embedding model load on
the cold-start path. Prebaked, that load is 173ms and runs in the background
where only the first search could wait on it. The justification does not survive
measurement. The setting stays — it plausibly helps concurrent requests, and at
scale-to-zero the cost difference is negligible — but the Bicep comment now says
what is true instead of what was assumed, so nobody re-derives the wrong reason.
- Index size is irrelevant to startup (60ms for 3.12 MB), so the corpus can grow
freely, and the embedding rounding earns its keep on file size alone.
- The only lever that would actually remove the cold start is minReplicas: 1, at
roughly 7x the monthly free consumption grant.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A cold start after scale-to-zero measures 24.5s to first byte against 60ms warm,
and nothing outside the process could say why. Log Analytics stamps every console
line of one ingestion batch with the same time, so the boot sequence collapses to
a single instant however it is queried, and piping stdout locally buffers until
exit. So the server times itself: `[boot +Nms]` marks derived from
process.uptime(), which unlike an in-process timer also covers Node's own startup
and module loading.
Measuring the precomputed path locally already contradicts two assumptions worth
recording:
+860ms node started (startup + module loading)
+1199ms corpora read from disk (+339ms)
+1242ms index read, validated, hydrated (+43ms for 3.12 MB)
+1273ms listening (+31ms)
+1497ms query embedder warm (+224ms, background)
The index is 43ms, not seconds — its size is not the cold-start cost, and growing
the corpus is cheap. The prebaked embedder is 224ms and loads in the background,
so only the first search could ever wait on it. That was the stated reason for
raising the container to 0.5 vCPU, which now looks like a weaker justification
than it did; the production marks will settle it. If they show the app listening
after a few seconds, the rest of the 24.5s is platform activation that neither CPU
nor index changes can touch.
Verified: 447 tests pass, marks appear on both boot paths locally. Incidentally
the run also confirmed the corpus hash guard and the staleness signal both work —
the local docs repo has moved to 6333d56 while the deployed image reports d8cb7f4,
so the precomputed index was correctly rejected and recomputed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The form offered trigger, phase scope and emit mode unconditionally, which
produced combinations that cannot do anything plus one silently wrong state:
- A scope on a Phase Start / Phase End rule that names a concrete phase can only
ever suppress the single firing — the phase IS the moment. Same for emit mode:
a one-shot has no second result to dedup against.
- Selecting "From a phase onwards" (or "Only during specific phases") and not
filling the detail dropdown serialised to null, i.e. the rule ran everywhere
while the form still read "From a phase onwards".
supportsPhaseScope / supportsEmitMode now drive visibility; buildScopeFields
nulls hidden-control state so a value from an earlier trigger choice cannot leak
into the payload; validateScopeSelection rejects an unfilled scope mode on both
create and save. Startup is the deliberate exception — a scope there defers the
one-shot instead of suppressing it, so the scope control stays and only emit mode
is hidden. One-shot triggers now state in-form when they collect.
Sub-dropdown hints no longer claim "No selection = all phases" (they turn amber
and name the required action instead).
Tests: app/gather-rules/__tests__/gatherRuleFormLogic.test.ts (15) pins the
applicability matrix, the payload nulling incl. the trigger-switch leak, the
unfilled-scope rejection, and the JSON-mode scopeMode derivation.
vitest 572/572; tsc silent.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live test found a third recall gap. The fallback only topped up when semantic
search returned FEWER than topK results — but the more common failure is a full
list of unconvincing ones. Querying "WOW6432Node" returned three chunks scoring
~0.29, none containing the word, while the single chunk that does was never
considered at all.
The fix is a ranking change, so it was measured rather than reasoned about. Adds
scripts/eval-docs-search.ts, which scores strategies over 2465 queries in three
deliberately different classes: every chunk heading (easy by construction, so a
regression detector), rare identifiers (the class this fallback serves), and
hand-written questions (the reality anchor — not derived from the corpus, so it
shares little vocabulary with it). Embeddings are computed once and each strategy
is a pure function over the same candidates, so sweeping a threshold costs no
extra inference.
known-item literal handwritten MRR fixed broken
append-only (before) 71.9% 66.2% 1.000 - -
weak < 0.30 71.9% 72.6% 1.000 142 0
weak < 0.35 73.7% 81.1% 1.000 335 0
weak < 0.40 76.3% 88.2% 0.944 502 3
weak < 0.45 77.2% 93.0% 0.889 613 5
0.35 is strictly dominant — better in every class, worse in none — so that is
what ships. Higher thresholds keep repairing identifier queries but start
demoting hand-written questions, the class that most resembles real use, so the
gain stops being free. When the head is weak, exact matches move to the front
while at least one semantic result is kept, so a weak-but-correct hit is never
pushed out entirely.
The measurement also resized the problem: identifier queries were succeeding only
66.2% of the time, so the reported case was one of roughly 750, not an outlier.
That is the part a single anecdote could not have shown, and the reason the
harness exists before the fix rather than after it.
Verified: 447 tests pass, including new cases for the weak path and for leaving a
confident ranking untouched. On the real corpus "WOW6432Node" now returns the
chunk that contains it at rank 1, while "how do I deploy the agent with Intune"
(0.819) and "what firewall URLs do I need to allow" (0.374) are unchanged and the
negative controls stay empty.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Two measured recall gaps in search_docs, both silent.
1. Truncation. MiniLM's context is 256 tokens; 87 of 324 chunks exceeded it and
13% of all tokens never reached the ranking — the tokenizer drops the tail
without complaint. Concretely, in built-in-rules.md :: Apps (734 tokens) the
strings ANALYZE-APP-013, ANALYZE-OFFICE-001, ANALYZE-CORR-003 and Click-to-Run
all sit past the cutoff, so "proxy configuration blocking app downloads" did not
retrieve that section at all.
Shrinking chunks cannot fix this: the corpus ranges 2.43 to 5.81 characters per
token, so no character budget bounds tokens, and a budget safe for the worst case
(~620 chars) would shred prose and tables alike. So the chunk stays whole and the
embedding is split instead — index() cuts long text into 250-token windows with
40 overlap, a document carries embeddings[][], and search() scores it by its BEST
window. Max rather than mean: averaging would blur a long section's several
topics into one mediocre direction, the same dilution that makes large context
windows worse than chunking.
The tokenizer is a build-time dependency only (vocabulary, not ONNX weights,
~93 ms); production boots from precomputed vectors and never loads it.
2. No general fallback. Proper nouns embed poorly — "RealmJoin" occurs in 3
chunks yet scores under the threshold. Fuzzy matching was built, measured and
rejected: at a threshold low enough to surface RealmJoin (0.28), Fuse also
returns 5 hits for "SCEPman", a word this corpus contains zero times. Literal
containment has neither problem, and every provider already exposes it. So a
short result set is now topped up via lexicalMatch on distinctive query terms,
requiring two matching terms as corroboration — without that rule "how do I bake
sourdough bread" matched a page about LLM providers, because "bake" is a
substring of "baked into the image". Keyword hits are appended and flagged, never
sorted together with cosine scores.
Also sharpens the tool description with the measured reason to prefer this over
fetching the docs site (~1k tokens for topK=3 against ~9k raw / ~600k rendered,
version-matched, and matching on meaning), and replaces the growing tail of
positional docs parameters with a DocsSearchBundle.
Verified: token coverage 87% -> 100%; "Office Click-to-Run install never
finished" 0.318 -> 0.442; "proxy configuration blocking app downloads" absent ->
0.559 at rank 1; the ten reference queries keep byte-identical scores and rank 1
(short chunks are one window, so the same vector as before — no long-chunk bias).
The fallback is not a dead path: across 904 rare corpus tokens queried alone it
contributes hits for 431, while negative controls stay empty. Precompute output
is accepted by the boot validator and a mutated corpus still rejected; index
2.63 -> 3.12 MB for 324 chunks / 424 vectors. The rules corpus gains the same
treatment unplanned: 128 rules now carry 172 vectors. 445 tests pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The MCP server could answer telemetry questions ("why did this session fail")
but not product ones ("how do I deploy the agent", "where is my data stored").
Those answers already exist in the autopilotmonitor-docs bundle; search_docs
makes them semantically searchable.
Deliberately a separate index from search_knowledge: rules and prose answer
different questions, and sharing one topK would dilute both. Registered for
every role — published documentation carries no tenant data — and skipped
entirely when no corpus was baked in, so a doc-less build advertises no
broken tool.
Corpus handling:
- Chunked at h2/h3 and <details>/<summary>, each chunk prefixed with a
"Page > Section > Subsection" breadcrumb so a mid-page hit is self-describing.
- Newlines are normalized before any parsing. This is a correctness
precondition, not cosmetics: the bundle ships CRLF and JS `.` does not match
\r, so /^(#{2,3})\s+(.*)$/ matched zero headings — silently, degrading every
page to unlabelled paragraph splits.
- Two size exceptions: tables are never split (a split table leaves `| --- |`
rows with no header), and unbroken changelog lists split at line level (left
whole, their tails fall outside the model window and become unsearchable).
- Short sections are kept — a one-line answer is often the best hit, and the
breadcrumb supplies the context a short body lacks.
Build and delivery:
- Docs are embedded at image build time into an optional third
PrecomputedSection. Optional so a checkout without the docs repo does not
invalidate the rules and event-type vectors; the reverse (server has docs,
file does not) is staleness and is rejected.
- The deploy workflow checks the public docs repo into the build context and
passes its SHA as DOCS_COMMIT, which /health reports alongside the chunk
count — editing the docs does not update a running server, so staleness is
made checkable rather than memorable.
- Embeddings are rounded to 6 decimals on serialize: 55% smaller index (2.63
vs 5.88 MB) for a ~1e-5 norm error against a 1e-3 tolerance. indexPrecomputed
now asserts unit norm, since the hydrate path never checked it.
Infra: the Bicep template was two changes behind the live deployment (EU
cutover and ACR hardening, both applied directly in Azure). Reconciled against
the running resources — managed-identity image pull replaces the admin
credentials and the acr-password secret — and infra/ is no longer gitignored so
it stops drifting. Container raised to 0.5 vCPU / 1.0 GiB for the cold-start
query-embedder load. The template does not model the mcp.autopilotmonitor.com
custom domain, so a full deployment would remove it; the header warns about it.
Verified: 428 tests pass; 45 pages produce 324 chunks; sample queries hit the
right page top-1/2; all 8 derived URLs return 200 live; the precompute output
is accepted by the boot validator and a mutated corpus is rejected.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The section was registered for routing but not in the sidebar, so it was
reachable only by typing the URL. Sidebar entries come from
globalNavConfig.tsx, which is not derived from tenantNavSections.ts -
both have to be touched for a section to exist.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Enforcement and service notices need somewhere to go. Until now the only
address on file was the Private Preview notification email, which is
onboarding data in the PreviewWhitelist table and not something a tenant
can maintain.
Adds TenantConfiguration.ContactEmail, owned and edited by the tenant's
own admins under Settings -> Tenant -> Contact:
- Seeded once when a preview notification address is saved, and only
while ContactEmail is still empty. It is never re-synced, so an edit in
the portal cannot be overwritten later by the onboarding source.
- Backfilled for tenants onboarded before the field existed, as a
manual-trigger maintenance task alongside the other backfills. It skips
any tenant that already has an address, so it converges to a no-op and
can be dropped once everyone is covered.
- Surfaced read-only in Admin -> Tenants next to the preview address, so
an operator who needs to warn a tenant does not have to read the table.
Purpose is stated where it is entered and in the Privacy Policy: service
matters only - a technical problem, a security matter, or a change
needing an administrator's attention. Never marketing, never shared.
The Terms already say a warning is sent "where a tenant contact address
is on file"; this is the field that makes that promise meaningful.
Wired through the full tenant-config round-trip (types, context state,
hydrate, save payload, reset, provider) so a save from any other section
cannot clobber it.
Verified: solution builds clean, 3101 backend tests and 557 web tests
pass, tsc --noEmit clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Intellectual Property & Licensing section: the repository licence
governs the code, not the hosted service, its infrastructure, its data,
or the name and branding. Customer data stays with the customer.
- Acceptable Use now states the enforcement ladder (warning, reduced rate
limits, suspension, termination) instead of an unqualified right to
suspend, and routes security research to GitHub Security Advisories
with explicit testing boundaries.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Privacy: "Sub-processors and Outbound Data Flows" becomes "External
Services". It still names who receives data, which is a controller's own
duty, but drops the Art. 28 vocabulary and the promise to announce new
sub-processors - that commitment lives in the data processing agreement.
Points at the renamed docs page and notes the DPA is available on
request.
Terms: the MSP clause now says a managing organization is responsible for
having the necessary data protection agreements with its customers,
instead of classifying it as a processor or sub-processor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dependabot alerts #63/#64/#65 (all high): brace-expansion exponential-time
expansion of consecutive non-expanding {} groups (GHSA-3jxr-9vmj-r5cp,
both the 1.x and 2.x trees) and js-yaml quadratic CPU consumption via
YAML merge-key chains (GHSA-52cp-r559-cp3m).
All three are transitive dev-only dependencies of the eslint toolchain
(eslint -> @eslint/eslintrc -> js-yaml; eslint/typescript-estree ->
minimatch -> brace-expansion) and resolve within their existing semver
ranges, so only the lockfile changes.
npx tsc --noEmit clean, vitest 557 passed, npm audit reports 0
vulnerabilities.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Dependabot alert #62 (protobufjs 7.6.4, DoS via infinite loop in .proto
option parsing, GHSA fixed in 7.6.5) plus the tar (critical) and
body-parser (low) advisories npm audit surfaced alongside it. All three
are transitive under @huggingface/transformers and resolve within their
existing semver ranges, so only the lockfile changes.
Lockfile also picks up the pending 1.5.0 -> 1.5.1 package version sync.
npx tsc --noEmit clean, vitest 400 passed / 41 skipped, npm audit
reports 0 vulnerabilities.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Oliver Kieselbach