Compare commits

...

14 Commits

Author SHA1 Message Date
Oliver Kieselbach a0d10b8471 test(cleanup): end-to-end incident replays for the self-destruct script
Six integration tests execute the generated cleanup script with a real
powershell.exe against GUID-unique scratch trees — proving what the string
pins cannot:

- locked tree ⇒ complete abort, marker + binaries survive (the 2026-07-22
  regression: the old script force-deleted around the locks)
- full incident replay: a renamed second process holds a file in the tree,
  is found by NAME, force-killed after the wait deadline, cleanup then
  completes with nothing left behind
- fresh foreign lock ⇒ loser backs off without touching the tree; stale
  lock (>10min) is taken over and cleanup proceeds
- happy path + keepLogs variant

BuildCleanupScript gains test-only timing parameters (defaults unchanged
for production) so the replays run in seconds. Agent suite 1969/1969.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 10:06:29 +02:00
Oliver Kieselbach ad3632e9fb fix(emergency-break): survivable self-destruct, reliable break report, ops visibility
Incident 2026-07-22: the 48h emergency break left exe + loaded DLLs orphaned
forever — a second agent instance held the images while the cleanup script
(waiting on the launcher PID only) force-deleted marker + state around the
locks AFTER removing the Scheduled Task, destroying the retry path.

Agent — cleanup script (LIFE-F7):
- wait for ALL agent processes by name (PID-reuse-proof), then force-kill
  stragglers; single-flight lock file serializes concurrent cleanup scripts
- probe-rename (GUID target) BEFORE task removal; any failure renames back
  and aborts with task + marker + files intact for a next-boot retry; the
  destructive Remove-Item-on-original fallback is gone
- script generation extracted to BuildCleanupScript, pinned by 13 tests;
  all 4 variants parser-validated + E2E-smoked (happy path, locked tree)

Agent — break report delivery:
- EmergencyReporter counts DELIVERED reports: reservation rolls back on
  failure (mark-before-send permanently suppressed a key after its first
  transport failure); optional in-reservation retries
- break path: NIC wait <=15s, 3 attempts x 15s timeout (survives Functions
  cold starts observed as >5s 503s), 75s overall bound

Backend:
- backend-materialized events (agent_emergency_break, session_timeout) now
  upsert the cross-session EventType index — they were invisible to every
  search-by-eventType surface (portal + MCP)
- new AgentEmergencyBreak ops event (Agent/Warning, once per session) so
  operators see silently-lost agents; dual-registered in OPS_EVENT_TYPES
- materialization extracted to a static seam, covered by 4 new tests

Agent 1963/1963, Backend 3198/3198, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:56:46 +02:00
Oliver Kieselbach 05e1569503 feat(agent): wildcard support for Local Admin Analyzer allowed accounts
Allowed Local Accounts entries may now contain glob wildcards (* / ?),
matching the hardware-whitelist semantics: Regex.Escape conversion,
case-insensitive, 100ms ReDoS timeout. Applies to both the account and
the profile-folder check. Entries without wildcard chars keep exact
matching; * and ? are invalid in Windows account names, so patterns and
literals cannot collide. UI documents the syntax and rejects patterns
with fewer than 2 literal characters (a bare * would disable the
analyzer). No schema change - old agents treat patterns as literals and
keep alerting (fail-safe).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:47:13 +02:00
Oliver Kieselbach ba3581e253 bootstrapper added download stopwatch 2026-07-23 09:35:30 +02:00
Oliver Kieselbach 829ec4c969 fix(release): stamp Cache-Control no-cache on every blob upload
The first release after the download-alias migration exposed Front Door
caching as a correctness hazard: the alias kept serving the previous
version.json (X-Cache: TCP_REMOTE_HIT) after the origin had rotated to
2.0.1370, pinning the portal's version badge to the old build. The dangerous
variant is worse than a stale badge -- manifest and ZIP are independent cache
entries, so a stale manifest paired with a fresh ZIP fails the bootstrap's
SHA-256 integrity check and the install dies.

Route caching on /agent/* has been disabled (infra side); this commit is the
defense-in-depth half: every upload now carries x-ms-blob-cache-control
no-cache -- the CI Send-Blob (eu and legacy mirror alike), the local build.ps1
counterpart (gitignored, changed in lockstep), and the versions/*.json
manifest publishes. Uniform no-cache rather than an immutable/mutable split
because every blob in the agent container rotates in place, including the
per-line AutopilotMonitor-Agent-v2.zip -- nothing there is safely cacheable.
Existing blobs get the header with the next release's re-upload.

Log/docs updated in url-registry.md; no code paths changed, YAML and ps1
parse-checked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:17:02 +02:00
Oliver Kieselbach b2d2d37c6f feat(realmjoin): detect aborted RJ ESP (missing 110) and release the completion gate
An interactive logon during the RealmJoin first deployment (phase 100/101)
makes RJ reclassify the run as a secondary-user deployment, whose branch
writes 200/210 and never CompletedFirstDeployment (110) — no crash, no
reboot request. The V2 completion gate waited for exactly 110, so these
sessions hung until the 60-min hard timeout (session 224b2087; siblings
558bf067/3cc18724 delayed ~50 min the same way).

- promote realmjoin_phase_changed to a typed RealmJoinPhaseChanged signal
  (dual-emitted by the adapter) so RealmJoinFacts.LastDeploymentPhase is
  persisted across agent restarts
- abort rule: prev in {100,101} and current in {200,210} with no outcome
  yet -> emit realmjoin_first_deployment_incomplete (Warning), cancel the
  hard-timeout deadline, release the gate with outcome
  FirstDeploymentIncomplete; also evaluated on the RealmJoinDetected
  replay path (restart swallows the transition)
- deliberately NOT triggered by 101->0 (RJ service restart may still
  complete the first deployment) nor by a first observation of 210
  (session 6f1959c0: RJ already completed before agent boot)
- fix realmjoin_timeout always claiming "last phase: 0" - the persisted
  phase now feeds the message
- backend EnrollmentTimeoutClassifier counts the new event as a gate
  terminal; MCP resource catalog dual-synced
- CompleteIfDeferredOrBookkeep widened to a leading-effect list

Verified by replaying session 224b2087's signal log: gate releases at
15:05:17 (101->200) instead of hanging until the 15:54 deadline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:14:30 +02:00
Oliver Kieselbach e0ad51b965 refactor(urls): one guard-enforced URL registry per component, agent downloads move to the alias
Every well-known host -- own subdomains, both blob accounts, the API origin,
Graph, Entra login -- now lives in exactly one registry per component:
Constants.cs (*BaseUrl) for C#, utils/config.ts for the web app, src/config.ts
for the MCP server. lib/hostRouting.ts derives its hostnames from the registry
and middleware.ts imports from it, retiring what turned out to be a fourth
independent copy of the www/portal host pair.

Three new guard tests (backend xUnit, web vitest, MCP vitest) scan non-test
source and fail on any enforced host outside its registry. This is the lesson
of the EU cutover: the registry existed then too, but LatestVersionsService
(feeds the portal's version badge) and ValidateBootstrapCodeFunction carried
their own literal copies of the legacy blob host and were missed -- kept
working only by the fail-soft legacy mirror in build-agent.yml, whose silent
failure would have pinned badge and self-update to stale bits with a green
release. Both guards proved themselves on their first run, flagging the
middleware host copy and a literal in an MCP tool description. Test files stay
exempt on purpose: literals there are independent oracles, and
ValidateBootstrapCodeResponseShapeTests now pins the full alias URL against
the Constants-built one so registry changes must be acknowledged consciously.

The two legacy readers plus the agent SelfUpdater now use
Constants.AgentDownloadBaseUrl (download.autopilotmonitor.com/agent); the
alias was verified live for version.json, version-v2.json and both ZIPs
before the switch. The portal's bootstrap allow-list accepts the alias AND
the legacy host (AGENT_DOWNLOAD_HOSTNAMES) for already-deployed customer
scripts, which imposes a deploy order: web before or with backend. The
legacy keepalive probes and the release mirror stay untouched -- they exist
for exactly this transition. Also: the bootstrap short-link now targets www
directly instead of riding the registrar's apex 301, and the welcome-mail
docs button points at docs. instead of bouncing through www./docs.

Suites: backend 3194, agent core 1925, DecisionCore 474, web 582, MCP 449 --
all green; next build verified (it, not tsc, proves the next.config.ts
registry import). Docs: docs/url-registry.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 08:37:42 +02:00
Oliver Kieselbach bbb55034e3 feat(versioning): one version contract for agent, backend and MCP
All three components now report <major>.<minor>.<build> plus a 7-char commit
and a build timestamp. The backend had been pinned at a hand-written 1.0.0
since inception and starts at 1.5.1000; the MCP server's package.json version
was bumped by hand and starts at 1.5.300, and it now reports its own commit on
/health instead of only the documentation bundle's.

The build number keeps coming from a counter blob reserved before the build
with ETag-CAS, but there is now one implementation instead of two copies:
.github/scripts/Request-BuildNumber.ps1, called by all three workflows and by
the local build.ps1. It lives under .github/ because scripts/Deployment/ is
gitignored (those scripts carry SAS tokens) and would therefore be absent from
a CI checkout; scripts/Build/ and scripts/Release/ are shadowed by the
[Bb]uild/ and [Rr]elease/ artifact ignores.

The counter cannot be replaced by the published manifest, which is the obvious
simplification to reach for: version.json is written only on stable publish, so
two consecutive dev builds would derive the same number and the second would
overwrite the first's artifact. git rev-list --count fails the same way, two
builds of one commit collide.

Both deploy workflows now poll their health endpoint until it reports exactly
the version and commit just deployed, and publish versions/<component>.json
only then. A green deploy step alone means the package was accepted, not that
it is being served -- so a manifest that disagrees with the live endpoint now
means the running instance drifted (rollback, stale revision) rather than a
pipeline that stopped halfway. The agent manifest gains the same commit and
buildUtc fields additively; the bootstrap reads it by field name, so older
installers ignore them, and sha256/bootstrapVersion stay untouched.

Requires the versions container plus AZURE_BLOB_VERSIONS_SAS_TOKEN and the two
seeded counters (backend 999, mcp 299) before the first deploy run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 17:41:03 +02:00
Oliver Kieselbach 58c8cf87a4 updated command 2026-07-22 15:38:02 +02:00
Oliver Kieselbach 71d2c0e47f fix(maintenance): remove the time-window auto-block, it never measured data volume
GetExcessiveDataSendersAsync gated on LastEventAt - StartedAt alone. No event
count, no bytes, no rate — the name asserted something the code never checked.
What it actually selected was every enrollment session that spanned a night,
which for Autopilot is the ordinary case: device unboxed in the evening, ESP
runs, machine powered off, user signs in the next morning.

The per-device alerts from the previous commit made this visible. All three
devices blocked at the time of the analysis were false positives:

  CND3320949 / 64ed21e8    23 events over 15.5h  (1.5 events/hour)
  5CG4303CPT / db5751bc   192 events over 16.8h  (blocked mid Account Setup 1/5)
  PF5YANFB   / 2069de87   781 events over 31.5h

14 such blocks in 30 days across 9 customer tenants. Over the same window the
event-count detector fired 5 times and was right every time (2742, 2714, 2712,
2506 events). Each false block costs a live enrollment 8 hours of telemetry.

Repairing it rather than removing it does not pay: once a volume condition is
added, "many events AND long span" is a subset of "many events", which is what
the count detector already does. Volume is the quantity that costs anything; a
23-event session spanning 15 hours costs nothing. So the detector, its two admin
settings, the ExcessiveDataBlocked ops event and the portal section go, and the
event-count path is the only automatic block left.

MaxSessionWindowHours and MaintenanceBlockDurationHours are deleted rather than
deprecated: the config row is written with Replace, so the columns drop out on
the next portal save, and nothing reads them meanwhile. A comment on the
surviving detector records why there is no second one, so it does not get
rebuilt a year from now.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 15:28:28 +02:00
Oliver Kieselbach 3b2f5ed46d feat(runaway-detection): name the device in the time-window alert, link it in the block list
The two runaway detectors emitted alerts of unequal usefulness. The event-count
path fires per session and carries sessionId + serialNumber, so the Ops Events
detail modal can offer View session / Block / Kill. The time-window path
aggregated a whole tenant into one event with {devicesBlocked, windowHours} —
no device, no session, no shortcuts, on precisely the alert where an operator
wants to act. It now writes one event per blocked device with the same payload
shape; the aggregate count stays in the audit entry, where a sum is the useful
view.

The Active Blocks table showed the serial as plain text although the API already
returns BlockedSessionIds. Maintenance blocks now link the serial to the session
that triggered them; manual whole-device blocks have no session and stay text.

Two defects in the event-count path, found while comparing the detectors:

- No isBlocked guard, so the auto-action re-blocked an already-blocked device and
  overwrote BlockedAt/UnblockAt/Reason of the existing block — usually the
  time-window block on the same session, whose reason then vanished from the list.
- No status filter, so a session that finished days ago with a high EventCount
  still cost its device a 24h block. The fix needed a layer below the obvious one:
  the runaway query never selected Status, and SessionStatus.InProgress is
  ordinal 0, so a guard on the unpopulated field would have been silently
  always-true. ParseStatusForAutoAction is stricter than MapToSessionSummary --
  Enum.TryParse maps "0" onto InProgress and ORs comma-separated names into an
  unrelated member, both of which would open the gate.

Not addressed, and deliberately: a block is session-scoped and stops the upload,
so whichever detector fires first freezes the other for that session. With
MaxSessionWindowHours=12 the time detector wins in practice. That is correct
behaviour, not a defect; merging the detectors would be its own change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:43:00 +02:00
Oliver Kieselbach 7329f12217 ci(codeql): skip analysis on documentation-only changes
CodeQL analyzes C# and TypeScript, so a commit touching only docs or markdown
cannot change its result - it spent ~10 minutes (8:50 for C#, 1:32 for
TypeScript) re-reporting an unchanged outcome.

paths-ignore on push and pull_request only. The weekly cron stays unfiltered:
it exists to catch advisories published against code that did not change, and
it doubles as the backstop should the filter ever skip too much.

A commit touching docs AND code still runs - GitHub skips only when every
changed path matches the filter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:10:18 +02:00
Oliver Kieselbach c9a9df99f1 docs(ci): describe what the CI workflow actually does
The header claimed it "runs the full test suite" - it also typechecks the web
and MCP projects and fails on drift between rules/guardrails.json and the
generated rule files. It also called itself "the gate that decides whether a
change is allowed to be correct", which was both garbled and false: main is
deliberately unprotected, so a red run reports rather than blocks.

Comment only, no behaviour change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:10:17 +02:00
Oliver Kieselbach 757427ffea chore(build): clear the compiler and xUnit analyzer warnings CI annotates
The CI test job surfaced 10 annotations, but GitHub caps them at 10 per job —
a full rebuild showed 31 nullable warnings plus 26 xUnit analyzer warnings.

Production code:
- CertificateValidator: X509Certificate2(byte[]) is obsolete (SYSLIB0057) ->
  X509CertificateLoader.LoadCertificate.
- AuthenticationMiddleware: IsValidTenantId now carries [NotNullWhen(true)], so
  the guard above BuildTenantAuthority is visible to flow analysis.
- AgentEndpointMigrationRules: the out params are genuinely nullable on the
  false path; netstandard2.0 has no usable [NotNullWhen], so the contract is
  documented instead.
- Constants.ClassifyEspAppsFailure: errorCode is optional by design (the
  "no HRESULT" branch is the timeout fallback) - signature says so now.

Tests: nullable annotations on helper signatures, Assert.Single/DoesNotContain
overloads instead of pre-filtering with Where, and [InlineData(null)] theories
declare nullable parameters. GatherRule helpers now seed Parameters with the
model default (empty) rather than null, matching a deserialized rule.

No behaviour change. 5616 tests pass; solution builds warning-free apart from
the two pre-existing NU1510 pruning notices on the pinned security packages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:09:11 +02:00
132 changed files with 3015 additions and 713 deletions
+102
View File
@@ -0,0 +1,102 @@
<#
.SYNOPSIS
Reserves a monotone build number from a counter blob (ETag compare-and-swap).
.DESCRIPTION
Single source of build numbers for every component that ships from this repo
(agent, backend, MCP). Each component owns one counter blob holding a plain
integer; this script reads it, writes value+1 back under If-Match, and returns
the reserved number.
Reserve-BEFORE-build: two concurrent builds can never mint the same number.
A failed build burns a number, which is harmless -- uniqueness matters,
density does not.
The blob is public-read; only the write needs the container SAS.
Why a counter and not the published version manifest: manifests are written
on publish, not on build (the agent writes version.json only for stable
releases), so two dev builds in a row would derive the same number and
overwrite each other's versioned artifact.
.PARAMETER CounterUrl
Full URL of the counter blob, without SAS.
.PARAMETER SasToken
Container SAS with create+write permission. A leading '?' is tolerated.
.PARAMETER Override
Explicit build number instead of counter+1. If it is not greater than the
current counter the blob is left untouched (re-build of a number that was
already reserved). 0 = no override.
.PARAMETER MaxAttempts
CAS retries before giving up.
.OUTPUTS
[int] the reserved build number. Progress goes to the host, not the pipeline.
.EXAMPLE
$n = ./Request-BuildNumber.ps1 -CounterUrl $url -SasToken $sas
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)][string]$CounterUrl,
[Parameter(Mandatory = $true)][string]$SasToken,
[int]$Override = 0,
[int]$MaxAttempts = 5
)
$ErrorActionPreference = 'Stop'
# Tolerate both secret formats ("sp=cw&..." and "?sp=cw&...").
$writeUrl = $CounterUrl + '?' + $SasToken.TrimStart('?')
function Get-HttpStatus {
param($ErrorRecord)
if ($ErrorRecord.Exception.PSObject.Properties['Response'] -and $ErrorRecord.Exception.Response) {
return [int]$ErrorRecord.Exception.Response.StatusCode
}
return 0
}
function Read-Counter {
try {
$resp = Invoke-WebRequest -Uri $CounterUrl -Method Get -UseBasicParsing
} catch {
if ((Get-HttpStatus $_) -eq 404) {
throw "Counter blob not found at $CounterUrl. Seed it once with the last used build number: PUT the plain integer with 'x-ms-blob-type: BlockBlob' and 'If-None-Match: *'. See docs/versioning.md."
}
throw
}
$raw = if ($resp.Content -is [byte[]]) { [System.Text.Encoding]::UTF8.GetString($resp.Content) } else { [string]$resp.Content }
# If-Match requires the quoted ETag form; PowerShell strips the quotes on read.
$etag = '"' + ([string]($resp.Headers['ETag'] | Select-Object -First 1)).Trim('"') + '"'
return [pscustomobject]@{ Value = [int]$raw.Trim(); ETag = $etag }
}
for ($attempt = 1; $attempt -le $MaxAttempts; $attempt++) {
$counter = Read-Counter
if ($Override -gt 0 -and $Override -le $counter.Value) {
Write-Host " Build number $Override <= counter $($counter.Value) -- already reserved, counter blob unchanged"
return $Override
}
$target = if ($Override -gt 0) { $Override } else { $counter.Value + 1 }
try {
$headers = @{ 'x-ms-blob-type' = 'BlockBlob'; 'If-Match' = $counter.ETag }
Invoke-RestMethod -Uri $writeUrl -Method Put -Headers $headers -Body "$target" -ContentType 'text/plain' | Out-Null
Write-Host " Reserved build number $target (CAS write ok, attempt $attempt)"
return $target
} catch {
if ((Get-HttpStatus $_) -eq 412) {
Write-Host " CAS conflict on attempt $attempt (another build reserved concurrently) -- retrying"
Start-Sleep -Seconds ($attempt * 2)
} else {
throw
}
}
}
throw "Could not reserve a build number after $MaxAttempts CAS attempts"
+20 -61
View File
@@ -64,70 +64,19 @@ jobs:
with:
dotnet-version: '10.0.x'
# Build numbers come from the shared counter blob (single source of truth for CI
# AND local build.ps1). Reserve-before-build with ETag-CAS: concurrent builds can
# never mint the same number; a failed build burns a number, which is harmless.
# The blob is public-read (like the rest of the container); writes need the SAS.
# Build numbers come from the shared counter blob -- one counter and one
# implementation for CI and the local build (.github/scripts/Request-BuildNumber.ps1).
# Reserve-before-build with ETag-CAS: concurrent builds can never mint the same
# number; a failed build burns a number, which is harmless.
- name: Reserve build number (counter blob CAS)
id: version
shell: pwsh
run: |
$counterBlob = "buildcounter-v$env:LINE_MAJOR.txt"
$counterUrl = "https://autopilotmonitoreu.blob.core.windows.net/agent/$counterBlob"
# Tolerate both secret formats ("sp=cw&..." and "?sp=cw&...") -- the URL below adds the '?'.
$sasToken = "${{ secrets.AZURE_BLOB_AGENT_SAS_TOKEN }}".TrimStart('?')
$override = "${{ inputs.build_number }}"
$override = 0
if ('${{ inputs.build_number }}' -ne '') { $override = [int]'${{ inputs.build_number }}' }
function Get-Counter {
try {
$resp = Invoke-WebRequest -Uri $counterUrl -Method Get
$raw = if ($resp.Content -is [byte[]]) { [System.Text.Encoding]::UTF8.GetString($resp.Content) } else { [string]$resp.Content }
# If-Match requires the quoted ETag form; pwsh strips the quotes on read.
$etag = '"' + ([string]($resp.Headers['ETag'] | Select-Object -First 1)).Trim('"') + '"'
return @{ Value = [int]$raw.Trim(); ETag = $etag }
} catch {
$status = if ($_.Exception.Response) { [int]$_.Exception.Response.StatusCode } else { 0 }
if ($status -eq 404) {
Write-Error "Counter blob '$counterBlob' not found. Seed it once with the last used build number (see docs/agent/build-counter-blob.md) -- do NOT seed from the repo's buildcounter.txt, it is a stale offline fallback."
}
throw
}
}
$buildNumber = $null
for ($attempt = 1; $attempt -le 5; $attempt++) {
$counter = Get-Counter
if ($override -ne '') {
$target = [int]$override
if ($target -le $counter.Value) {
# Re-build of an already-reserved number: counter stays monotonic, no write.
Write-Host "Manual build number $target <= counter $($counter.Value) -- counter blob unchanged"
$buildNumber = $target
break
}
Write-Host "Manual build number $target > counter $($counter.Value) -- advancing counter"
} else {
$target = $counter.Value + 1
}
try {
$headers = @{ 'x-ms-blob-type' = 'BlockBlob'; 'If-Match' = $counter.ETag }
Invoke-RestMethod -Uri "$counterUrl`?$sasToken" -Method Put -Headers $headers -Body "$target" -ContentType 'text/plain' | Out-Null
$buildNumber = $target
Write-Host "Reserved build number ${target} (CAS write ok, attempt $attempt)"
break
} catch {
$status = if ($_.Exception.Response) { [int]$_.Exception.Response.StatusCode } else { 0 }
if ($status -eq 412) {
Write-Host "CAS conflict on attempt $attempt (another build reserved concurrently) -- retrying"
Start-Sleep -Seconds ($attempt * 2)
} else {
throw
}
}
}
if ($null -eq $buildNumber) {
Write-Error "Could not reserve a build number after 5 CAS attempts"
}
$counterUrl = "https://autopilotmonitoreu.blob.core.windows.net/agent/buildcounter-v$($env:LINE_MAJOR).txt"
$buildNumber = & ./.github/scripts/Request-BuildNumber.ps1 -CounterUrl $counterUrl -SasToken '${{ secrets.AZURE_BLOB_AGENT_SAS_TOKEN }}' -Override $override
$csproj = Get-Content $env:CSPROJ_PATH -Raw
if ($csproj -match '<VersionPrefix>(\d+\.\d+)</VersionPrefix>') {
@@ -228,8 +177,13 @@ jobs:
Write-Warning "Could not parse bootstrap `$ScriptVersion from $env:BOOTSTRAP_SOURCE"
}
# Enrich the per-line manifest emitted by MSBuild (version-v2.json, read above)
# Enrich the per-line manifest emitted by MSBuild (version-v2.json, read above).
# commit/buildUtc are the same two fields the backend and MCP manifests carry —
# additive, and the bootstrap reads this file by field name, so older installers
# ignore them. sha256/bootstrapVersion stay: those are the customer contract.
$manifest | Add-Member -NotePropertyName "sha256" -NotePropertyValue $sha256Hash -Force
$manifest | Add-Member -NotePropertyName "commit" -NotePropertyValue '${{ github.sha }}'.Substring(0, 7) -Force
$manifest | Add-Member -NotePropertyName "buildUtc" -NotePropertyValue ([DateTime]::UtcNow.ToString('o')) -Force
if ($bootstrapVersion) {
$manifest | Add-Member -NotePropertyName "bootstrapVersion" -NotePropertyValue $bootstrapVersion -Force
}
@@ -277,7 +231,12 @@ jobs:
param([string]$Local, [string]$BlobName, [string]$ContentType)
$url = "$containerUrl/$BlobName`?$sasToken"
$bytes = [System.IO.File]::ReadAllBytes($Local)
$headers = @{ "x-ms-blob-type" = "BlockBlob"; "Content-Type" = $ContentType }
# no-cache on EVERY blob: they all rotate in place (even the per-line ZIP is
# overwritten each build), and the download alias fronts them with Front Door.
# Route caching is disabled there, but a stale manifest paired with a fresh ZIP
# (independent cache entries) would fail the bootstrap SHA check -- this header
# keeps that class of bug impossible even if caching is ever re-enabled.
$headers = @{ "x-ms-blob-type" = "BlockBlob"; "Content-Type" = $ContentType; "x-ms-blob-cache-control" = "no-cache" }
if ($ContentType -eq "application/zip") {
$md5 = [System.Security.Cryptography.MD5]::Create()
$headers["Content-MD5"] = [Convert]::ToBase64String($md5.ComputeHash($bytes))
+7 -3
View File
@@ -1,8 +1,12 @@
name: CI
# Runs the full test suite on every pull request and every push to main.
# The deploy workflows build but do not test — this is the gate that decides
# whether a change is allowed to be correct.
# Tests, typechecks and drift-checks every push to main and every pull request:
# the .NET suites, the web and MCP suites, and a guard that the generated rule
# files still match rules/guardrails.json.
#
# The deploy workflows build but never test, so this is the only place the suites
# actually run. It does not gate anything — main is deliberately unprotected, so a
# red run reports rather than blocks. Read the result and follow up.
on:
push:
+15 -1
View File
@@ -1,12 +1,26 @@
name: CodeQL
# Documentation-only changes cannot affect what CodeQL analyzes (C# and
# TypeScript), so they are filtered out rather than spending ~10 minutes to
# re-report an unchanged result. A commit that touches docs AND code still runs:
# GitHub skips only when EVERY changed path matches the filter.
on:
push:
branches: [main]
paths-ignore:
- 'docs/**'
- '**/*.md'
- 'LICENSE'
pull_request:
branches: [main]
paths-ignore:
- 'docs/**'
- '**/*.md'
- 'LICENSE'
schedule:
# Weekly, so advisories published after a merge still surface.
# Weekly, so advisories published after a merge still surface. Deliberately
# unfiltered — it re-analyzes the whole tree regardless of what changed, which
# is also the backstop if the filters above ever skip too much.
- cron: '17 3 * * 1'
permissions:
+84 -4
View File
@@ -7,6 +7,12 @@ permissions:
id-token: write # OIDC federated login (secret-less)
contents: read
env:
FUNCTION_APP: autopilotmonitor-api-eu
HEALTH_URL: https://autopilotmonitor-api-eu.azurewebsites.net/api/health
VERSIONS_CONTAINER: https://autopilotmonitoreu.blob.core.windows.net/versions
CSPROJ_PATH: src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj
jobs:
build-and-deploy:
runs-on: ubuntu-latest
@@ -27,17 +33,91 @@ jobs:
with:
dotnet-version: '10.0.x'
# Same counter mechanism as the agent line: reserve-before-build with ETag-CAS
# against a per-component counter blob. BuildTimestampUtc is pinned here rather
# than left to MSBuild's default so the published manifest and the value baked
# into the assembly (reported by /api/health) are the same instant.
- name: Reserve build number (counter blob CAS)
id: version
shell: pwsh
run: |
$buildNumber = & ./.github/scripts/Request-BuildNumber.ps1 -CounterUrl "${{ env.VERSIONS_CONTAINER }}/backend.txt" -SasToken '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'
$csproj = Get-Content ${{ env.CSPROJ_PATH }} -Raw
if ($csproj -notmatch '<VersionPrefix>(\d+\.\d+)</VersionPrefix>') {
throw "VersionPrefix not found in ${{ env.CSPROJ_PATH }}"
}
$version = "$($Matches[1]).$buildNumber"
$buildUtc = [DateTime]::UtcNow.ToString('o')
Write-Host "Backend version: $version (built $buildUtc)"
"build_number=$buildNumber" >> $env:GITHUB_OUTPUT
"version=$version" >> $env:GITHUB_OUTPUT
"build_utc=$buildUtc" >> $env:GITHUB_OUTPUT
- name: Restore dependencies
run: dotnet restore src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj
run: dotnet restore ${{ env.CSPROJ_PATH }}
- name: Build
run: dotnet build src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj --configuration Release --no-restore -p:SourceRevisionId=${{ github.sha }}
run: dotnet build ${{ env.CSPROJ_PATH }} --configuration Release --no-restore -p:SourceRevisionId=${{ github.sha }} -p:BuildNumber=${{ steps.version.outputs.build_number }} -p:BuildTimestampUtc=${{ steps.version.outputs.build_utc }}
- name: Publish
run: dotnet publish src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj --configuration Release --output ./publish --no-build -p:SourceRevisionId=${{ github.sha }}
run: dotnet publish ${{ env.CSPROJ_PATH }} --configuration Release --output ./publish --no-build -p:SourceRevisionId=${{ github.sha }} -p:BuildNumber=${{ steps.version.outputs.build_number }} -p:BuildTimestampUtc=${{ steps.version.outputs.build_utc }}
- name: Deploy to Azure Functions
uses: Azure/functions-action@v1
with:
app-name: autopilotmonitor-api-eu
app-name: ${{ env.FUNCTION_APP }}
package: ./publish
# A green deploy step only means the package was accepted. This gate makes the
# workflow assert what is actually being served, so the manifest published below
# can be trusted as "this is live" rather than "this was uploaded".
# /api/health is IP rate-limited at 30/min; one poll per 10s stays well under.
- name: Verify deployed build is live
shell: pwsh
run: |
$expectedCommit = '${{ github.sha }}'.Substring(0, 7)
$expectedVersion = '${{ steps.version.outputs.version }}'
$deadline = [DateTime]::UtcNow.AddMinutes(3)
$attempt = 0
while ([DateTime]::UtcNow -lt $deadline) {
$attempt++
try {
$health = Invoke-RestMethod -Uri '${{ env.HEALTH_URL }}' -Method Get -TimeoutSec 30
if ($health.commitHash -eq $expectedCommit -and $health.version -eq $expectedVersion) {
Write-Host "Live after $attempt attempt(s): $($health.version) @ $($health.commitHash), built $($health.buildUtc)"
exit 0
}
Write-Host "Attempt ${attempt}: serving $($health.version) @ $($health.commitHash) -- waiting for $expectedVersion @ $expectedCommit"
} catch {
Write-Host "Attempt ${attempt}: /api/health not answering yet -- $($_.Exception.Message)"
}
Start-Sleep -Seconds 10
}
throw "Backend did not report $expectedVersion @ $expectedCommit within 3 minutes -- deployment not verified, version manifest NOT published"
# The manifest is the cross-component record of what is deployed where. It is
# written last and only on a verified deploy, so a difference between the
# manifest and /api/health means the running app drifted after the fact
# (rollback, stale instance) rather than a half-finished pipeline.
- name: Publish version manifest
shell: pwsh
run: |
$manifest = [ordered]@{
component = 'backend'
version = '${{ steps.version.outputs.version }}'
commit = '${{ github.sha }}'.Substring(0, 7)
buildUtc = '${{ steps.version.outputs.build_utc }}'
deployedUtc = [DateTime]::UtcNow.ToString('o')
runId = '${{ github.run_id }}'
} | ConvertTo-Json
$sas = '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'.TrimStart('?')
$url = "${{ env.VERSIONS_CONTAINER }}/backend.json?$sas"
# no-cache: rotating pointer, must never be served stale by any fronting cache.
Invoke-RestMethod -Uri $url -Method Put -Headers @{ 'x-ms-blob-type' = 'BlockBlob'; 'x-ms-blob-cache-control' = 'no-cache' } -Body $manifest -ContentType 'application/json' | Out-Null
Write-Host "Published versions/backend.json"
Write-Host $manifest
+82
View File
@@ -12,6 +12,9 @@ env:
CONTAINER_APP_NAME: autopilotmonitor-mcp
RESOURCE_GROUP: ${{ secrets.AZURE_MCP_RESOURCE_GROUP }}
IMAGE_TAG: ${{ github.sha }}
HEALTH_URL: https://mcp.autopilotmonitor.com/health
VERSIONS_CONTAINER: https://autopilotmonitoreu.blob.core.windows.net/versions
PACKAGE_JSON: src/McpServer/autopilot-monitor-mcp/package.json
jobs:
build-and-deploy:
@@ -44,6 +47,30 @@ jobs:
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
# Same counter mechanism as agent and backend: reserve-before-build with
# ETag-CAS against the component's own counter blob. major.minor stays curated
# in package.json; only the build part is minted here.
- name: Reserve build number (counter blob CAS)
id: version
shell: pwsh
run: |
$buildNumber = & ./.github/scripts/Request-BuildNumber.ps1 -CounterUrl "${{ env.VERSIONS_CONTAINER }}/mcp.txt" -SasToken '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'
$pkgVersion = (Get-Content ${{ env.PACKAGE_JSON }} -Raw | ConvertFrom-Json).version
$prefix = ($pkgVersion -split '\.')[0..1] -join '.'
$version = "$prefix.$buildNumber"
$buildUtc = [DateTime]::UtcNow.ToString('o')
Write-Host "MCP version: $version (built $buildUtc)"
# 7 chars, matching what the backend's BackendBuildInfo reports — one commit
# shape across all components so the values can be compared as-is.
$shortSha = '${{ github.sha }}'.Substring(0, 7)
"build_number=$buildNumber" >> $env:GITHUB_OUTPUT
"version=$version" >> $env:GITHUB_OUTPUT
"build_utc=$buildUtc" >> $env:GITHUB_OUTPUT
"short_sha=$shortSha" >> $env:GITHUB_OUTPUT
- name: Login to ACR
run: az acr login --name ${{ env.ACR_NAME }}
@@ -52,6 +79,9 @@ jobs:
docker build \
-f src/McpServer/autopilot-monitor-mcp/Dockerfile \
--build-arg DOCS_COMMIT=${{ steps.docs.outputs.sha }} \
--build-arg BUILD_NUMBER=${{ steps.version.outputs.build_number }} \
--build-arg BUILD_COMMIT=${{ steps.version.outputs.short_sha }} \
--build-arg BUILD_UTC=${{ steps.version.outputs.build_utc }} \
-t ${{ env.ACR_NAME }}.azurecr.io/${{ env.CONTAINER_APP_NAME }}:${{ env.IMAGE_TAG }} \
-t ${{ env.ACR_NAME }}.azurecr.io/${{ env.CONTAINER_APP_NAME }}:latest \
.
@@ -63,3 +93,55 @@ jobs:
--name ${{ env.CONTAINER_APP_NAME }} \
--resource-group ${{ env.RESOURCE_GROUP }} \
--image ${{ env.ACR_NAME }}.azurecr.io/${{ env.CONTAINER_APP_NAME }}:${{ env.IMAGE_TAG }}
# `az containerapp update` returns once the revision is provisioned, not once
# it serves traffic — and with minReplicas=0 the first request also pays a cold
# start. Polling /health until it reports THIS build is what makes the manifest
# below mean "live"; a full 5 minutes because the cold start alone is ~25s.
- name: Verify deployed build is live
shell: pwsh
run: |
$expectedCommit = '${{ steps.version.outputs.short_sha }}'
$expectedVersion = '${{ steps.version.outputs.version }}'
$deadline = [DateTime]::UtcNow.AddMinutes(5)
$attempt = 0
while ([DateTime]::UtcNow -lt $deadline) {
$attempt++
try {
$health = Invoke-RestMethod -Uri '${{ env.HEALTH_URL }}' -Method Get -TimeoutSec 60
if ($health.commit -eq $expectedCommit -and $health.version -eq $expectedVersion) {
Write-Host "Live after $attempt attempt(s): $($health.version) @ $($health.commit), docs $($health.docs.commit) ($($health.docs.chunks) chunks)"
exit 0
}
Write-Host "Attempt ${attempt}: serving $($health.version) @ $($health.commit) -- waiting for $expectedVersion @ $expectedCommit"
} catch {
Write-Host "Attempt ${attempt}: /health not answering yet -- $($_.Exception.Message)"
}
Start-Sleep -Seconds 15
}
throw "MCP did not report $expectedVersion @ $expectedCommit within 5 minutes -- deployment not verified, version manifest NOT published"
# Cross-component record of what is deployed where; written last and only on a
# verified deploy. docsCommit rides along because for this component "which
# build" and "which documentation" are two separate questions.
- name: Publish version manifest
shell: pwsh
run: |
$manifest = [ordered]@{
component = 'mcp'
version = '${{ steps.version.outputs.version }}'
commit = '${{ steps.version.outputs.short_sha }}'
buildUtc = '${{ steps.version.outputs.build_utc }}'
deployedUtc = [DateTime]::UtcNow.ToString('o')
runId = '${{ github.run_id }}'
docsCommit = '${{ steps.docs.outputs.sha }}'.Substring(0, 7)
} | ConvertTo-Json
$sas = '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'.TrimStart('?')
$url = "${{ env.VERSIONS_CONTAINER }}/mcp.json?$sas"
# no-cache: rotating pointer, must never be served stale by any fronting cache.
Invoke-RestMethod -Uri $url -Method Put -Headers @{ 'x-ms-blob-type' = 'BlockBlob'; 'x-ms-blob-cache-control' = 'no-cache' } -Body $manifest -ContentType 'application/json' | Out-Null
Write-Host "Published versions/mcp.json"
Write-Host $manifest
+16 -19
View File
@@ -22,26 +22,21 @@ attestation on top.
# Schema
- **Counter blob**: `https://autopilotmonitor.blob.core.windows.net/agent/buildcounter-v{LINE}.txt`
(per release line, analogous to `version-v{LINE}.json`). Content: the **last used**
build number as plain text. Public-read like the rest of the container (the number
is visible in `version-v{LINE}.json` anyway); writes require the container SAS.
- **Protocol — reserve before build, ETag-CAS**:
1. Anonymous `GET` the blob, capture value `N` and the `ETag`.
2. `PUT N+1` with `If-Match: <etag>` (quoted form — pwsh strips the quotes on read,
they must be re-added or the request is rejected client-side).
3. `412 Precondition Failed` → another build reserved concurrently → re-read, retry
(5 attempts, linear backoff).
4. Build with `BUILD_NUMBER=N+1` in the environment. The csproj honors
`BUILD_NUMBER` and then skips its `PersistBuildCounter` target, so the repo file
is not touched.
- **Failed builds burn a number.** That is intentional: uniqueness matters, density
does not. Reserve-before-build is what makes concurrent minting impossible.
The reserve-before-build CAS protocol itself is shared with the backend and the MCP
server and is documented once in [Version Contract](../versioning.md); the single
implementation is `.github/scripts/Request-BuildNumber.ps1`. Agent-specific:
- **Counter blob**: `https://autopilotmonitoreu.blob.core.windows.net/agent/buildcounter-v{LINE}.txt`
— one per release line, analogous to `version-v{LINE}.json`, and kept in the `agent`
container next to the artifacts it numbers rather than in `versions/`. Content: the
**last used** build number as plain text. Public-read like the rest of the container
(the number is visible in `version-v{LINE}.json` anyway); writes require the SAS.
- The reserved number reaches the build as `BUILD_NUMBER` in the environment. The
csproj honors it and then skips its `PersistBuildCounter` target, so the repo file
is not touched.
- **CI manual override** (`build_number` input): numbers `<=` counter are re-builds
and leave the blob unchanged; numbers `>` counter advance it (monotonic guard).
- **Seeding** (one-time per line): `PUT <last used number>` with `If-None-Match: *`
so an existing counter can never be clobbered. V2 was seeded with `1356` on
2026-07-17.
- V2 was seeded with `1356` on 2026-07-17.
# Consequences
@@ -62,7 +57,9 @@ attestation on top.
# Citations
- [Version Contract](../versioning.md) — the shared CAS protocol and manifest format
- `.github/workflows/build-agent.yml` — "Reserve build number (counter blob CAS)" step
- `scripts/Deployment/build.ps1` (gitignored, contains SAS) — `Request-BuildNumber`
- `.github/scripts/Request-BuildNumber.ps1` — the shared implementation
- `scripts/Deployment/build.ps1` (gitignored, contains SAS) — local build path
- `src/Agent/AutopilotMonitor.Agent.V2/AutopilotMonitor.Agent.V2.csproj` —
`BUILD_NUMBER` property + `PersistBuildCounter` target
+11 -2
View File
@@ -137,8 +137,17 @@ false` and an arm holds), `EspExiting`, `ImeUserSessionCompleted`,
advisory backstop. All routes converge on `CompleteThroughFinalizingOrDefer`:
1. **Completion gates** (currently only the RealmJoin AND-gate): if RealmJoin was
detected and is neither resolved nor timed-out (hard 60-min timeout), completion is
*deferred* — the session stays in stage and emits `completion_waiting`.
detected and has no terminal outcome yet, completion is *deferred* — the session
stays in stage and emits `completion_waiting`. Three terminals open the gate:
`Resolved` (DeploymentPhase 110), `FirstDeploymentIncomplete` (phase left the
first-deployment window 100/101 for 200/210 without 110 — the RJ ESP was aborted,
typically by an interactive logon reclassifying the run as secondary-user; emits
the `realmjoin_first_deployment_incomplete` Warning, session 224b2087), and
`Timeout` (hard 60-min deadline). The abort rule reads the *persisted*
`RealmJoinFacts.LastDeploymentPhase` (updated by the typed `RealmJoinPhaseChanged`
signal), so it survives agent restarts — the `RealmJoinDetected` replay path
evaluates it too. A first observation of 210 does **not** release (RJ may have
completed before agent boot and deploy afterwards).
2. `Finalizing` stage + ~5 s `FinalizingGrace` deadline.
3. Grace fires → `Completed`, `EnrollmentComplete` outcome, `enrollment_complete`
timeline event with full audit payload.
+2
View File
@@ -12,6 +12,8 @@ https://docs.autopilotmonitor.com (separate repository).
# Architecture
* [Architecture Guide](architecture.md) - High-level architecture overview and solution layout for contributors.
* [Version Contract](versioning.md) - One version shape for agent, backend and MCP: the ETag-CAS counter blob that mints build numbers, and the manifests published only after a deploy is verified live.
* [URL Registry](url-registry.md) - Every well-known host lives in one registry file per component (Constants.cs / utils/config.ts / src/config.ts), enforced by guard tests; includes the agent-download migration to the download alias and its deploy-order constraint.
# Agent
+10
View File
@@ -1,7 +1,17 @@
# Log
## 2026-07-23
* **Update**: `url-registry.md` — the first release after the alias migration surfaced Front Door caching as a correctness hazard: the alias served the previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) after the origin had rotated, and since manifest and ZIP are independent cache entries, a stale manifest with a fresh ZIP fails the bootstrap SHA-256 check. Route caching on `/agent/*` is now disabled and every blob upload stamps `Cache-Control: no-cache` (CI `Send-Blob`, local `build.ps1`, `versions/*.json` manifests) — all agent-container blobs rotate in place, so nothing there is safely cacheable.
* **Update**: `agent/decision-engine.md` — the RealmJoin completion gate has a third terminal, `FirstDeploymentIncomplete`. Session 224b2087 showed why: an interactive logon 17 s after DeploymentPhase 101 makes RealmJoin reclassify the run as a secondary-user deployment, whose branch writes 200/210 and never 110 — the gate stayed closed until the 60-min hard timeout although Hello and desktop were long resolved. `realmjoin_phase_changed` is now a typed `RealmJoinPhaseChanged` signal, so `RealmJoinFacts.LastDeploymentPhase` is persisted across agent restarts (this also fixes every `realmjoin_timeout` event claiming "last phase: 0"); a transition out of 100/101 into 200/210 releases the gate with the `realmjoin_first_deployment_incomplete` Warning, and the `RealmJoinDetected` replay path evaluates the same rule for restarts that swallow the transition. A first observation of 210 deliberately does not release — session 6f1959c0 started at 210 and only deployed afterwards.
* **Creation**: Added `url-registry.md` — all well-known hosts (own subdomains, both blob accounts, API origin, Graph, Entra login) are now defined once per component (`Constants.cs`, web `utils/config.ts`, MCP `src/config.ts`) and referenced everywhere else; three guard tests fail the build on any literal outside the registries. Written because the EU cutover missed two hardcoded copies of the legacy blob host — `LatestVersionsService` (feeds the portal's version badge) and `ValidateBootstrapCodeFunction` — which kept working only through the fail-soft legacy mirror in `build-agent.yml`; those two plus the agent `SelfUpdater` now use the `download.autopilotmonitor.com` alias (verified serving all four artifacts first). The portal's bootstrap allow-list accepts alias AND legacy host during the transition, which imposes a deploy order: web before (or with) backend. The sweep also surfaced a fourth copy of the www/portal host pair in `middleware.ts` (now derived from the registry via `hostRouting.ts`) and a backend short-link built on the bare apex domain, which only worked through a registrar-level 301.
## 2026-07-22
* **Creation**: Added `versioning.md` — agent, backend and MCP now share one version shape (`<major>.<minor>.<build>` + 7-char commit + build timestamp) and one implementation of the build-number reservation (`.github/scripts/Request-BuildNumber.ps1`, previously copy-pasted between `build-agent.yml` and `build.ps1`). The backend had been pinned at a hand-written `1.0.0` since inception and now starts at `1.5.1000`; the MCP server's `package.json` version was bumped by hand and now starts at `1.5.300`, and it reports its own commit on `/health` instead of only the documentation bundle's. Both deploy workflows poll their health endpoint until it reports the build just deployed and publish `versions/<component>.json` only then, so a manifest that disagrees with the live endpoint means the instance drifted rather than the pipeline failing halfway. `agent/build-counter-blob.md` keeps the agent-specific parts and now points at the shared protocol; its counter URL was still the pre-cutover storage account.
* **Creation**: Added `rules/gather-rule-guardrails.md` — the gather-rule allowlists are enforced only on the agent (the backend validates no `collectorType`, `target`, or `parameters`), so a collector that forgets to call its guard *is* the vulnerability. Written because two had: `LogParserCollector` read any path in `rule.Target`, bypassing even the `C:\Users` hard block that every other file collector honours, and `EventLogCollector` read any channel including `Security` — while the customer-facing documentation already promised both were allowlisted. Both now guard, with `logparser` checking the directory for wildcard targets (`Path.GetFullPath` throws on `*`) and re-checking each resolved path. Added an event-log channel allowlist (`eventLogChannels`) plus hard-blocked channels held in C# rather than JSON, so a parse error cannot lift them. Also fixed the command timeout, which was inert: `ReadToEnd()` blocks until the pipe closes, so `WaitForExit(30000)` was only reached after the process had already exited and a hung command pinned its worker forever — now read asynchronously and killed at the timeout.
* **Update**: Added `.github/workflows/ci.yml` — the full suite (backend, agent V2 Core, DecisionCore, SummaryDialog, web, MCP) now runs on every pull request and every push to main, plus a job that fails if the generated guardrail files drift from `rules/guardrails.json`. Previously **no** workflow ran tests on a PR: the deploy workflows build without testing, and `build-agent.yml` tested only `DecisionCore.Tests` on manual dispatch — while `trust/security-faq.md` already told customers that automated tests run in CI. Added CodeQL (C# and TypeScript) and Dependabot (nuget, npm ×2, actions). Marked `OfficeBinaryWatcherTests` `[Collection("SerialThreading")]`: its bounded re-probe races the shared ThreadPool and failed once under full-suite load.
+85
View File
@@ -0,0 +1,85 @@
---
type: Concept
title: URL Registry — One Place per Component, Guard-Enforced
description: Every well-known own or Microsoft host lives in exactly one registry file per component (Constants.cs, utils/config.ts, src/config.ts); guard tests fail the build on any literal outside them. Includes the agent-download host migration to the download alias.
resource: /src/Shared/AutopilotMonitor.Shared/Constants.cs
tags:
- architecture
- urls
- ci
- convention
timestamp: 2026-07-23T00:00:00+02:00
---
# URL Registry
Every well-known base URL — own hosts (portal, www, docs, download alias, MCP,
API, both blob accounts) and Microsoft hosts (Graph, Entra login) — is defined
in exactly one registry file per component. Code references the constant; the
literal appears nowhere else.
| Component | Registry | Guard test |
|---|---|---|
| C# (Shared/Backend/Agent) | `AutopilotMonitor.Shared.Constants` (`*BaseUrl`) | `HardcodedUrlGuardTests` |
| Web | `utils/config.ts` (`DOCS_URL`, `SITE_URL`, `PORTAL_URL`, `ENTRA_LOGIN_URL`, `API_URL_PROD`, `BLOB_URL_PROD`, `AGENT_DOWNLOAD_HOSTNAMES`) | `utils/__tests__/hardcodedUrls.guard.test.ts` |
| MCP | `src/config.ts` (`API_BASE_URL`, `ENTRA_LOGIN_BASE_URL`, `DOCS_BASE_URL`) | `src/__tests__/hardcoded-urls.guard.test.ts` |
Derived registries build on these instead of repeating hosts:
`lib/hostRouting.ts` computes `PUBLIC_HOST`/`PORTAL_HOST`/`APEX_HOST` from the
URL constants, and `middleware.ts` imports from it.
# Schema
The guards scan non-test source for the enforced host list and fail with
file:line on any occurrence outside the registry. Comment lines are ignored
(docs may cite URLs). Test files are excluded on purpose: a literal in a test
is an independent oracle — `ValidateBootstrapCodeResponseShapeTests` spells the
full agent download URL out so a registry change must be acknowledged in two
places, backend and portal allow-list, consciously.
Why enforced rather than agreed: the registry existed before (both
`Constants.AgentBlobBaseUrl` and the MCP `config.ts` header document earlier
drift), yet the EU cutover still missed two literal copies of the blob host —
`LatestVersionsService` and `ValidateBootstrapCodeFunction` kept reading the
legacy account, silently kept alive only by the fail-soft legacy mirror in
`build-agent.yml`. A convention that is not checked decays; these guards make
the next stray literal a red build instead of a production surprise.
# Agent-download host migration (2026-07-23)
The two legacy readers above plus the agent's `SelfUpdater` now use
`Constants.AgentDownloadBaseUrl` (`download.autopilotmonitor.com/agent`, Front
Door in front of the current blob origin — verified serving `version.json`,
`version-v2.json` and both ZIPs before the switch). The portal's
`bootstrapValidation.ts` allow-lists BOTH the alias and the legacy blob host
(`AGENT_DOWNLOAD_HOSTNAMES`) because bootstrap scripts already deployed in
customer Intune tenants still carry legacy URLs.
Deploy-order constraint: the web allow-list must be live BEFORE (or with) the
backend that serves alias URLs — an old portal build rejects the alias host and
the bootstrap flow dies client-side. Remove the legacy hostname from the
allow-list only after the legacy account is torn down.
**Front Door caching is a correctness hazard here, not a staleness nuisance.**
The first release after the migration exposed it: the alias kept serving the
previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) while the origin was already
updated — and because manifest and ZIP are independent cache entries, a stale
manifest paired with a fresh ZIP fails the bootstrap's SHA-256 check. Two
defenses, both in place since 2026-07-23: route caching on `/agent/*` is
DISABLED, and every blob upload (CI `Send-Blob`, local `build.ps1`, the
`versions/*.json` manifests) stamps `Cache-Control: no-cache` — all these blobs
rotate in place, even the per-line ZIP. If caching is ever re-enabled, the
headers keep the mismatch class impossible.
Deliberately still on the legacy account: the `HealthCheckService` /
`MaintenanceService` legacy-keepalive probes and the fail-soft mirror in
`build-agent.yml` — they exist precisely to keep already-deployed customer
bootstrap scripts working until the customer migration completes.
# Citations
- `src/Shared/AutopilotMonitor.Shared/Constants.cs` — C# registry
- `src/Web/autopilot-monitor-web/utils/config.ts` — web registry
- `src/McpServer/autopilot-monitor-mcp/src/config.ts` — MCP registry
- `src/Backend/AutopilotMonitor.Functions.Tests/HardcodedUrlGuardTests.cs`
- [Version Contract](versioning.md) — the release pipeline this migration protects
+108
View File
@@ -0,0 +1,108 @@
---
type: Concept
title: Version Contract — Agent, Backend, MCP
description: One version shape for all three shipping components, the ETag-CAS counter blob that mints build numbers, and the published manifests that make "what is deployed" checkable from outside.
resource: /.github/scripts/Request-BuildNumber.ps1
tags:
- build
- ci
- release
- versioning
timestamp: 2026-07-22T00:00:00+02:00
---
# Version Contract
Three components ship from this repository — the agent, the Functions backend and the
MCP server. They share one version shape, one way of minting build numbers, and one
manifest format, so that a version string means the same thing whoever emits it.
```
version = <major>.<minor>.<build> major.minor curated by hand, build minted per build
commit = 7-char short SHA of this repo
buildUtc = ISO-8601 build moment
```
| Component | major.minor from | Counter blob | Reported by |
|---|---|---|---|
| Agent V2 | `VersionPrefix` in `AutopilotMonitor.Agent.V2.csproj` | `agent/buildcounter-v2.txt` | `agent/version.json` |
| Backend | `VersionPrefix` in `AutopilotMonitor.Functions.csproj` | `versions/backend.txt` | `GET /api/health` |
| MCP | `version` in `package.json` | `versions/mcp.txt` | `GET /health`, MCP handshake |
# Schema
## Build numbers — reserve before build, ETag-CAS
`.github/scripts/Request-BuildNumber.ps1` is the only implementation; the CI
workflows and the local `build.ps1` all call it.
1. Anonymous `GET` on the counter blob → value `N` and `ETag`. The blob holds the
**last used** number as plain text and is public-read; only the write needs the
container SAS.
2. `PUT N+1` with `If-Match: <etag>`. The quoted ETag form is required — PowerShell
strips the quotes on read, and an unquoted `If-Match` is rejected client-side.
3. `412 Precondition Failed` → a concurrent build reserved first → re-read and retry
(5 attempts, linear backoff).
4. Build with the reserved number.
Reserving *before* the build is what makes concurrent minting impossible. **A failed
build burns a number** — deliberate: uniqueness matters, density does not.
Why a counter and not the published manifest: manifests are written on publish, not on
build. The agent writes `version.json` only for stable releases, so deriving the next
number from it would hand two consecutive dev builds the same number and let the second
overwrite the first's artifact. A state-free scheme such as `git rev-list --count HEAD`
fails for the same reason — two builds of one commit collide.
**Seeding** (once per component): `PUT <last used number>` with `If-None-Match: *`, so
an existing counter can never be clobbered. Agent V2 was seeded with `1356` on
2026-07-17; `versions/backend.txt` with `999` and `versions/mcp.txt` with `299` on
2026-07-22.
## Local builds
No counter is reached, the build number stays `0`, and the commit is empty or
`unknown`. A `1.5.0` with no commit hash on `/api/health` is therefore a developer
machine, not a release — the fallback is the signal, not a defect. The agent is the
exception: `build.ps1` produces uploadable artifacts and reserves a real number unless
`-SkipUpload` is passed.
## Manifests
`versions/backend.json` and `versions/mcp.json` (public-read) record what is deployed:
```json
{ "component": "backend", "version": "1.5.1000", "commit": "abc1234",
"buildUtc": "…", "deployedUtc": "…", "runId": "…" }
```
The MCP manifest adds `docsCommit`, because for that component "which build" and "which
documentation" are separate questions.
Each is written **last and only after the deploy workflow has polled the component's
health endpoint until it reports that exact version and commit**. A green deploy step
alone means the package was accepted, not that it is being served. Consequently a
manifest that disagrees with the live endpoint means the running instance drifted after
the fact (rollback, stale revision) rather than a half-finished pipeline.
The agent keeps its own manifest at `agent/version.json` — the bootstrap derives that
URL from the download URL and verifies the ZIP against its `sha256`, so it cannot move.
It carries the same `commit` and `buildUtc` fields; `sha256` and `bootstrapVersion`
remain, being the customer-facing integrity contract.
# Consequences
- Bumping `major.minor` is a human decision, made in the component's own file
(`VersionPrefix`, or `version` in `package.json`). Nothing bumps it automatically.
- A new component joins by seeding a counter blob and calling the same script; no new
mechanism.
- The counter blobs and manifests live in the `versions` container on
`autopilotmonitoreu` (writes via `AZURE_BLOB_VERSIONS_SAS_TOKEN`). The agent's
counter stays in the `agent` container next to the artifacts it numbers.
# Citations
- `.github/scripts/Request-BuildNumber.ps1` — the CAS protocol
- `.github/workflows/deploy-backend.yml`, `deploy-mcp.yml` — reserve, verify, publish
- `.github/workflows/build-agent.yml`, `scripts/Deployment/build.ps1` — agent line
- [Build Counter Blob](agent/build-counter-blob.md) — agent-specific consequences
@@ -272,13 +272,16 @@ try {
$downloadAttempt++
try {
Write-Log "Download attempt ${downloadAttempt}/${maxDownloadAttempts}"
$downloadStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Invoke-WebRequest `
-Uri $AgentDownloadUrl `
-OutFile $zipPath `
-UseBasicParsing `
-TimeoutSec 30 `
-ErrorAction Stop
Write-Log "Downloaded agent to $zipPath"
$downloadStopwatch.Stop()
$downloadSeconds = [math]::Round($downloadStopwatch.Elapsed.TotalSeconds, 1)
Write-Log "Downloaded agent to $zipPath (took ${downloadSeconds}s)"
break
}
catch {
@@ -15,7 +15,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
private const string CurrentUrl = "https://autopilotmonitor-api-eu.azurewebsites.net";
private const string TargetUrl = "https://autopilotmonitor-api-us.azurewebsites.net";
private static AgentConfigResponse Config(string migrateUrl) =>
private static AgentConfigResponse Config(string? migrateUrl) =>
new AgentConfigResponse { MigrateToApiBaseUrl = migrateUrl };
[Fact]
@@ -70,6 +70,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// Read the file the production code actually wrote and deserialize with the same lib.
var raw = File.ReadAllText(_cachePath);
var persisted = JsonConvert.DeserializeObject<AgentConfigResponse>(raw);
Assert.NotNull(persisted);
Assert.False(persisted.UnrestrictedMode);
Assert.False(persisted.AllowAgentDowngrade);
@@ -193,7 +193,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
case BackendBehaviour.AlwaysAuthFailure401:
throw new BackendAuthException("Unauthorized", 401);
case BackendBehaviour.AlwaysReturnNull:
return Task.FromResult<AgentConfigResponse>(null);
return Task.FromResult<AgentConfigResponse>(null!);
default:
throw new InvalidOperationException("unknown behaviour");
}
@@ -204,9 +204,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// In-memory + per-instance cache so a prior test's successful fetch can't
// bleed into a later test that expects UsedDefaults.
private AgentConfigResponse _cached;
private AgentConfigResponse? _cached;
protected override void CacheConfig(AgentConfigResponse config) { _cached = config; }
protected override AgentConfigResponse LoadCachedConfig() => _cached;
protected override AgentConfigResponse LoadCachedConfig() => _cached!;
}
}
}
@@ -51,7 +51,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Integration
logger: logger,
helloTracker: null);
EspExitedEventArgs captured = null;
EspExitedEventArgs? captured = null;
tracker.EspExited += (_, args) => captured = args;
// Source time IS in the past relative to the clock — verify the tracker carries the
@@ -0,0 +1,71 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Analyzers
{
/// <summary>
/// LocalAdminAnalyzer.MatchesAllowedEntry — the allowed-list matching contract.
/// Entries without wildcard characters match exactly (case-insensitive); entries
/// containing * or ? are glob patterns with the same semantics as the hardware
/// whitelist. The same matcher is applied to local accounts and profile folders.
/// </summary>
public sealed class LocalAdminAnalyzerTests
{
// -------------------------------------------------------------- literal entries
[Theory]
[InlineData("SupportAdmin", "SupportAdmin")]
[InlineData("supportadmin", "SupportAdmin")]
[InlineData("SUPPORTADMIN", "supportadmin")]
public void Literal_entry_matches_exactly_case_insensitive(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("SupportAdmin2", "SupportAdmin")]
[InlineData("SupportAdmin", "SupportAdmin2")]
[InlineData("Support", "SupportAdmin")]
public void Literal_entry_does_not_substring_match(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
// -------------------------------------------------------------- glob patterns
[Theory]
[InlineData("adm-12345", "adm-*")]
[InlineData("adm-", "adm-*")]
[InlineData("ADM-9F3B2C", "adm-*")]
[InlineData("lapsAdmin01", "lapsAdmin??")]
[InlineData("svc-backup-01", "svc-*-01")]
public void Wildcard_entry_matches_glob(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("administrator2", "adm-*")] // no literal "adm-" prefix
[InlineData("xadm-123", "adm-*")] // pattern is anchored at start
[InlineData("lapsAdmin1", "lapsAdmin??")] // ? is exactly one char
[InlineData("adm.123", "adm-*")] // "-" is literal, not a regex class
public void Wildcard_entry_does_not_overmatch(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Fact]
public void Regex_metacharacters_in_entry_are_treated_literally()
{
// "adm.123" as an entry has no wildcard chars → exact match only,
// the dot must not act as a regex any-char.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.123"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.123"));
// Mixed: wildcard present, remaining metacharacters stay literal.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.*"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.*"));
}
}
}
@@ -40,7 +40,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
try { Directory.Delete(_outsideUsers, recursive: true); } catch { /* best effort */ }
}
private GatherRuleContext Context(bool unrestricted = false, string imeLogPathOverride = null)
private GatherRuleContext Context(bool unrestricted = false, string? imeLogPathOverride = null)
=> new GatherRuleContext(
new AgentLogger(_tmp.Path, AgentLogLevel.Info),
"sess", "tenant",
@@ -52,13 +52,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
};
private static GatherRule Rule(string collectorType, string target,
Dictionary<string, string> parameters = null) => new GatherRule
Dictionary<string, string>? parameters = null) => new GatherRule
{
RuleId = "GATHER-GUARD-TEST",
Title = "guard test",
CollectorType = collectorType,
Target = target,
Parameters = parameters,
// Match the model default (empty, never null) so the collectors see the
// same shape a deserialized rule without parameters gives them.
Parameters = parameters ?? new Dictionary<string, string>(),
Trigger = "startup",
OutputEventType = "guard_test",
};
@@ -192,7 +192,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = "1" }),
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = 1 }));
Assert.NotEqual(
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = null }),
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = null! }),
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = "<null>" }));
}
@@ -60,7 +60,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
}
private static GatherRule ExitRule(
string id, string triggerPhase, List<string> activePhases = null, string fromPhase = null)
string id, string triggerPhase, List<string>? activePhases = null, string? fromPhase = null)
=> new GatherRule
{
RuleId = id,
@@ -63,7 +63,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
/// <summary>Interval rule with a 1h period — the timer never ticks within a test run,
/// so these rules exist purely for synchronous scope-matrix assertions.</summary>
private static GatherRule ScopeProbeRule(string id, List<string> activePhases = null, string fromPhase = null)
private static GatherRule ScopeProbeRule(string id, List<string>? activePhases = null, string? fromPhase = null)
=> new GatherRule
{
RuleId = id,
@@ -78,7 +78,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
ActiveFromPhase = fromPhase,
};
private static GatherRule StartupRule(string id, List<string> activePhases = null, string fromPhase = null)
private static GatherRule StartupRule(string id, List<string>? activePhases = null, string? fromPhase = null)
=> new GatherRule
{
RuleId = id,
@@ -27,7 +27,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
[InlineData(" ")] // whitespace only
[InlineData("<![LOG[truncated line with no closing")] // truncated / missing fields
[InlineData("garbage prefix <![LOG[msg]LOG]!><time=\"06:08:04.8834397\" date=\"2-8-2026\" component=\"C\" context=\"\" type=\"1\" thread=\"1\" file=\"\">")] // leading junk (non-ignorable) fails the StartsWith gate
public void TryParseLine_returns_false_without_throwing_for_malformed_input(string line)
public void TryParseLine_returns_false_without_throwing_for_malformed_input(string? line)
{
var ok = CmTraceLogParser.TryParseLine(line, out var entry);
@@ -24,12 +24,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
/// </summary>
public sealed class ImeLogPatternReDoSTests
{
private static ImeLogPattern Pattern(string id, string regex, string action = null) => new ImeLogPattern
private static ImeLogPattern Pattern(string id, string regex, string? action = null) => new ImeLogPattern
{
PatternId = id,
Pattern = regex,
Category = "always",
Action = action,
// Action is optional on the wire (absent = null); the model declares it non-nullable.
Action = action!,
Enabled = true,
Parameters = new Dictionary<string, string>(),
};
@@ -0,0 +1,212 @@
using System;
using System.Diagnostics;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Executes the generated cleanup script against REAL scratch trees with a real
/// powershell.exe — end-to-end replays of the 2026-07-22 incident mechanics that the
/// string pins in <see cref="CleanupServiceScriptTests"/> cannot prove:
/// <list type="bullet">
/// <item>a locked tree must abort WITHOUT deleting anything (old script force-deleted
/// marker + state around the locks and orphaned the binaries),</item>
/// <item>a straggler agent process (the incident's second instance) must be found by
/// NAME, killed, and the cleanup must then complete,</item>
/// <item>concurrent cleanup scripts must be serialized by the lock file.</item>
/// </list>
/// Timings are dialed down via the test-only BuildCleanupScript parameters so each case
/// runs in seconds. Every test uses GUID-unique process/task/tree names so nothing real
/// (a dev-box agent, a real Scheduled Task) can ever be touched.
/// </summary>
[Collection("SerialThreading")]
public class CleanupServiceScriptIntegrationTests : IDisposable
{
private readonly string _root;
private readonly string _basePath;
private readonly string _taskName = $"AmTest-NoSuchTask-{Guid.NewGuid():N}";
private readonly string _fakeProcessName = $"AmTestAgent{Guid.NewGuid().ToString("N").Substring(0, 8)}";
public CleanupServiceScriptIntegrationTests()
{
_root = Path.Combine(Path.GetTempPath(), $"am-cleanup-it-{Guid.NewGuid():N}");
_basePath = Path.Combine(_root, "AutopilotMonitor");
Directory.CreateDirectory(Path.Combine(_basePath, "Agent"));
Directory.CreateDirectory(Path.Combine(_basePath, "State"));
Directory.CreateDirectory(Path.Combine(_basePath, "Logs"));
File.WriteAllText(Path.Combine(_basePath, "Agent", "fake.exe"), "x");
File.WriteAllText(Path.Combine(_basePath, "State", "enrollment-complete.marker"), "x");
File.WriteAllText(Path.Combine(_basePath, "Logs", "agent.log"), "x");
}
public void Dispose()
{
try { Directory.Delete(_root, recursive: true); } catch { }
}
private static string LockPath => Path.Combine(Path.GetTempPath(), "AutopilotMonitor-Cleanup.lock");
private string BuildScript(bool keepLogs = false, int processWaitSeconds = 1)
=> CleanupService.BuildCleanupScript(
agentProcessName: _fakeProcessName,
agentBasePath: _basePath,
keepLogs: keepLogs,
scheduledTaskName: _taskName,
rebootOnComplete: false,
initialDelaySeconds: 0,
processWaitSeconds: processWaitSeconds,
renameRetries: 2,
renameRetryDelaySeconds: 1);
/// <summary>Runs the script synchronously in a real powershell.exe. Returns the script path.</summary>
private string RunScript(string script)
{
var scriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
File.WriteAllText(scriptPath, script);
using (var proc = Process.Start(new ProcessStartInfo
{
FileName = SystemPaths.PowerShell,
Arguments = $"-NoProfile -ExecutionPolicy Bypass -File \"{scriptPath}\"",
UseShellExecute = false,
CreateNoWindow = true,
}))
{
Assert.True(proc.WaitForExit(90_000), "cleanup script did not finish within 90s");
}
return scriptPath;
}
[Fact]
public void Happy_path_deletes_tree_and_removes_lock_and_script()
{
var scriptPath = RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "tree should be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(scriptPath), "script should self-delete");
Assert.False(File.Exists(LockPath), "cleanup lock should be released");
}
[Fact]
public void Locked_tree_aborts_completely_marker_and_files_survive()
{
// The incident regression: with a file locked inside the tree the OLD script
// deleted everything else (incl. the enrollment-complete marker) and left the
// binaries orphaned with the task already gone. The hardened script must treat
// the failed probe-rename as "someone still holds this" and walk away entirely.
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.None))
{
RunScript(BuildScript());
}
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")), "binaries must stay in place");
Assert.True(File.Exists(Path.Combine(_basePath, "State", "enrollment-complete.marker")),
"marker must survive so the next boot retries cleanup");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(LockPath));
}
[Fact]
public void Straggler_process_holding_the_tree_is_killed_and_cleanup_completes()
{
// Full incident replay: a SECOND agent-named process (here: a renamed
// powershell.exe, matching how only the process NAME identifies agents) holds a
// file in the tree open. The old script waited on the launcher PID only and ran
// its force-delete against the live locks. The hardened script must find the
// straggler by name, kill it after the wait deadline, and then delete everything.
var fakeExe = Path.Combine(_root, _fakeProcessName + ".exe");
File.Copy(SystemPaths.PowerShell, fakeExe);
var lockTarget = Path.Combine(_basePath, "Agent", "fake.exe").Replace("'", "''");
Process straggler = null;
try
{
straggler = Process.Start(new ProcessStartInfo
{
FileName = fakeExe,
Arguments = "-NoProfile -Command \"$f = [IO.File]::Open('" + lockTarget +
"', 'Open', 'Read', 'None'); Start-Sleep -Seconds 120\"",
UseShellExecute = false,
CreateNoWindow = true,
});
// Give the straggler time to acquire its file handle before cleanup starts.
var acquireDeadline = DateTime.UtcNow.AddSeconds(20);
var acquired = false;
while (!acquired && DateTime.UtcNow < acquireDeadline)
{
try
{
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
}
}
catch (IOException)
{
acquired = true; // share violation ⇒ straggler holds the file
}
if (!acquired) System.Threading.Thread.Sleep(200);
}
Assert.True(acquired, "straggler never acquired its file lock");
RunScript(BuildScript(processWaitSeconds: 2));
Assert.True(straggler.HasExited, "straggler must be force-killed by the name-based backstop");
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
}
finally
{
try { if (straggler != null && !straggler.HasExited) straggler.Kill(); } catch { }
try { straggler?.Dispose(); } catch { }
}
}
[Fact]
public void Fresh_foreign_lock_backs_off_without_touching_the_tree()
{
File.WriteAllText(LockPath, "held by another cleanup");
try
{
var scriptPath = RunScript(BuildScript());
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")),
"loser script must not touch the tree — the lock holder owns the deletion");
Assert.True(File.Exists(LockPath), "foreign lock must not be deleted by the loser");
Assert.False(File.Exists(scriptPath), "loser still self-deletes its script");
}
finally
{
try { File.Delete(LockPath); } catch { }
}
}
[Fact]
public void Stale_foreign_lock_is_taken_over_and_cleanup_proceeds()
{
File.WriteAllText(LockPath, "crashed cleanup from a previous boot");
File.SetLastWriteTime(LockPath, DateTime.Now.AddMinutes(-15));
RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "stale lock must be taken over, cleanup must run");
Assert.False(File.Exists(LockPath));
}
[Fact]
public void KeepLogs_variant_deletes_everything_except_logs()
{
RunScript(BuildScript(keepLogs: true));
Assert.True(File.Exists(Path.Combine(_basePath, "Logs", "agent.log")), "Logs must survive");
Assert.False(Directory.Exists(Path.Combine(_basePath, "Agent")));
Assert.False(Directory.Exists(Path.Combine(_basePath, "State")));
Assert.Empty(Directory.GetFileSystemEntries(_basePath, "*.del-*"));
}
}
}
@@ -0,0 +1,135 @@
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Pins the LIFE-F7 hardening invariants of the self-destruct cleanup script
/// (incident 2026-07-22: a second agent instance held exe + loaded DLLs while the
/// old script — waiting on the launcher PID only — force-deleted around the locks
/// AFTER removing the Scheduled Task, orphaning the binaries with no retry path).
/// These are string-level pins on <see cref="CleanupService.BuildCleanupScript"/>:
/// weakening any of them silently re-introduces the orphaned-binaries failure mode.
/// </summary>
public class CleanupServiceScriptTests
{
private const string ProcessName = "AutopilotMonitor.Agent";
private const string BasePath = @"C:\ProgramData\AutopilotMonitor";
private const string TaskName = "AutopilotMonitorAgent";
private static string Script(bool keepLogs = false, bool reboot = false)
=> CleanupService.BuildCleanupScript(ProcessName, BasePath, keepLogs, TaskName, reboot);
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Waits_by_process_name_not_launcher_pid(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #2: name-based wait covers marker-retry / self-update / manual instances
// and is immune to PID reuse. Wait-Process -Id saw exactly one PID and nothing else.
Assert.Contains($"Get-Process -Name '{ProcessName}'", script);
Assert.DoesNotContain("Wait-Process", script);
// Backstop: stragglers are force-killed — their images are deleted right after.
Assert.Contains("Stop-Process -Force", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Never_deletes_the_original_path(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #3: deletion happens exclusively on successfully renamed targets
// ($entry.Renamed). A Remove-Item against the original tree is what deleted
// marker + state around locked binaries in the incident.
Assert.DoesNotContain($"Remove-Item -Path '{BasePath}'", script);
Assert.Contains("Remove-Item -Path $entry.Renamed", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Task_removal_is_gated_on_probe_and_deletion_on_task_gone(bool keepLogs)
{
var script = Script(keepLogs);
// Probe-rename first; only a successful probe reaches task removal, and only a
// removed task reaches deletion. Both failure branches rename BACK so the next
// boot retries with task + marker + files intact.
Assert.Contains("if ($probeOk)", script);
Assert.Contains("if ($taskGone)", script);
Assert.Contains("Rename-Item -Path $entry.Renamed -NewName $entry.Original", script);
// LIFE-F6 task-removal loop + module-independent fallback stay intact.
Assert.Contains("Unregister-ScheduledTask -TaskName $taskName", script);
Assert.Contains("schtasks.exe /Delete /TN $taskName /F", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Concurrent_cleanup_scripts_are_serialized_by_lock_file(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #1: CreateNew is the atomic single-flight gate; the loser exits without
// touching the tree. Stale locks (>10 min) are taken over.
Assert.Contains("AutopilotMonitor-Cleanup.lock", script);
Assert.Contains("[System.IO.FileMode]::CreateNew", script);
Assert.Contains("$lockAge.TotalMinutes -gt 10", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Rename_targets_carry_a_guid_suffix(bool keepLogs)
{
// A fixed '.del' target collides with leftovers of earlier attempts and with a
// concurrent script's rename — the GUID makes every attempt's target unique.
Assert.Contains(".del-' + [Guid]::NewGuid().ToString('N')", Script(keepLogs));
}
[Fact]
public void KeepLogs_variant_excludes_logs_directory()
{
var script = Script(keepLogs: true);
Assert.Contains("-Exclude 'Logs'", script);
Assert.DoesNotContain($"Rename-Item -Path '{BasePath}' -NewName", script);
}
[Fact]
public void Full_variant_renames_the_whole_tree()
{
var script = Script(keepLogs: false);
Assert.Contains($"Rename-Item -Path '{BasePath}' -NewName $renamedPath", script);
Assert.DoesNotContain("-Exclude 'Logs'", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Reboot_flag_controls_shutdown_invocation(bool reboot)
{
var script = Script(keepLogs: false, reboot: reboot);
if (reboot)
Assert.Contains("shutdown.exe /r /t 10", script);
else
Assert.DoesNotContain("shutdown.exe", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Script_and_lock_are_released_in_finally(bool keepLogs)
{
var script = Script(keepLogs);
Assert.Contains("finally", script);
Assert.Contains("$lockStream.Dispose()", script);
Assert.Contains("Remove-Item -Path $scriptPath -Force", script);
}
}
}
@@ -256,7 +256,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
[InlineData(null, false)]
[InlineData("", false)]
[InlineData("0x8007", false)]
public void IsAppxFamilyHresult_MatchesOnlyAppxDeploymentRange(string hresult, bool expected)
public void IsAppxFamilyHresult_MatchesOnlyAppxDeploymentRange(string? hresult, bool expected)
{
Assert.Equal(expected, AppxFailureAnalyzer.IsAppxFamilyHresult(hresult));
}
@@ -39,7 +39,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
public List<EspFailureDetectedEventArgs> EspFailures { get; } = new List<EspFailureDetectedEventArgs>();
public ProvisioningStatusTracker Tracker { get; }
public Fixture(Func<IReadOnlyList<AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState>> packageStatesProbe = null)
public Fixture(Func<IReadOnlyList<AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState>>? packageStatesProbe = null)
{
var clock = new VirtualClock(Fixed);
var post = new InformationalEventPost(Sink, clock);
@@ -179,7 +179,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
}");
var captured = f.CapturedEvents();
Assert.Single(captured.Where(e => e.EventType == "esp_failure_settle_started"));
Assert.Single(captured, e => e.EventType == "esp_failure_settle_started");
}
// =====================================================================
@@ -214,8 +214,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
// No terminal fire — the failure was retracted.
Assert.Empty(f.EspFailures);
var recovered = Assert.Single(f.CapturedEvents()
.Where(e => e.EventType == "esp_failure_settle_recovered"));
var recovered = Assert.Single(
f.CapturedEvents(), e => e.EventType == "esp_failure_settle_recovered");
Assert.Equal("AccountSetup", (string)recovered.Data["category"]);
Assert.Equal("Apps", (string)recovered.Data["failedSubcategory"]);
Assert.Equal("inProgress", (string)recovered.Data["observedState"]);
@@ -280,7 +280,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
f.Tracker.TriggerSettleTimerForTest("AccountSetupCategory.Status");
Assert.Single(f.EspFailures);
Assert.Empty(f.CapturedEvents().Where(e => e.EventType == "esp_failure_settle_recovered"));
Assert.DoesNotContain(f.CapturedEvents(), e => e.EventType == "esp_failure_settle_recovered");
}
// =====================================================================
@@ -326,8 +326,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
var settleStarted = Assert.Single(f.CapturedEvents()
.Where(e => e.EventType == "esp_failure_settle_started"));
var settleStarted = Assert.Single(
f.CapturedEvents(), e => e.EventType == "esp_failure_settle_started");
Assert.Equal(1, (int)settleStarted.Data["trackedAppsNotCompletedCount"]);
var apps = (List<Dictionary<string, object>>)settleStarted.Data["trackedAppsNotCompleted"];
var app = Assert.Single(apps);
@@ -354,8 +354,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
""SecurityPoliciesSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Failed""}
}");
var settleStarted = Assert.Single(f.CapturedEvents()
.Where(e => e.EventType == "esp_failure_settle_started"));
var settleStarted = Assert.Single(
f.CapturedEvents(), e => e.EventType == "esp_failure_settle_started");
Assert.False(settleStarted.Data.ContainsKey("trackedAppsNotCompleted"));
Assert.False(settleStarted.Data.ContainsKey("trackedAppsNotCompletedCount"));
}
@@ -0,0 +1,143 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
{
/// <summary>
/// Pins the anti-flood semantics of <see cref="EmergencyReporter.TrySendAsync"/> after the
/// delivery-hardening change (2026-07-23): the once-per-session dedup and the per-session
/// budget count DELIVERED reports. The previous mark-before-send permanently suppressed an
/// error key after its first transport failure — fatal for single-shot callers like the 48h
/// emergency break, whose process exits right after the attempt.
/// </summary>
public class EmergencyReporterTests
{
private sealed class FakeApiClient : BackendApiClient
{
public readonly List<TimeSpan?> Timeouts = new List<TimeSpan?>();
public readonly Queue<bool> Results = new Queue<bool>();
public FakeApiClient() : base(
httpClient: new HttpClient(),
baseUrl: "http://localhost",
manufacturer: string.Empty,
model: string.Empty,
serialNumber: string.Empty,
useBootstrapTokenAuth: false,
bootstrapToken: null,
agentVersion: "0.0.0",
logger: null)
{
}
public int Calls => Timeouts.Count;
public override Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
Timeouts.Add(timeout);
return Task.FromResult(Results.Count > 0 && Results.Dequeue());
}
}
private static EmergencyReporter Reporter(FakeApiClient client)
=> new EmergencyReporter(client, "session", "tenant", "0.0.0", logger: null);
[Fact]
public async Task Failed_send_rolls_back_reservation_so_a_later_call_retries()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // first call fails
client.Results.Enqueue(true); // second call succeeds
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
// Old behaviour: second call suppressed as "already reported" ⇒ 1 call.
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Delivered_send_is_deduplicated_per_key()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(1, client.Calls);
}
[Fact]
public async Task Retries_stay_inside_one_reservation_and_stop_on_success()
{
var client = new FakeApiClient();
client.Results.Enqueue(false);
client.Results.Enqueue(true); // second attempt delivers — third must not fire
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, perAttemptTimeout: TimeSpan.FromMilliseconds(50), retryDelay: TimeSpan.Zero);
Assert.Equal(2, client.Calls);
// Delivered ⇒ dedup holds for the session.
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Per_attempt_timeout_is_forwarded_to_the_api_client()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
var timeout = TimeSpan.FromSeconds(15);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 1, perAttemptTimeout: timeout);
Assert.Equal(timeout, Assert.Single(client.Timeouts));
}
[Fact]
public async Task All_attempts_failing_rolls_back_after_exhausting_attempts()
{
var client = new FakeApiClient(); // queue empty ⇒ every attempt fails
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, retryDelay: TimeSpan.Zero);
Assert.Equal(3, client.Calls);
// Rolled back ⇒ a later call may try again (budget + dedup + cooldown restored).
client.Results.Enqueue(true);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(4, client.Calls);
}
[Fact]
public async Task Different_keys_do_not_suppress_each_other_after_rollback()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // break report fails → rollback (incl. cooldown restore)
client.Results.Enqueue(true); // unrelated report must still go out
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.ConfigFetchFailed, "m");
Assert.Equal(2, client.Calls);
}
}
}
@@ -44,7 +44,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
private readonly FakeQuarantineSink _quarantine = new FakeQuarantineSink();
private readonly FakeSignalIngressSink _sink = new FakeSignalIngressSink();
private readonly DecisionEngine _engine = new DecisionEngine();
private DecisionStepProcessor _sut;
private DecisionStepProcessor? _sut;
public DecisionStepProcessorParkedTripwireTests()
{
@@ -107,7 +107,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
return (step, signal);
}
private FakeSignalIngressSink.PostedSignal FindTripwirePost()
private FakeSignalIngressSink.PostedSignal? FindTripwirePost()
{
foreach (var posted in _sink.Posted)
{
@@ -121,6 +121,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
return null;
}
/// <summary>The tripwire post, asserted present — for the assertions that only run
/// after <c>WaitForTripwire()</c> has already confirmed one was emitted.</summary>
private FakeSignalIngressSink.PostedSignal RequireTripwirePost()
{
var post = FindTripwirePost();
Assert.NotNull(post);
return post!;
}
private int CountTripwirePosts()
{
var count = 0;
@@ -165,9 +174,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
Thread.Sleep(SuppressWait);
Assert.Equal(1, CountTripwirePosts());
var post = FindTripwirePost();
var post = RequireTripwirePost();
Assert.Equal(DecisionSignalKind.InformationalEvent, post.Kind);
Assert.Equal(nameof(SessionStage.EspAccountSetup), post.Payload["stage"]);
Assert.Equal(nameof(SessionStage.EspAccountSetup), post.Payload!["stage"]);
Assert.Equal("Warning", post.Payload[SignalPayloadKeys.Severity]);
Assert.Equal("true", post.Payload[SignalPayloadKeys.ImmediateUpload]);
Assert.Contains("esp_final_exit", post.Payload["signalsSeen"]);
@@ -222,7 +231,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
sut.ApplyStep(step, signal);
Assert.True(WaitForTripwire());
Assert.Equal(DeadlineNames.ClassifierTick, FindTripwirePost().Payload["armedDeadlines"]);
Assert.Equal(DeadlineNames.ClassifierTick, RequireTripwirePost().Payload!["armedDeadlines"]);
}
[Fact]
@@ -237,7 +246,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
sut.ApplyStep(step, signal);
Assert.True(WaitForTripwire());
Assert.Equal(nameof(SessionStage.AwaitingDesktop), FindTripwirePost().Payload["stage"]);
Assert.Equal(nameof(SessionStage.AwaitingDesktop), RequireTripwirePost().Payload!["stage"]);
}
// ===================================================================== Dwell semantics
@@ -256,8 +256,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
RemoteConfigFetchOutcome outcome,
int configVersion = 0,
int attempts = 1,
string failureType = null,
string failureMessage = null,
string? failureType = null,
string? failureMessage = null,
int? authStatusCode = null)
{
// CurrentConfig is set via the protected SetConfig path — easiest:
@@ -33,7 +33,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
[InlineData("Network connections (1 of 1 added)", null)]
[InlineData("", null)]
[InlineData(null, null)]
public void TryExtractErrorCode_extracts_parenthesised_hex_tail(string statusText, string expected)
public void TryExtractErrorCode_extracts_parenthesised_hex_tail(string? statusText, string? expected)
{
var actual = ProvisioningStatusTracker.TryExtractErrorCode(statusText);
Assert.Equal(expected, actual);
@@ -195,7 +195,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
var all = f.InfoEvents(SharedEventTypes.ScriptCompleted);
Assert.Equal(2, all.Count);
Assert.Equal("65.00", all[1].Payload!["durationSeconds"]);
Assert.Equal("cycle_including_reporting_latency", all[1].Payload["durationBasis"]);
Assert.Equal("cycle_including_reporting_latency", all[1].Payload!["durationBasis"]);
}
[Fact]
@@ -151,17 +151,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
}
[Fact]
public void TriggerPhaseChangedFromTest_emits_informational_only_no_decision_signal()
public void TriggerPhaseChangedFromTest_emits_typed_signal_and_informational_event()
{
using var f = new Fixture();
using var adapter = new RealmJoinWatcherAdapter(f.Watcher, f.Ingress, f.Clock);
adapter.TriggerPhaseChangedFromTest(prev: 100, curr: 200);
// Phase change is observability-only. There is no DecisionSignalKind.RealmJoinPhaseChanged.
// 1) Typed signal — persists LastDeploymentPhase and drives the aborted-RJ-ESP
// detection in the reducer (session 224b2087). Not Detected/Resolved.
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.RealmJoinDetected);
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.RealmJoinResolved);
var decision = f.Ingress.Posted.Single(p => p.Kind == DecisionSignalKind.RealmJoinPhaseChanged);
Assert.Equal("RealmJoinWatcher", decision.SourceOrigin);
Assert.Equal("200", decision.Payload![DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase]);
Assert.Equal("100", decision.Payload[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase]);
// 2) Dual-emit informational event keeps the existing timeline entry.
var info = f.Ingress.Posted.Single(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
@@ -1,4 +1,4 @@
using System;
using System;
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic;
using Xunit;
@@ -19,7 +19,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
private static List<BandwidthJobSample> Jobs(params BandwidthJobSample[] jobs)
=> new List<BandwidthJobSample>(jobs);
private static BandwidthJobSample Job(string fileId, long wan, long lan = 0)
private static BandwidthJobSample Job(string? fileId, long wan, long lan = 0)
=> new BandwidthJobSample { FileId = fileId, WanBytes = wan, LanBytes = lan };
[Fact]
@@ -43,8 +43,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.NotNull(estimate);
Assert.Equal(1, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsMax.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsMax!.Value, 3);
Assert.Equal(6_000_000, estimate.WanBytesObserved);
Assert.Equal("10-50", estimate.Bucket);
}
@@ -58,7 +58,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
estimator.AddSnapshot(T0.AddSeconds(3), Jobs(Job("a", 3_000_000), Job("b", 3_000_000)));
var estimate = estimator.TryBuildEstimate();
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
}
[Fact]
@@ -74,7 +74,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.Equal(2, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsMax.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsMax!.Value, 3);
}
[Fact]
@@ -91,7 +91,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
estimator.AddSnapshot(T0.AddSeconds(63), Jobs(Job("a", 106_000_000)));
var estimate = estimator.TryBuildEstimate();
Assert.Equal(1, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
}
[Fact]
@@ -130,8 +130,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
estimator.AddSnapshot(T0.AddSeconds(3), Jobs(Job("a", wan: 6_000_000, lan: 375_000_000)));
var estimate = estimator.TryBuildEstimate();
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(1000.0, estimate.LanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
Assert.Equal(1000.0, estimate.LanMbpsP90!.Value, 3);
Assert.Equal("10-50", estimate.Bucket); // bucket keyed to WAN, not LAN
}
@@ -182,8 +182,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.Equal(20, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3); // p90 stays on the steady plateau
Assert.Equal(160.0, estimate.WanMbpsMax.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3); // p90 stays on the steady plateau
Assert.Equal(160.0, estimate.WanMbpsMax!.Value, 3);
}
[Fact]
@@ -265,7 +265,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.Equal(1, estimate.WanSampleCount); // only the finite positive sample survived
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
Assert.Equal(0, estimate.WanBytesObserved);
}
@@ -31,84 +31,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
_logger.Info($"Executing FULL SELF-DESTRUCT (Scheduled Task + Files{(_configuration.RebootOnComplete ? " + Reboot" : "")})");
var currentProcessId = Process.GetCurrentProcess().Id;
var currentProcess = Process.GetCurrentProcess();
var agentBasePath = Path.GetFullPath(Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"AutopilotMonitor"
));
var logDir = Path.GetFullPath(Path.Combine(agentBasePath, "Logs"));
var keepLogs = _configuration.KeepLogFile;
// Create a self-deleting PowerShell script
var cleanupScript = $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Wait for agent process to exit
Start-Sleep -Seconds 2
Wait-Process -Id {currentProcessId} -ErrorAction SilentlyContinue -Timeout 30
Start-Sleep -Seconds 1
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan task firing
# against a deleted exe spams Task Scheduler errors every boot forever and, once the files (incl.
# the enrollment-complete marker) are gone, has no cleanup-retry path. So confirm the task is gone
# first; if it cannot be removed, SKIP the file deletion and leave everything intact for a clean
# cleanup retry on the next boot (the marker survives, so bootstrap re-runs cleanup).
$taskName = '{_configuration.ScheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
{(keepLogs ? $@"
if ($taskGone) {{
# Delete everything except Logs directory
Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' | ForEach-Object {{
$dest = $_.FullName + '.del'
try {{ Rename-Item -Path $_.FullName -NewName $dest -Force -ErrorAction SilentlyContinue }} catch {{ }}
Remove-Item -Path $dest -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
" : $@"
if ($taskGone) {{
# Rename folder then delete. Retry a few times to let the OS release handles.
$renamedPath = '{agentBasePath}.del'
$renamed = $false
for ($i = 1; $i -le 10; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$renamed = $true
break
}} catch {{
Start-Sleep -Seconds 2
}}
}}
if ($renamed) {{
Remove-Item -Path $renamedPath -Recurse -Force -ErrorAction SilentlyContinue
}} else {{
Remove-Item -Path '{agentBasePath}' -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
")}
{(_configuration.RebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
";
var cleanupScript = BuildCleanupScript(
agentProcessName: currentProcess.ProcessName,
agentBasePath: agentBasePath,
keepLogs: _configuration.KeepLogFile,
scheduledTaskName: _configuration.ScheduledTaskName,
rebootOnComplete: _configuration.RebootOnComplete);
// Write cleanup script to temp location (outside of agent folder)
var tempScriptPath = Path.Combine(Path.GetTempPath(), $"AutopilotMonitor-Cleanup-{Guid.NewGuid()}.ps1");
@@ -143,5 +77,171 @@ Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}
}
/// <summary>
/// Builds the self-deleting cleanup PowerShell script. Extracted (internal static, pure)
/// so tests can pin the hardened invariants without launching processes.
/// <para>
/// LIFE-F7 hardening (incident 2026-07-22 — exe + loaded DLLs orphaned forever):
/// </para>
/// <list type="number">
/// <item><b>Single-flight lock</b> — the enrollment-complete marker is written BEFORE
/// cleanup launches, so any agent start during an in-flight cleanup spawns a SECOND
/// cleanup script. Two unsynchronized scripts interleave their rename/delete phases
/// and leave partial trees behind. A CreateNew lock file in the SYSTEM temp dir
/// serializes them; a stale lock (&gt;10 min) is taken over.</item>
/// <item><b>Wait by process NAME, not launcher PID</b> — marker-retry instances,
/// self-update restarts and manual runs all load the same images but have PIDs the
/// old <c>Wait-Process -Id</c> never saw (and an already-freed PID can be reused by
/// an unrelated boot-time process). After the deadline any straggler is force-killed:
/// its images are about to be deleted anyway.</item>
/// <item><b>Probe-rename BEFORE task removal, never delete the original path</b> — the
/// rename is the atomic lock probe. If it fails, something still holds the tree: abort
/// with task + files + marker intact so the next boot retries (file-side symmetry to
/// the LIFE-F6 task-side rule). If the task cannot be removed afterwards, rename BACK
/// and abort for the same reason. Only a fully renamed tree with the task gone is
/// deleted — the old <c>Remove-Item</c>-on-original fallback deleted marker + state
/// around locked binaries AFTER the task was gone, converting a transient lock into a
/// permanently orphaned exe.</item>
/// </list>
/// </summary>
/// <remarks>
/// The timing parameters exist ONLY so integration tests can replay the incident
/// scenarios (locked tree, straggler process) in seconds instead of minutes —
/// production call sites always use the defaults.
/// </remarks>
internal static string BuildCleanupScript(
string agentProcessName,
string agentBasePath,
bool keepLogs,
string scheduledTaskName,
bool rebootOnComplete,
int initialDelaySeconds = 2,
int processWaitSeconds = 60,
int renameRetries = 10,
int renameRetryDelaySeconds = 2)
{
return $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Single-flight lock (LIFE-F7 #1): serialize concurrent cleanup scripts. CreateNew is
# atomic; the loser backs off and lets the winner own the deletion.
$lockPath = Join-Path ([System.IO.Path]::GetTempPath()) 'AutopilotMonitor-Cleanup.lock'
$lockStream = $null
try {{
if (Test-Path $lockPath) {{
$lockAge = (Get-Date) - (Get-Item $lockPath -ErrorAction Stop).LastWriteTime
if ($lockAge.TotalMinutes -gt 10) {{ Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue }}
}}
$lockStream = [System.IO.File]::Open($lockPath, [System.IO.FileMode]::CreateNew, [System.IO.FileAccess]::Write, [System.IO.FileShare]::None)
}} catch {{
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
exit 0
}}
try {{
# Wait for EVERY agent process to exit, by NAME (LIFE-F7 #2) — covers marker-retry
# instances, self-update restarts and manual runs, and is immune to PID reuse.
Start-Sleep -Seconds {initialDelaySeconds}
$deadline = (Get-Date).AddSeconds({processWaitSeconds})
while ((Get-Date) -lt $deadline) {{
if (-not (Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue)) {{ break }}
Start-Sleep -Seconds 2
}}
# Backstop: force-kill stragglers — their images are about to be deleted anyway.
Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
Start-Sleep -Seconds 1
{(keepLogs ? $@"
# Probe-rename all children except Logs (LIFE-F7 #3). Any failure ⇒ rename back and
# abort with the Scheduled Task intact so the next boot retries cleanly.
$renamed = @()
$probeOk = $true
foreach ($item in @(Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' -ErrorAction SilentlyContinue)) {{
$dest = $item.FullName + '.del-' + [Guid]::NewGuid().ToString('N')
$itemOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path $item.FullName -NewName $dest -Force -ErrorAction Stop
$itemOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
if ($itemOk) {{ $renamed += @{{ Original = $item.FullName; Renamed = $dest }} }}
else {{ $probeOk = $false; break }}
}}
" : $@"
# Probe-rename the whole tree (LIFE-F7 #3). Failure ⇒ abort with task + files + marker
# intact so the next boot retries cleanly. GUID suffix avoids collisions with leftovers
# of earlier attempts.
$renamedPath = '{agentBasePath}.del-' + [Guid]::NewGuid().ToString('N')
$probeOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$probeOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
$renamed = @(@{{ Original = '{agentBasePath}'; Renamed = $renamedPath }})
")}
if ($probeOk) {{
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan
# task firing against a deleted exe spams Task Scheduler errors every boot forever
# and, once the files (incl. the enrollment-complete marker) are gone, has no
# cleanup-retry path.
$taskName = '{scheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
if ($taskGone) {{
# Task gone + tree renamed ⇒ safe to delete. Never touches the original path.
foreach ($entry in $renamed) {{
Remove-Item -Path $entry.Renamed -Recurse -Force -ErrorAction SilentlyContinue
}}
}} else {{
# Task survived ⇒ rename back and leave everything for the next-boot retry
# (the marker survives, so bootstrap re-runs cleanup).
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
}} else {{
# Probe failed ⇒ something still holds the tree. Rename back whatever succeeded and
# abort WITHOUT deleting anything — task + marker stay, next boot retries.
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
{(rebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
}} finally {{
if ($lockStream) {{ $lockStream.Dispose() }}
Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}}
";
}
}
}
@@ -370,7 +370,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
try
{
var versionUrl = $"{Constants.AgentBlobBaseUrl}/{Constants.AgentVersionFileNameForLine(2)}";
// Download alias, not the legacy blob account: the legacy mirror in the
// release pipeline is fail-soft, so it can silently lag behind eu.
var versionUrl = $"{Constants.AgentDownloadBaseUrl}/{Constants.AgentVersionFileNameForLine(2)}";
using (var handler = new HttpClientHandler())
using (var client = new HttpClient(handler) { Timeout = TimeSpan.FromMilliseconds(VersionCheckTimeoutMs) })
@@ -497,7 +499,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
try
{
var zipUrl = $"{Constants.AgentBlobBaseUrl}/{Constants.AgentZipFileNameForLine(2)}";
var zipUrl = $"{Constants.AgentDownloadBaseUrl}/{Constants.AgentZipFileNameForLine(2)}";
// Clean up any previous download (fragment from aborted run, or symlink/junction).
// Pre-delete + FileMode.CreateNew below: CreateNew refuses to open an existing
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Management;
using System.Text.RegularExpressions;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
@@ -28,6 +29,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
///
/// Emits a single "local_admin_analysis" event at startup and at shutdown,
/// enabling delta detection between pre- and post-enrollment state.
///
/// Tenant-supplied allowed accounts may contain glob wildcards (e.g. "adm-*");
/// see <see cref="MatchesAllowedEntry"/>.
/// </summary>
public class LocalAdminAnalyzer : IAgentAnalyzer
{
@@ -246,6 +250,43 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
}
}
// -----------------------------------------------------------------------
// Allowed-list matching
// -----------------------------------------------------------------------
/// <summary>
/// Matches an account or profile-folder name against one allowed-list entry.
/// Entries containing <c>*</c> or <c>?</c> are glob patterns (same semantics as the
/// hardware whitelist: <c>*</c> = any run, <c>?</c> = one char, case-insensitive);
/// all other entries match exactly. No ambiguity with literal names — <c>*</c> and
/// <c>?</c> are invalid characters in Windows account names.
/// Internal for direct unit-testing via InternalsVisibleTo.
/// </summary>
internal static bool MatchesAllowedEntry(string name, string entry)
{
if (entry.IndexOf('*') < 0 && entry.IndexOf('?') < 0)
return string.Equals(entry, name, StringComparison.OrdinalIgnoreCase);
var regexPattern = "^" + Regex.Escape(entry)
.Replace("\\*", ".*")
.Replace("\\?", ".") + "$";
try
{
// Timeout guards against pathological patterns (ReDoS)
return Regex.IsMatch(name, regexPattern, RegexOptions.IgnoreCase, TimeSpan.FromMilliseconds(100));
}
catch (RegexMatchTimeoutException)
{
return false;
}
}
private static bool IsAllowed(string name, List<string> allowedAccounts)
{
return allowedAccounts.Any(a => MatchesAllowedEntry(name, a));
}
// -----------------------------------------------------------------------
// Individual checks
// -----------------------------------------------------------------------
@@ -308,8 +349,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
if (disabled)
continue;
if (!allowedAccounts.Any(a =>
string.Equals(a, name, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(name, allowedAccounts))
{
unexpected.Add(name);
_logger.Debug($"{Name}: Unexpected local account: {name}");
@@ -349,8 +389,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
allFound.Add(folderName);
if (!allowedAccounts.Any(a =>
string.Equals(a, folderName, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(folderName, allowedAccounts))
{
unexpected.Add(folderName);
_logger.Debug($"{Name}: Unexpected profile folder: {folderName}");
@@ -45,10 +45,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
/// </summary>
private static readonly (string Name, string Url)[] MdmEndpoints =
{
("Entra ID", "https://login.microsoftonline.com"),
("Entra ID", Constants.EntraLoginBaseUrl),
("Device Registration", "https://enterpriseregistration.windows.net"),
("Intune Portal", "https://portal.manage.microsoft.com"),
("Microsoft Graph", "https://graph.microsoft.com"),
("Microsoft Graph", Constants.GraphBaseUrl),
};
public NetworkChangeDetector(
@@ -169,9 +169,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// <summary>
/// Sends a critical error report to the emergency channel endpoint.
/// Fire-and-forget: swallows all exceptions so a failure here never cascades.
/// Uses a 5-second per-request timeout so it cannot block the upload loop.
/// Default 5-second per-request timeout so it cannot block the upload loop; the
/// emergency-break path passes a longer <paramref name="timeout"/> because its first
/// request of a freshly booted process pays cold TLS + DNS + a possible backend
/// cold start (observed 503s take well over 5 s).
/// Returns <c>true</c> when the HTTP round-trip completed (the endpoint always
/// answers 200), <c>false</c> on timeout/transport failure — the caller-side
/// anti-flood layer uses this to decide whether the report counts as delivered.
/// Virtual for tests.
/// </summary>
public async Task ReportAgentErrorAsync(AgentErrorReport report)
public virtual async Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
try
{
@@ -190,16 +197,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
httpRequest.Headers.Add("X-Tenant-Id", report.TenantId);
AddSecurityHeaders(httpRequest);
using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5)))
using (var cts = new CancellationTokenSource(timeout ?? TimeSpan.FromSeconds(5)))
using (var response = await _httpClient.SendAsync(httpRequest, cts.Token).ConfigureAwait(false))
{
// Response status is deliberately ignored — endpoint always returns 200.
return true;
}
}
}
catch (Exception ex)
{
_logger?.Debug($"ReportAgentErrorAsync: emergency channel failed: {ex.Message}");
return false;
}
}
@@ -57,19 +57,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// (intended to be called as fire-and-forget via <c>_ = reporter.TrySendAsync(...)</c>).
///
/// All anti-flood checks are applied synchronously before the async HTTP call,
/// so repeated calls are cheap when suppressed.
/// so repeated calls are cheap when suppressed. The anti-flood reservation is
/// taken up front (so concurrent same-key calls cannot double-send) but ROLLED
/// BACK when every attempt fails: the budget and the once-per-session dedup are
/// meant to count DELIVERED reports — the previous mark-before-send made the very
/// first transport failure permanently suppress that error key for the session
/// (found via the 48h emergency-break report, which is a single shot at boot).
///
/// <paramref name="attempts"/>/<paramref name="perAttemptTimeout"/>/<paramref name="retryDelay"/>
/// exist for callers whose process is about to exit and cannot come back later
/// (emergency break): retries stay INSIDE the one reservation, so they never
/// multiply against the per-session budget.
/// </summary>
public virtual async Task TrySendAsync(
AgentErrorType errorType,
string message,
int? httpStatusCode = null,
long? sequenceNumber = null)
long? sequenceNumber = null,
int attempts = 1,
TimeSpan? perAttemptTimeout = null,
TimeSpan? retryDelay = null)
{
// Deduplicate by error type + status code: same failure category sent only once per session.
// All anti-flood checks are protected by a lock to prevent race conditions when
// TrySendAsync is called concurrently (fire-and-forget from multiple threads).
var key = $"{errorType}:{httpStatusCode}";
int currentReportCount;
DateTime? previousLastReportTime;
lock (_antiFloodLock)
{
@@ -92,6 +106,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
return;
}
previousLastReportTime = _lastReportTime;
_reportedKeys.Add(key);
_reportCount++;
_lastReportTime = DateTime.UtcNow;
@@ -113,15 +128,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
Timestamp = DateTime.UtcNow
};
var delivered = false;
var totalAttempts = attempts < 1 ? 1 : attempts;
try
{
await _apiClient.ReportAgentErrorAsync(report);
for (var attempt = 1; attempt <= totalAttempts && !delivered; attempt++)
{
if (attempt > 1 && retryDelay.HasValue)
{
await Task.Delay(retryDelay.Value).ConfigureAwait(false);
}
delivered = await _apiClient.ReportAgentErrorAsync(report, perAttemptTimeout).ConfigureAwait(false);
if (!delivered && totalAttempts > 1)
{
_logger?.Debug($"[EmergencyChannel] Attempt {attempt}/{totalAttempts} failed: {key}");
}
}
}
catch (Exception ex)
{
// Swallow — emergency channel must never cascade failures into the caller
_logger?.Debug($"[EmergencyChannel] Failed to send report: {ex.Message}");
}
if (!delivered)
{
lock (_antiFloodLock)
{
_reportedKeys.Remove(key);
_reportCount--;
_lastReportTime = previousLastReportTime;
}
_logger?.Debug($"[EmergencyChannel] Report not delivered — reservation rolled back: {key}");
}
}
}
}
@@ -137,7 +137,24 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
private void OnPhaseChanged(object? sender, RealmJoinPhaseChangedEventArgs e)
{
// Phase change is observability-only: no DecisionSignalKind, just a timeline event.
// Dual-emission like OnDetected/OnResolved: (1) the typed signal persists the phase
// into RealmJoinFacts.LastDeploymentPhase (restart-safe) and lets the reducer detect
// the aborted-RJ-ESP shape (phase left 100/101 for 200/210 without 110 — session
// 224b2087); (2) the InformationalEvent keeps the existing timeline entry.
var payload = new Dictionary<string, string>(StringComparer.Ordinal)
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = e.CurrentPhase.ToString(),
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = e.PreviousPhase.ToString(),
};
_ingress.Post(
kind: DecisionSignalKind.RealmJoinPhaseChanged,
occurredAtUtc: _clock.UtcNow,
sourceOrigin: SourceLabel,
evidence: BuildEvidence(
"realmjoin-phase-changed-v1",
$"RealmJoin DeploymentPhase {e.PreviousPhase} -> {e.CurrentPhase}"),
payload: payload);
// Both raw int (deploymentPhase / previousPhase) and human-readable name
// (deploymentPhaseName / previousPhaseName) flow through — int stays for downstream
// filters / KQL, name fields make the message + UI rows readable without a lookup.
@@ -1,4 +1,6 @@
using System;
using System.Net.NetworkInformation;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Models;
@@ -15,28 +17,78 @@ namespace AutopilotMonitor.Agent.V2.Runtime
/// classifier terminalize the session instead of waiting out the grace. Fully swallowed — a send
/// failure (no network, no cert) must never delay the cleanup/exit the break exists to perform.
/// See tasks/enrollment-status-reclassification.md.
/// <para>
/// Delivery hardening (2026-07-23): this is the FIRST request of a freshly booted process and
/// the process never comes back — a miss is gone forever. The old single shot with a 5 s cap
/// died on boot-time realities (NIC still associating, cold TLS/DNS, backend cold starts
/// observed as &gt;5 s 503s). Now: wait briefly for the NIC, then up to
/// <see cref="SendAttempts"/> attempts with a per-attempt timeout that survives a Functions
/// cold start. The session is 48 h dead — spending up to ~75 s here is irrelevant, losing the
/// only signal that the agent gave up is not.
/// </para>
/// </summary>
internal static class EmergencyBreakReporter
{
private static readonly TimeSpan NetworkWaitMax = TimeSpan.FromSeconds(15);
private const int SendAttempts = 3;
private static readonly TimeSpan PerAttemptTimeout = TimeSpan.FromSeconds(15);
private static readonly TimeSpan RetryDelay = TimeSpan.FromSeconds(5);
// Upper bound for the blocking wait: NIC wait + 3×15s attempts + 2×5s delays + slack.
private static readonly TimeSpan OverallBudget = TimeSpan.FromSeconds(75);
public static void TrySend(AgentConfiguration agentConfig, string agentVersion, AgentLogger logger)
{
try
{
WaitForNetwork(logger);
var auth = BackendClientFactory.BuildAuthClients(agentConfig, agentVersion, logger);
var message =
$"Agent absolute session-age emergency break fired (cap {agentConfig.AbsoluteMaxSessionHours}h) — cleaning up and exiting.";
// The process is about to exit, so block briefly on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP errors;
// the auth bundle's HttpClients are intentionally left for process teardown.
// The process is about to exit, so block on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP
// errors and keeps the retries inside one anti-flood reservation; the auth
// bundle's HttpClients are intentionally left for process teardown.
auth.EmergencyReporter
.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, message)
.Wait(TimeSpan.FromSeconds(5));
.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak,
message,
attempts: SendAttempts,
perAttemptTimeout: PerAttemptTimeout,
retryDelay: RetryDelay)
.Wait(OverallBudget);
}
catch (Exception ex)
{
logger?.Debug($"Emergency-break report send failed (best-effort): {ex.Message}");
}
}
/// <summary>
/// Boot-time NIC grace: the break fires during bootstrap, often seconds after boot while
/// Wi-Fi is still associating. Polls the cheap link-level signal — it cannot prove backend
/// reachability (the send attempts handle that), it only avoids burning the first attempts
/// into a known-dead link. Best-effort: probe errors abort the wait, never the send.
/// </summary>
private static void WaitForNetwork(AgentLogger logger)
{
try
{
if (NetworkInterface.GetIsNetworkAvailable()) return;
logger?.Info($"Emergency-break report: no network link yet — waiting up to {NetworkWaitMax.TotalSeconds:F0}s.");
var deadline = DateTime.UtcNow + NetworkWaitMax;
while (DateTime.UtcNow < deadline)
{
Thread.Sleep(TimeSpan.FromSeconds(1));
if (NetworkInterface.GetIsNetworkAvailable()) return;
}
}
catch (Exception ex)
{
logger?.Debug($"Emergency-break report: network probe failed ({ex.Message}) — sending anyway.");
}
}
}
}
@@ -241,6 +241,199 @@ namespace AutopilotMonitor.DecisionCore.Tests
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
}
// ============================================================== Aborted first deployment (session 224b2087)
[Fact]
public void PhaseChanged_leaving_first_deployment_for_200_releases_gate_with_warning()
{
// Session 224b2087: interactive logon 17 s after phase 101 → RJ reclassifies the run
// as secondary-user deployment and writes 200/210, never 110. The 101 -> 200
// transition is impossible in a healthy first deployment, so it releases the gate
// with Outcome=FirstDeploymentIncomplete and a Warning timeline entry.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
Assert.NotEqual(SessionStage.Finalizing, state.Stage);
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(10),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "101",
}));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("FirstDeploymentIncomplete", step.NewState.RealmJoinFacts.Outcome!.Value);
Assert.NotNull(step.NewState.RealmJoinFacts.ResolvedUtc);
Assert.Equal(200, step.NewState.RealmJoinFacts.LastDeploymentPhase!.Value);
// Hard-timeout deadline cancelled in state AND as a scheduler effect.
Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.CancelDeadline &&
e.CancelDeadlineName == DeadlineNames.RealmJoinTimeout);
// Warning timeline entry names both phases.
var warning = Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_first_deployment_incomplete");
Assert.Equal("Warning", warning.Parameters!["severity"]);
Assert.Equal("101", warning.Parameters["previousPhase"]);
Assert.Equal("200", warning.Parameters["deploymentPhase"]);
Assert.Contains("110", warning.Parameters["message"]);
}
[Fact]
public void PhaseChanged_within_first_deployment_window_is_bookkeeping_only()
{
// 100 -> 101 stays inside the first-deployment window — no release, but the phase
// fact must advance (it is the restart-safe witness for the abort rule).
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "100" })).NewState;
var step = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "100",
}));
Assert.Equal(101, step.NewState.RealmJoinFacts.LastDeploymentPhase!.Value);
Assert.Null(step.NewState.RealmJoinFacts.Outcome);
Assert.Contains(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Empty(step.Effects);
}
[Fact]
public void PhaseChanged_from_completed_deployment_start_does_not_release()
{
// Session 6f1959c0 protection: RJ already stood at CompletedDeployment (210) when
// the agent booted and only deployed AFTERWARDS (210 -> 200 -> 210). Neither
// transition starts from the first-deployment window, so the abort rule must not
// fire — this ambiguous shape keeps waiting for 110 / the hard timeout (status quo).
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "210" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "210",
})).NewState;
Assert.Null(state.RealmJoinFacts.Outcome);
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(7),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "210",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "200",
}));
Assert.Null(step.NewState.RealmJoinFacts.Outcome);
Assert.Contains(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
}
[Fact]
public void Detected_replay_after_restart_with_deployment_phase_releases_gate()
{
// Agent restarted between the 101 and 200 observations: the watcher re-fires
// Detected with the CURRENT phase (200) and no PhaseChanged ever carries the
// transition. The persisted LastDeploymentPhase=101 is the witness — the Detected
// replay path must release the gate exactly like the PhaseChanged path.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
Assert.NotEqual(SessionStage.Finalizing, state.Stage);
Assert.Equal(101, state.RealmJoinFacts.LastDeploymentPhase!.Value);
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(12),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200" }));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("FirstDeploymentIncomplete", step.NewState.RealmJoinFacts.Outcome!.Value);
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_first_deployment_incomplete");
}
[Fact]
public void Timeout_message_reports_persisted_last_phase_instead_of_zero()
{
// Before RealmJoinPhaseChanged became a typed signal, LastDeploymentPhase was only
// written at detection time — every realmjoin_timeout claimed "last phase: 0" no
// matter how far RJ actually got, hiding this whole failure class from ops.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "100" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "100",
})).NewState;
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.DesktopArrived, T0.AddMinutes(7))).NewState;
var step = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, T0.AddMinutes(65),
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
var timeout = Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
Assert.Contains("last phase: 101", timeout.Parameters!["message"]);
Assert.Equal("101", timeout.Parameters["lastSeenPhase"]);
}
[Fact]
public void SelfDeploying_deferred_terminal_released_by_first_deployment_incomplete()
{
// Same deferred-release contract as RealmJoinResolved: when the SelfDeploying
// terminal was deferred on the closed RJ gate, the aborted-first-deployment release
// must complete the session directly (Completed + enrollment_complete), with the
// Warning entry leading the effects.
var engine = new DecisionEngine();
var state = DecisionState.CreateInitial("rj-sd-3", "rj-tenant", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0)).NewState;
state = engine.Reduce(state, MakeSignal(1, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(1),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "DeviceSetup" })).NewState;
state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(2),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.DeviceSetupProvisioningComplete, T0.AddMinutes(3))).NewState;
state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.DeadlineFired, T0.AddMinutes(8),
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.DeviceOnlyEspDetection })).NewState;
Assert.True(state.RealmJoinFacts.SelfDeployingDeferredCompletion?.Value);
var step = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(10),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "101",
}));
Assert.Equal(SessionStage.Completed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentComplete, step.NewState.Outcome);
Assert.Equal("FirstDeploymentIncomplete", step.NewState.RealmJoinFacts.Outcome!.Value);
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_first_deployment_incomplete");
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "enrollment_complete");
}
// ============================================================== SelfDeploying flow
[Fact]
@@ -113,9 +113,11 @@ public class EnrollmentTimeoutClassifierTests
// Negative Hello terminals leave the agent waiting — must NOT count as resolved.
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("hello_provisioning_failed") }).HelloResolved);
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("hello_completion_timeout") }).HelloResolved);
// Both RealmJoin gate terminals count as resolved (phase 110 or 60-min hard timeout).
// All RealmJoin gate terminals count as resolved (phase 110, aborted first
// deployment — session 224b2087 — or 60-min hard timeout).
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("realmjoin_resolved") }).RealmJoinResolved);
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("realmjoin_timeout") }).RealmJoinResolved);
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("realmjoin_first_deployment_incomplete") }).RealmJoinResolved);
}
[Fact]
@@ -0,0 +1,101 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Enforces the URL registry: every well-known own or Microsoft host must be
/// referenced through <c>AutopilotMonitor.Shared.Constants</c>, never as a
/// repeated string literal. The EU cutover missed two hardcoded copies of the
/// blob host (LatestVersionsService, ValidateBootstrapCodeFunction) precisely
/// because they did not go through the registry — this test makes that class
/// of drift a build failure instead of a production surprise.
///
/// Scope: non-test C# source under src/. Comment lines are ignored (docs and
/// examples may cite URLs); test projects are excluded because tests use
/// literals deliberately as independent oracles.
/// </summary>
public class HardcodedUrlGuardTests
{
private static readonly string[] EnforcedHosts =
{
"portal.autopilotmonitor.com",
"www.autopilotmonitor.com",
"docs.autopilotmonitor.com",
"download.autopilotmonitor.com",
"mcp.autopilotmonitor.com",
"autopilotmonitor-api-eu.azurewebsites.net",
"autopilotmonitor.blob.core.windows.net",
"autopilotmonitoreu.blob.core.windows.net",
"graph.microsoft.com",
"login.microsoftonline.com",
};
/// <summary>Files allowed to carry the literals — the registry itself.</summary>
private static readonly string[] RegistryFiles =
{
Path.Combine("src", "Shared", "AutopilotMonitor.Shared", "Constants.cs"),
};
[Fact]
public void WellKnownHosts_OnlyAppearInTheRegistry()
{
var repoRoot = FindRepoRoot();
var srcDir = Path.Combine(repoRoot, "src");
var violations = new List<string>();
foreach (var file in Directory.EnumerateFiles(srcDir, "*.cs", SearchOption.AllDirectories))
{
var relative = Path.GetRelativePath(repoRoot, file);
var normalized = relative.Replace('/', Path.DirectorySeparatorChar);
// bin/obj artifacts and generated files
if (normalized.Contains($"{Path.DirectorySeparatorChar}bin{Path.DirectorySeparatorChar}") ||
normalized.Contains($"{Path.DirectorySeparatorChar}obj{Path.DirectorySeparatorChar}"))
continue;
// Test projects: literals there are deliberate independent oracles.
if (normalized.Contains("Tests"))
continue;
if (RegistryFiles.Any(r => normalized.EndsWith(r, StringComparison.OrdinalIgnoreCase)))
continue;
var lines = File.ReadAllLines(file);
for (var i = 0; i < lines.Length; i++)
{
var trimmed = lines[i].TrimStart();
// Comment approximation: full-line // and /// comments plus the
// body lines of XML docs and block comments. A URL after code on
// the same line still counts — that is the case being guarded.
if (trimmed.StartsWith("//") || trimmed.StartsWith("*") || trimmed.StartsWith("/*"))
continue;
foreach (var host in EnforcedHosts)
{
if (trimmed.Contains(host, StringComparison.OrdinalIgnoreCase))
violations.Add($"{relative}:{i + 1}: {host}");
}
}
}
Assert.True(violations.Count == 0,
"Hardcoded well-known host(s) found outside the URL registry " +
"(use AutopilotMonitor.Shared.Constants.*BaseUrl instead):\n " +
string.Join("\n ", violations));
}
private static string FindRepoRoot()
{
var dir = new DirectoryInfo(AppContext.BaseDirectory);
while (dir != null && !File.Exists(Path.Combine(dir.FullName, "AutopilotMonitor.sln")))
{
dir = dir.Parent;
}
Assert.NotNull(dir);
return dir!.FullName;
}
}
@@ -1,4 +1,5 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
@@ -87,4 +88,61 @@ public class MaintenanceServiceExcessiveEventAutoActionTests
// — better to no-op than block/kill on an unintended config.
Assert.Null(MaintenanceService.DecideAutoAction(eventCount: 5000, autoMode: mode, autoThreshold: 2500));
}
// ── Status eligibility ────────────────────────────────────────────────────
// A device may only be blocked over a session that can still upload. Without this gate
// a session that finished days ago with a high EventCount got its device blocked on the
// next sweep, long after it stopped sending.
[Fact]
public void IsAutoActionEligible_InProgress_IsEligible()
{
Assert.True(MaintenanceService.IsAutoActionEligible(SessionStatus.InProgress));
}
[Theory]
[InlineData(SessionStatus.Succeeded)]
[InlineData(SessionStatus.Failed)]
[InlineData(SessionStatus.Incomplete)]
[InlineData(SessionStatus.Stalled)]
[InlineData(SessionStatus.AwaitingUser)]
[InlineData(SessionStatus.Pending)]
[InlineData(SessionStatus.Unknown)]
public void IsAutoActionEligible_EverythingElse_IsRejected(SessionStatus status)
{
// Matches the time-window watchdog's `Status eq 'InProgress'` filter, so neither
// detector can block a device over a session that no longer sends. Stalled and
// AwaitingUser can heal back to InProgress — then the next sweep acts.
Assert.False(MaintenanceService.IsAutoActionEligible(status));
}
// ── Status parsing feeding that gate ──────────────────────────────────────
// The runaway query projects a subset of columns. SessionStatus.InProgress is ordinal 0,
// so an unparsed Status would leave every row looking in-progress and the gate above
// silently always-true — these pin the fail-closed mapping.
[Theory]
[InlineData("InProgress", SessionStatus.InProgress)]
[InlineData("inprogress", SessionStatus.InProgress)]
[InlineData("Succeeded", SessionStatus.Succeeded)]
[InlineData("AwaitingUser", SessionStatus.AwaitingUser)]
public void ParseStatusForAutoAction_KnownNames_RoundTrip(string stored, SessionStatus expected)
{
Assert.Equal(expected, TableStorageService.ParseStatusForAutoAction(stored));
}
[Theory]
[InlineData(null)] // column absent from the projection or the row
[InlineData("")]
[InlineData(" ")]
[InlineData("Enrolling")] // status name that never existed
[InlineData("0")] // Enum.TryParse would map this onto InProgress and open the gate
[InlineData("3")]
[InlineData("-1")]
[InlineData("InProgress,Failed")] // flags-style combination is not a real status
public void ParseStatusForAutoAction_UnusableInput_IsUnknown(string? stored)
{
Assert.Equal(SessionStatus.Unknown, TableStorageService.ParseStatusForAutoAction(stored));
Assert.False(MaintenanceService.IsAutoActionEligible(TableStorageService.ParseStatusForAutoAction(stored)));
}
}
@@ -117,7 +117,6 @@ public class OffboardingSessionEnumeratorTests
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetExcessiveDataSendersAsync(string tenantId, DateTime windowCutoff, int maxSessionWindowHours) => throw new NotImplementedException();
public Task<List<string>> GetAllTenantIdsAsync() => throw new NotImplementedException();
public Task<int> DeleteSessionEventsAsync(string tenantId, string sessionId) => throw new NotImplementedException();
public Task<int> DeleteSessionRuleResultsAsync(string tenantId, string sessionId) => throw new NotImplementedException();
@@ -1,7 +1,15 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Functions.Ingest;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Functions.Services.Notifications;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
@@ -80,4 +88,116 @@ public class ReportAgentErrorFunctionTests
var evt = ReportAgentErrorFunction.BuildAgentEmergencyBreakEvent(Report(Now, message: ""), TenantId, null!, Now);
Assert.Contains("emergency break", evt.Message, StringComparison.OrdinalIgnoreCase);
}
// =========================================================================
// MaterializeEmergencyBreakArtifactsAsync — the artifact set per break report
// (2026-07-23 hardening): timeline event + cross-session EventType index row +
// AgentEmergencyBreak ops event, once per session. The index upsert is load-bearing:
// StoreEventsBatchAsync alone leaves the event invisible to every search-by-eventType
// surface (found in the 2026-07-22 incident analysis).
// =========================================================================
private sealed class Harness
{
public readonly Mock<ISessionRepository> SessionRepo = new();
public readonly List<OpsEventEntry> OpsEvents = new();
public readonly OpsEventService OpsService;
public Harness(List<EnrollmentEvent>? existingEvents = null)
{
SessionRepo.Setup(r => r.GetSessionEventsAsync(TenantId, SessionId, It.IsAny<int>()))
.ReturnsAsync(existingEvents ?? new List<EnrollmentEvent>());
SessionRepo.Setup(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()))
.ReturnsAsync((List<EnrollmentEvent> e) => e);
SessionRepo.Setup(r => r.UpsertEventTypeIndexBatchAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()))
.Returns(Task.CompletedTask);
var opsRepo = new Mock<IOpsEventRepository>();
opsRepo.Setup(r => r.SaveOpsEventAsync(It.IsAny<OpsEventEntry>()))
.Callback<OpsEventEntry>(e => { lock (OpsEvents) OpsEvents.Add(e); })
.Returns(Task.CompletedTask);
var adminConfig = new Mock<AdminConfigurationService>(
Mock.Of<IConfigRepository>(), NullLogger<AdminConfigurationService>.Instance,
new MemoryCache(new MemoryCacheOptions()));
var alertDispatch = new OpsAlertDispatchService(
adminConfig.Object,
new TelegramNotificationService(new HttpClient(), Mock.Of<IConfigRepository>(),
NullLogger<TelegramNotificationService>.Instance),
new WebhookNotificationService(new HttpClient(),
NullLogger<WebhookNotificationService>.Instance),
NullLogger<OpsAlertDispatchService>.Instance);
OpsService = new OpsEventService(opsRepo.Object, NullLogger<OpsEventService>.Instance, alertDispatch);
}
public Task RunAsync(AgentErrorReport report)
=> ReportAgentErrorFunction.MaterializeEmergencyBreakArtifactsAsync(
report, TenantId, SessionRepo.Object, OpsService, NullLogger<ReportAgentErrorFunction>.Instance);
}
[Fact]
public async Task First_break_report_materializes_event_index_row_and_ops_event()
{
var h = new Harness();
await h.RunAsync(Report(Now));
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(
It.Is<List<EnrollmentEvent>>(e => e.Count == 1
&& e[0].EventType == AutopilotMonitor.Shared.Constants.EventTypes.AgentEmergencyBreak)), Times.Once);
h.SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
TenantId, SessionId, It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Once);
var ops = Assert.Single(h.OpsEvents);
Assert.Equal("AgentEmergencyBreak", ops.EventType);
Assert.Equal(TenantId, ops.TenantId);
}
[Fact]
public async Task Already_materialized_session_produces_no_second_artifact_set()
{
var h = new Harness(new List<EnrollmentEvent>
{
new() { EventType = AutopilotMonitor.Shared.Constants.EventTypes.AgentEmergencyBreak, Sequence = 45 },
});
await h.RunAsync(Report(Now));
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()), Times.Never);
h.SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Never);
Assert.Empty(h.OpsEvents);
}
[Fact]
public async Task Non_break_error_types_and_missing_session_id_are_ignored()
{
var h = new Harness();
var other = Report(Now);
other.ErrorType = AgentErrorType.ConfigFetchFailed;
await h.RunAsync(other);
var noSession = Report(Now);
noSession.SessionId = null!;
await h.RunAsync(noSession);
h.SessionRepo.Verify(r => r.GetSessionEventsAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<int>()), Times.Never);
Assert.Empty(h.OpsEvents);
}
[Fact]
public async Task Storage_failure_is_swallowed_and_never_throws()
{
var h = new Harness();
h.SessionRepo.Setup(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()))
.ThrowsAsync(new InvalidOperationException("storage down"));
// Must not throw — the always-200 emergency channel must never turn into a retry loop.
await h.RunAsync(Report(Now));
Assert.Empty(h.OpsEvents);
}
}
@@ -64,22 +64,26 @@ public class ValidateBootstrapCodeResponseShapeTests
}
/// <summary>
/// The hardcoded agent download URL in <c>ValidateBootstrapCodeFunction</c> must
/// The agent download URL served by <c>ValidateBootstrapCodeFunction</c> must
/// satisfy the frontend's host allow-list and path regex.
/// </summary>
[Fact]
public void AgentDownloadUrl_MatchesFrontendAllowList()
{
// Keep in sync with ValidateBootstrapCodeFunction.cs — if that literal
// changes, both this assertion and bootstrapValidation.ts must be updated.
const string HardcodedAgentUrl =
"https://autopilotmonitor.blob.core.windows.net/agent/AutopilotMonitor-Agent.zip";
// Independent oracle, deliberately spelled out: if Constants or the function
// change the URL, this literal AND bootstrapValidation.ts (AGENT_DOWNLOAD_HOSTNAMES)
// must be updated consciously — that pairing is what keeps portal and backend in sync.
const string ExpectedAgentUrl =
"https://download.autopilotmonitor.com/agent/AutopilotMonitor-Agent.zip";
Assert.True(Uri.TryCreate(HardcodedAgentUrl, UriKind.Absolute, out var uri),
var built = $"{AutopilotMonitor.Shared.Constants.AgentDownloadBaseUrl}/{AutopilotMonitor.Shared.Constants.AgentZipFileName}";
Assert.Equal(ExpectedAgentUrl, built);
Assert.True(Uri.TryCreate(built, UriKind.Absolute, out var uri),
"AgentDownloadUrl is not a well-formed absolute URI");
Assert.Equal(Uri.UriSchemeHttps, uri!.Scheme);
Assert.Equal("autopilotmonitor.blob.core.windows.net", uri.Host);
Assert.Equal("download.autopilotmonitor.com", uri.Host);
Assert.True(AgentDownloadUrlPath.IsMatch(uri.AbsolutePath),
$"AgentDownloadUrl path '{uri.AbsolutePath}' does not match the frontend allow-list regex");
}
@@ -12,8 +12,16 @@
BuildTimestampUtc is baked into the assembly as AssemblyMetadata so /api/health
reports the actual build moment — not the file mtime, which Azure Functions
resets to the zip-extract time on every cold start. CI may override via
-p:BuildTimestampUtc=<iso8601> to force a specific timestamp. -->
<Version>1.0.0</Version>
-p:BuildTimestampUtc=<iso8601> to force a specific timestamp.
VersionPrefix is the curated part (bump by hand for a release); BuildNumber is
reserved per build from the counter blob versions/backend.txt and passed in as
-p:BuildNumber=<n> by deploy-backend.yml — same mechanism as the agent line.
A local build has no counter and stays at x.y.0 with an empty commit hash,
which is how /api/health tells a developer machine from a CI artifact. -->
<VersionPrefix>1.5</VersionPrefix>
<BuildNumber Condition="'$(BuildNumber)' == ''">0</BuildNumber>
<Version>$(VersionPrefix).$(BuildNumber)</Version>
<GenerateAssemblyInfo>true</GenerateAssemblyInfo>
<IncludeSourceRevisionInInformationalVersion>true</IncludeSourceRevisionInInformationalVersion>
<BuildTimestampUtc Condition="'$(BuildTimestampUtc)' == ''">$([System.DateTime]::UtcNow.ToString('o'))</BuildTimestampUtc>
@@ -590,8 +590,6 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
{ "ExcessiveEventAutoActionMode", config.ExcessiveEventAutoActionMode ?? "Off" },
{ "ExcessiveEventAutoActionThreshold", config.ExcessiveEventAutoActionThreshold },
{ "ExcessiveEventAutoActionDurationHours", config.ExcessiveEventAutoActionDurationHours },
{ "MaxSessionWindowHours", config.MaxSessionWindowHours },
{ "MaintenanceBlockDurationHours", config.MaintenanceBlockDurationHours },
{ "DiagnosticsGlobalLogPathsJson", config.DiagnosticsGlobalLogPathsJson },
{ "ModernDeploymentHarmlessEventIdsJson", config.ModernDeploymentHarmlessEventIdsJson ?? string.Empty },
{ "WhiteGloveSealingPatternIdsJson", config.WhiteGloveSealingPatternIdsJson ?? string.Empty },
@@ -657,8 +655,6 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
ExcessiveEventAutoActionMode = entity.GetString("ExcessiveEventAutoActionMode") ?? "Off",
ExcessiveEventAutoActionThreshold = entity.GetInt32("ExcessiveEventAutoActionThreshold") ?? 2500,
ExcessiveEventAutoActionDurationHours = entity.GetInt32("ExcessiveEventAutoActionDurationHours") ?? 24,
MaxSessionWindowHours = entity.GetInt32("MaxSessionWindowHours") ?? 24,
MaintenanceBlockDurationHours = entity.GetInt32("MaintenanceBlockDurationHours") ?? 12,
DiagnosticsGlobalLogPathsJson = entity.GetString("DiagnosticsGlobalLogPathsJson"),
ModernDeploymentHarmlessEventIdsJson = entity.GetString("ModernDeploymentHarmlessEventIdsJson"),
WhiteGloveSealingPatternIdsJson = entity.GetString("WhiteGloveSealingPatternIdsJson"),
@@ -69,9 +69,6 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff)
=> _storage.GetAgentSilentSessionsAsync(tenantId, silenceCutoff, hardCutoff);
public Task<List<SessionSummary>> GetExcessiveDataSendersAsync(string tenantId, DateTime windowCutoff, int maxSessionWindowHours)
=> _storage.GetExcessiveDataSendersAsync(tenantId, windowCutoff, maxSessionWindowHours);
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults)
=> _storage.GetLegacyTimeoutFailedSessionsAsync(tenantId, maxResults);
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System.Linq;
using System.Net;
using System.Threading;
@@ -107,7 +108,7 @@ public class AutopilotDeviceValidationConsentFunction
}
var consentUrl =
$"https://login.microsoftonline.com/{Uri.EscapeDataString(requestCtx.TargetTenantId)}/adminconsent" +
$"{Constants.EntraLoginBaseUrl}/{Uri.EscapeDataString(requestCtx.TargetTenantId)}/adminconsent" +
$"?client_id={Uri.EscapeDataString(validatorClientId)}" +
$"&redirect_uri={Uri.EscapeDataString(redirectUri)}" +
$"&state={Uri.EscapeDataString("autopilot-device-validation-enable")}";
@@ -4,6 +4,7 @@ using System.Net;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Azure.Functions.Worker;
@@ -92,7 +93,9 @@ namespace AutopilotMonitor.Functions.Functions.Bootstrap
{
Success = true,
ShortCode = session.ShortCode,
BootstrapUrl = $"https://autopilotmonitor.com/go/{session.ShortCode}",
// www directly — the bare apex is only a registrar-level 301 to www,
// so pointing there just adds a redirect hop for every bootstrap.
BootstrapUrl = $"{Constants.WebsiteBaseUrl}/go/{session.ShortCode}",
ExpiresAt = session.ExpiresAt,
Message = $"Bootstrap session created. Valid for {validityHours} hours."
};
@@ -92,7 +92,9 @@ namespace AutopilotMonitor.Functions.Functions.Bootstrap
Success = true,
TenantId = session.TenantId,
Token = session.Token,
AgentDownloadUrl = "https://autopilotmonitor.blob.core.windows.net/agent/AutopilotMonitor-Agent.zip",
// Download alias — the portal's bootstrapValidation.ts allowlists this
// host (plus the legacy blob host for transition); keep the two in sync.
AgentDownloadUrl = $"{Constants.AgentDownloadBaseUrl}/{Constants.AgentZipFileName}",
ExpiresAt = session.ExpiresAt,
Message = "Bootstrap code validated"
};
@@ -37,6 +37,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
private readonly TelemetryClient _telemetryClient;
private readonly BootstrapSessionService _bootstrapSessionService;
private readonly ISessionRepository _sessionRepo;
private readonly OpsEventService _opsEventService;
public ReportAgentErrorFunction(
ILogger<ReportAgentErrorFunction> logger,
@@ -48,7 +49,8 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
DeviceAssociationValidator deviceAssociationValidator,
TelemetryClient telemetryClient,
BootstrapSessionService bootstrapSessionService,
ISessionRepository sessionRepo)
ISessionRepository sessionRepo,
OpsEventService opsEventService)
{
_logger = logger;
_configService = configService;
@@ -60,6 +62,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
_telemetryClient = telemetryClient;
_bootstrapSessionService = bootstrapSessionService;
_sessionRepo = sessionRepo;
_opsEventService = opsEventService;
}
[Function("ReportAgentError")]
@@ -169,35 +172,71 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
["AgentTimestamp"] = report.Timestamp.ToString("O"),
});
// Materialize the agent's silent 48h emergency break as a timeline event so it shows in the
// session and the timeout classifier can see it (tasks/enrollment-status-reclassification.md).
// Best-effort — a failure here must never turn the always-200 emergency channel into a retry loop.
if (report.ErrorType == AgentErrorType.SessionAgeEmergencyBreak
&& !string.IsNullOrEmpty(report.SessionId))
{
try
{
var existing = await _sessionRepo.GetSessionEventsAsync(tenantId, report.SessionId, maxResults: 1000);
// Idempotency: the agent's emergency channel can send up to a few reports per session;
// only ever materialize one timeline event.
var alreadyMaterialized = existing.Any(e =>
string.Equals(e.EventType, Constants.EventTypes.AgentEmergencyBreak, StringComparison.OrdinalIgnoreCase));
if (!alreadyMaterialized)
{
var evt = BuildAgentEmergencyBreakEvent(report, tenantId, existing, DateTime.UtcNow);
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
}
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"ReportAgentError: failed to materialize agent_emergency_break event for session {SessionId}", report.SessionId);
}
}
await MaterializeEmergencyBreakArtifactsAsync(report, tenantId, _sessionRepo, _opsEventService, _logger);
return req.CreateResponse(HttpStatusCode.OK);
}
/// <summary>
/// Materializes the agent's silent 48h emergency break as (1) a timeline event so it shows
/// in the session and the timeout classifier can see it, (2) the cross-session EventType
/// index row, and (3) an <c>AgentEmergencyBreak</c> ops event for operator visibility
/// (tasks/enrollment-status-reclassification.md). Static seam with explicit dependencies so
/// tests can pin the artifact set without booting the Functions HTTP stack.
/// Best-effort — a failure here must never turn the always-200 emergency channel into a
/// retry loop.
/// </summary>
internal static async Task MaterializeEmergencyBreakArtifactsAsync(
AgentErrorReport report,
string tenantId,
ISessionRepository sessionRepo,
OpsEventService opsEventService,
ILogger logger)
{
if (report.ErrorType != AgentErrorType.SessionAgeEmergencyBreak
|| string.IsNullOrEmpty(report.SessionId))
{
return;
}
try
{
var existing = await sessionRepo.GetSessionEventsAsync(tenantId, report.SessionId, maxResults: 1000);
// Idempotency: the agent's emergency channel can send up to a few reports per session;
// only ever materialize one timeline event (and one ops event).
var alreadyMaterialized = existing.Any(e =>
string.Equals(e.EventType, Constants.EventTypes.AgentEmergencyBreak, StringComparison.OrdinalIgnoreCase));
if (alreadyMaterialized)
{
return;
}
var evt = BuildAgentEmergencyBreakEvent(report, tenantId, existing, DateTime.UtcNow);
await sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
// StoreEventsBatchAsync writes the Events partition only — the cross-session
// EventType index is normally written by EventIngestProcessor, which this
// backend-materialized event never passes through. Without the upsert the event
// exists on the session timeline but is invisible to every search-by-eventType
// surface (portal cross-session search, MCP search_sessions_by_event /
// query_raw_events) — found the hard way in the 2026-07-22 incident analysis.
await sessionRepo.UpsertEventTypeIndexBatchAsync(
tenantId, report.SessionId, new List<EnrollmentEvent> { evt });
// Operator visibility: an emergency break means an agent silently gave up at its
// absolute age cap — exactly the "are we losing agents?" signal. Emitted only on
// first materialization so repeat reports for the same session cannot flood the
// ops feed. OpsEventService never throws.
await opsEventService.RecordAgentEmergencyBreakAsync(
tenantId, report.SessionId, report.AgentVersion, evt.Message);
}
catch (Exception ex)
{
logger.LogWarning(ex,
"ReportAgentError: failed to materialize agent_emergency_break event for session {SessionId}", report.SessionId);
}
}
/// <summary>
/// Builds the backend-materialized <c>agent_emergency_break</c> timeline event from the agent's
/// best-effort emergency report. Static + pure (analog to
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System.Net;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Services;
@@ -351,7 +352,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
if (startChannels.Count == 0)
return;
var sessionUrl = $"https://portal.autopilotmonitor.com/sessions/{sessionId}";
var sessionUrl = $"{Constants.PortalBaseUrl}/sessions/{sessionId}";
var alert = NotificationAlertBuilder.BuildEnrollmentStartedAlert(
session.DeviceName,
session.SerialNumber,
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Azure.Functions.Worker.Middleware;
@@ -6,6 +7,7 @@ using Microsoft.Extensions.Logging;
using Microsoft.IdentityModel.Protocols;
using Microsoft.IdentityModel.Protocols.OpenIdConnect;
using Microsoft.IdentityModel.Tokens;
using System.Diagnostics.CodeAnalysis;
using System.IdentityModel.Tokens.Jwt;
using Microsoft.AspNetCore.Http;
using System.Net;
@@ -248,7 +250,7 @@ public class AuthenticationMiddleware : IFunctionsWorkerMiddleware
/// missing/garbage tid here prevents an outbound OIDC-metadata fetch to a bogus authority on
/// malformed-token floods. Loss-free — every legitimate Entra token carries a GUID tid.
/// </summary>
internal static bool IsValidTenantId(string? tid) => Guid.TryParse(tid, out _);
internal static bool IsValidTenantId([NotNullWhen(true)] string? tid) => Guid.TryParse(tid, out _);
/// <summary>
/// Maps a token issuer + tenant id to the OIDC authority to fetch signing keys from. Entra v1.0
@@ -259,8 +261,8 @@ public class AuthenticationMiddleware : IFunctionsWorkerMiddleware
{
var isV1Token = issuer != null && issuer.Contains("sts.windows.net");
return isV1Token
? $"https://login.microsoftonline.com/{tenantId}" // v1.0
: $"https://login.microsoftonline.com/{tenantId}/v2.0"; // v2.0
? $"{Constants.EntraLoginBaseUrl}/{tenantId}" // v1.0
: $"{Constants.EntraLoginBaseUrl}/{tenantId}/v2.0"; // v2.0
}
/// <summary>
@@ -283,7 +285,7 @@ public class AuthenticationMiddleware : IFunctionsWorkerMiddleware
ValidateIssuer = true,
IssuerValidator = (iss, token, parameters) =>
{
if (iss.StartsWith("https://login.microsoftonline.com/") ||
if (iss.StartsWith(Constants.EntraLoginBaseUrl + "/") ||
iss.StartsWith("https://sts.windows.net/"))
{
return iss;
@@ -104,7 +104,8 @@ builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
options =>
{
// Multi-Tenant Configuration
options.Instance = "https://login.microsoftonline.com/";
// Fully qualified: Microsoft.Identity.Web also exports a 'Constants' type.
options.Instance = AutopilotMonitor.Shared.Constants.EntraLoginBaseUrl + "/";
options.TenantId = "organizations"; // Accept tokens from any Azure AD tenant
options.ClientId = builder.Configuration["EntraId:ClientId"];
@@ -117,7 +118,7 @@ builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
ValidateIssuerSigningKey = true,
ValidIssuers = new[]
{
"https://login.microsoftonline.com/organizations/v2.0",
AutopilotMonitor.Shared.Constants.EntraLoginBaseUrl + "/organizations/v2.0",
"https://sts.windows.net/{tenantid}/"
},
// Temporarily accept Microsoft Graph tokens (used by frontend with User.Read scope)
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Linq;
using System.Net.Http.Headers;
@@ -119,7 +120,7 @@ namespace AutopilotMonitor.Functions.Security
// Use contains for server-side narrowing, then perform exact match client-side.
var escapedSerial = normalizedSerial.Replace("'", "''");
var filter = Uri.EscapeDataString($"contains(serialNumber,'{escapedSerial}')");
var graphUrl = $"https://graph.microsoft.com/v1.0/deviceManagement/windowsAutopilotDeviceIdentities?$top=100&$filter={filter}";
var graphUrl = $"{Constants.GraphBaseUrl}/v1.0/deviceManagement/windowsAutopilotDeviceIdentities?$top=100&$filter={filter}";
var response = await graphClient.GetAsync(graphUrl);
var responseBody = await response.Content.ReadAsStringAsync();
@@ -127,7 +127,7 @@ namespace AutopilotMonitor.Functions.Security
// Parse certificate from Base64
var certBytes = Convert.FromBase64String(certificateBase64);
var certificate = new X509Certificate2(certBytes);
var certificate = X509CertificateLoader.LoadCertificate(certBytes);
var thumbprint = certificate.Thumbprint;
logger?.LogDebug($"Validating certificate: Subject={certificate.Subject}, Issuer={certificate.Issuer}, Thumbprint={thumbprint}");
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Net.Http;
using System.Net.Http.Headers;
@@ -132,7 +133,7 @@ namespace AutopilotMonitor.Functions.Security
Encoding.UTF8,
"application/json");
var graphUrl = "https://graph.microsoft.com/beta/deviceManagement/importedDeviceIdentities/searchExistingIdentities";
var graphUrl = Constants.GraphBaseUrl + "/beta/deviceManagement/importedDeviceIdentities/searchExistingIdentities";
var response = await graphClient.PostAsync(graphUrl, content);
var responseBody = await response.Content.ReadAsStringAsync();
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Linq;
using System.Net.Http;
@@ -118,7 +119,7 @@ namespace AutopilotMonitor.Functions.Security
var escapedSerial = normalizedSerial.Replace("'", "''");
var filter = Uri.EscapeDataString($"contains(serialNumber,'{escapedSerial}')");
var orderby = Uri.EscapeDataString("preAssociationDateTime desc");
var graphUrl = "https://graph.microsoft.com/beta/deviceManagement/tenantAssociatedDevices"
var graphUrl = Constants.GraphBaseUrl + "/beta/deviceManagement/tenantAssociatedDevices"
+ $"?$top=25&$filter={filter}&$orderby={orderby}";
var response = await graphClient.GetAsync(graphUrl);
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
@@ -169,7 +170,7 @@ namespace AutopilotMonitor.Functions.Security
// while this POST is in flight, the generation changes and the token we cache below is
// tagged with the now-stale value — so readers reject it and re-mint fresh.
var genAtStart = CurrentGen(tenantId);
var tokenUrl = $"https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token";
var tokenUrl = $"{Constants.EntraLoginBaseUrl}/{tenantId}/oauth2/v2.0/token";
// Retry with backoff to handle Azure AD consent propagation delays.
// After admin consent is granted, the service principal may not be immediately
@@ -196,7 +197,7 @@ namespace AutopilotMonitor.Functions.Security
{
["client_id"] = clientId,
["client_secret"] = clientSecret,
["scope"] = "https://graph.microsoft.com/.default",
["scope"] = Constants.GraphBaseUrl + "/.default",
["grant_type"] = "client_credentials"
});
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Collections.Generic;
using System.Linq;
@@ -176,7 +177,7 @@ namespace AutopilotMonitor.Functions.Services.Analyze
return;
var session = await _sessionRepo.GetSessionAsync(envelope.TenantId, envelope.SessionId).ConfigureAwait(false);
var sessionUrl = $"https://portal.autopilotmonitor.com/sessions/{envelope.SessionId}";
var sessionUrl = $"{Constants.PortalBaseUrl}/sessions/{envelope.SessionId}";
foreach (var (result, rule) in candidates)
{
@@ -1,3 +1,5 @@
using AutopilotMonitor.Shared;
namespace AutopilotMonitor.Functions.Services;
/// <summary>
@@ -97,7 +99,7 @@ public static class EmailTemplates
<table cellpadding=""0"" cellspacing=""0"" style=""margin:0 auto 32px;"">
<tr>
<td style=""background-color:#2563eb; border-radius:8px;"">
<a href=""https://www.autopilotmonitor.com/docs"" target=""_blank""
<a href=""{Constants.DocsBaseUrl}"" target=""_blank""
style=""display:inline-block; padding:14px 32px; color:#ffffff; font-size:15px; font-weight:600; text-decoration:none;"">
View Documentation
</a>
@@ -147,9 +147,12 @@ namespace AutopilotMonitor.Functions.Services
helloResolved = true;
else if (Eq(type, "realmjoin_detected"))
realmJoinDetected = true;
// Either terminal opens the agent-side RealmJoin gate: phase 110 or the 60-min
// hard timeout (both dual-emitted to the timeline by the agent/engine).
else if (Eq(type, "realmjoin_resolved") || Eq(type, "realmjoin_timeout"))
// Any terminal opens the agent-side RealmJoin gate: phase 110, the aborted
// first deployment (phase left 100/101 for 200/210 without 110 — session
// 224b2087) or the 60-min hard timeout (all emitted to the timeline by the
// agent/engine).
else if (Eq(type, "realmjoin_resolved") || Eq(type, "realmjoin_timeout")
|| Eq(type, "realmjoin_first_deployment_incomplete"))
realmJoinResolved = true;
// Policy facts for the Hello-disabled / User-ESP-skipped completion mirror.
// Contradicting observations are resolved pessimistically below (both-seen →
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System.Linq;
using AutopilotMonitor.Functions.Services.Notifications;
using AutopilotMonitor.Shared.Models;
@@ -35,7 +36,7 @@ namespace AutopilotMonitor.Functions.Services
return;
var sessionUrl = updatedSession != null
? $"https://portal.autopilotmonitor.com/sessions/{request.SessionId}"
? $"{Constants.PortalBaseUrl}/sessions/{request.SessionId}"
: null;
// A failure alert requires an actual failure-ish verdict: an agent_timeout
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Collections.Generic;
using System.Globalization;
@@ -37,7 +38,7 @@ public sealed class ScriptDisplayNameResolver : IScriptDisplayNameResolver
/// </summary>
internal static readonly TimeSpan TokenAcquireBudget = TimeSpan.FromMilliseconds(1500);
private const string GraphBetaBase = "https://graph.microsoft.com/beta";
private const string GraphBetaBase = Constants.GraphBaseUrl + "/beta";
private const int PageSize = 100;
private readonly IGraphFeatureDetector _detector;
@@ -3,6 +3,7 @@ using System.Net.Http;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Shared;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
@@ -27,7 +28,10 @@ namespace AutopilotMonitor.Functions.Services
public sealed class LatestVersionsService : ILatestVersionsService
{
public const string VersionJsonUrl = "https://autopilotmonitor.blob.core.windows.net/agent/version.json";
// Download alias (Front Door → current blob origin) — NOT the legacy blob
// account. The legacy mirror in build-agent.yml is fail-soft, so reading from
// it could silently serve a stale version while the release pipeline is green.
public const string VersionJsonUrl = Constants.AgentDownloadBaseUrl + "/" + Constants.AgentVersionFileName;
private const string CacheKey = "latest-versions";
private static readonly TimeSpan CacheDuration = TimeSpan.FromHours(12);
private static readonly TimeSpan FailureCacheDuration = TimeSpan.FromMinutes(5);
@@ -1,4 +1,4 @@
using System;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
@@ -134,7 +134,6 @@ namespace AutopilotMonitor.Functions.Services
{
await MarkStalledSessionsAsTimedOutAsync();
await DetectExcessiveEventSessionsAsync();
await BlockExcessiveDataSendersAsync();
await AggregateMetricsWithCatchUpAsync();
// Plan §5 PR6 / §16 R14: session retention fanout extracted out of the 2h timer
// into the dedicated 12h SessionDeletionMaintenanceFunction so cascade-lifecycle
@@ -440,6 +439,13 @@ namespace AutopilotMonitor.Functions.Services
var timeoutEvent = BuildSessionTimeoutEvent(session, timeoutHours, sessionEvents, now);
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { timeoutEvent });
// Backend-materialized events bypass EventIngestProcessor and thus the
// cross-session EventType index — without this upsert session_timeout is
// invisible to every search-by-eventType surface (portal cross-session
// search, MCP search_sessions_by_event / query_raw_events).
await _sessionRepo.UpsertEventTypeIndexBatchAsync(
session.TenantId, session.SessionId, new List<EnrollmentEvent> { timeoutEvent });
// StoreEventsBatchAsync only bumps the orphan side-index, not the
// session's EventCount, and the terminal reconcile already ran inside
// UpdateSessionStatusAsync *before* this synthetic event existed. Recount
@@ -563,6 +569,16 @@ namespace AutopilotMonitor.Functions.Services
return null;
}
/// <summary>
/// Pure eligibility gate for the runaway-session auto-action path: only a session that
/// can still upload may cost its device a block. Mirrors the time-window watchdog's
/// <c>Status eq 'InProgress'</c> filter, so neither detector ever blocks a device over a
/// session that stopped sending — without it, a session that finished days ago with a
/// high EventCount still gets its device blocked on the next sweep.
/// The warn tier deliberately stays status-agnostic: it is forensic and blocks nothing.
/// </summary>
internal static bool IsAutoActionEligible(SessionStatus status) => status == SessionStatus.InProgress;
/// <summary>
/// Scans every tenant for sessions whose EventCount exceeds the configured warn
/// threshold (<see cref="AdminConfiguration.ExcessiveEventCountThreshold"/>) or the
@@ -580,6 +596,14 @@ namespace AutopilotMonitor.Functions.Services
/// mode mid-run never re-fires the warn.
/// </para>
/// Both paths skip when their threshold is 0; the whole sweep no-ops when both are off.
/// <para>
/// This is the ONLY automatic device block. A second, time-window detector used to run
/// beside it (AdminConfiguration.MaxSessionWindowHours) and blocked any session whose
/// first and last event lay more than N hours apart. It measured span, never volume, so
/// it flagged every enrollment that spanned a night — 23-event sessions included — and
/// was removed 2026-07-22. Event count is the quantity that actually costs anything;
/// if a span-based guard is ever reintroduced it has to gate on volume too.
/// </para>
/// </summary>
private async Task DetectExcessiveEventSessionsAsync()
{
@@ -637,6 +661,14 @@ namespace AutopilotMonitor.Functions.Services
var action = DecideAutoAction(session.EventCount, autoMode, autoThreshold);
if (action == null) continue;
if (!IsAutoActionEligible(session.Status))
{
_logger.LogInformation(
"Skipping auto-action for runaway session {SessionId} (tenant {TenantId}): status {Status} can no longer upload",
session.SessionId, tenantId, session.Status);
continue;
}
if (string.IsNullOrEmpty(session.SerialNumber))
{
_logger.LogWarning(
@@ -645,6 +677,19 @@ namespace AutopilotMonitor.Functions.Services
continue;
}
// Don't re-block an already-blocked device: BlockDeviceAsync would
// overwrite BlockedAt/UnblockAt/Reason of the existing block — most
// often one the time-window watchdog placed on the same session,
// whose reason then silently disappears from the Active Blocks list.
var (alreadyBlocked, _, _, _) = await _blockedDeviceService.IsBlockedAsync(tenantId, session.SerialNumber);
if (alreadyBlocked)
{
_logger.LogInformation(
"Skipping auto-action for runaway session {SessionId} (tenant {TenantId}): device {SerialNumber} is already blocked",
session.SessionId, tenantId, session.SerialNumber);
continue;
}
var reason = $"Auto-action: excessive session events ({session.EventCount} events ≥ threshold {autoThreshold})";
await _blockedDeviceService.BlockDeviceAsync(
tenantId,
@@ -680,108 +725,6 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// Detects devices that are still actively sending data beyond the configured maximum session
/// window and blocks them automatically. Status-independent: uses LastEventAt (written on every
/// event batch) combined with StartedAt to identify sessions spanning too long a data window.
/// Controlled by AdminConfiguration.MaxSessionWindowHours (0 = disabled).
/// </summary>
private async Task<int> BlockExcessiveDataSendersAsync()
{
_logger.LogInformation("Checking for excessive data senders...");
var sw = Stopwatch.StartNew();
try
{
var adminConfig = await _adminConfigurationService.GetConfigurationAsync();
if (adminConfig.MaxSessionWindowHours == 0)
{
_logger.LogInformation("Excessive data sender detection disabled (MaxSessionWindowHours = 0)");
return 0;
}
var windowCutoff = DateTime.UtcNow.AddHours(-adminConfig.MaxSessionWindowHours);
var blockDurationHours = adminConfig.MaintenanceBlockDurationHours;
var tenantIds = await _maintenanceRepo.GetAllTenantIdsAsync();
int totalBlocked = 0;
foreach (var tenantId in tenantIds)
{
try
{
var sessions = await _maintenanceRepo.GetExcessiveDataSendersAsync(tenantId, windowCutoff, adminConfig.MaxSessionWindowHours);
if (sessions.Count == 0)
{
_logger.LogInformation($"Tenant {tenantId}: No excessive data sender sessions found (window: {adminConfig.MaxSessionWindowHours}h)");
continue;
}
int blockedCount = 0;
foreach (var session in sessions)
{
if (string.IsNullOrEmpty(session.SerialNumber))
continue;
var (isBlocked, _, _, _) = await _blockedDeviceService.IsBlockedAsync(tenantId, session.SerialNumber);
if (isBlocked)
continue;
var reason = $"Excessive data window: session active for >{adminConfig.MaxSessionWindowHours}h " +
$"(started {session.StartedAt:yyyy-MM-dd HH:mm:ss} UTC, " +
$"last event {session.LastEventAt:yyyy-MM-dd HH:mm:ss} UTC)";
await _blockedDeviceService.BlockDeviceAsync(
tenantId,
session.SerialNumber,
durationHours: blockDurationHours,
blockedByEmail: "System.Maintenance",
reason: reason,
blockedSessionId: session.SessionId);
blockedCount++;
}
if (blockedCount > 0)
{
await _maintenanceRepo.LogAuditEntryAsync(
tenantId,
"ExcessiveDataBlock",
"Device",
$"{blockedCount} devices",
"System.Maintenance",
new Dictionary<string, string>
{
{ "DevicesBlocked", blockedCount.ToString() },
{ "WindowHours", adminConfig.MaxSessionWindowHours.ToString() },
{ "BlockDurationHours", blockDurationHours.ToString() }
});
_logger.LogInformation($"Tenant {tenantId}: Blocked {blockedCount} excessive data sender device(s) (window: {adminConfig.MaxSessionWindowHours}h, block: {blockDurationHours}h)");
await _opsEventService.RecordExcessiveDataBlockedAsync(tenantId, blockedCount, adminConfig.MaxSessionWindowHours);
}
totalBlocked += blockedCount;
}
catch (Exception ex)
{
_logger.LogError(ex, $"Failed to check excessive data senders for tenant {tenantId}");
}
}
sw.Stop();
_logger.LogInformation($"Excessive data sender check completed: {totalBlocked} devices blocked in {sw.ElapsedMilliseconds}ms");
return totalBlocked;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to check for excessive data senders");
return 0;
}
}
/// <summary>
/// Aggregates metrics for any missed days in the last 7 days, plus yesterday.
/// Checks the UsageMetrics table for existing snapshots to avoid re-aggregation.
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Collections.Generic;
using System.Linq;
@@ -167,7 +168,7 @@ namespace AutopilotMonitor.Functions.Services.Notifications
},
Actions = new List<NotificationAction>
{
new() { Type = "openUrl", Title = "Open Autopilot Monitor", Url = "https://portal.autopilotmonitor.com" },
new() { Type = "openUrl", Title = "Open Autopilot Monitor", Url = Constants.PortalBaseUrl },
},
};
}
@@ -255,10 +255,9 @@ namespace AutopilotMonitor.Functions.Services
$"Device {serialNumber} blocked: {reason}",
tenantId, blockedBy, new { serialNumber, reason });
public Task RecordExcessiveDataBlockedAsync(string tenantId, int devicesBlocked, int windowHours)
=> WriteAsync(OpsEventCategory.Security, "ExcessiveDataBlocked", OpsEventSeverity.Warning,
$"{devicesBlocked} device(s) auto-blocked for excessive data (>{windowHours}h window)",
tenantId, "System.Maintenance", new { devicesBlocked, windowHours });
// ExcessiveDataBlocked was removed 2026-07-22 together with the time-window detector that
// raised it — it blocked on session span alone and fired on ordinary overnight enrollments.
// Automatic blocks now come only from the event-count path below.
public Task RecordVersionBlockedAsync(string pattern, string blockedBy)
=> WriteAsync(OpsEventCategory.Security, "VersionBlocked", OpsEventSeverity.Warning,
@@ -494,6 +493,20 @@ namespace AutopilotMonitor.Functions.Services
$"{sessionCount} session(s) timed out after {timeoutHours}h",
tenantId, "System.Maintenance", new { sessionCount, timeoutHours });
/// <summary>
/// An agent reported its absolute session-age emergency break (48h cap) over the
/// emergency channel — it cleaned itself up and exited on a session that never reached
/// a terminal state. This is the "are we silently losing agents?" signal; emitted by
/// <see cref="Functions.Ingest.ReportAgentErrorFunction"/> once per session (guarded by
/// the timeline-event idempotency check). Warning-tier so operators can wire a Telegram
/// rule; if real-world volume turns out noisy, downgrade or remove — the timeline event
/// on the session is the durable record.
/// </summary>
public Task RecordAgentEmergencyBreakAsync(string tenantId, string sessionId, string? agentVersion, string message)
=> WriteAsync(OpsEventCategory.Agent, "AgentEmergencyBreak", OpsEventSeverity.Warning,
$"Agent emergency break on session {sessionId} (agent {agentVersion ?? "?"}): {message}",
tenantId, "System.EmergencyChannel", new { sessionId, agentVersion });
public Task RecordExcessiveSessionEventsAsync(string tenantId, string sessionId, int eventCount, int threshold)
=> WriteAsync(OpsEventCategory.Agent, "ExcessiveSessionEvents", OpsEventSeverity.Warning,
$"Session {sessionId} has {eventCount} events (threshold {threshold}) — likely agent loop bug",
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Collections.Generic;
using System.Diagnostics;
@@ -29,7 +30,7 @@ namespace AutopilotMonitor.Functions.Services
/// </summary>
public class SlaBreachEvaluationService
{
private const string DashboardUrl = "https://www.autopilotmonitor.com/sla";
private const string DashboardUrl = Constants.WebsiteBaseUrl + "/sla";
private const int MaxConflictRetries = 4;
private const int MinAppInstallSampleSize = 5;
@@ -1,4 +1,4 @@
using Azure;
using Azure;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Pagination;
@@ -345,8 +345,8 @@ namespace AutopilotMonitor.Functions.Services
}
}
// Suppress automated maintenance entries (SessionTimeout, ExcessiveDataBlock,
// DataRetentionCleanup, …) from the human-facing audit list. They are written
// Suppress automated maintenance entries (SessionTimeout, DataRetentionCleanup, …)
// from the human-facing audit list. They are written
// by MaintenanceService for traceability but duplicate the OpsEvents stream
// and are not actionable as user-attributable audits. Rows stay in storage
// and remain reachable via raw table queries.
@@ -792,73 +792,6 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// Gets sessions where the device has been actively sending data for longer than
/// <paramref name="maxSessionWindowHours"/>.
/// Status-independent: detects excessive data senders regardless of session status.
/// Uses LastEventAt (written on every event batch) for the "still active" check.
/// Sessions without LastEventAt (predating this field) are not returned.
///
/// The OData pre-filter narrows candidates to sessions that straddle the cutoff boundary,
/// then a post-filter verifies the actual session duration (LastEventAt − StartedAt)
/// exceeds the allowed window. This prevents false positives from short sessions that
/// merely happen to straddle the cutoff time.
/// </summary>
public async Task<List<SessionSummary>> GetExcessiveDataSendersAsync(string tenantId, DateTime windowCutoff, int maxSessionWindowHours)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.Sessions);
// OData pre-filter: narrow to sessions that straddle the cutoff boundary.
// Status eq 'InProgress' → only block devices whose session is still actively
// sending data. Completed sessions (Succeeded/Failed) must NOT be re-blocked
// even if their LastEventAt is still within the window — they cannot continue
// to abuse data transfer. This also defends against ghost-blocks caused by
// devices with bad clocks: an agent that submits events with timestamps from
// weeks in the past pushes StartedAt back, making the session look long-lived;
// but once the session is Succeeded/Failed, no further data flows from it.
// IsPreProvisioned ne true → exclude WhiteGlove sessions: a pre-provisioned device
// that resumes after weeks in storage looks like an excessive sender (StartedAt old,
// LastEventAt recent) but is a legitimate resumption, not abuse.
var cutoffStr = windowCutoff.ToString("yyyy-MM-ddTHH:mm:ss");
var filter = $"PartitionKey eq '{tenantId}' " +
$"and Status eq 'InProgress' " +
$"and LastEventAt gt datetime'{cutoffStr}Z' " +
$"and StartedAt lt datetime'{cutoffStr}Z' " +
$"and IsPreProvisioned ne true";
var query = tableClient.QueryAsync<TableEntity>(filter: filter);
var maxDuration = TimeSpan.FromHours(maxSessionWindowHours);
var sessions = new List<SessionSummary>();
await foreach (var entity in query)
{
// Post-filter: verify actual session duration exceeds the window.
// OData cannot compute date differences, so we check in code.
var startedAt = entity.GetDateTimeOffset("StartedAt")?.UtcDateTime;
var lastEventAt = entity.GetDateTimeOffset("LastEventAt")?.UtcDateTime;
if (startedAt.HasValue && lastEventAt.HasValue
&& (lastEventAt.Value - startedAt.Value) < maxDuration)
{
continue;
}
sessions.Add(MapToSessionSummary(entity));
}
return sessions;
}
catch (Exception ex)
{
_logger.LogError(ex, $"Failed to get excessive data sender sessions for tenant {tenantId}");
return new List<SessionSummary>();
}
}
/// <summary>
/// Gets all known tenant IDs from the TenantConfiguration table (one "config" row per
/// tenant). This replaces an O(corpus) scan of the entire Sessions table — the maintenance
@@ -3005,6 +3005,26 @@ namespace AutopilotMonitor.Functions.Services
#region Excessive-Event Detection
/// <summary>
/// Status parse for the runaway-session rows. Deliberately stricter than
/// <c>MapToSessionSummary</c>, which reads a missing column as InProgress for legacy rows:
/// here a missing, empty, numeric or unknown value maps to <see cref="SessionStatus.Unknown"/>
/// so the auto-block/kill gate fails closed and no device is blocked over a session that
/// cannot be classified. The two explicit rejections exist because <c>Enum.TryParse</c> is
/// lenient in ways that would open the gate: it maps the string "0" straight onto
/// InProgress, and it ORs comma-separated names into a single unrelated member.
/// </summary>
internal static SessionStatus ParseStatusForAutoAction(string? statusString)
{
if (string.IsNullOrWhiteSpace(statusString)) return SessionStatus.Unknown;
if (!char.IsLetter(statusString[0])) return SessionStatus.Unknown;
if (statusString.Contains(',')) return SessionStatus.Unknown;
return Enum.TryParse<SessionStatus>(statusString, ignoreCase: true, out var status)
&& Enum.IsDefined(status)
? status
: SessionStatus.Unknown;
}
/// <summary>
/// Returns sessions in the given tenant whose EventCount exceeds the threshold.
/// Projects only the fields needed for the idempotency check — avoids full entity read.
@@ -3023,7 +3043,10 @@ namespace AutopilotMonitor.Functions.Services
// SerialNumber + ExcessiveEventsAutoActioned added for the auto-block/kill
// path in MaintenanceService — lets us decide and execute in one round-trip
// per qualifying session, without a follow-up GetSessionAsync.
select: new[] { "PartitionKey", "RowKey", "EventCount", "SerialNumber", "ExcessiveEventsAlerted", "ExcessiveEventsAutoActioned" });
// Status is mandatory for that path: SessionStatus.InProgress is ordinal 0, so
// an unselected column would leave every row looking in-progress and the
// auto-action eligibility gate silently always-true.
select: new[] { "PartitionKey", "RowKey", "EventCount", "SerialNumber", "Status", "ExcessiveEventsAlerted", "ExcessiveEventsAutoActioned" });
await foreach (var entity in query)
{
@@ -3033,6 +3056,7 @@ namespace AutopilotMonitor.Functions.Services
SessionId = entity.RowKey,
EventCount = SafeGetInt32(entity, "EventCount") ?? 0,
SerialNumber = entity.GetString("SerialNumber") ?? string.Empty,
Status = ParseStatusForAutoAction(entity.GetString("Status")),
ExcessiveEventsAlerted = entity.GetBoolean("ExcessiveEventsAlerted") ?? false,
ExcessiveEventsAutoActioned = entity.GetBoolean("ExcessiveEventsAutoActioned") ?? false
});
@@ -89,6 +89,16 @@ COPY --from=build /app/docs/ ./docs/
ARG DOCS_COMMIT=unknown
ENV DOCS_COMMIT=${DOCS_COMMIT}
# Build identity of the server itself (deploy-mcp.yml supplies all three).
# BUILD_NUMBER completes the version as <major>.<minor>.<build>; left unset in a
# local `docker build`, which keeps package.json's version and marks it as such.
ARG BUILD_NUMBER=
ARG BUILD_COMMIT=unknown
ARG BUILD_UTC=unknown
ENV BUILD_NUMBER=${BUILD_NUMBER}
ENV BUILD_COMMIT=${BUILD_COMMIT}
ENV BUILD_UTC=${BUILD_UTC}
ENV PORT=8080
ENV RULES_DIR=/app/rules
ENV DOCS_DIR=/app/docs
@@ -0,0 +1,69 @@
import { describe, it, expect } from 'vitest';
import { readdirSync, readFileSync, statSync } from 'node:fs';
import { join, relative } from 'node:path';
/**
* Enforces the URL registry: every well-known own or Microsoft host must be
* referenced through src/config.ts, never as a repeated string literal (the
* copy-paste drift config.ts was created to end — see its header comment).
* Comment lines are ignored; tests are excluded (literals there are deliberate
* independent oracles).
*/
const ENFORCED_HOSTS = [
'portal.autopilotmonitor.com',
'www.autopilotmonitor.com',
'docs.autopilotmonitor.com',
'download.autopilotmonitor.com',
'mcp.autopilotmonitor.com',
'autopilotmonitor-api-eu.azurewebsites.net',
'autopilotmonitor.blob.core.windows.net',
'autopilotmonitoreu.blob.core.windows.net',
'graph.microsoft.com',
'login.microsoftonline.com',
] as const;
const SRC_ROOT = join(__dirname, '..');
/** The registry itself — the only file allowed to carry the literals. */
const REGISTRY_FILES = ['config.ts'];
function collectFiles(dir: string, acc: string[]): void {
for (const entry of readdirSync(dir)) {
if (entry === '__tests__' || entry.startsWith('.')) continue;
const full = join(dir, entry);
if (statSync(full).isDirectory()) {
collectFiles(full, acc);
} else if (/\.ts$/.test(entry) && !/\.test\.ts$/.test(entry)) {
acc.push(full);
}
}
}
describe('hardcoded URL guard', () => {
it('well-known hosts only appear in the registry (src/config.ts)', () => {
const files: string[] = [];
collectFiles(SRC_ROOT, files);
const violations: string[] = [];
for (const file of files) {
const rel = relative(SRC_ROOT, file).replace(/\\/g, '/');
if (REGISTRY_FILES.includes(rel)) continue;
const lines = readFileSync(file, 'utf-8').split('\n');
lines.forEach((line, i) => {
const trimmed = line.trimStart();
if (trimmed.startsWith('//') || trimmed.startsWith('*') || trimmed.startsWith('/*')) return;
for (const host of ENFORCED_HOSTS) {
if (trimmed.includes(host)) violations.push(`src/${rel}:${i + 1}: ${host}`);
}
});
}
expect(
violations,
'Hardcoded well-known host(s) found outside src/config.ts — import the constant instead:\n ' +
violations.join('\n '),
).toEqual([]);
});
});
@@ -28,6 +28,12 @@ function resolveApiBaseUrl(): string {
/** Resolved backend API base URL (no trailing slash). */
export const API_BASE_URL = resolveApiBaseUrl();
/** Entra ID login/token authority host (no trailing slash). */
export const ENTRA_LOGIN_BASE_URL = 'https://login.microsoftonline.com';
/** Published customer documentation — base for citable search_docs links. */
export const DOCS_BASE_URL = 'https://docs.autopilotmonitor.com';
/**
* Parse a positive-integer environment variable, falling back to `fallback`
* for missing, non-numeric, or non-positive values. A bare `parseInt` returns
@@ -23,9 +23,7 @@ import { readdir, readFile } from 'node:fs/promises';
import type { Dirent } from 'node:fs';
import { join, relative, sep } from 'node:path';
import type { SearchDocument } from './search-provider.js';
/** Published base URL of the bundle — used to build a citable link per chunk. */
const DOCS_BASE_URL = 'https://docs.autopilotmonitor.com';
import { DOCS_BASE_URL } from './config.js';
/**
* Upper bound on chunk text. MiniLM-L6-v2's 256-token window is roughly 1000-1200
@@ -23,7 +23,21 @@ import { API_BASE_URL } from './config.js';
const __dirname = dirname(fileURLToPath(import.meta.url));
const pkg = JSON.parse(readFileSync(resolve(__dirname, '..', 'package.json'), 'utf-8')) as { version: string };
const SERVER_VERSION: string = pkg.version;
// Version contract shared with the agent and the backend: <major>.<minor>.<build>,
// where major.minor is curated in package.json and <build> is reserved per build
// from the counter blob versions/mcp.txt and baked in by deploy-mcp.yml. Without
// the env var (any local run) the package.json version stands as-is, which is how
// /health and the MCP handshake tell a workstation from a deployed image.
const BUILD_NUMBER = process.env.BUILD_NUMBER ?? '';
const SERVER_VERSION: string = BUILD_NUMBER
? `${pkg.version.split('.').slice(0, 2).join('.')}.${BUILD_NUMBER}`
: pkg.version;
// Commit of THIS repo the image was built from. Counterpart to DOCS_COMMIT below:
// that one tracks the documentation bundle, this one the server itself.
const BUILD_COMMIT = process.env.BUILD_COMMIT ?? 'unknown';
const BUILD_UTC = process.env.BUILD_UTC ?? 'unknown';
/**
* Boot stopwatch. minReplicas=0 means every idle period ends in a cold start a user
@@ -299,6 +313,9 @@ app.use(createOAuthRouter());
// The version is the same value already advertised in the MCP handshake
// (createMcpServer → { name, version }), so surfacing it here leaks nothing
// new; it lets the backend health dashboard show the deployed MCP build.
// commit/buildUtc mirror what the backend's own /api/health has always
// reported (also anonymous): an opaque SHA carries no repository content and
// turns "which build is this?" into one request instead of an image lookup.
//
// The docs block exists for one reason: the documentation bundle lives in its own
// repo and is baked in at image build time, so editing the docs does NOT update a
@@ -310,6 +327,8 @@ app.get('/health', (_req, res) => {
res.json({
status: 'healthy',
version: SERVER_VERSION,
commit: BUILD_COMMIT,
buildUtc: BUILD_UTC,
docs: { commit: DOCS_COMMIT, chunks: docsIndex?.size ?? 0, sections: docSections(docsDocs) },
});
});
@@ -19,7 +19,7 @@
import { Router } from 'express';
import crypto from 'node:crypto';
import { oauthRateLimit, oauthTokenRateLimit } from './access-guard.js';
import { getPublicBaseUrl } from './config.js';
import { ENTRA_LOGIN_BASE_URL, getPublicBaseUrl } from './config.js';
const CLIENT_ID: string = (() => {
const v = process.env.AUTOPILOT_ENTRA_CLIENT_ID;
@@ -32,7 +32,7 @@ const CLIENT_ID: string = (() => {
return v;
})();
const CLIENT_SECRET = process.env.AUTOPILOT_ENTRA_CLIENT_SECRET ?? '';
const AUTHORITY = process.env.AUTOPILOT_ENTRA_AUTHORITY ?? 'https://login.microsoftonline.com/organizations';
const AUTHORITY = process.env.AUTOPILOT_ENTRA_AUTHORITY ?? `${ENTRA_LOGIN_BASE_URL}/organizations`;
const SCOPES = `api://${CLIENT_ID}/access_as_user openid profile offline_access`;
// Dynamic client registration (RFC 7591) is STATELESS: instead of a
@@ -115,6 +115,7 @@ export const EVENT_TYPES_CATALOG = {
'realmjoin_phase_changed',
'realmjoin_resolved',
'realmjoin_timeout',
'realmjoin_first_deployment_incomplete',
'realmjoin_package_started',
'realmjoin_package_completed',
],
@@ -6,6 +6,7 @@ import type { SearchProvider, DocsSearchBundle } from '../search-provider.js';
import { READ_ONLY, MAX_RESULT_SIZE_CHARS, toolResultText, SessionIdSchema, isBenignHealthDetectionReport, tenantIdDescription } from './shared.js';
import { toolError } from './error-handler.js';
import { ALL_EVENT_TYPES } from '../resource-catalog.js';
import { DOCS_BASE_URL } from '../config.js';
// ── Helpers ─────────────────────────────────────────────────────────────
@@ -1230,7 +1231,7 @@ export function registerSearchTools(
title: 'Search Product Documentation',
description:
'PRODUCT DOCUMENTATION SEARCH — the published Autopilot Monitor customer documentation ' +
'(docs.autopilotmonitor.com), chunked by section and ranked semantically. ' +
`(${DOCS_BASE_URL}), chunked by section and ranked semantically. ` +
'Use this for questions about how the PRODUCT works rather than what a specific enrollment did: ' +
'setup and onboarding, agent deployment, portal features, roles and permissions, settings, ' +
'notifications, network endpoints, security/privacy/data-residency, plans, and troubleshooting guidance. ' +
@@ -9,11 +9,12 @@ using SharedConstants = AutopilotMonitor.Shared.Constants;
namespace AutopilotMonitor.DecisionCore.Engine
{
// RealmJoin (RJ) deployment-tracking handlers. Plan: tasks/zany-gathering-oasis plan.
// The agent posts five new DecisionSignalKinds when it observes the
// The agent posts six DecisionSignalKinds when it observes the
// HKLM\SYSTEM\CurrentControlSet\Services\realmjoin\Parameters key + HKLM\SOFTWARE\RealmJoin\Packages
// (and the HKU\<sid>\... user-scope counterpart). Detection arms a 60-min hard timeout
// and gates the enrollment-completion AND-gate. Resolution (phase 110) or timeout
// releases the gate so the session can complete.
// and gates the enrollment-completion AND-gate. Resolution (phase 110), the
// aborted-first-deployment release (phase left 100/101 for 200/210 without 110 —
// session 224b2087) or the timeout releases the gate so the session can complete.
public sealed partial class DecisionEngine
{
// Hard 60-minute deadline from RJ-detected. Not configurable (per design choice —
@@ -25,6 +26,10 @@ namespace AutopilotMonitor.DecisionCore.Engine
public static class RealmJoinPayloadKeys
{
public const string DeploymentPhase = "deploymentPhase";
// Set on RealmJoinPhaseChanged: the phase the watcher observed immediately before
// the transition. The reducer prefers the PERSISTED RealmJoinFacts.LastDeploymentPhase
// (restart-safe) and only falls back to this process-local value.
public const string PreviousPhase = "previousPhase";
public const string PackageId = "packageId";
public const string DisplayName = "displayName";
public const string Version = "version";
@@ -107,6 +112,25 @@ namespace AutopilotMonitor.DecisionCore.Engine
}
builder.RealmJoinFacts = updatedFacts;
// Replay path (agent restarted between the 101 and 200 observations): the watcher
// re-fires Detected with the CURRENT phase and no PhaseChanged ever carries the
// transition — the persisted LastDeploymentPhase is the only witness of the aborted
// first deployment. Evaluate the abort rule here too so the gate does not stay
// closed until the hard timeout.
if (alreadyDetected
&& phase.HasValue
&& IsFirstDeploymentAbortTransition(state.RealmJoinFacts, state.RealmJoinFacts.LastDeploymentPhase?.Value, phase.Value))
{
return ReleaseFirstDeploymentIncomplete(
state: state,
signal: signal,
builder: builder,
nextStep: nextStep,
previousPhase: state.RealmJoinFacts.LastDeploymentPhase!.Value,
currentPhase: phase.Value,
triggerBase: nameof(DecisionSignalKind.RealmJoinDetected));
}
var effects = Array.Empty<DecisionEffect>();
if (!alreadyDetected)
{
@@ -177,7 +201,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
preparedBuilder: builder,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.RealmJoinResolved),
leadingEffect: cancelEffect);
leadingEffects: cancelEffect != null ? new[] { cancelEffect } : null);
}
/// <summary>
@@ -237,7 +261,156 @@ namespace AutopilotMonitor.DecisionCore.Engine
preparedBuilder: builder,
nextStepIndex: nextStep,
trigger: $"DeadlineFired:{DeadlineNames.RealmJoinTimeout}",
leadingEffect: timeoutEffect);
leadingEffects: new[] { timeoutEffect });
}
// RJ DeploymentPhase enum values relevant to the abort rule (mirrors
// RealmJoin.Core.SoftwarePackaging.DeploymentPhase; the agent-side RealmJoinInfo
// constants are not referencable from this netstandard shared lib).
private const int RjPhaseRunningFirstDeployment = 100;
private const int RjPhaseRunningFirstDeploymentAuto = 101;
private const int RjPhaseCompletedFirstDeployment = 110;
private const int RjPhaseRunningDeployment = 200;
private const int RjPhaseCompletedDeployment = 210;
private static bool IsRealmJoinFirstDeploymentPhase(int phase) =>
phase == RjPhaseRunningFirstDeployment || phase == RjPhaseRunningFirstDeploymentAuto;
/// <summary>
/// The aborted-RJ-ESP shape (session 224b2087): the phase left the first-deployment
/// window (100/101) for a regular deployment phase (200/210) — a transition that cannot
/// occur in a healthy first deployment, whose only exit is CompletedFirstDeployment
/// (110). Root cause observed in the RJ logs: an interactive logon lands seconds after
/// 101, RJ reclassifies the run as a secondary-user deployment
/// (isFirstMachineDeployment=False) and that branch writes 200/210, never 110.
/// <para>
/// Deliberately NOT triggered by 101 → 0 (RJ service restart may still complete the
/// first deployment) nor by a first observation of 210 (RJ already completed before
/// the agent booted — session 6f1959c0 starts 210 → 200; releasing there would be
/// wrong). <paramref name="previousPhase"/> comes from the persisted
/// <see cref="RealmJoinFacts.LastDeploymentPhase"/> where available so the rule
/// survives an agent restart between the 101 and 200 observations.
/// </para>
/// </summary>
private static bool IsFirstDeploymentAbortTransition(RealmJoinFacts facts, int? previousPhase, int currentPhase) =>
facts.DetectedUtc != null
&& facts.Outcome == null
&& previousPhase.HasValue
&& IsRealmJoinFirstDeploymentPhase(previousPhase.Value)
&& (currentPhase == RjPhaseRunningDeployment || currentPhase == RjPhaseCompletedDeployment);
/// <summary>
/// Phase-transition observation from the watcher. Always persists the current phase
/// into <see cref="RealmJoinFacts.LastDeploymentPhase"/> (restart-safe; also fixes the
/// <c>realmjoin_timeout</c> event reporting "last phase: 0" regardless of how far RJ
/// actually got). When <see cref="IsFirstDeploymentAbortTransition"/> matches, emits
/// the <c>realmjoin_first_deployment_incomplete</c> Warning, cancels the hard-timeout
/// deadline and releases the completion gate — for us RJ is finished at that point.
/// </summary>
private DecisionStep HandleRealmJoinPhaseChangedV1(DecisionState state, DecisionSignal signal)
{
var currentPhase = TryReadPhase(signal);
if (currentPhase == null)
{
var deadEnd = BumpStepBookkeeping(state, signal);
var deadEndTransition = BuildDeadEndTransition(
state: state,
signal: signal,
nextStepIndex: deadEnd.StepIndex,
trigger: nameof(DecisionSignalKind.RealmJoinPhaseChanged),
deadEndReason: "realmjoin_phase_changed_missing_phase");
return new DecisionStep(deadEnd, deadEndTransition, Array.Empty<DecisionEffect>());
}
// Persisted fact first (survives agent restarts), watcher's process-local value
// as fallback for the very first phase observation of a session.
var previousPhase = state.RealmJoinFacts.LastDeploymentPhase?.Value
?? TryReadInt(signal, RealmJoinPayloadKeys.PreviousPhase);
var nextStep = state.StepIndex + 1;
var builder = state.ToBuilder()
.WithStepIndex(nextStep)
.WithLastAppliedSignalOrdinal(signal.SessionSignalOrdinal);
builder.RealmJoinFacts = state.RealmJoinFacts.WithLastPhase(currentPhase.Value, signal.SessionSignalOrdinal);
if (!IsFirstDeploymentAbortTransition(state.RealmJoinFacts, previousPhase, currentPhase.Value))
{
var bookkept = builder.Build();
var transition = BuildTakenTransition(
before: state,
signal: signal,
toStage: state.Stage,
nextStepIndex: nextStep,
trigger: nameof(DecisionSignalKind.RealmJoinPhaseChanged));
return new DecisionStep(bookkept, transition, Array.Empty<DecisionEffect>());
}
return ReleaseFirstDeploymentIncomplete(
state: state,
signal: signal,
builder: builder,
nextStep: nextStep,
previousPhase: previousPhase!.Value,
currentPhase: currentPhase.Value,
triggerBase: nameof(DecisionSignalKind.RealmJoinPhaseChanged));
}
/// <summary>
/// Shared gate-release for the aborted-first-deployment rule — called from
/// <see cref="HandleRealmJoinPhaseChangedV1"/> and from the
/// <see cref="HandleRealmJoinDetectedV1"/> replay path (agent restarted between the
/// 101 and 200 observations: the watcher re-fires Detected with the CURRENT phase and
/// no PhaseChanged ever carries the transition, so the persisted fact is the only
/// witness). Mirrors <see cref="HandleRealmJoinResolvedV1"/>'s release shape with a
/// Warning timeline entry ahead of the completion effects.
/// </summary>
private DecisionStep ReleaseFirstDeploymentIncomplete(
DecisionState state,
DecisionSignal signal,
DecisionStateBuilder builder,
int nextStep,
int previousPhase,
int currentPhase,
string triggerBase)
{
builder.CancelDeadline(DeadlineNames.RealmJoinTimeout);
builder.RealmJoinFacts = builder.RealmJoinFacts.WithFirstDeploymentIncomplete(
signal.OccurredAtUtc, currentPhase, signal.SessionSignalOrdinal);
var leadingEffects = new List<DecisionEffect>(capacity: 2);
var cancelEffect = BuildRealmJoinTimeoutCancelEffectIfArmed(state);
if (cancelEffect != null) leadingEffects.Add(cancelEffect);
leadingEffects.Add(BuildRealmJoinFirstDeploymentIncompleteEvent(previousPhase, currentPhase));
return CompleteIfDeferredOrBookkeep(
state: state,
signal: signal,
preparedBuilder: builder,
nextStepIndex: nextStep,
trigger: triggerBase + ":FirstDeploymentIncomplete",
leadingEffects: leadingEffects);
}
/// <summary>
/// Build the <c>realmjoin_first_deployment_incomplete</c> Warning timeline entry.
/// Reducer-owned (like <see cref="BuildRealmJoinTimeoutEvent"/>) — the agent adapter
/// only dual-emits the neutral <c>realmjoin_phase_changed</c> observation; the abort
/// interpretation lives here where the persisted facts are.
/// </summary>
private static DecisionEffect BuildRealmJoinFirstDeploymentIncompleteEvent(int previousPhase, int currentPhase)
{
return new DecisionEffect(
kind: DecisionEffectKind.EmitEventTimelineEntry,
parameters: new Dictionary<string, string>(StringComparer.Ordinal)
{
[SignalPayloadKeys.EventType] = SharedConstants.EventTypes.RealmJoinFirstDeploymentIncomplete,
[SignalPayloadKeys.Source] = "DecisionEngine",
[SignalPayloadKeys.Severity] = "Warning",
[SignalPayloadKeys.Message] =
$"RealmJoin left first deployment (phase {previousPhase} -> {currentPhase}) without CompletedFirstDeployment (110) — the RealmJoin ESP likely aborted. Treating the RealmJoin deployment as finished.",
["previousPhase"] = previousPhase.ToString(CultureInfo.InvariantCulture),
["deploymentPhase"] = currentPhase.ToString(CultureInfo.InvariantCulture),
});
}
/// <summary>
@@ -360,7 +533,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
DecisionStateBuilder preparedBuilder,
int nextStepIndex,
string trigger,
DecisionEffect? leadingEffect)
IReadOnlyList<DecisionEffect>? leadingEffects)
{
var selfDeployingDeferred = state.RealmJoinFacts.SelfDeployingDeferredCompletion?.Value == true;
var classicReady = state.HelloResolvedUtc != null && state.DesktopArrivedUtc != null;
@@ -421,8 +594,8 @@ namespace AutopilotMonitor.DecisionCore.Engine
// Plan v9 Phase 4 — UI phase coverage: emit FinalizingSetup + Complete phase
// declarations BEFORE enrollment_complete so the Web timeline opens both bars
// for RJ-deferred-completion just like the direct SelfDeploying-terminal path.
var effects = new List<DecisionEffect>(capacity: 4);
if (leadingEffect != null) effects.Add(leadingEffect);
var effects = new List<DecisionEffect>(capacity: (leadingEffects?.Count ?? 0) + 3);
if (leadingEffects != null && leadingEffects.Count > 0) effects.AddRange(leadingEffects);
effects.Add(BuildPhaseTransitionEffect(EnrollmentPhase.FinalizingSetup, completedState, trigger + ":SelfDeployingDeferred"));
effects.Add(BuildPhaseTransitionEffect(EnrollmentPhase.Complete, completedState, trigger + ":SelfDeployingDeferred"));
effects.Add(BuildEnrollmentCompleteEffect(completedState, trigger + ":SelfDeployingDeferred"));
@@ -434,19 +607,19 @@ namespace AutopilotMonitor.DecisionCore.Engine
if (classicReady)
{
// This IS the gate-release path: the RealmJoin gate has just opened (the caller
// wrote WithResolved / WithTimeoutOutcome into preparedBuilder), so complete
// directly. Note the completion gates read `state` (pre-resolution), where RJ is
// still closed — routing this through CompleteThroughFinalizingOrDefer would
// re-defer on the very gate we just released. A future second gate that must
// re-block here would re-check the *post* state (WDP-v2 follow-up, ARCH-F1).
var extra = leadingEffect != null ? new[] { leadingEffect } : null;
// wrote WithResolved / WithTimeoutOutcome / WithFirstDeploymentIncomplete into
// preparedBuilder), so complete directly. Note the completion gates read `state`
// (pre-resolution), where RJ is still closed — routing this through
// CompleteThroughFinalizingOrDefer would re-defer on the very gate we just
// released. A future second gate that must re-block here would re-check the
// *post* state (WDP-v2 follow-up, ARCH-F1).
return TransitionToFinalizing(
state: state,
signal: signal,
preparedBuilder: preparedBuilder,
nextStepIndex: nextStepIndex,
trigger: trigger,
extraLeadingEffects: extra);
extraLeadingEffects: leadingEffects);
}
// Bookkeeping only — defer to the next Hello / Desktop / SelfDeploying signal.
@@ -458,10 +631,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
nextStepIndex: nextStepIndex,
trigger: trigger);
var bookkeepEffects = leadingEffect != null
? new[] { leadingEffect }
: Array.Empty<DecisionEffect>();
return new DecisionStep(newState, transition, bookkeepEffects);
return new DecisionStep(newState, transition, MaterializeEffects(leadingEffects));
}
/// <summary>
@@ -143,6 +143,7 @@ namespace AutopilotMonitor.DecisionCore.Engine
// ----- RealmJoin deployment tracking (DecisionEngine.RealmJoin.cs) -----
(DecisionSignalKind.RealmJoinDetected, 1) => HandleRealmJoinDetectedV1(state, signal),
(DecisionSignalKind.RealmJoinResolved, 1) => HandleRealmJoinResolvedV1(state, signal),
(DecisionSignalKind.RealmJoinPhaseChanged, 1) => HandleRealmJoinPhaseChangedV1(state, signal),
(DecisionSignalKind.RealmJoinPackageStarted, 1) => HandleRealmJoinPackageStartedV1(state, signal),
(DecisionSignalKind.RealmJoinPackageCompleted, 1) => HandleRealmJoinPackageCompletedV1(state, signal),
// Note: RealmJoinTimeout is a deadline-only kind (no direct adapter signal) —
@@ -118,5 +118,12 @@ namespace AutopilotMonitor.DecisionCore.Signals
RealmJoinTimeout,
RealmJoinPackageStarted,
RealmJoinPackageCompleted,
// Session 224b2087 (2026-07-22): DeploymentPhase transition observed by the watcher.
// Promoted from InformationalEvent-only so RealmJoinFacts.LastDeploymentPhase is
// persisted across agent restarts AND so the reducer can detect the aborted-RJ-ESP
// shape: an interactive logon during the first deployment (100/101) makes RJ reclassify
// the run as a secondary-user deployment, which writes 200/210 and NEVER 110 — the
// gate would otherwise stay closed until the 60-min hard timeout.
RealmJoinPhaseChanged,
}
}
@@ -14,8 +14,8 @@ namespace AutopilotMonitor.DecisionCore.State
/// <list type="bullet">
/// <item>Immutable; <see cref="With"/>-methods return new instances.</item>
/// <item><see cref="Packages"/> is capped at <see cref="MaxPackages"/>; overflow is discarded.</item>
/// <item><see cref="ResolvedUtc"/> is set once on observing phase 110; later phase readings update <see cref="LastDeploymentPhase"/> only.</item>
/// <item><see cref="Outcome"/> is one of <c>"Resolved"</c> / <c>"Timeout"</c>; null while RJ is still running.</item>
/// <item><see cref="ResolvedUtc"/> is set once on observing phase 110 (or the aborted-first-deployment release); later phase readings update <see cref="LastDeploymentPhase"/> only.</item>
/// <item><see cref="Outcome"/> is one of <c>"Resolved"</c> / <c>"Timeout"</c> / <c>"FirstDeploymentIncomplete"</c>; null while RJ is still running.</item>
/// </list>
/// </remarks>
public sealed class RealmJoinFacts
@@ -25,6 +25,11 @@ namespace AutopilotMonitor.DecisionCore.State
public const string OutcomeResolved = "Resolved";
public const string OutcomeTimeout = "Timeout";
// Session 224b2087 (2026-07-22): the RJ ESP was aborted (interactive logon during the
// first deployment reclassified the run as secondary-user) — DeploymentPhase left
// 100/101 for 200/210 without ever writing CompletedFirstDeployment (110). The gate
// releases with this distinct outcome so reporting can tell the three cases apart.
public const string OutcomeFirstDeploymentIncomplete = "FirstDeploymentIncomplete";
public static readonly RealmJoinFacts Empty = new RealmJoinFacts(
detectedUtc: null,
@@ -161,6 +166,27 @@ namespace AutopilotMonitor.DecisionCore.State
packages: Packages);
}
/// <summary>
/// Terminal outcome for the aborted-RJ-ESP case: DeploymentPhase left the
/// first-deployment window (100/101) for a regular deployment phase (200/210) without
/// ever reaching CompletedFirstDeployment (110). Sets <see cref="ResolvedUtc"/> (this
/// IS our resolution moment — RJ counts as finished for the completion gate) with
/// <see cref="OutcomeFirstDeploymentIncomplete"/>. No-op when an outcome is already set.
/// </summary>
public RealmJoinFacts WithFirstDeploymentIncomplete(DateTime utc, int phase, long sourceSignalOrdinal)
{
if (Outcome != null) return this; // resolved/timeout already wins
return new RealmJoinFacts(
detectedUtc: DetectedUtc,
resolvedUtc: new SignalFact<DateTime>(utc, sourceSignalOrdinal),
lastDeploymentPhase: new SignalFact<int>(phase, sourceSignalOrdinal),
outcome: new SignalFact<string>(OutcomeFirstDeploymentIncomplete, sourceSignalOrdinal),
selfDeployingDeferredCompletion: SelfDeployingDeferredCompletion,
productVersion: ProductVersion,
releaseChannel: ReleaseChannel,
packages: Packages);
}
public RealmJoinFacts WithLastPhase(int phase, long sourceSignalOrdinal)
{
if (LastDeploymentPhase != null && LastDeploymentPhase.Value == phase) return this;
@@ -82,6 +82,28 @@ namespace AutopilotMonitor.Shared
/// </summary>
public const string McpServerBaseUrl = "https://mcp.autopilotmonitor.com";
// -----------------------------------------------------------------------
// Well-known base URLs — single registry. Every own or Microsoft host used
// in code MUST reference these constants instead of repeating the literal;
// HardcodedUrlGuardTests enforces this. The EU cutover missed two copies of
// the blob host precisely because they did not go through this file.
// -----------------------------------------------------------------------
/// <summary>Customer portal (deep links in notifications and e-mails).</summary>
public const string PortalBaseUrl = "https://portal.autopilotmonitor.com";
/// <summary>Public marketing/product website.</summary>
public const string WebsiteBaseUrl = "https://www.autopilotmonitor.com";
/// <summary>Published customer documentation.</summary>
public const string DocsBaseUrl = "https://docs.autopilotmonitor.com";
/// <summary>Microsoft Graph API root (no version segment).</summary>
public const string GraphBaseUrl = "https://graph.microsoft.com";
/// <summary>Entra ID (Azure AD) login/token authority host (no trailing slash).</summary>
public const string EntraLoginBaseUrl = "https://login.microsoftonline.com";
// -----------------------------------------------------------------------
// Agent self-update
// -----------------------------------------------------------------------
@@ -441,6 +463,11 @@ namespace AutopilotMonitor.Shared
public const string RealmJoinPhaseChanged = "realmjoin_phase_changed";
public const string RealmJoinResolved = "realmjoin_resolved";
public const string RealmJoinTimeout = "realmjoin_timeout";
// Warning: DeploymentPhase left the first-deployment window (100/101) for 200/210
// without CompletedFirstDeployment (110) — the RJ ESP was aborted (typically an
// interactive logon during the first deployment reclassified the run as
// secondary-user). The completion gate releases at this point.
public const string RealmJoinFirstDeploymentIncomplete = "realmjoin_first_deployment_incomplete";
public const string RealmJoinPackageStarted = "realmjoin_package_started";
public const string RealmJoinPackageCompleted = "realmjoin_package_completed";
@@ -672,9 +699,11 @@ namespace AutopilotMonitor.Shared
/// </list>
/// </summary>
public static (string FailureType, string Message) ClassifyEspAppsFailure(
string errorCode, int? espTimeoutMinutes)
string? errorCode, int? espTimeoutMinutes)
{
var normalised = string.IsNullOrEmpty(errorCode) ? null : errorCode.ToLowerInvariant();
// netstandard2.0's IsNullOrEmpty lacks [NotNullWhen(false)], so the compiler can't
// see that the else branch is non-null (CS8602) — assert it.
var normalised = string.IsNullOrEmpty(errorCode) ? null : errorCode!.ToLowerInvariant();
if (string.Equals(normalised, DetectionFailureHResult, System.StringComparison.Ordinal))
{
@@ -82,7 +82,6 @@ namespace AutopilotMonitor.Shared.DataAccess
Task<List<SessionSummary>> GetGeoWindowSessionsAsync(DateTime startDate, DateTime endDate, string? tenantId = null);
Task<List<SessionSummary>> GetStalledSessionsAsync(string tenantId, DateTime cutoffTime);
Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff);
Task<List<SessionSummary>> GetExcessiveDataSendersAsync(string tenantId, DateTime windowCutoff, int maxSessionWindowHours);
// --- Legacy Reclassification (misclassification audit 2026-07-16) ---
/// <summary>
@@ -111,27 +111,15 @@ namespace AutopilotMonitor.Shared.Models
public int ExcessiveEventAutoActionThreshold { get; set; } = 2500;
/// <summary>
/// Duration in hours for the device block created by the auto-action path. Independent
/// of <see cref="MaintenanceBlockDurationHours"/> (which governs the time-window
/// ExcessiveDataBlocked path) so the operator can tune both autonomously. Default: 24.
/// Duration in hours for the device block created by the auto-action path. Default: 24.
/// </summary>
public int ExcessiveEventAutoActionDurationHours { get; set; } = 24;
// ===== MAINTENANCE AUTO-BLOCK SETTINGS =====
/// <summary>
/// Max active data window in hours for maintenance auto-block detection.
/// Sessions with LastEventAt within the last MaxSessionWindowHours AND StartedAt older
/// than MaxSessionWindowHours will have their device blocked by the nightly maintenance function.
/// 0 = disabled. Default: 24.
/// </summary>
public int MaxSessionWindowHours { get; set; } = 24;
/// <summary>
/// Duration in hours for maintenance-triggered device blocks (excessive data senders).
/// Default: 12.
/// </summary>
public int MaintenanceBlockDurationHours { get; set; } = 12;
// MaxSessionWindowHours / MaintenanceBlockDurationHours were removed 2026-07-22 with the
// time-window auto-block. That detector blocked on session span alone, so every enrollment
// that spanned a night qualified — including 23-event sessions. The event-count settings
// above are now the only automatic block. Stored rows keep the two columns until their
// next write; nothing reads them.
/// <summary>
/// Retention period in days for operational events in the OpsEvents table.
@@ -531,9 +519,7 @@ namespace AutopilotMonitor.Shared.Models
GlobalAdminRateLimitRequestsPerMinute = 600,
PlatformStatsBlobSasUrl = string.Empty,
CollectorIdleTimeoutMinutes = 15,
DesktopDetectorNoCandidateTimeoutMinutes = 10,
MaxSessionWindowHours = 24,
MaintenanceBlockDurationHours = 12
DesktopDetectorNoCandidateTimeoutMinutes = 10
};
}
}
@@ -27,9 +27,11 @@ namespace AutopilotMonitor.Shared.Services
/// Validates <paramref name="candidate"/> as a migration target and returns the
/// normalized base URL (<c>https://{lowercase-host}</c>, no trailing slash) on success.
/// Rules: absolute https URI, default port, no userinfo/path/query/fragment, host
/// matching <see cref="AllowedHostSuffixes"/>.
/// matching <see cref="AllowedHostSuffixes"/>. <paramref name="normalized"/> is non-null
/// exactly when this returns <c>true</c> (netstandard2.0 has no usable
/// <c>[NotNullWhen]</c>, so callers must check the bool themselves).
/// </summary>
public static bool TryNormalizeTarget(string candidate, out string normalized)
public static bool TryNormalizeTarget(string candidate, out string? normalized)
{
normalized = null;
@@ -77,7 +79,8 @@ namespace AutopilotMonitor.Shared.Services
/// differs from <paramref name="currentBaseUrl"/> (both compared in normalized form —
/// serving the agent its own current URL is a no-op, not a migration).
/// </summary>
public static bool IsEffectiveMigration(string candidate, string currentBaseUrl, out string normalizedTarget)
public static bool IsEffectiveMigration(
string candidate, string currentBaseUrl, out string? normalizedTarget)
{
normalizedTarget = null;
@@ -1,4 +1,5 @@
import type { Metadata } from "next";
import { SITE_URL } from "@/utils/config";
export const metadata: Metadata = {
title: "About – Real-Time Windows Autopilot Monitoring",
@@ -28,7 +29,7 @@ export const metadata: Metadata = {
title: "About AutopilotMonitor – Real-Time Windows Autopilot Monitoring",
description:
"Free, open-source real-time monitoring and troubleshooting for Windows Autopilot enrollments. Track every phase, run analyze rules, and resolve issues faster — built by Oliver Kieselbach.",
url: "https://www.autopilotmonitor.com/about",
url: `${SITE_URL}/about`,
},
twitter: {
title: "About AutopilotMonitor – Real-Time Windows Autopilot Monitoring",
@@ -36,7 +37,7 @@ export const metadata: Metadata = {
"Free, open-source real-time monitoring and troubleshooting for Windows Autopilot enrollments. Track every phase, run analyze rules, and resolve issues faster.",
},
alternates: {
canonical: "https://www.autopilotmonitor.com/about",
canonical: `${SITE_URL}/about`,
},
};
@@ -1,5 +1,6 @@
import Link from "next/link";
import { PublicPageHeader } from "../../components/PublicPageHeader";
import { DOCS_URL } from "@/utils/config";
const FEATURES = [
{
@@ -295,10 +296,10 @@ export default function AboutPage() {
<section className="bg-white rounded-xl shadow-sm border border-gray-100 p-8">
<h2 className="text-2xl font-bold text-gray-900 mb-4">Explore Further</h2>
<div className="grid grid-cols-1 sm:grid-cols-2 gap-3 text-sm">
<a href="https://docs.autopilotmonitor.com" target="_blank" rel="noopener noreferrer" className="rounded-lg border border-gray-200 px-4 py-3 hover:border-blue-300 hover:bg-blue-50/50 transition-colors text-gray-700 font-medium">
<a href={DOCS_URL} target="_blank" rel="noopener noreferrer" className="rounded-lg border border-gray-200 px-4 py-3 hover:border-blue-300 hover:bg-blue-50/50 transition-colors text-gray-700 font-medium">
Documentation →
</a>
<a href="https://docs.autopilotmonitor.com/changelog/platform-changelog" target="_blank" rel="noopener noreferrer" className="rounded-lg border border-gray-200 px-4 py-3 hover:border-blue-300 hover:bg-blue-50/50 transition-colors text-gray-700 font-medium">
<a href={`${DOCS_URL}/changelog/platform-changelog`} target="_blank" rel="noopener noreferrer" className="rounded-lg border border-gray-200 px-4 py-3 hover:border-blue-300 hover:bg-blue-50/50 transition-colors text-gray-700 font-medium">
Changelog →
</a>
<Link href="/privacy" className="rounded-lg border border-gray-200 px-4 py-3 hover:border-blue-300 hover:bg-blue-50/50 transition-colors text-gray-700 font-medium">
@@ -34,10 +34,6 @@ interface AdminConfigContextValue {
setCollectorIdleTimeoutMinutes: (value: number) => void;
desktopDetectorNoCandidateTimeoutMinutes: number;
setDesktopDetectorNoCandidateTimeoutMinutes: (value: number) => void;
maxSessionWindowHours: number;
setMaxSessionWindowHours: (value: number) => void;
maintenanceBlockDurationHours: number;
setMaintenanceBlockDurationHours: (value: number) => void;
opsEventRetentionDays: number;
setOpsEventRetentionDays: (value: number) => void;
slaNotificationCooldownHours: number;
@@ -138,8 +134,6 @@ export function AdminConfigProvider({ children }: { children: React.ReactNode })
const [agentMigrateTenantOverridesJson, setAgentMigrateTenantOverridesJson] = useState("");
const [collectorIdleTimeoutMinutes, setCollectorIdleTimeoutMinutes] = useState(15);
const [desktopDetectorNoCandidateTimeoutMinutes, setDesktopDetectorNoCandidateTimeoutMinutes] = useState(10);
const [maxSessionWindowHours, setMaxSessionWindowHours] = useState(24);
const [maintenanceBlockDurationHours, setMaintenanceBlockDurationHours] = useState(12);
const [opsEventRetentionDays, setOpsEventRetentionDays] = useState(90);
const [slaNotificationCooldownHours, setSlaNotificationCooldownHours] = useState(24);
const [allowAgentDowngrade, setAllowAgentDowngrade] = useState(false);
@@ -201,8 +195,6 @@ export function AdminConfigProvider({ children }: { children: React.ReactNode })
setAgentMigrateTenantOverridesJson(data.agentMigrateTenantOverridesJson ?? "");
setCollectorIdleTimeoutMinutes(data.collectorIdleTimeoutMinutes ?? 15);
setDesktopDetectorNoCandidateTimeoutMinutes(data.desktopDetectorNoCandidateTimeoutMinutes ?? 10);
setMaxSessionWindowHours(data.maxSessionWindowHours ?? 24);
setMaintenanceBlockDurationHours(data.maintenanceBlockDurationHours ?? 12);
setOpsEventRetentionDays(data.opsEventRetentionDays ?? 90);
setSlaNotificationCooldownHours(data.slaNotificationCooldownHours ?? 24);
setAllowAgentDowngrade(data.allowAgentDowngrade ?? false);
@@ -308,8 +300,6 @@ export function AdminConfigProvider({ children }: { children: React.ReactNode })
agentMigrateTenantOverridesJson: agentMigrateTenantOverridesJson.trim(),
collectorIdleTimeoutMinutes,
desktopDetectorNoCandidateTimeoutMinutes,
maxSessionWindowHours,
maintenanceBlockDurationHours,
opsEventRetentionDays,
slaNotificationCooldownHours,
allowAgentDowngrade,
@@ -342,7 +332,7 @@ export function AdminConfigProvider({ children }: { children: React.ReactNode })
} finally {
setSavingConfig(false);
}
}, [isGlobalAdmin, adminConfig, globalRateLimit, userRateLimit, globalAdminRateLimit, platformStatsBlobSasUrl, agentMigrateApiBaseUrl, agentMigrateTenantOverridesJson, collectorIdleTimeoutMinutes, desktopDetectorNoCandidateTimeoutMinutes, maxSessionWindowHours, maintenanceBlockDurationHours, opsEventRetentionDays, slaNotificationCooldownHours, allowAgentDowngrade, modernDeploymentHarmlessEventIds, enableIndexDualWrite, sessionDeletionKillSwitch, getAccessToken]);
}, [isGlobalAdmin, adminConfig, globalRateLimit, userRateLimit, globalAdminRateLimit, platformStatsBlobSasUrl, agentMigrateApiBaseUrl, agentMigrateTenantOverridesJson, collectorIdleTimeoutMinutes, desktopDetectorNoCandidateTimeoutMinutes, opsEventRetentionDays, slaNotificationCooldownHours, allowAgentDowngrade, modernDeploymentHarmlessEventIds, enableIndexDualWrite, sessionDeletionKillSwitch, getAccessToken]);
// Reset admin config
const handleResetAdminConfig = useCallback(() => {
@@ -355,8 +345,6 @@ export function AdminConfigProvider({ children }: { children: React.ReactNode })
setAgentMigrateTenantOverridesJson(adminConfig.agentMigrateTenantOverridesJson ?? "");
setCollectorIdleTimeoutMinutes(adminConfig.collectorIdleTimeoutMinutes ?? 15);
setDesktopDetectorNoCandidateTimeoutMinutes(adminConfig.desktopDetectorNoCandidateTimeoutMinutes ?? 10);
setMaxSessionWindowHours(adminConfig.maxSessionWindowHours ?? 24);
setMaintenanceBlockDurationHours(adminConfig.maintenanceBlockDurationHours ?? 12);
setOpsEventRetentionDays(adminConfig.opsEventRetentionDays ?? 90);
setSlaNotificationCooldownHours(adminConfig.slaNotificationCooldownHours ?? 24);
setAllowAgentDowngrade(adminConfig.allowAgentDowngrade ?? false);
@@ -501,8 +489,6 @@ export function AdminConfigProvider({ children }: { children: React.ReactNode })
agentMigrateTenantOverridesJson, setAgentMigrateTenantOverridesJson,
collectorIdleTimeoutMinutes, setCollectorIdleTimeoutMinutes,
desktopDetectorNoCandidateTimeoutMinutes, setDesktopDetectorNoCandidateTimeoutMinutes,
maxSessionWindowHours, setMaxSessionWindowHours,
maintenanceBlockDurationHours, setMaintenanceBlockDurationHours,
opsEventRetentionDays, setOpsEventRetentionDays,
slaNotificationCooldownHours, setSlaNotificationCooldownHours,
allowAgentDowngrade, setAllowAgentDowngrade,
@@ -1,6 +1,7 @@
"use client";
import { useCallback, useEffect, useRef, useState } from "react";
import Link from "next/link";
import { useSearchParams } from "next/navigation";
import { api } from "@/lib/api";
import { authenticatedFetch, TokenExpiredError } from "@/lib/authenticatedFetch";
@@ -8,6 +9,7 @@ import { trackEvent } from "@/lib/appInsights";
import { TenantConfiguration } from "./TenantManagementSection";
import { TenantSearchSelect } from "./TenantSearchSelect";
import { useCanMutatePlatform } from "@/hooks/useCanMutatePlatform";
import { firstBlockedSessionId, blockedSessionCount } from "./blockedDeviceHelpers";
interface ResolvedDevice {
sessionId: string;
@@ -27,17 +29,16 @@ interface BlockedDevice {
durationHours: number;
reason?: string;
action?: string;
/**
* Comma-separated sessions this block is scoped to — set by the maintenance auto-block
* paths, absent on a manual whole-device block. Drives the serial's deep-link: without a
* session there is nothing honest to link to.
*/
blockedSessionIds?: string | null;
}
interface DeviceBlockSectionProps {
tenants: TenantConfiguration[];
maxSessionWindowHours: number;
setMaxSessionWindowHours: (value: number) => void;
maintenanceBlockDurationHours: number;
setMaintenanceBlockDurationHours: (value: number) => void;
savingConfig: boolean;
adminConfigExists: boolean;
onSaveAdminConfig: () => Promise<void>;
getAccessToken: () => Promise<string | null>;
setError: (error: string | null) => void;
setSuccessMessage: (message: string | null) => void;
@@ -45,13 +46,6 @@ interface DeviceBlockSectionProps {
export function DeviceBlockSection({
tenants,
maxSessionWindowHours,
setMaxSessionWindowHours,
maintenanceBlockDurationHours,
setMaintenanceBlockDurationHours,
savingConfig,
adminConfigExists,
onSaveAdminConfig,
getAccessToken,
setError,
setSuccessMessage,
@@ -312,60 +306,6 @@ export function DeviceBlockSection({
</div>
</div>
<div className="p-6 space-y-6">
{/* Maintenance Auto-Block Settings */}
<div className="border border-red-200 dark:border-red-700 rounded-lg p-4 bg-red-50/50 dark:bg-red-900/10">
<div className="flex items-center space-x-2 mb-3">
<svg className="w-4 h-4 text-red-500 dark:text-red-400 flex-shrink-0" fill="none" stroke="currentColor" viewBox="0 0 24 24">
<path strokeLinecap="round" strokeLinejoin="round" strokeWidth={2} d="M12 8v4l3 3m6-3a9 9 0 11-18 0 9 9 0 0118 0z" />
</svg>
<h3 className="text-sm font-semibold text-red-900 dark:text-red-100">Maintenance Auto-Block Settings</h3>
</div>
<p className="text-xs text-red-600 dark:text-red-400 mb-4">
The nightly maintenance function automatically blocks devices that are still actively sending data beyond the configured window. Sessions with a <code className="bg-red-100 dark:bg-red-900/40 px-1 rounded">LastEventAt</code> timestamp within the window <em>and</em> a <code className="bg-red-100 dark:bg-red-900/40 px-1 rounded">StartedAt</code> older than the window are flagged – regardless of session status.
</p>
<div className="grid grid-cols-1 sm:grid-cols-2 gap-4 mb-4">
<div>
<label className="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-1">
Max Session Window (Hours)
</label>
<input
type="number"
min={0}
max={168}
value={maxSessionWindowHours}
onChange={(e) => setMaxSessionWindowHours(parseInt(e.target.value) || 0)}
className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-lg text-sm text-gray-900 dark:text-gray-100 bg-white dark:bg-gray-700 focus:outline-none focus:ring-2 focus:ring-red-500 focus:border-red-500"
/>
<p className="text-xs text-gray-500 dark:text-gray-400 mt-1">0 = disabled</p>
</div>
<div>
<label className="block text-sm font-medium text-gray-700 dark:text-gray-300 mb-1">
Maintenance Block Duration (Hours)
</label>
<input
type="number"
min={1}
max={720}
value={maintenanceBlockDurationHours}
onChange={(e) => setMaintenanceBlockDurationHours(parseInt(e.target.value) || 12)}
className="w-full px-3 py-2 border border-gray-300 dark:border-gray-600 rounded-lg text-sm text-gray-900 dark:text-gray-100 bg-white dark:bg-gray-700 focus:outline-none focus:ring-2 focus:ring-red-500 focus:border-red-500"
/>
<p className="text-xs text-gray-500 dark:text-gray-400 mt-1">Applied by the maintenance function</p>
</div>
</div>
<button
onClick={onSaveAdminConfig}
disabled={!canMutate || savingConfig || !adminConfigExists}
className="px-4 py-2 bg-red-600 text-white rounded-lg text-sm font-medium hover:bg-red-700 disabled:opacity-50 flex items-center space-x-2"
>
{savingConfig ? (
<><div className="animate-spin rounded-full h-4 w-4 border-b-2 border-white"></div><span>Saving...</span></>
) : (
<span>Save Maintenance Settings</span>
)}
</button>
</div>
{/* Block / Kill a device form */}
<div>
<h3 className="text-sm font-semibold text-red-900 dark:text-red-100 mb-3">Block or Kill a Device</h3>
@@ -563,7 +503,29 @@ export function DeviceBlockSection({
<tbody className="divide-y divide-red-100 dark:divide-red-900/30">
{blockedDevices.map((d) => (
<tr key={`${d.tenantId}:${d.serialNumber}`} className="bg-white dark:bg-gray-800">
<td className="px-3 py-2 font-mono font-medium text-gray-900 dark:text-gray-100">{d.serialNumber}</td>
<td className="px-3 py-2 font-mono font-medium text-gray-900 dark:text-gray-100">
{(() => {
// Maintenance auto-blocks carry the session they were placed for —
// that session is what the operator needs to look at. Manual blocks
// have none and stay plain text.
const sessionId = firstBlockedSessionId(d.blockedSessionIds);
if (!sessionId) return d.serialNumber;
const count = blockedSessionCount(d.blockedSessionIds);
return (
<Link
href={`/sessions/${encodeURIComponent(sessionId)}`}
target="_blank"
rel="noopener noreferrer"
className="text-blue-700 dark:text-blue-300 hover:underline"
title={count > 1
? `Open the first of ${count} blocked sessions (${sessionId})`
: `Open blocked session ${sessionId}`}
>
{d.serialNumber}
</Link>
);
})()}
</td>
{!blockListTenantId && (
<td className="px-3 py-2">
<button
@@ -54,7 +54,7 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
],
Security: [
"DeviceBlocked",
"ExcessiveDataBlocked",
// ExcessiveDataBlocked removed 2026-07-22 with the time-window auto-block that raised it.
// Critical-tier auto-action: emitted when maintenance auto-blocks/kills a device
// after its session crosses `excessiveEventAutoActionThreshold`. Dual-register
// per memory feedback_ops_event_types_dual_register so operators can wire a
@@ -95,7 +95,17 @@ const OPS_EVENT_TYPES: Record<string, string[]> = {
"TenantTrialExpiring",
"TenantTrialExpired",
],
Agent: ["BlobStorageMissing", "BlobStorageUnreachable", "NewImeVersionDetected", "ExcessiveSessionEvents"],
Agent: [
"BlobStorageMissing",
"BlobStorageUnreachable",
"NewImeVersionDetected",
"ExcessiveSessionEvents",
// 48h session-age emergency break reported over the agent's emergency channel — the
// "are we silently losing agents?" signal. Emitted once per session by
// ReportAgentErrorFunction (RecordAgentEmergencyBreakAsync). Dual-register per memory
// feedback_ops_event_types_dual_register.
"AgentEmergencyBreak",
],
};
const SEVERITIES = ["Info", "Warning", "Error", "Critical"];
@@ -0,0 +1,24 @@
// Helpers for the Active Blocks table's serial-number deep-link. Lives in a `.ts`
// file so vitest (which only loads `.test.ts` and doesn't transform JSX) can import
// and test them — same split as `opsEventSessionHelpers.ts`.
/**
* First session ID of a session-scoped block, or null when the block is not tied to a
* session. Maintenance auto-blocks (time-window and event-count paths) always carry the
* session they were placed for; a manual whole-device block carries none, and inventing a
* target for it would send the operator to a session that never triggered anything.
*/
export function firstBlockedSessionId(blockedSessionIds?: string | null): string | null {
if (!blockedSessionIds) return null;
for (const part of blockedSessionIds.split(",")) {
const trimmed = part.trim();
if (trimmed.length > 0) return trimmed;
}
return null;
}
/** Number of sessions a block is scoped to — 0 for a whole-device block. */
export function blockedSessionCount(blockedSessionIds?: string | null): number {
if (!blockedSessionIds) return 0;
return blockedSessionIds.split(",").filter((part) => part.trim().length > 0).length;
}

Some files were not shown because too many files have changed in this diff Show More