Compare commits

...

34 Commits

Author SHA1 Message Date
Oliver Kieselbach a0d10b8471 test(cleanup): end-to-end incident replays for the self-destruct script
Six integration tests execute the generated cleanup script with a real
powershell.exe against GUID-unique scratch trees — proving what the string
pins cannot:

- locked tree ⇒ complete abort, marker + binaries survive (the 2026-07-22
  regression: the old script force-deleted around the locks)
- full incident replay: a renamed second process holds a file in the tree,
  is found by NAME, force-killed after the wait deadline, cleanup then
  completes with nothing left behind
- fresh foreign lock ⇒ loser backs off without touching the tree; stale
  lock (>10min) is taken over and cleanup proceeds
- happy path + keepLogs variant

BuildCleanupScript gains test-only timing parameters (defaults unchanged
for production) so the replays run in seconds. Agent suite 1969/1969.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 10:06:29 +02:00
Oliver Kieselbach ad3632e9fb fix(emergency-break): survivable self-destruct, reliable break report, ops visibility
Incident 2026-07-22: the 48h emergency break left exe + loaded DLLs orphaned
forever — a second agent instance held the images while the cleanup script
(waiting on the launcher PID only) force-deleted marker + state around the
locks AFTER removing the Scheduled Task, destroying the retry path.

Agent — cleanup script (LIFE-F7):
- wait for ALL agent processes by name (PID-reuse-proof), then force-kill
  stragglers; single-flight lock file serializes concurrent cleanup scripts
- probe-rename (GUID target) BEFORE task removal; any failure renames back
  and aborts with task + marker + files intact for a next-boot retry; the
  destructive Remove-Item-on-original fallback is gone
- script generation extracted to BuildCleanupScript, pinned by 13 tests;
  all 4 variants parser-validated + E2E-smoked (happy path, locked tree)

Agent — break report delivery:
- EmergencyReporter counts DELIVERED reports: reservation rolls back on
  failure (mark-before-send permanently suppressed a key after its first
  transport failure); optional in-reservation retries
- break path: NIC wait <=15s, 3 attempts x 15s timeout (survives Functions
  cold starts observed as >5s 503s), 75s overall bound

Backend:
- backend-materialized events (agent_emergency_break, session_timeout) now
  upsert the cross-session EventType index — they were invisible to every
  search-by-eventType surface (portal + MCP)
- new AgentEmergencyBreak ops event (Agent/Warning, once per session) so
  operators see silently-lost agents; dual-registered in OPS_EVENT_TYPES
- materialization extracted to a static seam, covered by 4 new tests

Agent 1963/1963, Backend 3198/3198, tsc clean.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:56:46 +02:00
Oliver Kieselbach 05e1569503 feat(agent): wildcard support for Local Admin Analyzer allowed accounts
Allowed Local Accounts entries may now contain glob wildcards (* / ?),
matching the hardware-whitelist semantics: Regex.Escape conversion,
case-insensitive, 100ms ReDoS timeout. Applies to both the account and
the profile-folder check. Entries without wildcard chars keep exact
matching; * and ? are invalid in Windows account names, so patterns and
literals cannot collide. UI documents the syntax and rejects patterns
with fewer than 2 literal characters (a bare * would disable the
analyzer). No schema change - old agents treat patterns as literals and
keep alerting (fail-safe).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:47:13 +02:00
Oliver Kieselbach ba3581e253 bootstrapper added download stopwatch 2026-07-23 09:35:30 +02:00
Oliver Kieselbach 829ec4c969 fix(release): stamp Cache-Control no-cache on every blob upload
The first release after the download-alias migration exposed Front Door
caching as a correctness hazard: the alias kept serving the previous
version.json (X-Cache: TCP_REMOTE_HIT) after the origin had rotated to
2.0.1370, pinning the portal's version badge to the old build. The dangerous
variant is worse than a stale badge -- manifest and ZIP are independent cache
entries, so a stale manifest paired with a fresh ZIP fails the bootstrap's
SHA-256 integrity check and the install dies.

Route caching on /agent/* has been disabled (infra side); this commit is the
defense-in-depth half: every upload now carries x-ms-blob-cache-control
no-cache -- the CI Send-Blob (eu and legacy mirror alike), the local build.ps1
counterpart (gitignored, changed in lockstep), and the versions/*.json
manifest publishes. Uniform no-cache rather than an immutable/mutable split
because every blob in the agent container rotates in place, including the
per-line AutopilotMonitor-Agent-v2.zip -- nothing there is safely cacheable.
Existing blobs get the header with the next release's re-upload.

Log/docs updated in url-registry.md; no code paths changed, YAML and ps1
parse-checked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:17:02 +02:00
Oliver Kieselbach b2d2d37c6f feat(realmjoin): detect aborted RJ ESP (missing 110) and release the completion gate
An interactive logon during the RealmJoin first deployment (phase 100/101)
makes RJ reclassify the run as a secondary-user deployment, whose branch
writes 200/210 and never CompletedFirstDeployment (110) — no crash, no
reboot request. The V2 completion gate waited for exactly 110, so these
sessions hung until the 60-min hard timeout (session 224b2087; siblings
558bf067/3cc18724 delayed ~50 min the same way).

- promote realmjoin_phase_changed to a typed RealmJoinPhaseChanged signal
  (dual-emitted by the adapter) so RealmJoinFacts.LastDeploymentPhase is
  persisted across agent restarts
- abort rule: prev in {100,101} and current in {200,210} with no outcome
  yet -> emit realmjoin_first_deployment_incomplete (Warning), cancel the
  hard-timeout deadline, release the gate with outcome
  FirstDeploymentIncomplete; also evaluated on the RealmJoinDetected
  replay path (restart swallows the transition)
- deliberately NOT triggered by 101->0 (RJ service restart may still
  complete the first deployment) nor by a first observation of 210
  (session 6f1959c0: RJ already completed before agent boot)
- fix realmjoin_timeout always claiming "last phase: 0" - the persisted
  phase now feeds the message
- backend EnrollmentTimeoutClassifier counts the new event as a gate
  terminal; MCP resource catalog dual-synced
- CompleteIfDeferredOrBookkeep widened to a leading-effect list

Verified by replaying session 224b2087's signal log: gate releases at
15:05:17 (101->200) instead of hanging until the 15:54 deadline.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 09:14:30 +02:00
Oliver Kieselbach e0ad51b965 refactor(urls): one guard-enforced URL registry per component, agent downloads move to the alias
Every well-known host -- own subdomains, both blob accounts, the API origin,
Graph, Entra login -- now lives in exactly one registry per component:
Constants.cs (*BaseUrl) for C#, utils/config.ts for the web app, src/config.ts
for the MCP server. lib/hostRouting.ts derives its hostnames from the registry
and middleware.ts imports from it, retiring what turned out to be a fourth
independent copy of the www/portal host pair.

Three new guard tests (backend xUnit, web vitest, MCP vitest) scan non-test
source and fail on any enforced host outside its registry. This is the lesson
of the EU cutover: the registry existed then too, but LatestVersionsService
(feeds the portal's version badge) and ValidateBootstrapCodeFunction carried
their own literal copies of the legacy blob host and were missed -- kept
working only by the fail-soft legacy mirror in build-agent.yml, whose silent
failure would have pinned badge and self-update to stale bits with a green
release. Both guards proved themselves on their first run, flagging the
middleware host copy and a literal in an MCP tool description. Test files stay
exempt on purpose: literals there are independent oracles, and
ValidateBootstrapCodeResponseShapeTests now pins the full alias URL against
the Constants-built one so registry changes must be acknowledged consciously.

The two legacy readers plus the agent SelfUpdater now use
Constants.AgentDownloadBaseUrl (download.autopilotmonitor.com/agent); the
alias was verified live for version.json, version-v2.json and both ZIPs
before the switch. The portal's bootstrap allow-list accepts the alias AND
the legacy host (AGENT_DOWNLOAD_HOSTNAMES) for already-deployed customer
scripts, which imposes a deploy order: web before or with backend. The
legacy keepalive probes and the release mirror stay untouched -- they exist
for exactly this transition. Also: the bootstrap short-link now targets www
directly instead of riding the registrar's apex 301, and the welcome-mail
docs button points at docs. instead of bouncing through www./docs.

Suites: backend 3194, agent core 1925, DecisionCore 474, web 582, MCP 449 --
all green; next build verified (it, not tsc, proves the next.config.ts
registry import). Docs: docs/url-registry.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-23 08:37:42 +02:00
Oliver Kieselbach bbb55034e3 feat(versioning): one version contract for agent, backend and MCP
All three components now report <major>.<minor>.<build> plus a 7-char commit
and a build timestamp. The backend had been pinned at a hand-written 1.0.0
since inception and starts at 1.5.1000; the MCP server's package.json version
was bumped by hand and starts at 1.5.300, and it now reports its own commit on
/health instead of only the documentation bundle's.

The build number keeps coming from a counter blob reserved before the build
with ETag-CAS, but there is now one implementation instead of two copies:
.github/scripts/Request-BuildNumber.ps1, called by all three workflows and by
the local build.ps1. It lives under .github/ because scripts/Deployment/ is
gitignored (those scripts carry SAS tokens) and would therefore be absent from
a CI checkout; scripts/Build/ and scripts/Release/ are shadowed by the
[Bb]uild/ and [Rr]elease/ artifact ignores.

The counter cannot be replaced by the published manifest, which is the obvious
simplification to reach for: version.json is written only on stable publish, so
two consecutive dev builds would derive the same number and the second would
overwrite the first's artifact. git rev-list --count fails the same way, two
builds of one commit collide.

Both deploy workflows now poll their health endpoint until it reports exactly
the version and commit just deployed, and publish versions/<component>.json
only then. A green deploy step alone means the package was accepted, not that
it is being served -- so a manifest that disagrees with the live endpoint now
means the running instance drifted (rollback, stale revision) rather than a
pipeline that stopped halfway. The agent manifest gains the same commit and
buildUtc fields additively; the bootstrap reads it by field name, so older
installers ignore them, and sha256/bootstrapVersion stay untouched.

Requires the versions container plus AZURE_BLOB_VERSIONS_SAS_TOKEN and the two
seeded counters (backend 999, mcp 299) before the first deploy run.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 17:41:03 +02:00
Oliver Kieselbach 58c8cf87a4 updated command 2026-07-22 15:38:02 +02:00
Oliver Kieselbach 71d2c0e47f fix(maintenance): remove the time-window auto-block, it never measured data volume
GetExcessiveDataSendersAsync gated on LastEventAt - StartedAt alone. No event
count, no bytes, no rate — the name asserted something the code never checked.
What it actually selected was every enrollment session that spanned a night,
which for Autopilot is the ordinary case: device unboxed in the evening, ESP
runs, machine powered off, user signs in the next morning.

The per-device alerts from the previous commit made this visible. All three
devices blocked at the time of the analysis were false positives:

  CND3320949 / 64ed21e8    23 events over 15.5h  (1.5 events/hour)
  5CG4303CPT / db5751bc   192 events over 16.8h  (blocked mid Account Setup 1/5)
  PF5YANFB   / 2069de87   781 events over 31.5h

14 such blocks in 30 days across 9 customer tenants. Over the same window the
event-count detector fired 5 times and was right every time (2742, 2714, 2712,
2506 events). Each false block costs a live enrollment 8 hours of telemetry.

Repairing it rather than removing it does not pay: once a volume condition is
added, "many events AND long span" is a subset of "many events", which is what
the count detector already does. Volume is the quantity that costs anything; a
23-event session spanning 15 hours costs nothing. So the detector, its two admin
settings, the ExcessiveDataBlocked ops event and the portal section go, and the
event-count path is the only automatic block left.

MaxSessionWindowHours and MaintenanceBlockDurationHours are deleted rather than
deprecated: the config row is written with Replace, so the columns drop out on
the next portal save, and nothing reads them meanwhile. A comment on the
surviving detector records why there is no second one, so it does not get
rebuilt a year from now.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 15:28:28 +02:00
Oliver Kieselbach 3b2f5ed46d feat(runaway-detection): name the device in the time-window alert, link it in the block list
The two runaway detectors emitted alerts of unequal usefulness. The event-count
path fires per session and carries sessionId + serialNumber, so the Ops Events
detail modal can offer View session / Block / Kill. The time-window path
aggregated a whole tenant into one event with {devicesBlocked, windowHours} —
no device, no session, no shortcuts, on precisely the alert where an operator
wants to act. It now writes one event per blocked device with the same payload
shape; the aggregate count stays in the audit entry, where a sum is the useful
view.

The Active Blocks table showed the serial as plain text although the API already
returns BlockedSessionIds. Maintenance blocks now link the serial to the session
that triggered them; manual whole-device blocks have no session and stay text.

Two defects in the event-count path, found while comparing the detectors:

- No isBlocked guard, so the auto-action re-blocked an already-blocked device and
  overwrote BlockedAt/UnblockAt/Reason of the existing block — usually the
  time-window block on the same session, whose reason then vanished from the list.
- No status filter, so a session that finished days ago with a high EventCount
  still cost its device a 24h block. The fix needed a layer below the obvious one:
  the runaway query never selected Status, and SessionStatus.InProgress is
  ordinal 0, so a guard on the unpopulated field would have been silently
  always-true. ParseStatusForAutoAction is stricter than MapToSessionSummary --
  Enum.TryParse maps "0" onto InProgress and ORs comma-separated names into an
  unrelated member, both of which would open the gate.

Not addressed, and deliberately: a block is session-scoped and stops the upload,
so whichever detector fires first freezes the other for that session. With
MaxSessionWindowHours=12 the time detector wins in practice. That is correct
behaviour, not a defect; merging the detectors would be its own change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 14:43:00 +02:00
Oliver Kieselbach 7329f12217 ci(codeql): skip analysis on documentation-only changes
CodeQL analyzes C# and TypeScript, so a commit touching only docs or markdown
cannot change its result - it spent ~10 minutes (8:50 for C#, 1:32 for
TypeScript) re-reporting an unchanged outcome.

paths-ignore on push and pull_request only. The weekly cron stays unfiltered:
it exists to catch advisories published against code that did not change, and
it doubles as the backstop should the filter ever skip too much.

A commit touching docs AND code still runs - GitHub skips only when every
changed path matches the filter.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:10:18 +02:00
Oliver Kieselbach c9a9df99f1 docs(ci): describe what the CI workflow actually does
The header claimed it "runs the full test suite" - it also typechecks the web
and MCP projects and fails on drift between rules/guardrails.json and the
generated rule files. It also called itself "the gate that decides whether a
change is allowed to be correct", which was both garbled and false: main is
deliberately unprotected, so a red run reports rather than blocks.

Comment only, no behaviour change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:10:17 +02:00
Oliver Kieselbach 757427ffea chore(build): clear the compiler and xUnit analyzer warnings CI annotates
The CI test job surfaced 10 annotations, but GitHub caps them at 10 per job —
a full rebuild showed 31 nullable warnings plus 26 xUnit analyzer warnings.

Production code:
- CertificateValidator: X509Certificate2(byte[]) is obsolete (SYSLIB0057) ->
  X509CertificateLoader.LoadCertificate.
- AuthenticationMiddleware: IsValidTenantId now carries [NotNullWhen(true)], so
  the guard above BuildTenantAuthority is visible to flow analysis.
- AgentEndpointMigrationRules: the out params are genuinely nullable on the
  false path; netstandard2.0 has no usable [NotNullWhen], so the contract is
  documented instead.
- Constants.ClassifyEspAppsFailure: errorCode is optional by design (the
  "no HRESULT" branch is the timeout fallback) - signature says so now.

Tests: nullable annotations on helper signatures, Assert.Single/DoesNotContain
overloads instead of pre-filtering with Where, and [InlineData(null)] theories
declare nullable parameters. GatherRule helpers now seed Parameters with the
model default (empty) rather than null, matching a deserialized rule.

No behaviour change. 5616 tests pass; solution builds warning-free apart from
the two pre-existing NU1510 pruning notices on the pinned security packages.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 11:09:11 +02:00
Oliver Kieselbach 71832b9deb docs(about): list hardware rejection among the webhook alert types
The integrations list named start, success, failure and SLA alerts but omitted
hardware rejection, understating what tenants can actually subscribe to.

Verified before claiming it: WebhookNotifyOnHardwareRejection is wired
end-to-end - model, portal toggle in SectionHardwareWhitelist, and dispatch in
ReportDistressFunction. This list occurs nowhere else in the repo or in
autopilotmonitor-docs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:31:49 +02:00
Oliver Kieselbach ce4d884abc test(gather-rules): drive the collectors through their guards
The guard functions had unit tests, but nothing proved that Registry, Wmi,
Json, Xml and File actually consult them before touching the registry, WMI or
disk. These drive the collectors themselves: each blocked target must come
back empty, emit exactly one security_warning naming the target, and produce
no rule output. Prefix-spoofing included, plus a positive case through
JsonCollector so it is pinned that the guard is a gate and not a wall.

Path traversal had no explicit coverage either. The sharpest case is
...\Logs\..\..\..\..\Secret.txt - the RAW string starts with an allowed
prefix, so a naive StartsWith would admit it; the test pins that
Path.GetFullPath canonicalizes first and the allowlist sees C:\Secret.txt.
Also traversal into the C:\Users privacy block and into System32\config, both
asserted under unrestrictedMode, which lifts the allowlist but never those.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:31:49 +02:00
Oliver Kieselbach b56c774103 fix(logging): strip Unicode line separators in LogSanitizer
The sanitizer matched \p{Cc}, which covers the C0/C1 control ranges but not
U+2028 LINE SEPARATOR or U+2029 PARAGRAPH SEPARATOR - those are categories
Zl/Zp. A log viewer, a JSON-lines consumer or any JavaScript-side reader still
breaks a line on them, so they were a residual log-forging vector.

Matching \p{Zl} and \p{Zp} rather than listing escapes: the two categories
contain exactly those two characters, and it keeps the source free of
invisible literals.

Verified the test is not vacuous - the old pattern left "a<U+2028>b" as "a b",
the new one yields "ab".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:31:48 +02:00
Oliver Kieselbach 68767f436f fix(config): validate ContactEmail and seed it conditionally
The tenant contact address had no server-side validation, so a direct API
caller could store any string in the field service notices are sent to.
UpdateTenantConfiguration now rejects non-addresses, recipient lists (a comma
would silently widen who receives those notices), display-name forms and
control characters (mail-header injection), and normalizes what it stores.
Empty stays valid - having no contact address is a legitimate state.

The seed path had a worse problem than missing validation. It was a
read-modify-write of the whole model through SaveTenantConfigurationAsync,
which upserts with TableUpdateMode.Replace and no ETag: a concurrent portal
save was clobbered by the seeder's stale snapshot of EVERY field, not just
ContactEmail. It also wrote straight through IConfigRepository, bypassing the
TenantConfigurationService cache, so a seeded address stayed invisible for the
full 5-minute TTL.

TrySeedTenantContactEmailAsync now writes ETag-conditionally in Merge mode -
one property, and a 412 means someone else wrote the row so the seed abandons
rather than overwriting the winner. A lost seed is recoverable on the next
maintenance run; a lost admin edit is not.

The same seed logic existed twice (preview notification-email save and the
maintenance backfill). Both now call a single owner on
TenantConfigurationService, which holds the never-overwrite invariant, the
cache eviction and the fail-soft guarantee. Fail-soft lives in the service
rather than the repository so it holds for any implementation.

Tests cover the validator, the service contract, and the repository against a
mocked TableClient: Merge mode, the ETag actually passed, only ContactEmail in
the patch, the 412 race, 404, storage faults. Plus a regression pin that
SaveTenantConfigurationAsync is never called from the seed path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:31:30 +02:00
Oliver Kieselbach c7b06be8b7 fix(ratelimit): key the user throttle on CallerId, not the UPN
Follow-up to 479247cd. Publishing only a genuine UPN closed the fleet-wide bucket
collapse but opened a narrower hole: AuthenticationMiddleware validates issuer,
audience, lifetime and signature but requires neither upn nor scp, so an app-only
(client-credentials) token authenticates and reaches every AuthenticatedUser route
(feedback, realtime/negotiate, progress/*, health/detailed, auth/mcp, ...). Such a
token carries no upn, so it resolved to an empty identity and UserRateLimitMiddleware
skipped it entirely — previously it at least landed in the shared anonymous bucket.

Rather than reinstating a placeholder as a bucket key, separate the two concepts:

- RequestContext.UserPrincipalName keeps its meaning — a real UPN or nothing. Audit,
  presence and admin flows continue to see only genuine user identities.
- RequestContext.CallerId is the throttle identity: UPN, else the token subject
  (oid -> sub -> appid/azp), else a fail-closed shared bucket. Empty means exactly one
  thing — no JWT at all (device/anonymous routes, which carry their own limits).
- UserRateLimitMiddleware gates and buckets on CallerId.

This matches how McpQuotaEnforcementMiddleware already keys its quota (oid), so the
subject-based identity is the existing house convention for enforcement.

10 new tests: the oid/sub/appid/azp fallback chain and its precedence, the fail-closed
bucket for a token with no identifying claim (asserting it is never "anonymous"), and
that empty CallerId occurs only on unauthenticated routes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 10:17:34 +02:00
Oliver Kieselbach 479247cdae fix(ratelimit): stop agent fleet sharing one anonymous user bucket
Device and anonymous routes published the "anonymous" placeholder as
RequestContext.UserPrincipalName. UserRateLimitMiddleware skips only on an EMPTY UPN, so
every agent of every tenant was bucketed under user_ratelimit_anonymous and throttled
against the 120/min per-user limit. On 2026-07-21 21:00-21:30 UTC that rejected 1821
agent requests with 429 (/api/agent/telemetry, config, register-session, error); the
7-day trend shows it recurring and growing with fleet size.

- PolicyEnforcementMiddleware: RequestContext.UserPrincipalName carries a genuine JWT
  UPN or nothing. decision.UserIdentifier keeps the placeholder for log lines.
- Device/bootstrap routes already hold their own limit in SecurityValidator (per cert
  thumbprint / per bootstrap token), so they lose nothing.
- stats/platform and diagnostics/download had NO limiter of their own and were riding on
  the accidental shared bucket. Both now limit per trusted client IP (60/min and 30/min)
  via ClientIpExtractor, matching the HealthCheck pattern.
- 13 new tests pin the empty-UPN contract across all 12 device/anonymous routes, plus a
  guard that an authenticated caller still surfaces its real UPN.

Solution suite 5552 tests green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 09:48:33 +02:00
Oliver Kieselbach b6ba87f072 fix(deps): pin patched DataProtection + Cryptography.Xml to clear NuGet advisories
Microsoft.Identity.Web 4.3.0 pulls Microsoft.AspNetCore.DataProtection and
System.Security.Cryptography.Xml transitively at 10.0.0, which carry known
advisories (GHSA-9mv3-2cwr-p262 critical on DataProtection; GHSA-37gx-xxp4-5rgx
and siblings, high, on Cryptography.Xml). Add explicit references pinning both
to the patched 10.0.10 servicing build (matching the existing JwtBearer pin) —
the NuGet equivalent of the npm overrides used for the web/MCP alerts.

Pinned in the production Functions project so the fix flows to the deployed
artifact and to Functions.Tests via project reference. `dotnet list package
--vulnerable` now reports 0 across every project; build clean; 3110 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:44:44 +02:00
Oliver Kieselbach fa640ebe61 fix(security): resolve CodeQL log-forging findings and suppress by-design queries
Bucket B (real fixes) + Bucket A (config) of the CodeQL alert triage.

Log-forging (cs/log-forging, 8 sinks): add Helpers/LogSanitizer that strips
CR/LF and other control characters, and apply it to the logged copy of every
user-controlled value in AuthenticationMiddleware, PolicyEnforcementMiddleware
(Invoke + DecideAsync), McpQuotaEnforcementMiddleware and TableUserUsageRepository.
Only the logged value is sanitized — routing, authorization and storage keys are
untouched, so tracing fidelity for legitimate values (paths, UPNs, GUIDs) is
preserved. Structured logging already isolated the fields; this closes the CRLF
vector as defense-in-depth and clears the query.

Three genuine JS/TS fixes:
- guardValidation.ts: replace the no-op `.replace(/%/g, "%")` with a real
  escapeRegExp() — the author intended to escape; this also closes the latent
  metacharacter gap (js/identity-replacement).
- access-guard.ts: use a `%s` format argument instead of interpolating the UPN
  into the console.error format string (js/tainted-format-string).
- docs-corpus.ts: strip HTML tags with a loop-until-stable stripHtmlTags() so the
  sanitizer is robust and recognized as complete (js/incomplete-multi-char-sanitization).

CodeQL config: filter the two queries that fire structurally on this codebase,
where the flagged behavior is the feature — cs/exposure-of-sensitive-information
(audit/notification logs legitimately record the actor UPN / recipient email) and
cs/cleartext-storage-of-sensitive-information (rate limiters key on the trusted
client IP). Default suites stay active; genuine one-off false positives from other
queries are dismissed individually so those queries stay sharp.

Tests: Backend xUnit 3110 (+9 LogSanitizer), MCP vitest 19 (+1 tag-strip),
Web vitest 49, all tsc clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:38:47 +02:00
Oliver Kieselbach dbc20a9f66 test(deletion-maintenance): de-flake watchdog tests via event gates
The three watchdog tests raced a fixed fanout-body sleep against sub-second
warn/severe thresholds (e.g. 150ms body vs 500ms severe). On a loaded CI
runner the whole RunCoreAsync body overran the severe threshold before the
finally-block cancelled the watchdog, so the severe OpsEvent fired and
"LongRunning past warning" flaked (3100/3101).

Replace the wall-clock race with event gates: the fanout body now blocks on
WaitForEventAsync(<watchdog OpsEvent>) instead of sleeping, so the target
watchdog is guaranteed to have fired before the body returns. The opposite
threshold is pushed to minutes (warn-only tests) or the body waits on the
severe event (both-fire test) so warn is necessarily first. OpsEvents
recording is now lock-guarded since watchdog tasks record concurrently.

No production change. Watchdog class ran 5x green (16/16); full suite 3110/3110.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:33:26 +02:00
Oliver Kieselbach cbda62d131 Merge branch 'main' of https://github.com/okieselbach/Autopilot-Monitor 2026-07-22 01:19:36 +02:00
Oliver Kieselbach 3951000b40 ci(deploy-web): skip SWA deploy when token absent instead of failing
Dependabot PRs run without repository secrets, so AZURE_SWA_DEPLOY_TOKEN is
empty and Azure/static-web-apps-deploy exits with an error, painting a red
"Build and Deploy Job" check on every web-touching Dependabot PR. Setting
skip_deploy_on_missing_secrets makes the action skip the deploy (green) when
the token is missing; real pushes to main still carry the token and deploy.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:19:26 +02:00
dependabot[bot] c59842262e chore(deps): bump the web group across 1 directory with 10 updates (#104)
Bumps the web group with 10 updates in the /src/Web/autopilot-monitor-web directory:

| Package | From | To |
| --- | --- | --- |
| [@microsoft/applicationinsights-web](https://github.com/microsoft/ApplicationInsights-JS) | `3.3.11` | `3.4.3` |
| [@xyflow/react](https://github.com/xyflow/xyflow/tree/HEAD/packages/react) | `12.10.2` | `12.11.2` |
| [fflate](https://github.com/101arrowz/fflate) | `0.8.2` | `0.8.3` |
| [next](https://github.com/vercel/next.js) | `15.5.18` | `15.5.21` |
| [@next/bundle-analyzer](https://github.com/vercel/next.js/tree/HEAD/packages/next-bundle-analyzer) | `16.2.7` | `16.2.11` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `22.19.11` | `22.20.1` |
| [autoprefixer](https://github.com/postcss/autoprefixer) | `10.4.24` | `10.5.4` |
| [eslint](https://github.com/eslint/eslint) | `9.39.2` | `9.39.5` |
| [eslint-config-next](https://github.com/vercel/next.js/tree/HEAD/packages/eslint-config-next) | `15.5.12` | `15.5.21` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `4.1.2` | `4.1.10` |



Updates `@microsoft/applicationinsights-web` from 3.3.11 to 3.4.3
- [Release notes](https://github.com/microsoft/ApplicationInsights-JS/releases)
- [Changelog](https://github.com/microsoft/ApplicationInsights-JS/blob/main/RELEASES.md)
- [Commits](https://github.com/microsoft/ApplicationInsights-JS/commits)

Updates `@xyflow/react` from 12.10.2 to 12.11.2
- [Release notes](https://github.com/xyflow/xyflow/releases)
- [Changelog](https://github.com/xyflow/xyflow/blob/main/packages/react/CHANGELOG.md)
- [Commits](https://github.com/xyflow/xyflow/commits/@xyflow/react@12.11.2/packages/react)

Updates `fflate` from 0.8.2 to 0.8.3
- [Release notes](https://github.com/101arrowz/fflate/releases)
- [Changelog](https://github.com/101arrowz/fflate/blob/master/CHANGELOG.md)
- [Commits](https://github.com/101arrowz/fflate/compare/v0.8.2...v0.8.3)

Updates `next` from 15.5.18 to 15.5.21
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/compare/v15.5.18...v15.5.21)

Updates `@next/bundle-analyzer` from 16.2.7 to 16.2.11
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/commits/v16.2.11/packages/next-bundle-analyzer)

Updates `@types/node` from 22.19.11 to 22.20.1
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `autoprefixer` from 10.4.24 to 10.5.4
- [Release notes](https://github.com/postcss/autoprefixer/releases)
- [Changelog](https://github.com/postcss/autoprefixer/blob/main/CHANGELOG.md)
- [Commits](https://github.com/postcss/autoprefixer/compare/10.4.24...10.5.4)

Updates `eslint` from 9.39.2 to 9.39.5
- [Release notes](https://github.com/eslint/eslint/releases)
- [Commits](https://github.com/eslint/eslint/compare/v9.39.2...v9.39.5)

Updates `eslint-config-next` from 15.5.12 to 15.5.21
- [Release notes](https://github.com/vercel/next.js/releases)
- [Commits](https://github.com/vercel/next.js/commits/v15.5.21/packages/eslint-config-next)

Updates `vitest` from 4.1.2 to 4.1.10
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.10/packages/vitest)

---
updated-dependencies:
- dependency-name: "@microsoft/applicationinsights-web"
  dependency-version: 3.4.3
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: web
- dependency-name: "@xyflow/react"
  dependency-version: 12.11.2
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: web
- dependency-name: fflate
  dependency-version: 0.8.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: next
  dependency-version: 15.5.21
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: "@next/bundle-analyzer"
  dependency-version: 16.2.11
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: "@types/node"
  dependency-version: 22.20.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: web
- dependency-name: autoprefixer
  dependency-version: 10.5.4
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: web
- dependency-name: eslint
  dependency-version: 9.39.5
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: eslint-config-next
  dependency-version: 15.5.21
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
- dependency-name: vitest
  dependency-version: 4.1.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: web
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:15:43 +02:00
dependabot[bot] c3e9621a78 chore(deps): bump the mcp group across 1 directory with 5 updates (#103)
Bumps the mcp group with 5 updates in the /src/McpServer/autopilot-monitor-mcp directory:

| Package | From | To |
| --- | --- | --- |
| [@modelcontextprotocol/sdk](https://github.com/modelcontextprotocol/typescript-sdk) | `1.28.0` | `1.29.0` |
| [fuse.js](https://github.com/krisk/Fuse) | `7.1.0` | `7.5.0` |
| [@types/node](https://github.com/DefinitelyTyped/DefinitelyTyped/tree/HEAD/types/node) | `22.19.15` | `22.20.1` |
| [tsx](https://github.com/privatenumber/tsx) | `4.22.4` | `4.23.1` |
| [vitest](https://github.com/vitest-dev/vitest/tree/HEAD/packages/vitest) | `4.1.2` | `4.1.10` |



Updates `@modelcontextprotocol/sdk` from 1.28.0 to 1.29.0
- [Release notes](https://github.com/modelcontextprotocol/typescript-sdk/releases)
- [Commits](https://github.com/modelcontextprotocol/typescript-sdk/compare/v1.28.0...v1.29.0)

Updates `fuse.js` from 7.1.0 to 7.5.0
- [Release notes](https://github.com/krisk/Fuse/releases)
- [Changelog](https://github.com/krisk/Fuse/blob/main/CHANGELOG.md)
- [Commits](https://github.com/krisk/Fuse/compare/v7.1.0...v7.5.0)

Updates `@types/node` from 22.19.15 to 22.20.1
- [Release notes](https://github.com/DefinitelyTyped/DefinitelyTyped/releases)
- [Commits](https://github.com/DefinitelyTyped/DefinitelyTyped/commits/HEAD/types/node)

Updates `tsx` from 4.22.4 to 4.23.1
- [Release notes](https://github.com/privatenumber/tsx/releases)
- [Changelog](https://github.com/privatenumber/tsx/blob/master/release.config.cjs)
- [Commits](https://github.com/privatenumber/tsx/compare/v4.22.4...v4.23.1)

Updates `vitest` from 4.1.2 to 4.1.10
- [Release notes](https://github.com/vitest-dev/vitest/releases)
- [Changelog](https://github.com/vitest-dev/vitest/blob/main/docs/releases.md)
- [Commits](https://github.com/vitest-dev/vitest/commits/v4.1.10/packages/vitest)

---
updated-dependencies:
- dependency-name: "@modelcontextprotocol/sdk"
  dependency-version: 1.29.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: mcp
- dependency-name: fuse.js
  dependency-version: 7.5.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: mcp
- dependency-name: "@types/node"
  dependency-version: 22.20.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: mcp
- dependency-name: tsx
  dependency-version: 4.23.1
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: mcp
- dependency-name: vitest
  dependency-version: 4.1.10
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: mcp
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:15:10 +02:00
dependabot[bot] 3ed21c37d1 chore(deps): bump actions/upload-artifact from 6 to 7 (#102)
Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 6 to 7.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/v6...v7)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:08:23 +02:00
dependabot[bot] 4a1194569e chore(deps): bump actions/checkout from 5 to 7 (#91)
Bumps [actions/checkout](https://github.com/actions/checkout) from 5 to 7.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/v5...v7)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:07:14 +02:00
dependabot[bot] d0794fc7c5 chore(deps): bump actions/setup-node from 5 to 7 (#90)
Bumps [actions/setup-node](https://github.com/actions/setup-node) from 5 to 7.
- [Release notes](https://github.com/actions/setup-node/releases)
- [Commits](https://github.com/actions/setup-node/compare/v5...v7)

---
updated-dependencies:
- dependency-name: actions/setup-node
  dependency-version: '7'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:06:47 +02:00
dependabot[bot] f34c9d2322 chore(deps): bump github/codeql-action from 3 to 4 (#89)
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3 to 4.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/v3...v4)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-version: '4'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:06:44 +02:00
dependabot[bot] 380a8417c8 chore(deps): bump actions/setup-dotnet from 5 to 6 (#88)
Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 5 to 6.
- [Release notes](https://github.com/actions/setup-dotnet/releases)
- [Commits](https://github.com/actions/setup-dotnet/compare/v5...v6)

---
updated-dependencies:
- dependency-name: actions/setup-dotnet
  dependency-version: '6'
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-07-22 01:06:40 +02:00
Oliver Kieselbach 63826a4869 chore(dependabot): stop auto-PRing npm major bumps
Group config already batches minor/patch; majors fall outside the group and
each opens its own PR (zod 4, typescript 7, transformers 4, msal-react 5,
react-leaflet 5, react 19, @types/node 26, eslint-config-next 16 — all
breaking). Ignore version-update:semver-major for both npm ecosystems so
majors are reviewed deliberately instead of flooding the PR list. Security
updates are exempt from this filter, so vulnerability fixes still flow.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 01:03:22 +02:00
Oliver Kieselbach 7df208fac9 fix(deps): pin patched transitive deps via overrides to clear Dependabot alerts
Force the security-patched versions of transitive dependencies without
downgrading their parents (next 15, @modelcontextprotocol/sdk 1.28), which
is what `npm audit fix` would otherwise do:

- sharp ^0.35.0   (web via next, mcp via @huggingface/transformers) — libvips CVEs
- fast-uri ^3.1.4 (mcp via ajv) — host confusion
- hono ^4.12.27   (mcp via sdk) — jsx context/XSS/header advisories
- @hono/node-server ^2.0.5 (mcp via sdk)

npm audit: 0 vulnerabilities in both projects. Builds clean; MCP 447 tests
and Web 572 tests pass.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-22 00:58:37 +02:00
163 changed files with 5360 additions and 1701 deletions
+31
View File
@@ -0,0 +1,31 @@
# CodeQL configuration.
#
# The default query suites stay active — this file only filters out two C# queries
# that fire structurally on this codebase, where the flagged behavior is the feature,
# not a defect:
#
# cs/exposure-of-sensitive-information
# Audit and operational logs record WHO performed an action (actor UPN / email)
# and notification services deliver mail to a recipient address. Writing that
# identity to the log / notification IS the purpose of the field. Every such site
# is a legitimate, reviewed use — the query cannot distinguish it from a leak.
#
# cs/cleartext-storage-of-sensitive-information
# The rate limiters and distress guard use the trusted client IP as their bucket
# key. The IP is an identifier for throttling, not a secret at rest; storing it in
# the in-memory / table rate-limit key is by design.
#
# Genuine one-off false positives from OTHER queries (e.g. an open-redirect the taint
# model can't see is allow-listed, a rate-limited route it doesn't recognize) are
# dismissed individually in the Security tab so those queries stay sharp — they are
# NOT suppressed here.
#
# Log-injection (cs/log-forging) is deliberately NOT filtered: user-controlled values
# are sanitized at the sink via Helpers/LogSanitizer, so the query stays useful for
# catching a future unsanitized sink.
query-filters:
- exclude:
id: cs/exposure-of-sensitive-information
- exclude:
id: cs/cleartext-storage-of-sensitive-information
+10
View File
@@ -20,6 +20,11 @@ updates:
web:
patterns: ['*']
update-types: [minor, patch]
# Major bumps are breaking and reviewed deliberately, not auto-PR'd.
# Security updates ignore this filter, so the Security tab stays clean.
ignore:
- dependency-name: '*'
update-types: ['version-update:semver-major']
- package-ecosystem: npm
directory: /src/McpServer/autopilot-monitor-mcp
@@ -30,6 +35,11 @@ updates:
mcp:
patterns: ['*']
update-types: [minor, patch]
# Major bumps are breaking and reviewed deliberately, not auto-PR'd.
# Security updates ignore this filter, so the Security tab stays clean.
ignore:
- dependency-name: '*'
update-types: ['version-update:semver-major']
- package-ecosystem: github-actions
directory: /
+102
View File
@@ -0,0 +1,102 @@
<#
.SYNOPSIS
Reserves a monotone build number from a counter blob (ETag compare-and-swap).
.DESCRIPTION
Single source of build numbers for every component that ships from this repo
(agent, backend, MCP). Each component owns one counter blob holding a plain
integer; this script reads it, writes value+1 back under If-Match, and returns
the reserved number.
Reserve-BEFORE-build: two concurrent builds can never mint the same number.
A failed build burns a number, which is harmless -- uniqueness matters,
density does not.
The blob is public-read; only the write needs the container SAS.
Why a counter and not the published version manifest: manifests are written
on publish, not on build (the agent writes version.json only for stable
releases), so two dev builds in a row would derive the same number and
overwrite each other's versioned artifact.
.PARAMETER CounterUrl
Full URL of the counter blob, without SAS.
.PARAMETER SasToken
Container SAS with create+write permission. A leading '?' is tolerated.
.PARAMETER Override
Explicit build number instead of counter+1. If it is not greater than the
current counter the blob is left untouched (re-build of a number that was
already reserved). 0 = no override.
.PARAMETER MaxAttempts
CAS retries before giving up.
.OUTPUTS
[int] the reserved build number. Progress goes to the host, not the pipeline.
.EXAMPLE
$n = ./Request-BuildNumber.ps1 -CounterUrl $url -SasToken $sas
#>
[CmdletBinding()]
param(
[Parameter(Mandatory = $true)][string]$CounterUrl,
[Parameter(Mandatory = $true)][string]$SasToken,
[int]$Override = 0,
[int]$MaxAttempts = 5
)
$ErrorActionPreference = 'Stop'
# Tolerate both secret formats ("sp=cw&..." and "?sp=cw&...").
$writeUrl = $CounterUrl + '?' + $SasToken.TrimStart('?')
function Get-HttpStatus {
param($ErrorRecord)
if ($ErrorRecord.Exception.PSObject.Properties['Response'] -and $ErrorRecord.Exception.Response) {
return [int]$ErrorRecord.Exception.Response.StatusCode
}
return 0
}
function Read-Counter {
try {
$resp = Invoke-WebRequest -Uri $CounterUrl -Method Get -UseBasicParsing
} catch {
if ((Get-HttpStatus $_) -eq 404) {
throw "Counter blob not found at $CounterUrl. Seed it once with the last used build number: PUT the plain integer with 'x-ms-blob-type: BlockBlob' and 'If-None-Match: *'. See docs/versioning.md."
}
throw
}
$raw = if ($resp.Content -is [byte[]]) { [System.Text.Encoding]::UTF8.GetString($resp.Content) } else { [string]$resp.Content }
# If-Match requires the quoted ETag form; PowerShell strips the quotes on read.
$etag = '"' + ([string]($resp.Headers['ETag'] | Select-Object -First 1)).Trim('"') + '"'
return [pscustomobject]@{ Value = [int]$raw.Trim(); ETag = $etag }
}
for ($attempt = 1; $attempt -le $MaxAttempts; $attempt++) {
$counter = Read-Counter
if ($Override -gt 0 -and $Override -le $counter.Value) {
Write-Host " Build number $Override <= counter $($counter.Value) -- already reserved, counter blob unchanged"
return $Override
}
$target = if ($Override -gt 0) { $Override } else { $counter.Value + 1 }
try {
$headers = @{ 'x-ms-blob-type' = 'BlockBlob'; 'If-Match' = $counter.ETag }
Invoke-RestMethod -Uri $writeUrl -Method Put -Headers $headers -Body "$target" -ContentType 'text/plain' | Out-Null
Write-Host " Reserved build number $target (CAS write ok, attempt $attempt)"
return $target
} catch {
if ((Get-HttpStatus $_) -eq 412) {
Write-Host " CAS conflict on attempt $attempt (another build reserved concurrently) -- retrying"
Start-Sleep -Seconds ($attempt * 2)
} else {
throw
}
}
}
throw "Could not reserve a build number after $MaxAttempts CAS attempts"
+24 -65
View File
@@ -56,78 +56,27 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v5
uses: actions/checkout@v7
# SDK 10 = single build toolchain (parity with local build.ps1 / global.json).
- name: Setup .NET
uses: actions/setup-dotnet@v5
uses: actions/setup-dotnet@v6
with:
dotnet-version: '10.0.x'
# Build numbers come from the shared counter blob (single source of truth for CI
# AND local build.ps1). Reserve-before-build with ETag-CAS: concurrent builds can
# never mint the same number; a failed build burns a number, which is harmless.
# The blob is public-read (like the rest of the container); writes need the SAS.
# Build numbers come from the shared counter blob -- one counter and one
# implementation for CI and the local build (.github/scripts/Request-BuildNumber.ps1).
# Reserve-before-build with ETag-CAS: concurrent builds can never mint the same
# number; a failed build burns a number, which is harmless.
- name: Reserve build number (counter blob CAS)
id: version
shell: pwsh
run: |
$counterBlob = "buildcounter-v$env:LINE_MAJOR.txt"
$counterUrl = "https://autopilotmonitoreu.blob.core.windows.net/agent/$counterBlob"
# Tolerate both secret formats ("sp=cw&..." and "?sp=cw&...") -- the URL below adds the '?'.
$sasToken = "${{ secrets.AZURE_BLOB_AGENT_SAS_TOKEN }}".TrimStart('?')
$override = "${{ inputs.build_number }}"
$override = 0
if ('${{ inputs.build_number }}' -ne '') { $override = [int]'${{ inputs.build_number }}' }
function Get-Counter {
try {
$resp = Invoke-WebRequest -Uri $counterUrl -Method Get
$raw = if ($resp.Content -is [byte[]]) { [System.Text.Encoding]::UTF8.GetString($resp.Content) } else { [string]$resp.Content }
# If-Match requires the quoted ETag form; pwsh strips the quotes on read.
$etag = '"' + ([string]($resp.Headers['ETag'] | Select-Object -First 1)).Trim('"') + '"'
return @{ Value = [int]$raw.Trim(); ETag = $etag }
} catch {
$status = if ($_.Exception.Response) { [int]$_.Exception.Response.StatusCode } else { 0 }
if ($status -eq 404) {
Write-Error "Counter blob '$counterBlob' not found. Seed it once with the last used build number (see docs/agent/build-counter-blob.md) -- do NOT seed from the repo's buildcounter.txt, it is a stale offline fallback."
}
throw
}
}
$buildNumber = $null
for ($attempt = 1; $attempt -le 5; $attempt++) {
$counter = Get-Counter
if ($override -ne '') {
$target = [int]$override
if ($target -le $counter.Value) {
# Re-build of an already-reserved number: counter stays monotonic, no write.
Write-Host "Manual build number $target <= counter $($counter.Value) -- counter blob unchanged"
$buildNumber = $target
break
}
Write-Host "Manual build number $target > counter $($counter.Value) -- advancing counter"
} else {
$target = $counter.Value + 1
}
try {
$headers = @{ 'x-ms-blob-type' = 'BlockBlob'; 'If-Match' = $counter.ETag }
Invoke-RestMethod -Uri "$counterUrl`?$sasToken" -Method Put -Headers $headers -Body "$target" -ContentType 'text/plain' | Out-Null
$buildNumber = $target
Write-Host "Reserved build number ${target} (CAS write ok, attempt $attempt)"
break
} catch {
$status = if ($_.Exception.Response) { [int]$_.Exception.Response.StatusCode } else { 0 }
if ($status -eq 412) {
Write-Host "CAS conflict on attempt $attempt (another build reserved concurrently) -- retrying"
Start-Sleep -Seconds ($attempt * 2)
} else {
throw
}
}
}
if ($null -eq $buildNumber) {
Write-Error "Could not reserve a build number after 5 CAS attempts"
}
$counterUrl = "https://autopilotmonitoreu.blob.core.windows.net/agent/buildcounter-v$($env:LINE_MAJOR).txt"
$buildNumber = & ./.github/scripts/Request-BuildNumber.ps1 -CounterUrl $counterUrl -SasToken '${{ secrets.AZURE_BLOB_AGENT_SAS_TOKEN }}' -Override $override
$csproj = Get-Content $env:CSPROJ_PATH -Raw
if ($csproj -match '<VersionPrefix>(\d+\.\d+)</VersionPrefix>') {
@@ -228,8 +177,13 @@ jobs:
Write-Warning "Could not parse bootstrap `$ScriptVersion from $env:BOOTSTRAP_SOURCE"
}
# Enrich the per-line manifest emitted by MSBuild (version-v2.json, read above)
# Enrich the per-line manifest emitted by MSBuild (version-v2.json, read above).
# commit/buildUtc are the same two fields the backend and MCP manifests carry —
# additive, and the bootstrap reads this file by field name, so older installers
# ignore them. sha256/bootstrapVersion stay: those are the customer contract.
$manifest | Add-Member -NotePropertyName "sha256" -NotePropertyValue $sha256Hash -Force
$manifest | Add-Member -NotePropertyName "commit" -NotePropertyValue '${{ github.sha }}'.Substring(0, 7) -Force
$manifest | Add-Member -NotePropertyName "buildUtc" -NotePropertyValue ([DateTime]::UtcNow.ToString('o')) -Force
if ($bootstrapVersion) {
$manifest | Add-Member -NotePropertyName "bootstrapVersion" -NotePropertyValue $bootstrapVersion -Force
}
@@ -277,7 +231,12 @@ jobs:
param([string]$Local, [string]$BlobName, [string]$ContentType)
$url = "$containerUrl/$BlobName`?$sasToken"
$bytes = [System.IO.File]::ReadAllBytes($Local)
$headers = @{ "x-ms-blob-type" = "BlockBlob"; "Content-Type" = $ContentType }
# no-cache on EVERY blob: they all rotate in place (even the per-line ZIP is
# overwritten each build), and the download alias fronts them with Front Door.
# Route caching is disabled there, but a stale manifest paired with a fresh ZIP
# (independent cache entries) would fail the bootstrap SHA check -- this header
# keeps that class of bug impossible even if caching is ever re-enabled.
$headers = @{ "x-ms-blob-type" = "BlockBlob"; "Content-Type" = $ContentType; "x-ms-blob-cache-control" = "no-cache" }
if ($ContentType -eq "application/zip") {
$md5 = [System.Security.Cryptography.MD5]::Create()
$headers["Content-MD5"] = [Convert]::ToBase64String($md5.ComputeHash($bytes))
@@ -352,13 +311,13 @@ jobs:
subject-path: ${{ steps.artifact.outputs.versioned_zip_release }}
- name: Upload build artifact
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@v7
with:
name: AutopilotMonitor-Agent-v${{ steps.version.outputs.version }}
path: ${{ steps.artifact.outputs.versioned_zip_release }}
- name: Upload version metadata
uses: actions/upload-artifact@v6
uses: actions/upload-artifact@v7
with:
name: version-metadata
path: ${{ env.MANIFEST_NAME }}
+15 -11
View File
@@ -1,8 +1,12 @@
name: CI
# Runs the full test suite on every pull request and every push to main.
# The deploy workflows build but do not test — this is the gate that decides
# whether a change is allowed to be correct.
# Tests, typechecks and drift-checks every push to main and every pull request:
# the .NET suites, the web and MCP suites, and a guard that the generated rule
# files still match rules/guardrails.json.
#
# The deploy workflows build but never test, so this is the only place the suites
# actually run. It does not gate anything — main is deliberately unprotected, so a
# red run reports rather than blocks. Read the result and follow up.
on:
push:
@@ -24,9 +28,9 @@ jobs:
# windows-latest is required: the agent test projects target net48.
runs-on: windows-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- uses: actions/setup-dotnet@v5
- uses: actions/setup-dotnet@v6
with:
dotnet-version: '10.0.x'
@@ -55,9 +59,9 @@ jobs:
run:
working-directory: src/Web/autopilot-monitor-web
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- uses: actions/setup-node@v5
- uses: actions/setup-node@v7
with:
node-version: '20'
cache: npm
@@ -78,9 +82,9 @@ jobs:
run:
working-directory: src/McpServer/autopilot-monitor-mcp
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- uses: actions/setup-node@v5
- uses: actions/setup-node@v7
with:
node-version: '20'
cache: npm
@@ -98,9 +102,9 @@ jobs:
name: Generated guardrails match rules/guardrails.json
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- uses: actions/setup-node@v5
- uses: actions/setup-node@v7
with:
node-version: '20'
+20 -5
View File
@@ -1,12 +1,26 @@
name: CodeQL
# Documentation-only changes cannot affect what CodeQL analyzes (C# and
# TypeScript), so they are filtered out rather than spending ~10 minutes to
# re-report an unchanged result. A commit that touches docs AND code still runs:
# GitHub skips only when EVERY changed path matches the filter.
on:
push:
branches: [main]
paths-ignore:
- 'docs/**'
- '**/*.md'
- 'LICENSE'
pull_request:
branches: [main]
paths-ignore:
- 'docs/**'
- '**/*.md'
- 'LICENSE'
schedule:
# Weekly, so advisories published after a merge still surface.
# Weekly, so advisories published after a merge still surface. Deliberately
# unfiltered — it re-analyzes the whole tree regardless of what changed, which
# is also the backstop if the filters above ever skip too much.
- cron: '17 3 * * 1'
permissions:
@@ -35,24 +49,25 @@ jobs:
build-mode: none
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- uses: actions/setup-dotnet@v5
- uses: actions/setup-dotnet@v6
if: matrix.language == 'csharp'
with:
dotnet-version: '10.0.x'
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
uses: github/codeql-action/init@v4
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
config-file: .github/codeql/codeql-config.yml
- name: Build
if: matrix.build-mode == 'manual'
run: dotnet build AutopilotMonitor.sln --configuration Release
- name: Analyze
uses: github/codeql-action/analyze@v3
uses: github/codeql-action/analyze@v4
with:
category: /language:${{ matrix.language }}
+2 -2
View File
@@ -22,9 +22,9 @@ jobs:
permissions:
contents: write
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- uses: actions/setup-node@v5
- uses: actions/setup-node@v7
with:
node-version: '20'
+86 -6
View File
@@ -7,13 +7,19 @@ permissions:
id-token: write # OIDC federated login (secret-less)
contents: read
env:
FUNCTION_APP: autopilotmonitor-api-eu
HEALTH_URL: https://autopilotmonitor-api-eu.azurewebsites.net/api/health
VERSIONS_CONTAINER: https://autopilotmonitoreu.blob.core.windows.net/versions
CSPROJ_PATH: src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj
jobs:
build-and-deploy:
runs-on: ubuntu-latest
name: Build and Deploy Backend
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
- name: Login to Azure (OIDC)
uses: azure/login@v3
@@ -23,21 +29,95 @@ jobs:
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
- name: Setup .NET 10
uses: actions/setup-dotnet@v5
uses: actions/setup-dotnet@v6
with:
dotnet-version: '10.0.x'
# Same counter mechanism as the agent line: reserve-before-build with ETag-CAS
# against a per-component counter blob. BuildTimestampUtc is pinned here rather
# than left to MSBuild's default so the published manifest and the value baked
# into the assembly (reported by /api/health) are the same instant.
- name: Reserve build number (counter blob CAS)
id: version
shell: pwsh
run: |
$buildNumber = & ./.github/scripts/Request-BuildNumber.ps1 -CounterUrl "${{ env.VERSIONS_CONTAINER }}/backend.txt" -SasToken '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'
$csproj = Get-Content ${{ env.CSPROJ_PATH }} -Raw
if ($csproj -notmatch '<VersionPrefix>(\d+\.\d+)</VersionPrefix>') {
throw "VersionPrefix not found in ${{ env.CSPROJ_PATH }}"
}
$version = "$($Matches[1]).$buildNumber"
$buildUtc = [DateTime]::UtcNow.ToString('o')
Write-Host "Backend version: $version (built $buildUtc)"
"build_number=$buildNumber" >> $env:GITHUB_OUTPUT
"version=$version" >> $env:GITHUB_OUTPUT
"build_utc=$buildUtc" >> $env:GITHUB_OUTPUT
- name: Restore dependencies
run: dotnet restore src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj
run: dotnet restore ${{ env.CSPROJ_PATH }}
- name: Build
run: dotnet build src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj --configuration Release --no-restore -p:SourceRevisionId=${{ github.sha }}
run: dotnet build ${{ env.CSPROJ_PATH }} --configuration Release --no-restore -p:SourceRevisionId=${{ github.sha }} -p:BuildNumber=${{ steps.version.outputs.build_number }} -p:BuildTimestampUtc=${{ steps.version.outputs.build_utc }}
- name: Publish
run: dotnet publish src/Backend/AutopilotMonitor.Functions/AutopilotMonitor.Functions.csproj --configuration Release --output ./publish --no-build -p:SourceRevisionId=${{ github.sha }}
run: dotnet publish ${{ env.CSPROJ_PATH }} --configuration Release --output ./publish --no-build -p:SourceRevisionId=${{ github.sha }} -p:BuildNumber=${{ steps.version.outputs.build_number }} -p:BuildTimestampUtc=${{ steps.version.outputs.build_utc }}
- name: Deploy to Azure Functions
uses: Azure/functions-action@v1
with:
app-name: autopilotmonitor-api-eu
app-name: ${{ env.FUNCTION_APP }}
package: ./publish
# A green deploy step only means the package was accepted. This gate makes the
# workflow assert what is actually being served, so the manifest published below
# can be trusted as "this is live" rather than "this was uploaded".
# /api/health is IP rate-limited at 30/min; one poll per 10s stays well under.
- name: Verify deployed build is live
shell: pwsh
run: |
$expectedCommit = '${{ github.sha }}'.Substring(0, 7)
$expectedVersion = '${{ steps.version.outputs.version }}'
$deadline = [DateTime]::UtcNow.AddMinutes(3)
$attempt = 0
while ([DateTime]::UtcNow -lt $deadline) {
$attempt++
try {
$health = Invoke-RestMethod -Uri '${{ env.HEALTH_URL }}' -Method Get -TimeoutSec 30
if ($health.commitHash -eq $expectedCommit -and $health.version -eq $expectedVersion) {
Write-Host "Live after $attempt attempt(s): $($health.version) @ $($health.commitHash), built $($health.buildUtc)"
exit 0
}
Write-Host "Attempt ${attempt}: serving $($health.version) @ $($health.commitHash) -- waiting for $expectedVersion @ $expectedCommit"
} catch {
Write-Host "Attempt ${attempt}: /api/health not answering yet -- $($_.Exception.Message)"
}
Start-Sleep -Seconds 10
}
throw "Backend did not report $expectedVersion @ $expectedCommit within 3 minutes -- deployment not verified, version manifest NOT published"
# The manifest is the cross-component record of what is deployed where. It is
# written last and only on a verified deploy, so a difference between the
# manifest and /api/health means the running app drifted after the fact
# (rollback, stale instance) rather than a half-finished pipeline.
- name: Publish version manifest
shell: pwsh
run: |
$manifest = [ordered]@{
component = 'backend'
version = '${{ steps.version.outputs.version }}'
commit = '${{ github.sha }}'.Substring(0, 7)
buildUtc = '${{ steps.version.outputs.build_utc }}'
deployedUtc = [DateTime]::UtcNow.ToString('o')
runId = '${{ github.run_id }}'
} | ConvertTo-Json
$sas = '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'.TrimStart('?')
$url = "${{ env.VERSIONS_CONTAINER }}/backend.json?$sas"
# no-cache: rotating pointer, must never be served stale by any fronting cache.
Invoke-RestMethod -Uri $url -Method Put -Headers @{ 'x-ms-blob-type' = 'BlockBlob'; 'x-ms-blob-cache-control' = 'no-cache' } -Body $manifest -ContentType 'application/json' | Out-Null
Write-Host "Published versions/backend.json"
Write-Host $manifest
+84 -2
View File
@@ -12,6 +12,9 @@ env:
CONTAINER_APP_NAME: autopilotmonitor-mcp
RESOURCE_GROUP: ${{ secrets.AZURE_MCP_RESOURCE_GROUP }}
IMAGE_TAG: ${{ github.sha }}
HEALTH_URL: https://mcp.autopilotmonitor.com/health
VERSIONS_CONTAINER: https://autopilotmonitoreu.blob.core.windows.net/versions
PACKAGE_JSON: src/McpServer/autopilot-monitor-mcp/package.json
jobs:
build-and-deploy:
@@ -19,7 +22,7 @@ jobs:
name: Build and Deploy MCP Server
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
# The published customer documentation lives in its own PUBLIC repo and is
# baked into the image as the search_docs corpus (see the Dockerfile). No
@@ -28,7 +31,7 @@ jobs:
# RUNNING SERVER; this workflow must be re-run. /health reports the SHA
# captured below so that staleness is checkable.
- name: Check out documentation bundle
uses: actions/checkout@v5
uses: actions/checkout@v7
with:
repository: autopilotmonitor/autopilotmonitor-docs
path: mcp-docs
@@ -44,6 +47,30 @@ jobs:
tenant-id: ${{ secrets.AZURE_TENANT_ID }}
subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
# Same counter mechanism as agent and backend: reserve-before-build with
# ETag-CAS against the component's own counter blob. major.minor stays curated
# in package.json; only the build part is minted here.
- name: Reserve build number (counter blob CAS)
id: version
shell: pwsh
run: |
$buildNumber = & ./.github/scripts/Request-BuildNumber.ps1 -CounterUrl "${{ env.VERSIONS_CONTAINER }}/mcp.txt" -SasToken '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'
$pkgVersion = (Get-Content ${{ env.PACKAGE_JSON }} -Raw | ConvertFrom-Json).version
$prefix = ($pkgVersion -split '\.')[0..1] -join '.'
$version = "$prefix.$buildNumber"
$buildUtc = [DateTime]::UtcNow.ToString('o')
Write-Host "MCP version: $version (built $buildUtc)"
# 7 chars, matching what the backend's BackendBuildInfo reports — one commit
# shape across all components so the values can be compared as-is.
$shortSha = '${{ github.sha }}'.Substring(0, 7)
"build_number=$buildNumber" >> $env:GITHUB_OUTPUT
"version=$version" >> $env:GITHUB_OUTPUT
"build_utc=$buildUtc" >> $env:GITHUB_OUTPUT
"short_sha=$shortSha" >> $env:GITHUB_OUTPUT
- name: Login to ACR
run: az acr login --name ${{ env.ACR_NAME }}
@@ -52,6 +79,9 @@ jobs:
docker build \
-f src/McpServer/autopilot-monitor-mcp/Dockerfile \
--build-arg DOCS_COMMIT=${{ steps.docs.outputs.sha }} \
--build-arg BUILD_NUMBER=${{ steps.version.outputs.build_number }} \
--build-arg BUILD_COMMIT=${{ steps.version.outputs.short_sha }} \
--build-arg BUILD_UTC=${{ steps.version.outputs.build_utc }} \
-t ${{ env.ACR_NAME }}.azurecr.io/${{ env.CONTAINER_APP_NAME }}:${{ env.IMAGE_TAG }} \
-t ${{ env.ACR_NAME }}.azurecr.io/${{ env.CONTAINER_APP_NAME }}:latest \
.
@@ -63,3 +93,55 @@ jobs:
--name ${{ env.CONTAINER_APP_NAME }} \
--resource-group ${{ env.RESOURCE_GROUP }} \
--image ${{ env.ACR_NAME }}.azurecr.io/${{ env.CONTAINER_APP_NAME }}:${{ env.IMAGE_TAG }}
# `az containerapp update` returns once the revision is provisioned, not once
# it serves traffic — and with minReplicas=0 the first request also pays a cold
# start. Polling /health until it reports THIS build is what makes the manifest
# below mean "live"; a full 5 minutes because the cold start alone is ~25s.
- name: Verify deployed build is live
shell: pwsh
run: |
$expectedCommit = '${{ steps.version.outputs.short_sha }}'
$expectedVersion = '${{ steps.version.outputs.version }}'
$deadline = [DateTime]::UtcNow.AddMinutes(5)
$attempt = 0
while ([DateTime]::UtcNow -lt $deadline) {
$attempt++
try {
$health = Invoke-RestMethod -Uri '${{ env.HEALTH_URL }}' -Method Get -TimeoutSec 60
if ($health.commit -eq $expectedCommit -and $health.version -eq $expectedVersion) {
Write-Host "Live after $attempt attempt(s): $($health.version) @ $($health.commit), docs $($health.docs.commit) ($($health.docs.chunks) chunks)"
exit 0
}
Write-Host "Attempt ${attempt}: serving $($health.version) @ $($health.commit) -- waiting for $expectedVersion @ $expectedCommit"
} catch {
Write-Host "Attempt ${attempt}: /health not answering yet -- $($_.Exception.Message)"
}
Start-Sleep -Seconds 15
}
throw "MCP did not report $expectedVersion @ $expectedCommit within 5 minutes -- deployment not verified, version manifest NOT published"
# Cross-component record of what is deployed where; written last and only on a
# verified deploy. docsCommit rides along because for this component "which
# build" and "which documentation" are two separate questions.
- name: Publish version manifest
shell: pwsh
run: |
$manifest = [ordered]@{
component = 'mcp'
version = '${{ steps.version.outputs.version }}'
commit = '${{ steps.version.outputs.short_sha }}'
buildUtc = '${{ steps.version.outputs.build_utc }}'
deployedUtc = [DateTime]::UtcNow.ToString('o')
runId = '${{ github.run_id }}'
docsCommit = '${{ steps.docs.outputs.sha }}'.Substring(0, 7)
} | ConvertTo-Json
$sas = '${{ secrets.AZURE_BLOB_VERSIONS_SAS_TOKEN }}'.TrimStart('?')
$url = "${{ env.VERSIONS_CONTAINER }}/mcp.json?$sas"
# no-cache: rotating pointer, must never be served stale by any fronting cache.
Invoke-RestMethod -Uri $url -Method Put -Headers @{ 'x-ms-blob-type' = 'BlockBlob'; 'x-ms-blob-cache-control' = 'no-cache' } -Body $manifest -ContentType 'application/json' | Out-Null
Write-Host "Published versions/mcp.json"
Write-Host $manifest
+4 -1
View File
@@ -22,7 +22,7 @@ jobs:
runs-on: ubuntu-latest
name: Build and Deploy Job
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v7
with:
lfs: false
fetch-depth: 0 # Full history needed for git-based sitemap lastmod dates
@@ -39,6 +39,9 @@ jobs:
with:
azure_static_web_apps_api_token: ${{ secrets.AZURE_SWA_DEPLOY_TOKEN }}
repo_token: ${{ secrets.GITHUB_TOKEN }} # Used for Github integrations (i.e. PR comments)
# Dependabot PRs run without repo secrets, so the deploy token is absent.
# Skip (green) instead of failing the check; real deploys on main still have the token.
skip_deploy_on_missing_secrets: true
action: "upload"
###### Repository/Build Configurations - These values can be configured to match your app requirements. ######
# For more information regarding Static Web App workflow configurations, please visit: https://aka.ms/swaworkflowconfig
+16 -19
View File
@@ -22,26 +22,21 @@ attestation on top.
# Schema
- **Counter blob**: `https://autopilotmonitor.blob.core.windows.net/agent/buildcounter-v{LINE}.txt`
(per release line, analogous to `version-v{LINE}.json`). Content: the **last used**
build number as plain text. Public-read like the rest of the container (the number
is visible in `version-v{LINE}.json` anyway); writes require the container SAS.
- **Protocol — reserve before build, ETag-CAS**:
1. Anonymous `GET` the blob, capture value `N` and the `ETag`.
2. `PUT N+1` with `If-Match: <etag>` (quoted form — pwsh strips the quotes on read,
they must be re-added or the request is rejected client-side).
3. `412 Precondition Failed` → another build reserved concurrently → re-read, retry
(5 attempts, linear backoff).
4. Build with `BUILD_NUMBER=N+1` in the environment. The csproj honors
`BUILD_NUMBER` and then skips its `PersistBuildCounter` target, so the repo file
is not touched.
- **Failed builds burn a number.** That is intentional: uniqueness matters, density
does not. Reserve-before-build is what makes concurrent minting impossible.
The reserve-before-build CAS protocol itself is shared with the backend and the MCP
server and is documented once in [Version Contract](../versioning.md); the single
implementation is `.github/scripts/Request-BuildNumber.ps1`. Agent-specific:
- **Counter blob**: `https://autopilotmonitoreu.blob.core.windows.net/agent/buildcounter-v{LINE}.txt`
— one per release line, analogous to `version-v{LINE}.json`, and kept in the `agent`
container next to the artifacts it numbers rather than in `versions/`. Content: the
**last used** build number as plain text. Public-read like the rest of the container
(the number is visible in `version-v{LINE}.json` anyway); writes require the SAS.
- The reserved number reaches the build as `BUILD_NUMBER` in the environment. The
csproj honors it and then skips its `PersistBuildCounter` target, so the repo file
is not touched.
- **CI manual override** (`build_number` input): numbers `<=` counter are re-builds
and leave the blob unchanged; numbers `>` counter advance it (monotonic guard).
- **Seeding** (one-time per line): `PUT <last used number>` with `If-None-Match: *`
so an existing counter can never be clobbered. V2 was seeded with `1356` on
2026-07-17.
- V2 was seeded with `1356` on 2026-07-17.
# Consequences
@@ -62,7 +57,9 @@ attestation on top.
# Citations
- [Version Contract](../versioning.md) — the shared CAS protocol and manifest format
- `.github/workflows/build-agent.yml` — "Reserve build number (counter blob CAS)" step
- `scripts/Deployment/build.ps1` (gitignored, contains SAS) — `Request-BuildNumber`
- `.github/scripts/Request-BuildNumber.ps1` — the shared implementation
- `scripts/Deployment/build.ps1` (gitignored, contains SAS) — local build path
- `src/Agent/AutopilotMonitor.Agent.V2/AutopilotMonitor.Agent.V2.csproj` —
`BUILD_NUMBER` property + `PersistBuildCounter` target
+11 -2
View File
@@ -137,8 +137,17 @@ false` and an arm holds), `EspExiting`, `ImeUserSessionCompleted`,
advisory backstop. All routes converge on `CompleteThroughFinalizingOrDefer`:
1. **Completion gates** (currently only the RealmJoin AND-gate): if RealmJoin was
detected and is neither resolved nor timed-out (hard 60-min timeout), completion is
*deferred* — the session stays in stage and emits `completion_waiting`.
detected and has no terminal outcome yet, completion is *deferred* — the session
stays in stage and emits `completion_waiting`. Three terminals open the gate:
`Resolved` (DeploymentPhase 110), `FirstDeploymentIncomplete` (phase left the
first-deployment window 100/101 for 200/210 without 110 — the RJ ESP was aborted,
typically by an interactive logon reclassifying the run as secondary-user; emits
the `realmjoin_first_deployment_incomplete` Warning, session 224b2087), and
`Timeout` (hard 60-min deadline). The abort rule reads the *persisted*
`RealmJoinFacts.LastDeploymentPhase` (updated by the typed `RealmJoinPhaseChanged`
signal), so it survives agent restarts — the `RealmJoinDetected` replay path
evaluates it too. A first observation of 210 does **not** release (RJ may have
completed before agent boot and deploy afterwards).
2. `Finalizing` stage + ~5 s `FinalizingGrace` deadline.
3. Grace fires → `Completed`, `EnrollmentComplete` outcome, `enrollment_complete`
timeline event with full audit payload.
+2
View File
@@ -12,6 +12,8 @@ https://docs.autopilotmonitor.com (separate repository).
# Architecture
* [Architecture Guide](architecture.md) - High-level architecture overview and solution layout for contributors.
* [Version Contract](versioning.md) - One version shape for agent, backend and MCP: the ETag-CAS counter blob that mints build numbers, and the manifests published only after a deploy is verified live.
* [URL Registry](url-registry.md) - Every well-known host lives in one registry file per component (Constants.cs / utils/config.ts / src/config.ts), enforced by guard tests; includes the agent-download migration to the download alias and its deploy-order constraint.
# Agent
+10
View File
@@ -1,7 +1,17 @@
# Log
## 2026-07-23
* **Update**: `url-registry.md` — the first release after the alias migration surfaced Front Door caching as a correctness hazard: the alias served the previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) after the origin had rotated, and since manifest and ZIP are independent cache entries, a stale manifest with a fresh ZIP fails the bootstrap SHA-256 check. Route caching on `/agent/*` is now disabled and every blob upload stamps `Cache-Control: no-cache` (CI `Send-Blob`, local `build.ps1`, `versions/*.json` manifests) — all agent-container blobs rotate in place, so nothing there is safely cacheable.
* **Update**: `agent/decision-engine.md` — the RealmJoin completion gate has a third terminal, `FirstDeploymentIncomplete`. Session 224b2087 showed why: an interactive logon 17 s after DeploymentPhase 101 makes RealmJoin reclassify the run as a secondary-user deployment, whose branch writes 200/210 and never 110 — the gate stayed closed until the 60-min hard timeout although Hello and desktop were long resolved. `realmjoin_phase_changed` is now a typed `RealmJoinPhaseChanged` signal, so `RealmJoinFacts.LastDeploymentPhase` is persisted across agent restarts (this also fixes every `realmjoin_timeout` event claiming "last phase: 0"); a transition out of 100/101 into 200/210 releases the gate with the `realmjoin_first_deployment_incomplete` Warning, and the `RealmJoinDetected` replay path evaluates the same rule for restarts that swallow the transition. A first observation of 210 deliberately does not release — session 6f1959c0 started at 210 and only deployed afterwards.
* **Creation**: Added `url-registry.md` — all well-known hosts (own subdomains, both blob accounts, API origin, Graph, Entra login) are now defined once per component (`Constants.cs`, web `utils/config.ts`, MCP `src/config.ts`) and referenced everywhere else; three guard tests fail the build on any literal outside the registries. Written because the EU cutover missed two hardcoded copies of the legacy blob host — `LatestVersionsService` (feeds the portal's version badge) and `ValidateBootstrapCodeFunction` — which kept working only through the fail-soft legacy mirror in `build-agent.yml`; those two plus the agent `SelfUpdater` now use the `download.autopilotmonitor.com` alias (verified serving all four artifacts first). The portal's bootstrap allow-list accepts alias AND legacy host during the transition, which imposes a deploy order: web before (or with) backend. The sweep also surfaced a fourth copy of the www/portal host pair in `middleware.ts` (now derived from the registry via `hostRouting.ts`) and a backend short-link built on the bare apex domain, which only worked through a registrar-level 301.
## 2026-07-22
* **Creation**: Added `versioning.md` — agent, backend and MCP now share one version shape (`<major>.<minor>.<build>` + 7-char commit + build timestamp) and one implementation of the build-number reservation (`.github/scripts/Request-BuildNumber.ps1`, previously copy-pasted between `build-agent.yml` and `build.ps1`). The backend had been pinned at a hand-written `1.0.0` since inception and now starts at `1.5.1000`; the MCP server's `package.json` version was bumped by hand and now starts at `1.5.300`, and it reports its own commit on `/health` instead of only the documentation bundle's. Both deploy workflows poll their health endpoint until it reports the build just deployed and publish `versions/<component>.json` only then, so a manifest that disagrees with the live endpoint means the instance drifted rather than the pipeline failing halfway. `agent/build-counter-blob.md` keeps the agent-specific parts and now points at the shared protocol; its counter URL was still the pre-cutover storage account.
* **Creation**: Added `rules/gather-rule-guardrails.md` — the gather-rule allowlists are enforced only on the agent (the backend validates no `collectorType`, `target`, or `parameters`), so a collector that forgets to call its guard *is* the vulnerability. Written because two had: `LogParserCollector` read any path in `rule.Target`, bypassing even the `C:\Users` hard block that every other file collector honours, and `EventLogCollector` read any channel including `Security` — while the customer-facing documentation already promised both were allowlisted. Both now guard, with `logparser` checking the directory for wildcard targets (`Path.GetFullPath` throws on `*`) and re-checking each resolved path. Added an event-log channel allowlist (`eventLogChannels`) plus hard-blocked channels held in C# rather than JSON, so a parse error cannot lift them. Also fixed the command timeout, which was inert: `ReadToEnd()` blocks until the pipe closes, so `WaitForExit(30000)` was only reached after the process had already exited and a hung command pinned its worker forever — now read asynchronously and killed at the timeout.
* **Update**: Added `.github/workflows/ci.yml` — the full suite (backend, agent V2 Core, DecisionCore, SummaryDialog, web, MCP) now runs on every pull request and every push to main, plus a job that fails if the generated guardrail files drift from `rules/guardrails.json`. Previously **no** workflow ran tests on a PR: the deploy workflows build without testing, and `build-agent.yml` tested only `DecisionCore.Tests` on manual dispatch — while `trust/security-faq.md` already told customers that automated tests run in CI. Added CodeQL (C# and TypeScript) and Dependabot (nuget, npm ×2, actions). Marked `OfficeBinaryWatcherTests` `[Collection("SerialThreading")]`: its bounded re-probe races the shared ThreadPool and failed once under full-suite load.
+85
View File
@@ -0,0 +1,85 @@
---
type: Concept
title: URL Registry — One Place per Component, Guard-Enforced
description: Every well-known own or Microsoft host lives in exactly one registry file per component (Constants.cs, utils/config.ts, src/config.ts); guard tests fail the build on any literal outside them. Includes the agent-download host migration to the download alias.
resource: /src/Shared/AutopilotMonitor.Shared/Constants.cs
tags:
- architecture
- urls
- ci
- convention
timestamp: 2026-07-23T00:00:00+02:00
---
# URL Registry
Every well-known base URL — own hosts (portal, www, docs, download alias, MCP,
API, both blob accounts) and Microsoft hosts (Graph, Entra login) — is defined
in exactly one registry file per component. Code references the constant; the
literal appears nowhere else.
| Component | Registry | Guard test |
|---|---|---|
| C# (Shared/Backend/Agent) | `AutopilotMonitor.Shared.Constants` (`*BaseUrl`) | `HardcodedUrlGuardTests` |
| Web | `utils/config.ts` (`DOCS_URL`, `SITE_URL`, `PORTAL_URL`, `ENTRA_LOGIN_URL`, `API_URL_PROD`, `BLOB_URL_PROD`, `AGENT_DOWNLOAD_HOSTNAMES`) | `utils/__tests__/hardcodedUrls.guard.test.ts` |
| MCP | `src/config.ts` (`API_BASE_URL`, `ENTRA_LOGIN_BASE_URL`, `DOCS_BASE_URL`) | `src/__tests__/hardcoded-urls.guard.test.ts` |
Derived registries build on these instead of repeating hosts:
`lib/hostRouting.ts` computes `PUBLIC_HOST`/`PORTAL_HOST`/`APEX_HOST` from the
URL constants, and `middleware.ts` imports from it.
# Schema
The guards scan non-test source for the enforced host list and fail with
file:line on any occurrence outside the registry. Comment lines are ignored
(docs may cite URLs). Test files are excluded on purpose: a literal in a test
is an independent oracle — `ValidateBootstrapCodeResponseShapeTests` spells the
full agent download URL out so a registry change must be acknowledged in two
places, backend and portal allow-list, consciously.
Why enforced rather than agreed: the registry existed before (both
`Constants.AgentBlobBaseUrl` and the MCP `config.ts` header document earlier
drift), yet the EU cutover still missed two literal copies of the blob host —
`LatestVersionsService` and `ValidateBootstrapCodeFunction` kept reading the
legacy account, silently kept alive only by the fail-soft legacy mirror in
`build-agent.yml`. A convention that is not checked decays; these guards make
the next stray literal a red build instead of a production surprise.
# Agent-download host migration (2026-07-23)
The two legacy readers above plus the agent's `SelfUpdater` now use
`Constants.AgentDownloadBaseUrl` (`download.autopilotmonitor.com/agent`, Front
Door in front of the current blob origin — verified serving `version.json`,
`version-v2.json` and both ZIPs before the switch). The portal's
`bootstrapValidation.ts` allow-lists BOTH the alias and the legacy blob host
(`AGENT_DOWNLOAD_HOSTNAMES`) because bootstrap scripts already deployed in
customer Intune tenants still carry legacy URLs.
Deploy-order constraint: the web allow-list must be live BEFORE (or with) the
backend that serves alias URLs — an old portal build rejects the alias host and
the bootstrap flow dies client-side. Remove the legacy hostname from the
allow-list only after the legacy account is torn down.
**Front Door caching is a correctness hazard here, not a staleness nuisance.**
The first release after the migration exposed it: the alias kept serving the
previous `version.json` (`X-Cache: TCP_REMOTE_HIT`) while the origin was already
updated — and because manifest and ZIP are independent cache entries, a stale
manifest paired with a fresh ZIP fails the bootstrap's SHA-256 check. Two
defenses, both in place since 2026-07-23: route caching on `/agent/*` is
DISABLED, and every blob upload (CI `Send-Blob`, local `build.ps1`, the
`versions/*.json` manifests) stamps `Cache-Control: no-cache` — all these blobs
rotate in place, even the per-line ZIP. If caching is ever re-enabled, the
headers keep the mismatch class impossible.
Deliberately still on the legacy account: the `HealthCheckService` /
`MaintenanceService` legacy-keepalive probes and the fail-soft mirror in
`build-agent.yml` — they exist precisely to keep already-deployed customer
bootstrap scripts working until the customer migration completes.
# Citations
- `src/Shared/AutopilotMonitor.Shared/Constants.cs` — C# registry
- `src/Web/autopilot-monitor-web/utils/config.ts` — web registry
- `src/McpServer/autopilot-monitor-mcp/src/config.ts` — MCP registry
- `src/Backend/AutopilotMonitor.Functions.Tests/HardcodedUrlGuardTests.cs`
- [Version Contract](versioning.md) — the release pipeline this migration protects
+108
View File
@@ -0,0 +1,108 @@
---
type: Concept
title: Version Contract — Agent, Backend, MCP
description: One version shape for all three shipping components, the ETag-CAS counter blob that mints build numbers, and the published manifests that make "what is deployed" checkable from outside.
resource: /.github/scripts/Request-BuildNumber.ps1
tags:
- build
- ci
- release
- versioning
timestamp: 2026-07-22T00:00:00+02:00
---
# Version Contract
Three components ship from this repository — the agent, the Functions backend and the
MCP server. They share one version shape, one way of minting build numbers, and one
manifest format, so that a version string means the same thing whoever emits it.
```
version = <major>.<minor>.<build> major.minor curated by hand, build minted per build
commit = 7-char short SHA of this repo
buildUtc = ISO-8601 build moment
```
| Component | major.minor from | Counter blob | Reported by |
|---|---|---|---|
| Agent V2 | `VersionPrefix` in `AutopilotMonitor.Agent.V2.csproj` | `agent/buildcounter-v2.txt` | `agent/version.json` |
| Backend | `VersionPrefix` in `AutopilotMonitor.Functions.csproj` | `versions/backend.txt` | `GET /api/health` |
| MCP | `version` in `package.json` | `versions/mcp.txt` | `GET /health`, MCP handshake |
# Schema
## Build numbers — reserve before build, ETag-CAS
`.github/scripts/Request-BuildNumber.ps1` is the only implementation; the CI
workflows and the local `build.ps1` all call it.
1. Anonymous `GET` on the counter blob → value `N` and `ETag`. The blob holds the
**last used** number as plain text and is public-read; only the write needs the
container SAS.
2. `PUT N+1` with `If-Match: <etag>`. The quoted ETag form is required — PowerShell
strips the quotes on read, and an unquoted `If-Match` is rejected client-side.
3. `412 Precondition Failed` → a concurrent build reserved first → re-read and retry
(5 attempts, linear backoff).
4. Build with the reserved number.
Reserving *before* the build is what makes concurrent minting impossible. **A failed
build burns a number** — deliberate: uniqueness matters, density does not.
Why a counter and not the published manifest: manifests are written on publish, not on
build. The agent writes `version.json` only for stable releases, so deriving the next
number from it would hand two consecutive dev builds the same number and let the second
overwrite the first's artifact. A state-free scheme such as `git rev-list --count HEAD`
fails for the same reason — two builds of one commit collide.
**Seeding** (once per component): `PUT <last used number>` with `If-None-Match: *`, so
an existing counter can never be clobbered. Agent V2 was seeded with `1356` on
2026-07-17; `versions/backend.txt` with `999` and `versions/mcp.txt` with `299` on
2026-07-22.
## Local builds
No counter is reached, the build number stays `0`, and the commit is empty or
`unknown`. A `1.5.0` with no commit hash on `/api/health` is therefore a developer
machine, not a release — the fallback is the signal, not a defect. The agent is the
exception: `build.ps1` produces uploadable artifacts and reserves a real number unless
`-SkipUpload` is passed.
## Manifests
`versions/backend.json` and `versions/mcp.json` (public-read) record what is deployed:
```json
{ "component": "backend", "version": "1.5.1000", "commit": "abc1234",
"buildUtc": "…", "deployedUtc": "…", "runId": "…" }
```
The MCP manifest adds `docsCommit`, because for that component "which build" and "which
documentation" are separate questions.
Each is written **last and only after the deploy workflow has polled the component's
health endpoint until it reports that exact version and commit**. A green deploy step
alone means the package was accepted, not that it is being served. Consequently a
manifest that disagrees with the live endpoint means the running instance drifted after
the fact (rollback, stale revision) rather than a half-finished pipeline.
The agent keeps its own manifest at `agent/version.json` — the bootstrap derives that
URL from the download URL and verifies the ZIP against its `sha256`, so it cannot move.
It carries the same `commit` and `buildUtc` fields; `sha256` and `bootstrapVersion`
remain, being the customer-facing integrity contract.
# Consequences
- Bumping `major.minor` is a human decision, made in the component's own file
(`VersionPrefix`, or `version` in `package.json`). Nothing bumps it automatically.
- A new component joins by seeding a counter blob and calling the same script; no new
mechanism.
- The counter blobs and manifests live in the `versions` container on
`autopilotmonitoreu` (writes via `AZURE_BLOB_VERSIONS_SAS_TOKEN`). The agent's
counter stays in the `agent` container next to the artifacts it numbers.
# Citations
- `.github/scripts/Request-BuildNumber.ps1` — the CAS protocol
- `.github/workflows/deploy-backend.yml`, `deploy-mcp.yml` — reserve, verify, publish
- `.github/workflows/build-agent.yml`, `scripts/Deployment/build.ps1` — agent line
- [Build Counter Blob](agent/build-counter-blob.md) — agent-specific consequences
@@ -272,13 +272,16 @@ try {
$downloadAttempt++
try {
Write-Log "Download attempt ${downloadAttempt}/${maxDownloadAttempts}"
$downloadStopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Invoke-WebRequest `
-Uri $AgentDownloadUrl `
-OutFile $zipPath `
-UseBasicParsing `
-TimeoutSec 30 `
-ErrorAction Stop
Write-Log "Downloaded agent to $zipPath"
$downloadStopwatch.Stop()
$downloadSeconds = [math]::Round($downloadStopwatch.Elapsed.TotalSeconds, 1)
Write-Log "Downloaded agent to $zipPath (took ${downloadSeconds}s)"
break
}
catch {
@@ -15,7 +15,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
private const string CurrentUrl = "https://autopilotmonitor-api-eu.azurewebsites.net";
private const string TargetUrl = "https://autopilotmonitor-api-us.azurewebsites.net";
private static AgentConfigResponse Config(string migrateUrl) =>
private static AgentConfigResponse Config(string? migrateUrl) =>
new AgentConfigResponse { MigrateToApiBaseUrl = migrateUrl };
[Fact]
@@ -70,6 +70,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// Read the file the production code actually wrote and deserialize with the same lib.
var raw = File.ReadAllText(_cachePath);
var persisted = JsonConvert.DeserializeObject<AgentConfigResponse>(raw);
Assert.NotNull(persisted);
Assert.False(persisted.UnrestrictedMode);
Assert.False(persisted.AllowAgentDowngrade);
@@ -193,7 +193,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
case BackendBehaviour.AlwaysAuthFailure401:
throw new BackendAuthException("Unauthorized", 401);
case BackendBehaviour.AlwaysReturnNull:
return Task.FromResult<AgentConfigResponse>(null);
return Task.FromResult<AgentConfigResponse>(null!);
default:
throw new InvalidOperationException("unknown behaviour");
}
@@ -204,9 +204,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Configuration
// In-memory + per-instance cache so a prior test's successful fetch can't
// bleed into a later test that expects UsedDefaults.
private AgentConfigResponse _cached;
private AgentConfigResponse? _cached;
protected override void CacheConfig(AgentConfigResponse config) { _cached = config; }
protected override AgentConfigResponse LoadCachedConfig() => _cached;
protected override AgentConfigResponse LoadCachedConfig() => _cached!;
}
}
}
@@ -51,7 +51,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Integration
logger: logger,
helloTracker: null);
EspExitedEventArgs captured = null;
EspExitedEventArgs? captured = null;
tracker.EspExited += (_, args) => captured = args;
// Source time IS in the past relative to the clock — verify the tracker carries the
@@ -0,0 +1,71 @@
#nullable enable
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Analyzers
{
/// <summary>
/// LocalAdminAnalyzer.MatchesAllowedEntry — the allowed-list matching contract.
/// Entries without wildcard characters match exactly (case-insensitive); entries
/// containing * or ? are glob patterns with the same semantics as the hardware
/// whitelist. The same matcher is applied to local accounts and profile folders.
/// </summary>
public sealed class LocalAdminAnalyzerTests
{
// -------------------------------------------------------------- literal entries
[Theory]
[InlineData("SupportAdmin", "SupportAdmin")]
[InlineData("supportadmin", "SupportAdmin")]
[InlineData("SUPPORTADMIN", "supportadmin")]
public void Literal_entry_matches_exactly_case_insensitive(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("SupportAdmin2", "SupportAdmin")]
[InlineData("SupportAdmin", "SupportAdmin2")]
[InlineData("Support", "SupportAdmin")]
public void Literal_entry_does_not_substring_match(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
// -------------------------------------------------------------- glob patterns
[Theory]
[InlineData("adm-12345", "adm-*")]
[InlineData("adm-", "adm-*")]
[InlineData("ADM-9F3B2C", "adm-*")]
[InlineData("lapsAdmin01", "lapsAdmin??")]
[InlineData("svc-backup-01", "svc-*-01")]
public void Wildcard_entry_matches_glob(string name, string entry)
{
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Theory]
[InlineData("administrator2", "adm-*")] // no literal "adm-" prefix
[InlineData("xadm-123", "adm-*")] // pattern is anchored at start
[InlineData("lapsAdmin1", "lapsAdmin??")] // ? is exactly one char
[InlineData("adm.123", "adm-*")] // "-" is literal, not a regex class
public void Wildcard_entry_does_not_overmatch(string name, string entry)
{
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry(name, entry));
}
[Fact]
public void Regex_metacharacters_in_entry_are_treated_literally()
{
// "adm.123" as an entry has no wildcard chars → exact match only,
// the dot must not act as a regex any-char.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.123"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.123"));
// Mixed: wildcard present, remaining metacharacters stay literal.
Assert.True(LocalAdminAnalyzer.MatchesAllowedEntry("adm.123", "adm.*"));
Assert.False(LocalAdminAnalyzer.MatchesAllowedEntry("admX123", "adm.*"));
}
}
}
@@ -40,7 +40,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
try { Directory.Delete(_outsideUsers, recursive: true); } catch { /* best effort */ }
}
private GatherRuleContext Context(bool unrestricted = false, string imeLogPathOverride = null)
private GatherRuleContext Context(bool unrestricted = false, string? imeLogPathOverride = null)
=> new GatherRuleContext(
new AgentLogger(_tmp.Path, AgentLogLevel.Info),
"sess", "tenant",
@@ -52,13 +52,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
};
private static GatherRule Rule(string collectorType, string target,
Dictionary<string, string> parameters = null) => new GatherRule
Dictionary<string, string>? parameters = null) => new GatherRule
{
RuleId = "GATHER-GUARD-TEST",
Title = "guard test",
CollectorType = collectorType,
Target = target,
Parameters = parameters,
// Match the model default (empty, never null) so the collectors see the
// same shape a deserialized rule without parameters gives them.
Parameters = parameters ?? new Dictionary<string, string>(),
Trigger = "startup",
OutputEventType = "guard_test",
};
@@ -266,6 +268,130 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
$"expected the hung command to be killed at the timeout; took {sw.Elapsed}");
}
// -------------------------------------------------------------------
// Every remaining collector actually calls its guard
//
// The guard functions have their own unit tests, but those prove nothing about
// whether a collector consults them before touching the registry, WMI or disk.
// These drive the collectors themselves: each blocked target must come back empty,
// emit exactly one security_warning naming the target, and produce no rule output.
// -------------------------------------------------------------------
private void AssertBlocked(Dictionary<string, object> result, string target)
{
Assert.Empty(result);
var warning = SingleSecurityWarning();
Assert.Equal(true, warning.Data["blocked"]);
Assert.Equal(target, warning.Data["target"]);
Assert.DoesNotContain(_events, e => e.EventType == "guard_test");
}
[Theory]
[InlineData(@"SAM\SAM\Domains\Account\Users")]
[InlineData(@"SECURITY\Policy\Secrets")]
[InlineData(@"HKLM\SOFTWARE\Microsoft\EnrollmentsSomethingElse")] // prefix spoofing
public void RegistryCollector_OffAllowlistPath_IsBlockedBeforeTheHiveIsOpened(string target)
=> AssertBlocked(new RegistryCollector().Execute(Rule("registry", target), Context()), target);
[Theory]
[InlineData("SELECT * FROM Win32_UserAccount")]
[InlineData("SELECT * FROM Win32_ShadowCopy")]
[InlineData("SELECT * FROM Win32_OperatingSystemExtra")] // prefix spoofing
public void WmiCollector_OffAllowlistQuery_IsBlockedBeforeTheQueryRuns(string query)
=> AssertBlocked(new WmiCollector().Execute(Rule("wmi", query), Context()), query);
[Fact]
public void JsonCollector_UserProfilePath_IsBlockedBeforeTheFileIsOpened()
{
const string target = @"C:\Users\Public\Documents\secrets.json";
AssertBlocked(
new JsonCollector().Execute(
Rule("json", target, new Dictionary<string, string> { ["jsonpath"] = "$.token" }),
Context()),
target);
}
[Fact]
public void XmlCollector_UserProfilePath_IsBlockedBeforeTheFileIsOpened()
{
const string target = @"C:\Users\Public\Documents\secrets.xml";
AssertBlocked(
new XmlCollector().Execute(
Rule("xml", target, new Dictionary<string, string> { ["xpath"] = "//token" }),
Context()),
target);
}
[Fact]
public void FileCollector_UserProfilePath_IsBlockedBeforeTheFileIsOpened()
{
const string target = @"C:\Users\Public\Documents\secrets.txt";
AssertBlocked(new FileCollector().Execute(Rule("file", target), Context()), target);
}
[Fact]
public void JsonCollector_AllowedPath_ReadsTheFile()
{
// Counterpart to the blocked cases: proves the guard is a gate, not a wall —
// the collector does reach the file once the path is admitted.
var path = Path.Combine(_outsideUsers, "inventory.json");
File.WriteAllText(path, "{ \"agent\": { \"version\": \"2.4.0\" } }");
var result = new JsonCollector().Execute(
Rule("json", path, new Dictionary<string, string> { ["jsonpath"] = "$.agent.version" }),
Context(unrestricted: true));
Assert.DoesNotContain(_events, e => e.EventType == Constants.EventTypes.SecurityWarning);
Assert.Contains("2.4.0", string.Join(",", result.Values));
}
// -------------------------------------------------------------------
// Path traversal
// -------------------------------------------------------------------
[Fact]
public void FilePath_TraversalOutOfAnAllowedPrefix_IsBlocked()
{
// The raw string starts with an allowed prefix, so a naive StartsWith check would
// admit it. Path.GetFullPath runs FIRST, so the allowlist sees C:\Secret.txt.
const string traversal =
@"C:\ProgramData\Microsoft\IntuneManagementExtension\Logs\..\..\..\..\Secret.txt";
Assert.Equal(@"C:\Secret.txt", Path.GetFullPath(traversal));
Assert.False(GatherRuleGuards.IsFilePathAllowed(traversal));
}
[Fact]
public void FilePath_TraversalIntoTheUsersHardBlock_StaysBlockedEvenUnrestricted()
{
// Unrestricted mode lifts the allowlist but never the C:\Users privacy block —
// and it must not be reachable by spelling the path with '..' either.
const string traversal =
@"C:\ProgramData\Microsoft\IntuneManagementExtension\Logs\..\..\..\..\Users\Public\ntuser.dat";
Assert.StartsWith(@"C:\Users\", Path.GetFullPath(traversal), StringComparison.OrdinalIgnoreCase);
Assert.False(GatherRuleGuards.IsFilePathAllowed(traversal));
Assert.False(GatherRuleGuards.IsFilePathAllowed(traversal, unrestrictedMode: true));
}
[Fact]
public void FilePath_TraversalIntoTheConfigHiveDirectory_StaysBlockedEvenUnrestricted()
{
const string traversal = @"C:\Windows\Logs\..\System32\config\SAM";
Assert.False(GatherRuleGuards.IsFilePathAllowed(traversal));
Assert.False(GatherRuleGuards.IsFilePathAllowed(traversal, unrestrictedMode: true));
}
[Fact]
public void FileCollector_TraversalTarget_IsBlockedAndReported()
{
const string traversal =
@"C:\ProgramData\Microsoft\IntuneManagementExtension\Logs\..\..\..\..\Users\Public\ntuser.dat";
AssertBlocked(new FileCollector().Execute(Rule("file", traversal), Context()), traversal);
}
[Fact]
public void LogParser_ImeLogPathOverride_StillHonoursTheUsersHardBlock()
{
@@ -192,7 +192,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = "1" }),
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = 1 }));
Assert.NotEqual(
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = null }),
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = null! }),
GatherRuleExecutor.ComputeCanonicalHash(new Dictionary<string, object> { ["v"] = "<null>" }));
}
@@ -60,7 +60,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
}
private static GatherRule ExitRule(
string id, string triggerPhase, List<string> activePhases = null, string fromPhase = null)
string id, string triggerPhase, List<string>? activePhases = null, string? fromPhase = null)
=> new GatherRule
{
RuleId = id,
@@ -63,7 +63,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
/// <summary>Interval rule with a 1h period — the timer never ticks within a test run,
/// so these rules exist purely for synchronous scope-matrix assertions.</summary>
private static GatherRule ScopeProbeRule(string id, List<string> activePhases = null, string fromPhase = null)
private static GatherRule ScopeProbeRule(string id, List<string>? activePhases = null, string? fromPhase = null)
=> new GatherRule
{
RuleId = id,
@@ -78,7 +78,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Gather
ActiveFromPhase = fromPhase,
};
private static GatherRule StartupRule(string id, List<string> activePhases = null, string fromPhase = null)
private static GatherRule StartupRule(string id, List<string>? activePhases = null, string? fromPhase = null)
=> new GatherRule
{
RuleId = id,
@@ -27,7 +27,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
[InlineData(" ")] // whitespace only
[InlineData("<![LOG[truncated line with no closing")] // truncated / missing fields
[InlineData("garbage prefix <![LOG[msg]LOG]!><time=\"06:08:04.8834397\" date=\"2-8-2026\" component=\"C\" context=\"\" type=\"1\" thread=\"1\" file=\"\">")] // leading junk (non-ignorable) fails the StartsWith gate
public void TryParseLine_returns_false_without_throwing_for_malformed_input(string line)
public void TryParseLine_returns_false_without_throwing_for_malformed_input(string? line)
{
var ok = CmTraceLogParser.TryParseLine(line, out var entry);
@@ -24,12 +24,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Ime
/// </summary>
public sealed class ImeLogPatternReDoSTests
{
private static ImeLogPattern Pattern(string id, string regex, string action = null) => new ImeLogPattern
private static ImeLogPattern Pattern(string id, string regex, string? action = null) => new ImeLogPattern
{
PatternId = id,
Pattern = regex,
Category = "always",
Action = action,
// Action is optional on the wire (absent = null); the model declares it non-nullable.
Action = action!,
Enabled = true,
Parameters = new Dictionary<string, string>(),
};
@@ -0,0 +1,212 @@
using System;
using System.Diagnostics;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using AutopilotMonitor.Agent.V2.Core.Security;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Executes the generated cleanup script against REAL scratch trees with a real
/// powershell.exe — end-to-end replays of the 2026-07-22 incident mechanics that the
/// string pins in <see cref="CleanupServiceScriptTests"/> cannot prove:
/// <list type="bullet">
/// <item>a locked tree must abort WITHOUT deleting anything (old script force-deleted
/// marker + state around the locks and orphaned the binaries),</item>
/// <item>a straggler agent process (the incident's second instance) must be found by
/// NAME, killed, and the cleanup must then complete,</item>
/// <item>concurrent cleanup scripts must be serialized by the lock file.</item>
/// </list>
/// Timings are dialed down via the test-only BuildCleanupScript parameters so each case
/// runs in seconds. Every test uses GUID-unique process/task/tree names so nothing real
/// (a dev-box agent, a real Scheduled Task) can ever be touched.
/// </summary>
[Collection("SerialThreading")]
public class CleanupServiceScriptIntegrationTests : IDisposable
{
private readonly string _root;
private readonly string _basePath;
private readonly string _taskName = $"AmTest-NoSuchTask-{Guid.NewGuid():N}";
private readonly string _fakeProcessName = $"AmTestAgent{Guid.NewGuid().ToString("N").Substring(0, 8)}";
public CleanupServiceScriptIntegrationTests()
{
_root = Path.Combine(Path.GetTempPath(), $"am-cleanup-it-{Guid.NewGuid():N}");
_basePath = Path.Combine(_root, "AutopilotMonitor");
Directory.CreateDirectory(Path.Combine(_basePath, "Agent"));
Directory.CreateDirectory(Path.Combine(_basePath, "State"));
Directory.CreateDirectory(Path.Combine(_basePath, "Logs"));
File.WriteAllText(Path.Combine(_basePath, "Agent", "fake.exe"), "x");
File.WriteAllText(Path.Combine(_basePath, "State", "enrollment-complete.marker"), "x");
File.WriteAllText(Path.Combine(_basePath, "Logs", "agent.log"), "x");
}
public void Dispose()
{
try { Directory.Delete(_root, recursive: true); } catch { }
}
private static string LockPath => Path.Combine(Path.GetTempPath(), "AutopilotMonitor-Cleanup.lock");
private string BuildScript(bool keepLogs = false, int processWaitSeconds = 1)
=> CleanupService.BuildCleanupScript(
agentProcessName: _fakeProcessName,
agentBasePath: _basePath,
keepLogs: keepLogs,
scheduledTaskName: _taskName,
rebootOnComplete: false,
initialDelaySeconds: 0,
processWaitSeconds: processWaitSeconds,
renameRetries: 2,
renameRetryDelaySeconds: 1);
/// <summary>Runs the script synchronously in a real powershell.exe. Returns the script path.</summary>
private string RunScript(string script)
{
var scriptPath = Path.Combine(_root, $"cleanup-{Guid.NewGuid():N}.ps1");
File.WriteAllText(scriptPath, script);
using (var proc = Process.Start(new ProcessStartInfo
{
FileName = SystemPaths.PowerShell,
Arguments = $"-NoProfile -ExecutionPolicy Bypass -File \"{scriptPath}\"",
UseShellExecute = false,
CreateNoWindow = true,
}))
{
Assert.True(proc.WaitForExit(90_000), "cleanup script did not finish within 90s");
}
return scriptPath;
}
[Fact]
public void Happy_path_deletes_tree_and_removes_lock_and_script()
{
var scriptPath = RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "tree should be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(scriptPath), "script should self-delete");
Assert.False(File.Exists(LockPath), "cleanup lock should be released");
}
[Fact]
public void Locked_tree_aborts_completely_marker_and_files_survive()
{
// The incident regression: with a file locked inside the tree the OLD script
// deleted everything else (incl. the enrollment-complete marker) and left the
// binaries orphaned with the task already gone. The hardened script must treat
// the failed probe-rename as "someone still holds this" and walk away entirely.
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.None))
{
RunScript(BuildScript());
}
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")), "binaries must stay in place");
Assert.True(File.Exists(Path.Combine(_basePath, "State", "enrollment-complete.marker")),
"marker must survive so the next boot retries cleanup");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
Assert.False(File.Exists(LockPath));
}
[Fact]
public void Straggler_process_holding_the_tree_is_killed_and_cleanup_completes()
{
// Full incident replay: a SECOND agent-named process (here: a renamed
// powershell.exe, matching how only the process NAME identifies agents) holds a
// file in the tree open. The old script waited on the launcher PID only and ran
// its force-delete against the live locks. The hardened script must find the
// straggler by name, kill it after the wait deadline, and then delete everything.
var fakeExe = Path.Combine(_root, _fakeProcessName + ".exe");
File.Copy(SystemPaths.PowerShell, fakeExe);
var lockTarget = Path.Combine(_basePath, "Agent", "fake.exe").Replace("'", "''");
Process straggler = null;
try
{
straggler = Process.Start(new ProcessStartInfo
{
FileName = fakeExe,
Arguments = "-NoProfile -Command \"$f = [IO.File]::Open('" + lockTarget +
"', 'Open', 'Read', 'None'); Start-Sleep -Seconds 120\"",
UseShellExecute = false,
CreateNoWindow = true,
});
// Give the straggler time to acquire its file handle before cleanup starts.
var acquireDeadline = DateTime.UtcNow.AddSeconds(20);
var acquired = false;
while (!acquired && DateTime.UtcNow < acquireDeadline)
{
try
{
using (File.Open(Path.Combine(_basePath, "Agent", "fake.exe"),
FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
{
}
}
catch (IOException)
{
acquired = true; // share violation ⇒ straggler holds the file
}
if (!acquired) System.Threading.Thread.Sleep(200);
}
Assert.True(acquired, "straggler never acquired its file lock");
RunScript(BuildScript(processWaitSeconds: 2));
Assert.True(straggler.HasExited, "straggler must be force-killed by the name-based backstop");
Assert.False(Directory.Exists(_basePath), "after the kill the tree must be fully deleted");
Assert.Empty(Directory.GetFileSystemEntries(_root, "*.del-*"));
}
finally
{
try { if (straggler != null && !straggler.HasExited) straggler.Kill(); } catch { }
try { straggler?.Dispose(); } catch { }
}
}
[Fact]
public void Fresh_foreign_lock_backs_off_without_touching_the_tree()
{
File.WriteAllText(LockPath, "held by another cleanup");
try
{
var scriptPath = RunScript(BuildScript());
Assert.True(File.Exists(Path.Combine(_basePath, "Agent", "fake.exe")),
"loser script must not touch the tree — the lock holder owns the deletion");
Assert.True(File.Exists(LockPath), "foreign lock must not be deleted by the loser");
Assert.False(File.Exists(scriptPath), "loser still self-deletes its script");
}
finally
{
try { File.Delete(LockPath); } catch { }
}
}
[Fact]
public void Stale_foreign_lock_is_taken_over_and_cleanup_proceeds()
{
File.WriteAllText(LockPath, "crashed cleanup from a previous boot");
File.SetLastWriteTime(LockPath, DateTime.Now.AddMinutes(-15));
RunScript(BuildScript());
Assert.False(Directory.Exists(_basePath), "stale lock must be taken over, cleanup must run");
Assert.False(File.Exists(LockPath));
}
[Fact]
public void KeepLogs_variant_deletes_everything_except_logs()
{
RunScript(BuildScript(keepLogs: true));
Assert.True(File.Exists(Path.Combine(_basePath, "Logs", "agent.log")), "Logs must survive");
Assert.False(Directory.Exists(Path.Combine(_basePath, "Agent")));
Assert.False(Directory.Exists(Path.Combine(_basePath, "State")));
Assert.Empty(Directory.GetFileSystemEntries(_basePath, "*.del-*"));
}
}
}
@@ -0,0 +1,135 @@
using AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Runtime
{
/// <summary>
/// Pins the LIFE-F7 hardening invariants of the self-destruct cleanup script
/// (incident 2026-07-22: a second agent instance held exe + loaded DLLs while the
/// old script — waiting on the launcher PID only — force-deleted around the locks
/// AFTER removing the Scheduled Task, orphaning the binaries with no retry path).
/// These are string-level pins on <see cref="CleanupService.BuildCleanupScript"/>:
/// weakening any of them silently re-introduces the orphaned-binaries failure mode.
/// </summary>
public class CleanupServiceScriptTests
{
private const string ProcessName = "AutopilotMonitor.Agent";
private const string BasePath = @"C:\ProgramData\AutopilotMonitor";
private const string TaskName = "AutopilotMonitorAgent";
private static string Script(bool keepLogs = false, bool reboot = false)
=> CleanupService.BuildCleanupScript(ProcessName, BasePath, keepLogs, TaskName, reboot);
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Waits_by_process_name_not_launcher_pid(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #2: name-based wait covers marker-retry / self-update / manual instances
// and is immune to PID reuse. Wait-Process -Id saw exactly one PID and nothing else.
Assert.Contains($"Get-Process -Name '{ProcessName}'", script);
Assert.DoesNotContain("Wait-Process", script);
// Backstop: stragglers are force-killed — their images are deleted right after.
Assert.Contains("Stop-Process -Force", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Never_deletes_the_original_path(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #3: deletion happens exclusively on successfully renamed targets
// ($entry.Renamed). A Remove-Item against the original tree is what deleted
// marker + state around locked binaries in the incident.
Assert.DoesNotContain($"Remove-Item -Path '{BasePath}'", script);
Assert.Contains("Remove-Item -Path $entry.Renamed", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Task_removal_is_gated_on_probe_and_deletion_on_task_gone(bool keepLogs)
{
var script = Script(keepLogs);
// Probe-rename first; only a successful probe reaches task removal, and only a
// removed task reaches deletion. Both failure branches rename BACK so the next
// boot retries with task + marker + files intact.
Assert.Contains("if ($probeOk)", script);
Assert.Contains("if ($taskGone)", script);
Assert.Contains("Rename-Item -Path $entry.Renamed -NewName $entry.Original", script);
// LIFE-F6 task-removal loop + module-independent fallback stay intact.
Assert.Contains("Unregister-ScheduledTask -TaskName $taskName", script);
Assert.Contains("schtasks.exe /Delete /TN $taskName /F", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Concurrent_cleanup_scripts_are_serialized_by_lock_file(bool keepLogs)
{
var script = Script(keepLogs);
// LIFE-F7 #1: CreateNew is the atomic single-flight gate; the loser exits without
// touching the tree. Stale locks (>10 min) are taken over.
Assert.Contains("AutopilotMonitor-Cleanup.lock", script);
Assert.Contains("[System.IO.FileMode]::CreateNew", script);
Assert.Contains("$lockAge.TotalMinutes -gt 10", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Rename_targets_carry_a_guid_suffix(bool keepLogs)
{
// A fixed '.del' target collides with leftovers of earlier attempts and with a
// concurrent script's rename — the GUID makes every attempt's target unique.
Assert.Contains(".del-' + [Guid]::NewGuid().ToString('N')", Script(keepLogs));
}
[Fact]
public void KeepLogs_variant_excludes_logs_directory()
{
var script = Script(keepLogs: true);
Assert.Contains("-Exclude 'Logs'", script);
Assert.DoesNotContain($"Rename-Item -Path '{BasePath}' -NewName", script);
}
[Fact]
public void Full_variant_renames_the_whole_tree()
{
var script = Script(keepLogs: false);
Assert.Contains($"Rename-Item -Path '{BasePath}' -NewName $renamedPath", script);
Assert.DoesNotContain("-Exclude 'Logs'", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Reboot_flag_controls_shutdown_invocation(bool reboot)
{
var script = Script(keepLogs: false, reboot: reboot);
if (reboot)
Assert.Contains("shutdown.exe /r /t 10", script);
else
Assert.DoesNotContain("shutdown.exe", script);
}
[Theory]
[InlineData(false)]
[InlineData(true)]
public void Script_and_lock_are_released_in_finally(bool keepLogs)
{
var script = Script(keepLogs);
Assert.Contains("finally", script);
Assert.Contains("$lockStream.Dispose()", script);
Assert.Contains("Remove-Item -Path $scriptPath -Force", script);
}
}
}
@@ -256,7 +256,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
[InlineData(null, false)]
[InlineData("", false)]
[InlineData("0x8007", false)]
public void IsAppxFamilyHresult_MatchesOnlyAppxDeploymentRange(string hresult, bool expected)
public void IsAppxFamilyHresult_MatchesOnlyAppxDeploymentRange(string? hresult, bool expected)
{
Assert.Equal(expected, AppxFailureAnalyzer.IsAppxFamilyHresult(hresult));
}
@@ -39,7 +39,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
public List<EspFailureDetectedEventArgs> EspFailures { get; } = new List<EspFailureDetectedEventArgs>();
public ProvisioningStatusTracker Tracker { get; }
public Fixture(Func<IReadOnlyList<AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState>> packageStatesProbe = null)
public Fixture(Func<IReadOnlyList<AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.Ime.AppPackageState>>? packageStatesProbe = null)
{
var clock = new VirtualClock(Fixed);
var post = new InformationalEventPost(Sink, clock);
@@ -179,7 +179,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
}");
var captured = f.CapturedEvents();
Assert.Single(captured.Where(e => e.EventType == "esp_failure_settle_started"));
Assert.Single(captured, e => e.EventType == "esp_failure_settle_started");
}
// =====================================================================
@@ -214,8 +214,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
// No terminal fire — the failure was retracted.
Assert.Empty(f.EspFailures);
var recovered = Assert.Single(f.CapturedEvents()
.Where(e => e.EventType == "esp_failure_settle_recovered"));
var recovered = Assert.Single(
f.CapturedEvents(), e => e.EventType == "esp_failure_settle_recovered");
Assert.Equal("AccountSetup", (string)recovered.Data["category"]);
Assert.Equal("Apps", (string)recovered.Data["failedSubcategory"]);
Assert.Equal("inProgress", (string)recovered.Data["observedState"]);
@@ -280,7 +280,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
f.Tracker.TriggerSettleTimerForTest("AccountSetupCategory.Status");
Assert.Single(f.EspFailures);
Assert.Empty(f.CapturedEvents().Where(e => e.EventType == "esp_failure_settle_recovered"));
Assert.DoesNotContain(f.CapturedEvents(), e => e.EventType == "esp_failure_settle_recovered");
}
// =====================================================================
@@ -326,8 +326,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
""AppsSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Apps (Error)""}
}");
var settleStarted = Assert.Single(f.CapturedEvents()
.Where(e => e.EventType == "esp_failure_settle_started"));
var settleStarted = Assert.Single(
f.CapturedEvents(), e => e.EventType == "esp_failure_settle_started");
Assert.Equal(1, (int)settleStarted.Data["trackedAppsNotCompletedCount"]);
var apps = (List<Dictionary<string, object>>)settleStarted.Data["trackedAppsNotCompleted"];
var app = Assert.Single(apps);
@@ -354,8 +354,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
""SecurityPoliciesSubcategory"": {""subcategoryState"":""failed"",""subcategoryStatusText"":""Failed""}
}");
var settleStarted = Assert.Single(f.CapturedEvents()
.Where(e => e.EventType == "esp_failure_settle_started"));
var settleStarted = Assert.Single(
f.CapturedEvents(), e => e.EventType == "esp_failure_settle_started");
Assert.False(settleStarted.Data.ContainsKey("trackedAppsNotCompleted"));
Assert.False(settleStarted.Data.ContainsKey("trackedAppsNotCompletedCount"));
}
@@ -0,0 +1,143 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
{
/// <summary>
/// Pins the anti-flood semantics of <see cref="EmergencyReporter.TrySendAsync"/> after the
/// delivery-hardening change (2026-07-23): the once-per-session dedup and the per-session
/// budget count DELIVERED reports. The previous mark-before-send permanently suppressed an
/// error key after its first transport failure — fatal for single-shot callers like the 48h
/// emergency break, whose process exits right after the attempt.
/// </summary>
public class EmergencyReporterTests
{
private sealed class FakeApiClient : BackendApiClient
{
public readonly List<TimeSpan?> Timeouts = new List<TimeSpan?>();
public readonly Queue<bool> Results = new Queue<bool>();
public FakeApiClient() : base(
httpClient: new HttpClient(),
baseUrl: "http://localhost",
manufacturer: string.Empty,
model: string.Empty,
serialNumber: string.Empty,
useBootstrapTokenAuth: false,
bootstrapToken: null,
agentVersion: "0.0.0",
logger: null)
{
}
public int Calls => Timeouts.Count;
public override Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
Timeouts.Add(timeout);
return Task.FromResult(Results.Count > 0 && Results.Dequeue());
}
}
private static EmergencyReporter Reporter(FakeApiClient client)
=> new EmergencyReporter(client, "session", "tenant", "0.0.0", logger: null);
[Fact]
public async Task Failed_send_rolls_back_reservation_so_a_later_call_retries()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // first call fails
client.Results.Enqueue(true); // second call succeeds
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
// Old behaviour: second call suppressed as "already reported" ⇒ 1 call.
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Delivered_send_is_deduplicated_per_key()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(1, client.Calls);
}
[Fact]
public async Task Retries_stay_inside_one_reservation_and_stop_on_success()
{
var client = new FakeApiClient();
client.Results.Enqueue(false);
client.Results.Enqueue(true); // second attempt delivers — third must not fire
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, perAttemptTimeout: TimeSpan.FromMilliseconds(50), retryDelay: TimeSpan.Zero);
Assert.Equal(2, client.Calls);
// Delivered ⇒ dedup holds for the session.
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Per_attempt_timeout_is_forwarded_to_the_api_client()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
var reporter = Reporter(client);
var timeout = TimeSpan.FromSeconds(15);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 1, perAttemptTimeout: timeout);
Assert.Equal(timeout, Assert.Single(client.Timeouts));
}
[Fact]
public async Task All_attempts_failing_rolls_back_after_exhausting_attempts()
{
var client = new FakeApiClient(); // queue empty ⇒ every attempt fails
var reporter = Reporter(client);
await reporter.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak, "m",
attempts: 3, retryDelay: TimeSpan.Zero);
Assert.Equal(3, client.Calls);
// Rolled back ⇒ a later call may try again (budget + dedup + cooldown restored).
client.Results.Enqueue(true);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
Assert.Equal(4, client.Calls);
}
[Fact]
public async Task Different_keys_do_not_suppress_each_other_after_rollback()
{
var client = new FakeApiClient();
client.Results.Enqueue(false); // break report fails → rollback (incl. cooldown restore)
client.Results.Enqueue(true); // unrelated report must still go out
var reporter = Reporter(client);
await reporter.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m");
await reporter.TrySendAsync(AgentErrorType.ConfigFetchFailed, "m");
Assert.Equal(2, client.Calls);
}
}
}
@@ -44,7 +44,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
private readonly FakeQuarantineSink _quarantine = new FakeQuarantineSink();
private readonly FakeSignalIngressSink _sink = new FakeSignalIngressSink();
private readonly DecisionEngine _engine = new DecisionEngine();
private DecisionStepProcessor _sut;
private DecisionStepProcessor? _sut;
public DecisionStepProcessorParkedTripwireTests()
{
@@ -107,7 +107,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
return (step, signal);
}
private FakeSignalIngressSink.PostedSignal FindTripwirePost()
private FakeSignalIngressSink.PostedSignal? FindTripwirePost()
{
foreach (var posted in _sink.Posted)
{
@@ -121,6 +121,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
return null;
}
/// <summary>The tripwire post, asserted present — for the assertions that only run
/// after <c>WaitForTripwire()</c> has already confirmed one was emitted.</summary>
private FakeSignalIngressSink.PostedSignal RequireTripwirePost()
{
var post = FindTripwirePost();
Assert.NotNull(post);
return post!;
}
private int CountTripwirePosts()
{
var count = 0;
@@ -165,9 +174,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
Thread.Sleep(SuppressWait);
Assert.Equal(1, CountTripwirePosts());
var post = FindTripwirePost();
var post = RequireTripwirePost();
Assert.Equal(DecisionSignalKind.InformationalEvent, post.Kind);
Assert.Equal(nameof(SessionStage.EspAccountSetup), post.Payload["stage"]);
Assert.Equal(nameof(SessionStage.EspAccountSetup), post.Payload!["stage"]);
Assert.Equal("Warning", post.Payload[SignalPayloadKeys.Severity]);
Assert.Equal("true", post.Payload[SignalPayloadKeys.ImmediateUpload]);
Assert.Contains("esp_final_exit", post.Payload["signalsSeen"]);
@@ -222,7 +231,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
sut.ApplyStep(step, signal);
Assert.True(WaitForTripwire());
Assert.Equal(DeadlineNames.ClassifierTick, FindTripwirePost().Payload["armedDeadlines"]);
Assert.Equal(DeadlineNames.ClassifierTick, RequireTripwirePost().Payload!["armedDeadlines"]);
}
[Fact]
@@ -237,7 +246,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
sut.ApplyStep(step, signal);
Assert.True(WaitForTripwire());
Assert.Equal(nameof(SessionStage.AwaitingDesktop), FindTripwirePost().Payload["stage"]);
Assert.Equal(nameof(SessionStage.AwaitingDesktop), RequireTripwirePost().Payload!["stage"]);
}
// ===================================================================== Dwell semantics
@@ -256,8 +256,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
RemoteConfigFetchOutcome outcome,
int configVersion = 0,
int attempts = 1,
string failureType = null,
string failureMessage = null,
string? failureType = null,
string? failureMessage = null,
int? authStatusCode = null)
{
// CurrentConfig is set via the protected SetConfig path — easiest:
@@ -33,7 +33,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
[InlineData("Network connections (1 of 1 added)", null)]
[InlineData("", null)]
[InlineData(null, null)]
public void TryExtractErrorCode_extracts_parenthesised_hex_tail(string statusText, string expected)
public void TryExtractErrorCode_extracts_parenthesised_hex_tail(string? statusText, string? expected)
{
var actual = ProvisioningStatusTracker.TryExtractErrorCode(statusText);
Assert.Equal(expected, actual);
@@ -195,7 +195,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
var all = f.InfoEvents(SharedEventTypes.ScriptCompleted);
Assert.Equal(2, all.Count);
Assert.Equal("65.00", all[1].Payload!["durationSeconds"]);
Assert.Equal("cycle_including_reporting_latency", all[1].Payload["durationBasis"]);
Assert.Equal("cycle_including_reporting_latency", all[1].Payload!["durationBasis"]);
}
[Fact]
@@ -151,17 +151,23 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
}
[Fact]
public void TriggerPhaseChangedFromTest_emits_informational_only_no_decision_signal()
public void TriggerPhaseChangedFromTest_emits_typed_signal_and_informational_event()
{
using var f = new Fixture();
using var adapter = new RealmJoinWatcherAdapter(f.Watcher, f.Ingress, f.Clock);
adapter.TriggerPhaseChangedFromTest(prev: 100, curr: 200);
// Phase change is observability-only. There is no DecisionSignalKind.RealmJoinPhaseChanged.
// 1) Typed signal — persists LastDeploymentPhase and drives the aborted-RJ-ESP
// detection in the reducer (session 224b2087). Not Detected/Resolved.
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.RealmJoinDetected);
Assert.DoesNotContain(f.Ingress.Posted, p => p.Kind == DecisionSignalKind.RealmJoinResolved);
var decision = f.Ingress.Posted.Single(p => p.Kind == DecisionSignalKind.RealmJoinPhaseChanged);
Assert.Equal("RealmJoinWatcher", decision.SourceOrigin);
Assert.Equal("200", decision.Payload![DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase]);
Assert.Equal("100", decision.Payload[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase]);
// 2) Dual-emit informational event keeps the existing timeline entry.
var info = f.Ingress.Posted.Single(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
@@ -1,4 +1,4 @@
using System;
using System;
using System.Collections.Generic;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic;
using Xunit;
@@ -19,7 +19,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
private static List<BandwidthJobSample> Jobs(params BandwidthJobSample[] jobs)
=> new List<BandwidthJobSample>(jobs);
private static BandwidthJobSample Job(string fileId, long wan, long lan = 0)
private static BandwidthJobSample Job(string? fileId, long wan, long lan = 0)
=> new BandwidthJobSample { FileId = fileId, WanBytes = wan, LanBytes = lan };
[Fact]
@@ -43,8 +43,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.NotNull(estimate);
Assert.Equal(1, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsMax.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsMax!.Value, 3);
Assert.Equal(6_000_000, estimate.WanBytesObserved);
Assert.Equal("10-50", estimate.Bucket);
}
@@ -58,7 +58,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
estimator.AddSnapshot(T0.AddSeconds(3), Jobs(Job("a", 3_000_000), Job("b", 3_000_000)));
var estimate = estimator.TryBuildEstimate();
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
}
[Fact]
@@ -74,7 +74,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.Equal(2, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsMax.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsMax!.Value, 3);
}
[Fact]
@@ -91,7 +91,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
estimator.AddSnapshot(T0.AddSeconds(63), Jobs(Job("a", 106_000_000)));
var estimate = estimator.TryBuildEstimate();
Assert.Equal(1, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
}
[Fact]
@@ -130,8 +130,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
estimator.AddSnapshot(T0.AddSeconds(3), Jobs(Job("a", wan: 6_000_000, lan: 375_000_000)));
var estimate = estimator.TryBuildEstimate();
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(1000.0, estimate.LanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
Assert.Equal(1000.0, estimate.LanMbpsP90!.Value, 3);
Assert.Equal("10-50", estimate.Bucket); // bucket keyed to WAN, not LAN
}
@@ -182,8 +182,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.Equal(20, estimate.WanSampleCount);
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3); // p90 stays on the steady plateau
Assert.Equal(160.0, estimate.WanMbpsMax.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3); // p90 stays on the steady plateau
Assert.Equal(160.0, estimate.WanMbpsMax!.Value, 3);
}
[Fact]
@@ -265,7 +265,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Telemetry
var estimate = estimator.TryBuildEstimate();
Assert.Equal(1, estimate.WanSampleCount); // only the finite positive sample survived
Assert.Equal(16.0, estimate.WanMbpsP90.Value, 3);
Assert.Equal(16.0, estimate.WanMbpsP90!.Value, 3);
Assert.Equal(0, estimate.WanBytesObserved);
}
@@ -31,84 +31,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
_logger.Info($"Executing FULL SELF-DESTRUCT (Scheduled Task + Files{(_configuration.RebootOnComplete ? " + Reboot" : "")})");
var currentProcessId = Process.GetCurrentProcess().Id;
var currentProcess = Process.GetCurrentProcess();
var agentBasePath = Path.GetFullPath(Path.Combine(
Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData),
"AutopilotMonitor"
));
var logDir = Path.GetFullPath(Path.Combine(agentBasePath, "Logs"));
var keepLogs = _configuration.KeepLogFile;
// Create a self-deleting PowerShell script
var cleanupScript = $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Wait for agent process to exit
Start-Sleep -Seconds 2
Wait-Process -Id {currentProcessId} -ErrorAction SilentlyContinue -Timeout 30
Start-Sleep -Seconds 1
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan task firing
# against a deleted exe spams Task Scheduler errors every boot forever and, once the files (incl.
# the enrollment-complete marker) are gone, has no cleanup-retry path. So confirm the task is gone
# first; if it cannot be removed, SKIP the file deletion and leave everything intact for a clean
# cleanup retry on the next boot (the marker survives, so bootstrap re-runs cleanup).
$taskName = '{_configuration.ScheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
{(keepLogs ? $@"
if ($taskGone) {{
# Delete everything except Logs directory
Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' | ForEach-Object {{
$dest = $_.FullName + '.del'
try {{ Rename-Item -Path $_.FullName -NewName $dest -Force -ErrorAction SilentlyContinue }} catch {{ }}
Remove-Item -Path $dest -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
" : $@"
if ($taskGone) {{
# Rename folder then delete. Retry a few times to let the OS release handles.
$renamedPath = '{agentBasePath}.del'
$renamed = $false
for ($i = 1; $i -le 10; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$renamed = $true
break
}} catch {{
Start-Sleep -Seconds 2
}}
}}
if ($renamed) {{
Remove-Item -Path $renamedPath -Recurse -Force -ErrorAction SilentlyContinue
}} else {{
Remove-Item -Path '{agentBasePath}' -Recurse -Force -ErrorAction SilentlyContinue
}}
}}
")}
{(_configuration.RebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
";
var cleanupScript = BuildCleanupScript(
agentProcessName: currentProcess.ProcessName,
agentBasePath: agentBasePath,
keepLogs: _configuration.KeepLogFile,
scheduledTaskName: _configuration.ScheduledTaskName,
rebootOnComplete: _configuration.RebootOnComplete);
// Write cleanup script to temp location (outside of agent folder)
var tempScriptPath = Path.Combine(Path.GetTempPath(), $"AutopilotMonitor-Cleanup-{Guid.NewGuid()}.ps1");
@@ -143,5 +77,171 @@ Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}
}
/// <summary>
/// Builds the self-deleting cleanup PowerShell script. Extracted (internal static, pure)
/// so tests can pin the hardened invariants without launching processes.
/// <para>
/// LIFE-F7 hardening (incident 2026-07-22 — exe + loaded DLLs orphaned forever):
/// </para>
/// <list type="number">
/// <item><b>Single-flight lock</b> — the enrollment-complete marker is written BEFORE
/// cleanup launches, so any agent start during an in-flight cleanup spawns a SECOND
/// cleanup script. Two unsynchronized scripts interleave their rename/delete phases
/// and leave partial trees behind. A CreateNew lock file in the SYSTEM temp dir
/// serializes them; a stale lock (&gt;10 min) is taken over.</item>
/// <item><b>Wait by process NAME, not launcher PID</b> — marker-retry instances,
/// self-update restarts and manual runs all load the same images but have PIDs the
/// old <c>Wait-Process -Id</c> never saw (and an already-freed PID can be reused by
/// an unrelated boot-time process). After the deadline any straggler is force-killed:
/// its images are about to be deleted anyway.</item>
/// <item><b>Probe-rename BEFORE task removal, never delete the original path</b> — the
/// rename is the atomic lock probe. If it fails, something still holds the tree: abort
/// with task + files + marker intact so the next boot retries (file-side symmetry to
/// the LIFE-F6 task-side rule). If the task cannot be removed afterwards, rename BACK
/// and abort for the same reason. Only a fully renamed tree with the task gone is
/// deleted — the old <c>Remove-Item</c>-on-original fallback deleted marker + state
/// around locked binaries AFTER the task was gone, converting a transient lock into a
/// permanently orphaned exe.</item>
/// </list>
/// </summary>
/// <remarks>
/// The timing parameters exist ONLY so integration tests can replay the incident
/// scenarios (locked tree, straggler process) in seconds instead of minutes —
/// production call sites always use the defaults.
/// </remarks>
internal static string BuildCleanupScript(
string agentProcessName,
string agentBasePath,
bool keepLogs,
string scheduledTaskName,
bool rebootOnComplete,
int initialDelaySeconds = 2,
int processWaitSeconds = 60,
int renameRetries = 10,
int renameRetryDelaySeconds = 2)
{
return $@"
$scriptPath = $MyInvocation.MyCommand.Path
# Single-flight lock (LIFE-F7 #1): serialize concurrent cleanup scripts. CreateNew is
# atomic; the loser backs off and lets the winner own the deletion.
$lockPath = Join-Path ([System.IO.Path]::GetTempPath()) 'AutopilotMonitor-Cleanup.lock'
$lockStream = $null
try {{
if (Test-Path $lockPath) {{
$lockAge = (Get-Date) - (Get-Item $lockPath -ErrorAction Stop).LastWriteTime
if ($lockAge.TotalMinutes -gt 10) {{ Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue }}
}}
$lockStream = [System.IO.File]::Open($lockPath, [System.IO.FileMode]::CreateNew, [System.IO.FileAccess]::Write, [System.IO.FileShare]::None)
}} catch {{
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
exit 0
}}
try {{
# Wait for EVERY agent process to exit, by NAME (LIFE-F7 #2) — covers marker-retry
# instances, self-update restarts and manual runs, and is immune to PID reuse.
Start-Sleep -Seconds {initialDelaySeconds}
$deadline = (Get-Date).AddSeconds({processWaitSeconds})
while ((Get-Date) -lt $deadline) {{
if (-not (Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue)) {{ break }}
Start-Sleep -Seconds 2
}}
# Backstop: force-kill stragglers — their images are about to be deleted anyway.
Get-Process -Name '{agentProcessName}' -ErrorAction SilentlyContinue | Stop-Process -Force -ErrorAction SilentlyContinue
Start-Sleep -Seconds 1
{(keepLogs ? $@"
# Probe-rename all children except Logs (LIFE-F7 #3). Any failure ⇒ rename back and
# abort with the Scheduled Task intact so the next boot retries cleanly.
$renamed = @()
$probeOk = $true
foreach ($item in @(Get-ChildItem -Path '{agentBasePath}' -Exclude 'Logs' -ErrorAction SilentlyContinue)) {{
$dest = $item.FullName + '.del-' + [Guid]::NewGuid().ToString('N')
$itemOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path $item.FullName -NewName $dest -Force -ErrorAction Stop
$itemOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
if ($itemOk) {{ $renamed += @{{ Original = $item.FullName; Renamed = $dest }} }}
else {{ $probeOk = $false; break }}
}}
" : $@"
# Probe-rename the whole tree (LIFE-F7 #3). Failure ⇒ abort with task + files + marker
# intact so the next boot retries cleanly. GUID suffix avoids collisions with leftovers
# of earlier attempts.
$renamedPath = '{agentBasePath}.del-' + [Guid]::NewGuid().ToString('N')
$probeOk = $false
for ($i = 1; $i -le {renameRetries}; $i++) {{
try {{
Rename-Item -Path '{agentBasePath}' -NewName $renamedPath -Force -ErrorAction Stop
$probeOk = $true
break
}} catch {{
Start-Sleep -Seconds {renameRetryDelaySeconds}
}}
}}
$renamed = @(@{{ Original = '{agentBasePath}'; Renamed = $renamedPath }})
")}
if ($probeOk) {{
# Remove Scheduled Task — verify removal BEFORE deleting files (LIFE-F6). An orphan
# task firing against a deleted exe spams Task Scheduler errors every boot forever
# and, once the files (incl. the enrollment-complete marker) are gone, has no
# cleanup-retry path.
$taskName = '{scheduledTaskName}'
$taskGone = $false
for ($i = 1; $i -le 5; $i++) {{
try {{ Stop-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
try {{ Unregister-ScheduledTask -TaskName $taskName -Confirm:$false -ErrorAction SilentlyContinue }} catch {{ }}
$still = $null
try {{ $still = Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue }} catch {{ }}
if (-not $still) {{ $taskGone = $true; break }}
Start-Sleep -Seconds 1
}}
if (-not $taskGone) {{
# Fallback: schtasks.exe /Delete does not depend on the ScheduledTasks PowerShell module.
try {{ & schtasks.exe /Delete /TN $taskName /F 2>$null | Out-Null }} catch {{ }}
try {{ $taskGone = -not (Get-ScheduledTask -TaskName $taskName -ErrorAction SilentlyContinue) }} catch {{ }}
}}
if ($taskGone) {{
# Task gone + tree renamed ⇒ safe to delete. Never touches the original path.
foreach ($entry in $renamed) {{
Remove-Item -Path $entry.Renamed -Recurse -Force -ErrorAction SilentlyContinue
}}
}} else {{
# Task survived ⇒ rename back and leave everything for the next-boot retry
# (the marker survives, so bootstrap re-runs cleanup).
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
}} else {{
# Probe failed ⇒ something still holds the tree. Rename back whatever succeeded and
# abort WITHOUT deleting anything — task + marker stay, next boot retries.
foreach ($entry in $renamed) {{
try {{ Rename-Item -Path $entry.Renamed -NewName $entry.Original -Force -ErrorAction SilentlyContinue }} catch {{ }}
}}
}}
{(rebootOnComplete ? @"
# shutdown.exe (NOT Restart-Computer): Restart-Computer has no -Comment parameter and -Delay is
# only honoured with -Wait, so the previous invocation failed parameter binding and the device
# silently never rebooted. shutdown.exe /r /t 10 mirrors the standalone reboot path and reboots
# after a 10 s delay (lets this script finish + the agent exit). Review LIFE-F2.
shutdown.exe /r /t 10 /c 'Autopilot enrollment completed - Autopilot Monitor is rebooting'
" : "")}
}} finally {{
if ($lockStream) {{ $lockStream.Dispose() }}
Remove-Item -Path $lockPath -Force -ErrorAction SilentlyContinue
Remove-Item -Path $scriptPath -Force -ErrorAction SilentlyContinue
}}
";
}
}
}
@@ -370,7 +370,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
try
{
var versionUrl = $"{Constants.AgentBlobBaseUrl}/{Constants.AgentVersionFileNameForLine(2)}";
// Download alias, not the legacy blob account: the legacy mirror in the
// release pipeline is fail-soft, so it can silently lag behind eu.
var versionUrl = $"{Constants.AgentDownloadBaseUrl}/{Constants.AgentVersionFileNameForLine(2)}";
using (var handler = new HttpClientHandler())
using (var client = new HttpClient(handler) { Timeout = TimeSpan.FromMilliseconds(VersionCheckTimeoutMs) })
@@ -497,7 +499,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Runtime
{
try
{
var zipUrl = $"{Constants.AgentBlobBaseUrl}/{Constants.AgentZipFileNameForLine(2)}";
var zipUrl = $"{Constants.AgentDownloadBaseUrl}/{Constants.AgentZipFileNameForLine(2)}";
// Clean up any previous download (fragment from aborted run, or symlink/junction).
// Pre-delete + FileMode.CreateNew below: CreateNew refuses to open an existing
@@ -3,6 +3,7 @@ using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Management;
using System.Text.RegularExpressions;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Interop;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
@@ -28,6 +29,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
///
/// Emits a single "local_admin_analysis" event at startup and at shutdown,
/// enabling delta detection between pre- and post-enrollment state.
///
/// Tenant-supplied allowed accounts may contain glob wildcards (e.g. "adm-*");
/// see <see cref="MatchesAllowedEntry"/>.
/// </summary>
public class LocalAdminAnalyzer : IAgentAnalyzer
{
@@ -246,6 +250,43 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
}
}
// -----------------------------------------------------------------------
// Allowed-list matching
// -----------------------------------------------------------------------
/// <summary>
/// Matches an account or profile-folder name against one allowed-list entry.
/// Entries containing <c>*</c> or <c>?</c> are glob patterns (same semantics as the
/// hardware whitelist: <c>*</c> = any run, <c>?</c> = one char, case-insensitive);
/// all other entries match exactly. No ambiguity with literal names — <c>*</c> and
/// <c>?</c> are invalid characters in Windows account names.
/// Internal for direct unit-testing via InternalsVisibleTo.
/// </summary>
internal static bool MatchesAllowedEntry(string name, string entry)
{
if (entry.IndexOf('*') < 0 && entry.IndexOf('?') < 0)
return string.Equals(entry, name, StringComparison.OrdinalIgnoreCase);
var regexPattern = "^" + Regex.Escape(entry)
.Replace("\\*", ".*")
.Replace("\\?", ".") + "$";
try
{
// Timeout guards against pathological patterns (ReDoS)
return Regex.IsMatch(name, regexPattern, RegexOptions.IgnoreCase, TimeSpan.FromMilliseconds(100));
}
catch (RegexMatchTimeoutException)
{
return false;
}
}
private static bool IsAllowed(string name, List<string> allowedAccounts)
{
return allowedAccounts.Any(a => MatchesAllowedEntry(name, a));
}
// -----------------------------------------------------------------------
// Individual checks
// -----------------------------------------------------------------------
@@ -308,8 +349,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
if (disabled)
continue;
if (!allowedAccounts.Any(a =>
string.Equals(a, name, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(name, allowedAccounts))
{
unexpected.Add(name);
_logger.Debug($"{Name}: Unexpected local account: {name}");
@@ -349,8 +389,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Analyzers
allFound.Add(folderName);
if (!allowedAccounts.Any(a =>
string.Equals(a, folderName, StringComparison.OrdinalIgnoreCase)))
if (!IsAllowed(folderName, allowedAccounts))
{
unexpected.Add(folderName);
_logger.Debug($"{Name}: Unexpected profile folder: {folderName}");
@@ -45,10 +45,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Telemetry.Periodic
/// </summary>
private static readonly (string Name, string Url)[] MdmEndpoints =
{
("Entra ID", "https://login.microsoftonline.com"),
("Entra ID", Constants.EntraLoginBaseUrl),
("Device Registration", "https://enterpriseregistration.windows.net"),
("Intune Portal", "https://portal.manage.microsoft.com"),
("Microsoft Graph", "https://graph.microsoft.com"),
("Microsoft Graph", Constants.GraphBaseUrl),
};
public NetworkChangeDetector(
@@ -169,9 +169,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// <summary>
/// Sends a critical error report to the emergency channel endpoint.
/// Fire-and-forget: swallows all exceptions so a failure here never cascades.
/// Uses a 5-second per-request timeout so it cannot block the upload loop.
/// Default 5-second per-request timeout so it cannot block the upload loop; the
/// emergency-break path passes a longer <paramref name="timeout"/> because its first
/// request of a freshly booted process pays cold TLS + DNS + a possible backend
/// cold start (observed 503s take well over 5 s).
/// Returns <c>true</c> when the HTTP round-trip completed (the endpoint always
/// answers 200), <c>false</c> on timeout/transport failure — the caller-side
/// anti-flood layer uses this to decide whether the report counts as delivered.
/// Virtual for tests.
/// </summary>
public async Task ReportAgentErrorAsync(AgentErrorReport report)
public virtual async Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
try
{
@@ -190,16 +197,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
httpRequest.Headers.Add("X-Tenant-Id", report.TenantId);
AddSecurityHeaders(httpRequest);
using (var cts = new CancellationTokenSource(TimeSpan.FromSeconds(5)))
using (var cts = new CancellationTokenSource(timeout ?? TimeSpan.FromSeconds(5)))
using (var response = await _httpClient.SendAsync(httpRequest, cts.Token).ConfigureAwait(false))
{
// Response status is deliberately ignored — endpoint always returns 200.
return true;
}
}
}
catch (Exception ex)
{
_logger?.Debug($"ReportAgentErrorAsync: emergency channel failed: {ex.Message}");
return false;
}
}
@@ -57,19 +57,33 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// (intended to be called as fire-and-forget via <c>_ = reporter.TrySendAsync(...)</c>).
///
/// All anti-flood checks are applied synchronously before the async HTTP call,
/// so repeated calls are cheap when suppressed.
/// so repeated calls are cheap when suppressed. The anti-flood reservation is
/// taken up front (so concurrent same-key calls cannot double-send) but ROLLED
/// BACK when every attempt fails: the budget and the once-per-session dedup are
/// meant to count DELIVERED reports — the previous mark-before-send made the very
/// first transport failure permanently suppress that error key for the session
/// (found via the 48h emergency-break report, which is a single shot at boot).
///
/// <paramref name="attempts"/>/<paramref name="perAttemptTimeout"/>/<paramref name="retryDelay"/>
/// exist for callers whose process is about to exit and cannot come back later
/// (emergency break): retries stay INSIDE the one reservation, so they never
/// multiply against the per-session budget.
/// </summary>
public virtual async Task TrySendAsync(
AgentErrorType errorType,
string message,
int? httpStatusCode = null,
long? sequenceNumber = null)
long? sequenceNumber = null,
int attempts = 1,
TimeSpan? perAttemptTimeout = null,
TimeSpan? retryDelay = null)
{
// Deduplicate by error type + status code: same failure category sent only once per session.
// All anti-flood checks are protected by a lock to prevent race conditions when
// TrySendAsync is called concurrently (fire-and-forget from multiple threads).
var key = $"{errorType}:{httpStatusCode}";
int currentReportCount;
DateTime? previousLastReportTime;
lock (_antiFloodLock)
{
@@ -92,6 +106,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
return;
}
previousLastReportTime = _lastReportTime;
_reportedKeys.Add(key);
_reportCount++;
_lastReportTime = DateTime.UtcNow;
@@ -113,15 +128,40 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
Timestamp = DateTime.UtcNow
};
var delivered = false;
var totalAttempts = attempts < 1 ? 1 : attempts;
try
{
await _apiClient.ReportAgentErrorAsync(report);
for (var attempt = 1; attempt <= totalAttempts && !delivered; attempt++)
{
if (attempt > 1 && retryDelay.HasValue)
{
await Task.Delay(retryDelay.Value).ConfigureAwait(false);
}
delivered = await _apiClient.ReportAgentErrorAsync(report, perAttemptTimeout).ConfigureAwait(false);
if (!delivered && totalAttempts > 1)
{
_logger?.Debug($"[EmergencyChannel] Attempt {attempt}/{totalAttempts} failed: {key}");
}
}
}
catch (Exception ex)
{
// Swallow — emergency channel must never cascade failures into the caller
_logger?.Debug($"[EmergencyChannel] Failed to send report: {ex.Message}");
}
if (!delivered)
{
lock (_antiFloodLock)
{
_reportedKeys.Remove(key);
_reportCount--;
_lastReportTime = previousLastReportTime;
}
_logger?.Debug($"[EmergencyChannel] Report not delivered — reservation rolled back: {key}");
}
}
}
}
@@ -137,7 +137,24 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
private void OnPhaseChanged(object? sender, RealmJoinPhaseChangedEventArgs e)
{
// Phase change is observability-only: no DecisionSignalKind, just a timeline event.
// Dual-emission like OnDetected/OnResolved: (1) the typed signal persists the phase
// into RealmJoinFacts.LastDeploymentPhase (restart-safe) and lets the reducer detect
// the aborted-RJ-ESP shape (phase left 100/101 for 200/210 without 110 — session
// 224b2087); (2) the InformationalEvent keeps the existing timeline entry.
var payload = new Dictionary<string, string>(StringComparer.Ordinal)
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = e.CurrentPhase.ToString(),
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = e.PreviousPhase.ToString(),
};
_ingress.Post(
kind: DecisionSignalKind.RealmJoinPhaseChanged,
occurredAtUtc: _clock.UtcNow,
sourceOrigin: SourceLabel,
evidence: BuildEvidence(
"realmjoin-phase-changed-v1",
$"RealmJoin DeploymentPhase {e.PreviousPhase} -> {e.CurrentPhase}"),
payload: payload);
// Both raw int (deploymentPhase / previousPhase) and human-readable name
// (deploymentPhaseName / previousPhaseName) flow through — int stays for downstream
// filters / KQL, name fields make the message + UI rows readable without a lookup.
@@ -1,4 +1,6 @@
using System;
using System.Net.NetworkInformation;
using System.Threading;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Models;
@@ -15,28 +17,78 @@ namespace AutopilotMonitor.Agent.V2.Runtime
/// classifier terminalize the session instead of waiting out the grace. Fully swallowed — a send
/// failure (no network, no cert) must never delay the cleanup/exit the break exists to perform.
/// See tasks/enrollment-status-reclassification.md.
/// <para>
/// Delivery hardening (2026-07-23): this is the FIRST request of a freshly booted process and
/// the process never comes back — a miss is gone forever. The old single shot with a 5 s cap
/// died on boot-time realities (NIC still associating, cold TLS/DNS, backend cold starts
/// observed as &gt;5 s 503s). Now: wait briefly for the NIC, then up to
/// <see cref="SendAttempts"/> attempts with a per-attempt timeout that survives a Functions
/// cold start. The session is 48 h dead — spending up to ~75 s here is irrelevant, losing the
/// only signal that the agent gave up is not.
/// </para>
/// </summary>
internal static class EmergencyBreakReporter
{
private static readonly TimeSpan NetworkWaitMax = TimeSpan.FromSeconds(15);
private const int SendAttempts = 3;
private static readonly TimeSpan PerAttemptTimeout = TimeSpan.FromSeconds(15);
private static readonly TimeSpan RetryDelay = TimeSpan.FromSeconds(5);
// Upper bound for the blocking wait: NIC wait + 3×15s attempts + 2×5s delays + slack.
private static readonly TimeSpan OverallBudget = TimeSpan.FromSeconds(75);
public static void TrySend(AgentConfiguration agentConfig, string agentVersion, AgentLogger logger)
{
try
{
WaitForNetwork(logger);
var auth = BackendClientFactory.BuildAuthClients(agentConfig, agentVersion, logger);
var message =
$"Agent absolute session-age emergency break fired (cap {agentConfig.AbsoluteMaxSessionHours}h) — cleaning up and exiting.";
// The process is about to exit, so block briefly on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP errors;
// the auth bundle's HttpClients are intentionally left for process teardown.
// The process is about to exit, so block on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP
// errors and keeps the retries inside one anti-flood reservation; the auth
// bundle's HttpClients are intentionally left for process teardown.
auth.EmergencyReporter
.TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, message)
.Wait(TimeSpan.FromSeconds(5));
.TrySendAsync(
AgentErrorType.SessionAgeEmergencyBreak,
message,
attempts: SendAttempts,
perAttemptTimeout: PerAttemptTimeout,
retryDelay: RetryDelay)
.Wait(OverallBudget);
}
catch (Exception ex)
{
logger?.Debug($"Emergency-break report send failed (best-effort): {ex.Message}");
}
}
/// <summary>
/// Boot-time NIC grace: the break fires during bootstrap, often seconds after boot while
/// Wi-Fi is still associating. Polls the cheap link-level signal — it cannot prove backend
/// reachability (the send attempts handle that), it only avoids burning the first attempts
/// into a known-dead link. Best-effort: probe errors abort the wait, never the send.
/// </summary>
private static void WaitForNetwork(AgentLogger logger)
{
try
{
if (NetworkInterface.GetIsNetworkAvailable()) return;
logger?.Info($"Emergency-break report: no network link yet — waiting up to {NetworkWaitMax.TotalSeconds:F0}s.");
var deadline = DateTime.UtcNow + NetworkWaitMax;
while (DateTime.UtcNow < deadline)
{
Thread.Sleep(TimeSpan.FromSeconds(1));
if (NetworkInterface.GetIsNetworkAvailable()) return;
}
}
catch (Exception ex)
{
logger?.Debug($"Emergency-break report: network probe failed ({ex.Message}) — sending anyway.");
}
}
}
}
@@ -241,6 +241,199 @@ namespace AutopilotMonitor.DecisionCore.Tests
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
}
// ============================================================== Aborted first deployment (session 224b2087)
[Fact]
public void PhaseChanged_leaving_first_deployment_for_200_releases_gate_with_warning()
{
// Session 224b2087: interactive logon 17 s after phase 101 → RJ reclassifies the run
// as secondary-user deployment and writes 200/210, never 110. The 101 -> 200
// transition is impossible in a healthy first deployment, so it releases the gate
// with Outcome=FirstDeploymentIncomplete and a Warning timeline entry.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
Assert.NotEqual(SessionStage.Finalizing, state.Stage);
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(10),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "101",
}));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("FirstDeploymentIncomplete", step.NewState.RealmJoinFacts.Outcome!.Value);
Assert.NotNull(step.NewState.RealmJoinFacts.ResolvedUtc);
Assert.Equal(200, step.NewState.RealmJoinFacts.LastDeploymentPhase!.Value);
// Hard-timeout deadline cancelled in state AND as a scheduler effect.
Assert.DoesNotContain(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.CancelDeadline &&
e.CancelDeadlineName == DeadlineNames.RealmJoinTimeout);
// Warning timeline entry names both phases.
var warning = Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_first_deployment_incomplete");
Assert.Equal("Warning", warning.Parameters!["severity"]);
Assert.Equal("101", warning.Parameters["previousPhase"]);
Assert.Equal("200", warning.Parameters["deploymentPhase"]);
Assert.Contains("110", warning.Parameters["message"]);
}
[Fact]
public void PhaseChanged_within_first_deployment_window_is_bookkeeping_only()
{
// 100 -> 101 stays inside the first-deployment window — no release, but the phase
// fact must advance (it is the restart-safe witness for the abort rule).
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "100" })).NewState;
var step = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "100",
}));
Assert.Equal(101, step.NewState.RealmJoinFacts.LastDeploymentPhase!.Value);
Assert.Null(step.NewState.RealmJoinFacts.Outcome);
Assert.Contains(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
Assert.Empty(step.Effects);
}
[Fact]
public void PhaseChanged_from_completed_deployment_start_does_not_release()
{
// Session 6f1959c0 protection: RJ already stood at CompletedDeployment (210) when
// the agent booted and only deployed AFTERWARDS (210 -> 200 -> 210). Neither
// transition starts from the first-deployment window, so the abort rule must not
// fire — this ambiguous shape keeps waiting for 110 / the hard timeout (status quo).
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "210" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "210",
})).NewState;
Assert.Null(state.RealmJoinFacts.Outcome);
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(7),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "210",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "200",
}));
Assert.Null(step.NewState.RealmJoinFacts.Outcome);
Assert.Contains(step.NewState.Deadlines, d => d.Name == DeadlineNames.RealmJoinTimeout);
}
[Fact]
public void Detected_replay_after_restart_with_deployment_phase_releases_gate()
{
// Agent restarted between the 101 and 200 observations: the watcher re-fires
// Detected with the CURRENT phase (200) and no PhaseChanged ever carries the
// transition. The persisted LastDeploymentPhase=101 is the witness — the Detected
// replay path must release the gate exactly like the PhaseChanged path.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.DesktopArrived, T0.AddMinutes(6))).NewState;
Assert.NotEqual(SessionStage.Finalizing, state.Stage);
Assert.Equal(101, state.RealmJoinFacts.LastDeploymentPhase!.Value);
var step = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(12),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200" }));
Assert.Equal(SessionStage.Finalizing, step.NewState.Stage);
Assert.Equal("FirstDeploymentIncomplete", step.NewState.RealmJoinFacts.Outcome!.Value);
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_first_deployment_incomplete");
}
[Fact]
public void Timeout_message_reports_persisted_last_phase_instead_of_zero()
{
// Before RealmJoinPhaseChanged became a typed signal, LastDeploymentPhase was only
// written at detection time — every realmjoin_timeout claimed "last phase: 0" no
// matter how far RJ actually got, hiding this whole failure class from ops.
var engine = new DecisionEngine();
var state = PrimeClassicAwaitingDesktop(engine);
state = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(5),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "100" })).NewState;
state = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(6),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "100",
})).NewState;
state = engine.Reduce(state, MakeSignal(7, DecisionSignalKind.DesktopArrived, T0.AddMinutes(7))).NewState;
var step = engine.Reduce(state, MakeSignal(8, DecisionSignalKind.DeadlineFired, T0.AddMinutes(65),
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.RealmJoinTimeout }));
var timeout = Assert.Single(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_timeout");
Assert.Contains("last phase: 101", timeout.Parameters!["message"]);
Assert.Equal("101", timeout.Parameters["lastSeenPhase"]);
}
[Fact]
public void SelfDeploying_deferred_terminal_released_by_first_deployment_incomplete()
{
// Same deferred-release contract as RealmJoinResolved: when the SelfDeploying
// terminal was deferred on the closed RJ gate, the aborted-first-deployment release
// must complete the session directly (Completed + enrollment_complete), with the
// Warning entry leading the effects.
var engine = new DecisionEngine();
var state = DecisionState.CreateInitial("rj-sd-3", "rj-tenant", T0);
state = engine.Reduce(state, MakeSignal(0, DecisionSignalKind.SessionStarted, T0)).NewState;
state = engine.Reduce(state, MakeSignal(1, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(1),
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = "DeviceSetup" })).NewState;
state = engine.Reduce(state, MakeSignal(2, DecisionSignalKind.RealmJoinDetected, T0.AddMinutes(2),
new Dictionary<string, string> { [DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "101" })).NewState;
state = engine.Reduce(state, MakeSignal(3, DecisionSignalKind.DeviceSetupProvisioningComplete, T0.AddMinutes(3))).NewState;
state = engine.Reduce(state, MakeSignal(4, DecisionSignalKind.DeadlineFired, T0.AddMinutes(8),
new Dictionary<string, string> { [SignalPayloadKeys.Deadline] = DeadlineNames.DeviceOnlyEspDetection })).NewState;
Assert.True(state.RealmJoinFacts.SelfDeployingDeferredCompletion?.Value);
var step = engine.Reduce(state, MakeSignal(5, DecisionSignalKind.RealmJoinPhaseChanged, T0.AddMinutes(10),
new Dictionary<string, string>
{
[DecisionEngine.RealmJoinPayloadKeys.DeploymentPhase] = "200",
[DecisionEngine.RealmJoinPayloadKeys.PreviousPhase] = "101",
}));
Assert.Equal(SessionStage.Completed, step.NewState.Stage);
Assert.Equal(SessionOutcome.EnrollmentComplete, step.NewState.Outcome);
Assert.Equal("FirstDeploymentIncomplete", step.NewState.RealmJoinFacts.Outcome!.Value);
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "realmjoin_first_deployment_incomplete");
Assert.Contains(step.Effects, e =>
e.Kind == DecisionEffectKind.EmitEventTimelineEntry &&
e.Parameters != null &&
e.Parameters.TryGetValue("eventType", out var et) && et == "enrollment_complete");
}
// ============================================================== SelfDeploying flow
[Fact]
@@ -55,7 +55,7 @@ public class AuthFunctionSideEffectTests
{ CallBase = false };
var previewMock = new Mock<PreviewWhitelistService>(
configRepo, cache, Mock.Of<ILogger<PreviewWhitelistService>>())
configRepo, cache, Mock.Of<ILogger<PreviewWhitelistService>>(), _tenantConfigMock.Object)
{ CallBase = false };
_telegramMock = new Mock<TelegramNotificationService>(
@@ -113,9 +113,11 @@ public class EnrollmentTimeoutClassifierTests
// Negative Hello terminals leave the agent waiting — must NOT count as resolved.
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("hello_provisioning_failed") }).HelloResolved);
Assert.False(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("hello_completion_timeout") }).HelloResolved);
// Both RealmJoin gate terminals count as resolved (phase 110 or 60-min hard timeout).
// All RealmJoin gate terminals count as resolved (phase 110, aborted first
// deployment — session 224b2087 — or 60-min hard timeout).
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("realmjoin_resolved") }).RealmJoinResolved);
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("realmjoin_timeout") }).RealmJoinResolved);
Assert.True(EnrollmentTimeoutClassifier.ExtractRollup(new[] { Evt("realmjoin_first_deployment_incomplete") }).RealmJoinResolved);
}
[Fact]
@@ -0,0 +1,101 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Enforces the URL registry: every well-known own or Microsoft host must be
/// referenced through <c>AutopilotMonitor.Shared.Constants</c>, never as a
/// repeated string literal. The EU cutover missed two hardcoded copies of the
/// blob host (LatestVersionsService, ValidateBootstrapCodeFunction) precisely
/// because they did not go through the registry — this test makes that class
/// of drift a build failure instead of a production surprise.
///
/// Scope: non-test C# source under src/. Comment lines are ignored (docs and
/// examples may cite URLs); test projects are excluded because tests use
/// literals deliberately as independent oracles.
/// </summary>
public class HardcodedUrlGuardTests
{
private static readonly string[] EnforcedHosts =
{
"portal.autopilotmonitor.com",
"www.autopilotmonitor.com",
"docs.autopilotmonitor.com",
"download.autopilotmonitor.com",
"mcp.autopilotmonitor.com",
"autopilotmonitor-api-eu.azurewebsites.net",
"autopilotmonitor.blob.core.windows.net",
"autopilotmonitoreu.blob.core.windows.net",
"graph.microsoft.com",
"login.microsoftonline.com",
};
/// <summary>Files allowed to carry the literals — the registry itself.</summary>
private static readonly string[] RegistryFiles =
{
Path.Combine("src", "Shared", "AutopilotMonitor.Shared", "Constants.cs"),
};
[Fact]
public void WellKnownHosts_OnlyAppearInTheRegistry()
{
var repoRoot = FindRepoRoot();
var srcDir = Path.Combine(repoRoot, "src");
var violations = new List<string>();
foreach (var file in Directory.EnumerateFiles(srcDir, "*.cs", SearchOption.AllDirectories))
{
var relative = Path.GetRelativePath(repoRoot, file);
var normalized = relative.Replace('/', Path.DirectorySeparatorChar);
// bin/obj artifacts and generated files
if (normalized.Contains($"{Path.DirectorySeparatorChar}bin{Path.DirectorySeparatorChar}") ||
normalized.Contains($"{Path.DirectorySeparatorChar}obj{Path.DirectorySeparatorChar}"))
continue;
// Test projects: literals there are deliberate independent oracles.
if (normalized.Contains("Tests"))
continue;
if (RegistryFiles.Any(r => normalized.EndsWith(r, StringComparison.OrdinalIgnoreCase)))
continue;
var lines = File.ReadAllLines(file);
for (var i = 0; i < lines.Length; i++)
{
var trimmed = lines[i].TrimStart();
// Comment approximation: full-line // and /// comments plus the
// body lines of XML docs and block comments. A URL after code on
// the same line still counts — that is the case being guarded.
if (trimmed.StartsWith("//") || trimmed.StartsWith("*") || trimmed.StartsWith("/*"))
continue;
foreach (var host in EnforcedHosts)
{
if (trimmed.Contains(host, StringComparison.OrdinalIgnoreCase))
violations.Add($"{relative}:{i + 1}: {host}");
}
}
}
Assert.True(violations.Count == 0,
"Hardcoded well-known host(s) found outside the URL registry " +
"(use AutopilotMonitor.Shared.Constants.*BaseUrl instead):\n " +
string.Join("\n ", violations));
}
private static string FindRepoRoot()
{
var dir = new DirectoryInfo(AppContext.BaseDirectory);
while (dir != null && !File.Exists(Path.Combine(dir.FullName, "AutopilotMonitor.sln")))
{
dir = dir.Parent;
}
Assert.NotNull(dir);
return dir!.FullName;
}
}
@@ -0,0 +1,72 @@
using AutopilotMonitor.Functions.Helpers;
namespace AutopilotMonitor.Functions.Tests;
public class LogSanitizerTests
{
[Fact]
public void Clean_WithCrlf_StripsNewlines()
{
// The core log-forging vector: a value that injects a fake log line.
var forged = "/api/sessions\r\n[Auth Middleware] FAKE: admin granted";
var cleaned = LogSanitizer.Clean(forged);
Assert.Equal("/api/sessions[Auth Middleware] FAKE: admin granted", cleaned);
Assert.DoesNotContain('\r', cleaned!);
Assert.DoesNotContain('\n', cleaned!);
}
[Fact]
public void Clean_WithBareLineFeedOrCarriageReturn_StripsBoth()
{
Assert.Equal("ab", LogSanitizer.Clean("a\nb"));
Assert.Equal("ab", LogSanitizer.Clean("a\rb"));
}
[Fact]
public void Clean_WithOtherControlChars_StripsThem()
{
// Tab (9), NUL (0) and ESC (27, terminal escape) are all removed. Built
// from char codes so the source file carries no literal control bytes.
var input = $"a{(char)9}b{(char)0}c{(char)27}de";
Assert.Equal("abcde", LogSanitizer.Clean(input));
}
[Fact]
public void Clean_WithUnicodeLineSeparators_StripsThem()
{
// U+2028 LINE SEPARATOR and U+2029 PARAGRAPH SEPARATOR are Unicode categories
// Zl/Zp, not Cc — so a \p{Cc}-only pass would let them through, and a log viewer
// or JavaScript-side reader would still break the line on them. Built from char
// codes so the source file carries no literal separator bytes.
var forged = $"/api/sessions{(char)0x2028}[Auth Middleware] FAKE: admin granted";
var cleaned = LogSanitizer.Clean(forged);
Assert.Equal("/api/sessions[Auth Middleware] FAKE: admin granted", cleaned);
Assert.DoesNotContain((char)0x2028, cleaned!);
Assert.Equal("ab", LogSanitizer.Clean($"a{(char)0x2029}b"));
}
[Theory]
[InlineData("/api/sessions/abc-123/events")]
[InlineData("user@contoso.com")]
[InlineData("11111111-2222-3333-4444-555555555555")]
[InlineData("GET")]
public void Clean_WithLegitimateValue_ReturnsUnchanged(string value)
{
// Real paths, UPNs, GUIDs and methods never contain control chars, so
// tracing fidelity is fully preserved.
Assert.Equal(value, LogSanitizer.Clean(value));
}
[Fact]
public void Clean_WithNull_ReturnsNull()
{
Assert.Null(LogSanitizer.Clean(null));
}
[Fact]
public void Clean_WithEmpty_ReturnsEmpty()
{
Assert.Equal("", LogSanitizer.Clean(""));
}
}
@@ -1,4 +1,5 @@
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.Models;
namespace AutopilotMonitor.Functions.Tests;
@@ -87,4 +88,61 @@ public class MaintenanceServiceExcessiveEventAutoActionTests
// — better to no-op than block/kill on an unintended config.
Assert.Null(MaintenanceService.DecideAutoAction(eventCount: 5000, autoMode: mode, autoThreshold: 2500));
}
// ── Status eligibility ────────────────────────────────────────────────────
// A device may only be blocked over a session that can still upload. Without this gate
// a session that finished days ago with a high EventCount got its device blocked on the
// next sweep, long after it stopped sending.
[Fact]
public void IsAutoActionEligible_InProgress_IsEligible()
{
Assert.True(MaintenanceService.IsAutoActionEligible(SessionStatus.InProgress));
}
[Theory]
[InlineData(SessionStatus.Succeeded)]
[InlineData(SessionStatus.Failed)]
[InlineData(SessionStatus.Incomplete)]
[InlineData(SessionStatus.Stalled)]
[InlineData(SessionStatus.AwaitingUser)]
[InlineData(SessionStatus.Pending)]
[InlineData(SessionStatus.Unknown)]
public void IsAutoActionEligible_EverythingElse_IsRejected(SessionStatus status)
{
// Matches the time-window watchdog's `Status eq 'InProgress'` filter, so neither
// detector can block a device over a session that no longer sends. Stalled and
// AwaitingUser can heal back to InProgress — then the next sweep acts.
Assert.False(MaintenanceService.IsAutoActionEligible(status));
}
// ── Status parsing feeding that gate ──────────────────────────────────────
// The runaway query projects a subset of columns. SessionStatus.InProgress is ordinal 0,
// so an unparsed Status would leave every row looking in-progress and the gate above
// silently always-true — these pin the fail-closed mapping.
[Theory]
[InlineData("InProgress", SessionStatus.InProgress)]
[InlineData("inprogress", SessionStatus.InProgress)]
[InlineData("Succeeded", SessionStatus.Succeeded)]
[InlineData("AwaitingUser", SessionStatus.AwaitingUser)]
public void ParseStatusForAutoAction_KnownNames_RoundTrip(string stored, SessionStatus expected)
{
Assert.Equal(expected, TableStorageService.ParseStatusForAutoAction(stored));
}
[Theory]
[InlineData(null)] // column absent from the projection or the row
[InlineData("")]
[InlineData(" ")]
[InlineData("Enrolling")] // status name that never existed
[InlineData("0")] // Enum.TryParse would map this onto InProgress and open the gate
[InlineData("3")]
[InlineData("-1")]
[InlineData("InProgress,Failed")] // flags-style combination is not a real status
public void ParseStatusForAutoAction_UnusableInput_IsUnknown(string? stored)
{
Assert.Equal(SessionStatus.Unknown, TableStorageService.ParseStatusForAutoAction(stored));
Assert.False(MaintenanceService.IsAutoActionEligible(TableStorageService.ParseStatusForAutoAction(stored)));
}
}
@@ -117,7 +117,6 @@ public class OffboardingSessionEnumeratorTests
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetSessionsLeanAsync(string tenantId) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff) => throw new NotImplementedException();
public Task<List<SessionSummary>> GetExcessiveDataSendersAsync(string tenantId, DateTime windowCutoff, int maxSessionWindowHours) => throw new NotImplementedException();
public Task<List<string>> GetAllTenantIdsAsync() => throw new NotImplementedException();
public Task<int> DeleteSessionEventsAsync(string tenantId, string sessionId) => throw new NotImplementedException();
public Task<int> DeleteSessionRuleResultsAsync(string tenantId, string sessionId) => throw new NotImplementedException();
@@ -868,4 +868,149 @@ public class PolicyEnforcementMiddlewareTests
Assert.False(result.Allowed);
Assert.Equal(403, result.StatusCode);
}
// ── Empty-UPN contract for device / anonymous routes ───────────────────────────
//
// UserRateLimitMiddleware buckets on RequestContext.UserPrincipalName and skips only when it is
// EMPTY. Publishing the "anonymous" placeholder there collapsed every agent of every tenant into a
// single shared user_ratelimit_anonymous bucket, which 429'd agent telemetry fleet-wide once the
// per-user limit was reached. These tests pin the contract: no JWT ⇒ empty UPN.
[Theory]
[InlineData("POST", "/api/agent/telemetry")]
[InlineData("GET", "/api/agent/config")]
[InlineData("POST", "/api/agent/register-session")]
[InlineData("POST", "/api/agent/upload-url")]
[InlineData("POST", "/api/agent/error")]
[InlineData("POST", "/api/bootstrap/register-session")]
[InlineData("GET", "/api/bootstrap/config")]
[InlineData("POST", "/api/bootstrap/error")]
public async Task DeviceRoute_WithoutJwt_PublishesEmptyUpn(string method, string path)
{
var h = BuildHarness();
var result = await h.Middleware.DecideAsync(method, path, null, principal: null);
Assert.True(result.Allowed);
// The rate-limit bucket key: empty means "no user bucket applies to this call".
Assert.Equal(string.Empty, result.Context!.UserPrincipalName);
// The placeholder still renders in log lines — it just never reaches the RequestContext.
Assert.Equal("anonymous", result.UserIdentifier);
}
[Theory]
[InlineData("GET", "/api/health")]
[InlineData("GET", "/api/stats/platform")]
[InlineData("POST", "/api/agent/distress")]
[InlineData("GET", "/api/diagnostics/download")]
public async Task AnonymousRoute_WithoutJwt_PublishesEmptyUpn(string method, string path)
{
var h = BuildHarness();
var result = await h.Middleware.DecideAsync(method, path, null, principal: null);
Assert.True(result.Allowed);
Assert.Equal(string.Empty, result.Context!.UserPrincipalName);
}
[Fact]
public async Task AuthenticatedRoute_StillPublishesTheRealUpn()
{
// Guard the other direction: the fix must not blank the UPN for genuine JWT callers, or
// per-user rate limiting and presence tracking would silently stop working.
const string upn = "admin@contoso.com";
var h = BuildHarness();
h.AsTenantAdmin(TenantA, upn);
var result = await h.Middleware.DecideAsync("GET", "/api/notifications", null, AuthedPrincipal(TenantA, upn));
Assert.True(result.Allowed);
Assert.Equal(upn, result.Context!.UserPrincipalName);
Assert.Equal(upn, result.Context!.CallerId);
}
// ── CallerId: throttle identity survives tokens that carry no UPN ──────────────
//
// AuthenticationMiddleware validates issuer/audience/lifetime/signature but requires NO upn and no
// scp, so an app-only (client-credentials) token authenticates and reaches every AuthenticatedUser
// route (feedback, realtime/negotiate, progress/*, health/detailed, …). UserRateLimitMiddleware
// skips on an EMPTY throttle identity — so if CallerId fell back to the (absent) UPN, those tokens
// would bypass rate limiting entirely. These tests pin the fallback chain.
/// <summary>Principal with a tid but NO upn/preferred_username — shaped like an app-only token.</summary>
private static ClaimsPrincipal AppOnlyPrincipal(string tenantId, params (string Type, string Value)[] claims)
{
var all = new List<Claim> { new("tid", tenantId) };
all.AddRange(claims.Select(c => new Claim(c.Type, c.Value)));
return new ClaimsPrincipal(new ClaimsIdentity(all, "TestAuth"));
}
[Theory]
[InlineData("oid", "aaaaaaaa-0000-0000-0000-000000000001")]
[InlineData("http://schemas.microsoft.com/identity/claims/objectidentifier", "aaaaaaaa-0000-0000-0000-000000000002")]
[InlineData("sub", "subject-abc")]
[InlineData("appid", "bbbbbbbb-0000-0000-0000-000000000003")]
[InlineData("azp", "cccccccc-0000-0000-0000-000000000004")]
public async Task AuthenticatedUserRoute_TokenWithoutUpn_StillGetsAThrottleIdentity(string claimType, string claimValue)
{
var h = BuildHarness();
// /api/feedback/status is AuthenticatedUser — admits ANY valid principal, role unresolved.
var result = await h.Middleware.DecideAsync(
"GET", "/api/feedback/status", null, AppOnlyPrincipal(TenantA, (claimType, claimValue)));
Assert.True(result.Allowed);
// No UPN to report — audit/presence correctly see nothing...
Assert.Equal(string.Empty, result.Context!.UserPrincipalName);
// ...but the throttle still has something stable to bucket on.
Assert.Equal(claimValue, result.Context!.CallerId);
}
[Fact]
public async Task AuthenticatedUserRoute_TokenWithoutUpn_PrefersOidOverSubAndAppId()
{
var h = BuildHarness();
var principal = AppOnlyPrincipal(TenantA,
("appid", "cccccccc-0000-0000-0000-000000000009"),
("sub", "subject-xyz"),
("oid", "aaaaaaaa-0000-0000-0000-00000000000a"));
var result = await h.Middleware.DecideAsync("GET", "/api/feedback/status", null, principal);
Assert.True(result.Allowed);
Assert.Equal("aaaaaaaa-0000-0000-0000-00000000000a", result.Context!.CallerId);
}
[Fact]
public async Task AuthenticatedUserRoute_TokenWithNoIdentifyingClaim_FailsClosedToASharedBucket()
{
// Pathological (every real Entra token has a sub): must NOT resolve to empty, because empty
// means "skip the limiter". A shared bucket is the safe answer.
var h = BuildHarness();
var result = await h.Middleware.DecideAsync(
"GET", "/api/feedback/status", null, AppOnlyPrincipal(TenantA));
Assert.True(result.Allowed);
Assert.NotEqual(string.Empty, result.Context!.CallerId);
Assert.Equal(PolicyEnforcementMiddleware.UnidentifiedCallerBucket, result.Context!.CallerId);
// Never reuse the log placeholder as a bucket key — that was the original fleet-wide collapse.
Assert.NotEqual("anonymous", result.Context!.CallerId);
}
[Theory]
[InlineData("POST", "/api/agent/telemetry")]
[InlineData("GET", "/api/health")]
[InlineData("GET", "/api/stats/platform")]
public async Task UnauthenticatedRoute_HasEmptyCallerId(string method, string path)
{
// Empty CallerId is the ONLY signal that skips the user limiter — it must mean
// "no JWT at all", never "authenticated but unnamed".
var h = BuildHarness();
var result = await h.Middleware.DecideAsync(method, path, null, principal: null);
Assert.True(result.Allowed);
Assert.Equal(string.Empty, result.Context!.CallerId);
}
}
@@ -1,7 +1,15 @@
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Functions.Ingest;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Functions.Services.Notifications;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
@@ -80,4 +88,116 @@ public class ReportAgentErrorFunctionTests
var evt = ReportAgentErrorFunction.BuildAgentEmergencyBreakEvent(Report(Now, message: ""), TenantId, null!, Now);
Assert.Contains("emergency break", evt.Message, StringComparison.OrdinalIgnoreCase);
}
// =========================================================================
// MaterializeEmergencyBreakArtifactsAsync — the artifact set per break report
// (2026-07-23 hardening): timeline event + cross-session EventType index row +
// AgentEmergencyBreak ops event, once per session. The index upsert is load-bearing:
// StoreEventsBatchAsync alone leaves the event invisible to every search-by-eventType
// surface (found in the 2026-07-22 incident analysis).
// =========================================================================
private sealed class Harness
{
public readonly Mock<ISessionRepository> SessionRepo = new();
public readonly List<OpsEventEntry> OpsEvents = new();
public readonly OpsEventService OpsService;
public Harness(List<EnrollmentEvent>? existingEvents = null)
{
SessionRepo.Setup(r => r.GetSessionEventsAsync(TenantId, SessionId, It.IsAny<int>()))
.ReturnsAsync(existingEvents ?? new List<EnrollmentEvent>());
SessionRepo.Setup(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()))
.ReturnsAsync((List<EnrollmentEvent> e) => e);
SessionRepo.Setup(r => r.UpsertEventTypeIndexBatchAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()))
.Returns(Task.CompletedTask);
var opsRepo = new Mock<IOpsEventRepository>();
opsRepo.Setup(r => r.SaveOpsEventAsync(It.IsAny<OpsEventEntry>()))
.Callback<OpsEventEntry>(e => { lock (OpsEvents) OpsEvents.Add(e); })
.Returns(Task.CompletedTask);
var adminConfig = new Mock<AdminConfigurationService>(
Mock.Of<IConfigRepository>(), NullLogger<AdminConfigurationService>.Instance,
new MemoryCache(new MemoryCacheOptions()));
var alertDispatch = new OpsAlertDispatchService(
adminConfig.Object,
new TelegramNotificationService(new HttpClient(), Mock.Of<IConfigRepository>(),
NullLogger<TelegramNotificationService>.Instance),
new WebhookNotificationService(new HttpClient(),
NullLogger<WebhookNotificationService>.Instance),
NullLogger<OpsAlertDispatchService>.Instance);
OpsService = new OpsEventService(opsRepo.Object, NullLogger<OpsEventService>.Instance, alertDispatch);
}
public Task RunAsync(AgentErrorReport report)
=> ReportAgentErrorFunction.MaterializeEmergencyBreakArtifactsAsync(
report, TenantId, SessionRepo.Object, OpsService, NullLogger<ReportAgentErrorFunction>.Instance);
}
[Fact]
public async Task First_break_report_materializes_event_index_row_and_ops_event()
{
var h = new Harness();
await h.RunAsync(Report(Now));
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(
It.Is<List<EnrollmentEvent>>(e => e.Count == 1
&& e[0].EventType == AutopilotMonitor.Shared.Constants.EventTypes.AgentEmergencyBreak)), Times.Once);
h.SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
TenantId, SessionId, It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Once);
var ops = Assert.Single(h.OpsEvents);
Assert.Equal("AgentEmergencyBreak", ops.EventType);
Assert.Equal(TenantId, ops.TenantId);
}
[Fact]
public async Task Already_materialized_session_produces_no_second_artifact_set()
{
var h = new Harness(new List<EnrollmentEvent>
{
new() { EventType = AutopilotMonitor.Shared.Constants.EventTypes.AgentEmergencyBreak, Sequence = 45 },
});
await h.RunAsync(Report(Now));
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()), Times.Never);
h.SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Never);
Assert.Empty(h.OpsEvents);
}
[Fact]
public async Task Non_break_error_types_and_missing_session_id_are_ignored()
{
var h = new Harness();
var other = Report(Now);
other.ErrorType = AgentErrorType.ConfigFetchFailed;
await h.RunAsync(other);
var noSession = Report(Now);
noSession.SessionId = null!;
await h.RunAsync(noSession);
h.SessionRepo.Verify(r => r.GetSessionEventsAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<int>()), Times.Never);
Assert.Empty(h.OpsEvents);
}
[Fact]
public async Task Storage_failure_is_swallowed_and_never_throws()
{
var h = new Harness();
h.SessionRepo.Setup(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()))
.ThrowsAsync(new InvalidOperationException("storage down"));
// Must not throw — the always-200 emergency channel must never turn into a retry loop.
await h.RunAsync(Report(Now));
Assert.Empty(h.OpsEvents);
}
}
@@ -78,15 +78,16 @@ public class SessionDeletionMaintenanceFunctionTests
[Fact]
public async Task RunCore_emits_LongRunning_when_body_runs_past_warning_threshold()
{
// Set watchdog thresholds to sub-second; make the fanout sleep long enough to trip warning
// but not severe.
var harness = new Harness(watchdogWarn: TimeSpan.FromMilliseconds(50), watchdogSevere: TimeSpan.FromMilliseconds(500));
// Deterministic (no wall-clock race): the fanout body blocks until the WARN watchdog has
// actually recorded its OpsEvent, then returns. Severe sits minutes out, so it can never
// fire in the moment between warn firing and the body completing (which cancels it).
var harness = new Harness(watchdogWarn: TimeSpan.FromMilliseconds(20), watchdogSevere: TimeSpan.FromMinutes(10));
harness.SetKillSwitch(false);
harness.Fanout.Setup(f => f.RunAsync(It.IsAny<SessionRetentionFanoutService.FanoutResult>(), It.IsAny<DateTime>(), It.IsAny<CancellationToken>()))
.Returns(async (SessionRetentionFanoutService.FanoutResult r, DateTime _, CancellationToken ct) =>
{
r.TenantsProcessed = 1;
await Task.Delay(150, ct);
await harness.WaitForEventAsync("SessionDeletionMaintenanceLongRunning");
});
await harness.Sut.RunCoreAsync("Test", CancellationToken.None);
@@ -102,14 +103,17 @@ public class SessionDeletionMaintenanceFunctionTests
// assigned only AFTER the fanout returned, so every LongRunning warning showed zeros.
// The fanout now mutates the caller-supplied result incrementally; the watchdog snapshot
// must reflect that progress mid-run.
var harness = new Harness(watchdogWarn: TimeSpan.FromMilliseconds(50), watchdogSevere: TimeSpan.FromSeconds(5));
// Deterministic: counters are set at fanout entry, then the body blocks until the WARN
// watchdog fires and snapshots them. Severe sits minutes out and never fires — no
// body-sleep-vs-threshold race.
var harness = new Harness(watchdogWarn: TimeSpan.FromMilliseconds(50), watchdogSevere: TimeSpan.FromMinutes(10));
harness.SetKillSwitch(false);
harness.Fanout.Setup(f => f.RunAsync(It.IsAny<SessionRetentionFanoutService.FanoutResult>(), It.IsAny<DateTime>(), It.IsAny<CancellationToken>()))
.Returns(async (SessionRetentionFanoutService.FanoutResult r, DateTime _, CancellationToken ct) =>
{
r.TenantsProcessed = 3;
r.SessionsEnqueued = 7;
await Task.Delay(200, ct); // long enough for the 50ms watchdog to fire mid-run
await harness.WaitForEventAsync("SessionDeletionMaintenanceLongRunning");
});
await harness.Sut.RunCoreAsync("Test", CancellationToken.None);
@@ -122,17 +126,20 @@ public class SessionDeletionMaintenanceFunctionTests
[Fact]
public async Task RunCore_emits_LongRunningSevere_when_body_runs_past_severe_threshold()
{
var harness = new Harness(watchdogWarn: TimeSpan.FromMilliseconds(50), watchdogSevere: TimeSpan.FromMilliseconds(150));
// Deterministic: the body blocks until the SEVERE watchdog records its OpsEvent. Warn has a
// smaller threshold, so it is guaranteed to have fired first — neither depends on a body
// sleep outlasting a fixed wall-clock threshold.
var harness = new Harness(watchdogWarn: TimeSpan.FromMilliseconds(20), watchdogSevere: TimeSpan.FromMilliseconds(60));
harness.SetKillSwitch(false);
harness.Fanout.Setup(f => f.RunAsync(It.IsAny<SessionRetentionFanoutService.FanoutResult>(), It.IsAny<DateTime>(), It.IsAny<CancellationToken>()))
.Returns(async (SessionRetentionFanoutService.FanoutResult _, DateTime _, CancellationToken ct) =>
{
await Task.Delay(300, ct);
await harness.WaitForEventAsync("SessionDeletionMaintenanceLongRunningSevere");
});
await harness.Sut.RunCoreAsync("Test", CancellationToken.None);
// Both watchdogs fire: warning at 50ms, severe at 150ms.
// Both watchdogs fire: warning first (smaller threshold), then severe (which the body waited for).
Assert.Contains(harness.OpsEvents, e => e.EventName == "SessionDeletionMaintenanceLongRunning");
Assert.Contains(harness.OpsEvents, e => e.EventName == "SessionDeletionMaintenanceLongRunningSevere");
}
@@ -345,6 +352,12 @@ public class SessionDeletionMaintenanceFunctionTests
private readonly Dictionary<string, List<TableEntity>> _sessionsByState = new(StringComparer.Ordinal);
private readonly List<DeletionManifestBlobSummary> _ttlBlobs = new();
// Watchdog OpsEvents are recorded from concurrent watchdog tasks; _opsSync serializes the
// OpsEvents append + waiter signalling, and _eventWaiters lets a test's fanout body block
// until a specific watchdog OpsEvent has actually fired (see WaitForEventAsync).
private readonly object _opsSync = new();
private readonly Dictionary<string, TaskCompletionSource> _eventWaiters = new(StringComparer.Ordinal);
public Harness(TimeSpan? watchdogWarn = null, TimeSpan? watchdogSevere = null)
{
StorageMock = new Mock<TableStorageService>(Mock.Of<TableServiceClient>(), NullLogger<TableStorageService>.Instance);
@@ -376,7 +389,12 @@ public class SessionDeletionMaintenanceFunctionTests
opsRepo.Setup(r => r.SaveOpsEventAsync(It.IsAny<OpsEventEntry>()))
.Returns<OpsEventEntry>(e =>
{
OpsEvents.Add(new CapturedOpsEvent(e.EventType, e.Severity, e.Message, e.Details ?? string.Empty));
lock (_opsSync)
{
OpsEvents.Add(new CapturedOpsEvent(e.EventType, e.Severity, e.Message, e.Details ?? string.Empty));
if (_eventWaiters.TryGetValue(e.EventType, out var waiter))
waiter.TrySetResult();
}
return Task.CompletedTask;
});
var alertDispatch = new OpsAlertDispatchService(
@@ -426,6 +444,27 @@ public class SessionDeletionMaintenanceFunctionTests
public void SetKillSwitch(bool active) =>
AdminConfig.Setup(a => a.IsSessionDeletionKillSwitchActiveAsync()).ReturnsAsync(active);
/// <summary>
/// Deterministic watchdog gate: returns a task that completes once an OpsEvent whose type is
/// <paramref name="eventName"/> has been recorded. A test's fanout body awaits this instead of
/// sleeping a fixed span, so the watchdog is guaranteed to have fired before the body returns —
/// removing the wall-clock race (body sleep vs sub-second thresholds) that flaked under CI load.
/// </summary>
public Task WaitForEventAsync(string eventName)
{
lock (_opsSync)
{
if (OpsEvents.Any(e => e.EventName == eventName))
return Task.CompletedTask;
if (!_eventWaiters.TryGetValue(eventName, out var waiter))
{
waiter = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
_eventWaiters[eventName] = waiter;
}
return waiter.Task;
}
}
public void SeedTtlBlobs(params (string TenantId, string SessionId, string ManifestId, DateTime LastModified)[] entries)
{
foreach (var e in entries)
@@ -1,4 +1,19 @@
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Azure;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Functions.Config;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
@@ -9,6 +24,8 @@ namespace AutopilotMonitor.Functions.Tests;
/// </summary>
public class TenantContactEmailTests
{
private const string TenantId = "77777777-7777-7777-7777-777777777777";
[Fact]
public void ContactEmail_defaults_to_null_so_absence_is_distinguishable_from_empty()
{
@@ -22,4 +39,348 @@ public class TenantContactEmailTests
var config = new TenantConfiguration { ContactEmail = "it-operations@contoso.com" };
Assert.Equal("it-operations@contoso.com", config.ContactEmail);
}
// ------------------------------------------------------------------
// Server-side validation (UpdateTenantConfigurationFunction)
// ------------------------------------------------------------------
[Theory]
[InlineData("it-operations@contoso.com")]
[InlineData("first.last+autopilot@sub.fabrikam.co.uk")]
[InlineData("UPPER@CONTOSO.COM")]
public void ValidateContactEmail_accepts_real_addresses(string email)
=> Assert.Null(UpdateTenantConfigurationFunction.ValidateContactEmail(email));
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public void ValidateContactEmail_treats_absence_as_valid(string? email)
{
// No contact address is a legitimate state — it means we cannot reach this tenant.
Assert.Null(UpdateTenantConfigurationFunction.ValidateContactEmail(email));
}
[Theory]
[InlineData("not-an-address", "single")] // no @ at all
[InlineData("@contoso.com", "single")] // empty local part
[InlineData("ops@", "single")] // empty domain
[InlineData("ops@@contoso.com", "single")] // two @
[InlineData("ops@localhost", "dotted")] // unreachable bare host
[InlineData("ops@.contoso.com", "dotted")] // leading dot
[InlineData("ops@contoso.", "dotted")] // trailing dot
[InlineData("ops@contoso.com, attacker@evil.test", "single address")] // recipient list
[InlineData("ops@contoso.com; attacker@evil.test", "single address")]
[InlineData("Ops Team <ops@contoso.com>", "single address")] // display-name form
public void ValidateContactEmail_rejects_values_that_are_not_a_single_address(string email, string expectedHint)
{
var error = UpdateTenantConfigurationFunction.ValidateContactEmail(email);
Assert.NotNull(error);
Assert.Contains(expectedHint, error!, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public void ValidateContactEmail_rejects_header_injection()
{
// Once this address is actually mailed, an embedded CR/LF would forge mail headers.
var error = UpdateTenantConfigurationFunction.ValidateContactEmail(
$"ops@contoso.com{(char)13}{(char)10}Bcc: attacker@evil.test");
Assert.NotNull(error);
}
[Fact]
public void ValidateContactEmail_rejects_an_over_long_value()
{
var local = new string('a', UpdateTenantConfigurationFunction.MaxContactEmailLength);
var error = UpdateTenantConfigurationFunction.ValidateContactEmail($"{local}@contoso.com");
Assert.NotNull(error);
Assert.Contains("at most", error!);
}
// ------------------------------------------------------------------
// Seed contract (TenantConfigurationService — the single owner)
// ------------------------------------------------------------------
private static (TenantConfigurationService service, Mock<IConfigRepository> repo, MemoryCache cache) BuildService()
{
var repo = new Mock<IConfigRepository>();
var cache = new MemoryCache(new MemoryCacheOptions());
var service = new TenantConfigurationService(
repo.Object, NullLogger<TenantConfigurationService>.Instance, cache);
return (service, repo, cache);
}
[Fact]
public async Task TrySeedContactEmail_when_the_write_lands_reports_success_and_drops_the_cached_row()
{
var (service, repo, cache) = BuildService();
repo.Setup(r => r.TrySeedTenantContactEmailAsync(TenantId, "ops@contoso.com")).ReturnsAsync(true);
cache.Set($"tenant-config:{TenantId}", new TenantConfiguration { TenantId = TenantId });
Assert.True(await service.TrySeedContactEmailAsync(TenantId, "ops@contoso.com"));
// The repository writes behind this cache; without the eviction the freshly seeded
// address would stay invisible for the full 5-minute TTL.
Assert.False(cache.TryGetValue($"tenant-config:{TenantId}", out _));
}
[Fact]
public async Task TrySeedContactEmail_when_the_tenant_already_owns_an_address_leaves_the_cache_alone()
{
var (service, repo, cache) = BuildService();
repo.Setup(r => r.TrySeedTenantContactEmailAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync(false);
var cached = new TenantConfiguration { TenantId = TenantId, ContactEmail = "owner@contoso.com" };
cache.Set($"tenant-config:{TenantId}", cached);
Assert.False(await service.TrySeedContactEmailAsync(TenantId, "ops@contoso.com"));
Assert.True(cache.TryGetValue($"tenant-config:{TenantId}", out TenantConfiguration? still));
Assert.Equal("owner@contoso.com", still!.ContactEmail);
}
[Theory]
[InlineData(null)]
[InlineData("")]
[InlineData(" ")]
public async Task TrySeedContactEmail_with_no_address_never_reaches_storage(string? email)
{
var (service, repo, _) = BuildService();
Assert.False(await service.TrySeedContactEmailAsync(TenantId, email));
repo.Verify(r => r.TrySeedTenantContactEmailAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
}
[Fact]
public async Task TrySeedContactEmail_with_no_tenant_never_reaches_storage()
{
var (service, repo, _) = BuildService();
Assert.False(await service.TrySeedContactEmailAsync(" ", "ops@contoso.com"));
repo.Verify(r => r.TrySeedTenantContactEmailAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
}
[Fact]
public async Task TrySeedContactEmail_trims_before_it_reaches_storage()
{
var (service, repo, _) = BuildService();
repo.Setup(r => r.TrySeedTenantContactEmailAsync(TenantId, "ops@contoso.com")).ReturnsAsync(true);
Assert.True(await service.TrySeedContactEmailAsync(TenantId, " ops@contoso.com "));
repo.Verify(r => r.TrySeedTenantContactEmailAsync(TenantId, "ops@contoso.com"), Times.Once);
}
[Fact]
public async Task TrySeedContactEmail_never_writes_the_whole_configuration()
{
// Regression guard: the seed used to be a read-modify-write of the full model through
// SaveTenantConfigurationAsync, which replaces the row unconditionally — a concurrent
// portal save was clobbered by the seeder's stale snapshot of EVERY field.
var (service, repo, _) = BuildService();
repo.Setup(r => r.TrySeedTenantContactEmailAsync(It.IsAny<string>(), It.IsAny<string>()))
.ReturnsAsync(true);
await service.TrySeedContactEmailAsync(TenantId, "ops@contoso.com");
repo.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>()), Times.Never);
}
// ------------------------------------------------------------------
// Seed trigger (PreviewWhitelistService)
// ------------------------------------------------------------------
private static (PreviewWhitelistService preview, Mock<IConfigRepository> repo) BuildPreview()
{
var repo = new Mock<IConfigRepository>();
var cache = new MemoryCache(new MemoryCacheOptions());
var tenantConfig = new TenantConfigurationService(
repo.Object, NullLogger<TenantConfigurationService>.Instance, cache);
var preview = new PreviewWhitelistService(
repo.Object, cache, NullLogger<PreviewWhitelistService>.Instance, tenantConfig);
return (preview, repo);
}
[Fact]
public async Task SaveNotificationEmail_persists_the_address_and_seeds_the_contact_field()
{
var (preview, repo) = BuildPreview();
repo.Setup(r => r.TrySeedTenantContactEmailAsync(TenantId, "ops@contoso.com")).ReturnsAsync(true);
await preview.SaveNotificationEmailAsync(TenantId, "ops@contoso.com");
repo.Verify(r => r.SaveNotificationEmailAsync(TenantId, "ops@contoso.com"), Times.Once);
repo.Verify(r => r.TrySeedTenantContactEmailAsync(TenantId, "ops@contoso.com"), Times.Once);
}
[Fact]
public async Task SaveNotificationEmail_clearing_the_address_does_not_seed()
{
var (preview, repo) = BuildPreview();
await preview.SaveNotificationEmailAsync(TenantId, null);
repo.Verify(r => r.SaveNotificationEmailAsync(TenantId, null), Times.Once);
repo.Verify(r => r.TrySeedTenantContactEmailAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
}
[Fact]
public async Task SaveNotificationEmail_still_succeeds_when_the_seed_blows_up()
{
// The notification email has already been persisted at that point; a failing side
// effect must not turn a completed write into a failed request.
var (preview, repo) = BuildPreview();
repo.Setup(r => r.TrySeedTenantContactEmailAsync(It.IsAny<string>(), It.IsAny<string>()))
.ThrowsAsync(new RequestFailedException(503, "throttled"));
await preview.SaveNotificationEmailAsync(TenantId, "ops@contoso.com");
repo.Verify(r => r.SaveNotificationEmailAsync(TenantId, "ops@contoso.com"), Times.Once);
}
// ------------------------------------------------------------------
// Conditional write (TableConfigRepository) — the actual invariant
// ------------------------------------------------------------------
private sealed class RepoHarness
{
public TableConfigRepository Sut { get; }
public Mock<TableClient> Table { get; } = new();
public List<(TableEntity Entity, ETag IfMatch, TableUpdateMode Mode)> Updates { get; } = new();
public RepoHarness(TableEntity? stored, RequestFailedException? throwOnUpdate = null)
{
if (stored == null)
{
Table.Setup(c => c.GetEntityAsync<TableEntity>(
It.IsAny<string>(), It.IsAny<string>(),
It.IsAny<IEnumerable<string>>(), It.IsAny<CancellationToken>()))
.ThrowsAsync(new RequestFailedException(404, "Not Found", "ResourceNotFound", null));
}
else
{
Table.Setup(c => c.GetEntityAsync<TableEntity>(
It.IsAny<string>(), It.IsAny<string>(),
It.IsAny<IEnumerable<string>>(), It.IsAny<CancellationToken>()))
.ReturnsAsync(Response.FromValue(stored, Mock.Of<Response>()));
}
Table.Setup(c => c.UpdateEntityAsync(
It.IsAny<TableEntity>(), It.IsAny<ETag>(),
It.IsAny<TableUpdateMode>(), It.IsAny<CancellationToken>()))
.Returns<TableEntity, ETag, TableUpdateMode, CancellationToken>((e, tag, mode, _) =>
{
if (throwOnUpdate != null)
throw throwOnUpdate;
Updates.Add((e, tag, mode));
return Task.FromResult(Mock.Of<Response>());
});
var serviceClient = new Mock<TableServiceClient>();
serviceClient.Setup(c => c.GetTableClient(It.IsAny<string>())).Returns(Table.Object);
Sut = new TableConfigRepository(
new TableStorageService(serviceClient.Object, NullLogger<TableStorageService>.Instance),
NullLogger<TableConfigRepository>.Instance);
}
}
private static TableEntity StoredConfig(string? contactEmail, string etag = "W/\"stored\"")
{
var entity = new TableEntity(TenantId, "config") { ["DomainName"] = "contoso.com" };
if (contactEmail != null)
entity["ContactEmail"] = contactEmail;
entity.ETag = new ETag(etag);
return entity;
}
[Fact]
public async Task Repo_seeds_only_the_contact_field_and_only_conditionally()
{
var harness = new RepoHarness(StoredConfig(contactEmail: null));
Assert.True(await harness.Sut.TrySeedTenantContactEmailAsync(TenantId, " ops@contoso.com "));
var (entity, ifMatch, mode) = Assert.Single(harness.Updates);
// Merge, not Replace: every other stored property must survive untouched.
Assert.Equal(TableUpdateMode.Merge, mode);
Assert.Equal(new ETag("W/\"stored\""), ifMatch);
Assert.Equal("ops@contoso.com", entity["ContactEmail"]);
Assert.False(entity.ContainsKey("DomainName"));
}
[Fact]
public async Task Repo_never_overwrites_an_address_the_tenant_already_owns()
{
var harness = new RepoHarness(StoredConfig("owner@contoso.com"));
Assert.False(await harness.Sut.TrySeedTenantContactEmailAsync(TenantId, "ops@contoso.com"));
Assert.Empty(harness.Updates);
}
[Fact]
public async Task Repo_treats_a_whitespace_only_stored_value_as_absent()
{
var harness = new RepoHarness(StoredConfig(" "));
Assert.True(await harness.Sut.TrySeedTenantContactEmailAsync(TenantId, "ops@contoso.com"));
Assert.Single(harness.Updates);
}
[Fact]
public async Task Repo_loses_the_race_rather_than_clobbering_a_concurrent_write()
{
// Someone wrote the row between our read and our write. The ETag precondition turns
// that into a 412; the seed abandons rather than overwriting the winner.
var harness = new RepoHarness(
StoredConfig(contactEmail: null),
throwOnUpdate: new RequestFailedException(412, "Precondition Failed", "UpdateConditionNotSatisfied", null));
Assert.False(await harness.Sut.TrySeedTenantContactEmailAsync(TenantId, "ops@contoso.com"));
}
[Fact]
public async Task Repo_with_no_configuration_row_reports_no_seed()
{
var harness = new RepoHarness(stored: null);
Assert.False(await harness.Sut.TrySeedTenantContactEmailAsync(TenantId, "ops@contoso.com"));
Assert.Empty(harness.Updates);
}
[Fact]
public async Task Repo_is_fail_soft_on_a_storage_fault()
{
// Callers are side effects of an already-committed write — a lost seed is recoverable
// on the next maintenance run, a thrown exception is not.
var harness = new RepoHarness(
StoredConfig(contactEmail: null),
throwOnUpdate: new RequestFailedException(503, "Server Busy"));
Assert.False(await harness.Sut.TrySeedTenantContactEmailAsync(TenantId, "ops@contoso.com"));
}
[Theory]
[InlineData(null, "ops@contoso.com")]
[InlineData("", "ops@contoso.com")]
[InlineData(TenantId, null)]
[InlineData(TenantId, " ")]
public async Task Repo_rejects_incomplete_input_without_touching_storage(string? tenantId, string? email)
{
var harness = new RepoHarness(StoredConfig(contactEmail: null));
Assert.False(await harness.Sut.TrySeedTenantContactEmailAsync(tenantId!, email!));
Assert.Empty(harness.Updates);
harness.Table.Verify(c => c.GetEntityAsync<TableEntity>(
It.IsAny<string>(), It.IsAny<string>(),
It.IsAny<IEnumerable<string>>(), It.IsAny<CancellationToken>()), Times.Never);
}
}
@@ -87,7 +87,11 @@ public sealed class TenantOffboardFunctionTests
return new PreviewWhitelistService(
configRepoMock.Object,
cache,
NullLogger<PreviewWhitelistService>.Instance);
NullLogger<PreviewWhitelistService>.Instance,
new TenantConfigurationService(
configRepoMock.Object,
NullLogger<TenantConfigurationService>.Instance,
cache));
}
/// <summary>
@@ -64,22 +64,26 @@ public class ValidateBootstrapCodeResponseShapeTests
}
/// <summary>
/// The hardcoded agent download URL in <c>ValidateBootstrapCodeFunction</c> must
/// The agent download URL served by <c>ValidateBootstrapCodeFunction</c> must
/// satisfy the frontend's host allow-list and path regex.
/// </summary>
[Fact]
public void AgentDownloadUrl_MatchesFrontendAllowList()
{
// Keep in sync with ValidateBootstrapCodeFunction.cs — if that literal
// changes, both this assertion and bootstrapValidation.ts must be updated.
const string HardcodedAgentUrl =
"https://autopilotmonitor.blob.core.windows.net/agent/AutopilotMonitor-Agent.zip";
// Independent oracle, deliberately spelled out: if Constants or the function
// change the URL, this literal AND bootstrapValidation.ts (AGENT_DOWNLOAD_HOSTNAMES)
// must be updated consciously — that pairing is what keeps portal and backend in sync.
const string ExpectedAgentUrl =
"https://download.autopilotmonitor.com/agent/AutopilotMonitor-Agent.zip";
Assert.True(Uri.TryCreate(HardcodedAgentUrl, UriKind.Absolute, out var uri),
var built = $"{AutopilotMonitor.Shared.Constants.AgentDownloadBaseUrl}/{AutopilotMonitor.Shared.Constants.AgentZipFileName}";
Assert.Equal(ExpectedAgentUrl, built);
Assert.True(Uri.TryCreate(built, UriKind.Absolute, out var uri),
"AgentDownloadUrl is not a well-formed absolute URI");
Assert.Equal(Uri.UriSchemeHttps, uri!.Scheme);
Assert.Equal("autopilotmonitor.blob.core.windows.net", uri.Host);
Assert.Equal("download.autopilotmonitor.com", uri.Host);
Assert.True(AgentDownloadUrlPath.IsMatch(uri.AbsolutePath),
$"AgentDownloadUrl path '{uri.AbsolutePath}' does not match the frontend allow-list regex");
}
@@ -12,8 +12,16 @@
BuildTimestampUtc is baked into the assembly as AssemblyMetadata so /api/health
reports the actual build moment — not the file mtime, which Azure Functions
resets to the zip-extract time on every cold start. CI may override via
-p:BuildTimestampUtc=<iso8601> to force a specific timestamp. -->
<Version>1.0.0</Version>
-p:BuildTimestampUtc=<iso8601> to force a specific timestamp.
VersionPrefix is the curated part (bump by hand for a release); BuildNumber is
reserved per build from the counter blob versions/backend.txt and passed in as
-p:BuildNumber=<n> by deploy-backend.yml — same mechanism as the agent line.
A local build has no counter and stays at x.y.0 with an empty commit hash,
which is how /api/health tells a developer machine from a CI artifact. -->
<VersionPrefix>1.5</VersionPrefix>
<BuildNumber Condition="'$(BuildNumber)' == ''">0</BuildNumber>
<Version>$(VersionPrefix).$(BuildNumber)</Version>
<GenerateAssemblyInfo>true</GenerateAssemblyInfo>
<IncludeSourceRevisionInInformationalVersion>true</IncludeSourceRevisionInInformationalVersion>
<BuildTimestampUtc Condition="'$(BuildTimestampUtc)' == ''">$([System.DateTime]::UtcNow.ToString('o'))</BuildTimestampUtc>
@@ -47,6 +55,12 @@
<PackageReference Include="Newtonsoft.Json" Version="13.0.4" />
<PackageReference Include="Microsoft.Identity.Web" Version="4.3.0" />
<PackageReference Include="Microsoft.AspNetCore.Authentication.JwtBearer" Version="10.0.10" />
<!-- Security pins: Microsoft.Identity.Web 4.3.0 pulls these transitively at 10.0.0, which carries
known advisories (GHSA-9mv3-2cwr-p262 DataProtection; GHSA-37gx-xxp4-5rgx and siblings on
Cryptography.Xml). Force the patched 10.0.10 servicing build. Remove once Identity.Web ships
a dependency floor past the fix. -->
<PackageReference Include="Microsoft.AspNetCore.DataProtection" Version="10.0.10" />
<PackageReference Include="System.Security.Cryptography.Xml" Version="10.0.10" />
<PackageReference Include="Microsoft.Extensions.Http.Polly" Version="8.0.25" />
<PackageReference Include="Resend" Version="0.2.1" />
</ItemGroup>
@@ -71,6 +71,54 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
}
}
/// <summary>
/// Conditional, single-property seed of ContactEmail (see <see cref="IConfigRepository"/>).
/// Merge mode leaves every other stored property untouched, and the ETag precondition turns
/// a concurrent write into a 412 rather than a silent overwrite. Both matter: the caller is
/// a background/side-effect path that must never beat a tenant admin's own edit.
/// </summary>
public async Task<bool> TrySeedTenantContactEmailAsync(string tenantId, string email)
{
if (string.IsNullOrWhiteSpace(tenantId) || string.IsNullOrWhiteSpace(email))
return false;
try
{
var existing = await _tenantConfigTableClient.GetEntityAsync<TableEntity>(tenantId, "config");
// The tenant already owns an address — the seed is one-way and never re-syncs.
if (!string.IsNullOrWhiteSpace(existing.Value.GetString("ContactEmail")))
return false;
var patch = new TableEntity(tenantId, "config")
{
["ContactEmail"] = email.Trim()
};
await _tenantConfigTableClient.UpdateEntityAsync(
patch, existing.Value.ETag, TableUpdateMode.Merge);
return true;
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
// No configuration row yet — nothing to seed onto.
return false;
}
catch (RequestFailedException ex) when (ex.Status == 412)
{
// Someone wrote the row between the read and the write. They win by construction:
// a lost seed is recoverable (the backfill retries), a lost admin edit is not.
_logger.LogInformation(
"Contact address seed for {TenantId} skipped — the configuration changed concurrently", tenantId);
return false;
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Error seeding contact address for tenant {TenantId}", tenantId);
return false;
}
}
public async Task<List<TenantConfiguration>> GetAllTenantConfigurationsAsync()
{
try
@@ -542,8 +590,6 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
{ "ExcessiveEventAutoActionMode", config.ExcessiveEventAutoActionMode ?? "Off" },
{ "ExcessiveEventAutoActionThreshold", config.ExcessiveEventAutoActionThreshold },
{ "ExcessiveEventAutoActionDurationHours", config.ExcessiveEventAutoActionDurationHours },
{ "MaxSessionWindowHours", config.MaxSessionWindowHours },
{ "MaintenanceBlockDurationHours", config.MaintenanceBlockDurationHours },
{ "DiagnosticsGlobalLogPathsJson", config.DiagnosticsGlobalLogPathsJson },
{ "ModernDeploymentHarmlessEventIdsJson", config.ModernDeploymentHarmlessEventIdsJson ?? string.Empty },
{ "WhiteGloveSealingPatternIdsJson", config.WhiteGloveSealingPatternIdsJson ?? string.Empty },
@@ -609,8 +655,6 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
ExcessiveEventAutoActionMode = entity.GetString("ExcessiveEventAutoActionMode") ?? "Off",
ExcessiveEventAutoActionThreshold = entity.GetInt32("ExcessiveEventAutoActionThreshold") ?? 2500,
ExcessiveEventAutoActionDurationHours = entity.GetInt32("ExcessiveEventAutoActionDurationHours") ?? 24,
MaxSessionWindowHours = entity.GetInt32("MaxSessionWindowHours") ?? 24,
MaintenanceBlockDurationHours = entity.GetInt32("MaintenanceBlockDurationHours") ?? 12,
DiagnosticsGlobalLogPathsJson = entity.GetString("DiagnosticsGlobalLogPathsJson"),
ModernDeploymentHarmlessEventIdsJson = entity.GetString("ModernDeploymentHarmlessEventIdsJson"),
WhiteGloveSealingPatternIdsJson = entity.GetString("WhiteGloveSealingPatternIdsJson"),
@@ -69,9 +69,6 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
public Task<List<SessionSummary>> GetAgentSilentSessionsAsync(string tenantId, DateTime silenceCutoff, DateTime hardCutoff)
=> _storage.GetAgentSilentSessionsAsync(tenantId, silenceCutoff, hardCutoff);
public Task<List<SessionSummary>> GetExcessiveDataSendersAsync(string tenantId, DateTime windowCutoff, int maxSessionWindowHours)
=> _storage.GetExcessiveDataSendersAsync(tenantId, windowCutoff, maxSessionWindowHours);
public Task<List<SessionSummary>> GetLegacyTimeoutFailedSessionsAsync(string tenantId, int maxResults)
=> _storage.GetLegacyTimeoutFailedSessionsAsync(tenantId, maxResults);
@@ -1,3 +1,4 @@
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
@@ -72,7 +73,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
}
_logger.LogWarning("Failed to increment user usage after {MaxRetries} retries: user={UserId}, endpoint={Endpoint}",
maxRetries, userId, endpoint);
maxRetries, LogSanitizer.Clean(userId), LogSanitizer.Clean(endpoint));
}
public async Task<List<UserUsageRecord>> GetUsageByUserAsync(string userId, string? dateFrom = null, string? dateTo = null)
@@ -52,6 +52,34 @@ public static class ClaimsPrincipalExtensions
?? principal.FindFirst("preferred_username")?.Value;
}
/// <summary>
/// Stable per-caller identity for THROTTLING keys — never for authorization or audit.
/// <para>
/// Prefers the UPN so a human caller keeps the same bucket as before. Falls back to the token
/// subject (<c>oid</c> → <c>sub</c> → <c>appid</c>/<c>azp</c>) because a valid token need not carry
/// a user identity at all: AuthenticationMiddleware validates issuer/audience/lifetime/signature
/// but requires no <c>upn</c>, so an app-only (client-credentials) token authenticates fine and
/// reaches every AuthenticatedUser route. Without this fallback such a caller would present an
/// empty identity and skip per-user rate limiting entirely.
/// </para>
/// <para>
/// Both the raw and the WS-Fed-mapped claim names are checked: JwtSecurityTokenHandler runs with
/// inbound claim mapping ON, so <c>oid</c>/<c>sub</c> may arrive under their schema URIs.
/// </para>
/// Returns null only for a principal carrying no identifying claim whatsoever; callers must
/// decide fail-closed what that means (see <c>RequestContext.CallerId</c>).
/// </summary>
public static string? GetCallerId(this ClaimsPrincipal principal)
{
return principal.GetUserPrincipalName()
?? principal.FindFirst("oid")?.Value
?? principal.FindFirst("http://schemas.microsoft.com/identity/claims/objectidentifier")?.Value
?? principal.FindFirst("sub")?.Value
?? principal.FindFirst(ClaimTypes.NameIdentifier)?.Value
?? principal.FindFirst("appid")?.Value
?? principal.FindFirst("azp")?.Value;
}
/// <summary>
/// Gets the user's display name from token claims
/// Supports both v1.0 and v2.0 token formats
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System.Linq;
using System.Net;
using System.Threading;
@@ -107,7 +108,7 @@ public class AutopilotDeviceValidationConsentFunction
}
var consentUrl =
$"https://login.microsoftonline.com/{Uri.EscapeDataString(requestCtx.TargetTenantId)}/adminconsent" +
$"{Constants.EntraLoginBaseUrl}/{Uri.EscapeDataString(requestCtx.TargetTenantId)}/adminconsent" +
$"?client_id={Uri.EscapeDataString(validatorClientId)}" +
$"&redirect_uri={Uri.EscapeDataString(redirectUri)}" +
$"&state={Uri.EscapeDataString("autopilot-device-validation-enable")}";
@@ -4,6 +4,7 @@ using System.Net;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Azure.Functions.Worker;
@@ -92,7 +93,9 @@ namespace AutopilotMonitor.Functions.Functions.Bootstrap
{
Success = true,
ShortCode = session.ShortCode,
BootstrapUrl = $"https://autopilotmonitor.com/go/{session.ShortCode}",
// www directly — the bare apex is only a registrar-level 301 to www,
// so pointing there just adds a redirect hop for every bootstrap.
BootstrapUrl = $"{Constants.WebsiteBaseUrl}/go/{session.ShortCode}",
ExpiresAt = session.ExpiresAt,
Message = $"Bootstrap session created. Valid for {validityHours} hours."
};
@@ -92,7 +92,9 @@ namespace AutopilotMonitor.Functions.Functions.Bootstrap
Success = true,
TenantId = session.TenantId,
Token = session.Token,
AgentDownloadUrl = "https://autopilotmonitor.blob.core.windows.net/agent/AutopilotMonitor-Agent.zip",
// Download alias — the portal's bootstrapValidation.ts allowlists this
// host (plus the legacy blob host for transition); keep the two in sync.
AgentDownloadUrl = $"{Constants.AgentDownloadBaseUrl}/{Constants.AgentZipFileName}",
ExpiresAt = session.ExpiresAt,
Message = "Bootstrap code validated"
};
@@ -76,6 +76,19 @@ namespace AutopilotMonitor.Functions.Functions.Config
return badRequest;
}
// The contact address is where enforcement actions and service notices are sent,
// so it must be an address rather than whatever string a direct API caller posts.
var contactEmailError = ValidateContactEmail(config.ContactEmail);
if (contactEmailError != null)
{
var badRequest = req.CreateResponse(HttpStatusCode.BadRequest);
await badRequest.WriteAsJsonAsync(new { success = false, message = $"Invalid contact email: {contactEmailError}" });
return badRequest;
}
// Normalize so the stored value never carries surrounding whitespace, and an
// all-whitespace submission clears the field instead of masquerading as a value.
config.ContactEmail = string.IsNullOrWhiteSpace(config.ContactEmail) ? null : config.ContactEmail.Trim();
// Load the stored config up-front so we can (1) restore any redacted secret placeholders
// before validation/save and (2) protect GA-only fields below.
var existingConfig = await _configService.GetConfigurationAsync(requestCtx.TargetTenantId);
@@ -251,6 +264,52 @@ namespace AutopilotMonitor.Functions.Functions.Config
private const int MaxCustomHeaderCount = 25;
private const int MaxNotificationChannelsJsonLength = 65536;
// RFC 5321 caps a forward path at 254 characters.
internal const int MaxContactEmailLength = 254;
/// <summary>
/// Validates the tenant contact address. Returns an error message, or null when valid/empty.
/// Empty is legitimate — it means we have no way to reach the tenant.
/// <para>
/// Deliberately not an RFC 5322 parser: the job is to reject values that are not addresses
/// at all. Specifically it rejects recipient lists (a comma would silently widen who receives
/// service notices), display-name forms, and control characters (which would let a caller
/// forge mail headers once this address is actually mailed).
/// </para>
/// </summary>
internal static string? ValidateContactEmail(string? email)
{
if (string.IsNullOrWhiteSpace(email))
return null;
var trimmed = email.Trim();
if (trimmed.Length > MaxContactEmailLength)
return $"must be at most {MaxContactEmailLength} characters.";
foreach (var ch in trimmed)
{
if (char.IsControl(ch))
return "must not contain control characters.";
if (char.IsWhiteSpace(ch) || ch == ',' || ch == ';' || ch == '<' || ch == '>')
return "must be a single address, without spaces, separators or angle brackets.";
}
var at = trimmed.IndexOf('@');
if (at <= 0 || at != trimmed.LastIndexOf('@') || at == trimmed.Length - 1)
return "must contain a single \"@\" with text on both sides.";
// A bare host with no dot is unreachable from our sender, so it is a typo, not an address.
var domain = trimmed.Substring(at + 1);
if (!domain.Contains('.') || domain.StartsWith(".", StringComparison.Ordinal)
|| domain.EndsWith(".", StringComparison.Ordinal))
{
return "the domain part must be a dotted host name.";
}
return null;
}
/// <summary>
/// Validates the notification-channel list JSON. Returns an error message, or null when
/// valid/empty. Strict counterpart of the fail-soft <c>NotificationChannel.ParseList</c>:
@@ -1,5 +1,7 @@
using System.Net;
using System.Web;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Functions.Services.Diagnostics;
using AutopilotMonitor.Shared.Diagnostics;
using Microsoft.Azure.Functions.Worker;
@@ -17,21 +19,29 @@ namespace AutopilotMonitor.Functions.Functions.Diagnostics
/// from the query string, so a ticket cannot be retargeted.
/// <para>
/// Replay is bounded by the 10-min ticket TTL, the admin size cap, and the fact that the ticket
/// only ever points at the minting caller's own authorized blob; HMAC forgery is infeasible, so
/// no additional rate limiting is layered here.
/// only ever points at the minting caller's own authorized blob; HMAC forgery is infeasible. On top
/// of that the route carries a per-client-IP limit as defense in depth — it is unauthenticated and
/// streams whole blobs, so a valid ticket replayed in a loop would otherwise be unbounded egress.
/// </para>
/// </summary>
public class DiagnosticsTicketDownloadFunction
{
// Bounds replay of a still-valid ticket. A real client downloads a package once; the ceiling
// only has to stay clear of a legitimate retry-after-timeout.
private const int MaxRequestsPerMinutePerIp = 30;
private readonly ILogger<DiagnosticsTicketDownloadFunction> _logger;
private readonly DiagnosticsBlobStreamer _streamer;
private readonly RateLimitService _rateLimitService;
public DiagnosticsTicketDownloadFunction(
ILogger<DiagnosticsTicketDownloadFunction> logger,
DiagnosticsBlobStreamer streamer)
DiagnosticsBlobStreamer streamer,
RateLimitService rateLimitService)
{
_logger = logger;
_streamer = streamer;
_rateLimitService = rateLimitService;
}
[Function("DiagnosticsTicketDownload")]
@@ -40,6 +50,23 @@ namespace AutopilotMonitor.Functions.Functions.Diagnostics
{
try
{
// Rightmost X-Forwarded-For hop only — leftmost entries are caller-controlled.
var clientIp = ClientIpExtractor.GetTrustedClientIp(req);
var rateLimitResult = _rateLimitService.CheckRateLimit(
$"diag_ticket_download_{clientIp}", MaxRequestsPerMinutePerIp);
if (!rateLimitResult.IsAllowed)
{
_logger.LogWarning("DiagnosticsTicketDownload rate limit exceeded for IP {ClientIp} ({Count} requests)",
clientIp, rateLimitResult.RequestsInWindow);
var tooMany = req.CreateResponse(HttpStatusCode.TooManyRequests);
if (rateLimitResult.RetryAfter.HasValue)
tooMany.Headers.Add("Retry-After", ((int)rateLimitResult.RetryAfter.Value.TotalSeconds).ToString());
await tooMany.WriteAsJsonAsync(new { success = false, message = "Rate limit exceeded." });
return tooMany;
}
var ticket = HttpUtility.ParseQueryString(req.Url.Query)["t"];
if (string.IsNullOrEmpty(ticket))
{
@@ -37,6 +37,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
private readonly TelemetryClient _telemetryClient;
private readonly BootstrapSessionService _bootstrapSessionService;
private readonly ISessionRepository _sessionRepo;
private readonly OpsEventService _opsEventService;
public ReportAgentErrorFunction(
ILogger<ReportAgentErrorFunction> logger,
@@ -48,7 +49,8 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
DeviceAssociationValidator deviceAssociationValidator,
TelemetryClient telemetryClient,
BootstrapSessionService bootstrapSessionService,
ISessionRepository sessionRepo)
ISessionRepository sessionRepo,
OpsEventService opsEventService)
{
_logger = logger;
_configService = configService;
@@ -60,6 +62,7 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
_telemetryClient = telemetryClient;
_bootstrapSessionService = bootstrapSessionService;
_sessionRepo = sessionRepo;
_opsEventService = opsEventService;
}
[Function("ReportAgentError")]
@@ -169,35 +172,71 @@ namespace AutopilotMonitor.Functions.Functions.Ingest
["AgentTimestamp"] = report.Timestamp.ToString("O"),
});
// Materialize the agent's silent 48h emergency break as a timeline event so it shows in the
// session and the timeout classifier can see it (tasks/enrollment-status-reclassification.md).
// Best-effort — a failure here must never turn the always-200 emergency channel into a retry loop.
if (report.ErrorType == AgentErrorType.SessionAgeEmergencyBreak
&& !string.IsNullOrEmpty(report.SessionId))
{
try
{
var existing = await _sessionRepo.GetSessionEventsAsync(tenantId, report.SessionId, maxResults: 1000);
// Idempotency: the agent's emergency channel can send up to a few reports per session;
// only ever materialize one timeline event.
var alreadyMaterialized = existing.Any(e =>
string.Equals(e.EventType, Constants.EventTypes.AgentEmergencyBreak, StringComparison.OrdinalIgnoreCase));
if (!alreadyMaterialized)
{
var evt = BuildAgentEmergencyBreakEvent(report, tenantId, existing, DateTime.UtcNow);
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
}
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"ReportAgentError: failed to materialize agent_emergency_break event for session {SessionId}", report.SessionId);
}
}
await MaterializeEmergencyBreakArtifactsAsync(report, tenantId, _sessionRepo, _opsEventService, _logger);
return req.CreateResponse(HttpStatusCode.OK);
}
/// <summary>
/// Materializes the agent's silent 48h emergency break as (1) a timeline event so it shows
/// in the session and the timeout classifier can see it, (2) the cross-session EventType
/// index row, and (3) an <c>AgentEmergencyBreak</c> ops event for operator visibility
/// (tasks/enrollment-status-reclassification.md). Static seam with explicit dependencies so
/// tests can pin the artifact set without booting the Functions HTTP stack.
/// Best-effort — a failure here must never turn the always-200 emergency channel into a
/// retry loop.
/// </summary>
internal static async Task MaterializeEmergencyBreakArtifactsAsync(
AgentErrorReport report,
string tenantId,
ISessionRepository sessionRepo,
OpsEventService opsEventService,
ILogger logger)
{
if (report.ErrorType != AgentErrorType.SessionAgeEmergencyBreak
|| string.IsNullOrEmpty(report.SessionId))
{
return;
}
try
{
var existing = await sessionRepo.GetSessionEventsAsync(tenantId, report.SessionId, maxResults: 1000);
// Idempotency: the agent's emergency channel can send up to a few reports per session;
// only ever materialize one timeline event (and one ops event).
var alreadyMaterialized = existing.Any(e =>
string.Equals(e.EventType, Constants.EventTypes.AgentEmergencyBreak, StringComparison.OrdinalIgnoreCase));
if (alreadyMaterialized)
{
return;
}
var evt = BuildAgentEmergencyBreakEvent(report, tenantId, existing, DateTime.UtcNow);
await sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { evt });
// StoreEventsBatchAsync writes the Events partition only — the cross-session
// EventType index is normally written by EventIngestProcessor, which this
// backend-materialized event never passes through. Without the upsert the event
// exists on the session timeline but is invisible to every search-by-eventType
// surface (portal cross-session search, MCP search_sessions_by_event /
// query_raw_events) — found the hard way in the 2026-07-22 incident analysis.
await sessionRepo.UpsertEventTypeIndexBatchAsync(
tenantId, report.SessionId, new List<EnrollmentEvent> { evt });
// Operator visibility: an emergency break means an agent silently gave up at its
// absolute age cap — exactly the "are we losing agents?" signal. Emitted only on
// first materialization so repeat reports for the same session cannot flood the
// ops feed. OpsEventService never throws.
await opsEventService.RecordAgentEmergencyBreakAsync(
tenantId, report.SessionId, report.AgentVersion, evt.Message);
}
catch (Exception ex)
{
logger.LogWarning(ex,
"ReportAgentError: failed to materialize agent_emergency_break event for session {SessionId}", report.SessionId);
}
}
/// <summary>
/// Builds the backend-materialized <c>agent_emergency_break</c> timeline event from the agent's
/// best-effort emergency report. Static + pure (analog to
@@ -1,4 +1,6 @@
using System.Net;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
@@ -9,16 +11,27 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
/// <summary>
/// Public endpoint - no authentication required.
/// Returns pre-computed platform stats for the landing page.
/// Rate-limited per client IP: the route is unauthenticated and hits Table Storage on every call,
/// so it needs a limit of its own — no user bucket applies to an anonymous caller.
/// </summary>
public class GetPlatformStatsFunction
{
// Landing-page widget: one call per page load. Generous enough that a shared corporate NAT
// never trips it, tight enough to bound an unauthenticated storage read.
private const int MaxRequestsPerMinutePerIp = 60;
private readonly ILogger<GetPlatformStatsFunction> _logger;
private readonly IMetricsRepository _metricsRepo;
private readonly RateLimitService _rateLimitService;
public GetPlatformStatsFunction(ILogger<GetPlatformStatsFunction> logger, IMetricsRepository metricsRepo)
public GetPlatformStatsFunction(
ILogger<GetPlatformStatsFunction> logger,
IMetricsRepository metricsRepo,
RateLimitService rateLimitService)
{
_logger = logger;
_metricsRepo = metricsRepo;
_rateLimitService = rateLimitService;
}
[Function("GetPlatformStats")]
@@ -27,6 +40,24 @@ namespace AutopilotMonitor.Functions.Functions.Metrics
{
try
{
// Rightmost X-Forwarded-For hop only — leftmost entries are caller-controlled and would
// let one client rotate the rate-limit key on every request.
var clientIp = ClientIpExtractor.GetTrustedClientIp(req);
var rateLimitResult = _rateLimitService.CheckRateLimit(
$"platform_stats_{clientIp}", MaxRequestsPerMinutePerIp);
if (!rateLimitResult.IsAllowed)
{
_logger.LogWarning("Platform stats rate limit exceeded for IP {ClientIp} ({Count} requests)",
clientIp, rateLimitResult.RequestsInWindow);
var tooMany = req.CreateResponse(HttpStatusCode.TooManyRequests);
if (rateLimitResult.RetryAfter.HasValue)
tooMany.Headers.Add("Retry-After", ((int)rateLimitResult.RetryAfter.Value.TotalSeconds).ToString());
await tooMany.WriteAsJsonAsync(new { error = "Rate limit exceeded" });
return tooMany;
}
var stats = await _metricsRepo.GetPlatformStatsAsync();
var response = req.CreateResponse(HttpStatusCode.OK);
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System.Net;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Services;
@@ -351,7 +352,7 @@ namespace AutopilotMonitor.Functions.Functions.Sessions
if (startChannels.Count == 0)
return;
var sessionUrl = $"https://portal.autopilotmonitor.com/sessions/{sessionId}";
var sessionUrl = $"{Constants.PortalBaseUrl}/sessions/{sessionId}";
var alert = NotificationAlertBuilder.BuildEnrollmentStartedAlert(
session.DeviceName,
session.SerialNumber,
@@ -0,0 +1,36 @@
using System.Text.RegularExpressions;
namespace AutopilotMonitor.Functions.Helpers;
/// <summary>
/// Strips control characters from user-controlled values before they reach a log
/// sink, so a caller cannot forge additional log lines (CRLF injection) or smuggle
/// terminal escape sequences through the message. Structured logging already stores
/// each value as a discrete property, so this is defense-in-depth — it also clears
/// the CodeQL cs/log-forging alerts by removing the newline characters they key on.
/// Only ever apply to the LOGGED copy of a value; never to a value used for routing,
/// authorization, or storage keys.
/// </summary>
public static class LogSanitizer
{
// \p{Cc} = Unicode "Other, control" (C0 + C1 ranges), which includes CR and LF.
// \p{Zl} and \p{Zp} are the line and paragraph separators (U+2028 / U+2029). They
// are separators, not control characters, so \p{Cc} misses them — yet a log viewer,
// a JSON-lines consumer, or any JavaScript-side reader still breaks a line on them.
private static readonly Regex ControlChars = new(@"[\p{Cc}\p{Zl}\p{Zp}]", RegexOptions.Compiled);
/// <summary>
/// Removes CR, LF, the Unicode line/paragraph separators, and every other control
/// character from <paramref name="value"/>. Returns null/empty inputs unchanged.
/// </summary>
public static string? Clean(string? value)
{
if (string.IsNullOrEmpty(value))
return value;
// Explicit CR/LF removal first — this is the log-injection barrier the CodeQL
// taint model recognizes; the regex pass then drops any remaining separators.
var sanitized = value.Replace("\r", string.Empty).Replace("\n", string.Empty);
return ControlChars.Replace(sanitized, string.Empty);
}
}
@@ -19,6 +19,23 @@ public sealed record RequestContext
/// <summary>The user's UPN (from JWT upn/preferred_username claim).</summary>
public string UserPrincipalName { get; init; } = string.Empty;
/// <summary>
/// Stable identity of the authenticated caller, for THROTTLING keys only — never authorization,
/// never audit (use <see cref="UserPrincipalName"/> there; this may be a bare object id).
/// <para>
/// Empty ⇔ the request carried no JWT at all (device/bootstrap or anonymous route). Those routes
/// bring their own limits — per cert thumbprint / per bootstrap token in SecurityValidator, per
/// client IP on the public endpoints — so an empty value means "no user bucket applies here".
/// </para>
/// <para>
/// It is deliberately NOT the same field as <see cref="UserPrincipalName"/>: a valid token need
/// not carry a UPN (app-only/client-credentials tokens do not), and such a caller still reaches
/// every AuthenticatedUser route. Keying the throttle off the UPN alone let those tokens through
/// unlimited; keying audit off this would write object ids where a UPN is expected.
/// </para>
/// </summary>
public string CallerId { get; init; } = string.Empty;
/// <summary>True if the user is a Global Admin of the platform (full read + write).</summary>
public bool IsGlobalAdmin { get; init; }
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Azure.Functions.Worker.Middleware;
@@ -6,11 +7,13 @@ using Microsoft.Extensions.Logging;
using Microsoft.IdentityModel.Protocols;
using Microsoft.IdentityModel.Protocols.OpenIdConnect;
using Microsoft.IdentityModel.Tokens;
using System.Diagnostics.CodeAnalysis;
using System.IdentityModel.Tokens.Jwt;
using Microsoft.AspNetCore.Http;
using System.Net;
using System.Security.Claims;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Helpers;
namespace AutopilotMonitor.Functions.Middleware;
@@ -57,9 +60,12 @@ public class AuthenticationMiddleware : IFunctionsWorkerMiddleware
// JWT validation. The exempt set is DERIVED from EndpointAccessPolicyCatalog — never a
// hand-kept parallel list — so a new anonymous/device route can't drift out of sync here.
var requestPath = httpContext.Request.Path.Value ?? string.Empty;
// Sanitized copies for logging only — never used for routing/authorization decisions.
var logPath = LogSanitizer.Clean(requestPath);
var logMethod = LogSanitizer.Clean(httpContext.Request.Method);
if (SkipsJwtValidation(httpContext.Request.Method, requestPath))
{
_logger.LogDebug("[Auth Middleware] JWT-exempt route: {Method} {Path}", httpContext.Request.Method, requestPath);
_logger.LogDebug("[Auth Middleware] JWT-exempt route: {Method} {Path}", logMethod, logPath);
await next(context);
return;
}
@@ -68,7 +74,7 @@ public class AuthenticationMiddleware : IFunctionsWorkerMiddleware
var authHeader = httpContext.Request.Headers.Authorization.ToString();
if (string.IsNullOrEmpty(authHeader) || !authHeader.StartsWith("Bearer ", StringComparison.OrdinalIgnoreCase))
{
_logger.LogWarning("[Auth Middleware] Blocked unauthenticated request to {Path}", requestPath);
_logger.LogWarning("[Auth Middleware] Blocked unauthenticated request to {Path}", logPath);
httpContext.Response.StatusCode = (int)HttpStatusCode.Unauthorized;
httpContext.Response.ContentType = "application/json";
await httpContext.Response.WriteAsJsonAsync(new
@@ -96,7 +102,7 @@ public class AuthenticationMiddleware : IFunctionsWorkerMiddleware
// MSA, tid 9188040d-…) carries a GUID tid, so this reject is loss-free.
if (!IsValidTenantId(tenantId))
{
_logger.LogWarning("[Auth Middleware] Rejected token with missing/non-GUID tid for {Path}", requestPath);
_logger.LogWarning("[Auth Middleware] Rejected token with missing/non-GUID tid for {Path}", logPath);
throw new SecurityTokenValidationException("Token tid claim is missing or not a valid GUID");
}
@@ -207,7 +213,7 @@ public class AuthenticationMiddleware : IFunctionsWorkerMiddleware
if (!authenticated)
{
_logger.LogWarning("[Auth Middleware] Blocked request with invalid token to {Path}", requestPath);
_logger.LogWarning("[Auth Middleware] Blocked request with invalid token to {Path}", logPath);
httpContext.Response.StatusCode = (int)HttpStatusCode.Unauthorized;
httpContext.Response.ContentType = "application/json";
await httpContext.Response.WriteAsJsonAsync(new
@@ -244,7 +250,7 @@ public class AuthenticationMiddleware : IFunctionsWorkerMiddleware
/// missing/garbage tid here prevents an outbound OIDC-metadata fetch to a bogus authority on
/// malformed-token floods. Loss-free — every legitimate Entra token carries a GUID tid.
/// </summary>
internal static bool IsValidTenantId(string? tid) => Guid.TryParse(tid, out _);
internal static bool IsValidTenantId([NotNullWhen(true)] string? tid) => Guid.TryParse(tid, out _);
/// <summary>
/// Maps a token issuer + tenant id to the OIDC authority to fetch signing keys from. Entra v1.0
@@ -255,8 +261,8 @@ public class AuthenticationMiddleware : IFunctionsWorkerMiddleware
{
var isV1Token = issuer != null && issuer.Contains("sts.windows.net");
return isV1Token
? $"https://login.microsoftonline.com/{tenantId}" // v1.0
: $"https://login.microsoftonline.com/{tenantId}/v2.0"; // v2.0
? $"{Constants.EntraLoginBaseUrl}/{tenantId}" // v1.0
: $"{Constants.EntraLoginBaseUrl}/{tenantId}/v2.0"; // v2.0
}
/// <summary>
@@ -279,7 +285,7 @@ public class AuthenticationMiddleware : IFunctionsWorkerMiddleware
ValidateIssuer = true,
IssuerValidator = (iss, token, parameters) =>
{
if (iss.StartsWith("https://login.microsoftonline.com/") ||
if (iss.StartsWith(Constants.EntraLoginBaseUrl + "/") ||
iss.StartsWith("https://sts.windows.net/"))
{
return iss;
@@ -159,7 +159,7 @@ public class McpQuotaEnforcementMiddleware : IFunctionsWorkerMiddleware
}
catch (Exception ex)
{
logger.LogWarning(ex, "[McpQuota] Failed to record usage: user={UserId}, endpoint={Endpoint}", oid, normalizedEndpoint);
logger.LogWarning(ex, "[McpQuota] Failed to record usage: user={UserId}, endpoint={Endpoint}", LogSanitizer.Clean(oid), LogSanitizer.Clean(normalizedEndpoint));
}
});
}
@@ -23,6 +23,15 @@ namespace AutopilotMonitor.Functions.Middleware;
/// </summary>
public class PolicyEnforcementMiddleware : IFunctionsWorkerMiddleware
{
/// <summary>
/// Throttle bucket for an authenticated token that carries no identifying claim at all (no upn,
/// oid, sub, appid or azp). Every real Entra token has at least a <c>sub</c>, so this is a
/// fail-closed backstop rather than a live code path: such callers share one bucket instead of
/// escaping the limiter. Deliberately NOT "anonymous" — that value is reserved for the log
/// rendering of unauthenticated calls and must never become a rate-limit key again.
/// </summary>
internal const string UnidentifiedCallerBucket = "unidentified-token";
private readonly ILogger<PolicyEnforcementMiddleware> _logger;
private readonly GlobalAdminService _globalAdminService;
private readonly DelegatedAdminService _delegatedAdminService;
@@ -57,6 +66,10 @@ public class PolicyEnforcementMiddleware : IFunctionsWorkerMiddleware
var httpMethod = httpContext.Request.Method;
var queryTenantId = httpContext.Request.Query["tenantId"].FirstOrDefault();
// Sanitized copies for logging only — never used for routing/authorization decisions.
var logPath = LogSanitizer.Clean(requestPath);
var logMethod = LogSanitizer.Clean(httpMethod);
// Authenticated principal set by AuthenticationMiddleware (null on anonymous/device routes).
// Read from FunctionContext.Items (reliable in the isolated worker) rather than httpContext.User.
ClaimsPrincipal? principal = null;
@@ -76,7 +89,7 @@ public class PolicyEnforcementMiddleware : IFunctionsWorkerMiddleware
}
catch (Exception ex)
{
_logger.LogError(ex, "[PolicyEnforcement] Service error evaluating policy for {Method} {Path}", httpMethod, requestPath);
_logger.LogError(ex, "[PolicyEnforcement] Service error evaluating policy for {Method} {Path}", logMethod, logPath);
httpContext.Response.StatusCode = (int)HttpStatusCode.ServiceUnavailable;
httpContext.Response.ContentType = "application/json";
httpContext.Response.Headers["Retry-After"] = "5";
@@ -92,13 +105,13 @@ public class PolicyEnforcementMiddleware : IFunctionsWorkerMiddleware
{
context.Items[RequestContext.ItemsKey] = result.Context!;
_logger.LogDebug("[PolicyEnforcement] ALLOW {Method} {Path} user={User} role={Role}",
httpMethod, requestPath, result.UserIdentifier, result.UserRole);
logMethod, logPath, LogSanitizer.Clean(result.UserIdentifier), result.UserRole);
await next(context);
return;
}
_logger.LogWarning("[PolicyEnforcement] DENIED {Method} {Path} status={Status} user={User} role={Role} reason={Reason}",
httpMethod, requestPath, result.StatusCode, result.UserIdentifier, result.UserRole, result.LogReason);
logMethod, logPath, result.StatusCode, LogSanitizer.Clean(result.UserIdentifier), result.UserRole, result.LogReason);
httpContext.Response.StatusCode = result.StatusCode;
httpContext.Response.ContentType = "application/json";
await httpContext.Response.WriteAsJsonAsync(new { error = result.ErrorCode, message = result.ErrorMessage });
@@ -115,12 +128,16 @@ public class PolicyEnforcementMiddleware : IFunctionsWorkerMiddleware
internal async Task<PolicyResult> DecideAsync(
string httpMethod, string requestPath, string? queryTenantId, ClaimsPrincipal? principal)
{
// Sanitized copies for logging only — never used for routing/authorization decisions.
var logPath = LogSanitizer.Clean(requestPath);
var logMethod = LogSanitizer.Clean(httpMethod);
var catalogEntry = EndpointAccessPolicyCatalog.FindPolicy(httpMethod, requestPath);
if (catalogEntry == null)
{
// Fail-closed: unregistered route → 403
_logger.LogError("[PolicyEnforcement] BLOCKED unregistered route: {Method} {Path} — not in catalog (fail-closed)",
httpMethod, requestPath);
logMethod, logPath);
return PolicyResult.Deny((int)HttpStatusCode.Forbidden, "Forbidden", "Access denied.", "anonymous", "N/A", "Unregistered");
}
@@ -188,7 +205,7 @@ public class PolicyEnforcementMiddleware : IFunctionsWorkerMiddleware
if (crossTenant && !hasGlobalScope && !delegatedGrantsRead)
{
_logger.LogWarning("[PolicyEnforcement] BLOCKED cross-tenant access: user={User} jwtTenant={JwtTenant} target={Target} scoping={Scoping} path={Path}",
decision.UserIdentifier, jwtTenantId, namedTarget, catalogEntry.TenantScoping, requestPath);
LogSanitizer.Clean(decision.UserIdentifier), LogSanitizer.Clean(jwtTenantId), LogSanitizer.Clean(namedTarget), catalogEntry.TenantScoping, logPath);
return PolicyResult.Deny((int)HttpStatusCode.Forbidden, "CrossTenantAccessDenied",
"Access denied. You can only access your own tenant's resources.",
decision.UserIdentifier, decision.UserRole, "CrossTenant");
@@ -206,7 +223,21 @@ public class PolicyEnforcementMiddleware : IFunctionsWorkerMiddleware
{
TenantId = jwtTenantId,
TargetTenantId = targetTenantId,
UserPrincipalName = decision.UserIdentifier,
// ONLY a genuine JWT UPN lands here — never the "anonymous" placeholder that
// decision.UserIdentifier carries on device/anonymous routes. The placeholder used to be
// published here, which collapsed the entire agent fleet (every tenant) into one shared
// user_ratelimit_anonymous bucket and 429'd agent telemetry fleet-wide. It stays on
// decision.UserIdentifier for log lines, where "anonymous" is the useful rendering.
UserPrincipalName = upn ?? string.Empty,
// Throttle identity — NOT the same thing as the UPN. Empty only when the request carried
// no JWT at all (device/anonymous routes, which bring their own limits). An authenticated
// caller ALWAYS gets a non-empty value, including app-only tokens that carry no upn:
// those authenticate fine and reach every AuthenticatedUser route, so keying the throttle
// off the UPN alone would let them through unlimited. Fail closed if a token somehow
// carries no identifying claim at all — a shared bucket is the safe answer there.
CallerId = principal is null
? string.Empty
: (principal.GetCallerId() ?? UnidentifiedCallerBucket),
IsGlobalAdmin = isGlobalAdmin,
IsGlobalReader = isGlobalReader,
// Delegated flags reflect the caller's role for THIS request (null unless a delegated grant
@@ -9,9 +9,14 @@ using Microsoft.Extensions.Logging;
namespace AutopilotMonitor.Functions.Middleware;
/// <summary>
/// Per-user rate limiting for authenticated (JWT) requests.
/// Runs after PolicyEnforcementMiddleware so RequestContext (UPN, IsGlobalAdmin) is already resolved.
/// Agent/device routes have no RequestContext and are automatically skipped.
/// Per-caller rate limiting for authenticated (JWT) requests.
/// Runs after PolicyEnforcementMiddleware so RequestContext (CallerId, IsGlobalAdmin) is already resolved.
/// Buckets on RequestContext.CallerId — the throttle identity, which is the UPN for a human caller and
/// the token subject (oid/sub/appid) for a token that carries none. Device/bootstrap and anonymous
/// routes resolve to an EMPTY CallerId and are skipped here; they carry their own limits (per cert
/// thumbprint / per bootstrap token in SecurityValidator, per client IP on the public endpoints).
/// Keying on CallerId rather than UserPrincipalName is what keeps BOTH holes shut: the agent fleet out
/// of one shared bucket, and app-only tokens inside a bucket at all.
/// </summary>
public class UserRateLimitMiddleware : IFunctionsWorkerMiddleware
{
@@ -45,9 +50,12 @@ public class UserRateLimitMiddleware : IFunctionsWorkerMiddleware
}
var requestContext = context.GetRequestContext();
if (string.IsNullOrEmpty(requestContext.UserPrincipalName))
if (string.IsNullOrEmpty(requestContext.CallerId))
{
// Anonymous or device-auth route — no user to rate limit
// No JWT on this request — device-auth or anonymous route. Those limit themselves
// (per cert thumbprint / bootstrap token in SecurityValidator, per client IP on the
// public endpoints). Gate on CallerId, NOT UserPrincipalName: a valid app-only token
// carries no upn, and skipping on that would exempt it from throttling entirely.
await next(context);
return;
}
@@ -96,13 +104,13 @@ public class UserRateLimitMiddleware : IFunctionsWorkerMiddleware
config.GlobalAdminRateLimitRequestsPerMinute,
entitlementFloor);
var key = $"user_ratelimit_{requestContext.UserPrincipalName.ToLowerInvariant()}";
var key = $"user_ratelimit_{requestContext.CallerId.ToLowerInvariant()}";
result = _rateLimitService.CheckRateLimit(key, limit);
}
catch (Exception ex)
{
_logger.LogError(ex, "[UserRateLimit] Rate limit check failed for user={User}, allowing request (fail-open)",
requestContext.UserPrincipalName);
_logger.LogError(ex, "[UserRateLimit] Rate limit check failed for caller={Caller}, allowing request (fail-open)",
requestContext.CallerId);
await next(context);
return;
}
@@ -127,8 +135,8 @@ public class UserRateLimitMiddleware : IFunctionsWorkerMiddleware
: 60;
_logger.LogWarning(
"[UserRateLimit] THROTTLED user={User} requests={Count}/{Max} retryAfter={RetryAfter}s",
requestContext.UserPrincipalName, result.RequestsInWindow, result.MaxRequests, retryAfterSeconds);
"[UserRateLimit] THROTTLED caller={Caller} requests={Count}/{Max} retryAfter={RetryAfter}s",
requestContext.CallerId, result.RequestsInWindow, result.MaxRequests, retryAfterSeconds);
httpContext.Response.StatusCode = (int)HttpStatusCode.TooManyRequests;
httpContext.Response.Headers["Retry-After"] = retryAfterSeconds.ToString();
@@ -104,7 +104,8 @@ builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
options =>
{
// Multi-Tenant Configuration
options.Instance = "https://login.microsoftonline.com/";
// Fully qualified: Microsoft.Identity.Web also exports a 'Constants' type.
options.Instance = AutopilotMonitor.Shared.Constants.EntraLoginBaseUrl + "/";
options.TenantId = "organizations"; // Accept tokens from any Azure AD tenant
options.ClientId = builder.Configuration["EntraId:ClientId"];
@@ -117,7 +118,7 @@ builder.Services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
ValidateIssuerSigningKey = true,
ValidIssuers = new[]
{
"https://login.microsoftonline.com/organizations/v2.0",
AutopilotMonitor.Shared.Constants.EntraLoginBaseUrl + "/organizations/v2.0",
"https://sts.windows.net/{tenantid}/"
},
// Temporarily accept Microsoft Graph tokens (used by frontend with User.Read scope)
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Linq;
using System.Net.Http.Headers;
@@ -119,7 +120,7 @@ namespace AutopilotMonitor.Functions.Security
// Use contains for server-side narrowing, then perform exact match client-side.
var escapedSerial = normalizedSerial.Replace("'", "''");
var filter = Uri.EscapeDataString($"contains(serialNumber,'{escapedSerial}')");
var graphUrl = $"https://graph.microsoft.com/v1.0/deviceManagement/windowsAutopilotDeviceIdentities?$top=100&$filter={filter}";
var graphUrl = $"{Constants.GraphBaseUrl}/v1.0/deviceManagement/windowsAutopilotDeviceIdentities?$top=100&$filter={filter}";
var response = await graphClient.GetAsync(graphUrl);
var responseBody = await response.Content.ReadAsStringAsync();
@@ -127,7 +127,7 @@ namespace AutopilotMonitor.Functions.Security
// Parse certificate from Base64
var certBytes = Convert.FromBase64String(certificateBase64);
var certificate = new X509Certificate2(certBytes);
var certificate = X509CertificateLoader.LoadCertificate(certBytes);
var thumbprint = certificate.Thumbprint;
logger?.LogDebug($"Validating certificate: Subject={certificate.Subject}, Issuer={certificate.Issuer}, Thumbprint={thumbprint}");
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Net.Http;
using System.Net.Http.Headers;
@@ -132,7 +133,7 @@ namespace AutopilotMonitor.Functions.Security
Encoding.UTF8,
"application/json");
var graphUrl = "https://graph.microsoft.com/beta/deviceManagement/importedDeviceIdentities/searchExistingIdentities";
var graphUrl = Constants.GraphBaseUrl + "/beta/deviceManagement/importedDeviceIdentities/searchExistingIdentities";
var response = await graphClient.PostAsync(graphUrl, content);
var responseBody = await response.Content.ReadAsStringAsync();
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Linq;
using System.Net.Http;
@@ -118,7 +119,7 @@ namespace AutopilotMonitor.Functions.Security
var escapedSerial = normalizedSerial.Replace("'", "''");
var filter = Uri.EscapeDataString($"contains(serialNumber,'{escapedSerial}')");
var orderby = Uri.EscapeDataString("preAssociationDateTime desc");
var graphUrl = "https://graph.microsoft.com/beta/deviceManagement/tenantAssociatedDevices"
var graphUrl = Constants.GraphBaseUrl + "/beta/deviceManagement/tenantAssociatedDevices"
+ $"?$top=25&$filter={filter}&$orderby={orderby}";
var response = await graphClient.GetAsync(graphUrl);
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
@@ -169,7 +170,7 @@ namespace AutopilotMonitor.Functions.Security
// while this POST is in flight, the generation changes and the token we cache below is
// tagged with the now-stale value — so readers reject it and re-mint fresh.
var genAtStart = CurrentGen(tenantId);
var tokenUrl = $"https://login.microsoftonline.com/{tenantId}/oauth2/v2.0/token";
var tokenUrl = $"{Constants.EntraLoginBaseUrl}/{tenantId}/oauth2/v2.0/token";
// Retry with backoff to handle Azure AD consent propagation delays.
// After admin consent is granted, the service principal may not be immediately
@@ -196,7 +197,7 @@ namespace AutopilotMonitor.Functions.Security
{
["client_id"] = clientId,
["client_secret"] = clientSecret,
["scope"] = "https://graph.microsoft.com/.default",
["scope"] = Constants.GraphBaseUrl + "/.default",
["grant_type"] = "client_credentials"
});
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Collections.Generic;
using System.Linq;
@@ -176,7 +177,7 @@ namespace AutopilotMonitor.Functions.Services.Analyze
return;
var session = await _sessionRepo.GetSessionAsync(envelope.TenantId, envelope.SessionId).ConfigureAwait(false);
var sessionUrl = $"https://portal.autopilotmonitor.com/sessions/{envelope.SessionId}";
var sessionUrl = $"{Constants.PortalBaseUrl}/sessions/{envelope.SessionId}";
foreach (var (result, rule) in candidates)
{
@@ -1,3 +1,5 @@
using AutopilotMonitor.Shared;
namespace AutopilotMonitor.Functions.Services;
/// <summary>
@@ -97,7 +99,7 @@ public static class EmailTemplates
<table cellpadding=""0"" cellspacing=""0"" style=""margin:0 auto 32px;"">
<tr>
<td style=""background-color:#2563eb; border-radius:8px;"">
<a href=""https://www.autopilotmonitor.com/docs"" target=""_blank""
<a href=""{Constants.DocsBaseUrl}"" target=""_blank""
style=""display:inline-block; padding:14px 32px; color:#ffffff; font-size:15px; font-weight:600; text-decoration:none;"">
View Documentation
</a>
@@ -147,9 +147,12 @@ namespace AutopilotMonitor.Functions.Services
helloResolved = true;
else if (Eq(type, "realmjoin_detected"))
realmJoinDetected = true;
// Either terminal opens the agent-side RealmJoin gate: phase 110 or the 60-min
// hard timeout (both dual-emitted to the timeline by the agent/engine).
else if (Eq(type, "realmjoin_resolved") || Eq(type, "realmjoin_timeout"))
// Any terminal opens the agent-side RealmJoin gate: phase 110, the aborted
// first deployment (phase left 100/101 for 200/210 without 110 — session
// 224b2087) or the 60-min hard timeout (all emitted to the timeline by the
// agent/engine).
else if (Eq(type, "realmjoin_resolved") || Eq(type, "realmjoin_timeout")
|| Eq(type, "realmjoin_first_deployment_incomplete"))
realmJoinResolved = true;
// Policy facts for the Hello-disabled / User-ESP-skipped completion mirror.
// Contradicting observations are resolved pessimistically below (both-seen →
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System.Linq;
using AutopilotMonitor.Functions.Services.Notifications;
using AutopilotMonitor.Shared.Models;
@@ -35,7 +36,7 @@ namespace AutopilotMonitor.Functions.Services
return;
var sessionUrl = updatedSession != null
? $"https://portal.autopilotmonitor.com/sessions/{request.SessionId}"
? $"{Constants.PortalBaseUrl}/sessions/{request.SessionId}"
: null;
// A failure alert requires an actual failure-ish verdict: an agent_timeout
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Collections.Generic;
using System.Globalization;
@@ -37,7 +38,7 @@ public sealed class ScriptDisplayNameResolver : IScriptDisplayNameResolver
/// </summary>
internal static readonly TimeSpan TokenAcquireBudget = TimeSpan.FromMilliseconds(1500);
private const string GraphBetaBase = "https://graph.microsoft.com/beta";
private const string GraphBetaBase = Constants.GraphBaseUrl + "/beta";
private const int PageSize = 100;
private readonly IGraphFeatureDetector _detector;
@@ -3,6 +3,7 @@ using System.Net.Http;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Shared;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
@@ -27,7 +28,10 @@ namespace AutopilotMonitor.Functions.Services
public sealed class LatestVersionsService : ILatestVersionsService
{
public const string VersionJsonUrl = "https://autopilotmonitor.blob.core.windows.net/agent/version.json";
// Download alias (Front Door → current blob origin) — NOT the legacy blob
// account. The legacy mirror in build-agent.yml is fail-soft, so reading from
// it could silently serve a stale version while the release pipeline is green.
public const string VersionJsonUrl = Constants.AgentDownloadBaseUrl + "/" + Constants.AgentVersionFileName;
private const string CacheKey = "latest-versions";
private static readonly TimeSpan CacheDuration = TimeSpan.FromHours(12);
private static readonly TimeSpan FailureCacheDuration = TimeSpan.FromMinutes(5);
@@ -527,6 +527,13 @@ namespace AutopilotMonitor.Functions.Services
/// the seed happens when the notification address is saved, so this converges to a
/// no-op and can be removed once every existing tenant has been covered.
/// </para>
/// <para>
/// The enumerated snapshot is only a cheap pre-filter; the authoritative "is it still
/// empty" check happens under an ETag inside
/// <see cref="TenantConfigurationService.TrySeedContactEmailAsync"/>. A full-model save
/// here would write a snapshot that is minutes old by the time the loop reaches this
/// tenant, silently reverting whatever an admin changed in between.
/// </para>
/// </summary>
/// <returns>Number of tenants that received a contact address.</returns>
private async Task<int> BackfillTenantContactEmailsAsync()
@@ -546,9 +553,8 @@ namespace AutopilotMonitor.Functions.Services
if (string.IsNullOrWhiteSpace(email))
continue;
config.ContactEmail = email.Trim();
await _tenantConfigService.SaveConfigurationAsync(config);
filled++;
if (await _tenantConfigService.TrySeedContactEmailAsync(config.TenantId, email))
filled++;
}
if (filled > 0)
@@ -1,4 +1,4 @@
using System;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
@@ -134,7 +134,6 @@ namespace AutopilotMonitor.Functions.Services
{
await MarkStalledSessionsAsTimedOutAsync();
await DetectExcessiveEventSessionsAsync();
await BlockExcessiveDataSendersAsync();
await AggregateMetricsWithCatchUpAsync();
// Plan §5 PR6 / §16 R14: session retention fanout extracted out of the 2h timer
// into the dedicated 12h SessionDeletionMaintenanceFunction so cascade-lifecycle
@@ -440,6 +439,13 @@ namespace AutopilotMonitor.Functions.Services
var timeoutEvent = BuildSessionTimeoutEvent(session, timeoutHours, sessionEvents, now);
await _sessionRepo.StoreEventsBatchAsync(new List<EnrollmentEvent> { timeoutEvent });
// Backend-materialized events bypass EventIngestProcessor and thus the
// cross-session EventType index — without this upsert session_timeout is
// invisible to every search-by-eventType surface (portal cross-session
// search, MCP search_sessions_by_event / query_raw_events).
await _sessionRepo.UpsertEventTypeIndexBatchAsync(
session.TenantId, session.SessionId, new List<EnrollmentEvent> { timeoutEvent });
// StoreEventsBatchAsync only bumps the orphan side-index, not the
// session's EventCount, and the terminal reconcile already ran inside
// UpdateSessionStatusAsync *before* this synthetic event existed. Recount
@@ -563,6 +569,16 @@ namespace AutopilotMonitor.Functions.Services
return null;
}
/// <summary>
/// Pure eligibility gate for the runaway-session auto-action path: only a session that
/// can still upload may cost its device a block. Mirrors the time-window watchdog's
/// <c>Status eq 'InProgress'</c> filter, so neither detector ever blocks a device over a
/// session that stopped sending — without it, a session that finished days ago with a
/// high EventCount still gets its device blocked on the next sweep.
/// The warn tier deliberately stays status-agnostic: it is forensic and blocks nothing.
/// </summary>
internal static bool IsAutoActionEligible(SessionStatus status) => status == SessionStatus.InProgress;
/// <summary>
/// Scans every tenant for sessions whose EventCount exceeds the configured warn
/// threshold (<see cref="AdminConfiguration.ExcessiveEventCountThreshold"/>) or the
@@ -580,6 +596,14 @@ namespace AutopilotMonitor.Functions.Services
/// mode mid-run never re-fires the warn.
/// </para>
/// Both paths skip when their threshold is 0; the whole sweep no-ops when both are off.
/// <para>
/// This is the ONLY automatic device block. A second, time-window detector used to run
/// beside it (AdminConfiguration.MaxSessionWindowHours) and blocked any session whose
/// first and last event lay more than N hours apart. It measured span, never volume, so
/// it flagged every enrollment that spanned a night — 23-event sessions included — and
/// was removed 2026-07-22. Event count is the quantity that actually costs anything;
/// if a span-based guard is ever reintroduced it has to gate on volume too.
/// </para>
/// </summary>
private async Task DetectExcessiveEventSessionsAsync()
{
@@ -637,6 +661,14 @@ namespace AutopilotMonitor.Functions.Services
var action = DecideAutoAction(session.EventCount, autoMode, autoThreshold);
if (action == null) continue;
if (!IsAutoActionEligible(session.Status))
{
_logger.LogInformation(
"Skipping auto-action for runaway session {SessionId} (tenant {TenantId}): status {Status} can no longer upload",
session.SessionId, tenantId, session.Status);
continue;
}
if (string.IsNullOrEmpty(session.SerialNumber))
{
_logger.LogWarning(
@@ -645,6 +677,19 @@ namespace AutopilotMonitor.Functions.Services
continue;
}
// Don't re-block an already-blocked device: BlockDeviceAsync would
// overwrite BlockedAt/UnblockAt/Reason of the existing block — most
// often one the time-window watchdog placed on the same session,
// whose reason then silently disappears from the Active Blocks list.
var (alreadyBlocked, _, _, _) = await _blockedDeviceService.IsBlockedAsync(tenantId, session.SerialNumber);
if (alreadyBlocked)
{
_logger.LogInformation(
"Skipping auto-action for runaway session {SessionId} (tenant {TenantId}): device {SerialNumber} is already blocked",
session.SessionId, tenantId, session.SerialNumber);
continue;
}
var reason = $"Auto-action: excessive session events ({session.EventCount} events ≥ threshold {autoThreshold})";
await _blockedDeviceService.BlockDeviceAsync(
tenantId,
@@ -680,108 +725,6 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// Detects devices that are still actively sending data beyond the configured maximum session
/// window and blocks them automatically. Status-independent: uses LastEventAt (written on every
/// event batch) combined with StartedAt to identify sessions spanning too long a data window.
/// Controlled by AdminConfiguration.MaxSessionWindowHours (0 = disabled).
/// </summary>
private async Task<int> BlockExcessiveDataSendersAsync()
{
_logger.LogInformation("Checking for excessive data senders...");
var sw = Stopwatch.StartNew();
try
{
var adminConfig = await _adminConfigurationService.GetConfigurationAsync();
if (adminConfig.MaxSessionWindowHours == 0)
{
_logger.LogInformation("Excessive data sender detection disabled (MaxSessionWindowHours = 0)");
return 0;
}
var windowCutoff = DateTime.UtcNow.AddHours(-adminConfig.MaxSessionWindowHours);
var blockDurationHours = adminConfig.MaintenanceBlockDurationHours;
var tenantIds = await _maintenanceRepo.GetAllTenantIdsAsync();
int totalBlocked = 0;
foreach (var tenantId in tenantIds)
{
try
{
var sessions = await _maintenanceRepo.GetExcessiveDataSendersAsync(tenantId, windowCutoff, adminConfig.MaxSessionWindowHours);
if (sessions.Count == 0)
{
_logger.LogInformation($"Tenant {tenantId}: No excessive data sender sessions found (window: {adminConfig.MaxSessionWindowHours}h)");
continue;
}
int blockedCount = 0;
foreach (var session in sessions)
{
if (string.IsNullOrEmpty(session.SerialNumber))
continue;
var (isBlocked, _, _, _) = await _blockedDeviceService.IsBlockedAsync(tenantId, session.SerialNumber);
if (isBlocked)
continue;
var reason = $"Excessive data window: session active for >{adminConfig.MaxSessionWindowHours}h " +
$"(started {session.StartedAt:yyyy-MM-dd HH:mm:ss} UTC, " +
$"last event {session.LastEventAt:yyyy-MM-dd HH:mm:ss} UTC)";
await _blockedDeviceService.BlockDeviceAsync(
tenantId,
session.SerialNumber,
durationHours: blockDurationHours,
blockedByEmail: "System.Maintenance",
reason: reason,
blockedSessionId: session.SessionId);
blockedCount++;
}
if (blockedCount > 0)
{
await _maintenanceRepo.LogAuditEntryAsync(
tenantId,
"ExcessiveDataBlock",
"Device",
$"{blockedCount} devices",
"System.Maintenance",
new Dictionary<string, string>
{
{ "DevicesBlocked", blockedCount.ToString() },
{ "WindowHours", adminConfig.MaxSessionWindowHours.ToString() },
{ "BlockDurationHours", blockDurationHours.ToString() }
});
_logger.LogInformation($"Tenant {tenantId}: Blocked {blockedCount} excessive data sender device(s) (window: {adminConfig.MaxSessionWindowHours}h, block: {blockDurationHours}h)");
await _opsEventService.RecordExcessiveDataBlockedAsync(tenantId, blockedCount, adminConfig.MaxSessionWindowHours);
}
totalBlocked += blockedCount;
}
catch (Exception ex)
{
_logger.LogError(ex, $"Failed to check excessive data senders for tenant {tenantId}");
}
}
sw.Stop();
_logger.LogInformation($"Excessive data sender check completed: {totalBlocked} devices blocked in {sw.ElapsedMilliseconds}ms");
return totalBlocked;
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to check for excessive data senders");
return 0;
}
}
/// <summary>
/// Aggregates metrics for any missed days in the last 7 days, plus yesterday.
/// Checks the UsageMetrics table for existing snapshots to avoid re-aggregation.
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Collections.Generic;
using System.Linq;
@@ -167,7 +168,7 @@ namespace AutopilotMonitor.Functions.Services.Notifications
},
Actions = new List<NotificationAction>
{
new() { Type = "openUrl", Title = "Open Autopilot Monitor", Url = "https://portal.autopilotmonitor.com" },
new() { Type = "openUrl", Title = "Open Autopilot Monitor", Url = Constants.PortalBaseUrl },
},
};
}
@@ -255,10 +255,9 @@ namespace AutopilotMonitor.Functions.Services
$"Device {serialNumber} blocked: {reason}",
tenantId, blockedBy, new { serialNumber, reason });
public Task RecordExcessiveDataBlockedAsync(string tenantId, int devicesBlocked, int windowHours)
=> WriteAsync(OpsEventCategory.Security, "ExcessiveDataBlocked", OpsEventSeverity.Warning,
$"{devicesBlocked} device(s) auto-blocked for excessive data (>{windowHours}h window)",
tenantId, "System.Maintenance", new { devicesBlocked, windowHours });
// ExcessiveDataBlocked was removed 2026-07-22 together with the time-window detector that
// raised it — it blocked on session span alone and fired on ordinary overnight enrollments.
// Automatic blocks now come only from the event-count path below.
public Task RecordVersionBlockedAsync(string pattern, string blockedBy)
=> WriteAsync(OpsEventCategory.Security, "VersionBlocked", OpsEventSeverity.Warning,
@@ -494,6 +493,20 @@ namespace AutopilotMonitor.Functions.Services
$"{sessionCount} session(s) timed out after {timeoutHours}h",
tenantId, "System.Maintenance", new { sessionCount, timeoutHours });
/// <summary>
/// An agent reported its absolute session-age emergency break (48h cap) over the
/// emergency channel — it cleaned itself up and exited on a session that never reached
/// a terminal state. This is the "are we silently losing agents?" signal; emitted by
/// <see cref="Functions.Ingest.ReportAgentErrorFunction"/> once per session (guarded by
/// the timeline-event idempotency check). Warning-tier so operators can wire a Telegram
/// rule; if real-world volume turns out noisy, downgrade or remove — the timeline event
/// on the session is the durable record.
/// </summary>
public Task RecordAgentEmergencyBreakAsync(string tenantId, string sessionId, string? agentVersion, string message)
=> WriteAsync(OpsEventCategory.Agent, "AgentEmergencyBreak", OpsEventSeverity.Warning,
$"Agent emergency break on session {sessionId} (agent {agentVersion ?? "?"}): {message}",
tenantId, "System.EmergencyChannel", new { sessionId, agentVersion });
public Task RecordExcessiveSessionEventsAsync(string tenantId, string sessionId, int eventCount, int threshold)
=> WriteAsync(OpsEventCategory.Agent, "ExcessiveSessionEvents", OpsEventSeverity.Warning,
$"Session {sessionId} has {eventCount} events (threshold {threshold}) — likely agent loop bug",
@@ -16,16 +16,19 @@ public class PreviewWhitelistService
private readonly IConfigRepository _configRepo;
private readonly IMemoryCache _cache;
private readonly ILogger<PreviewWhitelistService> _logger;
private readonly TenantConfigurationService _tenantConfigService;
private static readonly TimeSpan CacheDuration = TimeSpan.FromMinutes(5);
public PreviewWhitelistService(
IConfigRepository configRepo,
IMemoryCache cache,
ILogger<PreviewWhitelistService> logger)
ILogger<PreviewWhitelistService> logger,
TenantConfigurationService tenantConfigService)
{
_configRepo = configRepo;
_cache = cache;
_logger = logger;
_tenantConfigService = tenantConfigService;
}
/// <summary>
@@ -104,42 +107,15 @@ public class PreviewWhitelistService
/// <summary>
/// Saves (or clears) the notification email for a tenant, and seeds the tenant's
/// contact address from it the first time one is given.
/// <para>
/// The seed is a side effect of a write that has already been persisted, so it must never
/// fail this call — <see cref="TenantConfigurationService.TrySeedContactEmailAsync"/> is
/// fail-soft and owns the "never overwrite what the tenant owns" invariant.
/// </para>
/// </summary>
public async Task SaveNotificationEmailAsync(string tenantId, string? email)
{
await _configRepo.SaveNotificationEmailAsync(tenantId, email);
await SeedContactEmailAsync(tenantId, email);
}
/// <summary>
/// One-way seed: copies the preview notification address into
/// <see cref="TenantConfiguration.ContactEmail"/> only while that field is still empty.
/// Once the tenant owns a contact address — seeded or edited in the portal — this never
/// touches it again, so a later change here cannot silently overwrite the tenant's choice.
/// Best-effort: the notification email has already been persisted by the caller, and a
/// failure to seed must not fail that write.
/// </summary>
private async Task SeedContactEmailAsync(string tenantId, string? email)
{
if (string.IsNullOrWhiteSpace(email))
return;
try
{
var config = await _configRepo.GetTenantConfigurationAsync(tenantId);
if (config == null || !string.IsNullOrWhiteSpace(config.ContactEmail))
return;
config.ContactEmail = email.Trim();
await _configRepo.SaveTenantConfigurationAsync(config);
_logger.LogInformation(
"Seeded tenant contact address for {TenantId} from the preview notification email", tenantId);
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"Could not seed tenant contact address for {TenantId} — the notification email was still saved", tenantId);
}
await _tenantConfigService.TrySeedContactEmailAsync(tenantId, email);
}
}
@@ -1,3 +1,4 @@
using AutopilotMonitor.Shared;
using System;
using System.Collections.Generic;
using System.Diagnostics;
@@ -29,7 +30,7 @@ namespace AutopilotMonitor.Functions.Services
/// </summary>
public class SlaBreachEvaluationService
{
private const string DashboardUrl = "https://www.autopilotmonitor.com/sla";
private const string DashboardUrl = Constants.WebsiteBaseUrl + "/sla";
private const int MaxConflictRetries = 4;
private const int MinAppInstallSampleSize = 5;
@@ -1,4 +1,4 @@
using Azure;
using Azure;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Pagination;
@@ -345,8 +345,8 @@ namespace AutopilotMonitor.Functions.Services
}
}
// Suppress automated maintenance entries (SessionTimeout, ExcessiveDataBlock,
// DataRetentionCleanup, …) from the human-facing audit list. They are written
// Suppress automated maintenance entries (SessionTimeout, DataRetentionCleanup, …)
// from the human-facing audit list. They are written
// by MaintenanceService for traceability but duplicate the OpsEvents stream
// and are not actionable as user-attributable audits. Rows stay in storage
// and remain reachable via raw table queries.
@@ -792,73 +792,6 @@ namespace AutopilotMonitor.Functions.Services
}
}
/// <summary>
/// Gets sessions where the device has been actively sending data for longer than
/// <paramref name="maxSessionWindowHours"/>.
/// Status-independent: detects excessive data senders regardless of session status.
/// Uses LastEventAt (written on every event batch) for the "still active" check.
/// Sessions without LastEventAt (predating this field) are not returned.
///
/// The OData pre-filter narrows candidates to sessions that straddle the cutoff boundary,
/// then a post-filter verifies the actual session duration (LastEventAt − StartedAt)
/// exceeds the allowed window. This prevents false positives from short sessions that
/// merely happen to straddle the cutoff time.
/// </summary>
public async Task<List<SessionSummary>> GetExcessiveDataSendersAsync(string tenantId, DateTime windowCutoff, int maxSessionWindowHours)
{
SecurityValidator.EnsureValidGuid(tenantId, nameof(tenantId));
try
{
var tableClient = _tableServiceClient.GetTableClient(Constants.TableNames.Sessions);
// OData pre-filter: narrow to sessions that straddle the cutoff boundary.
// Status eq 'InProgress' → only block devices whose session is still actively
// sending data. Completed sessions (Succeeded/Failed) must NOT be re-blocked
// even if their LastEventAt is still within the window — they cannot continue
// to abuse data transfer. This also defends against ghost-blocks caused by
// devices with bad clocks: an agent that submits events with timestamps from
// weeks in the past pushes StartedAt back, making the session look long-lived;
// but once the session is Succeeded/Failed, no further data flows from it.
// IsPreProvisioned ne true → exclude WhiteGlove sessions: a pre-provisioned device
// that resumes after weeks in storage looks like an excessive sender (StartedAt old,
// LastEventAt recent) but is a legitimate resumption, not abuse.
var cutoffStr = windowCutoff.ToString("yyyy-MM-ddTHH:mm:ss");
var filter = $"PartitionKey eq '{tenantId}' " +
$"and Status eq 'InProgress' " +
$"and LastEventAt gt datetime'{cutoffStr}Z' " +
$"and StartedAt lt datetime'{cutoffStr}Z' " +
$"and IsPreProvisioned ne true";
var query = tableClient.QueryAsync<TableEntity>(filter: filter);
var maxDuration = TimeSpan.FromHours(maxSessionWindowHours);
var sessions = new List<SessionSummary>();
await foreach (var entity in query)
{
// Post-filter: verify actual session duration exceeds the window.
// OData cannot compute date differences, so we check in code.
var startedAt = entity.GetDateTimeOffset("StartedAt")?.UtcDateTime;
var lastEventAt = entity.GetDateTimeOffset("LastEventAt")?.UtcDateTime;
if (startedAt.HasValue && lastEventAt.HasValue
&& (lastEventAt.Value - startedAt.Value) < maxDuration)
{
continue;
}
sessions.Add(MapToSessionSummary(entity));
}
return sessions;
}
catch (Exception ex)
{
_logger.LogError(ex, $"Failed to get excessive data sender sessions for tenant {tenantId}");
return new List<SessionSummary>();
}
}
/// <summary>
/// Gets all known tenant IDs from the TenantConfiguration table (one "config" row per
/// tenant). This replaces an O(corpus) scan of the entire Sessions table — the maintenance
@@ -3005,6 +3005,26 @@ namespace AutopilotMonitor.Functions.Services
#region Excessive-Event Detection
/// <summary>
/// Status parse for the runaway-session rows. Deliberately stricter than
/// <c>MapToSessionSummary</c>, which reads a missing column as InProgress for legacy rows:
/// here a missing, empty, numeric or unknown value maps to <see cref="SessionStatus.Unknown"/>
/// so the auto-block/kill gate fails closed and no device is blocked over a session that
/// cannot be classified. The two explicit rejections exist because <c>Enum.TryParse</c> is
/// lenient in ways that would open the gate: it maps the string "0" straight onto
/// InProgress, and it ORs comma-separated names into a single unrelated member.
/// </summary>
internal static SessionStatus ParseStatusForAutoAction(string? statusString)
{
if (string.IsNullOrWhiteSpace(statusString)) return SessionStatus.Unknown;
if (!char.IsLetter(statusString[0])) return SessionStatus.Unknown;
if (statusString.Contains(',')) return SessionStatus.Unknown;
return Enum.TryParse<SessionStatus>(statusString, ignoreCase: true, out var status)
&& Enum.IsDefined(status)
? status
: SessionStatus.Unknown;
}
/// <summary>
/// Returns sessions in the given tenant whose EventCount exceeds the threshold.
/// Projects only the fields needed for the idempotency check — avoids full entity read.
@@ -3023,7 +3043,10 @@ namespace AutopilotMonitor.Functions.Services
// SerialNumber + ExcessiveEventsAutoActioned added for the auto-block/kill
// path in MaintenanceService — lets us decide and execute in one round-trip
// per qualifying session, without a follow-up GetSessionAsync.
select: new[] { "PartitionKey", "RowKey", "EventCount", "SerialNumber", "ExcessiveEventsAlerted", "ExcessiveEventsAutoActioned" });
// Status is mandatory for that path: SessionStatus.InProgress is ordinal 0, so
// an unselected column would leave every row looking in-progress and the
// auto-action eligibility gate silently always-true.
select: new[] { "PartitionKey", "RowKey", "EventCount", "SerialNumber", "Status", "ExcessiveEventsAlerted", "ExcessiveEventsAutoActioned" });
await foreach (var entity in query)
{
@@ -3033,6 +3056,7 @@ namespace AutopilotMonitor.Functions.Services
SessionId = entity.RowKey,
EventCount = SafeGetInt32(entity, "EventCount") ?? 0,
SerialNumber = entity.GetString("SerialNumber") ?? string.Empty,
Status = ParseStatusForAutoAction(entity.GetString("Status")),
ExcessiveEventsAlerted = entity.GetBoolean("ExcessiveEventsAlerted") ?? false,
ExcessiveEventsAutoActioned = entity.GetBoolean("ExcessiveEventsAutoActioned") ?? false
});

Some files were not shown because too many files have changed in this diff Show More