Compare commits

...

53 Commits

Author SHA1 Message Date
Oliver Kieselbach ec025686ce bump pointer: internal (hello wait timeout plan, D-301)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-30 15:38:30 +02:00
Oliver Kieselbach 821c41e2ba Tenant Hello wait timeout up to 1 hour (#167)
* feat: tenant Hello wait timeout up to 1 hour

The "Hello Wait Timeout" setting (30..300 s in the UI) had no effect in the
V2 agent: its only consumer, HelloTracker.StartHelloWaitTimer, has no
production caller, while the decision engine armed a fixed 300 s
hello_safety deadline after the ESP exit and the tracker ran a fixed 300 s
completion timer after the wizard start.

The setting is now the real Hello budget, clamped to 300..3600 s
(HelloWaitTimeout in AutopilotMonitor.Shared). Every value that was valid
before (<= 300, including the old 30 s default) keeps today's 5 minutes.

- Engine: the agent stamps helloWaitTimeoutSeconds on EnrollmentFactsObserved
  (only when > 300), recorded set-once in EnrollmentScenarioObservations.
  All six hello_safety arm sites share one builder with the tenant window;
  a fact that arrives after hello_safety was armed moves it out. The WDP
  backstop is never shorter than the tenant window; the signal census
  surfaces hello_wait_extended.
- Agent: the Hello tracker's wizard completion timer uses the same budget;
  the stall probe skips session_stalled while hello_safety is pending.
- Backend: ValidateModel rejects values outside 300..3600; stored rows are
  read as their effective value, so legacy 30 s rows never block unrelated
  saves or backup reverts. Default 30 -> 300; shared manifest and wire types
  regenerated.
- Web: input 300..3600, bounds pinned against the C# constants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S17pipjWqjZ8ULAu1JrFyG

* fix: keep the Hello wait's 30 s minimum and default, raise only the maximum

Product decision: the setting keeps its UI contract (30 s minimum and
default, same description); only the maximum moves to 3600 s. Configured
values up to 300 s keep the built-in 5-minute Hello window, larger values
extend it.

- HelloWaitTimeout: MinSeconds/DefaultSeconds 30; BuiltInSeconds 300 is the
  floor of EffectiveSeconds.
- Backend validates only a changed value (30..3600), like the retention cap;
  the read-time normalization is gone again.
- The agent stamps the tenant window only above BuiltInSeconds.
- Web keeps the original text and default; only the maximum and its hint
  change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S17pipjWqjZ8ULAu1JrFyG

* fix: Hello wait default 300 s, stale-fire guard, deferred stall report

Product decision: default 300 s (what V2 has always waited), minimum 30 s
still accepted, maximum 3600 s. Values up to 300 s keep the built-in
5-minute window, which the settings hint now says.

Review follow-ups:
- Engine: a hello_safety fire whose incarnation was replaced (armed due
  later than the fire's due time) is dead-ended, same guard as RealmJoin —
  a late tenant window could otherwise be undercut by the queued fire of
  the built-in deadline. The tenant fact is recorded only when it extends
  the built-in window.
- Stall probe: session_stalled is held back while a tenant-extended Hello
  wait (hello_safety or the WDP backstop) is pending and emitted once it is
  over and the session is still idle; real activity drops it. Default
  tenants never hit the guard.
- Comments: WDP backstop wording, 5-min reference, AgentConfiguration.
- Web pins the default against the shared manifest; tracker constructor
  defaults use HelloWaitTimeout.DefaultSeconds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S17pipjWqjZ8ULAu1JrFyG

---------

Co-authored-by: Claude <noreply@anthropic.com>
2026-09-30 15:36:36 +02:00
Oliver Kieselbach ec3eaa6437 fix(web): sitemap and llms.txt list public pages at their canonical /page/ URL (#166) 2026-09-29 14:13:59 +02:00
Oliver Kieselbach d36c9a659d Merge branch 'main' of https://github.com/okieselbach/AutopilotMonitor 2026-09-29 12:02:55 +02:00
Oliver Kieselbach c27dbf2aa2 bump pointer: internal (client role gates plan)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:46:31 +02:00
Oliver Kieselbach 28443d11fa fix(web): don't send SignalR joins and bootstrap reads the backend refuses by role
Every visit by a user without a tenant role joined the tenant broadcast
group before the dashboard redirected to /progress, and the settings
area loaded the bootstrap session list for every role. The backend
refused both by rule: 134 AddToGroup 403s, 19 serial-less session
joins and 27 bootstrap-list 403s in 30 days, each one also an
api_request_failed event.

- lib/signalrGroupAccess mirrors the backend's same-tenant join rules
  (broadcast and member-notify groups need an own role or platform
  scope, the admin-notify group the Admin or a GA, a session group a
  serial for roleless callers, global-admins platform scope, an
  unparseable name is a 400). Cross-tenant joins stay with the
  backend's platform/delegated admission. The gate keys on
  hasOwnTenantOrPlatformRole, not hasTenantReadScope: a delegated admin
  without an own role is refused the home tenant's groups too.
- SignalRContext.joinGroup applies it before sending: a refused join
  reaches onDenied with the server's status and is never held for the
  reconnect rejoin. joinGroup depends on the caller's role fields as
  primitives, so its identity changes only when auth/me resolves, the
  same commit that delivers the tenant id consumers join with. That
  also ends the "tenant-" join the dashboard sent before the tenant id
  was known (the 400s).
- TenantConfigContext loads bootstrap sessions only for
  canManageBootstrapSessions (mirror of BootstrapManagerOrGA: GA, Admin,
  Operator with the bootstrap grant).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:46:05 +02:00
Oliver Kieselbach 0c41b0f449 bump pointer: .claude/commands (ux-insights rejoin baseline)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:22:44 +02:00
Oliver Kieselbach 7799fdb725 bump pointer: .claude/commands (platform-health baselines 2026-09-29)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 11:04:44 +02:00
dependabot[bot] 0c1a2a4192 chore(deps-dev): bump tsx (#165)
Bumps the mcp group in /src/McpServer/autopilot-monitor-mcp with 1 update: [tsx](https://github.com/privatenumber/tsx).


Updates `tsx` from 4.23.13 to 4.23.15
- [Release notes](https://github.com/privatenumber/tsx/releases)
- [Changelog](https://github.com/privatenumber/tsx/blob/master/release.config.cjs)
- [Commits](https://github.com/privatenumber/tsx/compare/v4.23.13...v4.23.15)

---
updated-dependencies:
- dependency-name: tsx
  dependency-version: 4.23.15
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: mcp
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-09-29 11:00:30 +02:00
Oliver Kieselbach 0356770ae8 bump pointer: internal (operator group customer label, D-300)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:58:39 +02:00
Oliver Kieselbach 3a9a074be1 Operator groups can carry a customer label; customers still never see the group name
Under D-267 a managed tenant saw every operator-provisioned access as
"Platform support". A customer in two operator groups run by different teams
got two identical entries under "Who can read this tenant" and could not tell
them apart.

A Global Admin can now give an operator group a customer label (at most 80
characters) with the existing PATCH global/tenant-groups/{groupId}, which takes
only the fields to change: name and/or customerLabel, where an empty label
clears it. The customer sees the label as the entry name; a group without one
and direct grants keep "Platform support". Owned (self-service) groups refuse a
label with 409, because their customers see the managing tenant's name.
RenameTenantGroupAsync becomes UpdateTenantGroupAsync, one read-modify-write on
the meta row.

DelegationSelfService.CustomerNameOf is now the one source of the name a
customer sees for a group. The managers view and the Group detail of the audit
rows written under every member tenant (assign, unassign, group delete) both use
it. Those audit rows carried the internal group name to the customer's audit
page before this change.

Admin, Tenant Groups: operator groups show "Customers see: ..." and get a
"Customer label" button next to Rename. The customer page is unchanged.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 10:58:25 +02:00
Oliver Kieselbach a3870e7dcc bump pointer: internal (pre-provisioning seal + registration failure reason live)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 01:59:09 +02:00
Oliver Kieselbach 8cb5d42edc Seal pre-provisioning despite IME's AccountSetup line; report why a session never registered
WhiteGlove seal: IME reports "EspPhase: AccountSetup" from the device session
as soon as the device apps are done when the user ESP is enabled. The sealing
classifier counted that as user activity (-40) and left pre-provisioning Part 1
unsealed, so no whiteglove.complete marker was written and the Part-2 boot hit
the 48-hour emergency break. The excluder now also needs the ESP registry to
show AccountSetup progress: the adapter stamps accountSetupProgress on
WhiteGloveShellCoreSuccess, only after the tracker has actually read the
AccountSetup subcategories (unread is never "untouched"). The new excluder is a
subset of the old one, so a verdict can only rise; signals without the key
score as before. WhiteGloveShellCoreSuccess v1 gets its replay fixture.

The Part-1 exit writes whiteglove.complete as its first step instead of after
analyzers, diagnostics upload, grace and drain, where a reseal could kill the
process before it.

Registration failures: a run that exits because register-session failed
records outcome, last error, network link at start and end, the wall-clock
attempt window and the config-fetch outcome in registration-failure.json. The
emergency break sends the record as AgentErrorReport.PriorRegistrationFailure
(null for older agents); the backend bounds it and adds it to the ops details,
the App Insights dimensions and the timeline data. Retry behaviour is unchanged.

The raw registry trigger whiteglove_signal_* is renamed save_result_step_*: the
step exists in every 25H2 enrollment and the old name read as a
pre-provisioning signal.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 01:38:45 +02:00
Oliver Kieselbach fa637369e1 bump pointer: internal (emergency break session age live)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 00:05:44 +02:00
Oliver Kieselbach edf30d35e0 Emergency break reports the session age; a break without a session row is Info
The agent measured the session age at its 48 h break but never sent it.
AgentErrorReport now carries SessionAgeHours (optional, null for older
agents), and the backend puts it into the ops details, the timeline data and
the App Insights dimension. It is information only: the backend cannot derive
it (no row, or a row registered seconds before the break after a hibernate
resume), and it never feeds the severity.

The backend now reads the Sessions row once for the owner binding and the
break verdict. A report for a session without a row (in practice one that
never registered) is Info and writes no timeline event and no EventType index
row: nothing can close or open that session, and the orphan sweep deleted the
event a day later with a Warning of its own. A row under cascade delete gets
nothing written past the lock. Terminal stays Info with lateCleanup; an open
status or a failed read stays Warning with the historical write. Break
materialization now requires GUID ids.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 23:40:36 +02:00
Oliver Kieselbach 3b4228e3a1 bump pointer: internal (session-owner binding stage 2 live)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 23:38:45 +02:00
Oliver Kieselbach a029d69d5e Session-owner binding stage 2: refuse sessions bound to another device
Each Sessions row is bound to the device identity that registered it. The
binding ran in shadow; it now refuses every foreign-identity outcome:
register-session answers 403 with session_owner_mismatch before anything is
written, agent telemetry 403 in the error envelope (SessionOwnerMismatch),
the hosted diagnostics upload URL 403. The error channel stays 200 and only
skips its write into the named session. An evaluation defect fails open.

The agent already rotates its session id on the 403 at registration; it now
does the same for the coded 409 (session id claimed by another tenant) instead
of retrying six times and exiting, and fails without retry on any other coded
409 (BackendRefusalException, never an auth failure). A snapshot restored
after a rotation adopts the current session id.

A guard test pins the four call sites and fails when a site observes the
binding without acting on the refusal. Privacy page: session binding under
how data is protected.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 23:12:18 +02:00
Oliver Kieselbach daae1ce1fa bump pointer: internal (paged cross-partition merge released)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 21:02:35 +02:00
Oliver Kieselbach d90cf24508 bump pointer: internal (D-294/D-295 paged cross-partition merge)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:53:35 +02:00
Oliver Kieselbach 32f1fd4132 Click-next lists change page only after the page loaded
The audit log, ops events, rule submissions and session reports each
carried a copy of the same paging state. Next bumped the page number and
cursor stack before the request and kept them when it failed, so a failed
request showed "Page 3" over the rows of page 1.

useCursorPager (pure transitions in lib/cursorPager) owns rows, cursors
and the stack; the page number is derived from the stack. A move commits
only after its page loaded, and only the latest move commits. The fetch is
read through useLatest, which retires the per-page workarounds for the
token getter's changing identity.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:53:35 +02:00
Oliver Kieselbach ffb09e60fa Global audit and ops-event pages continue from one RowKey watermark
The cross-partition pages (audit log over all tenants, ops events over
all categories) kept one continuation entry per partition that still had
rows. On the tenant fan-out the nextLink outgrew the host's URL limit:
the follow-up request was rejected with HTTP 414 before the function ran,
so Next never loaded a second page for a Global Admin.

The continuation is now the (RowKey, Partition) of the last returned row.
Every page reads each partition through BoundedFanOut (pageSize + 1 rows
after the watermark) and merges by (RowKey, Partition), which is the
display order in both tables. Partitions after the watermark's partition
continue with `RowKey ge`, the others with `RowKey gt`, so equal ops
RowKeys across categories survive a page cut. The merge no longer sorts
by the resolved OccurredUtc while advancing by RowKey, which could skip
rows where the two disagree.

The fan-out was unbounded (Task.WhenAll over every tenant); it now runs
with the shared cross-tenant concurrency ceiling like the session list.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 20:53:35 +02:00
Oliver Kieselbach f920b6228e fix(web): suspended and offboarded tenants get their page again (#164) 2026-09-28 18:23:28 +02:00
Oliver Kieselbach 1765773116 bump pointer: internal (RealmJoin app install rows released)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 18:21:38 +02:00
Oliver Kieselbach 5c7295019b bump pointer: internal (D-292/D-293 RealmJoin app install rows)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:19:18 +02:00
Oliver Kieselbach a0ad769231 RealmJoin packages count as app installs next to Intune apps
App install rows now carry their install channel. The ingest folds the RealmJoin
watcher's package started/completed events into AppInstallSummaries rows (channel
"realmjoin", keyed by scope + package id; duration = completion minus the observed
start). IME rows keep their key and have no channel column, so they read as before.

Every per-app aggregate groups on (channel, name): apps list, app analytics and
sessions (new ?source=, default ime), metrics/app, SLA top failing apps and the
version-regression radar (tracker key, episode key, bell link, ops event). The app
install SLA rate includes RealmJoin packages. The ESP-blocking stamp stays IME-only.
RealmJoin packages do not feed the software inventory or CVE correlation.

Web: a RealmJoin pill and an All / Intune / RealmJoin filter on Software > Installs,
the pill on app detail, Fleet Health and SLA, source-aware app links and PDF meta
line, and a hint in the empty Inventory tab. The pill moves out of InstallProgress
into a shared component. MCP: get_app_install_metrics names the source field.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 10:19:18 +02:00
Oliver Kieselbach 64da9f850b bump pointer: internal (D-291 chain build without downloads)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:07:10 +02:00
Oliver Kieselbach 5dae21968b Client-certificate chain building never downloads an issuer
The platform forwards whatever certificate the TLS client presented, so an
unauthenticated caller can hand in a leaf signed by a CA it made up. The
chain engine then fetched the leaf's AIA caIssuers URL before rejecting it:
an attacker-directed GET from the backend, and a worker blocked for the
10 s retrieval timeout. Every legitimate issuer is embedded and Intune
certificates carry no AIA, so downloads are now disabled and the unused
retrieval timeout is gone.

The regression test signs a leaf with a CA that is neither embedded nor
sent and asserts that the listener behind its AIA URL sees no connection;
a self-signed leaf never reaches that path.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-28 09:06:09 +02:00
Oliver Kieselbach 59897dfbd8 bump pointer: internal (B-g9t editor verify)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 20:33:30 +02:00
Oliver Kieselbach f4551ea1ed bump pointer: internal (D-288/D-289/D-290 live)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 20:21:58 +02:00
Oliver Kieselbach e816c92b88 Tenant configuration has one conditional write path; the full-model PUT is gone
A tenant configuration row is never written blind any more (D-290). Three writers saved
from the 5-minute instance cache and four wrote a fresh read without an ETag: auto-re-enable
could lift a suspension a Global Admin had just renewed, the PUT could write the cached
Disabled=false over the offboarding tombstone during the drain barrier, and a concurrent
first login could overwrite the write-once OnboardedBy.

- TenantConfigurationService.CreateOrUpdateAsync next to UpdateAsync (shared core): fresh
  read with ETag, tombstone refused, mutation decides, conditional replace, conditional
  insert for a missing row, re-read and re-decide on a lost race.
- SaveConfigurationAsync and IConfigRepository.SaveTenantConfigurationAsync deleted; new
  TryCreateTenantConfigurationAsync. The default row of GetConfigurationAsync is a
  conditional insert.
- Converted: first-login seed (writes only while the stored row has no domain; only the
  winning login fires the signup side effects), auto-re-enable (decided on the fresh row),
  auth client-id tracking, app-homing flip, conferred-Pro grace stamp, offboarding tombstone.
- PUT/POST config/{tenantId} removed (function, catalog entries, response DTO). The Global
  Admin tenant editor sends only its changed fields to the field PATCH; the Collect Logs
  quick-config PATCHes its two diagnostics fields with ?intent=collect-logs, which keeps
  the CollectLogsQuickConfigEnabled ops event. Shared diff helper lib/tenantConfigSave.ts.
- Tests: create-or-update and default-row races, concurrent first login, renewed
  suspension, tombstone paths; repository and service doubles in ConfigRepoMockExtensions.
  Regenerated shared manifest and wire types (web, MCP).

Deploy order: backend, then web right after (the old web's editor save and quick-config
use the removed PUT).

Bumps internal (D-290, plan, docs).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 20:11:56 +02:00
Oliver Kieselbach a16a70fba5 Self-service trial stays once per tenant across offboarding; plan writers decide on the fresh row
Offboarding deletes the tenant configuration, and the returning tenant's first login recreated
it with TrialConsumed=false, so the one-time 30-day Pro trial could be chained by offboarding
and re-onboarding.

- The offboard click reads the configuration fail-loud and ORs TrialConsumed into the
  OffboardingByTenantPointer (sticky across rounds, same ETag CAS as OffboardCount).
- The first-login write (HandleNewTenantDomainAsync, the one write that sets OnboardedBy)
  carries it onto the new row; an unreadable pointer means no write, retried next login.
  Re-onboarding itself is unchanged, auto-approve included.
- TenantConfigurationService.UpdateAsync: fresh projected row with ETag, offboarding
  tombstone refused (403 TenantSuspended), mutation may decline, conditional replace with
  the ordinary pre-write backup, re-read and re-decide on a lost race (409 after 3).
  SetTenantPlanTier and StartTrial use it, so a warm instance's cached row is never saved
  (it could lift the tombstone during the drain barrier) and two concurrent trial starts
  cannot both pass.
- Tests: TenantConfigUpdateTests (service + both endpoints), pointer stickiness and the
  offboard endpoint in TenantOffboardFunctionTests, first-login carry-over in
  AuthFunctionSideEffectTests; shared EndpointHarness.

Bumps internal (D-288, D-289, plan, docs).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 19:08:34 +02:00
Oliver Kieselbach 1b2bdd8f35 bump pointer: internal (SSRF gate released)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 13:43:22 +02:00
Oliver Kieselbach 942bd86238 bump pointer: internal (SSRF gate live-verify)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 13:08:24 +02:00
Oliver Kieselbach 7a920049ca bump pointer: internal (D-287 Node/Next version plan)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 12:48:30 +02:00
Oliver Kieselbach 2eeb4da950 bump pointer: internal (B-h4m CodeQL note)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 12:38:08 +02:00
Oliver Kieselbach 20a80cbdc6 bump pointer: internal (SSRF address gate D-286, B-h4m)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 12:37:03 +02:00
Oliver Kieselbach c59e65ee0a fix(backend): webhook SSRF gate moves onto the connection
SsrfGuard.ValidateDestinationAsync resolved the webhook host and checked
it, then HttpClient resolved it again for the POST, so a DNS-rebinding
host could pass the check and be connected to an internal address.

The webhook client's SocketsHttpHandler now connects through
SsrfGuard.ConnectToPublicAddress: it resolves once per connection,
refuses the connection if any address is blocked, enforces HTTPS and
connects the socket only to the checked addresses; TLS still runs against
the request host. The pre-flight check is removed, the registration moves
to AddWebhookNotificationHttpClient so a test can prove the wiring, the
notification retry policy no longer retries a refused destination, and
the test endpoint reports the refusal with the same message as before.

IsBlockedAddress follows the same model as the MCP server: all IANA
special-purpose space, IPv6 only 2000::/3, zone ids and IPv4-mapped
addresses refused (IPNetwork.Contains judges a mapped address by its
IPv4 form, so those are refused first).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 12:36:06 +02:00
Oliver Kieselbach 89aa9e1f10 fix(mcp): CIMD fetch checks the address inside the socket's own DNS lookup
The SSRF gate resolved the client_id host, checked the answer and then
handed the hostname to fetch(), which resolved it again at connect time.
A DNS-rebinding host could pass the check with a public answer and be
connected to an internal one.

The metadata fetch now runs through a dedicated undici dispatcher whose
connect.lookup is the gate: one resolution per connection, every address
must be public, the socket connects only to those addresses. The global
fetch is untouched.

The address classifier now refuses every IANA special-purpose block
(IPv4), allows only global unicast 2000::/3 minus its special-purpose
blocks (IPv6), and refuses zone ids and IPv4-mapped addresses. The tests
enumerate the unedited IANA registries (tests/fixtures/
iana-special-purpose-addresses) and prove on loopback sockets that a
refused host opens no socket and that an allowed one is connected to
exactly the address the gate returned.

Adds undici ^8.11.2 (engines >=22.19.0).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-27 12:35:56 +02:00
Oliver Kieselbach 2d3700cfaf bump pointer: internal (B-5p4 release state)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 17:16:02 +02:00
Oliver Kieselbach 5a182f3f16 bump pointer: internal (B-5p4 DO-geo deploy/verify)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 17:10:32 +02:00
Oliver Kieselbach 889b843d6d feat(agent): Delivery Optimization geo endpoint as last-resort geolocation provider
When ipinfo.io (plus retry) and ifconfig.co both fail, the agent now asks
geo.prod.do.dsp.mp.microsoft.com/geo. It returns only CountryCode and
ExternalIpAddress, but it belongs to the DO/Windows Update endpoint set Autopilot
needs anyway, so it stays reachable where public geo services are blocked.
The session then gets a country-only device_location (source "do-geo", no
timezone, so timezone auto-set stays pending) instead of none.

- GeoLocationService: the three providers share one TryProvider skeleton;
  response parsers are internal and unit-tested; GeoLocationResult.Describe()
  skips missing parts ("DE" instead of ", , DE") in the event message and log.
- Failure trace carries doGeoError + lastResortProvider.
- Backend: geographic location keys skip empty region/city, so country-only
  sessions group as "DE" rather than ", DE"; drilldown still round-trips.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 17:08:56 +02:00
Oliver Kieselbach 9b91119239 bump pointer: internal (way 3 docs live)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 14:58:33 +02:00
Oliver Kieselbach 1e74d67da9 fix(web): service-principal member form names both uses — automation (access_as_application) and an app sending its users' tokens (delegated access_as_user, users read-only)
The ClientAppNotRegistered refusal (D-279, now enforced) sends Tenant Admins
to this form; its hint required access_as_application, which an app acting
on behalf of users does not need.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 14:51:52 +02:00
Oliver Kieselbach ed013bbdda bump pointers: internal (hash repair via v2.0.1466), .claude/commands (platform-health repair note)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 14:51:37 +02:00
Oliver Kieselbach 4b40619d1d bump pointer: internal (client-app binding enforced, way 3 documented)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 14:48:21 +02:00
Oliver Kieselbach 3ce1f2b036 bump pointer: internal (D-285 plan live)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 14:06:30 +02:00
Oliver Kieselbach 673f045212 fix(web): platform settings saves send only the changed fields via PATCH global/config (D-285)
Each save handler still builds the edited configuration but now sends changedAdminConfigFields(loaded, edited) through lib/adminConfigSave.ts, and nothing at all when nothing changed. This covers the Global Settings, the diagnostics paths, the ops alerts and the vulnerability section. The MCP access policy patches its single field without reading first. A page loaded before another writer (the agent release pipeline, a second operator, a sync job) can no longer revert that writer's fields. Deployed after the backend that serves PATCH global/config.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 13:55:43 +02:00
Oliver Kieselbach df640e3684 fix(backend): admin configuration is written only in its changed columns; PATCH global/config replaces the full-model PUT (D-285)
On 2026-09-25 a Global Settings save from a page loaded before an agent release wrote the whole configuration back and reverted the release's LatestAgentV2* hashes. After that, every 2.0.1465 agent reported IntegrityCheckFailed. The flaw was structural: every writer (portal PUT, plan tiers, three vulnerability-sync stamp paths) replaced the whole row from the copy it held, usually the 5-minute cache.

- IConfigRepository.UpdateAdminConfigurationAsync(mutate, updatedBy) is the one write path. It reads the row fresh with its ETag and applies the mutation to a copy materialized from that read. It computes the changed columns from the converter projections and writes only those onto the freshly read raw entity; a null value drops the column. The replace is conditional, and a conflict triggers a re-read and retry. A no-op writes nothing. Pipeline-owned and unknown columns survive. The full-model SaveAdminConfigurationAsync / SaveConfigurationAsync are removed, and the first row is created with AddEntity.
- PATCH global/config with { fields } (PatchAdminConfigurationRequest) replaces PUT/POST. Its field gate is derived from the persisted columns and rejects server stamps, LatestAgentV2* / LatestBootstrapV2ScriptVersion, PlanTierDefinitionsJson, the sync stamps and redacted placeholders. The route catalog lists PATCH.
- Plan tiers, TriggerMsrcSync, TriggerVulnerabilityDataSync, VulnerabilityDataSyncFunction and VulnerabilityCacheRefreshService.StampAsync each hand in a mutation.
- Tests: AdminConfigurationChangedFieldsTests (stale-page toggle keeps the release hashes, concurrent pipeline write re-read and kept, no-op, rejection, null drop, first row, gate); existing tests adapted; the request guard allows the admin field map.
- Regenerated shared manifest and wire types (web utils + MCP). The web still sends PUT until the web commit that follows the backend deploy.
- The internal pointer is bumped to D-285.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 13:51:53 +02:00
Oliver Kieselbach 59bea9d888 bump pointers: internal (plan envelope live), .claude/commands (platform-health markers)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 13:11:39 +02:00
Oliver Kieselbach e2c91afda2 feat(mcp): seal refresh tokens of registered self-hosted clients so that deleting the registration also ends copies (D-283)
A portal-registered client (amc_) now receives its Entra refresh token only as a compact JWE (RFC 7516, dir + A256GCM, kid amc-rt-v1). The key is derived from OAuthStateSigningKey. At refresh, the registration comes from the envelope, never from the client_id parameter. Before, a copy of the token redeemed without client_id went to the organizations authority and skipped the registration check.

A raw refresh token presented with an amc_ client id is accepted and returned sealed, so these clients migrate on their next refresh. All other clients are unchanged. jose becomes a production dependency; it was a dev-only transitive dependency before. The internal pointer is bumped to D-283/D-284.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 13:00:42 +02:00
Oliver Kieselbach eb47c4d69a bump pointer
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 12:32:14 +02:00
Oliver Kieselbach 91d56f2bbb bump pointer
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 11:58:20 +02:00
Oliver Kieselbach 12b847360c feat(web): Intune Enrollment Validation in the tenant settings; banners follow any device validation (D-282)
The option is a regular tenant-admin toggle in its own "Without pre-registration" block, with the
add-on permission state read from the existing graph-permissions status endpoint. The dashboard
banner and the admin Ready badge key on "any validation method enabled" (feature flag
deviceValidationEnabled, manifest-driven validate* fields) instead of Autopilot alone, so a
device-preparation tenant that validates by device association is no longer told that agent
ingestion is blocked. Stale "shadow mode" wording for Device Association is gone, the
Not-registered report names every method, sessions show "Intune Enrollment", and the privacy
page and FAQ list the new method. Grant script help describes the IntuneDeviceBinding feature
as Intune Enrollment Validation.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-25 11:57:42 +02:00
272 changed files with 10180 additions and 2909 deletions
@@ -29,10 +29,9 @@
Cloud PCs are never Autopilot-registered, so this enables agent monitoring
of Cloud PC first-connect enrollment)
- IntuneDeviceBinding -> DeviceManagementManagedDevices.Read.All
(cert-to-device binding: verifies that the Intune device id carried in the
agent's client certificate is a device THIS tenant actually enrolled --
currently a Global-Admin-only preview that records telemetry and does not
block enrollment)
(Intune Enrollment Validation: accepts devices enrolled in THIS tenant's
Intune, matched by the Intune device id in the agent's client certificate,
without Autopilot registration, corporate identifier or device association)
- All -> every optional permission above, in one run
(also works with -Revoke to remove all optional grants at once)
@@ -76,15 +75,17 @@
.NOTES
Requires:
- Microsoft.Graph.Authentication PowerShell module (auto-installs if missing).
- Sign-in as Global Administrator OR Privileged Role Administrator OR Cloud Application Administrator.
- Sign-in as Global Administrator OR Privileged Role Administrator (with PIM, activate the
role first). Only these roles may assign Microsoft Graph application permissions;
Application Administrator and Cloud Application Administrator are not enough.
- The Autopilot Monitor app must be admin-consented in your tenant first
(its service principal must exist before grants can be added).
Sign-in behaviour:
In Azure Cloud Shell the script signs in via the ambient identity endpoint (a delegated
token for the already-signed-in user) - no device-code prompt, which Conditional Access
policies commonly block. Outside Cloud Shell it falls back to a normal interactive
Microsoft Graph sign-in. -TenantId only applies to the interactive fallback.
- The Autopilot Monitor app must be admin-consented in your tenant first
(its service principal must exist before grants can be added).
After granting:
The Autopilot Monitor backend may cache an older token for up to ~1 hour. Use the
@@ -140,6 +141,36 @@ if ($PSCmdlet.ParameterSetName -eq 'ByFeatures') {
# ---- Constants ----
$MicrosoftGraphAppId = '00000003-0000-0000-c000-000000000000'
# ---- Graph failure details ----
# Invoke-MgGraphRequest puts only "Response status code does not indicate success: ..." into the
# exception message. Graph's own error (code + message) is the response body in ErrorDetails,
# behind the request line, the status line and the headers. Some headers carry JSON themselves
# (x-ms-ags-diagnostic), so the body is taken from behind the blank line that ends the headers.
function Get-GraphFailure {
param([System.Management.Automation.ErrorRecord]$ErrorRecord)
$status = 0
if ($ErrorRecord.Exception.PSObject.Properties['Response'] -and $ErrorRecord.Exception.Response) {
$status = [int]$ErrorRecord.Exception.Response.StatusCode
}
$text = $ErrorRecord.Exception.Message
if ($ErrorRecord.ErrorDetails -and $ErrorRecord.ErrorDetails.Message) {
$body = ($ErrorRecord.ErrorDetails.Message -split '\r?\n\r?\n', 2)[-1].Trim()
if ($body.StartsWith('{')) {
try {
$graphError = ($body | ConvertFrom-Json).error
if ($graphError.code) { $text = "$($graphError.code): $($graphError.message)" }
}
catch {
# Not a Graph JSON body - keep the exception message.
}
}
}
[pscustomobject]@{ Status = $status; Text = $text }
}
# ---- Pre-flight: module ----
if (-not (Get-Module -ListAvailable -Name Microsoft.Graph.Authentication)) {
Write-Host "[Setup] Microsoft.Graph.Authentication not installed. Installing for current user..." -ForegroundColor Yellow
@@ -189,7 +220,7 @@ Write-Host "[Auth] Granted scopes: $($ctx.Scopes -join ', ')"
# carries a fixed scope set (incl. Directory.AccessAsUser.All) instead of the requested scopes,
# and its effective rights follow the signed-in user's directory role.
if (-not $identityConnected -and -not ($ctx.Scopes -contains 'AppRoleAssignment.ReadWrite.All')) {
Write-Warning "Scope 'AppRoleAssignment.ReadWrite.All' was NOT granted. The signed-in user likely lacks the required role (Global Admin / Privileged Role Admin / Cloud App Admin)."
Write-Warning "Scope 'AppRoleAssignment.ReadWrite.All' was NOT granted. The signed-in user likely lacks the required role (Global Administrator or Privileged Role Administrator)."
if (-not $VerifyOnly) { return }
}
@@ -263,6 +294,7 @@ $action = if ($Revoke) { 'Revoke' } else { 'Grant' }
Write-Host "[4/4] $action mode - processing $($Permissions.Count) permission(s)..." -ForegroundColor Cyan
$results = @()
$graphForbidden = $false
foreach ($permName in $Permissions) {
$appRole = $graphSp.appRoles | Where-Object {
$_.value -eq $permName -and $_.allowedMemberTypes -contains 'Application'
@@ -289,7 +321,9 @@ foreach ($permName in $Permissions) {
$results += [pscustomobject]@{ Permission = $permName; Status = 'Revoked' }
}
catch {
Write-Warning " [FAIL] '$permName' revoke failed: $($_.Exception.Message)"
$failure = Get-GraphFailure $_
if ($failure.Status -eq 403) { $graphForbidden = $true }
Write-Warning " [FAIL] '$permName' revoke failed: $($failure.Text)"
$results += [pscustomobject]@{ Permission = $permName; Status = 'Failed' }
}
}
@@ -316,7 +350,9 @@ foreach ($permName in $Permissions) {
$results += [pscustomobject]@{ Permission = $permName; Status = 'Granted' }
}
catch {
Write-Warning " [FAIL] '$permName' grant failed: $($_.Exception.Message)"
$failure = Get-GraphFailure $_
if ($failure.Status -eq 403) { $graphForbidden = $true }
Write-Warning " [FAIL] '$permName' grant failed: $($failure.Text)"
$results += [pscustomobject]@{ Permission = $permName; Status = 'Failed' }
}
}
@@ -325,6 +361,9 @@ foreach ($permName in $Permissions) {
Write-Host ""
Write-Host "=========== Summary ===========" -ForegroundColor Cyan
$results | Format-Table -AutoSize | Out-Host
if ($graphForbidden) {
Write-Warning "Microsoft Graph refused the change (403): the signed-in account may not manage Microsoft Graph application permissions. Sign in as Global Administrator or Privileged Role Administrator (with PIM, activate the role first) and run the script again."
}
Write-Host ""
Write-Host "Note: token-cached backends may need up to ~1h to see the new permission set." -ForegroundColor Yellow
Write-Host " Use 'Refresh permission status' in the admin UI to apply changes immediately." -ForegroundColor Yellow
@@ -0,0 +1,71 @@
#nullable enable
using System;
using System.Collections.Generic;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring
{
/// <summary>
/// The completion timer a started Hello wizard runs against is the tenant's Hello wait
/// (<c>HelloWaitTimeoutSeconds</c>), resolved to 300..3600 s — the same budget the engine's
/// hello_safety window spends from the ESP exit, so a tenant allowing an hour is not cut off
/// by a fixed 5-min tracker timer while the user is still inside the wizard.
/// </summary>
public sealed class HelloTrackerCompletionBudgetTests
{
private static readonly DateTime Fixed = new DateTime(2026, 9, 29, 8, 0, 0, DateTimeKind.Utc);
[Theory]
[InlineData(30, 300)] // the old default, still stored for many tenants → the built-in window
[InlineData(0, 300)]
[InlineData(300, 300)]
[InlineData(1800, 1800)]
[InlineData(3600, 3600)]
[InlineData(86400, 3600)] // clamped to the ceiling
public void Completion_budget_is_the_resolved_tenant_hello_wait(int configured, int expected)
{
using var tmp = new TempDirectory();
using var tracker = NewTracker(tmp, new FakeSignalIngressSink(), configured);
Assert.Equal(expected, tracker.CompletionTimeoutSecondsForTest);
}
[Fact]
public void Completion_timeout_event_reports_the_tenant_budget()
{
using var tmp = new TempDirectory();
var ingress = new FakeSignalIngressSink();
using var tracker = NewTracker(tmp, ingress, helloWaitTimeoutSeconds: 2400);
tracker.NotifyHelloWizardStarted();
Assert.True(tracker.IsCompletionTimerActiveForTest);
tracker.TriggerCompletionTimeoutForTest();
var timeout = ingress.Posted.Single(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == Constants.EventTypes.HelloCompletionTimeout);
var data = Assert.IsAssignableFrom<IDictionary<string, object>>(timeout.TypedPayload);
Assert.Equal(2400, data["timeoutSeconds"]);
Assert.Equal("timeout", tracker.HelloOutcome);
}
private static HelloTracker NewTracker(TempDirectory tmp, FakeSignalIngressSink ingress, int helloWaitTimeoutSeconds) =>
new HelloTracker(
sessionId: "S1",
tenantId: "T1",
post: new InformationalEventPost(ingress, new VirtualClock(Fixed)),
logger: new AgentLogger(tmp.Path, AgentLogLevel.Info),
helloWaitTimeoutSeconds: helloWaitTimeoutSeconds);
}
}
@@ -83,6 +83,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
using var f = new Fixture();
Assert.False(f.Tracker.HasAccountSetupActivity);
Assert.False(f.Tracker.HasAccountSetupProgress);
Assert.Null(f.Tracker.AccountSetupProgressIfRead);
}
[Fact]
@@ -97,6 +98,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
Assert.True(f.Tracker.HasAccountSetupActivity);
Assert.False(f.Tracker.HasAccountSetupProgress);
// Read and untouched — the only state the WhiteGlove sealing gate treats as "false".
Assert.False(f.Tracker.AccountSetupProgressIfRead);
}
[Fact]
@@ -108,6 +111,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
Assert.True(f.Tracker.HasAccountSetupActivity);
Assert.True(f.Tracker.HasAccountSetupProgress);
Assert.True(f.Tracker.AccountSetupProgressIfRead);
}
[Theory]
@@ -138,5 +142,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
Assert.False(f.Tracker.HasAccountSetupProgress);
}
[Fact]
public void OnlyDeviceSetupRead_AccountSetupProgressIsUnknown()
{
// Adversarial for the WhiteGlove sealing gate: the tracker holds DeviceSetup data but
// has not read the AccountSetup category — that must be "unknown", never "untouched".
using var f = new Fixture();
f.Tracker.ProcessCategoryStatusForTest("DeviceSetupCategory.Status",
"{\"categoryState\":\"inProgress\",\"DeviceSetup.AppsSubcategory\":{\"subcategoryState\":\"inProgress\"}}");
Assert.Null(f.Tracker.AccountSetupProgressIfRead);
Assert.False(f.Tracker.HasAccountSetupProgress);
}
}
}
@@ -0,0 +1,123 @@
#nullable enable
using System;
using System.Linq;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.SystemSignals
{
/// <summary>
/// A tenant-extended Hello wait runs up to 60 min — the default session_stalled threshold. While
/// it is pending the session waits on a bounded timer that resolves it either way, so the stall
/// probe holds the report back instead of flagging the session Stalled; once the wait is over
/// and the session is still idle the report fires, and real activity drops it. The collector
/// runs with no scan sources, so only the session_stalled decision is exercised.
/// </summary>
public sealed class StallProbeHelloWaitGuardTests
{
private static readonly DateTime Fixed = new DateTime(2026, 9, 29, 8, 0, 0, DateTimeKind.Utc);
private const double IdleAtStallThreshold = 60;
[Fact]
public void Session_stalled_is_held_back_while_the_hello_wait_is_pending()
{
using var tmp = new TempDirectory();
var ingress = new FakeSignalIngressSink();
var collector = NewCollector(tmp, ingress, helloWaitPending: () => true);
collector.CheckAndRunProbes(IdleAtStallThreshold);
collector.CheckAndRunProbes(IdleAtStallThreshold + 1);
Assert.Equal(0, StalledCount(ingress));
}
[Fact]
public void Held_back_report_fires_once_the_hello_wait_is_over_and_the_session_is_still_idle()
{
using var tmp = new TempDirectory();
var ingress = new FakeSignalIngressSink();
var pending = true;
var collector = NewCollector(tmp, ingress, helloWaitPending: () => pending);
collector.CheckAndRunProbes(IdleAtStallThreshold);
Assert.Equal(0, StalledCount(ingress));
// hello_safety resolved (a deadline fire is no activity, so no ResetProbes): the next
// tick emits the held-back report — once.
pending = false;
collector.CheckAndRunProbes(IdleAtStallThreshold + 1);
collector.CheckAndRunProbes(IdleAtStallThreshold + 2);
Assert.Equal(1, StalledCount(ingress));
}
[Fact]
public void Real_activity_drops_the_held_back_report()
{
using var tmp = new TempDirectory();
var ingress = new FakeSignalIngressSink();
var pending = true;
var collector = NewCollector(tmp, ingress, helloWaitPending: () => pending);
collector.CheckAndRunProbes(IdleAtStallThreshold);
pending = false;
collector.ResetProbes();
collector.CheckAndRunProbes(1);
Assert.Equal(0, StalledCount(ingress));
// A new idle window reaches the threshold again — the fire-once report is still unused.
collector.CheckAndRunProbes(IdleAtStallThreshold);
Assert.Equal(1, StalledCount(ingress));
}
[Fact]
public void Without_a_hello_wait_probe_session_stalled_fires_as_before()
{
using var tmp = new TempDirectory();
var ingress = new FakeSignalIngressSink();
var collector = NewCollector(tmp, ingress, helloWaitPending: null);
collector.CheckAndRunProbes(IdleAtStallThreshold);
Assert.Equal(1, StalledCount(ingress));
}
[Fact]
public void A_throwing_hello_wait_probe_never_costs_the_stall_report()
{
using var tmp = new TempDirectory();
var ingress = new FakeSignalIngressSink();
var collector = NewCollector(tmp, ingress, helloWaitPending: () => throw new InvalidOperationException("probe down"));
collector.CheckAndRunProbes(IdleAtStallThreshold);
Assert.Equal(1, StalledCount(ingress));
}
private static StallProbeCollector NewCollector(TempDirectory tmp, FakeSignalIngressSink ingress, Func<bool>? helloWaitPending) =>
new StallProbeCollector(
sessionId: "S1",
tenantId: "T1",
post: new InformationalEventPost(ingress, new VirtualClock(Fixed)),
logger: new AgentLogger(tmp.Path, AgentLogLevel.Info),
thresholdsMinutes: new[] { 60 },
traceIndices: Array.Empty<int>(),
sources: Array.Empty<string>(),
sessionStalledAfterProbeIndex: 1,
helloWaitPending: helloWaitPending);
private static int StalledCount(FakeSignalIngressSink ingress) =>
ingress.Posted.Count(p =>
p.Kind == DecisionSignalKind.InformationalEvent
&& p.Payload != null
&& p.Payload.TryGetValue(SignalPayloadKeys.EventType, out var et)
&& et == Constants.EventTypes.SessionStalled);
}
}
@@ -20,6 +20,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
private sealed class FakeApiClient : BackendApiClient
{
public readonly List<TimeSpan?> Timeouts = new List<TimeSpan?>();
public readonly List<AgentErrorReport> Reports = new List<AgentErrorReport>();
public readonly Queue<bool> Results = new Queue<bool>();
public FakeApiClient() : base(
@@ -40,6 +41,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
public override Task<bool> ReportAgentErrorAsync(AgentErrorReport report, TimeSpan? timeout = null)
{
Timeouts.Add(timeout);
Reports.Add(report);
return Task.FromResult(Results.Count > 0 && Results.Dequeue());
}
}
@@ -139,5 +141,37 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
Assert.Equal(2, client.Calls);
}
[Fact]
public async Task Session_age_is_carried_in_the_report_and_absent_by_default()
{
// One reporter per report: a delivered report starts the 10-minute cooldown.
var client = new FakeApiClient();
client.Results.Enqueue(true);
client.Results.Enqueue(true);
await Reporter(client).TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m", sessionAgeHours: 478.2);
await Reporter(client).TrySendAsync(AgentErrorType.ConfigFetchFailed, "m");
Assert.Equal(2, client.Reports.Count);
Assert.Equal(478.2, client.Reports[0].SessionAgeHours);
Assert.Null(client.Reports[1].SessionAgeHours);
}
[Fact]
public async Task Prior_registration_failure_is_carried_in_the_report_and_absent_by_default()
{
var client = new FakeApiClient();
client.Results.Enqueue(true);
client.Results.Enqueue(true);
var record = new RegistrationFailureSummary { FailedRuns = 2, Outcome = "Failed", NetworkLinkUpAtEnd = false };
await Reporter(client).TrySendAsync(AgentErrorType.SessionAgeEmergencyBreak, "m", priorRegistrationFailure: record);
await Reporter(client).TrySendAsync(AgentErrorType.ConfigFetchFailed, "m");
Assert.Equal(2, client.Reports.Count);
Assert.Same(record, client.Reports[0].PriorRegistrationFailure);
Assert.Null(client.Reports[1].PriorRegistrationFailure);
}
}
}
@@ -0,0 +1,88 @@
using AutopilotMonitor.Agent.V2.Core.Monitoring.Transport;
using Newtonsoft.Json.Linq;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Monitoring.Transport
{
/// <summary>
/// Response parsers of the geo providers. Addresses come from the RFC 5737 documentation
/// ranges — never real ones.
/// </summary>
public class GeoLocationServiceTests
{
[Fact]
public void ParseDoGeo_maps_country_and_egress_ip_and_ignores_the_do_service_fields()
{
var json = JObject.Parse(@"{
""ExternalIpAddress"": ""203.0.113.7"",
""CountryCode"": ""DE"",
""KeyValue_EndpointFullUri"": ""https://kv601.prod.do.dsp.mp.microsoft.com/all"",
""Version"": ""26A6D4D6994E8DA5"",
""CacheId"": ""7"",
""ContentCert"": false,
""DownloadModeFailSafe"": """"
}");
var result = GeoLocationService.ParseDoGeo(json);
Assert.NotNull(result);
Assert.Equal("DE", result.Country);
Assert.Equal("203.0.113.7", result.Ip);
Assert.Equal("do-geo", result.Source);
Assert.Null(result.Region);
Assert.Null(result.City);
Assert.Null(result.Loc);
Assert.Null(result.Timezone);
}
[Theory]
[InlineData(@"{ ""ExternalIpAddress"": ""203.0.113.7"" }")]
[InlineData(@"{ ""ExternalIpAddress"": ""203.0.113.7"", ""CountryCode"": """" }")]
public void ParseDoGeo_without_country_is_a_failure(string body)
{
Assert.Null(GeoLocationService.ParseDoGeo(JObject.Parse(body)));
}
[Fact]
public void ParseIpInfo_maps_all_fields()
{
var json = JObject.Parse(@"{
""ip"": ""198.51.100.4"", ""city"": ""Stuttgart"", ""region"": ""Baden-Wurttemberg"",
""country"": ""DE"", ""loc"": ""48.7823,9.1770"", ""timezone"": ""Europe/Berlin""
}");
var result = GeoLocationService.ParseIpInfo(json);
Assert.Equal("DE", result.Country);
Assert.Equal("Baden-Wurttemberg", result.Region);
Assert.Equal("Stuttgart", result.City);
Assert.Equal("48.7823,9.1770", result.Loc);
Assert.Equal("Europe/Berlin", result.Timezone);
Assert.Equal("198.51.100.4", result.Ip);
Assert.Equal("ipinfo", result.Source);
}
[Fact]
public void ParseIfConfigCo_joins_coordinates_and_leaves_loc_empty_without_them()
{
var withCoords = GeoLocationService.ParseIfConfigCo(JObject.Parse(@"{
""ip"": ""198.51.100.4"", ""country_iso"": ""DE"", ""region_name"": ""Bavaria"",
""city"": ""Munich"", ""latitude"": 48.13, ""longitude"": 11.58, ""time_zone"": ""Europe/Berlin""
}"));
var withoutCoords = GeoLocationService.ParseIfConfigCo(JObject.Parse(@"{ ""country_iso"": ""DE"" }"));
Assert.Equal("48.13,11.58", withCoords.Loc);
Assert.Equal("DE", withCoords.Country);
Assert.Equal("Bavaria", withCoords.Region);
Assert.Equal("ifconfig.co", withCoords.Source);
Assert.Equal("", withoutCoords.Loc);
}
[Fact]
public void Describe_skips_the_parts_a_provider_did_not_return()
{
Assert.Equal("Stuttgart, BW, DE", new GeoLocationResult { City = "Stuttgart", Region = "BW", Country = "DE" }.Describe());
Assert.Equal("DE", new GeoLocationResult { Country = "DE", Region = "" }.Describe());
}
}
}
@@ -162,6 +162,28 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Orchestration
Assert.Equal(SessionStage.AwaitingHello, result.InitialState.Stage);
}
[Fact]
public void Clean_snapshot_of_a_rotated_session_adopts_the_current_session_id()
{
// SESSION-OWNER-BINDING: the backend refused the persisted session id at registration,
// the agent rotated, and the snapshot on disk still names the old session. The
// enrollment state carries over; the session id must be the current one.
using var rig = new Rig();
var saved = DecisionState.CreateInitial("S-REFUSED", TenantId)
.ToBuilder()
.WithStage(SessionStage.EspAccountSetup)
.WithStepIndex(7)
.WithLastAppliedSignalOrdinal(6)
.Build();
SeedSnapshot(rig, saved);
var result = rig.Recover();
Assert.Equal(SessionId, result.InitialState.SessionId);
Assert.Equal(SessionStage.EspAccountSetup, result.InitialState.Stage);
Assert.Equal(7, result.InitialState.StepIndex);
}
[Fact]
public void Snapshot_plus_pending_tail_replays_tail_onto_snapshot()
{
@@ -118,5 +118,88 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
Assert.Throws<ArgumentNullException>(
() => BackendSessionRegistration.Register(config, auth, http, "1.0", consoleMode: false, logger: null));
}
// ============================================================ failed-registration record
[Fact]
public void RecordRegistrationRun_records_a_failed_run_with_its_window_and_bounded_error()
{
using var tmp = new TempDirectory();
var persistence = new SessionIdPersistence(tmp.Path);
var started = new DateTime(2026, 8, 20, 9, 0, 0, DateTimeKind.Utc);
BackendSessionRegistration.RecordRegistrationRun(
persistence, SessionRegistrationResult.Failed(new string('e', 1000)),
started, started.AddSeconds(1204.46), linkUpAtStart: false, linkUpAtEnd: true,
configFetchOutcome: "UsedDefaults", logger: NewLogger(tmp.Path));
var record = persistence.LoadRegistrationFailure();
Assert.NotNull(record);
Assert.Equal(1, record!.FailedRuns);
Assert.Equal("Failed", record.Outcome);
Assert.Equal(BackendSessionRegistration.MaxRecordedErrorLength, record.LastError!.Length);
Assert.False(record.NetworkLinkUpAtStart);
Assert.True(record.NetworkLinkUpAtEnd);
Assert.Equal(1204.5, record.AttemptWindowSeconds);
Assert.Equal("UsedDefaults", record.ConfigFetchOutcome);
Assert.Equal(started.AddSeconds(1204.46), record.LastFailedAtUtc);
}
[Fact]
public void RecordRegistrationRun_records_auth_failures_and_clears_on_success()
{
using var tmp = new TempDirectory();
var persistence = new SessionIdPersistence(tmp.Path);
var at = new DateTime(2026, 8, 20, 9, 0, 0, DateTimeKind.Utc);
BackendSessionRegistration.RecordRegistrationRun(
persistence, SessionRegistrationResult.AuthFailed(401, "unauthorized"),
at, at.AddSeconds(3), true, true, "Succeeded", NewLogger(tmp.Path));
Assert.Equal("AuthFailed", persistence.LoadRegistrationFailure()!.Outcome);
BackendSessionRegistration.RecordRegistrationRun(
persistence, SessionRegistrationResult.Succeeded(new RegisterSessionResponse { Success = true }),
at, at.AddSeconds(1), true, true, "Succeeded", NewLogger(tmp.Path));
Assert.Null(persistence.LoadRegistrationFailure());
}
[Fact]
public void RecordRegistrationRun_without_persistence_is_a_no_op()
{
using var tmp = new TempDirectory();
BackendSessionRegistration.RecordRegistrationRun(
null, SessionRegistrationResult.Failed("x"), DateTime.UtcNow, DateTime.UtcNow,
null, null, null, NewLogger(tmp.Path));
}
// ============================================================ emergency-break message
[Fact]
public void Break_message_names_failed_registrations_in_invariant_culture()
{
var previous = System.Threading.Thread.CurrentThread.CurrentCulture;
try
{
System.Threading.Thread.CurrentThread.CurrentCulture = new System.Globalization.CultureInfo("de-DE");
var record = new RegistrationFailureSummary { FailedRuns = 3, Outcome = "Failed", NetworkLinkUpAtEnd = false };
var message = EmergencyBreakReporter.BuildMessage(478.2, 48, record);
Assert.Contains("session age 478.2h (cap 48h)", message);
Assert.EndsWith(" 3 earlier start(s) failed to register (last: Failed, network link at end: down).", message);
}
finally
{
System.Threading.Thread.CurrentThread.CurrentCulture = previous;
}
}
[Fact]
public void Break_message_without_record_keeps_the_historical_text()
{
Assert.Equal(
"Agent absolute session-age emergency break fired at session age 50.0h (cap 48h) — cleaning up and exiting.",
EmergencyBreakReporter.BuildMessage(50.0, 48, null));
}
}
}
@@ -4,6 +4,7 @@ using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Orchestration;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Agent.V2.Core.Tests.Orchestration;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.Shared.Models;
using Xunit;
@@ -29,7 +30,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
var sink = new FakeSignalIngressSink();
var logger = NewLogger(tmp.Path);
AutopilotMonitor.Agent.V2.Runtime.LifecycleEmitters.PostEnrollmentFactsObserved(sink, logger);
AutopilotMonitor.Agent.V2.Runtime.LifecycleEmitters.PostEnrollmentFactsObserved(sink, HelloWaitTimeout.DefaultSeconds, logger);
Assert.Single(sink.Posted);
var posted = sink.Posted[0];
@@ -47,7 +48,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
var sink = new FakeSignalIngressSink();
var logger = NewLogger(tmp.Path);
AutopilotMonitor.Agent.V2.Runtime.LifecycleEmitters.PostEnrollmentFactsObserved(sink, logger);
AutopilotMonitor.Agent.V2.Runtime.LifecycleEmitters.PostEnrollmentFactsObserved(sink, HelloWaitTimeout.DefaultSeconds, logger);
var payload = sink.Posted[0].Payload;
Assert.NotNull(payload);
@@ -73,6 +74,34 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
Assert.True(cloudPc == "true" || cloudPc == "false");
}
[Theory]
[InlineData(300)] // the default — the payload stays exactly what it was before the key existed
[InlineData(30)] // the old default, still stored for many tenants
[InlineData(0)]
public void Hello_wait_not_extending_the_built_in_window_is_not_stamped(int helloWaitTimeoutSeconds)
{
using var tmp = new TempDirectory();
var sink = new FakeSignalIngressSink();
AutopilotMonitor.Agent.V2.Runtime.LifecycleEmitters.PostEnrollmentFactsObserved(sink, helloWaitTimeoutSeconds, NewLogger(tmp.Path));
Assert.DoesNotContain(SignalPayloadKeys.HelloWaitTimeoutSeconds, (IDictionary<string, string>)sink.Posted[0].Payload!);
}
[Theory]
[InlineData(1800, "1800")]
[InlineData(3600, "3600")]
[InlineData(86400, "3600")] // the effective (clamped) value is what the engine should arm
public void Extended_hello_wait_is_stamped_as_its_effective_value(int helloWaitTimeoutSeconds, string expected)
{
using var tmp = new TempDirectory();
var sink = new FakeSignalIngressSink();
AutopilotMonitor.Agent.V2.Runtime.LifecycleEmitters.PostEnrollmentFactsObserved(sink, helloWaitTimeoutSeconds, NewLogger(tmp.Path));
Assert.Equal(expected, sink.Posted[0].Payload![SignalPayloadKeys.HelloWaitTimeoutSeconds]);
}
[Fact]
public void Swallows_sink_exceptions()
{
@@ -82,7 +111,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
var logger = NewLogger(tmp.Path);
var ex = Record.Exception(() =>
AutopilotMonitor.Agent.V2.Runtime.LifecycleEmitters.PostEnrollmentFactsObserved(sink, logger));
AutopilotMonitor.Agent.V2.Runtime.LifecycleEmitters.PostEnrollmentFactsObserved(sink, HelloWaitTimeout.DefaultSeconds, logger));
Assert.Null(ex);
}
@@ -149,6 +149,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
persistence.SaveSessionCreatedAt(DateTime.UtcNow.AddHours(-100));
var stateDir = Path.Combine(tmp.Path, "State");
double? reportedAgeHours = null;
var tripped = AutopilotMonitor.Agent.V2.Program.CheckSessionAgeEmergencyBreak(
dataDirectory: tmp.Path,
@@ -157,9 +158,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Program
selfDestructOnComplete: false,
cleanupServiceFactory: null,
logger: logger,
consoleMode: false);
consoleMode: false,
onBreakFired: ageHours => reportedAgeHours = ageHours);
Assert.True(tripped);
// The measured age reaches the break report (the backend cannot derive it).
Assert.NotNull(reportedAgeHours);
Assert.InRange(reportedAgeHours.Value, 99.9, 100.1);
// Marker must have been written so the next restart exits cleanly.
Assert.True(File.Exists(Path.Combine(stateDir, "enrollment-complete.marker")));
// Session must have been cleared.
@@ -62,6 +62,18 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Runtime
Assert.Equal("ipinfo.io", evt.Data["source"]);
}
[Fact]
public void BuildGeoEvent_country_only_location_has_no_empty_parts_in_message()
{
var loc = new GeoLocationResult { Country = "DE", Source = "do-geo" };
var evt = StartupEnvironmentProbes.BuildGeoEvent(Config(), loc);
Assert.Equal("Device location: DE (via do-geo)", evt.Message);
Assert.Equal("DE", evt.Data["country"]);
Assert.Equal("do-geo", evt.Data["source"]);
}
// ================================================================= Outbound IP
[Fact]
@@ -98,6 +110,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Runtime
PrimaryError = "connection refused",
PrimaryRetryError = "DNS failure",
FallbackError = "timeout",
DoGeoError = "HTTP 403 (Forbidden)",
};
var evt = StartupEnvironmentProbes.BuildGeoFailureEvent(cfg, attempt);
@@ -108,6 +121,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Runtime
Assert.Equal("connection refused", evt.Data["primaryError"]);
Assert.Equal("DNS failure", evt.Data["primaryRetryError"]);
Assert.Equal("timeout", evt.Data["fallbackError"]);
Assert.Equal("HTTP 403 (Forbidden)", evt.Data["doGeoError"]);
Assert.Equal("do-geo", evt.Data["lastResortProvider"]);
}
[Fact]
@@ -119,6 +134,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Runtime
Assert.Equal("unknown", evt.Data["primaryError"]);
Assert.Equal("unknown", evt.Data["primaryRetryError"]);
Assert.Equal("unknown", evt.Data["fallbackError"]);
Assert.Equal("unknown", evt.Data["doGeoError"]);
}
// ================================================================= Timezone
@@ -3,6 +3,7 @@ using System.IO;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Agent.V2.Core.Security;
using AutopilotMonitor.Agent.V2.Core.Tests.Harness;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
@@ -234,5 +235,78 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
Assert.False(File.Exists(Path.Combine(tmp.Path, "session.created")));
Assert.False(File.Exists(Path.Combine(tmp.Path, "whiteglove.complete")));
}
// ============================================================ failed-registration record
private static RegistrationFailureSummary FailedRun(DateTime at, string outcome = "Failed", bool? linkUpAtEnd = false) =>
new RegistrationFailureSummary
{
FailedRuns = 1,
FirstFailedAtUtc = at,
LastFailedAtUtc = at,
Outcome = outcome,
LastError = "No such host is known.",
NetworkLinkUpAtStart = false,
NetworkLinkUpAtEnd = linkUpAtEnd,
AttemptWindowSeconds = 91.3,
ConfigFetchOutcome = "UsedDefaults",
};
[Fact]
public void RecordRegistrationFailure_counts_runs_keeps_the_first_time_and_describes_the_latest_run()
{
using var tmp = new TempDirectory();
var sut = new SessionIdPersistence(tmp.Path);
var first = new DateTime(2026, 8, 20, 9, 0, 0, DateTimeKind.Utc);
var second = new DateTime(2026, 9, 12, 7, 30, 0, DateTimeKind.Utc);
Assert.Null(sut.LoadRegistrationFailure());
sut.RecordRegistrationFailure(FailedRun(first));
sut.RecordRegistrationFailure(FailedRun(second, outcome: "AuthFailed", linkUpAtEnd: true));
// A fresh instance reads it back from disk, as the next boot's emergency break does.
var record = new SessionIdPersistence(tmp.Path).LoadRegistrationFailure();
Assert.NotNull(record);
Assert.Equal(2, record!.FailedRuns);
Assert.Equal(first, record.FirstFailedAtUtc);
Assert.Equal(second, record.LastFailedAtUtc);
Assert.Equal("AuthFailed", record.Outcome);
Assert.True(record.NetworkLinkUpAtEnd);
Assert.Equal(91.3, record.AttemptWindowSeconds);
Assert.Equal("UsedDefaults", record.ConfigFetchOutcome);
}
[Fact]
public void Registration_failure_record_is_cleared_by_success_delete_and_rotate()
{
using var tmp = new TempDirectory();
var sut = new SessionIdPersistence(tmp.Path);
sut.GetOrCreate();
var at = new DateTime(2026, 8, 20, 9, 0, 0, DateTimeKind.Utc);
sut.RecordRegistrationFailure(FailedRun(at));
sut.ClearRegistrationFailure();
Assert.Null(sut.LoadRegistrationFailure());
sut.RecordRegistrationFailure(FailedRun(at));
sut.Rotate();
Assert.Null(sut.LoadRegistrationFailure());
sut.RecordRegistrationFailure(FailedRun(at));
sut.Delete();
Assert.Null(sut.LoadRegistrationFailure());
Assert.False(File.Exists(Path.Combine(tmp.Path, "registration-failure.json")));
}
[Theory]
[InlineData("not json")]
[InlineData("{\"FailedRuns\":0}")]
public void LoadRegistrationFailure_reports_nothing_for_a_corrupt_or_empty_record(string content)
{
using var tmp = new TempDirectory();
File.WriteAllText(Path.Combine(tmp.Path, "registration-failure.json"), content);
Assert.Null(new SessionIdPersistence(tmp.Path).LoadRegistrationFailure());
}
}
}
@@ -137,6 +137,86 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Security
Assert.Equal(originalSessionId, result.RotatedFromSessionId);
}
private static BackendRefusalException TenantConflict()
=> new BackendRefusalException("Backend returned 409 Conflict with error code session_owner_mismatch.", 409,
AutopilotMonitor.Shared.Constants.AgentErrorCodes.SessionOwnerMismatch);
[Fact]
public async Task Tenant_conflict_409_rotates_once_like_an_owner_mismatch()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
var config = NewConfig();
var originalSessionId = config.SessionId;
var rotatedTo = Guid.NewGuid().ToString();
var seenSessionIds = new System.Collections.Generic.List<string>();
var apiClient = new FakeApiClient(
attempt => attempt == 1
? throw TenantConflict()
: new RegisterSessionResponse { Success = true, SessionId = rotatedTo, ValidatedBy = ValidatorType.AutopilotV1 },
onRegister: r => seenSessionIds.Add(r.SessionId));
var tracker = new AuthFailureTracker(maxFailures: 5, timeoutMinutes: 0,
clock: SystemClock.Instance, logger: logger);
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
apiClient, config, agentVersion: "0.0.0", logger: logger,
authFailureTracker: tracker,
backoffDelay: _ => throw new InvalidOperationException("rotation must retry without backoff"),
networkLinkWait: _ => Task.CompletedTask,
rotateSession: () => { config.SessionId = rotatedTo; return rotatedTo; });
Assert.Equal(SessionRegistrationOutcome.Succeeded, result.Outcome);
Assert.Equal(new[] { originalSessionId, rotatedTo }, seenSessionIds);
Assert.Equal(originalSessionId, result.RotatedFromSessionId);
Assert.Equal(0, tracker.ConsecutiveFailures);
}
[Fact]
public async Task Coded_refusal_after_rotation_fails_without_retry_or_auth_failure()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
var config = NewConfig();
var apiClient = new FakeApiClient(_ => throw TenantConflict());
var tracker = new AuthFailureTracker(maxFailures: 5, timeoutMinutes: 0,
clock: SystemClock.Instance, logger: logger);
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
apiClient, config, agentVersion: "0.0.0", logger: logger,
authFailureTracker: tracker,
backoffDelay: _ => throw new InvalidOperationException("a coded refusal must not be retried"),
networkLinkWait: _ => Task.CompletedTask,
rotateSession: () => { config.SessionId = Guid.NewGuid().ToString(); return config.SessionId; });
Assert.Equal(SessionRegistrationOutcome.Failed, result.Outcome);
Assert.Equal(2, apiClient.CallCount);
Assert.Equal(0, tracker.ConsecutiveFailures);
}
[Fact]
public async Task Rotation_hook_failure_on_a_tenant_conflict_fails_without_auth_failure()
{
using var tmp = new TempDirectory();
var logger = NewLogger(tmp);
var config = NewConfig();
var apiClient = new FakeApiClient(_ => throw TenantConflict());
var tracker = new AuthFailureTracker(maxFailures: 5, timeoutMinutes: 0,
clock: SystemClock.Instance, logger: logger);
var result = await SessionRegistrationHelper.RegisterWithRetryAsync(
apiClient, config, agentVersion: "0.0.0", logger: logger,
authFailureTracker: tracker, networkLinkWait: _ => Task.CompletedTask,
rotateSession: () => throw new IOException("disk full"));
Assert.Equal(SessionRegistrationOutcome.Failed, result.Outcome);
Assert.Equal(1, apiClient.CallCount);
Assert.Equal(0, tracker.ConsecutiveFailures);
}
[Fact]
public async Task Second_owner_mismatch_after_rotation_is_an_auth_failure()
{
@@ -99,6 +99,52 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.SignalAdapters
Assert.Equal(DecisionSignalKind.WhiteGloveShellCoreSuccess, posted.Kind);
}
[Theory]
[InlineData(true, "true")]
[InlineData(false, "false")]
public void WhiteGloveEvent_stamps_read_AccountSetup_progress_on_payload_and_evidence(bool progress, string expected)
{
using var f = new Fixture();
using var adapter = new EspAndHelloTrackerAdapter(
f.Coordinator, f.Ingress, f.Clock, accountSetupProgressProbe: () => progress);
adapter.TriggerWhiteGloveFromTest();
var posted = Assert.Single(f.Ingress.Posted);
Assert.Equal(expected, posted.Payload![SignalPayloadKeys.AccountSetupProgress]);
Assert.Equal(expected, posted.Evidence.DerivationInputs![SignalPayloadKeys.AccountSetupProgress]);
}
[Fact]
public void WhiteGloveEvent_with_unread_AccountSetup_registry_sends_no_progress_key()
{
// Unknown must keep the sealing classifier's AccountSetup excluder — the payload key
// is absent, the evidence records why.
using var f = new Fixture();
using var adapter = new EspAndHelloTrackerAdapter(
f.Coordinator, f.Ingress, f.Clock, accountSetupProgressProbe: () => null);
adapter.TriggerWhiteGloveFromTest();
var posted = Assert.Single(f.Ingress.Posted);
Assert.True(posted.Payload == null || !posted.Payload.ContainsKey(SignalPayloadKeys.AccountSetupProgress));
Assert.Equal("unread", posted.Evidence.DerivationInputs![SignalPayloadKeys.AccountSetupProgress]);
}
[Fact]
public void WhiteGloveEvent_default_probe_on_unstarted_coordinator_reports_unread()
{
// Production probe: the provisioning tracker does not exist before Start (and never on
// Device Preparation) — that is "unread", not "untouched".
using var f = new Fixture();
using var adapter = new EspAndHelloTrackerAdapter(f.Coordinator, f.Ingress, f.Clock);
adapter.TriggerWhiteGloveFromTest();
var posted = Assert.Single(f.Ingress.Posted);
Assert.True(posted.Payload == null || !posted.Payload.ContainsKey(SignalPayloadKeys.AccountSetupProgress));
}
[Fact]
public void EspFailureEvent_emits_EspTerminalFailure_with_merged_subSource_annotation()
{
@@ -564,6 +564,37 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Termination
Assert.False(File.Exists(Path.Combine(rig.StateDir, "enrollment-complete.marker")));
}
[Fact]
public void Handle_whiteglove_part1_writes_marker_before_the_later_exit_steps()
{
// The technician's reseal can kill the process anywhere after the seal. The marker
// is what lets the next boot skip the 48-hour break and resume Part 2, so it must
// exist before collectors stop, diagnostics upload, grace and drain run.
using var rig = new Rig();
rig.State = new DecisionStateBuilder(DecisionState.CreateInitial("S1", "T1")) { Stage = SessionStage.WhiteGloveSealed }.Build();
bool? markerPresentAtCollectorStop = null;
rig.StopPeripheralCollectorsHook = () => markerPresentAtCollectorStop = rig.SessionPersistence.IsWhiteGloveResume();
rig.Build().Handle(sender: null!,
Args(EnrollmentTerminationReason.DecisionTerminalStage, EnrollmentTerminationOutcome.Succeeded, SessionStage.WhiteGloveSealed));
Assert.True(markerPresentAtCollectorStop, "whiteglove.complete must be written before the peripheral collectors are stopped.");
var emitted = rig.EmittedEventTypes.ToList();
Assert.True(emitted.IndexOf("agent_shutting_down") >= 0, "agent_shutting_down must still be emitted.");
}
[Fact]
public void Handle_non_whiteglove_termination_never_writes_the_marker()
{
using var rig = new Rig();
rig.State = new DecisionStateBuilder(DecisionState.CreateInitial("S1", "T1")) { Stage = SessionStage.Completed }.Build();
rig.Build().Handle(sender: null!,
Args(EnrollmentTerminationReason.DecisionTerminalStage, EnrollmentTerminationOutcome.Succeeded, SessionStage.Completed));
Assert.False(rig.SessionPersistence.IsWhiteGloveResume());
}
[Fact]
public void Handle_whiteglove_part1_stops_collectors_before_emitting_part1_complete()
{
@@ -88,6 +88,35 @@ namespace AutopilotMonitor.Agent.V2.Core.Tests.Transport
Assert.Null(ex.ErrorCode);
}
// ── coded 409: a refusal, never an auth failure ─────────────────────
[Fact]
public async Task Coded_409_is_a_refusal_not_an_auth_failure()
{
using var client = BuildClient(
HttpStatusCode.Conflict,
"{\"success\":false,\"message\":\"Session id is already registered to another tenant\",\"errorCode\":\"session_owner_mismatch\"}",
"application/json");
var ex = await Assert.ThrowsAsync<BackendRefusalException>(
() => client.RegisterSessionAsync(new AutopilotMonitor.Shared.Models.SessionRegistration()));
Assert.Equal(409, ex.StatusCode);
Assert.Equal("session_owner_mismatch", ex.ErrorCode);
}
[Fact]
public async Task Uncoded_409_stays_a_plain_http_error()
{
using var client = BuildClient(
HttpStatusCode.Conflict,
"{\"success\":false,\"message\":\"A diagnostics package with this name already exists\"}",
"application/json");
await Assert.ThrowsAsync<HttpRequestException>(
() => client.GetDiagnosticsUploadUrlAsync("tenant-1", "session-1", "package.zip"));
}
[Theory]
[InlineData(null, null)]
[InlineData("", null)]
@@ -157,12 +157,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Configuration
public double ReplaySpeedFactor { get; set; } = 50;
/// <summary>
/// Wait time in seconds after ESP exit before marking Hello as skipped.
/// When ESP exits, we wait this duration for Hello wizard to start (event 62404).
/// Default: 30 seconds (reasonable for systems under load).
/// If Hello wizard starts within this window, we continue waiting for Hello completion (300/301).
/// Tenant Hello wait in seconds after the ESP exit, merged from RemoteConfig. Stamped on
/// <c>EnrollmentFactsObserved</c> for the decision engine's hello_safety window; the Hello
/// tracker gets the same remote value through the collector configuration. Both resolve it
/// through <see cref="HelloWaitTimeout.EffectiveSeconds"/> (at least the built-in 300 s,
/// at most 3600 s). Default: 300 seconds.
/// </summary>
public int HelloWaitTimeoutSeconds { get; set; } = 30;
public int HelloWaitTimeoutSeconds { get; set; } = HelloWaitTimeout.DefaultSeconds;
/// <summary>
/// Maximum consecutive authentication failures (401/403) before the agent shuts down.
@@ -216,6 +216,13 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// </summary>
public bool HasAccountSetupProgress => _provisioningTracker?.HasAccountSetupProgress ?? false;
/// <summary>
/// <see cref="ProvisioningStatusTracker.AccountSetupProgressIfRead"/>; <c>null</c> also when
/// the provisioning tracker is not running (Device Preparation, before Start). Stamped on
/// the WhiteGloveShellCoreSuccess signal for the sealing classifier.
/// </summary>
public bool? AccountSetupProgressIfRead => _provisioningTracker?.AccountSetupProgressIfRead;
/// <summary>
/// True once WhiteGlove start has been detected (EventID 509 or persisted from prior run).
/// Forwarded from ModernDeploymentTracker.
@@ -234,7 +241,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
string tenantId,
InformationalEventPost post,
AgentLogger logger,
int helloWaitTimeoutSeconds = 30,
int helloWaitTimeoutSeconds = HelloWaitTimeout.DefaultSeconds,
bool modernDeploymentWatcherEnabled = true,
int modernDeploymentLogLevelMax = 3,
bool modernDeploymentBackfillEnabled = true,
@@ -47,7 +47,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
internal const string HResult_UserSkippedHello = "0x801C044F";
internal const int HelloCompletionTimeoutSeconds = 300;
internal const int BackfillLookbackMinutes = 5;
private static readonly HashSet<int> TrackedUdrEventIds = new HashSet<int>
@@ -81,6 +80,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
private readonly InformationalEventPost _post;
private readonly int _helloWaitTimeoutSeconds;
// How long a started Hello wizard may take to reach a terminal event (300/301/362): the
// tenant's Hello wait resolved through HelloWaitTimeout (300..3600 s). The engine's
// hello_safety window spends the same budget from the ESP exit, so this timer never cuts
// a user off inside the wizard before the tenant's budget is used up.
private readonly int _helloCompletionTimeoutSeconds;
private EventLogWatcher _udrWatcher;
private EventLogWatcher _helloForBusinessWatcher;
private System.Threading.Timer _policyCheckTimer;
@@ -121,13 +126,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
string tenantId,
InformationalEventPost post,
AgentLogger logger,
int helloWaitTimeoutSeconds = 30)
int helloWaitTimeoutSeconds = HelloWaitTimeout.DefaultSeconds)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_helloWaitTimeoutSeconds = helloWaitTimeoutSeconds;
_helloCompletionTimeoutSeconds = HelloWaitTimeout.EffectiveSeconds(helloWaitTimeoutSeconds);
}
// =====================================================================
@@ -1094,7 +1100,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
if (policyKnownEnabled)
{
_logger.Info($"Hello wait timeout ({_helloWaitTimeoutSeconds}s) expired but Hello policy is enabled — " +
$"wizard not yet visible, starting long completion timer ({HelloCompletionTimeoutSeconds}s)");
$"wizard not yet visible, starting long completion timer ({_helloCompletionTimeoutSeconds}s)");
_post.Emit(new EnrollmentEvent
{
@@ -1105,7 +1111,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = $"Hello wizard did not start within {_helloWaitTimeoutSeconds}s after ESP exit — " +
$"Hello policy is enabled, waiting up to {HelloCompletionTimeoutSeconds}s for wizard",
$"Hello policy is enabled, waiting up to {_helloCompletionTimeoutSeconds}s for wizard",
Data = new Dictionary<string, object>
{
{ "timeoutSeconds", _helloWaitTimeoutSeconds },
@@ -1205,11 +1211,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
if (_isHelloCompleted) return;
if (_helloCompletionTimer != null) return;
_logger.Info($"Starting Hello completion timer ({HelloCompletionTimeoutSeconds}s) - waiting for terminal Hello event (300/301/362)");
_logger.Info($"Starting Hello completion timer ({_helloCompletionTimeoutSeconds}s) - waiting for terminal Hello event (300/301/362)");
_helloCompletionTimer = new System.Threading.Timer(
OnHelloCompletionTimeout,
null,
TimeSpan.FromSeconds(HelloCompletionTimeoutSeconds),
TimeSpan.FromSeconds(_helloCompletionTimeoutSeconds),
TimeSpan.FromMilliseconds(-1));
}
@@ -1230,7 +1236,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
return;
}
_logger.Warning($"Hello completion timeout ({HelloCompletionTimeoutSeconds}s) expired after wizard start without terminal event");
_logger.Warning($"Hello completion timeout ({_helloCompletionTimeoutSeconds}s) expired after wizard start without terminal event");
_isHelloCompleted = true;
HelloOutcome = _helloWizardStarted ? "timeout" : "wizard_not_started";
StopHelloCompletionTimerLocked();
@@ -1243,10 +1249,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
Severity = EventSeverity.Warning,
Source = "EspAndHelloTracker",
Phase = EnrollmentPhase.Unknown,
Message = $"Hello wizard started but no terminal event (300/301/362) arrived within {HelloCompletionTimeoutSeconds}s",
Message = $"Hello wizard started but no terminal event (300/301/362) arrived within {_helloCompletionTimeoutSeconds}s",
Data = new Dictionary<string, object>
{
{ "timeoutSeconds", HelloCompletionTimeoutSeconds },
{ "timeoutSeconds", _helloCompletionTimeoutSeconds },
{ "helloWizardStarted", _helloWizardStarted }
},
ImmediateUpload = true
@@ -1304,6 +1310,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
internal void TriggerWaitTimeoutForTest() => OnHelloWaitTimeout(null);
internal void TriggerCompletionTimeoutForTest() => OnHelloCompletionTimeout(null);
internal int CompletionTimeoutSecondsForTest => _helloCompletionTimeoutSeconds;
internal bool IsWaitTimerActiveForTest { get { lock (_stateLock) { return _helloWaitTimer != null; } } }
internal bool IsCompletionTimerActiveForTest { get { lock (_stateLock) { return _helloCompletionTimer != null; } } }
internal bool IsHelloWizardStartedForTest { get { lock (_stateLock) { return _helloWizardStarted; } } }
@@ -254,16 +254,27 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// host to tell the real user-ESP page exit from the Device→Account handoff exit; not a
/// completion input.
/// </summary>
public bool HasAccountSetupProgress
public bool HasAccountSetupProgress => AccountSetupProgressIfRead == true;
/// <summary>
/// Tri-state form of <see cref="HasAccountSetupProgress"/>: <c>null</c> until the tracker
/// has actually read AccountSetup subcategories (not started yet, category not written,
/// empty), otherwise whether one has left <c>notStarted</c>. The WhiteGlove sealing
/// classifier treats only a read <c>false</c> as "user ESP untouched" — an unread
/// registry must never pass for an untouched one.
/// </summary>
public bool? AccountSetupProgressIfRead
{
get
{
lock (_stateLock)
{
if (_lastSubcategoryStates == null)
return false;
if (!_lastSubcategoryStates.TryGetValue("AccountSetupCategory.Status", out var subs) || subs == null)
return false;
return null;
if (!_lastSubcategoryStates.TryGetValue("AccountSetupCategory.Status", out var subs)
|| subs == null
|| subs.Count == 0)
return null;
foreach (var state in subs.Values)
{
if (!string.Equals(state, "notStarted", StringComparison.OrdinalIgnoreCase))
@@ -279,7 +290,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// with subcategoryState=succeeded. NOT a WhiteGlove indicator: on Win11 25H2 (build
/// 26200+) Windows writes this property as a trivially-succeeding step in EVERY
/// enrollment (validated 2026-08-18 — user-driven sessions f475e697/c601a24b/4219203f
/// all carry it). Kept for observability only (whiteglove_signal_* raw dumps); must not
/// all carry it). Kept for observability only (save_result_step_* raw dumps); must not
/// be used for classification. Reliable WhiteGlove anchors are ModernDeployment
/// EventID 509 and the Shell-Core WhiteGlove-success events.
/// </summary>
@@ -966,8 +977,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
_logger.Info($"ProvisioningStatusTracker: SaveWhiteGloveSuccessResult state " +
$"transition: {previousState} -> {state}");
// Neutral trigger name: the step exists in every 25H2 enrollment,
// and "whiteglove_*" was read as a pre-provisioning signal.
EmitRawRegistryDump(categoryLabel, jsonValue,
$"whiteglove_signal_{state}");
$"save_result_step_{state}");
if (string.Equals(state, "succeeded", StringComparison.OrdinalIgnoreCase))
{
@@ -39,7 +39,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// - Trace-indexed probes: emit a <c>stall_probe_check</c> Trace event on quiet/activeInstalls.
/// - Non-trace probes: emit nothing when no anomaly is found (agent log only).
/// - Anomaly found: always emit <c>stall_probe_result</c> Warning (plus EspFailureDetected if terminal).
/// - Probe index configured as "session-stalled": emit fire-once <c>session_stalled</c> Warning.
/// - Probe index configured as "session-stalled": emit fire-once <c>session_stalled</c> Warning
/// (held back while a tenant-extended Hello wait is pending, emitted once it is over).
/// </summary>
public class StallProbeCollector
{
@@ -54,10 +55,20 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
private readonly HashSet<int> _harmlessModernDeploymentEventIds;
private readonly bool _isDevicePreparation;
// True while a tenant-extended Hello wait (up to 60 min) is pending in the decision
// engine. It is a bounded wait that resolves the session either way — not a stall — so
// session_stalled is held back until it is over.
private readonly Func<bool> _helloWaitPending;
private readonly object _stateLock = new object();
private readonly HashSet<int> _firedProbeIndices = new HashSet<int>();
private bool _sessionStalledFired;
// The session_stalled report held back while the Hello wait was pending (the probe result
// of that idle window). Emitted on a later tick once the wait is over and the session is
// still idle; dropped by real activity (ResetProbes).
private ProbeResult _deferredSessionStalled;
// P2 — monotonic generation, bumped by ResetProbes. RunProbe runs OUTSIDE _stateLock; if a
// real-activity reset lands while a probe is in flight, the post-probe "mark fired" must be
// dropped (its generation is stale) so the next idle period can fire that probe again.
@@ -153,9 +164,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
string[] sources,
int sessionStalledAfterProbeIndex,
int[] harmlessModernDeploymentEventIds = null,
bool isDevicePreparation = false)
bool isDevicePreparation = false,
Func<bool> helloWaitPending = null)
{
_sessionId = sessionId ?? throw new ArgumentNullException(nameof(sessionId));
_helloWaitPending = helloWaitPending;
_tenantId = tenantId ?? throw new ArgumentNullException(nameof(tenantId));
_post = post ?? throw new ArgumentNullException(nameof(post));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
@@ -192,6 +205,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// Always bump the generation, even when no probes are currently marked fired — a
// probe may be running right now (outside the lock) and must not re-mark itself.
_probeGeneration++;
_deferredSessionStalled = null;
if (_firedProbeIndices.Count > 0)
{
_logger.Trace($"StallProbeCollector: resetting {_firedProbeIndices.Count} fired probes");
@@ -206,6 +220,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
/// </summary>
public void CheckAndRunProbes(double idleMinutes)
{
EmitDeferredSessionStalledIfDue(idleMinutes);
// Iterate thresholds from lowest to highest so probes fire in order.
for (int i = 0; i < _thresholdsMinutes.Length; i++)
{
@@ -225,7 +241,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
if (alreadyFired)
continue;
RunProbe(probeIndex, thresholdMinutes, idleMinutes);
RunProbe(probeIndex, thresholdMinutes, idleMinutes, genAtStart);
lock (_stateLock)
{
@@ -237,7 +253,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
}
}
private void RunProbe(int probeIndex, int thresholdMinutes, double idleMinutes)
private void RunProbe(int probeIndex, int thresholdMinutes, double idleMinutes, int generation)
{
var started = DateTime.UtcNow;
var result = new ProbeResult { ProbeIndex = probeIndex, IdleMinutes = Math.Round(idleMinutes, 1) };
@@ -293,8 +309,21 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
// session_stalled is fire-once across the whole session. It fires at the configured
// probe index UNLESS activeInstalls is observed (activeInstalls counts as progress).
// While a tenant Hello wait is pending it is held back, not dropped.
if (probeIndex == _sessionStalledAfterProbeIndex && !hasActiveInstalls)
{
if (IsHelloWaitPending())
{
lock (_stateLock)
{
// P2: a reset that landed while this probe ran ended the idle window.
if (!_sessionStalledFired && _probeGeneration == generation)
_deferredSessionStalled = result;
}
_logger.Info($"StallProbeCollector: session_stalled held back at probe {probeIndex} — a tenant Hello wait is still pending");
return;
}
lock (_stateLock)
{
if (_sessionStalledFired)
@@ -305,6 +334,52 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Enrollment.SystemSignals
}
}
/// <summary>
/// Emits the held-back session_stalled report once the Hello wait is over and the session
/// is still idle (no ResetProbes since), with the idle time of this tick.
/// </summary>
private void EmitDeferredSessionStalledIfDue(double idleMinutes)
{
ProbeResult deferred;
lock (_stateLock)
{
deferred = _deferredSessionStalled;
if (deferred == null || _sessionStalledFired)
return;
}
if (IsHelloWaitPending())
return;
lock (_stateLock)
{
if (_sessionStalledFired || !ReferenceEquals(_deferredSessionStalled, deferred))
return;
_sessionStalledFired = true;
_deferredSessionStalled = null;
}
deferred.IdleMinutes = Math.Round(idleMinutes, 1);
_logger.Info($"StallProbeCollector: Hello wait over, still idle at {deferred.IdleMinutes}min — emitting the held-back session_stalled");
EmitSessionStalledEvent(deferred);
}
private bool IsHelloWaitPending()
{
if (_helloWaitPending == null)
return false;
try
{
return _helloWaitPending();
}
catch (Exception ex)
{
// A failing probe must never cost the stall report — treat as not pending.
_logger.Warning($"StallProbeCollector: Hello-wait probe threw, treating as not pending: {ex.Message}");
return false;
}
}
private void RunWithTimeout(Action<ProbeResult> scan, int timeoutMs, string sourceName, ProbeResult result)
{
// Enforce a real hard timeout: run the source on a dedicated background thread and
@@ -237,6 +237,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
using (response)
{
await ThrowOnAuthFailureAsync(response).ConfigureAwait(false);
await ThrowOnCodedConflictAsync(response).ConfigureAwait(false);
response.EnsureSuccessStatusCode();
var responseJson = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
@@ -338,6 +339,38 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
errorCode: errorCode);
}
/// <summary>
/// Throws <see cref="BackendRefusalException"/> for a 409 whose JSON body carries a
/// machine-readable <c>errorCode</c> — register-session answers
/// <c>session_owner_mismatch</c> when another tenant claimed the session id, which no retry
/// can clear but a session rotation can. A 409 without a code falls through to
/// <c>EnsureSuccessStatusCode</c> as before.
/// </summary>
private static async Task ThrowOnCodedConflictAsync(HttpResponseMessage response)
{
if (response.StatusCode != System.Net.HttpStatusCode.Conflict)
return;
string body = null;
try
{
body = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
}
catch
{
// No body, no code — the generic 409 handling applies.
}
var errorCode = TryExtractErrorCode(body);
if (errorCode == null)
return;
throw new BackendRefusalException(
$"Backend returned 409 Conflict with error code {errorCode}.",
(int)response.StatusCode,
errorCode);
}
/// <summary>
/// Pulls <c>errorCode</c> out of a backend JSON error body. Tolerant: any parse failure
/// or a non-string value yields null, never an exception.
@@ -471,4 +504,24 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
ErrorCode = errorCode;
}
}
/// <summary>
/// Thrown when the backend refuses a request on a non-auth status (409) with a machine-readable
/// code from Shared <c>Constants.AgentErrorCodes</c>. Unlike <see cref="BackendAuthException"/>
/// it says nothing about the device's authorization and must never feed the auth-failure tracker.
/// </summary>
public class BackendRefusalException : Exception
{
/// <summary>HTTP status code returned by the backend (409).</summary>
public int StatusCode { get; }
/// <summary>Machine-readable code from the backend's JSON body (never null).</summary>
public string ErrorCode { get; }
public BackendRefusalException(string message, int statusCode, string errorCode) : base(message)
{
StatusCode = statusCode;
ErrorCode = errorCode;
}
}
}
@@ -79,6 +79,11 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// exist for callers whose process is about to exit and cannot come back later
/// (emergency break): retries stay INSIDE the one reservation, so they never
/// multiply against the per-session budget.
///
/// <paramref name="sessionAgeHours"/> is the emergency break's measured session age
/// (<see cref="AgentErrorReport.SessionAgeHours"/>) and <paramref name="priorRegistrationFailure"/>
/// its record of earlier starts that never registered
/// (<see cref="AgentErrorReport.PriorRegistrationFailure"/>); null for every other error type.
/// </summary>
public virtual async Task TrySendAsync(
AgentErrorType errorType,
@@ -87,7 +92,9 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
long? sequenceNumber = null,
int attempts = 1,
TimeSpan? perAttemptTimeout = null,
TimeSpan? retryDelay = null)
TimeSpan? retryDelay = null,
double? sessionAgeHours = null,
RegistrationFailureSummary priorRegistrationFailure = null)
{
// Deduplicate by error type + status code: same failure category sent only once per session.
// All anti-flood checks are protected by a lock to prevent race conditions when
@@ -135,6 +142,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
Message = message,
HttpStatusCode = httpStatusCode,
SequenceNumber = sequenceNumber,
SessionAgeHours = sessionAgeHours,
PriorRegistrationFailure = priorRegistrationFailure,
AgentVersion = _agentVersion,
Timestamp = DateTime.UtcNow
};
@@ -1,5 +1,6 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Threading;
using System.Threading.Tasks;
@@ -25,6 +26,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
/// </summary>
public string Ip { get; set; }
/// <summary>"City, Region, Country" without the parts a provider did not return.</summary>
public string Describe() =>
string.Join(", ", new[] { City, Region, Country }.Where(p => !string.IsNullOrEmpty(p)));
public Dictionary<string, object> ToDictionary()
{
return new Dictionary<string, object>
@@ -45,6 +50,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
public string PrimaryError { get; set; }
public string PrimaryRetryError { get; set; }
public string FallbackError { get; set; }
public string DoGeoError { get; set; }
}
public static class GeoLocationService
@@ -52,6 +58,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
private static readonly TimeSpan RequestTimeout = TimeSpan.FromSeconds(5);
private static readonly TimeSpan RetryDelay = TimeSpan.FromSeconds(2);
internal const string IpInfoUrl = "https://ipinfo.io/json";
internal const string IfConfigCoUrl = "https://ifconfig.co/json";
/// <summary>
/// Delivery Optimization's own geo endpoint. Country and egress IP only (no region, city,
/// coordinates or timezone), but it is part of the Windows Update / DO endpoint set that
/// Autopilot needs anyway, so it stays reachable in networks that block public geo services.
/// </summary>
internal const string DoGeoUrl = "https://geo.prod.do.dsp.mp.microsoft.com/geo";
// Static HttpClient to avoid socket exhaustion from repeated short-lived instances.
// HttpClient is designed to be reused across requests.
private static readonly HttpClient SharedHttpClient = CreateSharedHttpClient();
@@ -68,7 +84,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
var attempt = new GeoLocationAttemptResult();
// Try ipinfo.io first
var (result, error) = await TryIpInfo(logger);
var (result, error) = await TryProvider("ipinfo.io", IpInfoUrl, ParseIpInfo, logger);
if (result != null)
{
attempt.Location = result;
@@ -80,7 +96,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
logger?.Info($"GeoLocation: Retrying ipinfo.io after {RetryDelay.TotalSeconds}s...");
await Task.Delay(RetryDelay);
(result, error) = await TryIpInfo(logger);
(result, error) = await TryProvider("ipinfo.io", IpInfoUrl, ParseIpInfo, logger);
if (result != null)
{
attempt.Location = result;
@@ -89,7 +105,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
attempt.PrimaryRetryError = error;
// Fallback to ifconfig.co
(result, error) = await TryIfConfigCo(logger);
(result, error) = await TryProvider("ifconfig.co", IfConfigCoUrl, ParseIfConfigCo, logger);
if (result != null)
{
attempt.Location = result;
@@ -97,108 +113,111 @@ namespace AutopilotMonitor.Agent.V2.Core.Monitoring.Transport
}
attempt.FallbackError = error;
// Last resort: Delivery Optimization geo — country only, but better than no location
(result, error) = await TryProvider("do-geo", DoGeoUrl, ParseDoGeo, logger);
if (result != null)
{
attempt.Location = result;
return attempt;
}
attempt.DoGeoError = error;
logger?.Warning("GeoLocation: All providers failed, skipping location event");
return attempt;
}
private static async Task<(GeoLocationResult result, string error)> TryIpInfo(AgentLogger logger)
private static async Task<(GeoLocationResult result, string error)> TryProvider(
string name, string url, Func<JObject, GeoLocationResult> parse, AgentLogger logger)
{
try
{
logger?.Info("GeoLocation: Querying ipinfo.io...");
logger?.Info($"GeoLocation: Querying {name}...");
using (var httpResponse = await SharedHttpClient.GetAsync("https://ipinfo.io/json"))
using (var httpResponse = await SharedHttpClient.GetAsync(url))
{
if (!httpResponse.IsSuccessStatusCode)
{
var error = $"HTTP {(int)httpResponse.StatusCode} ({httpResponse.ReasonPhrase})";
logger?.Warning($"GeoLocation: ipinfo.io failed: {error}");
logger?.Warning($"GeoLocation: {name} failed: {error}");
return (null, error);
}
var response = await httpResponse.Content.ReadAsStringAsync();
var json = JObject.Parse(response);
var result = new GeoLocationResult
var result = parse(JObject.Parse(response));
if (result == null)
{
Country = json.Value<string>("country"),
Region = json.Value<string>("region"),
City = json.Value<string>("city"),
Loc = json.Value<string>("loc"),
Timezone = json.Value<string>("timezone"),
Ip = json.Value<string>("ip"),
Source = "ipinfo"
};
var error = "No country in response";
logger?.Warning($"GeoLocation: {name} failed: {error}");
return (null, error);
}
logger?.Info($"GeoLocation: ipinfo.io returned {result.City}, {result.Region}, {result.Country}");
logger?.Info($"GeoLocation: {name} returned {result.Describe()}");
return (result, null);
}
}
catch (TaskCanceledException)
{
var error = "Timeout (5s)";
logger?.Warning($"GeoLocation: ipinfo.io failed: {error}");
var error = $"Timeout ({RequestTimeout.TotalSeconds}s)";
logger?.Warning($"GeoLocation: {name} failed: {error}");
return (null, error);
}
catch (Exception ex)
{
var error = ex.Message;
logger?.Warning($"GeoLocation: ipinfo.io failed: {error}");
logger?.Warning($"GeoLocation: {name} failed: {error}");
return (null, error);
}
}
private static async Task<(GeoLocationResult result, string error)> TryIfConfigCo(AgentLogger logger)
internal static GeoLocationResult ParseIpInfo(JObject json) =>
new GeoLocationResult
{
Country = json.Value<string>("country"),
Region = json.Value<string>("region"),
City = json.Value<string>("city"),
Loc = json.Value<string>("loc"),
Timezone = json.Value<string>("timezone"),
Ip = json.Value<string>("ip"),
Source = "ipinfo"
};
internal static GeoLocationResult ParseIfConfigCo(JObject json)
{
try
var latitude = json.Value<string>("latitude") ?? "";
var longitude = json.Value<string>("longitude") ?? "";
var loc = !string.IsNullOrEmpty(latitude) && !string.IsNullOrEmpty(longitude)
? $"{latitude},{longitude}"
: "";
return new GeoLocationResult
{
logger?.Info("GeoLocation: Falling back to ifconfig.co...");
Country = json.Value<string>("country_iso"),
Region = json.Value<string>("region_name"),
City = json.Value<string>("city"),
Loc = loc,
Timezone = json.Value<string>("time_zone"),
Ip = json.Value<string>("ip"),
Source = "ifconfig.co"
};
}
using (var httpResponse = await SharedHttpClient.GetAsync("https://ifconfig.co/json"))
{
if (!httpResponse.IsSuccessStatusCode)
{
var error = $"HTTP {(int)httpResponse.StatusCode} ({httpResponse.ReasonPhrase})";
logger?.Warning($"GeoLocation: ifconfig.co failed: {error}");
return (null, error);
}
/// <summary>
/// The DO geo response carries DO service configuration next to the geo fields; only
/// <c>CountryCode</c> and <c>ExternalIpAddress</c> are location data. Without a country
/// the response is worthless as a location, so it counts as a failure (null).
/// </summary>
internal static GeoLocationResult ParseDoGeo(JObject json)
{
var country = json.Value<string>("CountryCode");
if (string.IsNullOrEmpty(country))
return null;
var response = await httpResponse.Content.ReadAsStringAsync();
var json = JObject.Parse(response);
var latitude = json.Value<string>("latitude") ?? "";
var longitude = json.Value<string>("longitude") ?? "";
var loc = !string.IsNullOrEmpty(latitude) && !string.IsNullOrEmpty(longitude)
? $"{latitude},{longitude}"
: "";
var result = new GeoLocationResult
{
Country = json.Value<string>("country_iso"),
Region = json.Value<string>("region_name"),
City = json.Value<string>("city"),
Loc = loc,
Timezone = json.Value<string>("time_zone"),
Ip = json.Value<string>("ip"),
Source = "ifconfig.co"
};
logger?.Info($"GeoLocation: ifconfig.co returned {result.City}, {result.Region}, {result.Country}");
return (result, null);
}
}
catch (TaskCanceledException)
return new GeoLocationResult
{
var error = "Timeout (5s)";
logger?.Warning($"GeoLocation: ifconfig.co failed: {error}");
return (null, error);
}
catch (Exception ex)
{
var error = ex.Message;
logger?.Warning($"GeoLocation: ifconfig.co failed: {error}");
return (null, error);
}
Country = country,
Ip = json.Value<string>("ExternalIpAddress"),
Source = "do-geo"
};
}
}
}
@@ -479,6 +479,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
if (collectors.StallProbeEnabled)
{
// A pending tenant Hello wait (up to 60 min) is a bounded wait that resolves the
// session either way — the probe holds session_stalled back until it is over.
Func<bool>? helloWaitPending = null;
if (decisionStateProbe != null)
helloWaitPending = () => DecisionEngine.IsTenantHelloWaitPending(decisionStateProbe(), clock.UtcNow);
hosts.Add(new StallProbeHost(
sessionId: sessionId,
tenantId: tenantId,
@@ -490,7 +496,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
sources: collectors.StallProbeSources,
sessionStalledAfterProbeIndex: collectors.SessionStalledAfterProbeIndex,
harmlessModernDeploymentEventIds: collectors.ModernDeploymentHarmlessEventIds,
isDevicePreparation: isDevicePreparation));
isDevicePreparation: isDevicePreparation,
helloWaitPending: helloWaitPending));
}
// ESP policy-provider stall tripwire — always-on kernel host (no config gate, like
@@ -58,7 +58,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
string[]? sources,
int sessionStalledAfterProbeIndex,
int[]? harmlessModernDeploymentEventIds,
bool isDevicePreparation = false)
bool isDevicePreparation = false,
Func<bool>? helloWaitPending = null)
{
if (ingress == null) throw new ArgumentNullException(nameof(ingress));
if (clock == null) throw new ArgumentNullException(nameof(clock));
@@ -74,7 +75,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
sources: sources ?? new[] { "provisioning_registry", "diagnostics_registry", "eventlog", "appworkload_log" },
sessionStalledAfterProbeIndex: sessionStalledAfterProbeIndex,
harmlessModernDeploymentEventIds: harmlessModernDeploymentEventIds,
isDevicePreparation: isDevicePreparation);
isDevicePreparation: isDevicePreparation,
helloWaitPending: helloWaitPending);
_adapter = new StallProbeCollectorAdapter(_collector, ingress, clock);
_observableIngress = ingress as SignalIngress;
@@ -420,7 +420,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Orchestration
// Re-stamp AgentBootUtc so the current run's deadlines floor at "now" — the
// persisted boot anchor is from the prior run and using it would let replayed
// tail signals arm deadlines that are already past-due (premature fire).
seed = loadedState.ToBuilder().WithAgentBootUtc(agentBootUtc).Build();
// Re-stamp SessionId too: after a session rotation at registration
// (SESSION-OWNER-BINDING) the snapshot still names the refused session, and the
// timeline emitter takes each event's SessionId from this state.
var seedBuilder = loadedState.ToBuilder().WithAgentBootUtc(agentBootUtc);
seedBuilder.SessionId = sessionId;
seed = seedBuilder.Build();
signalsToReplay = CollectSignalLogTailAfter(signalLog, loadedState.LastAppliedSignalOrdinal);
branchTag = signalsToReplay.Count > 0 ? "b-tail-replay" : "b-snapshot-current";
}
@@ -18,7 +18,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Runtime
/// </para>
/// <list type="bullet">
/// <item><b>device_location</b> (or <c>agent_trace</c> on failure) — IP-based geo via
/// <see cref="GeoLocationService"/>. Skipped when <c>EnableGeoLocation=false</c>.</item>
/// <see cref="GeoLocationService"/>. Skipped when <c>EnableGeoLocation=false</c>. The
/// last-resort provider (DO geo) yields a country-only location without timezone.</item>
/// <item><b>timezone_auto_set</b> — only when <c>EnableTimezoneAutoSet=true</c> AND the
/// geo lookup returned an IANA timezone. Uses <see cref="TimezoneService"/> (tzutil).</item>
/// <item><b>ntp_time_check</b> — NTP offset from <c>NtpServer</c> (default time.windows.com).
@@ -225,7 +226,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Runtime
Source = "StartupEnvironmentProbes",
Phase = EnrollmentPhase.Unknown,
Timestamp = DateTime.UtcNow,
Message = $"Device location: {location.City}, {location.Region}, {location.Country} (via {location.Source})",
Message = $"Device location: {location.Describe()} (via {location.Source})",
Data = location.ToDictionary(),
ImmediateUpload = true,
};
@@ -266,8 +267,10 @@ namespace AutopilotMonitor.Agent.V2.Core.Runtime
{ "primaryError", attempt?.PrimaryError ?? "unknown" },
{ "primaryRetryError", attempt?.PrimaryRetryError ?? "unknown" },
{ "fallbackError", attempt?.FallbackError ?? "unknown" },
{ "doGeoError", attempt?.DoGeoError ?? "unknown" },
{ "primaryProvider", "ipinfo.io" },
{ "fallbackProvider", "ifconfig.co" },
{ "lastResortProvider", "do-geo" },
},
};
@@ -1,6 +1,8 @@
using System;
using System.IO;
using AutopilotMonitor.Agent.V2.Core.Logging;
using AutopilotMonitor.Shared.Models;
using Newtonsoft.Json;
namespace AutopilotMonitor.Agent.V2.Core.Security
{
@@ -22,6 +24,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
private readonly string _sessionFilePath;
private readonly string _sessionCreatedFilePath;
private readonly string _whiteGloveMarkerPath;
private readonly string _registrationFailurePath;
private readonly object _lockObject = new object();
public SessionIdPersistence(string dataDirectory)
@@ -35,6 +38,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
_sessionFilePath = Path.Combine(dataDirectory, "session.id");
_sessionCreatedFilePath = Path.Combine(dataDirectory, "session.created");
_whiteGloveMarkerPath = Path.Combine(dataDirectory, "whiteglove.complete");
_registrationFailurePath = Path.Combine(dataDirectory, "registration-failure.json");
}
/// <summary>
@@ -164,6 +168,55 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
lock (_lockObject) TryDelete(_whiteGloveMarkerPath, logger);
}
/// <summary>
/// Folds one failed registration run into <c>registration-failure.json</c>: the run
/// counter grows, the first failure time is kept, everything else describes the latest
/// run. A run that never registers exits and leaves nothing else behind — the emergency
/// break reads this file so the backend learns why the session never appeared.
/// Best-effort; never throws.
/// </summary>
public void RecordRegistrationFailure(RegistrationFailureSummary latestRun, AgentLogger logger = null)
{
if (latestRun == null) return;
lock (_lockObject)
{
try
{
var previous = LoadRegistrationFailureInternal();
var merged = new RegistrationFailureSummary
{
FailedRuns = (previous?.FailedRuns ?? 0) + 1,
FirstFailedAtUtc = previous?.FirstFailedAtUtc ?? latestRun.LastFailedAtUtc,
LastFailedAtUtc = latestRun.LastFailedAtUtc,
Outcome = latestRun.Outcome,
LastError = latestRun.LastError,
NetworkLinkUpAtStart = latestRun.NetworkLinkUpAtStart,
NetworkLinkUpAtEnd = latestRun.NetworkLinkUpAtEnd,
AttemptWindowSeconds = latestRun.AttemptWindowSeconds,
ConfigFetchOutcome = latestRun.ConfigFetchOutcome,
};
File.WriteAllText(_registrationFailurePath, JsonConvert.SerializeObject(merged));
logger?.Info($"SessionIdPersistence: recorded failed registration run #{merged.FailedRuns}.");
}
catch (Exception ex)
{
logger?.Warning($"SessionIdPersistence: RecordRegistrationFailure failed: {ex.Message}");
}
}
}
/// <summary>The failed-registration record since the last success, or <c>null</c> (none, unreadable).</summary>
public RegistrationFailureSummary LoadRegistrationFailure()
{
lock (_lockObject) return LoadRegistrationFailureInternal();
}
/// <summary>Drops the failed-registration record — called once registration succeeds.</summary>
public void ClearRegistrationFailure(AgentLogger logger = null)
{
lock (_lockObject) TryDelete(_registrationFailurePath, logger);
}
/// <summary>Deletes the persisted SessionId. Next <see cref="GetOrCreate"/> starts a fresh session.</summary>
public void Delete(AgentLogger logger = null)
{
@@ -172,6 +225,7 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
TryDelete(_sessionFilePath, logger);
TryDelete(_sessionCreatedFilePath, logger);
TryDelete(_whiteGloveMarkerPath, logger);
TryDelete(_registrationFailurePath, logger);
}
}
@@ -191,11 +245,28 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
var fresh = Guid.NewGuid().ToString();
WriteAtomic(fresh, logger);
SaveSessionCreatedAtInternal(DateTime.UtcNow);
// Earlier failed runs belonged to the refused session id.
TryDelete(_registrationFailurePath, logger);
logger?.Info($"SessionIdPersistence: rotated SessionId to {fresh}.");
return fresh;
}
}
private RegistrationFailureSummary LoadRegistrationFailureInternal()
{
try
{
if (!File.Exists(_registrationFailurePath)) return null;
var record = JsonConvert.DeserializeObject<RegistrationFailureSummary>(File.ReadAllText(_registrationFailurePath));
return record != null && record.FailedRuns > 0 ? record : null;
}
catch
{
// Corrupt record: report nothing rather than a half-read account.
return null;
}
}
private static void TryDelete(string path, AgentLogger logger)
{
try { if (File.Exists(path)) File.Delete(path); }
@@ -34,13 +34,16 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
/// <see cref="AuthFailureTracker"/> (which fires the first-failure distress + may trip
/// the shutdown threshold) and return <see cref="SessionRegistrationOutcome.AuthFailed"/>
/// without retrying — the backend has definitively rejected the device cert.</item>
/// <item>Exception to the above — a 403 carrying error code
/// <item>Exception to the above — a 403 or 409 carrying error code
/// <c>session_owner_mismatch</c> (SESSION-OWNER-BINDING): the persisted SessionId names a
/// session bound to another device identity (typically an Intune re-enrollment without a
/// wipe, where <c>session.id</c> survived but the certificate identity changed). That is
/// not an auth failure: the session is rotated through <c>rotateSession</c> and
/// registration retried exactly once. The tracker is NOT fed — five of these would
/// otherwise soft-shutdown a perfectly authorized agent.</item>
/// session that is not this device's — bound to another device identity (403, typically an
/// Intune re-enrollment without a wipe, where <c>session.id</c> survived but the certificate
/// identity changed) or claimed by another tenant (409). That is not an auth failure: the
/// session is rotated through <c>rotateSession</c> and registration retried exactly once.
/// The tracker is NOT fed — five of these would otherwise soft-shutdown a perfectly
/// authorized agent.</item>
/// <item>Any other coded 409 (<see cref="BackendRefusalException"/>) → <see cref="SessionRegistrationOutcome.Failed"/>
/// without retrying — a refusal with a code does not clear by waiting.</item>
/// <item>Any other exception on the last attempt → <see cref="EmergencyReporter.TrySendAsync"/>
/// with <c>AgentErrorType.RegisterSessionFailed</c> so operators see the final cause.</item>
/// </list>
@@ -113,14 +116,14 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
lastError = response?.Message ?? "(null response)";
logger.Warning($"Session registration failed: {lastError}");
}
catch (BackendAuthException ex) when (
ex.ErrorCode == Constants.AgentErrorCodes.SessionOwnerMismatch
catch (Exception ex) when (
IsSessionOwnerMismatch(ex)
&& rotateSession != null
&& rotatedFromSessionId == null)
{
// SESSION-OWNER-BINDING: the id on disk belongs to a session this device
// identity does not own. Become a new session and register once more —
// immediately, no backoff, no auth-failure bookkeeping.
// SESSION-OWNER-BINDING: the id on disk names a session this device does not
// own. Become a new session and register once more — immediately, no backoff,
// no auth-failure bookkeeping.
rotatedFromSessionId = registration.SessionId;
string newSessionId;
try
@@ -130,15 +133,25 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
catch (Exception rotateEx)
{
logger.Error($"Session owner mismatch reported by backend but session rotation failed: {rotateEx.Message}", rotateEx);
authFailureTracker?.RecordFailure(ex.StatusCode, "agent/register-session", ex.EndpointUnavailable);
return SessionRegistrationResult.AuthFailed(ex.StatusCode, ex.Message);
if (ex is BackendAuthException authEx)
{
authFailureTracker?.RecordFailure(authEx.StatusCode, "agent/register-session", authEx.EndpointUnavailable);
return SessionRegistrationResult.AuthFailed(authEx.StatusCode, authEx.Message);
}
return SessionRegistrationResult.Failed(ex.Message);
}
logger.Warning(
$"Backend refused registration with {Constants.AgentErrorCodes.SessionOwnerMismatch}: session {rotatedFromSessionId} " +
$"is bound to another device identity (re-enrollment without wipe?). Rotated to {newSessionId}; re-registering.");
$"is not this device's (bound to another device identity after a re-enrollment without wipe, or claimed by another tenant). " +
$"Rotated to {newSessionId}; re-registering.");
registration = BuildRegistration(agentConfig, agentVersion, deviceHardware);
continue;
}
catch (BackendRefusalException ex)
{
logger.Error($"Session registration refused ({ex.StatusCode}, {ex.ErrorCode}): {ex.Message}");
return SessionRegistrationResult.Failed(ex.Message);
}
catch (BackendAuthException ex)
{
logger.Error($"Session registration authentication failed ({ex.StatusCode}): {ex.Message}");
@@ -195,6 +208,12 @@ namespace AutopilotMonitor.Agent.V2.Core.Security
return SessionRegistrationResult.Failed(lastError ?? "max retries exceeded");
}
private static bool IsSessionOwnerMismatch(Exception ex)
{
var code = (ex as BackendAuthException)?.ErrorCode ?? (ex as BackendRefusalException)?.ErrorCode;
return code == Constants.AgentErrorCodes.SessionOwnerMismatch;
}
/// <summary>V1 parity exponential backoff: 2^attempt seconds.</summary>
private static Task DefaultBackoffDelay(int attempt)
=> Task.Delay(((int)Math.Pow(2, attempt)) * 1000);
@@ -24,7 +24,7 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
/// <list type="bullet">
/// <item><c>HelloCompleted</c> → <see cref="DecisionSignalKind.HelloResolved"/></item>
/// <item><c>FinalizingSetupPhaseTriggered</c> → <see cref="DecisionSignalKind.EspPhaseChanged"/> (phase=FinalizingSetup)</item>
/// <item><c>WhiteGloveCompleted</c> → <see cref="DecisionSignalKind.WhiteGloveShellCoreSuccess"/></item>
/// <item><c>WhiteGloveCompleted</c> → <see cref="DecisionSignalKind.WhiteGloveShellCoreSuccess"/> (payload: registry AccountSetup progress at that moment)</item>
/// <item><c>EspFailureDetected</c> → <see cref="DecisionSignalKind.EspTerminalFailure"/> (merged aus ShellCore + Provisioning)</item>
/// <item><c>DeviceSetupProvisioningComplete</c> → <see cref="DecisionSignalKind.DeviceSetupProvisioningComplete"/></item>
/// <item><c>AccountSetupProvisioningComplete</c> → <see cref="DecisionSignalKind.AccountSetupProvisioningComplete"/> (session 330f73f3 fix — strong AccountSetup-done gate)</item>
@@ -46,6 +46,7 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
private readonly EspAndHelloTracker _coordinator;
private readonly ISignalIngressSink _ingress;
private readonly IClock _clock;
private readonly Func<bool?> _accountSetupProgressProbe;
private bool _helloPosted;
private bool _helloWizardStartedPosted;
@@ -60,11 +61,14 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
public EspAndHelloTrackerAdapter(
EspAndHelloTracker coordinator,
ISignalIngressSink ingress,
IClock clock)
IClock clock,
Func<bool?>? accountSetupProgressProbe = null)
{
_coordinator = coordinator ?? throw new ArgumentNullException(nameof(coordinator));
_ingress = ingress ?? throw new ArgumentNullException(nameof(ingress));
_clock = clock ?? throw new ArgumentNullException(nameof(clock));
// Test seam; production reads the provisioning tracker's registry-derived flag.
_accountSetupProgressProbe = accountSetupProgressProbe ?? (() => _coordinator.AccountSetupProgressIfRead);
_coordinator.HelloCompleted += OnHelloCompleted;
_coordinator.FinalizingSetupPhaseTriggered += OnFinalizing;
@@ -216,13 +220,29 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
if (_whiteGloveSuccessPosted) return;
_whiteGloveSuccessPosted = true;
// Registry AccountSetup progress at the moment of the success page. The sealing
// classifier lifts its AccountSetup excluder only on an explicit "false" — IME logs
// AccountSetup from the device session once the device apps are done, which is no
// user activity. Sampled here so replay reads it from the SignalLog (L.2/L.5). An
// unread registry sends no key: unknown must keep the excluder, never lift it.
var progress = _accountSetupProgressProbe();
var progressText = progress.HasValue ? (progress.Value ? "true" : "false") : null;
var now = ResolveOccurredAt(out var derivedFromClock);
var derivationInputs = new Dictionary<string, string>(StringComparer.Ordinal)
{
["subSource"] = "ShellCoreTracker",
[SignalPayloadKeys.AccountSetupProgress] = progressText ?? "unread",
};
TagDerivedTimestamp(derivationInputs, derivedFromClock);
var payload = progressText == null
? null
: new Dictionary<string, string>(StringComparer.Ordinal)
{
[SignalPayloadKeys.AccountSetupProgress] = progressText,
};
_ingress.Post(
kind: DecisionSignalKind.WhiteGloveShellCoreSuccess,
occurredAtUtc: now,
@@ -231,7 +251,8 @@ namespace AutopilotMonitor.Agent.V2.Core.SignalAdapters
kind: EvidenceKind.Derived,
identifier: DetectorId,
summary: "WhiteGlove sealing success (coordinator-forwarded)",
derivationInputs: derivationInputs));
derivationInputs: derivationInputs),
payload: payload);
}
private void EmitEspFailure(EspFailureDetectedEventArgs args)
@@ -169,6 +169,15 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
_logger.Info(
$"EnrollmentTerminationHandler: handling Terminated (reason={args.Reason}, outcome={args.Outcome}, stage={args.StageName}).");
// The seal is already persisted in the decision snapshot. The marker goes down
// before anything else: the technician's reseal can kill this process at any
// later step, and a missing marker makes the next boot run the 48-hour break
// against the Part-1 session age instead of resuming Part 2.
if (isWhiteGlovePart1)
{
TrySaveWhiteGloveComplete();
}
var state = TryGetCurrentState();
// M4.6.δ — shutdown analyzers run BEFORE the dialog / diagnostics upload so
@@ -254,7 +263,8 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
// WhiteGlove Part-1 exit: keep the session alive, but announce the handoff so
// the timeline clearly marks the transition. The `whiteglove.complete` marker
// lets the next agent boot classify itself as a Part-2 resume.
// (written at the top of this handler) lets the next agent boot classify itself
// as a Part-2 resume.
if (isWhiteGlovePart1)
{
// Stop peripheral collectors BEFORE the Part-1-complete marker so their
@@ -298,7 +308,6 @@ namespace AutopilotMonitor.Agent.V2.Core.Termination
DelayLateEventGrace();
DrainSpool();
TrySaveWhiteGloveComplete();
// Option 2 (WG Part 1 graceful-exit hardening, 2026-04-30): write the
// clean-exit marker BEFORE _signalShutdown returns control to the main
// thread. The AppDomain.ProcessExit handler still writes it as a second
@@ -308,7 +308,7 @@ namespace AutopilotMonitor.Agent.V2
Func<CleanupService> cleanupServiceFactory,
AgentLogger logger,
bool consoleMode,
Action onBreakFired = null)
Action<double> onBreakFired = null)
{
try
{
@@ -347,7 +347,7 @@ namespace AutopilotMonitor.Agent.V2
// (tasks/enrollment-status-reclassification.md). Fired BEFORE cleanup, while the
// session state and network are still intact. Never throws — a send failure (e.g. no
// network) must not block the cleanup/exit that is the whole point of this guard.
try { onBreakFired?.Invoke(); }
try { onBreakFired?.Invoke(sessionAgeHours); }
catch (Exception cbEx) { logger.Debug($"Emergency-break notify callback failed: {cbEx.Message}"); }
// Write enrollment-complete.marker so the next start exits cleanly even if
@@ -273,7 +273,9 @@ namespace AutopilotMonitor.Agent.V2
var registration = Runtime.BackendSessionRegistration.Register(
bootstrap.AgentConfig, auth, telemetry.MtlsHttpClient, GetAgentVersion(), consoleMode, logger,
rotateSession: () => Runtime.BackendSessionRegistration.RotateSession(
bootstrap.AgentConfig, bootstrap.SessionPersistence, auth, transportDir, logger));
bootstrap.AgentConfig, bootstrap.SessionPersistence, auth, transportDir, logger),
sessionPersistence: bootstrap.SessionPersistence,
configFetchOutcome: runtimeConfig.RemoteConfigService.LastFetchOutcome.ToString());
if (registration.ShouldExit)
{
return registration.ExitCode;
@@ -87,7 +87,11 @@ namespace AutopilotMonitor.Agent.V2.Runtime
cleanupServiceFactory, logger, consoleMode,
// Best-effort: surface the otherwise-silent 48h break to the backend before cleanup
// (tasks/enrollment-status-reclassification.md). Never blocks the exit.
onBreakFired: () => EmergencyBreakReporter.TrySend(agentConfig, Program.GetAgentVersion(), logger)))
// The failed-registration record is read here, before the break deletes the
// session files.
onBreakFired: sessionAgeHours => EmergencyBreakReporter.TrySend(
agentConfig, Program.GetAgentVersion(), sessionAgeHours,
sessionPersistence.LoadRegistrationFailure(), logger)))
{
logger.Info("Emergency break fired — agent exiting.");
return BootstrapResult.Exit(0);
@@ -526,7 +526,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
// analytics. WhiteGlove Part-2 resume runs as a fresh Classic
// enrollment after PR-A's archive-and-reset, so the engine state is
// empty and needs the facts re-seeded — same as a first boot.
LifecycleEmitters.PostEnrollmentFactsObserved(orchestrator.IngressSink, logger);
LifecycleEmitters.PostEnrollmentFactsObserved(orchestrator.IngressSink, agentConfig.HelloWaitTimeoutSeconds, logger);
// V2 parity — post SessionStarted so the reducer establishes the session
// anchor (HandleSessionStartedV1 in DecisionEngine.Shared.cs). Skipped only
@@ -1,6 +1,7 @@
using System;
using System.Net;
using System.Net.Http;
using System.Net.NetworkInformation;
using System.Threading.Tasks;
using AutopilotMonitor.Agent.V2.Core.Configuration;
using AutopilotMonitor.Agent.V2.Core.Logging;
@@ -17,7 +18,8 @@ namespace AutopilotMonitor.Agent.V2.Runtime
/// <c>IncrementSessionEventCountAsync</c> / <c>UpdateSessionStatusAsync</c> silently
/// no-op and session status / phase / admin-overrides / validator reconcile all break.
/// On registration failure we follow V1's rule: collectors MUST NOT start and the
/// agent exits cleanly so the next Scheduled-Task tick can retry.
/// agent exits cleanly; the Scheduled Task has only a boot trigger, so the next attempt is
/// the next boot. The failed run is recorded for the emergency break report.
/// </summary>
internal static class BackendSessionRegistration
{
@@ -28,12 +30,19 @@ namespace AutopilotMonitor.Agent.V2.Runtime
string agentVersion,
bool consoleMode,
AgentLogger logger,
Func<string> rotateSession = null)
Func<string> rotateSession = null,
SessionIdPersistence sessionPersistence = null,
string configFetchOutcome = null)
{
if (agentConfig == null) throw new ArgumentNullException(nameof(agentConfig));
if (auth == null) throw new ArgumentNullException(nameof(auth));
if (logger == null) throw new ArgumentNullException(nameof(logger));
// Wall clock on purpose: a window far above the retry schedule is the evidence that
// the device slept or was suspended while this run tried to register.
var startedUtc = DateTime.UtcNow;
var linkUpAtStart = TryIsNetworkLinkUp();
var registrationResult = SessionRegistrationHelper.RegisterWithRetryAsync(
apiClient: auth.BackendApiClient,
agentConfig: agentConfig,
@@ -48,6 +57,10 @@ namespace AutopilotMonitor.Agent.V2.Runtime
rotateSession: rotateSession)
.GetAwaiter().GetResult();
RecordRegistrationRun(
sessionPersistence, registrationResult, startedUtc, DateTime.UtcNow,
linkUpAtStart, TryIsNetworkLinkUp(), configFetchOutcome, logger);
if (registrationResult.Outcome != SessionRegistrationOutcome.Succeeded)
{
logger.Error(
@@ -65,13 +78,71 @@ namespace AutopilotMonitor.Agent.V2.Runtime
return SessionRegistrationOutcomeResult.Continue(registrationResult);
}
/// <summary>Longest <see cref="RegistrationFailureSummary.LastError"/> kept on disk and on the wire.</summary>
internal const int MaxRecordedErrorLength = 256;
/// <summary>
/// Keeps <c>registration-failure.json</c> in step with this run: a failed run is folded
/// into the record (the emergency break later reports it — for a session that never
/// registered, the only account of why), a successful one clears it. Never throws.
/// </summary>
internal static void RecordRegistrationRun(
SessionIdPersistence sessionPersistence,
SessionRegistrationResult registrationResult,
DateTime startedUtc,
DateTime endedUtc,
bool? linkUpAtStart,
bool? linkUpAtEnd,
string configFetchOutcome,
AgentLogger logger)
{
if (sessionPersistence == null || registrationResult == null) return;
try
{
if (registrationResult.Outcome == SessionRegistrationOutcome.Succeeded)
{
sessionPersistence.ClearRegistrationFailure(logger);
return;
}
var error = registrationResult.ErrorMessage;
if (error != null && error.Length > MaxRecordedErrorLength)
error = error.Substring(0, MaxRecordedErrorLength);
sessionPersistence.RecordRegistrationFailure(new RegistrationFailureSummary
{
FailedRuns = 1,
FirstFailedAtUtc = endedUtc,
LastFailedAtUtc = endedUtc,
Outcome = registrationResult.Outcome.ToString(),
LastError = error,
NetworkLinkUpAtStart = linkUpAtStart,
NetworkLinkUpAtEnd = linkUpAtEnd,
AttemptWindowSeconds = Math.Round(Math.Max(0, (endedUtc - startedUtc).TotalSeconds), 1),
ConfigFetchOutcome = configFetchOutcome,
}, logger);
}
catch (Exception ex)
{
logger?.Warning($"Registration run record failed (best-effort): {ex.Message}");
}
}
private static bool? TryIsNetworkLinkUp()
{
try { return NetworkInterface.GetIsNetworkAvailable(); }
catch { return null; }
}
/// <summary>
/// SESSION-OWNER-BINDING: the one place that turns "backend says this session id is not
/// ours" into a new session. Rotates the persisted id, points the runtime config at it,
/// drops the transport spool (its lines carry the refused session's partition key and
/// would otherwise be uploaded against it after the rotation) and re-targets the
/// emergency reporter, which snapshotted the id before registration. Everything else
/// reads <c>agentConfig.SessionId</c> live after Phase 6.
/// emergency reporter, which snapshotted the id before registration. Everything else that
/// holds a session id (orchestrator, emitters, spool) is built in Phase 7 from the rotated
/// <c>agentConfig.SessionId</c> — keep it that way: a component built before Phase 6 needs
/// its own re-target here.
/// </summary>
internal static string RotateSession(
AgentConfiguration agentConfig,
@@ -36,15 +36,20 @@ namespace AutopilotMonitor.Agent.V2.Runtime
// Upper bound for the blocking wait: NIC wait + 3×15s attempts + 2×5s delays + slack.
private static readonly TimeSpan OverallBudget = TimeSpan.FromSeconds(75);
public static void TrySend(AgentConfiguration agentConfig, string agentVersion, AgentLogger logger)
public static void TrySend(
AgentConfiguration agentConfig,
string agentVersion,
double sessionAgeHours,
RegistrationFailureSummary priorRegistrationFailure,
AgentLogger logger)
{
try
{
WaitForNetwork(logger);
var auth = BackendClientFactory.BuildAuthClients(agentConfig, agentVersion, logger);
var message =
$"Agent absolute session-age emergency break fired (cap {agentConfig.AbsoluteMaxSessionHours}h) — cleaning up and exiting.";
var roundedAgeHours = Math.Round(sessionAgeHours, 1);
var message = BuildMessage(roundedAgeHours, agentConfig.AbsoluteMaxSessionHours, priorRegistrationFailure);
// The process is about to exit, so block on the otherwise fire-and-forget send
// rather than let it be abandoned mid-flight. TrySendAsync swallows its own HTTP
@@ -56,7 +61,9 @@ namespace AutopilotMonitor.Agent.V2.Runtime
message,
attempts: SendAttempts,
perAttemptTimeout: PerAttemptTimeout,
retryDelay: RetryDelay)
retryDelay: RetryDelay,
sessionAgeHours: roundedAgeHours,
priorRegistrationFailure: priorRegistrationFailure)
.Wait(OverallBudget);
}
catch (Exception ex)
@@ -65,6 +72,25 @@ namespace AutopilotMonitor.Agent.V2.Runtime
}
}
/// <summary>
/// Invariant culture: the text lands in the backend's ops feed and timeline, and a device
/// locale would otherwise print "478,2h". The registration clause names what the ops feed
/// cannot otherwise show for a session that never registered.
/// </summary>
internal static string BuildMessage(double roundedAgeHours, int capHours, RegistrationFailureSummary priorRegistrationFailure)
{
var message = FormattableString.Invariant(
$"Agent absolute session-age emergency break fired at session age {roundedAgeHours:F1}h (cap {capHours}h) — cleaning up and exiting.");
if (priorRegistrationFailure != null && priorRegistrationFailure.FailedRuns > 0)
{
message += FormattableString.Invariant(
$" {priorRegistrationFailure.FailedRuns} earlier start(s) failed to register (last: {priorRegistrationFailure.Outcome}, network link at end: {FormatLink(priorRegistrationFailure.NetworkLinkUpAtEnd)}).");
}
return message;
}
private static string FormatLink(bool? up) => up == null ? "unknown" : (up.Value ? "up" : "down");
/// <summary>
/// Boot-time NIC grace: the break fires during bootstrap, often seconds after boot while
/// Wi-Fi is still associating. Polls the cheap link-level signal — it cannot prove backend
@@ -314,7 +314,8 @@ namespace AutopilotMonitor.Agent.V2.Runtime
/// <see cref="DecisionSignalKind.EnrollmentFactsObserved"/> signal carrying the
/// registry-derived enrollment facts (<c>enrollmentType</c> + <c>isHybridJoin</c>)
/// so the reducer can seed <see cref="State.EnrollmentScenarioProfile"/> via the
/// stage-agnostic <c>HandleEnrollmentFactsObservedV1</c> handler.
/// stage-agnostic <c>HandleEnrollmentFactsObservedV1</c> handler — plus the tenant's
/// Hello wait when it extends the engine's built-in hello_safety window.
/// <para>
/// Posted immediately before <see cref="PostSessionStarted"/> so the Inspector
/// timeline reads naturally (facts → anchor) — but the reducer correctness does
@@ -329,6 +330,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
/// </summary>
public static void PostEnrollmentFactsObserved(
ISignalIngressSink ingressSink,
int helloWaitTimeoutSeconds,
AgentLogger logger)
{
try
@@ -352,6 +354,16 @@ namespace AutopilotMonitor.Agent.V2.Runtime
["isCloudPc"] = isCloudPc ? "true" : "false",
};
// Tenant Hello wait (remote config, not a registry fact) — stamped only when it
// widens the engine's built-in hello_safety window, so sessions on the default
// post the same payload as before.
var helloWaitSeconds = HelloWaitTimeout.EffectiveSeconds(helloWaitTimeoutSeconds);
if (helloWaitSeconds > HelloWaitTimeout.BuiltInSeconds)
{
payload[SignalPayloadKeys.HelloWaitTimeoutSeconds] =
helloWaitSeconds.ToString(System.Globalization.CultureInfo.InvariantCulture);
}
var evidence = new Evidence(
kind: EvidenceKind.Synthetic,
identifier: "enrollment_registry_facts_read",
@@ -364,7 +376,7 @@ namespace AutopilotMonitor.Agent.V2.Runtime
evidence: evidence,
payload: payload);
logger.Debug($"EnrollmentFactsObserved signal posted (enrollmentType={enrollmentType}, isHybridJoin={isHybridJoin}, isSelfDeployingProfile={isSelfDeploying}, isCloudPc={isCloudPc}).");
logger.Debug($"EnrollmentFactsObserved signal posted (enrollmentType={enrollmentType}, isHybridJoin={isHybridJoin}, isSelfDeployingProfile={isSelfDeploying}, isCloudPc={isCloudPc}, helloWaitSeconds={helloWaitSeconds}).");
}
catch (Exception ex)
{
@@ -0,0 +1,357 @@
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using Xunit;
namespace AutopilotMonitor.DecisionCore.Tests
{
/// <summary>
/// Tenant-extendable Hello window. The agent stamps the tenant's <c>HelloWaitTimeoutSeconds</c>
/// on <see cref="DecisionSignalKind.EnrollmentFactsObserved"/> when it exceeds the built-in
/// 300 s; every <see cref="DeadlineNames.HelloSafety"/> arm site then uses it, clamped to
/// 3600 s. Without the key the window stays exactly the built-in 300 s.
/// </summary>
public sealed class HelloSafetyTenantWindowTests
{
private static readonly DateTime T0 = new DateTime(2026, 9, 29, 8, 0, 0, DateTimeKind.Utc);
private static readonly DateTime EspExit = T0.AddMinutes(6);
[Theory]
[InlineData(null, 300)]
[InlineData("1800", 1800)]
[InlineData("301", 301)]
[InlineData("3600", 3600)]
[InlineData("86400", 3600)] // beyond the ceiling — clamped
[InlineData("300", 300)] // not an extension — ignored (the agent never stamps it)
[InlineData("120", 300)]
public void Esp_exit_arms_hello_safety_with_the_tenant_window(string? helloWait, int expectedSeconds)
{
var engine = new DecisionEngine();
var state = ReduceAll(engine, ClassicUntilEspExit(helloWait));
Assert.Equal(SessionStage.AwaitingHello, state.Stage);
var helloSafety = Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
Assert.Equal(EspExit.AddSeconds(expectedSeconds), helloSafety.DueAtUtc);
}
[Fact]
public void Tenant_window_is_recorded_set_once_with_its_source_ordinal()
{
var engine = new DecisionEngine();
var state = ReduceAll(engine, ClassicUntilEspExit("1800"));
var fact = state.ScenarioObservations.HelloWaitTimeoutSeconds;
Assert.NotNull(fact);
Assert.Equal(1800, fact!.Value);
Assert.Equal(1, fact.SourceSignalOrdinal);
// A later re-post (every agent start posts the facts again) never re-bases the window.
var repost = engine.Reduce(state, Facts(20, EspExit.AddSeconds(30), "3600"));
Assert.Equal(1800, repost.NewState.ScenarioObservations.HelloWaitTimeoutSeconds!.Value);
Assert.Equal(1, repost.NewState.ScenarioObservations.HelloWaitTimeoutSeconds!.SourceSignalOrdinal);
Assert.Empty(repost.Effects);
Assert.Equal(
EspExit.AddSeconds(1800),
Assert.Single(repost.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety).DueAtUtc);
}
[Fact]
public void Census_surfaces_the_tenant_window_only_when_recorded()
{
var engine = new DecisionEngine();
var extended = DecisionStateSignalCensus.Build(ReduceAll(engine, ClassicUntilEspExit("2700")));
Assert.Contains("hello_wait_extended", extended.SignalsSeen);
var evidence = Assert.IsAssignableFrom<IDictionary<string, object>>(extended.SignalEvidence["helloWaitExtended"]);
Assert.Equal(2700, evidence["seconds"]);
var builtIn = DecisionStateSignalCensus.Build(ReduceAll(engine, ClassicUntilEspExit(helloWait: null)));
Assert.DoesNotContain("hello_wait_extended", builtIn.SignalsSeen);
}
[Theory]
[InlineData("abc")]
[InlineData("")]
[InlineData("0")]
[InlineData("-5")]
public void Malformed_tenant_window_is_ignored(string helloWait)
{
var engine = new DecisionEngine();
var state = ReduceAll(engine, ClassicUntilEspExit(helloWait));
Assert.Null(state.ScenarioObservations.HelloWaitTimeoutSeconds);
Assert.Equal(
EspExit.AddSeconds(300),
Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.HelloSafety).DueAtUtc);
}
[Fact]
public void Late_tenant_window_extends_the_armed_hello_safety_from_its_original_anchor()
{
// A backfilled ESP exit armed the built-in window before the facts signal landed.
var engine = new DecisionEngine();
var state = ReduceAll(engine, ClassicUntilEspExit(helloWait: null));
Assert.Equal(
EspExit.AddSeconds(300),
Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.HelloSafety).DueAtUtc);
var step = engine.Reduce(state, Facts(20, EspExit.AddSeconds(10), "1800"));
Assert.Equal(SessionStage.AwaitingHello, step.NewState.Stage);
var extended = Assert.Single(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
Assert.Equal(EspExit.AddSeconds(1800), extended.DueAtUtc);
Assert.Equal(DecisionSignalKind.DeadlineFired, extended.FiresSignalKind);
Assert.Equal(DeadlineNames.HelloSafety, extended.FiresPayload![SignalPayloadKeys.Deadline]);
// The live scheduler is told to replace its timer (same name → replaced).
var effect = Assert.Single(step.Effects);
Assert.Equal(DecisionEffectKind.ScheduleDeadline, effect.Kind);
Assert.Equal(EspExit.AddSeconds(1800), effect.Deadline!.DueAtUtc);
}
[Theory]
[InlineData("300")]
[InlineData("120")]
public void Late_tenant_window_not_longer_than_the_built_in_leaves_the_deadline_alone(string helloWait)
{
var engine = new DecisionEngine();
var state = ReduceAll(engine, ClassicUntilEspExit(helloWait: null));
var step = engine.Reduce(state, Facts(20, EspExit.AddSeconds(10), helloWait));
Assert.Empty(step.Effects);
Assert.Null(step.NewState.ScenarioObservations.HelloWaitTimeoutSeconds);
Assert.Equal(
EspExit.AddSeconds(300),
Assert.Single(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety).DueAtUtc);
}
[Fact]
public void Fire_of_the_replaced_built_in_deadline_is_dead_ended_after_the_late_extension()
{
// The old timer fired while the facts signal was still queued ahead of it: the facts
// step moves hello_safety out, then the queued fire of the replaced incarnation
// arrives. It must not time Hello out at the built-in window.
var engine = new DecisionEngine();
var state = ReduceAll(engine, ClassicUntilEspExit(helloWait: null));
var builtInDue = EspExit.AddSeconds(300);
state = engine.Reduce(state, Facts(20, EspExit.AddSeconds(10), "1800")).NewState;
var stale = engine.Reduce(state, MakeSignal(21, DecisionSignalKind.DeadlineFired, builtInDue.AddSeconds(2),
Fired(DeadlineNames.HelloSafety, builtInDue)));
Assert.False(stale.Transition.Taken);
Assert.Equal("hello_safety_stale_superseded_by_rearm", stale.Transition.DeadEndReason);
Assert.Empty(stale.Effects);
Assert.Null(stale.NewState.HelloResolvedUtc);
Assert.Equal(SessionStage.AwaitingHello, stale.NewState.Stage);
var extendedDue = EspExit.AddSeconds(1800);
Assert.Equal(
extendedDue,
Assert.Single(stale.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety).DueAtUtc);
// The fire of the current incarnation still times Hello out (desktop is in → Finalizing).
var current = engine.Reduce(stale.NewState, MakeSignal(22, DecisionSignalKind.DeadlineFired, extendedDue,
Fired(DeadlineNames.HelloSafety, extendedDue)));
Assert.Equal(SessionStage.Finalizing, current.NewState.Stage);
Assert.Equal("Timeout", current.NewState.HelloOutcome!.Value);
}
[Fact]
public void Late_tenant_window_after_the_built_in_expired_leaves_the_pending_fire_to_decide()
{
var engine = new DecisionEngine();
var state = ReduceAll(engine, ClassicUntilEspExit(helloWait: null));
var step = engine.Reduce(state, Facts(20, EspExit.AddSeconds(301), "1800"));
Assert.Empty(step.Effects);
Assert.Equal(
EspExit.AddSeconds(300),
Assert.Single(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety).DueAtUtc);
// The fact is still recorded — later arm sites use it.
Assert.Equal(1800, step.NewState.ScenarioObservations.HelloWaitTimeoutSeconds!.Value);
}
[Fact]
public void Late_extension_is_replay_deterministic()
{
var engine = new DecisionEngine();
var signals = ClassicUntilEspExit(helloWait: null)
.Concat(new[] { Facts(20, EspExit.AddSeconds(10), "2400") })
.ToList();
var stepwise = ReduceAll(engine, signals);
var replayed = ReducerReplay.Replay(engine, Seed(), signals);
Assert.Equal(
Assert.Single(stepwise.Deadlines, d => d.Name == DeadlineNames.HelloSafety).DueAtUtc,
Assert.Single(replayed.Deadlines, d => d.Name == DeadlineNames.HelloSafety).DueAtUtc);
Assert.Equal(EspExit.AddSeconds(2400), replayed.Deadlines.Single(d => d.Name == DeadlineNames.HelloSafety).DueAtUtc);
}
[Fact]
public void Tenant_window_also_applies_to_the_AccountSetup_complete_deferred_promote()
{
// Strong gate arrives after the final exit and before the desktop: the deferred
// promote arms hello_safety at the gate signal — with the tenant window.
var engine = new DecisionEngine();
var state = ReduceAll(engine, new[]
{
MakeSignal(0, DecisionSignalKind.SessionStarted, T0),
Facts(1, T0.AddSeconds(5), "2400"),
MakeSignal(2, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(1), Phase("DeviceSetup")),
MakeSignal(3, DecisionSignalKind.HelloPolicyDetected, T0.AddMinutes(1).AddSeconds(5),
new Dictionary<string, string> { [SignalPayloadKeys.HelloEnabled] = "true" }),
MakeSignal(4, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(12), Phase("AccountSetup")),
MakeSignal(5, DecisionSignalKind.EspExiting, T0.AddMinutes(24)),
});
Assert.Equal(SessionStage.EspAccountSetup, state.Stage);
var gate = T0.AddMinutes(25);
var step = engine.Reduce(state, MakeSignal(6, DecisionSignalKind.AccountSetupProvisioningComplete, gate));
Assert.Equal(SessionStage.AwaitingHello, step.NewState.Stage);
Assert.Equal(
gate.AddSeconds(2400),
Assert.Single(step.NewState.Deadlines, d => d.Name == DeadlineNames.HelloSafety).DueAtUtc);
}
[Theory]
[InlineData(null, 30 * 60)]
[InlineData("1200", 30 * 60)] // a shorter tenant window never shortens the WDP backstop
[InlineData("3600", 3600)]
public void Device_preparation_backstop_is_never_shorter_than_the_tenant_window(string? helloWait, int expectedSeconds)
{
// WDP has no ESP, so hello_safety never arms; the desktop-armed backstop is the last
// net behind the tracker's Hello completion timer, which runs on the tenant budget.
var engine = new DecisionEngine();
var facts = new Dictionary<string, string>
{
[SignalPayloadKeys.EnrollmentType] = "v2",
[SignalPayloadKeys.EnrollmentTypeDeterministic] = "true",
};
if (helloWait != null)
facts[SignalPayloadKeys.HelloWaitTimeoutSeconds] = helloWait;
var desktop = T0.AddMinutes(9);
var state = ReduceAll(engine, new[]
{
MakeSignal(0, DecisionSignalKind.SessionStarted, T0),
MakeSignal(1, DecisionSignalKind.EnrollmentFactsObserved, T0, facts),
MakeSignal(2, DecisionSignalKind.DesktopArrived, desktop),
});
Assert.Equal(
desktop.AddSeconds(expectedSeconds),
Assert.Single(state.Deadlines, d => d.Name == DeadlineNames.DevicePrepCompletion).DueAtUtc);
}
[Fact]
public void IsTenantHelloWaitPending_is_true_only_while_an_extended_window_is_armed_and_not_yet_due()
{
var engine = new DecisionEngine();
var awaiting = ReduceAll(engine, ClassicUntilEspExit("1800"));
var due = EspExit.AddSeconds(1800);
Assert.True(DecisionEngine.IsTenantHelloWaitPending(awaiting, EspExit.AddMinutes(1)));
Assert.True(DecisionEngine.IsTenantHelloWaitPending(awaiting, due.AddSeconds(-1)));
Assert.False(DecisionEngine.IsTenantHelloWaitPending(awaiting, due));
Assert.False(DecisionEngine.IsTenantHelloWaitPending(awaiting, due.AddSeconds(1)));
Assert.False(DecisionEngine.IsTenantHelloWaitPending(null, EspExit));
Assert.False(DecisionEngine.IsTenantHelloWaitPending(Seed(), EspExit));
// The built-in window never counts: default tenants keep their stall reporting.
var builtIn = ReduceAll(engine, ClassicUntilEspExit(helloWait: null));
Assert.Contains(builtIn.Deadlines, d => d.Name == DeadlineNames.HelloSafety);
Assert.False(DecisionEngine.IsTenantHelloWaitPending(builtIn, EspExit.AddMinutes(1)));
// Hello resolved → the deadline is cancelled → nothing pending any more.
var resolved = engine.Reduce(awaiting, MakeSignal(20, DecisionSignalKind.HelloResolved, EspExit.AddMinutes(10),
new Dictionary<string, string> { [SignalPayloadKeys.HelloOutcome] = "completed" })).NewState;
Assert.False(DecisionEngine.IsTenantHelloWaitPending(resolved, EspExit.AddMinutes(11)));
}
[Fact]
public void IsTenantHelloWaitPending_covers_the_device_preparation_backstop()
{
var engine = new DecisionEngine();
var desktop = T0.AddMinutes(9);
var state = ReduceAll(engine, new[]
{
MakeSignal(0, DecisionSignalKind.SessionStarted, T0),
MakeSignal(1, DecisionSignalKind.EnrollmentFactsObserved, T0, new Dictionary<string, string>
{
[SignalPayloadKeys.EnrollmentType] = "v2",
[SignalPayloadKeys.EnrollmentTypeDeterministic] = "true",
[SignalPayloadKeys.HelloWaitTimeoutSeconds] = "3600",
}),
MakeSignal(2, DecisionSignalKind.DesktopArrived, desktop),
});
Assert.True(DecisionEngine.IsTenantHelloWaitPending(state, desktop.AddMinutes(59)));
Assert.False(DecisionEngine.IsTenantHelloWaitPending(state, desktop.AddMinutes(60)));
}
// ============================================================ helpers
private static DecisionState Seed() => DecisionState.CreateInitial("sess-hello-window", "tenant", T0);
/// <summary>
/// Classic user-driven flow up to the final ESP exit: strong gate and desktop in place,
/// Hello policy unknown — so the exit promotes to AwaitingHello and arms hello_safety.
/// </summary>
private static List<DecisionSignal> ClassicUntilEspExit(string? helloWait)
{
var signals = new List<DecisionSignal> { MakeSignal(0, DecisionSignalKind.SessionStarted, T0) };
if (helloWait != null)
signals.Add(Facts(1, T0.AddSeconds(5), helloWait));
signals.Add(MakeSignal(2, DecisionSignalKind.EspPhaseChanged, T0.AddMinutes(1), Phase("AccountSetup")));
signals.Add(MakeSignal(3, DecisionSignalKind.AccountSetupProvisioningComplete, T0.AddMinutes(2)));
signals.Add(MakeSignal(4, DecisionSignalKind.DesktopArrived, T0.AddMinutes(3)));
signals.Add(MakeSignal(5, DecisionSignalKind.EspExiting, EspExit));
return signals;
}
private static DecisionState ReduceAll(DecisionEngine engine, IEnumerable<DecisionSignal> signals)
{
var state = Seed();
foreach (var signal in signals)
state = engine.Reduce(state, signal).NewState;
return state;
}
private static DecisionSignal Facts(long ordinal, DateTime occurredAtUtc, string helloWait) =>
MakeSignal(ordinal, DecisionSignalKind.EnrollmentFactsObserved, occurredAtUtc,
new Dictionary<string, string> { [SignalPayloadKeys.HelloWaitTimeoutSeconds] = helloWait });
private static Dictionary<string, string> Fired(string deadlineName, DateTime dueAtUtc) =>
new Dictionary<string, string>
{
[SignalPayloadKeys.Deadline] = deadlineName,
[SignalPayloadKeys.DeadlineDueAtUtc] = dueAtUtc.ToString("O", CultureInfo.InvariantCulture),
};
private static Dictionary<string, string> Phase(string phase) =>
new Dictionary<string, string> { [SignalPayloadKeys.EspPhase] = phase };
private static DecisionSignal MakeSignal(
long ordinal,
DecisionSignalKind kind,
DateTime occurredAtUtc,
IReadOnlyDictionary<string, string>? payload = null) =>
new DecisionSignal(
sessionSignalOrdinal: ordinal,
sessionTraceOrdinal: ordinal,
kind: kind,
kindSchemaVersion: 1,
occurredAtUtc: occurredAtUtc,
sourceOrigin: "test",
evidence: new Evidence(EvidenceKind.Synthetic, $"t-{kind}-{ordinal}", $"synthetic {kind}"),
payload: payload);
}
}
@@ -522,7 +522,9 @@ namespace AutopilotMonitor.DecisionCore.Tests
.WithSkipUserEsp(value: true, sourceSignalOrdinal: 4)
.WithEspSyncFailureTimeoutMinutes(value: 90, sourceSignalOrdinal: 5)
.WithRegistrySelfDeployingProfile(value: false, sourceSignalOrdinal: 6)
.WithCloudPc(value: true, sourceSignalOrdinal: 7))
.WithCloudPc(value: true, sourceSignalOrdinal: 7)
.WithAccountSetupProgressAtWhiteGloveSuccess(value: false, sourceSignalOrdinal: 8)
.WithHelloWaitTimeoutSeconds(value: 1800, sourceSignalOrdinal: 9))
.Build();
var json = StateSerializer.Serialize(state);
@@ -538,6 +540,10 @@ namespace AutopilotMonitor.DecisionCore.Tests
Assert.Equal(6, o.RegistrySelfDeployingProfile!.SourceSignalOrdinal);
Assert.True(o.CloudPc!.Value);
Assert.Equal(7, o.CloudPc!.SourceSignalOrdinal);
Assert.False(o.AccountSetupProgressAtWhiteGloveSuccess!.Value);
Assert.Equal(8, o.AccountSetupProgressAtWhiteGloveSuccess!.SourceSignalOrdinal);
Assert.Equal(1800, o.HelloWaitTimeoutSeconds!.Value);
Assert.Equal(9, o.HelloWaitTimeoutSeconds!.SourceSignalOrdinal);
// Never-observed facts must rehydrate as null (null ≠ veto — session 62e603c9).
Assert.Null(o.WhiteGloveSealingPatternSeen);
@@ -31,7 +31,6 @@ namespace AutopilotMonitor.DecisionCore.Tests
DecisionSignalKind.HelloResolved,
DecisionSignalKind.ImeUserSessionCompleted,
DecisionSignalKind.DeviceSetupProvisioningComplete,
DecisionSignalKind.WhiteGloveShellCoreSuccess,
DecisionSignalKind.WhiteGloveSealingPatternDetected,
DecisionSignalKind.AadUserJoinedLate,
DecisionSignalKind.DeviceInfoCollected,
@@ -0,0 +1,155 @@
using System;
using System.Collections.Generic;
using AutopilotMonitor.DecisionCore.Classifiers;
using AutopilotMonitor.DecisionCore.Engine;
using AutopilotMonitor.DecisionCore.Serialization;
using AutopilotMonitor.DecisionCore.Signals;
using AutopilotMonitor.DecisionCore.State;
using Xunit;
namespace AutopilotMonitor.DecisionCore.Tests
{
/// <summary>
/// The sealing classifier's AccountSetup excluder (-40) applies only when the IME reported
/// the AccountSetup phase AND the WhiteGlove_Success signal did not report an untouched
/// AccountSetup registry. IME logs "EspPhase: AccountSetup" from the device session once
/// the device apps are done — on pre-provisioning with the user ESP enabled that line
/// routinely precedes the technician's success page and used to leave Part 1 unsealed.
/// The gate may only ever lift the excluder, never add one.
/// </summary>
public sealed class WhiteGloveAccountSetupProgressGateTests
{
private static readonly DateTime T0 = new DateTime(2026, 9, 29, 7, 25, 9, DateTimeKind.Utc);
[Fact]
public void Ime_account_setup_with_untouched_registry_seals_on_the_fast_path()
{
var step = new DecisionEngine().Reduce(
StateWithImeAccountSetup(), MakeShellCoreSignal(ordinal: 3, accountSetupProgress: "false"));
Assert.Equal(SessionStage.WhiteGloveSealed, step.NewState.Stage);
Assert.Equal(SessionOutcome.WhiteGlovePart1Sealed, step.NewState.Outcome);
Assert.Equal(HypothesisLevel.Confirmed, step.NewState.ClassifierOutcomes.WhiteGloveSealing.Level);
Assert.Equal(WhiteGloveSealingClassifier.WeightShellCoreWhiteGloveSuccess,
step.NewState.ClassifierOutcomes.WhiteGloveSealing.Score);
var fact = step.NewState.ScenarioObservations.AccountSetupProgressAtWhiteGloveSuccess;
Assert.NotNull(fact);
Assert.False(fact!.Value);
Assert.Equal(3, fact.SourceSignalOrdinal);
}
[Fact]
public void Ime_account_setup_with_registry_progress_keeps_the_excluder()
{
var step = new DecisionEngine().Reduce(
StateWithImeAccountSetup(), MakeShellCoreSignal(ordinal: 3, accountSetupProgress: "true"));
Assert.Equal(SessionStage.EspAccountSetup, step.NewState.Stage);
Assert.Contains(step.Effects, e => e.Kind == DecisionEffectKind.RunClassifier);
Assert.True(DecisionEngine.BuildWhiteGloveSealingSnapshot(step.NewState).HasAccountSetupActivity);
}
[Fact]
public void Missing_progress_payload_keeps_the_excluder_for_older_agents()
{
var step = new DecisionEngine().Reduce(
StateWithImeAccountSetup(), MakeShellCoreSignal(ordinal: 3, accountSetupProgress: null));
Assert.Equal(SessionStage.EspAccountSetup, step.NewState.Stage);
Assert.Null(step.NewState.ScenarioObservations.AccountSetupProgressAtWhiteGloveSuccess);
Assert.True(DecisionEngine.BuildWhiteGloveSealingSnapshot(step.NewState).HasAccountSetupActivity);
}
[Theory]
[InlineData("")]
[InlineData("yes")]
public void Unparseable_progress_payload_is_treated_as_missing(string raw)
{
var step = new DecisionEngine().Reduce(
StateWithImeAccountSetup(), MakeShellCoreSignal(ordinal: 3, accountSetupProgress: raw));
Assert.Null(step.NewState.ScenarioObservations.AccountSetupProgressAtWhiteGloveSuccess);
Assert.NotEqual(SessionStage.WhiteGloveSealed, step.NewState.Stage);
}
[Fact]
public void Registry_progress_without_ime_account_setup_adds_no_excluder()
{
var state = DecisionState.CreateInitial("s-gate", "t-gate").ToBuilder()
.WithStage(SessionStage.EspDeviceSetup)
.WithStepIndex(1)
.WithLastAppliedSignalOrdinal(0)
.Build();
var step = new DecisionEngine().Reduce(state, MakeShellCoreSignal(ordinal: 1, accountSetupProgress: "true"));
Assert.Equal(SessionStage.WhiteGloveSealed, step.NewState.Stage);
}
[Fact]
public void Untouched_registry_does_not_lift_a_hard_excluder()
{
var builder = DecisionState.CreateInitial("s-gate", "t-gate").ToBuilder()
.WithStage(SessionStage.EspAccountSetup)
.WithStepIndex(3)
.WithLastAppliedSignalOrdinal(2);
builder.AccountSetupEnteredUtc = new SignalFact<DateTime>(T0.AddMinutes(-3), 1);
builder.DesktopArrivedUtc = new SignalFact<DateTime>(T0.AddMinutes(-1), 2);
var step = new DecisionEngine().Reduce(builder.Build(), MakeShellCoreSignal(ordinal: 3, accountSetupProgress: "false"));
Assert.NotEqual(SessionStage.WhiteGloveSealed, step.NewState.Stage);
var snapshot = DecisionEngine.BuildWhiteGloveSealingSnapshot(step.NewState);
Assert.False(snapshot.HasAccountSetupActivity);
Assert.True(snapshot.DesktopArrived);
}
[Fact]
public void Progress_fact_survives_the_state_serializer()
{
var step = new DecisionEngine().Reduce(
StateWithImeAccountSetup(), MakeShellCoreSignal(ordinal: 3, accountSetupProgress: "false"));
var roundTripped = StateSerializer.Deserialize(StateSerializer.Serialize(step.NewState));
var fact = roundTripped.ScenarioObservations.AccountSetupProgressAtWhiteGloveSuccess;
Assert.NotNull(fact);
Assert.False(fact!.Value);
Assert.Equal(3, fact.SourceSignalOrdinal);
}
private static DecisionState StateWithImeAccountSetup()
{
// The IME device-session line: AccountSetup entered before the success page.
var builder = DecisionState.CreateInitial("s-gate", "t-gate").ToBuilder()
.WithStage(SessionStage.EspAccountSetup)
.WithStepIndex(3)
.WithLastAppliedSignalOrdinal(2);
builder.AccountSetupEnteredUtc = new SignalFact<DateTime>(T0.AddMinutes(-3), 2);
return builder.Build();
}
private static DecisionSignal MakeShellCoreSignal(long ordinal, string? accountSetupProgress)
{
var payload = new Dictionary<string, string>(StringComparer.Ordinal);
if (accountSetupProgress != null)
{
payload[SignalPayloadKeys.AccountSetupProgress] = accountSetupProgress;
}
return new DecisionSignal(
sessionSignalOrdinal: ordinal,
sessionTraceOrdinal: ordinal,
kind: DecisionSignalKind.WhiteGloveShellCoreSuccess,
kindSchemaVersion: 1,
occurredAtUtc: T0,
sourceOrigin: "EspAndHelloTracker",
evidence: new Evidence(
kind: EvidenceKind.Raw,
identifier: "ShellCore-62407",
summary: "WhiteGlove_Success"),
payload: payload);
}
}
}
@@ -0,0 +1,276 @@
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Functions.Config;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Azure;
using Azure.Data.Tables;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Newtonsoft.Json.Linq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// D-285: the admin configuration row is changed only through the changed-columns write — fresh
/// read, mutation on a copy, only the columns whose value changed are laid onto the freshly read
/// row, conditional on its ETag. Pins the 2026-09-25 incident: a Global Settings page loaded before
/// an agent release saved one toggle and reverted the release's hashes with the full-model PUT.
/// </summary>
public class AdminConfigurationChangedFieldsTests
{
private const string ReleaseVersion = "2.0.1465";
private const string ReleaseZipSha = "9f0b5403da7e49202274fdf8f38675bc5cfea85f7d881e85d5fe4bd71308250e";
private const string ReleaseExeSha = "503dd02b3c3690acfe20c44dfce82c367ca1e56dcedaba3ecc91ff6ce893d726";
/// <summary>
/// A one-row table that behaves like storage: reads hand out copies with the current ETag,
/// a replace with a stale ETag fails 412, a create on an existing row fails 409.
/// </summary>
private sealed class FakeAdminTable
{
private int _version = 1;
public TableEntity? Current;
public int Writes;
public TableUpdateMode? LastMode;
/// <summary>Runs once, right before the first replace is checked — a concurrent writer.</summary>
public Action<FakeAdminTable>? BeforeFirstReplace;
public TableConfigRepository Sut { get; }
public FakeAdminTable(TableEntity? current)
{
Current = current;
var table = new Mock<TableClient>();
table.Setup(c => c.GetEntityAsync<TableEntity>(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<string>>(), It.IsAny<CancellationToken>()))
.Returns(() => Current == null
? throw new RequestFailedException(404, "Not Found", "ResourceNotFound", null)
: Task.FromResult(Response.FromValue(Copy(Current, _version), Mock.Of<Response>())));
table.Setup(c => c.UpdateEntityAsync(
It.IsAny<TableEntity>(), It.IsAny<ETag>(), It.IsAny<TableUpdateMode>(), It.IsAny<CancellationToken>()))
.Returns<TableEntity, ETag, TableUpdateMode, CancellationToken>((e, ifMatch, mode, _) =>
{
if (BeforeFirstReplace != null)
{
var concurrent = BeforeFirstReplace;
BeforeFirstReplace = null;
concurrent(this);
}
if (ifMatch != Etag(_version))
throw new RequestFailedException(412, "Precondition Failed", "UpdateConditionNotSatisfied", null);
Current = Copy(e, _version);
_version++;
Writes++;
LastMode = mode;
return Task.FromResult(Mock.Of<Response>());
});
table.Setup(c => c.AddEntityAsync(It.IsAny<TableEntity>(), It.IsAny<CancellationToken>()))
.Returns<TableEntity, CancellationToken>((e, _) =>
{
if (Current != null)
throw new RequestFailedException(409, "Conflict", "EntityAlreadyExists", null);
Current = Copy(e, _version);
Writes++;
return Task.FromResult(Mock.Of<Response>());
});
var serviceClient = new Mock<TableServiceClient>();
serviceClient.Setup(c => c.GetTableClient(It.IsAny<string>())).Returns(table.Object);
Sut = new TableConfigRepository(
new TableStorageService(serviceClient.Object, NullLogger<TableStorageService>.Instance),
new Mock<IConfigBackupRepository>().Object,
NullLogger<TableConfigRepository>.Instance);
}
/// <summary>A write by someone else (the release pipeline's Merge), bumping the ETag.</summary>
public void ExternalMerge(string column, object value)
{
Current![column] = value;
_version++;
}
private static ETag Etag(int version) => new($"W/\"v{version}\"");
private static TableEntity Copy(TableEntity source, int version)
=> new(source.Where(kv => kv.Key != "odata.etag").ToDictionary(kv => kv.Key, kv => kv.Value)) { ETag = Etag(version) };
}
/// <summary>The stored row right after the agent release: the pipeline's hashes plus a column this build does not know.</summary>
private static TableEntity ReleasedRow()
{
var row = TableConfigRepository.ConvertToAdminTableEntity(new AdminConfiguration
{
UpdatedBy = "ga@operator.example",
GlobalRateLimitRequestsPerMinute = 100,
NvdApiKey = "stored-key",
LatestAgentV2Version = ReleaseVersion,
LatestAgentV2Sha256 = ReleaseZipSha,
LatestAgentV2ExeSha256 = ReleaseExeSha,
});
row["ColumnOfANewerBuild"] = "keep-me";
return row;
}
[Fact]
public async Task Stale_page_toggle_writes_only_that_field_and_keeps_the_release_hashes()
{
var table = new FakeAdminTable(ReleasedRow());
// The page was loaded before the release; it sends only what the operator changed.
var fields = JObject.Parse("{ \"enforceClientAppBinding\": true }");
Assert.Null(PatchAdminConfigurationFunction.CheckFields(fields));
var result = await table.Sut.UpdateAdminConfigurationAsync(
draft => PatchAdminConfigurationFunction.ApplyFields(draft, fields), "ga@operator.example");
Assert.Null(result.Error);
Assert.Equal(new[] { "EnforceClientAppBinding" }, result.ChangedColumns);
var row = table.Current!;
Assert.True(row.GetBoolean("EnforceClientAppBinding"));
Assert.Equal(ReleaseVersion, row.GetString("LatestAgentV2Version"));
Assert.Equal(ReleaseZipSha, row.GetString("LatestAgentV2Sha256"));
Assert.Equal(ReleaseExeSha, row.GetString("LatestAgentV2ExeSha256"));
Assert.Equal("keep-me", row.GetString("ColumnOfANewerBuild"));
Assert.Equal("stored-key", row.GetString("NvdApiKey"));
Assert.Equal(TableUpdateMode.Replace, table.LastMode);
Assert.Equal(1, table.Writes);
}
[Fact]
public async Task A_concurrent_pipeline_write_is_reread_and_kept()
{
var table = new FakeAdminTable(ReleasedRow())
{
// The release pipeline lands between the backend's read and its write.
BeforeFirstReplace = t => t.ExternalMerge("LatestAgentV2Version", "2.0.1466"),
};
var result = await table.Sut.UpdateAdminConfigurationAsync(
c => { c.GlobalRateLimitRequestsPerMinute = 250; return null; }, "System (test)");
Assert.Null(result.Error);
Assert.Equal("2.0.1466", table.Current!.GetString("LatestAgentV2Version"));
Assert.Equal(250, table.Current.GetInt32("GlobalRateLimitRequestsPerMinute"));
Assert.Equal(1, table.Writes);
}
[Fact]
public async Task A_mutation_that_changes_nothing_writes_nothing()
{
var table = new FakeAdminTable(ReleasedRow());
var result = await table.Sut.UpdateAdminConfigurationAsync(
c => { c.GlobalRateLimitRequestsPerMinute = 100; return null; }, "ga@operator.example");
Assert.Null(result.Error);
Assert.Empty(result.ChangedColumns);
Assert.Equal(0, table.Writes);
}
[Fact]
public async Task A_rejected_mutation_writes_nothing()
{
var table = new FakeAdminTable(ReleasedRow());
var result = await table.Sut.UpdateAdminConfigurationAsync(
c => { c.GlobalRateLimitRequestsPerMinute = 999; return "nope"; }, "ga@operator.example");
Assert.Equal("nope", result.Error);
Assert.Equal(0, table.Writes);
Assert.Equal(100, table.Current!.GetInt32("GlobalRateLimitRequestsPerMinute"));
}
[Fact]
public async Task A_null_value_drops_its_column()
{
var table = new FakeAdminTable(ReleasedRow());
var result = await table.Sut.UpdateAdminConfigurationAsync(
c => { c.NvdApiKey = null!; return null; }, "ga@operator.example"); // what a JSON null in a patch does
Assert.Equal(new[] { "NvdApiKey" }, result.ChangedColumns);
Assert.False(table.Current!.ContainsKey("NvdApiKey"));
Assert.Equal(ReleaseExeSha, table.Current.GetString("LatestAgentV2ExeSha256"));
}
[Fact]
public async Task Without_a_row_the_first_change_creates_it()
{
var table = new FakeAdminTable(current: null);
var result = await table.Sut.UpdateAdminConfigurationAsync(
c => { c.EnforceClientAppBinding = true; return null; }, "ga@operator.example");
Assert.Null(result.Error);
Assert.True(table.Current!.GetBoolean("EnforceClientAppBinding"));
Assert.Equal(1, table.Writes);
}
[Fact]
public async Task Default_creation_never_replaces_an_existing_row()
{
var table = new FakeAdminTable(ReleasedRow());
var created = await table.Sut.CreateAdminConfigurationIfMissingAsync(AdminConfiguration.CreateDefault());
Assert.False(created);
Assert.Equal(ReleaseVersion, table.Current!.GetString("LatestAgentV2Version"));
}
[Theory]
[InlineData("latestAgentV2Version")]
[InlineData("LatestAgentV2Sha256")]
[InlineData("latestAgentV2ExeSha256")]
[InlineData("latestBootstrapV2ScriptVersion")]
[InlineData("updatedBy")]
[InlineData("lastUpdated")]
[InlineData("planTierDefinitionsJson")]
[InlineData("msrcLastSyncUtc")]
public void The_gate_rejects_pipeline_stamp_and_flow_owned_fields(string field)
{
var error = PatchAdminConfigurationFunction.CheckFields(new JObject { [field] = "x" });
Assert.NotNull(error);
Assert.Contains("not writable", error);
}
[Fact]
public void The_gate_rejects_unknown_fields_and_redacted_placeholders()
{
Assert.Contains("Unknown field", PatchAdminConfigurationFunction.CheckFields(new JObject { ["noSuchField"] = 1 }));
Assert.Contains("placeholder", PatchAdminConfigurationFunction.CheckFields(
new JObject { ["nvdApiKey"] = Constants.RedactedSecretPlaceholder }));
}
[Fact]
public void Patchable_fields_are_the_persisted_settings()
{
var patchable = PatchAdminConfigurationFunction.PatchableFields;
Assert.Contains("EnforceClientAppBinding", patchable);
Assert.Contains("McpClientRegistrationEnabled", patchable);
Assert.Contains("McpAccessPolicy", patchable);
Assert.Contains("NvdApiKey", patchable);
Assert.Contains("OpsNotificationChannelsJson", patchable);
Assert.DoesNotContain("LatestAgentV2ExeSha256", patchable);
Assert.DoesNotContain("PartitionKey", patchable);
Assert.DoesNotContain("UpdatedBy", patchable);
}
[Fact]
public void Applying_validates_the_result_without_echoing_values()
{
var draft = AdminConfiguration.CreateDefault();
Assert.Contains("at least 1", PatchAdminConfigurationFunction.ApplyFields(
draft, JObject.Parse("{ \"globalRateLimitRequestsPerMinute\": 0 }")));
var typeError = PatchAdminConfigurationFunction.ApplyFields(
AdminConfiguration.CreateDefault(), JObject.Parse("{ \"collectorIdleTimeoutMinutes\": \"https://secret.example/sas?sig=abc\" }"));
Assert.NotNull(typeError);
Assert.DoesNotContain("secret.example", typeError);
}
}
@@ -14,7 +14,7 @@ namespace AutopilotMonitor.Functions.Tests;
/// for the device path and UserRateLimitMiddleware for the user path). These tests pin the
/// removal of the former background sync job:
/// <list type="bullet">
/// <item>Saving the admin config must NOT enumerate or write ANY tenant configuration.</item>
/// <item>Changing the admin config must NOT enumerate or write ANY tenant configuration.</item>
/// </list>
/// <para>
/// Historical context: the sync used to copy <c>GlobalRateLimitRequestsPerMinute</c> into every
@@ -28,30 +28,35 @@ namespace AutopilotMonitor.Functions.Tests;
public class AdminConfigurationRateLimitSyncTests
{
[Fact]
public async Task SaveConfigurationAsync_DoesNotEnumerateOrWriteTenantConfigs()
public async Task UpdateAsync_DoesNotEnumerateOrWriteTenantConfigs()
{
var repo = new Mock<IConfigRepository>(MockBehavior.Strict);
repo.Setup(r => r.SaveAdminConfigurationAsync(It.IsAny<AdminConfiguration>()))
.ReturnsAsync(true);
repo.Setup(r => r.UpdateAdminConfigurationAsync(
It.IsAny<Func<AdminConfiguration, string?>>(), It.IsAny<string>(), It.IsAny<string?>()))
.ReturnsAsync(new AdminConfigurationUpdateResult
{
Before = new AdminConfiguration(),
After = new AdminConfiguration(),
ChangedColumns = new[] { "GlobalRateLimitRequestsPerMinute", "UserRateLimitRequestsPerMinute" },
});
var sut = new AdminConfigurationService(
repo.Object,
NullLogger<AdminConfigurationService>.Instance,
new MemoryCache(new MemoryCacheOptions()));
await sut.SaveConfigurationAsync(new AdminConfiguration
await sut.UpdateAsync(c =>
{
UpdatedBy = "global-admin@contoso.com",
GlobalRateLimitRequestsPerMinute = 200,
UserRateLimitRequestsPerMinute = 240,
});
c.GlobalRateLimitRequestsPerMinute = 200;
c.UserRateLimitRequestsPerMinute = 240;
return null;
}, "global-admin@contoso.com");
// The admin config row is written...
repo.Verify(r => r.SaveAdminConfigurationAsync(It.IsAny<AdminConfiguration>()), Times.Once);
repo.Verify(r => r.UpdateAdminConfigurationAsync(
It.IsAny<Func<AdminConfiguration, string?>>(), "global-admin@contoso.com", It.IsAny<string?>()), Times.Once);
// ...but NO tenant configuration is enumerated or mutated (sync removed).
repo.Verify(r => r.GetAllTenantConfigurationsAsync(), Times.Never);
repo.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>()), Times.Never);
repo.Verify(r => r.SaveTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
repo.VerifyNoTenantConfigWrite();
}
}
@@ -1,4 +1,5 @@
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Functions.Admin;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Functions.Services.GraphResolution;
using AutopilotMonitor.Functions.Services.Notifications;
@@ -32,6 +33,7 @@ public class AppHomingServiceTests
private readonly Mock<AdminConfigurationService> _adminConfigMock;
private readonly Mock<TenantConfigurationService> _tenantConfigMock;
private readonly StoredTenantConfig _store;
private readonly Mock<GraphTokenService> _graphTokenMock;
private readonly Mock<IGraphFeatureDetector> _detectorMock;
private readonly Mock<IMaintenanceRepository> _maintenanceMock;
@@ -99,13 +101,10 @@ public class AppHomingServiceTests
new TelemetryClient(new TelemetryConfiguration { DisableTelemetry = true }));
// Defaults: flag on, legacy-homed config exists, both apps mint a role-less token (the
// superset rule passes trivially). FlipAsync reads via the cache-bypassing
// GetConfigurationFreshAsync (read-modify-write on the whole entity).
// superset rule passes trivially). FlipAsync writes through UpdateAsync (conditional
// write on the fresh row), stubbed over one stored row.
_adminConfigMock.Setup(x => x.IsSelfServiceAppHomingEnabledAsync()).ReturnsAsync(true);
var config = LegacyHomedConfig();
_tenantConfigMock.Setup(x => x.GetConfigurationFreshAsync(TenantId)).ReturnsAsync(config);
_tenantConfigMock.Setup(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.Returns(Task.CompletedTask);
_store = _tenantConfigMock.StubWrites(TenantId, LegacyHomedConfig());
SetupProbe(GraphTokenResult.Success(Jwt()));
SetupLegacyToken(GraphTokenResult.Success(Jwt()));
}
@@ -143,8 +142,7 @@ public class AppHomingServiceTests
It.IsAny<CancellationToken>()))
.ReturnsAsync(result);
private void VerifyNeverSaved() =>
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
private void VerifyNeverSaved() => Assert.Empty(_store.Writes);
// ── IsFunnelEligibleAsync ───────────────────────────────────────────────
@@ -340,9 +338,10 @@ public class AppHomingServiceTests
var outcome = await _sut.TryAutoFlipToPrimaryAsync(LegacyHomedConfig(), Actor);
Assert.Equal(AppHomingAutoFlipOutcome.Flipped, outcome.Outcome);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.Is<TenantConfiguration>(c =>
string.Equals(c.HomedAppClientId, PrimaryId, StringComparison.OrdinalIgnoreCase)
&& c.UpdatedBy == Actor), It.IsAny<string?>(), It.IsAny<string?>()), Times.Once);
var (written, source, _) = Assert.Single(_store.Writes);
Assert.Equal(PrimaryId, written.HomedAppClientId, ignoreCase: true);
Assert.Equal(Actor, written.UpdatedBy);
Assert.Equal("app-homing", source);
_detectorMock.Verify(x => x.InvalidateTenant(TenantId), Times.Once);
_maintenanceMock.Verify(m => m.LogAuditEntryAsync(TenantId, "UPDATE", "TenantConfiguration",
TenantId, Actor, It.Is<Dictionary<string, string>?>(d =>
@@ -359,7 +358,7 @@ public class AppHomingServiceTests
var outcome = await _sut.TryAutoFlipToPrimaryAsync(LegacyHomedConfig(), Actor);
Assert.Equal(AppHomingAutoFlipOutcome.ProbeFailed, outcome.Outcome);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
VerifyNeverSaved();
}
[Fact]
@@ -370,7 +369,7 @@ public class AppHomingServiceTests
var outcome = await _sut.TryAutoFlipToPrimaryAsync(LegacyHomedConfig(), Actor);
Assert.Equal(AppHomingAutoFlipOutcome.ProbeTransient, outcome.Outcome);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
VerifyNeverSaved();
}
[Fact]
@@ -394,7 +393,7 @@ public class AppHomingServiceTests
var outcome = await _sut.TryAutoFlipToPrimaryAsync(LegacyHomedConfig(), Actor);
Assert.Equal(AppHomingAutoFlipOutcome.NotEligible, outcome.Outcome);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
VerifyNeverSaved();
}
// ── FlipAsync ───────────────────────────────────────────────────────────
@@ -404,7 +403,7 @@ public class AppHomingServiceTests
{
var config = LegacyHomedConfig();
config.EntraAppRolesEnabled = true;
_tenantConfigMock.Setup(x => x.GetConfigurationFreshAsync(TenantId)).ReturnsAsync(config);
_store.Row = config;
await _sut.FlipAsync(TenantId, PrimaryId, Actor, "manual-ga");
@@ -417,11 +416,11 @@ public class AppHomingServiceTests
{
var config = LegacyHomedConfig();
config.HomedAppClientId = PrimaryId;
_tenantConfigMock.Setup(x => x.GetConfigurationFreshAsync(TenantId)).ReturnsAsync(config);
_store.Row = config;
await _sut.FlipAsync(TenantId, PrimaryId, Actor, "manual-ga");
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
VerifyNeverSaved();
_detectorMock.Verify(x => x.InvalidateTenant(It.IsAny<string>()), Times.Never);
Assert.Empty(_savedOpsEvents);
}
@@ -430,30 +429,68 @@ public class AppHomingServiceTests
public async Task Flip_throws_when_config_row_missing_and_saves_nothing()
{
// A flip must never materialize a default row (e.g. resurrect an offboarded tenant);
// the fresh read returning null is a hard stop.
_tenantConfigMock.Setup(x => x.GetConfigurationFreshAsync(TenantId))
.ReturnsAsync((TenantConfiguration?)null);
// a missing row is a hard stop.
_store.Row = null;
await Assert.ThrowsAsync<InvalidOperationException>(
() => _sut.FlipAsync(TenantId, PrimaryId, Actor, "manual-ga"));
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
VerifyNeverSaved();
_detectorMock.Verify(x => x.InvalidateTenant(It.IsAny<string>()), Times.Never);
Assert.Empty(_savedOpsEvents);
}
[Fact]
public async Task Flip_decides_on_the_stored_row_not_the_callers_view()
{
// The caller's (cached) view is legacy-homed; the stored row was flipped meanwhile.
var stored = LegacyHomedConfig();
stored.HomedAppClientId = PrimaryId;
_store.Row = stored;
await _sut.FlipAsync(TenantId, PrimaryId, Actor, "consent-auto-flip");
VerifyNeverSaved();
Assert.Empty(_savedOpsEvents);
}
[Fact]
public async Task Flip_skips_a_tenant_being_offboarded()
{
var tombstone = LegacyHomedConfig();
tombstone.Disabled = true;
tombstone.DisabledReason = TenantOffboardFunction.OffboardingDisabledReason;
_store.Row = tombstone;
await _sut.FlipAsync(TenantId, PrimaryId, Actor, "manual-ga");
VerifyNeverSaved();
_detectorMock.Verify(x => x.InvalidateTenant(It.IsAny<string>()), Times.Never);
Assert.Empty(_savedOpsEvents);
}
[Fact]
public async Task Flip_throws_when_every_conditional_write_lost()
{
_store.Force = TenantConfigUpdateStatus.Conflict;
await Assert.ThrowsAsync<InvalidOperationException>(
() => _sut.FlipAsync(TenantId, PrimaryId, Actor, "manual-ga"));
Assert.Empty(_savedOpsEvents);
}
[Fact]
public async Task Flip_back_to_legacy_persists_null_homing()
{
var config = LegacyHomedConfig();
config.HomedAppClientId = PrimaryId;
_tenantConfigMock.Setup(x => x.GetConfigurationFreshAsync(TenantId)).ReturnsAsync(config);
_store.Row = config;
await _sut.FlipAsync(TenantId, null, Actor, "manual-ga");
// The null-homing invariant: legacy is represented as null, never as the legacy GUID.
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.Is<TenantConfiguration>(c =>
c.HomedAppClientId == null), It.IsAny<string?>(), It.IsAny<string?>()), Times.Once);
Assert.Null(Assert.Single(_store.Writes).Row.HomedAppClientId);
_detectorMock.Verify(x => x.InvalidateTenant(TenantId), Times.Once);
}
}
@@ -306,7 +306,7 @@ public class AppInstallIdentityAndEspBlockingTests
};
var root = TestWire.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, repo.Object, "Company Portal", days: 30));
summaries, repo.Object, "Company Portal", AppInstallSources.Ime, days: 30));
Assert.Equal(1, root.GetProperty("collisionExcluded").GetInt32());
var stats = root.GetProperty("summary");
@@ -325,7 +325,7 @@ public class AppInstallIdentityAndEspBlockingTests
var summaries = new List<AppInstallSummary> { Summary("Company Portal", collision: true) };
var root = TestWire.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, repo.Object, "Company Portal", days: 30));
summaries, repo.Object, "Company Portal", AppInstallSources.Ime, days: 30));
Assert.Equal(1, root.GetProperty("collisionExcluded").GetInt32());
Assert.Equal(0, root.GetProperty("summary").GetProperty("totalInstalls").GetInt32());
@@ -195,6 +195,37 @@ public class AppVersionRegressionRadarTests
Assert.Empty(AppVersionRegressionRadar.Evaluate(rows2));
}
[Fact]
public void Evaluate_SameNameInTwoChannels_IsTwoApps()
{
// The RealmJoin package regressed; the Intune app of the same name did not. A shared
// group would dilute the RealmJoin medians with the Intune rows and hide the regression.
var rows = new List<AppInstallSummary>();
rows.AddRange(Batch("1.0", 10, 300, Now.AddDays(-20)).Select(s => { s.Source = AppInstallSources.RealmJoin; return s; }));
rows.AddRange(Batch("2.0", 10, 900, Now.AddDays(-5)).Select(s => { s.Source = AppInstallSources.RealmJoin; return s; }));
rows.AddRange(Batch("1.0", 30, 300, Now.AddDays(-20)));
rows.AddRange(Batch("2.0", 30, 300, Now.AddDays(-5)));
var finding = Assert.Single(AppVersionRegressionRadar.Evaluate(rows));
Assert.Equal(AppInstallSources.RealmJoin, finding.Source);
Assert.Equal(App, finding.AppName);
Assert.Equal(900, finding.CurrentMedianSeconds);
}
[Fact]
public void ShouldReArm_ReadsTheAlertsOwnChannel()
{
// The IME app still regresses; the RealmJoin episode's version has no measured installs.
var rows = new List<AppInstallSummary>();
rows.AddRange(Batch("1.0", 10, 300, Now.AddDays(-20)));
rows.AddRange(Batch("2.0", 10, 900, Now.AddDays(-5)));
var rjAlert = Alert();
rjAlert.Source = AppInstallSources.RealmJoin;
Assert.True(AppVersionRegressionRadar.ShouldReArm(rows, rjAlert));
Assert.False(AppVersionRegressionRadar.ShouldReArm(rows, Alert()));
}
[Fact]
public void Evaluate_DeterministicOrdering_LiftDesc_ThenAppNameOrdinal()
{
@@ -279,13 +310,25 @@ public class AppVersionRegressionRadarTests
{
Assert.Equal("appversionregression|contoso app_2|1.0_beta_",
TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(
" Contoso App/2 ", " 1.0#Beta? "));
AppInstallSources.Ime, " Contoso App/2 ", " 1.0#Beta? "));
}
[Fact]
public void AppVersionRegressionRowKey_ImeKeepsPreChannelShape_OtherChannelsPrefixTheApp()
{
// Episodes written before channels existed must keep resolving (re-arm deletes them by key).
Assert.Equal("appversionregression|contoso vpn|2.0",
TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(null, App, "2.0"));
Assert.Equal("appversionregression|contoso vpn|2.0",
TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(string.Empty, App, "2.0"));
Assert.Equal("appversionregression|realmjoin:contoso vpn|2.0",
TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(AppInstallSources.RealmJoin, App, "2.0"));
}
[Fact]
public void AppVersionRegressionRowKey_NoPrefixCollisionWithOtherKeyspaces()
{
var key = TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(App, "1.0");
var key = TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionRowKey(AppInstallSources.RealmJoin, App, "1.0");
Assert.StartsWith("appversionregression|", key, StringComparison.Ordinal);
// The prefix range [prefix, "appversionregression}") must not swallow the sibling
// keyspaces ("ruleregression|", "tpmpss|", "{mfr}|{model}").
@@ -293,6 +336,39 @@ public class AppVersionRegressionRadarTests
Assert.True(string.CompareOrdinal("tpmpss|", "appversionregression}") > 0);
}
[Fact]
public void AppVersionRegressionEntity_RealmJoinChannel_RoundTripsWithPrefixedKey()
{
var alert = new AppVersionRegressionAlert
{
TenantId = TenantA,
AppName = App,
Source = AppInstallSources.RealmJoin,
CurrentVersion = "2.4.0",
PreviousVersion = "2.3.9",
FirstNotifiedAt = Now,
LastEvaluatedAt = Now,
};
var entity = TableHardwareRejectionNotificationTracker.BuildAppVersionRegressionEntity(TenantA, alert);
Assert.Equal("appversionregression|realmjoin:contoso vpn|2.4.0", entity.RowKey);
Assert.Equal(AppInstallSources.RealmJoin,
TableHardwareRejectionNotificationTracker.MapToAppVersionRegressionAlert(entity).Source);
}
[Fact]
public void AppVersionRegressionEntity_RowWithoutSourceColumn_MapsToIme()
{
var entity = new Azure.Data.Tables.TableEntity(TenantA.ToLowerInvariant(), "appversionregression|contoso vpn|2.4.0")
{
["TenantId"] = TenantA,
["AppName"] = App,
["CurrentVersion"] = "2.4.0",
};
Assert.Equal(AppInstallSources.Ime,
TableHardwareRejectionNotificationTracker.MapToAppVersionRegressionAlert(entity).Source);
}
[Fact]
public void AppVersionRegressionEntity_RoundTripsAllFields()
{
@@ -318,6 +394,7 @@ public class AppVersionRegressionRadarTests
var mapped = TableHardwareRejectionNotificationTracker.MapToAppVersionRegressionAlert(entity);
Assert.Equal(TenantA, mapped.TenantId);
Assert.Equal(App, mapped.AppName);
Assert.Equal(AppInstallSources.Ime, mapped.Source);
Assert.Equal("2.4.0", mapped.CurrentVersion);
Assert.Equal("2.3.9", mapped.PreviousVersion);
Assert.Equal(1740, mapped.CurrentMedianSeconds);
@@ -371,7 +448,7 @@ public class AppVersionRegressionRadarTests
};
var root = TestWire.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
rows, sessionRepo.Object, App, days: 30));
rows, sessionRepo.Object, App, AppInstallSources.Ime, days: 30));
var version = root.GetProperty("versionBreakdown")[0];
Assert.Equal("1.0", version.GetProperty("appVersion").GetString());
@@ -395,12 +472,12 @@ public class AppVersionRegressionRadarTests
// Empty-data early return still surfaces the block (episodes can outlive window data).
var emptyRoot = TestWire.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
new List<AppInstallSummary>(), sessionRepo.Object, App, days: 30,
new List<AppInstallSummary>(), sessionRepo.Object, App, AppInstallSources.Ime, days: 30,
new List<AppVersionRegressionAlert> { alert }));
Assert.Equal(1, emptyRoot.GetProperty("versionRegressions").GetArrayLength());
var root = TestWire.SerializeToElement(await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
new List<AppInstallSummary> { Install("1.0", 100, Recent) }, sessionRepo.Object, App, days: 30,
new List<AppInstallSummary> { Install("1.0", 100, Recent) }, sessionRepo.Object, App, AppInstallSources.Ime, days: 30,
new List<AppVersionRegressionAlert> { alert }));
var regression = root.GetProperty("versionRegressions")[0];
@@ -131,7 +131,7 @@ public class AppsAnalyticsFailureRateTests
sessionRepo = mock.Object;
}
var result = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
summaries, sessionRepo, "Contoso App", days: 30);
summaries, sessionRepo, "Contoso App", AppInstallSources.Ime, days: 30);
return TestWire.SerializeToElement(result);
}
@@ -31,9 +31,9 @@ public class AuditLogDeletionExclusionTests
public void Global_fanout_filter_includes_deletion_exclusion_only_when_requested()
{
var without = TableStorageService.BuildAuditLogFilterWithRowKeyBound(
Tenant.ToString(), null, null, lastRowKey: null, excludeDeletions: false);
Tenant.ToString(), null, null, rowKeyBound: null, excludeDeletions: false);
var with = TableStorageService.BuildAuditLogFilterWithRowKeyBound(
Tenant.ToString(), null, null, lastRowKey: null, excludeDeletions: true);
Tenant.ToString(), null, null, rowKeyBound: null, excludeDeletions: true);
Assert.DoesNotContain("deletion_started", without);
Assert.Contains("Action ne 'deletion_started'", with);
@@ -47,7 +47,7 @@ public class AuditLogDeletionExclusionTests
// noise leak back into one of the two views.
var tenant = TableStorageService.BuildAuditLogFilter(Tenant.ToString(), null, null, excludeDeletions: true);
var global = TableStorageService.BuildAuditLogFilterWithRowKeyBound(
Tenant.ToString(), null, null, lastRowKey: null, excludeDeletions: true);
Tenant.ToString(), null, null, rowKeyBound: null, excludeDeletions: true);
var clause = TableStorageService.DeletionExclusionClause();
Assert.Contains(clause, tenant);
@@ -51,7 +51,7 @@ public class AuditLogFieldFilterTests
var filters = new AuditLogQueryFilters { Action = "device_blocked", EntityType = "Device" };
var f = TableStorageService.BuildAuditLogFilterWithRowKeyBound(
Tenant, null, null, lastRowKey: null, excludeDeletions: false, filters);
Tenant, null, null, rowKeyBound: null, excludeDeletions: false, filters);
Assert.Contains("Action eq 'device_blocked'", f);
Assert.Contains("EntityType eq 'Device'", f);
@@ -1,5 +1,9 @@
using AutopilotMonitor.Functions.Functions.Config;
using AutopilotMonitor.Functions.Functions.Infrastructure;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Tests.Offboarding;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.Models.Offboarding;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
@@ -32,6 +36,8 @@ public class AuthFunctionSideEffectTests
private readonly Mock<IMetricsRepository> _metricsRepoMock;
private readonly Mock<AutopilotMonitor.Functions.Services.Activation.ITenantAutoApproveEnqueuer> _autoApproveEnqueuerMock;
private readonly FakeSignalRNotificationService _signalR;
private readonly FakeOffboardingAuditRepository _offboardingRepo = new();
private readonly StoredTenantConfig _store;
private readonly AuthFunction _sut;
public AuthFunctionSideEffectTests()
@@ -108,12 +114,12 @@ public class AuthFunctionSideEffectTests
new Mock<AdminIdentityResolver>(
_metricsRepoMock.Object, _tenantConfigMock.Object, Mock.Of<ILogger<AdminIdentityResolver>>()) { CallBase = false }.Object,
_signalR,
adminConfigService.Object);
adminConfigService.Object,
_offboardingRepo);
// Default: all fire-and-forget calls succeed
_tenantConfigMock
.Setup(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.Returns(Task.CompletedTask);
// Writes go through UpdateAsync / CreateOrUpdateAsync over one stored row (none by default:
// a brand-new tenant). Default: all fire-and-forget calls succeed.
_store = _tenantConfigMock.StubWrites(TenantId, row: null);
_telegramMock
.Setup(x => x.SendNewTenantSignupAsync(It.IsAny<string>(), It.IsAny<string>()))
.Returns(Task.CompletedTask);
@@ -202,7 +208,7 @@ public class AuthFunctionSideEffectTests
Assert.True(string.IsNullOrEmpty(config.DomainName));
Assert.Null(config.OnboardedBy);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Empty(_store.Writes);
_telegramMock.Verify(x => x.SendNewTenantSignupAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
}
@@ -222,7 +228,11 @@ public class AuthFunctionSideEffectTests
Assert.Equal(Upn, config.OnboardedBy);
// The DPA version rides along in the same onboarding write (D-252).
Assert.Equal(CurrentDpa, config.DpaVersion);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(config, It.IsAny<string?>(), It.IsAny<string?>()), Times.Once);
var (written, source, _) = Assert.Single(_store.Writes);
Assert.Equal("contoso.com", written.DomainName);
Assert.Equal(Upn, written.OnboardedBy);
Assert.Equal(CurrentDpa, written.DpaVersion);
Assert.Equal("auth", source);
_telegramMock.Verify(x => x.SendNewTenantSignupAsync(TenantId, Upn), Times.Once);
_globalNotificationMock.Verify(x => x.CreateNotificationAsync(
"preview_signup", "New Tenant Signup",
@@ -237,6 +247,122 @@ public class AuthFunctionSideEffectTests
It.IsAny<CancellationToken>()), Times.Once);
}
// -------------------------------------------------------------------------
// Returning tenant (D-288): the offboarding deleted the configuration, the pointer
// carries the consumed self-service trial into the first-login write.
// -------------------------------------------------------------------------
private void SeedOffboardingPointer(bool trialConsumed) =>
_offboardingRepo.Pointers[TenantId] = (new OffboardingByTenantPointer
{
PartitionKey = Constants.OffboardingPartitionKeys.ByTenant,
RowKey = TenantId,
TenantId = TenantId,
LatestStatus = "Completed",
OffboardCount = 1,
TrialConsumed = trialConsumed,
}, "\"etag\"");
private static TenantConfiguration ReturningTenantDefault()
{
// What auth/me hands over once the offboarding deleted the row: an unpersisted default.
var config = TenantConfiguration.CreateDefault(TenantId);
Assert.False(config.TrialConsumed);
return config;
}
[Fact]
public async Task HandleNewTenantDomain_ReturningTenantThatUsedTheTrial_CarriesItIntoTheOnboardingWrite()
{
SeedOffboardingPointer(trialConsumed: true);
var config = ReturningTenantDefault();
await _sut.HandleNewTenantDomainAsync(config, TenantId, Upn);
Assert.True(config.TrialConsumed);
Assert.Equal(Upn, config.OnboardedBy);
var written = Assert.Single(_store.Writes).Row;
Assert.True(written.TrialConsumed);
Assert.Equal(Upn, written.OnboardedBy);
// Re-onboarding itself stays unrestricted: same signup path, auto-approve included.
_autoApproveEnqueuerMock.Verify(x => x.EnqueueAsync(
It.IsAny<AutopilotMonitor.Functions.Services.Activation.TenantAutoApproveEnvelope>(),
It.IsAny<TimeSpan?>(), It.IsAny<CancellationToken>()), Times.Once);
// The verify criterion of the finding: the trial gate refuses the returning tenant.
var deny = PlanManagementFunction.EvaluateTrialStart(config, DateTime.UtcNow);
Assert.Equal(Constants.ApiErrorCodes.TrialAlreadyConsumed, deny?.Code);
}
[Theory]
[InlineData(false)] // offboarded without ever using the trial
[InlineData(null)] // never offboarded
public async Task HandleNewTenantDomain_NoConsumedTrialOnRecord_LeavesTheTrialAvailable(bool? pointerTrialConsumed)
{
if (pointerTrialConsumed is { } consumed)
SeedOffboardingPointer(consumed);
var config = ReturningTenantDefault();
await _sut.HandleNewTenantDomainAsync(config, TenantId, Upn);
Assert.False(config.TrialConsumed);
Assert.Equal("contoso.com", config.DomainName);
Assert.False(Assert.Single(_store.Writes).Row.TrialConsumed);
}
[Fact]
public async Task HandleNewTenantDomain_OffboardingLookupFails_WritesNothingAndRetriesOnTheNextLogin()
{
// An onboarded row without the carry-over would hand the trial back; no write at all instead.
_offboardingRepo.ThrowOnPointerRead = new Azure.RequestFailedException(503, "Server busy");
var config = ReturningTenantDefault();
await _sut.HandleNewTenantDomainAsync(config, TenantId, Upn);
Assert.True(string.IsNullOrEmpty(config.DomainName));
Assert.Null(config.OnboardedBy);
Assert.Empty(_store.Writes);
_telegramMock.Verify(x => x.SendNewTenantSignupAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
_autoApproveEnqueuerMock.Verify(x => x.EnqueueAsync(
It.IsAny<AutopilotMonitor.Functions.Services.Activation.TenantAutoApproveEnvelope>(),
It.IsAny<TimeSpan?>(), It.IsAny<CancellationToken>()), Times.Never);
}
[Fact]
public async Task HandleNewTenantDomain_ConcurrentFirstLogin_NeverOverwritesOnboardedBy()
{
// This login's view still has no domain, but another login seeded the stored row meanwhile:
// OnboardedBy is write-once (D-061), and the signup side effects belong to the winner.
var stored = DefaultConfig();
stored.DomainName = "contoso.com";
stored.OnboardedBy = "first@contoso.com";
_store.Row = stored;
var view = DefaultConfig();
await _sut.HandleNewTenantDomainAsync(view, TenantId, "second@contoso.com");
Assert.Empty(_store.Writes);
Assert.Equal("first@contoso.com", _store.Row!.OnboardedBy);
_telegramMock.Verify(x => x.SendNewTenantSignupAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
_autoApproveEnqueuerMock.Verify(x => x.EnqueueAsync(
It.IsAny<AutopilotMonitor.Functions.Services.Activation.TenantAutoApproveEnvelope>(),
It.IsAny<TimeSpan?>(), It.IsAny<CancellationToken>()), Times.Never);
}
[Fact]
public async Task HandleNewTenantDomain_OffboardingTombstone_IsNeverSeeded()
{
// The offboard click writes a Disabled tombstone even onto a row without a domain.
var tombstone = DefaultConfig();
tombstone.Disabled = true;
tombstone.DisabledReason = AutopilotMonitor.Functions.Functions.Admin.TenantOffboardFunction.OffboardingDisabledReason;
_store.Row = tombstone;
await _sut.HandleNewTenantDomainAsync(DefaultConfig(), TenantId, Upn);
Assert.Empty(_store.Writes);
_telegramMock.Verify(x => x.SendNewTenantSignupAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
}
// -------------------------------------------------------------------------
// HandleAuthClientIdTrackingAsync — the write must mutate a cache-BYPASSING
// read, never the (possibly stale) cached view. Regression for the prod
@@ -261,16 +387,16 @@ public class AuthFunctionSideEffectTests
fresh.LastAuthClientId = LegacyAppId;
fresh.HomedAppClientId = PrimaryAppId; // the concurrent flip that must survive
_tenantConfigMock.Setup(x => x.GetConfigurationFreshAsync(TenantId)).ReturnsAsync(fresh);
_store.Row = fresh;
await _sut.HandleAuthClientIdTrackingAsync(cached, TenantId, $"api://{PrimaryAppId}");
// The FRESH entity was mutated and saved — the flip survives, the stale view is discarded.
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(fresh, It.IsAny<string?>(), It.IsAny<string?>()), Times.Once);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(cached, It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Equal(PrimaryAppId, fresh.HomedAppClientId);
Assert.Equal(PrimaryAppId, fresh.LastAuthClientId);
Assert.NotNull(fresh.LastAuthClientIdSince);
// The FRESH row was mutated and written — the flip survives, the stale view is discarded.
var written = Assert.Single(_store.Writes).Row;
Assert.Equal(PrimaryAppId, written.HomedAppClientId);
Assert.Equal(PrimaryAppId, written.LastAuthClientId);
Assert.NotNull(written.LastAuthClientIdSince);
Assert.Null(cached.HomedAppClientId);
}
[Fact]
@@ -286,11 +412,11 @@ public class AuthFunctionSideEffectTests
fresh.DomainName = "contoso.com";
fresh.LastAuthClientId = PrimaryAppId;
_tenantConfigMock.Setup(x => x.GetConfigurationFreshAsync(TenantId)).ReturnsAsync(fresh);
_store.Row = fresh;
await _sut.HandleAuthClientIdTrackingAsync(cached, TenantId, PrimaryAppId);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Empty(_store.Writes);
}
[Fact]
@@ -303,8 +429,9 @@ public class AuthFunctionSideEffectTests
await _sut.HandleAuthClientIdTrackingAsync(cached, TenantId, $"api://{PrimaryAppId}");
_tenantConfigMock.Verify(x => x.GetConfigurationFreshAsync(It.IsAny<string>()), Times.Never);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
_tenantConfigMock.Verify(x => x.UpdateAsync(It.IsAny<string>(), It.IsAny<Func<TenantConfiguration, bool>>(),
It.IsAny<string>(), It.IsAny<string>()), Times.Never);
Assert.Empty(_store.Writes);
}
private const string CurrentDpa = AutopilotMonitor.Shared.Constants.CurrentDpaVersion;
@@ -321,7 +448,7 @@ public class AuthFunctionSideEffectTests
await _sut.HandleNewTenantDomainAsync(config, TenantId, Upn);
Assert.Null(config.DpaVersion);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Empty(_store.Writes);
}
[Fact]
@@ -332,10 +459,12 @@ public class AuthFunctionSideEffectTests
var config = DefaultConfig();
config.DomainName = null!;
config.OnboardedBy = "original.requester@contoso.com";
_store.Row = ConfigRepoMockExtensions.Clone(config);
await _sut.HandleNewTenantDomainAsync(config, TenantId, Upn);
Assert.Equal("original.requester@contoso.com", config.OnboardedBy);
Assert.Equal("original.requester@contoso.com", Assert.Single(_store.Writes).Row.OnboardedBy);
}
[Fact]
@@ -346,7 +475,7 @@ public class AuthFunctionSideEffectTests
await _sut.HandleNewTenantDomainAsync(config, TenantId, Upn);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Empty(_store.Writes);
_telegramMock.Verify(x => x.SendNewTenantSignupAsync(It.IsAny<string>(), It.IsAny<string>()), Times.Never);
}
@@ -358,7 +487,7 @@ public class AuthFunctionSideEffectTests
await _sut.HandleNewTenantDomainAsync(config, TenantId, "");
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Empty(_store.Writes);
}
[Fact]
@@ -369,7 +498,7 @@ public class AuthFunctionSideEffectTests
await _sut.HandleNewTenantDomainAsync(config, TenantId, "nodomain");
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Empty(_store.Writes);
}
[Fact]
@@ -385,8 +514,8 @@ public class AuthFunctionSideEffectTests
// Should not throw — Telegram is fire-and-forget
await _sut.HandleNewTenantDomainAsync(config, TenantId, Upn);
// SaveConfig should still have been called before the fire-and-forget
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(config, It.IsAny<string?>(), It.IsAny<string?>()), Times.Once);
// The write still landed before the fire-and-forget
Assert.Single(_store.Writes);
}
// -------------------------------------------------------------------------
@@ -400,14 +529,68 @@ public class AuthFunctionSideEffectTests
config.Disabled = true;
config.DisabledReason = "Maintenance";
config.DisabledUntil = DateTime.UtcNow.AddHours(-1); // expired
_store.Row = ConfigRepoMockExtensions.Clone(config);
await _sut.HandleAutoReEnableAsync(config, TenantId);
Assert.False(config.Disabled);
Assert.Null(config.DisabledReason);
Assert.Null(config.DisabledUntil);
Assert.Equal("System (auto-re-enable)", config.UpdatedBy);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(config, It.IsAny<string?>(), It.IsAny<string?>()), Times.Once);
var written = Assert.Single(_store.Writes).Row;
Assert.False(written.Disabled);
Assert.Null(written.DisabledUntil);
Assert.Equal("System (auto-re-enable)", written.UpdatedBy);
}
[Fact]
public async Task HandleAutoReEnable_SuspensionRenewedMeanwhile_StaysSuspended()
{
// This instance cached the expired suspension; a Global Admin renewed it since.
var view = DefaultConfig();
view.Disabled = true;
view.DisabledReason = "Maintenance";
view.DisabledUntil = DateTime.UtcNow.AddHours(-1);
var stored = ConfigRepoMockExtensions.Clone(view);
stored.DisabledReason = "Abuse review";
stored.DisabledUntil = DateTime.UtcNow.AddDays(2);
_store.Row = stored;
await _sut.HandleAutoReEnableAsync(view, TenantId);
Assert.Empty(_store.Writes);
Assert.True(view.IsCurrentlyDisabled());
Assert.Equal("Abuse review", view.DisabledReason);
}
[Fact]
public async Task HandleAutoReEnable_RowIsTheOffboardingTombstone_StaysSuspended()
{
var view = DefaultConfig();
view.Disabled = true;
view.DisabledUntil = DateTime.UtcNow.AddHours(-1);
var tombstone = ConfigRepoMockExtensions.Clone(view);
tombstone.DisabledReason = AutopilotMonitor.Functions.Functions.Admin.TenantOffboardFunction.OffboardingDisabledReason;
tombstone.DisabledUntil = null;
_store.Row = tombstone;
await _sut.HandleAutoReEnableAsync(view, TenantId);
Assert.Empty(_store.Writes);
Assert.True(view.IsCurrentlyDisabled());
}
[Fact]
public async Task HandleAutoReEnable_StorageFails_ThisLoginStillPasses()
{
var view = DefaultConfig();
view.Disabled = true;
view.DisabledUntil = DateTime.UtcNow.AddHours(-1);
_store.ThrowOnRead = new Azure.RequestFailedException(503, "Server busy");
await _sut.HandleAutoReEnableAsync(view, TenantId);
Assert.False(view.Disabled);
Assert.Empty(_store.Writes);
}
[Fact]
@@ -420,7 +603,7 @@ public class AuthFunctionSideEffectTests
await _sut.HandleAutoReEnableAsync(config, TenantId);
Assert.True(config.Disabled);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Empty(_store.Writes);
}
[Fact]
@@ -431,7 +614,7 @@ public class AuthFunctionSideEffectTests
await _sut.HandleAutoReEnableAsync(config, TenantId);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Empty(_store.Writes);
}
[Fact]
@@ -444,7 +627,7 @@ public class AuthFunctionSideEffectTests
await _sut.HandleAutoReEnableAsync(config, TenantId);
Assert.True(config.Disabled);
_tenantConfigMock.Verify(x => x.SaveConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Empty(_store.Writes);
}
// -------------------------------------------------------------------------
@@ -63,7 +63,7 @@ public class AuthenticationMiddlewareTests
[InlineData("GET", "/api/progress/sessions/lookup")] // AuthenticatedUserWithRole
[InlineData("GET", "/api/sessions")] // MemberRead
[InlineData("GET", "/api/audit/logs")] // MemberRead
[InlineData("PUT", "/api/config/11111111-1111-1111-1111-111111111111")] // TenantAdminOrGA
[InlineData("PATCH", "/api/config/11111111-1111-1111-1111-111111111111/fields")] // TenantAdminOrGA
[InlineData("GET", "/api/global/sessions")] // GlobalReadOrDelegatedSubset
[InlineData("POST", "/api/global/raw/logs")] // GlobalAdminOnly (KQL proxy)
[InlineData("GET", "/api/health/detailed")] // AuthenticatedUser (sub-route is NOT blanket-exempt)
@@ -152,7 +152,7 @@ public class BusinessTimestampFilterClauseTests
var plain = TableStorageService.BuildAuditLogFilter(Tenant, from, to);
var fanOut = TableStorageService.BuildAuditLogFilterWithRowKeyBound(
Tenant, from, to, lastRowKey: "!0123_x", excludeDeletions: false);
Tenant, from, to, rowKeyBound: new PerPartitionFanOutMerge.RowKeyBound("!0123_x", Inclusive: false), excludeDeletions: false);
foreach (var f in new[] { plain!, fanOut })
{
@@ -1,3 +1,5 @@
using System.Net;
using System.Net.Sockets;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using AutopilotMonitor.Functions.Security;
@@ -13,6 +15,7 @@ namespace AutopilotMonitor.Functions.Tests;
/// must validate successfully. This guards against rollouts where the embedded root/intermediate
/// bundle drifts from the chain Microsoft is actually issuing.
/// - Negative path: self-signed leaf certs (with or without Intune-shaped DNs) must NOT validate.
/// - No network I/O: a leaf from an unknown CA must be rejected without fetching its AIA URL.
/// </summary>
public class CertificateValidatorTests
{
@@ -131,6 +134,31 @@ public class CertificateValidatorTests
Assert.Contains("expired", result.ErrorMessage, StringComparison.OrdinalIgnoreCase);
}
[Fact]
public void ValidateCertificate_WithUnknownIssuerAndAiaUrl_NeverFetchesTheIssuer()
{
// The caller mints the leaf, so the AIA caIssuers URL points wherever they like. A
// self-signed leaf never reaches the issuer-download path; it takes a leaf signed by a
// CA that is neither embedded nor sent.
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
try
{
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
var b64 = CreateLeafFromUnknownCaBase64($"http://127.0.0.1:{port}/{Guid.NewGuid():N}.cer");
var result = CertificateValidator.ValidateCertificate(b64);
Assert.False(result.IsValid);
Assert.Contains("chain validation failed", result.ErrorMessage, StringComparison.OrdinalIgnoreCase);
Assert.False(listener.Pending(), "Chain building connected to the certificate's AIA URL");
}
finally
{
listener.Stop();
}
}
private static string ResolveSamplePath()
{
var assemblyDir = Path.GetDirectoryName(typeof(CertificateValidatorTests).Assembly.Location)!;
@@ -165,4 +193,24 @@ public class CertificateValidatorTests
return Convert.ToBase64String(cert.Export(X509ContentType.Cert));
}
private static string CreateLeafFromUnknownCaBase64(string aiaCaIssuersUrl)
{
using var caKey = RSA.Create(2048);
var caReq = new CertificateRequest("CN=Unknown Test CA", caKey, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
caReq.CertificateExtensions.Add(new X509BasicConstraintsExtension(true, false, 0, true));
using var ca = caReq.CreateSelfSigned(DateTimeOffset.UtcNow.AddDays(-1), DateTimeOffset.UtcNow.AddYears(1));
using var leafKey = RSA.Create(2048);
var leafReq = new CertificateRequest("CN=test-device.contoso.example", leafKey, HashAlgorithmName.SHA256, RSASignaturePadding.Pkcs1);
leafReq.CertificateExtensions.Add(
new X509EnhancedKeyUsageExtension(
new OidCollection { new Oid("1.3.6.1.5.5.7.3.2") },
critical: false));
leafReq.CertificateExtensions.Add(
new X509AuthorityInformationAccessExtension(ocspUris: null, caIssuersUris: new[] { aiaCaIssuersUrl }));
using var leaf = leafReq.Create(ca, DateTimeOffset.UtcNow.AddHours(-1), DateTimeOffset.UtcNow.AddDays(30), RandomNumberGenerator.GetBytes(16));
return Convert.ToBase64String(leaf.Export(X509ContentType.Cert));
}
}
@@ -40,14 +40,23 @@ public class CollectLogsQuickConfigOpsEventTests
new TenantConfiguration { TenantId = "t1", DiagnosticsUploadMode = "OnFailure", DiagnosticsUploadDestination = "Hosted" })!;
[Theory]
[InlineData("collect-logs", UpdateTenantConfigurationFunction.CollectLogsSource)]
[InlineData("COLLECT-LOGS", UpdateTenantConfigurationFunction.CollectLogsSource)]
[InlineData(null, "portal-put")]
[InlineData("", "portal-put")]
[InlineData("something-else", "portal-put")]
public void ResolveWriteSource_only_honours_the_allow_listed_intent(string? intent, string expected)
[InlineData("?intent=collect-logs", "patch", PatchTenantConfigurationFieldsFunction.CollectLogsSource)]
[InlineData("?intent=COLLECT-LOGS", "patch", PatchTenantConfigurationFieldsFunction.CollectLogsSource)]
[InlineData("", "patch", "api-patch")]
[InlineData("?intent=", "patch", "api-patch")]
[InlineData("?intent=something-else", "patch", "api-patch")]
[InlineData("?intent=collect-logs", "revert", "api-revert")] // a revert is never the quick-config dialog
public void ResolveSource_only_honours_the_allow_listed_intent(string query, string operation, string expected)
{
Assert.Equal(expected, UpdateTenantConfigurationFunction.ResolveWriteSource(intent));
var (req, _) = EndpointHarness.Request("t1", queryString: query);
Assert.Equal(expected, PatchTenantConfigurationFieldsFunction.ResolveSource(req, operation));
}
[Fact]
public void ResolveSource_mcp_header_without_intent_is_mcp()
{
var (req, _) = EndpointHarness.Request("t1", headers: new Dictionary<string, string> { ["X-Client-Source"] = "mcp" });
Assert.Equal("mcp-patch", PatchTenantConfigurationFieldsFunction.ResolveSource(req, "patch"));
}
[Fact]
@@ -56,7 +65,7 @@ public class CollectLogsQuickConfigOpsEventTests
var (service, saved) = Rig();
await service.RecordDiagnosticsUploadConfigChangedAsync(
"t1", "contoso.com", EnableFlip(), "admin@contoso.com", UpdateTenantConfigurationFunction.CollectLogsSource);
"t1", "contoso.com", EnableFlip(), "admin@contoso.com", PatchTenantConfigurationFieldsFunction.CollectLogsSource);
var evt = Assert.Single(saved);
Assert.Equal("CollectLogsQuickConfigEnabled", evt.EventType);
@@ -85,7 +94,7 @@ public class CollectLogsQuickConfigOpsEventTests
new TenantConfiguration { TenantId = "t1", DiagnosticsUploadMode = "Always", DiagnosticsUploadDestination = "Hosted" },
new TenantConfiguration { TenantId = "t1", DiagnosticsUploadMode = "Off", DiagnosticsUploadDestination = "Hosted" })!;
await service.RecordDiagnosticsUploadConfigChangedAsync("t1", null, disable, "ga@x", UpdateTenantConfigurationFunction.CollectLogsSource);
await service.RecordDiagnosticsUploadConfigChangedAsync("t1", null, disable, "ga@x", PatchTenantConfigurationFieldsFunction.CollectLogsSource);
Assert.Equal("DiagnosticsUploadDisabled", Assert.Single(saved).EventType);
}
@@ -56,6 +56,19 @@ public class ConfigBackupHookTests
Upserts.Add(e);
return Task.FromResult(Mock.Of<Response>());
});
Table.Setup(c => c.AddEntityAsync(It.IsAny<TableEntity>(), It.IsAny<CancellationToken>()))
.Returns<TableEntity, CancellationToken>((e, _) =>
{
Upserts.Add(e);
return Task.FromResult(Mock.Of<Response>());
});
Table.Setup(c => c.UpdateEntityAsync(
It.IsAny<TableEntity>(), It.IsAny<ETag>(), It.IsAny<TableUpdateMode>(), It.IsAny<CancellationToken>()))
.Returns<TableEntity, ETag, TableUpdateMode, CancellationToken>((e, _, _, _) =>
{
Upserts.Add(e);
return Task.FromResult(Mock.Of<Response>());
});
var serviceClient = new Mock<TableServiceClient>();
serviceClient.Setup(c => c.GetTableClient(It.IsAny<string>())).Returns(Table.Object);
@@ -102,13 +115,13 @@ public class ConfigBackupHookTests
.Callback<ConfigBackupEntry, CancellationToken>((e, _) => snapshot = e)
.Returns(Task.CompletedTask);
var saved = await harness.Sut.SaveTenantConfigurationAsync(incoming, "portal-put", "test change");
var saved = await harness.Sut.TryReplaceTenantConfigurationAsync(incoming, "\"etag\"", "api-patch", "test change");
Assert.True(saved);
Assert.Single(harness.Upserts); // the config row itself
Assert.NotNull(snapshot);
Assert.Equal(TenantId, snapshot!.PartitionKey);
Assert.Equal("portal-put", snapshot.Source);
Assert.Equal("api-patch", snapshot.Source);
Assert.Equal("test change", snapshot.Reason);
Assert.Equal("ga@operator.example", snapshot.ChangedBy); // identity about to write
// The snapshot holds the STORED row (retention 30), not the incoming one.
@@ -123,11 +136,11 @@ public class ConfigBackupHookTests
}
[Fact]
public async Task RowCreation_NoExistingRow_NoSnapshot()
public async Task RowCreation_ConditionalInsert_NoSnapshot()
{
var harness = new Harness(stored: null);
var saved = await harness.Sut.SaveTenantConfigurationAsync(StoredConfig(), "portal-put", null);
var saved = await harness.Sut.TryCreateTenantConfigurationAsync(StoredConfig());
Assert.True(saved);
Assert.Single(harness.Upserts);
@@ -137,6 +150,16 @@ public class ConfigBackupHookTests
b => b.PruneAsync(It.IsAny<string>(), It.IsAny<int>(), It.IsAny<CancellationToken>()), Times.Never);
}
[Fact]
public async Task RowCreation_RowAlreadyExists_ReturnsFalse()
{
var harness = new Harness(stored: null);
harness.Table.Setup(c => c.AddEntityAsync(It.IsAny<TableEntity>(), It.IsAny<CancellationToken>()))
.ThrowsAsync(new RequestFailedException(409, "Conflict", "EntityAlreadyExists", null));
Assert.False(await harness.Sut.TryCreateTenantConfigurationAsync(StoredConfig()));
}
[Fact]
public async Task NoiseOnlyChange_AuthClientIdFlip_NoSnapshot()
{
@@ -150,7 +173,7 @@ public class ConfigBackupHookTests
incoming.LastUpdated = DateTime.UtcNow;
incoming.UpdatedBy = "System (auth)";
var saved = await harness.Sut.SaveTenantConfigurationAsync(incoming, "auth", null);
var saved = await harness.Sut.TryReplaceTenantConfigurationAsync(incoming, "\"etag\"", "auth", null);
Assert.True(saved); // the save itself still goes through
Assert.Single(harness.Upserts);
@@ -164,7 +187,7 @@ public class ConfigBackupHookTests
var stored = StoredConfig();
var harness = new Harness(TableConfigRepository.ConvertToTenantTableEntity(stored));
var saved = await harness.Sut.SaveTenantConfigurationAsync(RoundtripClone(stored), "portal-put", null);
var saved = await harness.Sut.TryReplaceTenantConfigurationAsync(RoundtripClone(stored), "\"etag\"", "api-patch", null);
Assert.True(saved);
harness.Backup.Verify(
@@ -183,14 +206,14 @@ public class ConfigBackupHookTests
var incoming = RoundtripClone(stored);
incoming.DataRetentionDays = 90;
var saved = await harness.Sut.SaveTenantConfigurationAsync(incoming, "portal-put", null);
var saved = await harness.Sut.TryReplaceTenantConfigurationAsync(incoming, "\"etag\"", "api-patch", null);
Assert.True(saved);
Assert.Single(harness.Upserts); // the config write went through regardless
}
[Fact]
public async Task LegacyOneArgOverload_StillSnapshots_WithUnknownSource()
public async Task NoSource_StillSnapshots_WithUnknownSource()
{
var stored = StoredConfig();
var harness = new Harness(TableConfigRepository.ConvertToTenantTableEntity(stored));
@@ -203,7 +226,7 @@ public class ConfigBackupHookTests
var incoming = RoundtripClone(stored);
incoming.DataRetentionDays = 90;
await harness.Sut.SaveTenantConfigurationAsync(incoming);
await harness.Sut.TryReplaceTenantConfigurationAsync(incoming, "\"etag\"", null, null);
Assert.NotNull(snapshot);
Assert.Equal("unknown", snapshot!.Source);
@@ -225,15 +248,11 @@ public class ConfigBackupHookTests
.Callback<ConfigBackupEntry, CancellationToken>((e, _) => snapshot = e)
.Returns(Task.CompletedTask);
var incoming = new AdminConfiguration
{
UpdatedBy = "ga@operator.example",
GlobalRateLimitRequestsPerMinute = 200,
};
var result = await harness.Sut.UpdateAdminConfigurationAsync(
c => { c.GlobalRateLimitRequestsPerMinute = 200; return null; }, "ga@operator.example");
var saved = await harness.Sut.SaveAdminConfigurationAsync(incoming);
Assert.True(saved);
Assert.Null(result.Error);
Assert.Equal(new[] { "GlobalRateLimitRequestsPerMinute" }, result.ChangedColumns);
Assert.NotNull(snapshot);
Assert.Equal("GlobalConfig", snapshot!.PartitionKey);
Assert.Equal("admin-config", snapshot.Source);
@@ -0,0 +1,132 @@
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Functions.Admin;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
internal static class ConfigRepoMockExtensions
{
/// <summary>
/// Asserts that no tenant configuration row was created or replaced — the only two ways the
/// repository can write one (D-290).
/// </summary>
public static void VerifyNoTenantConfigWrite(this Mock<IConfigRepository> repo)
{
repo.Verify(r => r.TryCreateTenantConfigurationAsync(It.IsAny<TenantConfiguration>()), Times.Never);
repo.Verify(r => r.TryReplaceTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string>()), Times.Never);
repo.Verify(r => r.TryReplaceTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), It.IsAny<string>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
}
/// <summary>
/// Backs <see cref="TenantConfigurationService.UpdateAsync"/> and
/// <see cref="TenantConfigurationService.CreateOrUpdateAsync"/> of a service mock with one stored row
/// and the real contract: the mutation sees a fresh copy (never the caller's object), the offboarding
/// tombstone is refused, a declined mutation writes nothing, and every write is recorded.
/// </summary>
public static StoredTenantConfig StubWrites(this Mock<TenantConfigurationService> service, string tenantId, TenantConfiguration? row)
{
var store = new StoredTenantConfig { Row = row };
Task<TenantConfigUpdate> Run(Func<TenantConfiguration, bool> mutate, string source, string reason, bool createIfMissing)
{
if (store.Force is { } forced)
return Task.FromResult(new TenantConfigUpdate(forced, null));
if (store.ThrowOnRead is { } ex)
throw ex;
TenantConfiguration fresh;
if (store.Row == null)
{
if (!createIfMissing)
return Task.FromResult(new TenantConfigUpdate(TenantConfigUpdateStatus.NotFound, null));
fresh = TenantConfiguration.CreateDefault(tenantId);
}
else
{
fresh = Clone(store.Row);
if (TenantOffboardFunction.IsOffboardingTombstone(fresh))
return Task.FromResult(new TenantConfigUpdate(TenantConfigUpdateStatus.OffboardingInProgress, fresh));
}
if (!mutate(fresh))
return Task.FromResult(new TenantConfigUpdate(TenantConfigUpdateStatus.Declined, fresh));
fresh.LastUpdated = DateTime.UtcNow;
store.Row = fresh;
store.Writes.Add((fresh, source, reason));
return Task.FromResult(new TenantConfigUpdate(TenantConfigUpdateStatus.Updated, fresh));
}
service.Setup(x => x.UpdateAsync(tenantId, It.IsAny<Func<TenantConfiguration, bool>>(), It.IsAny<string>(), It.IsAny<string>()))
.Returns((string _, Func<TenantConfiguration, bool> mutate, string source, string reason) => Run(mutate, source, reason, false));
service.Setup(x => x.CreateOrUpdateAsync(tenantId, It.IsAny<Func<TenantConfiguration, bool>>(), It.IsAny<string>(), It.IsAny<string>()))
.Returns((string _, Func<TenantConfiguration, bool> mutate, string source, string reason) => Run(mutate, source, reason, true));
return store;
}
/// <summary>
/// Backs the tenant-configuration reads and writes of a repository mock with one stored row and
/// the storage contract: every read hands back a fresh copy, the replace succeeds only with the
/// current ETag, the insert only while no row exists, and every write is recorded.
/// </summary>
public static StoredTenantConfig BackWithRow(this Mock<IConfigRepository> repo, TenantConfiguration? row)
{
var store = new StoredTenantConfig { Row = row };
var version = 0;
string ETag() => $"etag-{version}";
TenantConfiguration? Read()
{
if (store.ThrowOnRead is { } ex) throw ex;
return store.Row == null ? null : Clone(store.Row);
}
bool Write(TenantConfiguration config, string source, string reason)
{
if (store.ThrowOnWrite is { } ex) throw ex;
if (store.RejectWrites) return false;
store.Row = Clone(config);
version++;
store.Writes.Add((store.Row, source, reason));
return true;
}
repo.Setup(r => r.GetTenantConfigurationAsync(It.IsAny<string>())).ReturnsAsync(() => Read());
repo.Setup(r => r.GetTenantConfigurationWithEtagAsync(It.IsAny<string>()))
.ReturnsAsync(() => Read() is { } fresh ? (fresh, ETag()) : null);
repo.Setup(r => r.TryReplaceTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), It.IsAny<string>(), It.IsAny<string?>(), It.IsAny<string?>()))
.ReturnsAsync((TenantConfiguration c, string etag, string? source, string? reason) =>
store.Row != null && etag == ETag() && Write(c, source ?? "", reason ?? ""));
repo.Setup(r => r.TryCreateTenantConfigurationAsync(It.IsAny<TenantConfiguration>()))
.ReturnsAsync((TenantConfiguration c) => store.Row == null && Write(c, "(create)", ""));
return store;
}
/// <summary>A storage round trip: what a fresh read of the stored row hands back.</summary>
public static TenantConfiguration Clone(TenantConfiguration config)
=> TableConfigRepository.ConvertFromTenantTableEntity(TableConfigRepository.ConvertToTenantTableEntity(config));
}
/// <summary>The stored row behind <see cref="ConfigRepoMockExtensions.StubWrites"/> and what was written to it.</summary>
internal sealed class StoredTenantConfig
{
public TenantConfiguration? Row { get; set; }
public List<(TenantConfiguration Row, string Source, string Reason)> Writes { get; } = new();
/// <summary>When set, every write returns this status without touching the row (e.g. Conflict).</summary>
public TenantConfigUpdateStatus? Force { get; set; }
/// <summary>When set, every read (and, for the service stub, every write) throws it — a storage outage.</summary>
public Exception? ThrowOnRead { get; set; }
/// <summary>Repository double only: when set, every write throws it.</summary>
public Exception? ThrowOnWrite { get; set; }
/// <summary>Repository double only: every conditional write loses (as if a concurrent writer always won).</summary>
public bool RejectWrites { get; set; }
}
@@ -713,31 +713,6 @@ public class ConfigWireParityTests
// ---- UpdateTenantConfiguration -------------------------------------------------------
[Fact]
public void UpdateTenantConfigurationResponse_matches_the_ack_plus_config_shape()
{
var config = new TenantConfiguration
{
TenantId = "6a6a35a2-30b2-4f2f-9a1b-6d9f1a2b3c4d",
DomainName = "contoso.com",
UpdatedBy = "admin@contoso.com",
};
AssertParity(
new
{
success = true,
message = "Configuration updated successfully",
config = config
},
new UpdateTenantConfigurationResponse
{
Success = true,
Message = "Configuration updated successfully",
Config = config
});
}
private static void AssertParity(object anonymousLiteral, IApiResponse typed)
=> ApiResponseWireParityTests.AssertWireIdentical(anonymousLiteral, typed);
}
@@ -74,8 +74,7 @@ public class CrossTenantAccessTests
[Theory]
[InlineData("GET", "/api/config/{0}")]
[InlineData("PUT", "/api/config/{0}")]
[InlineData("POST", "/api/config/{0}")]
[InlineData("PATCH", "/api/config/{0}/fields")]
[InlineData("GET", "/api/config/{0}/feature-flags")]
[InlineData("GET", "/api/tenants/{0}/admins")]
[InlineData("POST", "/api/tenants/{0}/admins")]
@@ -96,8 +95,7 @@ public class CrossTenantAccessTests
[Theory]
[InlineData("GET", "/api/config/{0}")]
[InlineData("PUT", "/api/config/{0}")]
[InlineData("POST", "/api/config/{0}")]
[InlineData("PATCH", "/api/config/{0}/fields")]
[InlineData("GET", "/api/config/{0}/feature-flags")]
[InlineData("GET", "/api/tenants/{0}/admins")]
[InlineData("POST", "/api/tenants/{0}/admins")]
@@ -118,7 +116,7 @@ public class CrossTenantAccessTests
[Theory]
[InlineData("GET", "/api/config/{0}")]
[InlineData("PUT", "/api/config/{0}")]
[InlineData("PATCH", "/api/config/{0}/fields")]
[InlineData("GET", "/api/tenants/{0}/admins")]
[InlineData("DELETE", "/api/tenants/{0}/offboard")]
public void CrossTenant_GlobalAdmin_IsAllowed(string httpMethod, string pathTemplate)
@@ -584,17 +584,18 @@ public class DelegatedAdminServiceTests
}
[Fact]
public async Task RenameGroup_DoesNotInvalidateCache()
public async Task UpdateGroup_DoesNotInvalidateCache()
{
var (svc, repo) = Build();
ReturnsGroups(repo, Assignment(GroupId1));
GroupTenants(repo, GroupId1, TenantA);
repo.Setup(r => r.RenameTenantGroupAsync(It.IsAny<string>(), It.IsAny<string>())).ReturnsAsync(true);
repo.Setup(r => r.UpdateTenantGroupAsync(It.IsAny<string>(), It.IsAny<string?>(), It.IsAny<string?>())).ReturnsAsync(true);
await svc.GetScopeAsync(Id(Upn));
await svc.RenameGroupAsync(GroupId1, "Renamed");
await svc.GetScopeAsync(Id(Upn)); // name-only change → scope stays cached
await svc.UpdateGroupAsync(GroupId1, "Renamed", "Service Desk");
await svc.GetScopeAsync(Id(Upn)); // name/label change → scope stays cached
repo.Verify(r => r.UpdateTenantGroupAsync(GroupId1, "Renamed", "Service Desk"), Times.Once);
repo.Verify(r => r.GetGroupAssignmentsForUpnAsync(It.IsAny<string>()), Times.Once);
}
}
@@ -563,4 +563,62 @@ public class DelegationSelfServiceTests
Assert.False(group.Revocable);
Assert.Equal(2, group.Assignees.Count);
}
[Fact]
public async Task ListManagers_TwoLabelledOperatorGroups_AreDistinguishable_DirectGrantsStayNeutral()
{
const string serviceGroupId = "0f1e2d3c4b5a69788796a5b4c3d2e1f0";
const string careGroupId = "1a2b3c4d5e6f70819293a4b5c6d7e8f9";
var h = Build();
h.Repo.Setup(x => x.GetGroupIdsContainingTenantAsync(Customer)).ReturnsAsync(new List<string> { serviceGroupId, careGroupId });
h.Repo.Setup(x => x.GetTenantGroupAsync(serviceGroupId)).ReturnsAsync(new TenantGroup
{
GroupId = serviceGroupId, Name = "Internal managed service tenants", CustomerLabel = "Contoso Managed Service Team",
TenantIds = new List<string> { Customer },
});
h.Repo.Setup(x => x.GetTenantGroupAsync(careGroupId)).ReturnsAsync(new TenantGroup
{
GroupId = careGroupId, Name = "Internal care unit", CustomerLabel = " Contoso Customer Care Unit ",
TenantIds = new List<string> { Customer },
});
h.Repo.Setup(x => x.GetDelegatedAssigneesAsync(Customer)).ReturnsAsync(new List<DelegatedAdminEntry>
{
new() { Upn = "ops@vendor.example", TenantId = Customer, Role = Constants.DelegatedRoles.DelegatedReader, IsEnabled = true, Status = Constants.DelegatedStatus.Active, GrantedAt = Now.AddDays(-10) },
});
var managers = await h.Svc.ListManagersAsync(Customer);
Assert.Equal("Contoso Managed Service Team", managers.Single(m => m.GroupId == serviceGroupId).Name);
Assert.Equal("Contoso Customer Care Unit", managers.Single(m => m.GroupId == careGroupId).Name);
Assert.All(managers.Where(m => m.GroupId != null), m =>
{
Assert.Equal(DelegationSelfService.SourceOperator, m.Source);
Assert.False(m.Revocable);
Assert.DoesNotContain("Internal", m.Name, StringComparison.Ordinal);
});
// A direct grant has no group, hence no label — it keeps the neutral name.
Assert.Equal(DelegationSelfService.OperatorLabel, managers.Single(m => m.GroupId == null).Name);
}
// ── Customer-facing group name ──────────────────────────────────────────────
[Theory]
[InlineData(null, "Platform support")]
[InlineData("", "Platform support")]
[InlineData(" ", "Platform support")]
[InlineData(" Contoso Service Desk ", "Contoso Service Desk")]
public void CustomerNameOf_OperatorGroup_LabelElseNeutral_NeverTheInternalName(string? label, string expected)
{
var group = new TenantGroup { GroupId = "0f1e2d3c4b5a69788796a5b4c3d2e1f0", Name = "Internal name", CustomerLabel = label };
Assert.Equal(expected, DelegationSelfService.CustomerNameOf(group));
}
[Fact]
public void CustomerNameOf_OwnedGroup_IsItsName_LabelIgnored()
{
var group = new TenantGroup { GroupId = GroupId, Name = "partner.example", OwnerTenantId = Home, CustomerLabel = "Ignored" };
Assert.Equal("partner.example", DelegationSelfService.CustomerNameOf(group));
}
}
@@ -0,0 +1,74 @@
using System.Net;
using System.Text;
using System.Text.Json;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared;
using Azure.Core.Serialization;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.DependencyInjection;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Minimal HTTP endpoint harness: a mocked <see cref="HttpRequestData"/> whose FunctionContext carries the
/// <see cref="RequestContext"/> the policy middleware would have resolved, serialized with the production
/// wire settings.
/// </summary>
internal static class EndpointHarness
{
public static (HttpRequestData Req, FunctionContext Context) Request(
string tenantId, string upn = "admin@contoso.com", string? jsonBody = null, string queryString = "",
IDictionary<string, string>? headers = null)
{
var services = new ServiceCollection();
services.AddOptions();
services.Configure<WorkerOptions>(o => o.Serializer = new JsonObjectSerializer(ApiJsonOptions.Create()));
var provider = services.BuildServiceProvider();
var items = new Dictionary<object, object>
{
["RequestContext"] = new RequestContext
{
TenantId = tenantId,
TargetTenantId = tenantId,
UserPrincipalName = upn,
IsTenantAdmin = true,
UserRole = Constants.TenantRoles.Admin,
},
};
var context = new Mock<FunctionContext>();
context.SetupGet(c => c.Items).Returns(items);
context.SetupGet(c => c.InstanceServices).Returns(provider);
var req = new Mock<HttpRequestData>(context.Object);
var headerCollection = new HttpHeadersCollection();
foreach (var (name, value) in headers ?? new Dictionary<string, string>())
headerCollection.Add(name, value);
req.SetupGet(r => r.Headers).Returns(headerCollection);
// Query parses the Url in the base class (a bare mock would hand back null).
req.SetupGet(r => r.Url).Returns(new Uri("https://localhost/api/test" + queryString));
req.SetupGet(r => r.Query).CallBase();
req.SetupGet(r => r.Body).Returns(new MemoryStream(Encoding.UTF8.GetBytes(jsonBody ?? "")));
req.Setup(r => r.CreateResponse()).Returns(() => new FakeResponse(context.Object));
return (req.Object, context.Object);
}
/// <summary>The <c>code</c> of an error envelope; null when the body carries none.</summary>
public static string? ErrorCode(HttpResponseData response)
{
response.Body.Position = 0;
using var doc = JsonDocument.Parse(response.Body);
return doc.RootElement.TryGetProperty("code", out var code) ? code.GetString() : null;
}
private sealed class FakeResponse : HttpResponseData
{
public FakeResponse(FunctionContext context) : base(context) { }
public override HttpStatusCode StatusCode { get; set; } = HttpStatusCode.OK;
public override HttpHeadersCollection Headers { get; set; } = new();
public override Stream Body { get; set; } = new MemoryStream();
public override HttpCookies Cookies { get; } = new Mock<HttpCookies>().Object;
}
}
@@ -156,14 +156,17 @@ public class EndpointPolicyCatalogCompletenessTests
public void SameRoute_DifferentMethods_ResolveToDifferentPolicies()
{
var getConfig = EndpointAccessPolicyCatalog.FindPolicy("GET", "/api/config/tenant-1");
var putConfig = EndpointAccessPolicyCatalog.FindPolicy("PUT", "/api/config/tenant-1");
var patchConfig = EndpointAccessPolicyCatalog.FindPolicy("PATCH", "/api/config/tenant-1/fields");
Assert.NotNull(getConfig);
Assert.NotNull(putConfig);
Assert.NotNull(patchConfig);
// GET config read is member-tier (Operators/Viewers get the redacted read-only Settings
// view; the handler redacts for every non-admin caller); PUT config write stays Admin/GA only.
// view; the handler redacts for every non-admin caller); the field PATCH stays Admin/GA only.
Assert.Equal(EndpointPolicy.MemberRead, getConfig.Policy);
Assert.Equal(EndpointPolicy.TenantAdminOrGA, putConfig.Policy);
Assert.Equal(EndpointPolicy.TenantAdminOrGA, patchConfig.Policy);
// There is no full-model write of the tenant configuration (D-290).
Assert.Null(EndpointAccessPolicyCatalog.FindPolicy("PUT", "/api/config/tenant-1"));
Assert.Null(EndpointAccessPolicyCatalog.FindPolicy("POST", "/api/config/tenant-1"));
var getRules = EndpointAccessPolicyCatalog.FindPolicy("GET", "/api/rules/gather");
var postRules = EndpointAccessPolicyCatalog.FindPolicy("POST", "/api/rules/gather");
@@ -1,4 +1,5 @@
using System.Text.Json;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Functions.Config;
using AutopilotMonitor.Functions.Services.Notifications;
using AutopilotMonitor.Shared.Models;
@@ -140,7 +141,7 @@ public class GenericWebhookTests
Assert.Empty(config.GetGenericWebhookHeaders());
}
// ── UpdateTenantConfigurationFunction.ValidateWebhookCustomHeaders ────
// ── TenantConfigValidation.ValidateWebhookCustomHeaders ────
[Theory]
[InlineData(null)]
@@ -149,7 +150,7 @@ public class GenericWebhookTests
[InlineData("{\"Authorization\":\"Bearer abc\"}")]
public void ValidateWebhookCustomHeaders_Accepts_EmptyOrValid(string? json)
{
Assert.Null(UpdateTenantConfigurationFunction.ValidateWebhookCustomHeaders(json));
Assert.Null(TenantConfigValidation.ValidateWebhookCustomHeaders(json));
}
[Theory]
@@ -160,7 +161,7 @@ public class GenericWebhookTests
[InlineData("{\"X-Key\":\"line1\\nline2\"}", "must not contain line breaks")]
public void ValidateWebhookCustomHeaders_Rejects_Invalid(string json, string expectedFragment)
{
var error = UpdateTenantConfigurationFunction.ValidateWebhookCustomHeaders(json);
var error = TenantConfigValidation.ValidateWebhookCustomHeaders(json);
Assert.NotNull(error);
Assert.Contains(expectedFragment, error);
}
@@ -115,4 +115,21 @@ public class GeographicDrilldownFilterTests
Assert.Equal("Saxony", fal.Region);
Assert.Equal("Falkenstein", fal.City);
}
[Theory]
[InlineData("city")]
[InlineData("region")]
[InlineData("country")]
public void CountryOnlySession_KeyIsTheCountryAndDrillsDownToItself(string groupBy)
{
// The agent's last-resort geo provider returns the country only.
var countryOnly = Geo("de-only", "DE", "", "");
var sessions = new List<SessionSummary> { countryOnly, Geo("de-bav", "DE", "Bavaria", "Munich") };
var key = GetGeographicMetricsFunction.GetLocationKey(countryOnly, groupBy);
var drilled = GetGeographicLocationSessionsFunction.FilterSessionsByLocation(sessions, key, groupBy);
Assert.Equal("DE", key); // never ", DE" or ", , DE"
Assert.Contains(drilled, s => s.SessionId == "de-only");
}
}
@@ -0,0 +1,123 @@
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.Models;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Tenant Hello wait (<see cref="TenantConfiguration.HelloWaitTimeoutSeconds"/>): a changed value
/// must lie in 30..3600 s. Like the retention cap, only a CHANGED value is checked, so a value
/// stored before the rule existed never blocks an unrelated save or a backup revert (both
/// validate the whole model).
/// </summary>
public class HelloWaitTimeoutConfigTests
{
private const string TenantId = "11111111-1111-1111-1111-111111111111";
[Theory]
[InlineData(30)]
[InlineData(300)]
[InlineData(1800)]
[InlineData(3600)]
public void ValidateModel_accepts_the_supported_range(int seconds)
{
var candidate = TenantConfiguration.CreateDefault(TenantId);
candidate.HelloWaitTimeoutSeconds = seconds;
Assert.Null(TenantConfigValidation.ValidateModel(candidate, TenantConfiguration.CreateDefault(TenantId), isGlobalAdmin: false));
}
[Theory]
[InlineData(29)]
[InlineData(3601)]
[InlineData(0)]
[InlineData(-1)]
public void ValidateModel_rejects_a_changed_value_outside_the_range(int seconds)
{
var candidate = TenantConfiguration.CreateDefault(TenantId);
candidate.HelloWaitTimeoutSeconds = seconds;
var error = TenantConfigValidation.ValidateModel(candidate, TenantConfiguration.CreateDefault(TenantId), isGlobalAdmin: true);
Assert.Equal("Hello wait timeout must be between 30 and 3600 seconds.", error);
}
[Fact]
public void ValidateModel_lets_an_unchanged_out_of_range_value_through()
{
// Written through the API before the range was validated — an unrelated save (or a
// revert to a snapshot holding the same value) must not fail on it.
var existing = TenantConfiguration.CreateDefault(TenantId);
existing.HelloWaitTimeoutSeconds = 86400;
var candidate = TenantConfiguration.CreateDefault(TenantId);
candidate.HelloWaitTimeoutSeconds = 86400;
candidate.ContactEmail = "it@example.com";
Assert.Null(TenantConfigValidation.ValidateModel(candidate, existing, isGlobalAdmin: false));
}
[Fact]
public void Default_is_the_built_in_five_minutes()
{
Assert.Equal(300, TenantConfiguration.CreateDefault(TenantId).HelloWaitTimeoutSeconds);
Assert.Equal(300, new CollectorConfiguration().HelloWaitTimeoutSeconds);
Assert.Equal(300, HelloWaitTimeout.EffectiveSeconds(HelloWaitTimeout.DefaultSeconds));
}
[Fact]
public void Stored_old_default_stays_valid_and_keeps_the_built_in_window()
{
// Every tenant that ever saved the agent settings page has the old 30 s default stored.
var existing = TenantConfiguration.CreateDefault(TenantId);
existing.HelloWaitTimeoutSeconds = 30;
var candidate = TenantConfiguration.CreateDefault(TenantId);
candidate.HelloWaitTimeoutSeconds = 30;
candidate.ContactEmail = "it@example.com";
Assert.Null(TenantConfigValidation.ValidateModel(candidate, existing, isGlobalAdmin: false));
Assert.Equal(300, HelloWaitTimeout.EffectiveSeconds(30));
}
[Theory]
[InlineData(30, 300)]
[InlineData(300, 300)]
[InlineData(301, 301)]
[InlineData(3600, 3600)]
[InlineData(86400, 3600)]
[InlineData(0, 300)]
public void EffectiveSeconds_keeps_the_built_in_window_as_floor_and_caps_at_one_hour(int configured, int expected)
{
Assert.Equal(expected, HelloWaitTimeout.EffectiveSeconds(configured));
}
[Fact]
public void Row_without_the_column_reads_as_the_default()
{
var entity = TableConfigRepository.ConvertToTenantTableEntity(TenantConfiguration.CreateDefault(TenantId));
entity.Remove("HelloWaitTimeoutSeconds");
Assert.Equal(300, TableConfigRepository.ConvertFromTenantTableEntity(entity).HelloWaitTimeoutSeconds);
}
[Fact]
public void Stored_value_is_read_verbatim()
{
var entity = TableConfigRepository.ConvertToTenantTableEntity(TenantConfiguration.CreateDefault(TenantId));
entity["HelloWaitTimeoutSeconds"] = 30;
Assert.Equal(30, TableConfigRepository.ConvertFromTenantTableEntity(entity).HelloWaitTimeoutSeconds);
}
[Fact]
public void Configured_value_roundtrips_store_and_map()
{
var config = TenantConfiguration.CreateDefault(TenantId);
config.HelloWaitTimeoutSeconds = 2400;
var mapped = TableConfigRepository.ConvertFromTenantTableEntity(
TableConfigRepository.ConvertToTenantTableEntity(config));
Assert.Equal(2400, mapped.HelloWaitTimeoutSeconds);
}
}
@@ -153,8 +153,7 @@ public class McpUserServiceTenantMcpSwitchTests
var result = await _sut.IsAllowedAsync(Upn, HomeTenant, Oid);
Assert.True(result.IsAllowed);
_configRepo.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>()), Times.Never);
_configRepo.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
_configRepo.VerifyNoTenantConfigWrite();
}
[Fact]
@@ -218,14 +218,14 @@ public class NotificationChannelsTests
Assert.Equal("https://second.example/hook", channels[1].Url);
}
// ── Save-time validation (UpdateTenantConfigurationFunction) ─────────
// ── Save-time validation (TenantConfigValidation) ─────────
[Fact]
public void ValidateNotificationChannels_NullEmptyOrValidList_PassesThrough()
{
Assert.Null(UpdateTenantConfigurationFunction.ValidateNotificationChannels(null));
Assert.Null(UpdateTenantConfigurationFunction.ValidateNotificationChannels(""));
Assert.Null(UpdateTenantConfigurationFunction.ValidateNotificationChannels(
Assert.Null(TenantConfigValidation.ValidateNotificationChannels(null));
Assert.Null(TenantConfigValidation.ValidateNotificationChannels(""));
Assert.Null(TenantConfigValidation.ValidateNotificationChannels(
"[" + ChannelJson("ch-1") + "," + ChannelJson("ch-2", providerType: 10) + "]"));
}
@@ -236,7 +236,7 @@ public class NotificationChannelsTests
[InlineData("[{\"id\":\"a\",\"providerType\":20,\"url\":\"http://plain.example\"}]", "")] // non-https rejected by SsrfGuard format check
public void ValidateNotificationChannels_RejectsInvalidEntries(string json, string expectedFragment)
{
var error = UpdateTenantConfigurationFunction.ValidateNotificationChannels(json);
var error = TenantConfigValidation.ValidateNotificationChannels(json);
Assert.NotNull(error);
if (expectedFragment.Length > 0)
Assert.Contains(expectedFragment, error);
@@ -245,7 +245,7 @@ public class NotificationChannelsTests
[Fact]
public void ValidateNotificationChannels_RejectsDuplicateIds()
{
var error = UpdateTenantConfigurationFunction.ValidateNotificationChannels(
var error = TenantConfigValidation.ValidateNotificationChannels(
"[" + ChannelJson("ch-1") + "," + ChannelJson("ch-1") + "]");
Assert.NotNull(error);
Assert.Contains("duplicate", error);
@@ -255,7 +255,7 @@ public class NotificationChannelsTests
public void ValidateNotificationChannels_RejectsBadPerChannelHeaders()
{
var json = "[" + ChannelJson("ch-1", extra: ",\"customHeadersJson\":\"not-json\"") + "]";
var error = UpdateTenantConfigurationFunction.ValidateNotificationChannels(json);
var error = TenantConfigValidation.ValidateNotificationChannels(json);
Assert.NotNull(error);
Assert.Contains("headers", error);
}
@@ -291,7 +291,7 @@ public class NotificationChannelsTests
public void ValidateNotificationChannels_RejectsBadPerChannelSigningSecret()
{
var json = "[" + ChannelJson("ch-1", extra: ",\"signingSecret\":\"too-short\"") + "]";
var error = UpdateTenantConfigurationFunction.ValidateNotificationChannels(json);
var error = TenantConfigValidation.ValidateNotificationChannels(json);
Assert.NotNull(error);
Assert.Contains("signing secret", error);
}
@@ -88,9 +88,14 @@ internal sealed class FakeOffboardingAuditRepository : IOffboardingAuditReposito
return Task.CompletedTask;
}
/// <summary>When set, every pointer read throws — a storage outage on the re-onboarding lookup.</summary>
public Exception? ThrowOnPointerRead { get; set; }
public Task<(OffboardingByTenantPointer? Pointer, string? ETag)> TryGetByTenantPointerAsync(
string normalizedTenantId, CancellationToken ct = default)
{
if (ThrowOnPointerRead is { } ex)
throw ex;
if (Pointers.TryGetValue(normalizedTenantId, out var entry))
{
return Task.FromResult<(OffboardingByTenantPointer?, string?)>((Clone(entry.Pointer), entry.ETag));
@@ -152,5 +157,6 @@ internal sealed class FakeOffboardingAuditRepository : IOffboardingAuditReposito
PartitionKey = p.PartitionKey, RowKey = p.RowKey, TenantId = p.TenantId,
LatestHistoryRowKey = p.LatestHistoryRowKey, LatestStatus = p.LatestStatus,
LatestUpdatedAt = p.LatestUpdatedAt, OffboardCount = p.OffboardCount,
TrialConsumed = p.TrialConsumed,
};
}
@@ -175,7 +175,8 @@ public class OpsEventFieldFilterTests
// unfiltered rows — the exact bug this test exists to prevent.
var filters = new OpsEventQueryFilters { EventType = "AgentEmergencyBreak", MinSeverity = "Error" };
var fanOut = TableOpsEventRepository.BuildFilterWithRowKeyBound("Agent", null, null, "0009", filters);
var fanOut = TableOpsEventRepository.BuildFilterWithRowKeyBound(
"Agent", null, null, new PerPartitionFanOutMerge.RowKeyBound("0009", Inclusive: false), filters);
Assert.Contains("EventType eq 'AgentEmergencyBreak'", fanOut);
Assert.Contains("(Severity eq 'Error' or Severity eq 'Critical')", fanOut);
@@ -1,325 +1,283 @@
using AutopilotMonitor.Functions.DataAccess.TableStorage;
using AutopilotMonitor.Shared.DataAccess;
using static AutopilotMonitor.Functions.DataAccess.TableStorage.PerPartitionFanOutMerge;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Cross-partition merge used by both the global all-category ops-events page
/// and the global cross-tenant audit-logs page. Without this fan-out, Azure
/// pages cross-partition queries by (PK asc, RK asc) — so page 1 would be
/// entirely the alphabetically-first partition, masking newer items from
/// later-alphabet partitions. The merge picks top-pageSize by timestamp across
/// all per-partition fetches.
/// Cross-partition paging behind the global all-category ops-events page and the global
/// cross-tenant audit-log page. Every page fans out one read per partition and merges by
/// (RowKey, Partition); the continuation is the watermark of the last returned row. The tests
/// drive the helper against an in-memory partition store that honours the RowKey bound exactly
/// like the OData clause does.
/// </summary>
public class PerPartitionFanOutMergeTests
{
private static OpsEventEntry Entry(string partition, DateTime ts, string id)
=> new() { Category = partition, Id = id, Timestamp = ts, EventType = "synthetic" };
private static (string RowKey, OpsEventEntry Item) Row(DateTime ts, string id)
// Reverse-tick RowKey like the storage layer writes it: a smaller string is a newer row.
private static string Rk(int minutesAgo, string suffix = "")
{
// Mimics the storage-layer revtick rowkey: smaller string = newer timestamp.
var rk = $"{(DateTime.MaxValue.Ticks - ts.Ticks):D19}_{id}";
return (rk, Entry(partition: "<set-by-caller>", ts, id));
var t0 = new DateTime(2026, 9, 28, 12, 0, 0, DateTimeKind.Utc);
return $"{DateTime.MaxValue.Ticks - t0.AddMinutes(-minutesAgo).Ticks:D19}{suffix}";
}
private static PartitionFetchResult<OpsEventEntry> Fetched(
string partition, params (string RowKey, OpsEventEntry Item)[] rows)
private sealed class Store
{
var aligned = rows
.Select(r => (r.RowKey, Item: new OpsEventEntry
private readonly Dictionary<string, List<string>> _rows;
private readonly Func<string, bool> _matches;
public List<(string Partition, RowKeyBound? Bound, int Limit)> Calls { get; } = new();
public Store(Dictionary<string, List<string>> rows, Func<string, bool>? matches = null)
{
_rows = rows;
_matches = matches ?? (_ => true);
}
public IEnumerable<string> Partitions => _rows.Keys;
public PartitionReader<string> Reader => (partition, bound, limit, _) =>
{
lock (Calls) Calls.Add((partition, bound, limit));
var read = _rows[partition]
.Where(_matches)
.Where(rk => bound == null
|| (bound.Inclusive
? string.CompareOrdinal(rk, bound.RowKey) >= 0
: string.CompareOrdinal(rk, bound.RowKey) > 0))
.OrderBy(rk => rk, StringComparer.Ordinal)
.Take(limit)
.Select(rk => (rk, $"{partition}/{rk}"))
.ToList();
return Task.FromResult(read);
};
/// <summary>The full newest-first order the pages must reproduce.</summary>
public List<string> Expected() => _rows
.SelectMany(kv => kv.Value.Where(_matches).Select(rk => (Partition: kv.Key, RowKey: rk)))
.OrderBy(r => r.RowKey, StringComparer.Ordinal)
.ThenBy(r => r.Partition, StringComparer.Ordinal)
.Select(r => $"{r.Partition}/{r.RowKey}")
.ToList();
}
private static async Task<List<List<string>>> WalkAsync(Store store, int pageSize)
{
var pages = new List<List<string>>();
string? continuation = null;
do
{
var page = await FetchPageAsync(store.Partitions, pageSize, continuation, store.Reader);
pages.Add(page.Items.ToList());
continuation = page.NextRawToken;
if (pages.Count > 1000) Assert.Fail("Walk did not terminate");
}
while (continuation != null);
return pages;
}
// ────────── Order ──────────────────────────────────────────────────────
[Fact]
public async Task First_page_is_newest_across_partitions_not_alphabetically_first()
{
var store = new Store(new()
{
["Agent"] = new() { Rk(5) },
["Security"] = new() { Rk(0) },
["Tenant"] = new() { Rk(10) },
});
var page = await FetchPageAsync(store.Partitions, 3, null, store.Reader);
Assert.Equal(new[] { $"Security/{Rk(0)}", $"Agent/{Rk(5)}", $"Tenant/{Rk(10)}" }, page.Items);
Assert.Null(page.NextRawToken);
}
[Fact]
public async Task First_page_reads_every_partition_unbounded_with_one_row_of_lookahead()
{
var store = new Store(new() { ["A"] = new() { Rk(1) }, ["B"] = new(), ["C"] = new() { Rk(2) } });
await FetchPageAsync(store.Partitions, 15, null, store.Reader);
Assert.Equal(new[] { "A", "B", "C" }, store.Calls.Select(c => c.Partition).OrderBy(p => p));
Assert.All(store.Calls, c => Assert.Null(c.Bound));
Assert.All(store.Calls, c => Assert.Equal(16, c.Limit));
}
// ────────── Watermark and bounds ───────────────────────────────────────
[Fact]
public void Bound_is_exclusive_up_to_the_watermark_partition_and_inclusive_after_it()
{
var w = new Watermark("0000000000000000042", "Maintenance");
Assert.Null(BoundAfter(null, "Agent"));
Assert.Equal(new RowKeyBound(w.RowKey, Inclusive: false), BoundAfter(w, "Agent"));
Assert.Equal(new RowKeyBound(w.RowKey, Inclusive: false), BoundAfter(w, "Maintenance"));
Assert.Equal(new RowKeyBound(w.RowKey, Inclusive: true), BoundAfter(w, "Security"));
}
[Fact]
public void Bound_renders_as_an_escaped_odata_rowkey_clause()
{
Assert.Equal("RowKey gt '!0042_a'", new RowKeyBound("!0042_a", false).ToODataClause());
Assert.Equal("RowKey ge 'it''s'", new RowKeyBound("it's", true).ToODataClause());
}
[Fact]
public async Task Equal_rowkeys_in_different_partitions_all_survive_a_page_boundary()
{
// Ops-event RowKeys are bare reverse ticks, so two categories can share one. With a
// RowKey-only watermark the second row would be skipped after the cut.
var shared = Rk(3);
var store = new Store(new()
{
["Agent"] = new() { shared },
["Security"] = new() { shared },
["Tenant"] = new() { shared, Rk(4) },
});
var pages = await WalkAsync(store, pageSize: 1);
Assert.Equal(store.Expected(), pages.SelectMany(p => p));
Assert.Equal(4, pages.Count);
}
// ────────── Exact end of list ──────────────────────────────────────────
[Fact]
public async Task No_continuation_when_the_rows_exactly_fill_the_page()
{
var store = new Store(new() { ["A"] = new() { Rk(1), Rk(2), Rk(3) }, ["B"] = new() });
var page = await FetchPageAsync(store.Partitions, 3, null, store.Reader);
Assert.Equal(3, page.Items.Count);
Assert.Null(page.NextRawToken);
}
[Fact]
public async Task Continuation_when_one_row_is_left_over()
{
var store = new Store(new() { ["A"] = new() { Rk(1), Rk(2), Rk(3), Rk(4) } });
var page = await FetchPageAsync(store.Partitions, 3, null, store.Reader);
Assert.Equal(3, page.Items.Count);
Assert.NotNull(page.NextRawToken);
}
// ────────── Full walks ─────────────────────────────────────────────────
[Theory]
[InlineData(1)]
[InlineData(2)]
[InlineData(3)]
[InlineData(5)]
[InlineData(15)]
[InlineData(100)]
public async Task Walk_returns_every_row_once_in_global_order_and_never_an_empty_page(int pageSize)
{
// Uneven partitions: one empty, one dominant, interleaved timestamps and a tie.
var store = new Store(new()
{
["00000000-0000-0000-0000-000000000000"] = new() { Rk(0, "_g0"), Rk(40, "_g1") },
["1a"] = new() { Rk(1, "_a"), Rk(2, "_a"), Rk(3, "_a"), Rk(4, "_a"), Rk(5, "_a"), Rk(6, "_a"), Rk(30, "_a") },
["2b"] = new(),
["3c"] = new() { Rk(2, "_c"), Rk(7, "_c"), Rk(8, "_c") },
["4d"] = new() { Rk(9, "_a") },
["5e"] = new() { Rk(9, "_a"), Rk(50, "_e") },
});
var pages = await WalkAsync(store, pageSize);
var expected = store.Expected();
Assert.Equal(expected, pages.SelectMany(p => p));
Assert.All(pages, p => Assert.NotEmpty(p));
Assert.All(pages.Take(pages.Count - 1), p => Assert.Equal(pageSize, p.Count));
}
[Fact]
public async Task Walk_with_a_server_side_filter_returns_exactly_the_matching_rows()
{
// The reader applies the filter (like `Action ne 'deletion_started'`), so excluded rows
// never take a page slot and never become a watermark.
var store = new Store(
new()
{
Id = r.Item.Id,
Category = partition,
EventType = r.Item.EventType,
Timestamp = r.Item.Timestamp,
}))
.ToArray();
return new PartitionFetchResult<OpsEventEntry>(partition, aligned);
}
["A"] = new() { Rk(1, "_keep"), Rk(2, "_drop"), Rk(3, "_drop"), Rk(4, "_keep") },
["B"] = new() { Rk(2, "_drop"), Rk(5, "_keep"), Rk(6, "_drop") },
},
matches: rk => rk.EndsWith("_keep", StringComparison.Ordinal));
// ────────── Merge picks newest-across-partitions, not within-partition ──
var pages = await WalkAsync(store, pageSize: 2);
[Fact]
public void First_page_returns_newest_across_partitions_not_alphabetically_first()
{
// Without the merge, Azure's (PK asc, RK asc) ordering would surface
// "Agent" partition items first (alphabetically), even though "Security"
// has a newer event. The merge must pick by timestamp, not partition.
var t0 = DateTime.UtcNow;
var fetched = new[]
{
Fetched("Agent", Row(t0.AddMinutes(-5), "agent-1")),
Fetched("Security", Row(t0, "sec-1")),
Fetched("Tenant", Row(t0.AddMinutes(-10), "tenant-1")),
};
var (items, _) = MergeAndAdvance(
fetched,
new Dictionary<string, PartitionContinuation>(),
pageSize: 3,
timestampSelector: e => e.Timestamp);
Assert.Equal(3, items.Count);
Assert.Equal("sec-1", items[0].Id); // newest first
Assert.Equal("agent-1", items[1].Id);
Assert.Equal("tenant-1", items[2].Id);
Assert.Equal(store.Expected(), pages.SelectMany(p => p));
Assert.Equal(3, store.Expected().Count);
}
[Fact]
public void Empty_fetch_drops_partition_from_continuation_map()
public async Task Empty_store_yields_one_empty_page_without_continuation()
{
// Wire-format v2 invariant: exhausted partitions are absent from the
// continuation, not encoded with `x:true`. Consumers treat "missing"
// as exhausted on subsequent pages — that's what shrinks the audit
// fan-out token from 30+ KB to a few hundred bytes.
var t0 = DateTime.UtcNow;
var fetched = new[]
{
Fetched("Agent"), // empty fetch
Fetched("Security", Row(t0, "sec-1")),
};
var store = new Store(new() { ["A"] = new(), ["B"] = new() });
var (_, conts) = MergeAndAdvance(
fetched,
new Dictionary<string, PartitionContinuation>(),
pageSize: 5,
timestampSelector: e => e.Timestamp);
var page = await FetchPageAsync(store.Partitions, 15, null, store.Reader);
Assert.False(conts.ContainsKey("Agent"));
Assert.True(conts.ContainsKey("Security"));
Assert.Empty(page.Items);
Assert.Null(page.NextRawToken);
}
// ────────── Continuation advance: avoid losing leftovers when a partition loses the cut ──
// ────────── Token ──────────────────────────────────────────────────────
[Fact]
public void Continuation_keeps_position_for_partitions_whose_items_lost_the_merge_cut()
public async Task Continuation_size_does_not_grow_with_the_number_of_partitions()
{
// Page 1: pageSize=3. Agent fetched 5 items, Security fetched 5 items, but
// Security's are all newer. Merge takes 3 from Security; Agent's 5 items
// all lost the cut. Agent's continuation MUST stay at its previous
// position so the items get re-fetched and compete on page 2.
var t0 = DateTime.UtcNow;
var fetched = new[]
{
Fetched("Agent",
Row(t0.AddMinutes(-100), "a1"),
Row(t0.AddMinutes(-101), "a2"),
Row(t0.AddMinutes(-102), "a3"),
Row(t0.AddMinutes(-103), "a4"),
Row(t0.AddMinutes(-104), "a5")),
Fetched("Security",
Row(t0, "s1"),
Row(t0.AddMinutes(-1), "s2"),
Row(t0.AddMinutes(-2), "s3"),
Row(t0.AddMinutes(-3), "s4"),
Row(t0.AddMinutes(-4), "s5")),
};
// The per-partition map this replaced grew with every partition holding rows until the
// nextLink failed with HTTP 414. Every partition here has rows that lose the merge cut.
static Store Tenants(int count) => new(Enumerable.Range(0, count).ToDictionary(
i => Guid.Parse($"00000000-0000-0000-0000-{i:D12}").ToString(),
i => new List<string> { Rk(i, "_" + Guid.Empty.ToString("N")), Rk(i + 1000, "_" + Guid.Empty.ToString("N")) }));
var (items, conts) = MergeAndAdvance(
fetched,
new Dictionary<string, PartitionContinuation>(),
pageSize: 3,
timestampSelector: e => e.Timestamp);
var few = await FetchPageAsync(Tenants(3).Partitions, 1, null, Tenants(3).Reader);
var many = await FetchPageAsync(Tenants(500).Partitions, 1, null, Tenants(500).Reader);
// Security wins all three slots.
Assert.Equal(new[] { "s1", "s2", "s3" }, items.Select(i => i.Id).ToArray());
// Agent had no items returned and a non-empty fetch → keep prior continuation
// (null = restart). The partition stays in the map (active with no rk),
// so the next page will re-fetch its items and they'll be re-considered
// at a lower floor.
Assert.True(conts.ContainsKey("Agent"));
Assert.Null(conts["Agent"].LastRowKey);
Assert.True(conts.ContainsKey("Security"));
Assert.NotNull(conts["Security"].LastRowKey);
Assert.NotNull(many.NextRawToken);
Assert.Equal(few.NextRawToken!.Length, many.NextRawToken!.Length);
Assert.True(many.NextRawToken.Length < 200, $"raw continuation is {many.NextRawToken.Length} characters");
}
[Fact]
public void Continuation_advances_to_oldest_returned_rowkey_per_partition()
public void Watermark_round_trips()
{
// When merge takes some-but-not-all from a partition, advance only past the
// items we actually returned. The leftover (older) items in that fetch
// will be re-fetched and re-considered on the next page.
var t0 = DateTime.UtcNow;
var (rkNew, _) = Row(t0, "x-new");
var (rkMid, _) = Row(t0.AddMinutes(-5), "x-mid");
var (rkOld, _) = Row(t0.AddMinutes(-10), "x-old");
var fetched = new[]
{
Fetched("Agent",
(rkNew, Entry("Agent", t0, "x-new")),
(rkMid, Entry("Agent", t0.AddMinutes(-5), "x-mid")),
(rkOld, Entry("Agent", t0.AddMinutes(-10), "x-old"))),
// A second partition whose only event is between Agent's mid and old
// items, ensuring the merge ends after Agent's mid item.
Fetched("Security",
Row(t0.AddMinutes(-7), "s-only")),
};
var w = new Watermark("!2516116980007133310_a4bb51498cad468f89431b15f1e701f7", "tenant-a");
var (items, conts) = MergeAndAdvance(
fetched,
new Dictionary<string, PartitionContinuation>(),
pageSize: 3,
timestampSelector: e => e.Timestamp);
// Top 3 by timestamp desc: x-new, x-mid, s-only. x-old is dropped.
Assert.Equal(new[] { "x-new", "x-mid", "s-only" }, items.Select(i => i.Id).ToArray());
// Agent's continuation advances to rkMid (the oldest Agent rowkey we returned).
// x-old will be re-fetched and re-considered next page (filter: RowKey gt rkMid).
Assert.Equal(rkMid, conts["Agent"].LastRowKey);
}
// ────────── Codec round-trip ────────────────────────────────────────────
[Fact]
public void Encode_then_decode_round_trips_active_partitions()
{
var original = new Dictionary<string, PartitionContinuation>
{
["Agent"] = new PartitionContinuation("0123456789012345678_aaa"),
["Consent"] = new PartitionContinuation("0987654321098765432_bbb"),
["Tenant"] = new PartitionContinuation(null), // active, restart from top
};
var encoded = EncodeMultiContinuation(original);
var decoded = DecodeMultiContinuation(encoded);
Assert.Equal(3, decoded.Count);
Assert.Equal("0123456789012345678_aaa", decoded["Agent"].LastRowKey);
Assert.Equal("0987654321098765432_bbb", decoded["Consent"].LastRowKey);
Assert.Null(decoded["Tenant"].LastRowKey);
Assert.Equal(w, DecodeWatermark(EncodeWatermark(w)));
}
[Fact]
public void Decode_silently_drops_legacy_v1_exhausted_entries()
public void Decode_rejects_anything_that_is_not_a_v3_watermark()
{
// Wire format v1 (pre-fix) explicitly encoded `x:true` for exhausted
// partitions; v2 just omits them. Tokens that were already in flight
// when the deploy went out must continue to page through cleanly —
// those legacy `x:true` entries are dropped silently so the
// consumer's "missing = exhausted" rule applies.
var legacyJson = """{"v":1,"c":{"Agent":{"rk":"0123456789012345678_aaa","x":false},"Security":{"rk":null,"x":true},"Consent":{"rk":"0987654321098765432_bbb","x":false}}}""";
var legacyToken = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(legacyJson));
static string B64(string json) => Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes(json));
var decoded = DecodeMultiContinuation(legacyToken);
Assert.Equal(2, decoded.Count);
Assert.True(decoded.ContainsKey("Agent"));
Assert.True(decoded.ContainsKey("Consent"));
Assert.False(decoded.ContainsKey("Security")); // dropped: was x:true in v1 wire
Assert.Null(DecodeWatermark(null));
Assert.Null(DecodeWatermark(""));
Assert.Null(DecodeWatermark("not-base64-!@#"));
Assert.Null(DecodeWatermark(B64("this is not json")));
Assert.Null(DecodeWatermark(B64("""{"v":2,"c":{"Agent":{"rk":"0123"}}}""")));
Assert.Null(DecodeWatermark(B64("""{"v":3,"rk":"","p":"Agent"}""")));
Assert.Null(DecodeWatermark(B64("""{"v":3,"rk":"0123"}""")));
Assert.Null(DecodeWatermark(B64("""{"v":"3","rk":"0123","p":"Agent"}""")));
}
[Fact]
public void Encode_omits_partition_state_field_to_keep_token_compact()
public async Task A_token_from_an_older_deploy_ends_the_list_without_reading_storage()
{
// Sanity-check the wire compactness gain: 200 partitions of v1 with
// `x:false` per entry produced ~30 KB after base64; v2 should be
// dramatically smaller because we drop the redundant flag and only
// encode active entries.
var many = Enumerable.Range(0, 200)
.ToDictionary(i => $"tenant-{i:D3}", i => new PartitionContinuation($"!1234567890123456789_{i:D8}"));
var store = new Store(new() { ["Agent"] = new() { Rk(1) } });
var v2 = Convert.ToBase64String(System.Text.Encoding.UTF8.GetBytes("""{"v":2,"c":{"Agent":{"rk":null}}}"""));
var encoded = EncodeMultiContinuation(many);
var page = await FetchPageAsync(store.Partitions, 15, v2, store.Reader);
// 200 entries × (~30 chars name + ~30 chars rk + JSON overhead) → JSON
// ~12 KB → base64 ~16 KB. The legacy `x:false` flag adds another
// ~10 bytes per entry (~2 KB total) which we now save. The hard
// assertion below is on absolute size: we must stay well below the
// ~30 KB that broke audit Next clicks.
Assert.True(encoded.Length < 20_000,
$"Encoded continuation should stay under 20 KB even at 200 partitions; got {encoded.Length} bytes");
}
[Fact]
public void Decode_returns_empty_for_malformed_or_empty_continuation()
{
Assert.Empty(DecodeMultiContinuation(null));
Assert.Empty(DecodeMultiContinuation(""));
Assert.Empty(DecodeMultiContinuation("not-base64-!@#"));
Assert.Empty(DecodeMultiContinuation("dGhpcyBpcyBub3QganNvbg==")); // valid base64, not JSON
}
// ────────── End-to-end: drain across pages preserves all items, in order ──
[Fact]
public void Multi_page_drain_returns_all_items_globally_newest_first_no_duplicates()
{
var t0 = new DateTime(2026, 5, 5, 12, 0, 0, DateTimeKind.Utc);
var allItems = new List<(string Part, OpsEventEntry Item, string Rk)>();
var partitions = new[] { "Agent", "Security", "Tenant" };
var minute = 0;
foreach (var part in partitions)
{
for (int i = 0; i < 4; i++)
{
minute++;
var ts = t0.AddMinutes(-minute);
var (rk, _) = Row(ts, $"{part}-{i}");
allItems.Add((part, new OpsEventEntry
{
Category = part,
Id = $"{part}-{i}",
Timestamp = ts,
EventType = "synthetic",
}, rk));
}
}
IReadOnlyList<PartitionFetchResult<OpsEventEntry>> SimulateFetch(
Dictionary<string, PartitionContinuation> continuations, int pageSize, bool isFirstPage)
{
var fetched = new List<PartitionFetchResult<OpsEventEntry>>();
foreach (var part in partitions)
{
// Mirrors the consumer rule: on first page query everything; on
// subsequent pages only partitions still in the continuation.
if (!isFirstPage && !continuations.ContainsKey(part)) continue;
continuations.TryGetValue(part, out var prior);
var lastRk = prior?.LastRowKey;
var rows = allItems
.Where(t => t.Part == part)
.Where(t => lastRk == null || string.Compare(t.Rk, lastRk, StringComparison.Ordinal) > 0)
.OrderBy(t => t.Rk, StringComparer.Ordinal)
.Take(pageSize)
.Select(t => (t.Rk, t.Item))
.ToList();
fetched.Add(new PartitionFetchResult<OpsEventEntry>(part, rows));
}
return fetched;
}
const int pageSize = 5;
var seen = new List<OpsEventEntry>();
var conts = new Dictionary<string, PartitionContinuation>();
var loopGuard = 0;
bool isFirstPage = true;
while (true)
{
var fetched = SimulateFetch(conts, pageSize, isFirstPage);
var (page, nextConts) = MergeAndAdvance(fetched, conts, pageSize, e => e.Timestamp);
seen.AddRange(page);
conts = nextConts;
isFirstPage = false;
// Map only carries active partitions now — empty == drain complete.
if (conts.Count == 0) break;
if (page.Count == 0) break;
if (++loopGuard > 50) Assert.Fail("Drain loop did not terminate");
}
Assert.Equal(12, seen.Count);
Assert.Equal(12, seen.Select(e => e.Id).Distinct().Count());
for (int i = 1; i < seen.Count; i++)
{
Assert.True(seen[i - 1].Timestamp >= seen[i].Timestamp,
$"Out-of-order at index {i}: {seen[i - 1].Id}({seen[i - 1].Timestamp}) before {seen[i].Id}({seen[i].Timestamp})");
}
Assert.Empty(page.Items);
Assert.Null(page.NextRawToken);
Assert.Empty(store.Calls);
}
}
@@ -346,7 +346,7 @@ public class PolicyEnforcementMiddlewareTests
h.AsTenantAdmin(TenantA, upn);
// PUT config/{tenantId} is TenantAdminOrGA (write). Route tenant == JWT tenant ⇒ own-tenant write.
var result = await h.Middleware.DecideAsync("PUT", $"/api/config/{TenantA}", null, AuthedPrincipal(TenantA, upn));
var result = await h.Middleware.DecideAsync("PATCH", $"/api/config/{TenantA}/fields", null, AuthedPrincipal(TenantA, upn));
Assert.True(result.Allowed); // additive: the tenant-admin hat still grants write in the own tenant
}
@@ -361,7 +361,7 @@ public class PolicyEnforcementMiddlewareTests
h.AsGlobalRole(Constants.GlobalRoles.GlobalReader);
// No tenant membership ⇒ no write path.
var result = await h.Middleware.DecideAsync("PUT", $"/api/config/{TenantA}", null, AuthedPrincipal(TenantA, upn));
var result = await h.Middleware.DecideAsync("PATCH", $"/api/config/{TenantA}/fields", null, AuthedPrincipal(TenantA, upn));
Assert.False(result.Allowed);
Assert.Equal(403, result.StatusCode);
@@ -700,7 +700,7 @@ public class PolicyEnforcementMiddlewareTests
h.AsDelegated(TenantB, Constants.DelegatedRoles.DelegatedAdmin); // even a DelegatedAdmin row…
// PUT config/{B} is TenantAdminOrGA (write). Phase 1: delegation never satisfies a write tier.
var result = await h.Middleware.DecideAsync("PUT", $"/api/config/{TenantB}", null, AuthedPrincipal(TenantA, upn));
var result = await h.Middleware.DecideAsync("PATCH", $"/api/config/{TenantB}/fields", null, AuthedPrincipal(TenantA, upn));
Assert.False(result.Allowed);
Assert.Equal(403, result.StatusCode);
@@ -986,9 +986,7 @@ public class PolicyEnforcementMiddlewareTests
Assert.True(result.Allowed); // the delegated read itself is allowed…
// …but no config row was persisted for the MSP's home tenant A (nor any tenant).
h.ConfigRepo.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>()), Times.Never);
h.ConfigRepo.Verify(r => r.SaveTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
h.ConfigRepo.VerifyNoTenantConfigWrite();
}
// ── Phase 2a: delegated single-tenant access to cross-tenant /api/global/* read endpoints ──
@@ -1547,7 +1545,7 @@ public class PolicyEnforcementMiddlewareTests
h.AsTenantAdmin(TenantA, upn);
AsSuspended(h, TenantA); // indefinite suspension (no DisabledUntil)
var result = await h.Middleware.DecideAsync("PUT", $"/api/config/{TenantA}", null, AuthedPrincipal(TenantA, upn));
var result = await h.Middleware.DecideAsync("PATCH", $"/api/config/{TenantA}/fields", null, AuthedPrincipal(TenantA, upn));
Assert.False(result.Allowed);
Assert.Equal("TenantSuspended", result.ErrorCode);
@@ -1576,7 +1574,7 @@ public class PolicyEnforcementMiddlewareTests
AsSuspended(h, TenantB);
var own = await h.Middleware.DecideAsync("GET", "/api/sessions", null, AuthedPrincipal(TenantA, upn));
var cross = await h.Middleware.DecideAsync("PUT", $"/api/config/{TenantB}", null, AuthedPrincipal(TenantA, upn));
var cross = await h.Middleware.DecideAsync("PATCH", $"/api/config/{TenantB}/fields", null, AuthedPrincipal(TenantA, upn));
Assert.True(own.Allowed);
Assert.True(cross.Allowed);
@@ -43,8 +43,10 @@ public sealed class ProConferralServiceTests
var configRepo = new Mock<IConfigRepository>();
configRepo.Setup(c => c.GetTenantConfigurationAsync(It.IsAny<string>()))
.ReturnsAsync((string id) => rows.TryGetValue(id, out var cfg) ? cfg : null);
configRepo.Setup(c => c.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.ReturnsAsync((TenantConfiguration cfg, string? source, string? reason) => { saves.Add((cfg, source, reason)); return true; });
configRepo.Setup(c => c.GetTenantConfigurationWithEtagAsync(It.IsAny<string>()))
.ReturnsAsync((string id) => rows.TryGetValue(id, out var cfg) ? (cfg, "etag") : null);
configRepo.Setup(c => c.TryReplaceTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), "etag", It.IsAny<string?>(), It.IsAny<string?>()))
.ReturnsAsync((TenantConfiguration cfg, string _, string? source, string? reason) => { saves.Add((cfg, source, reason)); return true; });
var adminRepo = new Mock<IAdminRepository>();
adminRepo.Setup(a => a.GetGroupTenantsAsync(Constants.TenantGroupIds.ForHomeTenant(Home))).ReturnsAsync(members.ToList());
@@ -0,0 +1,314 @@
using System.Text.Json;
using Azure.Data.Tables;
using AutopilotMonitor.Functions.Functions.Apps;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// RealmJoin packages as <c>AppInstallSummaries</c> rows (<see cref="AppInstallSources.RealmJoin"/>):
/// the ingest fold of <c>realmjoin_package_started/completed</c>, the scope + package-id row
/// identity, the store-side channel columns, and the channel dimension of the per-app aggregates
/// (an Intune app and a RealmJoin package with the same display name never merge).
/// </summary>
public class RealmJoinPackageInstallRowsTests
{
private static readonly DateTime T0 = new(2026, 9, 17, 8, 29, 26, DateTimeKind.Utc);
private const string SessionId = "11111111-2222-3333-4444-555555555555";
private static Dictionary<string, AppInstallAggregationState> Aggregate(params EnrollmentEvent[] events)
{
// Same comparer as the production classification (EventIngestProcessor.Classification).
var summaries = new Dictionary<string, AppInstallAggregationState>(StringComparer.OrdinalIgnoreCase);
foreach (var evt in events)
EventIngestProcessor.AggregateAppInstallEvent(evt, "tenant", SessionId, summaries);
return summaries;
}
private static AppInstallSummary Row(Dictionary<string, AppInstallAggregationState> summaries, string packageId, string scope = "machine")
=> summaries[EventIngestProcessor.RealmJoinAggregationKey(scope, packageId)].Summary;
private static EnrollmentEvent Started(DateTime at, string packageId = "generic-github-desktop",
string displayName = "GitHub Desktop", string scope = "machine", string? version = "3.6.5.0")
{
var data = new Dictionary<string, object>
{
["packageId"] = packageId,
["displayName"] = displayName,
["scope"] = scope,
};
if (version != null) data["version"] = version;
return new EnrollmentEvent { EventType = Constants.EventTypes.RealmJoinPackageStarted, Timestamp = at, Data = data };
}
private static EnrollmentEvent Completed(DateTime at, bool success = true, int exitCode = 0,
string packageId = "generic-github-desktop", string displayName = "GitHub Desktop", string scope = "machine")
{
var evt = Started(at, packageId, displayName, scope);
evt.EventType = Constants.EventTypes.RealmJoinPackageCompleted;
// Wire form of the agent adapter: strings.
evt.Data!["success"] = success ? "true" : "false";
evt.Data["lastExitCode"] = exitCode.ToString();
return evt;
}
// ── ingest fold: live watcher events ────────────────────────────────────
[Fact]
public void StartedThenCompleted_OneRealmJoinRow_WithMeasuredDuration()
{
var s = Row(Aggregate(Started(T0), Completed(T0.AddSeconds(22))), "generic-github-desktop");
Assert.Equal(AppInstallSources.RealmJoin, s.Source);
Assert.Equal("generic-github-desktop", s.AppId);
Assert.Equal("machine", s.InstallScope);
Assert.Equal("GitHub Desktop", s.AppName);
Assert.Equal("3.6.5.0", s.AppVersion);
Assert.Equal("Succeeded", s.Status);
Assert.Equal("Installed", s.TerminalState);
Assert.Equal(0, s.ExitCode);
Assert.Equal(string.Empty, s.FailureCode);
Assert.Equal(T0, s.StartedAt);
Assert.Equal(T0, s.LastAttemptStartedAt);
Assert.Equal(T0.AddSeconds(22), s.CompletedAt);
Assert.Equal(22, s.DurationSeconds);
Assert.Equal(1, s.InstallPassCount);
Assert.True(MetricsMath.HasMeasuredDuration(s));
}
[Fact]
public void FailedPackage_ExitCodeIsTheFailureCode()
{
var s = Row(Aggregate(Started(T0), Completed(T0.AddSeconds(40), success: false, exitCode: 1603)), "generic-github-desktop");
Assert.Equal("Failed", s.Status);
Assert.Equal("Error", s.TerminalState);
Assert.Equal(1603, s.ExitCode);
Assert.Equal("1603", s.FailureCode);
}
[Fact]
public void StartedAndCompletedInOneRegistryPass_IsUnmeasured()
{
// Older agents report both events in the same watcher pass, milliseconds apart: that is
// no install duration.
var s = Row(Aggregate(Started(T0), Completed(T0.AddMilliseconds(4))), "generic-github-desktop");
Assert.Equal("Succeeded", s.Status);
Assert.Equal(0, s.DurationSeconds);
Assert.False(MetricsMath.HasMeasuredDuration(s));
}
[Fact]
public void MissingDisplayName_UsesPackageId_UntilANameArrives()
{
var summaries = Aggregate(Started(T0, displayName: string.Empty));
Assert.Equal("generic-github-desktop", Row(summaries, "generic-github-desktop").AppName);
EventIngestProcessor.AggregateAppInstallEvent(Completed(T0.AddSeconds(5)), "tenant", SessionId, summaries);
Assert.Equal("GitHub Desktop", Row(summaries, "generic-github-desktop").AppName);
}
[Fact]
public void SamePackageIdInBothScopes_IsTwoRows()
{
var summaries = Aggregate(
Completed(T0, scope: "machine"),
Completed(T0, success: false, exitCode: 1, scope: "user"));
Assert.Equal("Succeeded", Row(summaries, "generic-github-desktop", "machine").Status);
Assert.Equal("Failed", Row(summaries, "generic-github-desktop", "user").Status);
}
[Fact]
public void RealmJoinPackage_AndIntuneApp_WithTheSameName_AreTwoRows()
{
var ime = new EnrollmentEvent
{
EventType = "app_install_completed",
Timestamp = T0,
Data = new Dictionary<string, object> { ["appName"] = "GitHub Desktop", ["state"] = "Installed" },
};
var summaries = Aggregate(ime, Completed(T0.AddSeconds(1)));
Assert.Equal(2, summaries.Count);
Assert.Equal(string.Empty, summaries["GitHub Desktop"].Summary.Source);
Assert.Equal(AppInstallSources.RealmJoin, Row(summaries, "generic-github-desktop").Source);
}
[Fact]
public void EventWithoutPackageId_IsIgnored()
{
var evt = Completed(T0);
evt.Data!.Remove("packageId");
Assert.Empty(Aggregate(evt));
}
// ── store: row identity, channel columns, reconcile ─────────────────────
[Fact]
public void RowKey_ImeUnchanged_RealmJoinKeyedByScopeAndPackageId()
{
Assert.Equal($"{SessionId}_Contoso App_2",
TableStorageService.BuildAppInstallSummaryRowKey(new AppInstallSummary { SessionId = SessionId, AppName = "Contoso App/2" }));
var rj = new AppInstallSummary
{
SessionId = SessionId,
Source = AppInstallSources.RealmJoin,
InstallScope = "user",
AppId = "generic-7zip-usersettings",
AppName = "7-Zip Settings",
};
var key = TableStorageService.BuildAppInstallSummaryRowKey(rj);
Assert.Equal($"{SessionId}_realmjoin|user|generic-7zip-usersettings", key);
// The per-session range scans (ESP stamp, close step, deletion) match on this prefix.
Assert.StartsWith($"{SessionId}_", key, StringComparison.Ordinal);
}
[Fact]
public void Entity_ImeRowHasNoChannelColumns_RealmJoinRowRoundTrips()
{
var ime = TableStorageService.BuildAppInstallSummaryEntity(
new AppInstallSummary { TenantId = "t", SessionId = SessionId, AppName = "App" }, "rk");
Assert.False(ime.ContainsKey("Source"));
Assert.False(ime.ContainsKey("InstallScope"));
var summary = new AppInstallSummary
{
TenantId = "t",
SessionId = SessionId,
AppName = "7-Zip",
Source = AppInstallSources.RealmJoin,
InstallScope = "machine",
AppId = "generic-7zip",
};
var entity = TableStorageService.BuildAppInstallSummaryEntity(summary, "rk");
var sut = new TableStorageService(new Mock<TableServiceClient>().Object, NullLogger<TableStorageService>.Instance);
var mapped = sut.MapToAppInstallSummary(entity);
Assert.Equal(AppInstallSources.RealmJoin, mapped.Source);
Assert.Equal("machine", mapped.InstallScope);
Assert.Equal("generic-7zip", mapped.AppId);
}
[Fact]
public void Reconcile_PackageIdFallbackName_KeepsTheStoredDisplayName()
{
var existing = new TableEntity("t", "rk") { ["AppName"] = "7-Zip", ["StartedAt"] = T0 };
var summary = new AppInstallSummary
{
Source = AppInstallSources.RealmJoin,
AppId = "generic-7zip",
AppName = "generic-7zip",
StartedAt = T0.AddMinutes(30),
};
TableStorageService.ReconcileAppInstallSummaryWithExisting(summary, existing);
Assert.Equal("7-Zip", summary.AppName);
}
[Fact]
public void EspBlockingStamp_NeverTouchesRealmJoinRows()
{
var sets = EspBlockingSets.FromEventData(new Dictionary<string, object>
{
["espTrackedWin32AppIds"] = new[] { "generic-7zip" },
})!;
var rj = new TableEntity("t", "rk") { ["Source"] = AppInstallSources.RealmJoin, ["AppId"] = "generic-7zip" };
var ime = new TableEntity("t", "rk2") { ["AppId"] = "generic-7zip" };
Assert.False(TableStorageService.ShouldStampEspBlocking(rj, sets));
Assert.True(TableStorageService.ShouldStampEspBlocking(ime, sets));
}
// ── per-app aggregates: the channel is part of the app ──────────────────
private static AppInstallSummary Summary(string appName, string source, string status = "Succeeded", int duration = 60)
=> new()
{
TenantId = "t",
SessionId = Guid.NewGuid().ToString(),
AppName = appName,
Source = source,
Status = status,
TerminalState = status == "Failed" ? "Error" : "Installed",
DurationSeconds = duration,
StartedAt = DateTime.UtcNow.AddDays(-1),
LastAttemptStartedAt = DateTime.UtcNow.AddDays(-1),
};
private static List<AppInstallSummary> MixedChannelRows() => new()
{
Summary("GitHub Desktop", string.Empty),
Summary("GitHub Desktop", AppInstallSources.RealmJoin, status: "Failed"),
Summary("GitHub Desktop", AppInstallSources.RealmJoin),
};
[Fact]
public void AppsList_SameNameInTwoChannels_IsTwoItems()
{
var root = TestWire.SerializeToElement(AppsAnalyticsHelper.BuildAppsListResponse(MixedChannelRows(), days: 30));
var apps = root.GetProperty("apps").EnumerateArray().ToList();
Assert.Equal(2, apps.Count);
var ime = apps.Single(a => a.GetProperty("source").GetString() == AppInstallSources.Ime);
var rj = apps.Single(a => a.GetProperty("source").GetString() == AppInstallSources.RealmJoin);
Assert.Equal(1, ime.GetProperty("totalInstalls").GetInt32());
Assert.Equal(2, rj.GetProperty("totalInstalls").GetInt32());
Assert.Equal(1, rj.GetProperty("failed").GetInt32());
}
[Fact]
public async Task Analytics_And_Sessions_FilterByChannel()
{
var sessionRepo = new Mock<ISessionRepository>();
sessionRepo.Setup(r => r.GetSessionAsync(It.IsAny<string>(), It.IsAny<string>())).ReturnsAsync((SessionSummary?)null);
var rj = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
MixedChannelRows(), sessionRepo.Object, "GitHub Desktop", AppInstallSources.RealmJoin, days: 30);
Assert.Equal(AppInstallSources.RealmJoin, rj.Source);
Assert.Equal(2, rj.Summary.TotalInstalls);
Assert.Equal(1, rj.Summary.Failed);
var ime = await AppsAnalyticsHelper.BuildAnalyticsResponseAsync(
MixedChannelRows(), sessionRepo.Object, "GitHub Desktop", AppInstallSources.Ime, days: 30);
Assert.Equal(AppInstallSources.Ime, ime.Source);
Assert.Equal(1, ime.Summary.TotalInstalls);
var sessions = await AppsAnalyticsHelper.BuildSessionsResponseAsync(
MixedChannelRows(), sessionRepo.Object, "GitHub Desktop", AppInstallSources.RealmJoin, days: 30,
statusFilter: "all", modelFilter: null, versionFilter: null, offset: 0, limit: 50);
Assert.Equal(2, sessions.Total);
}
[Fact]
public void AppMetrics_GroupsPerChannel()
{
var payload = MetricsMath.BuildAppMetricsPayload(MixedChannelRows());
Assert.Equal(2, payload.TotalApps);
var failing = Assert.Single(payload.TopFailingApps);
Assert.Equal(AppInstallSources.RealmJoin, failing.Source);
Assert.Equal("GitHub Desktop", failing.AppName);
}
[Theory]
[InlineData(null, true, "ime")]
[InlineData("", true, "ime")]
[InlineData("realmjoin", true, "realmjoin")]
[InlineData(" RealmJoin ", true, "realmjoin")]
[InlineData("office-c2r", false, "office-c2r")]
public void SourceQueryParam(string? raw, bool valid, string expected)
{
Assert.Equal(valid, AppsAnalyticsHelper.TryParseSourceQueryParam(raw, out var source));
Assert.Equal(expected, source);
}
}
@@ -4,9 +4,11 @@ using System.Net.Http;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Functions.Ingest;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Functions.Services.Deletion;
using AutopilotMonitor.Functions.Services.Notifications;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Azure.Data.Tables;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
@@ -94,14 +96,29 @@ public class ReportAgentErrorFunctionTests
// (2026-07-23 hardening): timeline event + cross-session EventType index row +
// AgentEmergencyBreak ops event, once per session. The index upsert is load-bearing:
// StoreEventsBatchAsync alone leaves the event invisible to every search-by-eventType
// surface (found in the 2026-07-22 incident analysis).
// surface (found in the 2026-07-22 incident analysis). A session without a writable
// row gets the ops event only.
// =========================================================================
private static TableEntity Row(SessionStatus? status = SessionStatus.InProgress, string? deletionState = null)
{
var row = new TableEntity(TenantId, SessionId);
if (status.HasValue) row["Status"] = status.Value.ToString();
if (deletionState != null) row["DeletionState"] = deletionState;
return row;
}
private static ReportAgentErrorFunction.SessionRowRead Read(TableEntity? row) => new(row, Failed: false);
private static readonly ReportAgentErrorFunction.SessionRowRead ReadFailed = new(null, Failed: true);
private sealed class Harness
{
public readonly Mock<ISessionRepository> SessionRepo = new();
public readonly List<OpsEventEntry> OpsEvents = new();
public readonly OpsEventService OpsService;
private readonly SessionDeletionGuard _deletionGuard = new(
Mock.Of<ISessionDeletionInventoryReader>(), NullLogger<SessionDeletionGuard>.Instance);
public Harness(List<EnrollmentEvent>? existingEvents = null)
{
@@ -125,9 +142,23 @@ public class ReportAgentErrorFunctionTests
OpsService = new OpsEventService(opsRepo.Object, NullLogger<OpsEventService>.Instance, alertDispatch);
}
/// <summary>Default read: an existing, open session row — the historical path.</summary>
public Task RunAsync(AgentErrorReport report)
=> RunAsync(report, Read(Row()));
public Task RunAsync(AgentErrorReport report, ReportAgentErrorFunction.SessionRowRead? sessionRead)
=> ReportAgentErrorFunction.MaterializeEmergencyBreakArtifactsAsync(
report, TenantId, SessionRepo.Object, OpsService, NullLogger<ReportAgentErrorFunction>.Instance);
report, TenantId, sessionRead, SessionRepo.Object, _deletionGuard, OpsService,
NullLogger<ReportAgentErrorFunction>.Instance);
public void VerifyNothingWrittenIntoTheSession()
{
SessionRepo.Verify(r => r.GetSessionEventsAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<int>()), Times.Never);
SessionRepo.Verify(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()), Times.Never);
SessionRepo.Verify(r => r.UpsertEventTypeIndexBatchAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<IEnumerable<EnrollmentEvent>>()), Times.Never);
}
}
[Fact]
@@ -165,7 +196,7 @@ public class ReportAgentErrorFunctionTests
}
[Fact]
public async Task Non_break_error_types_and_missing_session_id_are_ignored()
public async Task Non_break_error_types_and_ids_without_a_row_read_are_ignored()
{
var h = new Harness();
@@ -173,12 +204,10 @@ public class ReportAgentErrorFunctionTests
other.ErrorType = AgentErrorType.ConfigFetchFailed;
await h.RunAsync(other);
var noSession = Report(Now);
noSession.SessionId = null!;
await h.RunAsync(noSession);
// No read = the ids were not GUIDs (ReadSessionRowAsync) — no session can exist to break.
await h.RunAsync(Report(Now), sessionRead: null);
h.SessionRepo.Verify(r => r.GetSessionEventsAsync(
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<int>()), Times.Never);
h.VerifyNothingWrittenIntoTheSession();
Assert.Empty(h.OpsEvents);
}
@@ -192,10 +221,8 @@ public class ReportAgentErrorFunctionTests
public async Task Break_on_already_terminal_session_is_info_late_cleanup(SessionStatus status)
{
var h = new Harness();
h.SessionRepo.Setup(r => r.GetSessionAsync(TenantId, SessionId))
.ReturnsAsync(new SessionSummary { SessionId = SessionId, TenantId = TenantId, Status = status });
await h.RunAsync(Report(Now));
await h.RunAsync(Report(Now), Read(Row(status)));
var ops = Assert.Single(h.OpsEvents);
Assert.Equal(OpsEventSeverity.Info, ops.Severity);
@@ -216,44 +243,220 @@ public class ReportAgentErrorFunctionTests
public async Task Break_on_open_session_stays_warning(SessionStatus status)
{
var h = new Harness();
h.SessionRepo.Setup(r => r.GetSessionAsync(TenantId, SessionId))
.ReturnsAsync(new SessionSummary { SessionId = SessionId, TenantId = TenantId, Status = status });
await h.RunAsync(Report(Now));
await h.RunAsync(Report(Now), Read(Row(status)));
var ops = Assert.Single(h.OpsEvents);
Assert.Equal(OpsEventSeverity.Warning, ops.Severity);
Assert.Contains($"still {status}", ops.Message);
Assert.Contains("\"lateCleanup\":false", ops.Details);
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()), Times.Once);
}
// 2026-09-28 sweep: every "unknown" Warning named a session that never registered. The break is
// the only thing the backend ever hears of it — nothing to close, and a timeline event would be
// an orphan the maintenance sweep deletes a day later (with a Warning of its own).
[Fact]
public async Task Break_on_session_without_row_is_info_and_writes_nothing_into_the_session()
{
var h = new Harness();
await h.RunAsync(Report(Now), Read(null));
var ops = Assert.Single(h.OpsEvents);
Assert.Equal(OpsEventSeverity.Info, ops.Severity);
Assert.Contains("no session row", ops.Message);
Assert.Contains("\"sessionStatusAtBreak\":\"missing\"", ops.Details);
Assert.Contains("\"lateCleanup\":false", ops.Details);
h.VerifyNothingWrittenIntoTheSession();
}
[Theory]
[InlineData(AutopilotMonitor.Shared.Models.Deletion.SessionDeletionState.Preparing)]
[InlineData(AutopilotMonitor.Shared.Models.Deletion.SessionDeletionState.Queued)]
[InlineData(AutopilotMonitor.Shared.Models.Deletion.SessionDeletionState.Running)]
[InlineData(AutopilotMonitor.Shared.Models.Deletion.SessionDeletionState.Poisoned)]
public async Task Break_on_session_under_cascade_delete_writes_nothing_past_the_lock(string deletionState)
{
var h = new Harness();
await h.RunAsync(Report(Now), Read(Row(SessionStatus.InProgress, deletionState)));
var ops = Assert.Single(h.OpsEvents);
Assert.Equal(OpsEventSeverity.Info, ops.Severity);
Assert.Contains("\"sessionStatusAtBreak\":\"deleting\"", ops.Details);
h.VerifyNothingWrittenIntoTheSession();
}
[Fact]
public async Task Unknown_or_unreadable_session_status_keeps_warning()
public async Task Unreadable_row_keeps_the_historical_warning_and_write()
{
// No row (Harness default: GetSessionAsync → null) — the historical Warning semantics.
// Existence unknown — fail safe: the break may be the only signal for an open session.
var h = new Harness();
await h.RunAsync(Report(Now));
await h.RunAsync(Report(Now), ReadFailed);
var ops = Assert.Single(h.OpsEvents);
Assert.Equal(OpsEventSeverity.Warning, ops.Severity);
Assert.Contains("\"sessionStatusAtBreak\":\"unknown\"", ops.Details);
// Row read throws — the break must still be recorded, still as Warning.
var h2 = new Harness();
h2.SessionRepo.Setup(r => r.GetSessionAsync(TenantId, SessionId))
.ThrowsAsync(new InvalidOperationException("table down"));
await h2.RunAsync(Report(Now));
Assert.Equal(OpsEventSeverity.Warning, Assert.Single(h2.OpsEvents).Severity);
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(It.IsAny<List<EnrollmentEvent>>()), Times.Once);
}
[Fact]
public void ClassifyEmergencyBreak_is_terminal_set_only()
public async Task Row_without_parseable_status_stays_warning()
{
Assert.Equal((OpsEventSeverity.Info, true), ReportAgentErrorFunction.ClassifyEmergencyBreak(SessionStatus.Succeeded));
Assert.Equal((OpsEventSeverity.Info, true), ReportAgentErrorFunction.ClassifyEmergencyBreak(SessionStatus.Failed));
Assert.Equal((OpsEventSeverity.Info, true), ReportAgentErrorFunction.ClassifyEmergencyBreak(SessionStatus.Incomplete));
Assert.Equal((OpsEventSeverity.Warning, false), ReportAgentErrorFunction.ClassifyEmergencyBreak(SessionStatus.AwaitingUser));
Assert.Equal((OpsEventSeverity.Warning, false), ReportAgentErrorFunction.ClassifyEmergencyBreak(SessionStatus.Unknown));
Assert.Equal((OpsEventSeverity.Warning, false), ReportAgentErrorFunction.ClassifyEmergencyBreak(null));
var h = new Harness();
await h.RunAsync(Report(Now), Read(Row(status: null)));
var ops = Assert.Single(h.OpsEvents);
Assert.Equal(OpsEventSeverity.Warning, ops.Severity);
Assert.Contains("\"sessionStatusAtBreak\":\"unknown\"", ops.Details);
}
// The agent-measured age separates a break just past the cap from a device that slept for
// weeks; the backend cannot derive it (no row, or a row registered seconds before the break).
[Fact]
public async Task Reported_session_age_lands_in_ops_details_and_timeline_data()
{
var h = new Harness();
var report = Report(Now);
report.SessionAgeHours = 478.2;
await h.RunAsync(report);
Assert.Contains("\"sessionAgeHours\":478.2", Assert.Single(h.OpsEvents).Details);
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(
It.Is<List<EnrollmentEvent>>(e => (double)e[0].Data["sessionAgeHours"] == 478.2)), Times.Once);
}
// A session that never registered leaves the backend nothing but this report; the agent's
// record of its failed starts is the only account of why.
[Fact]
public async Task Prior_registration_failure_lands_in_ops_details_even_without_a_row()
{
var h = new Harness();
var report = Report(Now);
report.PriorRegistrationFailure = ReportAgentErrorFunction.SanitizeRegistrationFailure(Failure());
await h.RunAsync(report, Read(null));
var details = Assert.Single(h.OpsEvents).Details;
Assert.Contains("\"priorRegistrationFailure\":{\"failedRuns\":3", details);
Assert.Contains("\"networkLinkUpAtEnd\":false", details);
Assert.Contains("\"attemptWindowSeconds\":1204.5", details);
Assert.Contains("\"configFetchOutcome\":\"UsedDefaults\"", details);
h.VerifyNothingWrittenIntoTheSession();
}
[Fact]
public async Task Prior_registration_failure_lands_in_the_timeline_data_of_a_registered_session()
{
var h = new Harness();
var report = Report(Now);
report.PriorRegistrationFailure = ReportAgentErrorFunction.SanitizeRegistrationFailure(Failure());
await h.RunAsync(report);
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(
It.Is<List<EnrollmentEvent>>(e =>
((Dictionary<string, object?>)e[0].Data["priorRegistrationFailure"])["failedRuns"]!.Equals(3))), Times.Once);
}
[Fact]
public async Task Agent_without_registration_record_leaves_it_null_and_out_of_the_timeline()
{
var h = new Harness();
await h.RunAsync(Report(Now));
Assert.Contains("\"priorRegistrationFailure\":null", Assert.Single(h.OpsEvents).Details);
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(
It.Is<List<EnrollmentEvent>>(e => !e[0].Data.ContainsKey("priorRegistrationFailure"))), Times.Once);
}
[Fact]
public void SanitizeRegistrationFailure_bounds_device_input()
{
var raw = Failure();
raw.LastError = new string('x', 5000);
raw.Outcome = new string('o', 500);
raw.ConfigFetchOutcome = new string('c', 500);
raw.AttemptWindowSeconds = -3;
var clean = ReportAgentErrorFunction.SanitizeRegistrationFailure(raw)!;
Assert.Equal(ReportAgentErrorFunction.MaxRegistrationErrorLength, clean.LastError!.Length);
Assert.Equal(ReportAgentErrorFunction.MaxRegistrationTokenLength, clean.Outcome.Length);
Assert.Equal(ReportAgentErrorFunction.MaxRegistrationTokenLength, clean.ConfigFetchOutcome!.Length);
Assert.Equal(0, clean.AttemptWindowSeconds);
Assert.Equal(3, clean.FailedRuns);
Assert.NotSame(raw, clean);
}
[Theory]
[InlineData(0)]
[InlineData(-1)]
public void SanitizeRegistrationFailure_drops_a_record_without_failed_runs(int failedRuns)
{
var raw = Failure();
raw.FailedRuns = failedRuns;
Assert.Null(ReportAgentErrorFunction.SanitizeRegistrationFailure(raw));
Assert.Null(ReportAgentErrorFunction.SanitizeRegistrationFailure(null));
}
private static RegistrationFailureSummary Failure() => new()
{
FailedRuns = 3,
FirstFailedAtUtc = Now.AddDays(-20),
LastFailedAtUtc = Now.AddDays(-19),
Outcome = "Failed",
LastError = "No such host is known.",
NetworkLinkUpAtStart = false,
NetworkLinkUpAtEnd = false,
AttemptWindowSeconds = 1204.5,
ConfigFetchOutcome = "UsedDefaults",
};
[Fact]
public async Task Agent_without_session_age_leaves_it_null_and_out_of_the_timeline()
{
var h = new Harness();
await h.RunAsync(Report(Now));
Assert.Contains("\"sessionAgeHours\":null", Assert.Single(h.OpsEvents).Details);
h.SessionRepo.Verify(r => r.StoreEventsBatchAsync(
It.Is<List<EnrollmentEvent>>(e => !e[0].Data.ContainsKey("sessionAgeHours"))), Times.Once);
}
[Fact]
public void ClassifyEmergencyBreak_follows_row_state()
{
static ReportAgentErrorFunction.EmergencyBreakVerdict Classify(TableEntity? row, bool failed = false, bool locked = false)
=> ReportAgentErrorFunction.ClassifyEmergencyBreak(row, failed, locked);
foreach (var terminal in new[] { SessionStatus.Succeeded, SessionStatus.Failed, SessionStatus.Incomplete })
{
var v = Classify(Row(terminal));
Assert.Equal((OpsEventSeverity.Info, true, true, terminal.ToString()), (v.Severity, v.LateCleanup, v.Materialize, v.SessionStatusAtBreak));
}
foreach (var open in new[] { SessionStatus.InProgress, SessionStatus.Pending, SessionStatus.AwaitingUser, SessionStatus.Stalled, SessionStatus.Unknown })
{
var v = Classify(Row(open));
Assert.Equal((OpsEventSeverity.Warning, false, true, open.ToString()), (v.Severity, v.LateCleanup, v.Materialize, v.SessionStatusAtBreak));
}
var missing = Classify(null);
Assert.Equal((OpsEventSeverity.Info, false, false, "missing"), (missing.Severity, missing.LateCleanup, missing.Materialize, missing.SessionStatusAtBreak));
var deleting = Classify(Row(SessionStatus.Succeeded), locked: true);
Assert.Equal((OpsEventSeverity.Info, false, false, "deleting"), (deleting.Severity, deleting.LateCleanup, deleting.Materialize, deleting.SessionStatusAtBreak));
var unreadable = Classify(null, failed: true);
Assert.Equal((OpsEventSeverity.Warning, false, true, "unknown"), (unreadable.Severity, unreadable.LateCleanup, unreadable.Materialize, unreadable.SessionStatusAtBreak));
}
[Fact]
@@ -373,4 +373,106 @@ public class RevokeEnforcementTests
proConferral.Verify(p => p.RecordLossAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>()), Times.Never);
}
// --- Customer-visible audit rows name the group as the customer sees it, never internally ---
[Theory]
[InlineData(null, "Platform support")]
[InlineData("Contoso Service Desk", "Contoso Service Desk")]
public async Task UnassignCore_AuditNamesTheGroupByItsCustomerName(string? label, string expected)
{
var (fn, repo, audit, _) = BuildGroupFn();
repo.Setup(r => r.GetTenantGroupAsync(GroupId)).ReturnsAsync(new TenantGroup
{
GroupId = GroupId,
Name = "Internal MSP customers north",
CustomerLabel = label,
TenantIds = new List<string> { TenantA },
AssigneeCount = 1,
});
repo.Setup(r => r.GetGroupAssignmentsForUpnAsync(UpnLower))
.ReturnsAsync(new List<TenantGroupAssignment> { new() { Upn = UpnLower, GroupId = GroupId } });
repo.Setup(r => r.UnassignGroupAsync(UpnLower, GroupId)).ReturnsAsync(true);
await fn.UnassignCoreAsync(GroupId, Upn, "ga@vendor.example");
audit.Verify(a => a.LogAuditEntryAsync(
TenantA, "DELETE", "DelegatedGroupAccess", UpnLower, "ga@vendor.example",
It.Is<Dictionary<string, string>?>(d => d != null && d["Group"] == expected && d["GroupId"] == GroupId)), Times.Once);
}
// --- Group update: name and/or customer label (PATCH only what changes) ---
[Theory]
[InlineData(null, null, "name or customerLabel is required")]
[InlineData(" ", null, "name must not be empty")]
[InlineData("Renamed", null, null)]
[InlineData(null, "", null)] // clears the label
[InlineData(null, "Contoso Service Desk", null)]
public void ValidateUpdateRequest_Shapes(string? name, string? label, string? expectedError)
=> Assert.Equal(expectedError,
TenantGroupManagementFunction.ValidateUpdateRequest(new UpdateTenantGroupRequest { Name = name, CustomerLabel = label }));
[Fact]
public void ValidateUpdateRequest_LabelLongerThanCap_Rejected_TrimmedLengthCounts()
{
var atCap = " " + new string('x', DelegationSelfService.CustomerLabelMaxLength) + " ";
Assert.Null(TenantGroupManagementFunction.ValidateUpdateRequest(new UpdateTenantGroupRequest { CustomerLabel = atCap }));
var tooLong = new string('x', DelegationSelfService.CustomerLabelMaxLength + 1);
Assert.NotNull(TenantGroupManagementFunction.ValidateUpdateRequest(new UpdateTenantGroupRequest { CustomerLabel = tooLong }));
}
[Fact]
public async Task UpdateGroupCore_Label_OnOperatorGroup_WritesOnlyTheLabel_NoAudit()
{
var (fn, repo, audit, _) = BuildGroupFn();
repo.Setup(r => r.GetTenantGroupAsync(GroupId)).ReturnsAsync(new TenantGroup { GroupId = GroupId, Name = "MSP Customers" });
repo.Setup(r => r.UpdateTenantGroupAsync(GroupId, null, "Contoso Service Desk")).ReturnsAsync(true);
var outcome = await fn.UpdateGroupCoreAsync(GroupId, new UpdateTenantGroupRequest { CustomerLabel = "Contoso Service Desk" }, "ga@vendor.example");
Assert.Equal(TenantGroupManagementFunction.GroupUpdateOutcome.Updated, outcome);
repo.Verify(r => r.UpdateTenantGroupAsync(GroupId, null, "Contoso Service Desk"), Times.Once);
VerifyNoAudit(audit); // naming is not an access change
}
[Fact]
public async Task UpdateGroupCore_Label_OnOwnedGroup_Refused_NothingWritten()
{
var (fn, repo, _, _) = BuildGroupFn();
var ownedGroupId = Constants.TenantGroupIds.ForHomeTenant(OwnerTenant);
repo.Setup(r => r.GetTenantGroupAsync(ownedGroupId)).ReturnsAsync(new TenantGroup { GroupId = ownedGroupId, Name = "partner.example", OwnerTenantId = OwnerTenant });
var outcome = await fn.UpdateGroupCoreAsync(ownedGroupId, new UpdateTenantGroupRequest { CustomerLabel = "Anything" }, "ga@vendor.example");
Assert.Equal(TenantGroupManagementFunction.GroupUpdateOutcome.LabelOnOwnedGroup, outcome);
repo.Verify(r => r.UpdateTenantGroupAsync(It.IsAny<string>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
}
[Fact]
public async Task UpdateGroupCore_Label_UnknownGroup_NotFound()
{
var (fn, repo, _, _) = BuildGroupFn(); // default: no group
var outcome = await fn.UpdateGroupCoreAsync(GroupId, new UpdateTenantGroupRequest { CustomerLabel = "Anything" }, "ga@vendor.example");
Assert.Equal(TenantGroupManagementFunction.GroupUpdateOutcome.NotFound, outcome);
repo.Verify(r => r.UpdateTenantGroupAsync(It.IsAny<string>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
}
[Fact]
public async Task UpdateGroupCore_RenameOnly_LeavesTheLabelAlone_MissIsNotFound()
{
var (fn, repo, _, _) = BuildGroupFn();
repo.Setup(r => r.UpdateTenantGroupAsync(GroupId, "Renamed", null)).ReturnsAsync(true);
Assert.Equal(TenantGroupManagementFunction.GroupUpdateOutcome.Updated,
await fn.UpdateGroupCoreAsync(GroupId, new UpdateTenantGroupRequest { Name = "Renamed" }, "ga@vendor.example"));
repo.Verify(r => r.UpdateTenantGroupAsync(GroupId, "Renamed", null), Times.Once);
repo.Setup(r => r.UpdateTenantGroupAsync("tpl-missing", "Renamed", null)).ReturnsAsync(false);
Assert.Equal(TenantGroupManagementFunction.GroupUpdateOutcome.NotFound,
await fn.UpdateGroupCoreAsync("tpl-missing", new UpdateTenantGroupRequest { Name = "Renamed" }, "ga@vendor.example"));
}
}
@@ -0,0 +1,63 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// SESSION-OWNER-BINDING call-site inventory. The functions are too dependency-heavy to boot in a
/// unit test, and the refusal is a one-line branch at each site — so the guard is structural:
/// every file that observes the binding must also act on <c>Decision.Rejected</c>, and the set of
/// sites is pinned so a new session-scoped agent endpoint is added here on purpose (and decides
/// how it refuses) instead of silently observing without enforcing.
/// </summary>
public class SessionOwnerBindingCallSiteGuardTests
{
private const string ObserveCall = "_ownerBinding.Observe(";
private static readonly string[] ExpectedSites =
{
"Functions/Diagnostics/GetDiagnosticsUploadUrlFunction.cs",
"Functions/Ingest/IngestTelemetryFunction.cs",
"Functions/Ingest/ReportAgentErrorFunction.cs",
"Functions/Sessions/RegisterSessionFunction.cs",
};
[Fact]
public void Every_observing_site_acts_on_the_refusal()
{
var functionsRoot = Path.Combine(FindRepoRoot(), "src", "Backend", "AutopilotMonitor.Functions");
var sites = new Dictionary<string, string>(StringComparer.Ordinal);
foreach (var file in Directory.EnumerateFiles(functionsRoot, "*.cs", SearchOption.AllDirectories))
{
var relative = Path.GetRelativePath(functionsRoot, file).Replace(Path.DirectorySeparatorChar, '/');
if (relative.StartsWith("bin/", StringComparison.OrdinalIgnoreCase) ||
relative.StartsWith("obj/", StringComparison.OrdinalIgnoreCase))
continue;
var text = File.ReadAllText(file);
if (text.Contains(ObserveCall, StringComparison.Ordinal))
sites[relative] = text;
}
Assert.Equal(ExpectedSites, sites.Keys.OrderBy(k => k, StringComparer.Ordinal).ToArray());
var missing = sites.Where(kv => !kv.Value.Contains(".Rejected", StringComparison.Ordinal)).Select(kv => kv.Key).ToList();
Assert.True(missing.Count == 0,
"These sites observe the session owner binding but never act on Decision.Rejected:\n " + string.Join("\n ", missing));
}
private static string FindRepoRoot()
{
var dir = new DirectoryInfo(AppContext.BaseDirectory);
while (dir != null && !File.Exists(Path.Combine(dir.FullName, "AutopilotMonitor.sln")))
{
dir = dir.Parent;
}
Assert.NotNull(dir);
return dir!.FullName;
}
}
@@ -19,9 +19,9 @@ using Xunit;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// SESSION-OWNER-BINDING-SHADOW carriers: the outcome lands on the request row for every
/// request (denominator), non-Match outcomes produce a Warning, would-reject outcomes raise one
/// throttled ops event, stamping is fail-soft, and nothing is ever refused.
/// SESSION-OWNER-BINDING carriers: the outcome lands on the request row for every request
/// (denominator), non-Match outcomes produce a Warning, rejected outcomes raise one throttled
/// ops event, stamping is fail-soft, and an evaluation defect fails open.
/// </summary>
public class SessionOwnerBindingObserverTests
{
@@ -105,7 +105,7 @@ public class SessionOwnerBindingObserverTests
}
[Fact]
public async Task Would_reject_outcome_warns_and_raises_one_ops_event_per_session_and_outcome()
public async Task Rejected_outcome_warns_and_raises_one_ops_event_per_session_and_outcome()
{
var rig = new Rig();
var (req, items) = Request("2.0.1500");
@@ -115,12 +115,12 @@ public class SessionOwnerBindingObserverTests
var second = rig.Sut.Observe(req, Tenant, Session, row, Cert(Thumb2, "SN-ATTACKER"), "agent/telemetry");
Assert.Equal(SessionOwnershipPolicy.Outcome.MismatchCert, first.Outcome);
Assert.True(first.WouldReject);
Assert.True(first.Rejected);
Assert.Equal(SessionOwnershipPolicy.Outcome.MismatchCert, items[SessionOwnershipPolicy.RequestItemKey]);
var warnings = rig.Logger.Entries.Where(e => e.Level == LogLevel.Warning).ToList();
Assert.Equal(2, warnings.Count);
Assert.All(warnings, w => Assert.Contains("AgentSessionOwnerBinding outcome=MismatchCert enforced=False wouldReject=True", w.Message));
Assert.All(warnings, w => Assert.Contains("AgentSessionOwnerBinding outcome=MismatchCert enforced=True wouldReject=True", w.Message));
Assert.Contains("serialMatch=False", warnings[0].Message);
Assert.Contains("ver=2.0.1500", warnings[0].Message);
@@ -130,11 +130,11 @@ public class SessionOwnerBindingObserverTests
Assert.Equal(OpsEventCategory.Security, evt.Category);
Assert.Equal(Tenant, evt.TenantId);
Assert.Contains("MismatchCert", evt.Message);
Assert.Contains("shadow", evt.Message);
Assert.Contains("refused", evt.Message);
Assert.DoesNotContain(Thumb1, evt.Message);
Assert.DoesNotContain(Thumb2, evt.Message);
Assert.Contains("\"serialMatch\":false", evt.Details);
Assert.Contains("\"enforced\":false", evt.Details);
Assert.Contains("\"enforced\":true", evt.Details);
_ = second;
}
@@ -154,6 +154,22 @@ public class SessionOwnerBindingObserverTests
Assert.Empty(rig.Ops);
}
[Fact]
public void Evaluation_defect_fails_open()
{
var rig = new Rig();
var (req, _) = Request();
// A null validation throws inside the policy — our defect, never evidence of a foreign device.
var d = rig.Sut.Observe(req, Tenant, Session, OwnedRow(Thumb1), null!, "agent/telemetry");
Assert.Equal(SessionOwnershipPolicy.Outcome.Match, d.Outcome);
Assert.False(d.Rejected);
Assert.Null(d.OwnerToStamp);
Assert.Contains(rig.Logger.Entries, e => e.Level == LogLevel.Warning && e.Message.Contains("request allowed"));
Assert.Empty(rig.Ops);
}
[Fact]
public async Task Stamp_writes_the_owner_and_swallows_storage_failures()
{
@@ -11,8 +11,8 @@ namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// SESSION-OWNER-BINDING rule matrix. The policy is pure, so every lifecycle shape that must keep
/// working (restart, WhiteGlove Part 2, bootstrap→cert handoff, certificate re-issue, rows from
/// before the binding) and every shape that must be visible as a would-reject (foreign cert,
/// foreign bootstrap code, bootstrap on a cert-owned row, re-enroll without wipe) is pinned here.
/// before the binding) and every shape that is refused (foreign cert, foreign bootstrap code,
/// bootstrap on a cert-owned row, re-enroll without wipe) is pinned here.
/// </summary>
public class SessionOwnershipPolicyTests
{
@@ -65,7 +65,7 @@ public class SessionOwnershipPolicyTests
Assert.Equal(Thumb1, d.OwnerToStamp.Thumbprint);
Assert.Equal(Dev1, d.OwnerToStamp.DeviceId);
Assert.Equal("SN-1", d.OwnerToStamp.Serial);
Assert.False(d.WouldReject);
Assert.False(d.Rejected);
}
[Fact]
@@ -75,7 +75,7 @@ public class SessionOwnershipPolicyTests
var d = SessionOwnershipPolicy.Evaluate(Row(owner: CertOwner(Thumb1, Dev1)), v, Now);
Assert.Equal(SessionOwnershipPolicy.Outcome.CallerUnidentified, d.Outcome);
Assert.Null(d.OwnerToStamp);
Assert.False(d.WouldReject);
Assert.False(d.Rejected);
}
// ── legacy rows (pre-binding) ────────────────────────────────────────────
@@ -87,7 +87,7 @@ public class SessionOwnershipPolicyTests
Assert.Equal(SessionOwnershipPolicy.Outcome.ClaimLegacy, d.Outcome);
Assert.NotNull(d.OwnerToStamp);
Assert.True(d.SerialMatch);
Assert.False(d.WouldReject);
Assert.False(d.Rejected);
}
[Fact]
@@ -96,7 +96,7 @@ public class SessionOwnershipPolicyTests
var d = SessionOwnershipPolicy.Evaluate(Row(serial: "OTHER"), Cert(Thumb1, Dev1), Now);
Assert.Equal(SessionOwnershipPolicy.Outcome.LegacySerialMismatch, d.Outcome);
Assert.Null(d.OwnerToStamp);
Assert.True(d.WouldReject);
Assert.True(d.Rejected);
}
[Fact]
@@ -123,7 +123,7 @@ public class SessionOwnershipPolicyTests
Assert.Equal(SessionOwnershipPolicy.Outcome.RebindCertRotation, d.Outcome);
Assert.Equal(Thumb2, d.OwnerToStamp!.Thumbprint);
Assert.Equal(Dev1, d.OwnerToStamp.DeviceId);
Assert.False(d.WouldReject);
Assert.False(d.Rejected);
}
[Fact]
@@ -133,7 +133,7 @@ public class SessionOwnershipPolicyTests
Assert.Equal(SessionOwnershipPolicy.Outcome.MismatchCert, d.Outcome);
Assert.False(d.SerialMatch);
Assert.Null(d.OwnerToStamp);
Assert.True(d.WouldReject);
Assert.True(d.Rejected);
}
[Fact]
@@ -143,7 +143,7 @@ public class SessionOwnershipPolicyTests
var d = SessionOwnershipPolicy.Evaluate(Row(owner: CertOwner(Thumb1, Dev1)), Cert(Thumb2, Dev2, "SN-1"), Now);
Assert.Equal(SessionOwnershipPolicy.Outcome.MismatchCert, d.Outcome);
Assert.True(d.SerialMatch);
Assert.True(d.WouldReject);
Assert.True(d.Rejected);
}
[Fact]
@@ -160,7 +160,7 @@ public class SessionOwnershipPolicyTests
Assert.Equal(SessionOwnershipPolicy.Outcome.RebindBootstrapHandoff, d.Outcome);
Assert.Equal(SessionOwner.Kinds.Cert, d.OwnerToStamp!.Kind);
Assert.Null(d.OwnerToStamp.BootstrapCode);
Assert.False(d.WouldReject);
Assert.False(d.Rejected);
}
[Fact]
@@ -168,7 +168,7 @@ public class SessionOwnershipPolicyTests
{
var d = SessionOwnershipPolicy.Evaluate(Row(owner: BootstrapOwner("ABC123")), Cert(Thumb1, Dev1, "OTHER"), Now);
Assert.Equal(SessionOwnershipPolicy.Outcome.MismatchBootstrapOwned, d.Outcome);
Assert.True(d.WouldReject);
Assert.True(d.Rejected);
}
// ── bootstrap caller ─────────────────────────────────────────────────────
@@ -187,7 +187,7 @@ public class SessionOwnershipPolicyTests
{
var d = SessionOwnershipPolicy.Evaluate(Row(owner: BootstrapOwner("ABC123")), Bootstrap(code, serial), Now);
Assert.Equal(SessionOwnershipPolicy.Outcome.MismatchBootstrap, d.Outcome);
Assert.True(d.WouldReject);
Assert.True(d.Rejected);
}
[Fact]
@@ -195,7 +195,7 @@ public class SessionOwnershipPolicyTests
{
var d = SessionOwnershipPolicy.Evaluate(Row(owner: CertOwner(Thumb1, Dev1)), Bootstrap("ABC123"), Now);
Assert.Equal(SessionOwnershipPolicy.Outcome.DowngradeToBootstrap, d.Outcome);
Assert.True(d.WouldReject);
Assert.True(d.Rejected);
}
[Fact]
@@ -244,7 +244,7 @@ public class SessionOwnershipPolicyTests
[InlineData(SessionOwnershipPolicy.Outcome.RebindBootstrapHandoff)]
[InlineData(SessionOwnershipPolicy.Outcome.CallerUnidentified)]
public void Lifecycle_outcomes_are_deliberately_tolerated(string outcome)
=> Assert.False(SessionOwnershipPolicy.WouldRejectUnderEnforcement(outcome));
=> Assert.False(SessionOwnershipPolicy.Rejects(outcome));
[Theory]
[InlineData(SessionOwnershipPolicy.Outcome.LegacySerialMismatch)]
@@ -252,16 +252,22 @@ public class SessionOwnershipPolicyTests
[InlineData(SessionOwnershipPolicy.Outcome.MismatchCert)]
[InlineData(SessionOwnershipPolicy.Outcome.MismatchBootstrap)]
[InlineData(SessionOwnershipPolicy.Outcome.DowngradeToBootstrap)]
public void Foreign_identity_outcomes_would_reject(string outcome)
=> Assert.True(SessionOwnershipPolicy.WouldRejectUnderEnforcement(outcome));
public void Foreign_identity_outcomes_are_rejected(string outcome)
=> Assert.True(SessionOwnershipPolicy.Rejects(outcome));
[Fact]
public void Stage1_has_no_Rejects_rule()
public void Every_outcome_is_classified_and_the_telemetry_field_reports_the_rule()
{
// Enforcement is a deliberate, visible code change (stage 2) — not something that can
// slip in through a refactor. Delete this test in the change that adds Rejects.
var rejects = typeof(SessionOwnershipPolicy).GetMethods(BindingFlags.Public | BindingFlags.Static)
.Where(m => m.Name == "Rejects").ToArray();
Assert.Empty(rejects);
// A new Outcome constant must land in one of the two theories above on purpose, and the
// shadow-era wouldReject field (KQL keeps querying it) must say exactly what Rejects does.
var outcomes = typeof(SessionOwnershipPolicy.Outcome)
.GetFields(BindingFlags.Public | BindingFlags.Static)
.Where(f => f.IsLiteral)
.Select(f => (string)f.GetRawConstantValue()!)
.ToArray();
Assert.Equal(11, outcomes.Length);
Assert.Equal(5, outcomes.Count(SessionOwnershipPolicy.Rejects));
Assert.All(outcomes, o => Assert.Equal(SessionOwnershipPolicy.Rejects(o), SessionOwnershipPolicy.WouldRejectUnderEnforcement(o)));
}
}
@@ -88,6 +88,8 @@ public sealed class SharedManifestParityTests
["webhookProviderTypes"] = EnumMap<WebhookProviderType>(),
["annotationLanes"] = AnnotationLanes.All,
["annotationVerdicts"] = AnnotationVerdicts.All,
// Install channels of the app rows (AppInstallSummary.Source and the per-app DTOs).
["appInstallSources"] = AppInstallSources.All,
// Community rule submissions: kinds, effective statuses (incl. the derived "published"),
// attribution modes and reviewer decisions — the MCP tool schemas enumerate these.
["ruleSubmissionKinds"] = RuleSubmissionKinds.All,
@@ -1,8 +1,14 @@
using System.Net;
using System.Text;
using System.Text.RegularExpressions;
using AutopilotMonitor.Functions.Security;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// Format checks and the address classifier. The connection gate itself is covered on real sockets
/// in <see cref="WebhookConnectGateTests"/>.
/// </summary>
public class SsrfGuardTests
{
// ── ValidateWebhookUrlFormat: valid URLs ──
@@ -145,37 +151,124 @@ public class SsrfGuardTests
Assert.False(SsrfGuard.IsBlockedAddress(IPAddress.Parse(ip)));
}
// ── ValidateDestinationAsync: integration ──
// ── IsBlockedAddress: IPv6 beyond the classic ranges ──
[Fact]
public async Task ValidateDestination_Localhost_ThrowsSsrfException()
[Theory]
[InlineData("::")] // unspecified
[InlineData("::ffff:8.8.8.8")] // IPv4-mapped, even of a public address
[InlineData("::8.8.8.8")] // IPv4-compatible (deprecated)
[InlineData("64:ff9b::a00:1")] // NAT64 of 10.0.0.1
[InlineData("2002:a00:1::1")] // 6to4 of 10.0.0.1
[InlineData("2001::1")] // Teredo
[InlineData("2001:db8::1")] // documentation
[InlineData("fec0::1")] // deprecated site-local
[InlineData("4000::1")] // outside global unicast
[InlineData("e000::1")] // outside global unicast
public void IsBlocked_Ipv6OutsidePublicUnicast_ReturnsTrue(string ip)
{
var ex = await Assert.ThrowsAsync<SsrfException>(
() => SsrfGuard.ValidateDestinationAsync("https://localhost/webhook"));
Assert.Contains("private or reserved", ex.Message);
Assert.True(SsrfGuard.IsBlockedAddress(IPAddress.Parse(ip)));
}
[Fact]
public async Task ValidateDestination_HttpScheme_ThrowsSsrfException()
[Theory]
[InlineData("fe80::1%5")]
[InlineData("2606:4700:4700::1111%5")]
public void IsBlocked_Ipv6WithZoneId_ReturnsTrue(string ip)
{
var ex = await Assert.ThrowsAsync<SsrfException>(
() => SsrfGuard.ValidateDestinationAsync("http://example.com/webhook"));
Assert.Contains("HTTPS", ex.Message);
Assert.True(SsrfGuard.IsBlockedAddress(IPAddress.Parse(ip)));
}
[Fact]
public async Task ValidateDestination_InvalidUrl_ThrowsSsrfException()
[Theory]
[InlineData("2606:4700:4700::1111")] // Cloudflare
[InlineData("2a00:1450:4001::1")] // Google
[InlineData("2001:200::1")] // just above 2001::/23
[InlineData("2003::1")] // just above 6to4 2002::/16
[InlineData("2001:db9::1")] // just above documentation 2001:db8::/32
[InlineData("192.0.1.0")] // just above 192.0.0.0/24
[InlineData("198.20.0.0")] // just above benchmarking 198.18.0.0/15
[InlineData("223.255.255.255")] // just below multicast
public void IsBlocked_PublicNeighbourOfRefusedBlock_ReturnsFalse(string ip)
{
await Assert.ThrowsAsync<SsrfException>(
() => SsrfGuard.ValidateDestinationAsync("not-a-url"));
Assert.False(SsrfGuard.IsBlockedAddress(IPAddress.Parse(ip)));
}
// ── IsBlockedAddress: every IANA special-purpose block (tests/fixtures/iana-special-purpose-addresses) ──
private static readonly string[] RegistryFiles = { "iana-ipv4-special-registry-1.csv", "iana-ipv6-special-registry-1.csv" };
public static IEnumerable<object[]> RegistryBlocks() =>
RegistryFiles.SelectMany(file => ReadRegistryBlocks(file).Select(block => new object[] { file, block }));
[Fact]
public async Task ValidateDestination_UnresolvableHost_ThrowsSsrfException()
public void Registry_IsReadCompletely()
{
var ex = await Assert.ThrowsAsync<SsrfException>(
() => SsrfGuard.ValidateDestinationAsync("https://this-host-does-not-exist-xyz123.invalid/webhook"));
Assert.Contains("resolve", ex.Message);
foreach (var file in RegistryFiles)
{
var blocks = ReadRegistryBlocks(file);
Assert.True(blocks.Count >= 20, $"{file}: only {blocks.Count} blocks read");
Assert.All(blocks, block => Assert.Matches(@"^[0-9a-f.:]+/\d{1,3}$", block));
}
}
[Theory]
[MemberData(nameof(RegistryBlocks))]
public void IsBlocked_EveryIanaSpecialPurposeBlock_FirstAndLastAddress(string file, string block)
{
var network = IPNetwork.Parse(block);
var first = network.BaseAddress.GetAddressBytes();
var last = (byte[])first.Clone();
for (var bit = network.PrefixLength; bit < last.Length * 8; bit++)
last[bit / 8] |= (byte)(0x80 >> (bit % 8));
foreach (var ip in new[] { new IPAddress(first), new IPAddress(last) })
Assert.True(SsrfGuard.IsBlockedAddress(ip), $"{file}: {block} -> {ip}");
}
private static List<string> ReadRegistryBlocks(string file)
{
var dir = new DirectoryInfo(AppContext.BaseDirectory);
while (dir != null && !File.Exists(Path.Combine(dir.FullName, "AutopilotMonitor.sln")))
dir = dir.Parent;
Assert.NotNull(dir);
var rows = ParseCsv(File.ReadAllText(Path.Combine(dir!.FullName, "tests", "fixtures", "iana-special-purpose-addresses", file)));
Assert.Equal("Address Block", rows[0][0]);
// Footnote markers ("2002::/16 [3]") dropped, cells that list two blocks split.
return rows.Skip(1)
.SelectMany(row => row[0].Split(','))
.Select(block => Regex.Replace(block, @"\[\d+\]", "").Trim())
.Where(block => block.Length > 0)
.ToList();
}
/// <summary>RFC 4180 reader — the registry quotes cells, wraps the RFC column over lines and doubles quotes.</summary>
private static List<List<string>> ParseCsv(string text)
{
var rows = new List<List<string>>();
var row = new List<string>();
var cell = new StringBuilder();
var quoted = false;
for (var i = 0; i < text.Length; i++)
{
var c = text[i];
if (quoted)
{
if (c == '"' && i + 1 < text.Length && text[i + 1] == '"') { cell.Append('"'); i++; }
else if (c == '"') quoted = false;
else cell.Append(c);
}
else if (c == '"') quoted = true;
else if (c == ',') { row.Add(cell.ToString()); cell.Clear(); }
else if (c == '\n' || c == '\r')
{
if (c == '\r' && i + 1 < text.Length && text[i + 1] == '\n') i++;
row.Add(cell.ToString());
rows.Add(row);
row = new List<string>();
cell.Clear();
}
else cell.Append(c);
}
if (cell.Length > 0 || row.Count > 0) { row.Add(cell.ToString()); rows.Add(row); }
return rows;
}
// ── ValidateAzureBlobSasUrlFormat ──
@@ -0,0 +1,368 @@
using System.Net;
using AutopilotMonitor.Functions.Functions.Admin;
using AutopilotMonitor.Functions.Functions.Config;
using AutopilotMonitor.Functions.Services;
using AutopilotMonitor.Shared;
using AutopilotMonitor.Shared.DataAccess;
using AutopilotMonitor.Shared.Models;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging.Abstractions;
using Moq;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// <see cref="TenantConfigurationService.UpdateAsync"/> and the plan writers built on it: the decision runs
/// on the fresh row and the write is conditional on that row's ETag — a cached view (up to 5 minutes stale
/// per instance) is never saved, so it can neither rewind another writer's change nor lift the offboarding
/// tombstone during the drain barrier, and two concurrent trial starts cannot both see TrialConsumed=false.
/// </summary>
public class TenantConfigUpdateTests
{
private const string TenantId = "aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee";
private readonly Mock<IConfigRepository> _repo = new();
private readonly Queue<(TenantConfiguration Row, string ETag)> _freshReads = new();
private readonly List<(TenantConfiguration Config, string ETag, string? Source, string? Reason)> _replaces = new();
private readonly Queue<bool> _replaceResults = new();
private readonly List<TenantConfiguration> _creates = new();
private readonly Queue<bool> _createResults = new();
public TenantConfigUpdateTests()
{
_repo.Setup(r => r.GetTenantConfigurationWithEtagAsync(TenantId))
.ReturnsAsync(() => _freshReads.Count > 0 ? _freshReads.Dequeue() : null);
_repo.Setup(r => r.TryReplaceTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), It.IsAny<string>(), It.IsAny<string?>(), It.IsAny<string?>()))
.Callback<TenantConfiguration, string, string?, string?>((c, e, s, r) => _replaces.Add((c, e, s, r)))
.ReturnsAsync(() => _replaceResults.Count == 0 || _replaceResults.Dequeue());
_repo.Setup(r => r.TryCreateTenantConfigurationAsync(It.IsAny<TenantConfiguration>()))
.Callback<TenantConfiguration>(c => _creates.Add(c))
.ReturnsAsync(() => _createResults.Count == 0 || _createResults.Dequeue());
}
private TenantConfigurationService Service(ManagedTenantProIndex? proIndex = null) =>
new(_repo.Object, NullLogger<TenantConfigurationService>.Instance,
new MemoryCache(new MemoryCacheOptions()), proIndex ?? ManagedTenantProIndex.None);
private static TenantConfiguration Row(Action<TenantConfiguration>? shape = null)
{
var row = TenantConfiguration.CreateDefault(TenantId);
row.DomainName = "contoso.com";
row.ContactEmail = "it@contoso.com";
row.CompanyName = "Contoso";
shape?.Invoke(row);
return row;
}
private static TenantConfiguration Tombstone() => Row(r =>
{
r.Disabled = true;
r.DisabledReason = TenantOffboardFunction.OffboardingDisabledReason;
});
/// <summary>Primes the service's cache with <paramref name="cached"/>, the view a warm instance would hold.</summary>
private async Task PrimeCacheAsync(TenantConfigurationService service, TenantConfiguration cached)
{
_repo.Setup(r => r.GetTenantConfigurationAsync(TenantId)).ReturnsAsync(cached);
Assert.Same(cached, await service.GetConfigurationIfExistsAsync(TenantId));
}
// ── UpdateAsync ─────────────────────────────────────────────────────────
[Fact]
public async Task UpdateAsync_MutatesTheFreshRow_UnderItsETag_NeverTheCachedView()
{
var service = Service();
await PrimeCacheAsync(service, Row(r => r.HomedAppClientId = null));
_freshReads.Enqueue((Row(r => r.HomedAppClientId = "flipped-meanwhile"), "etag-1"));
var update = await service.UpdateAsync(TenantId, row => { row.PayingCustomer = true; return true; }, "plan", "why");
Assert.Equal(TenantConfigUpdateStatus.Updated, update.Status);
var (written, etag, source, reason) = Assert.Single(_replaces);
Assert.Equal("etag-1", etag);
Assert.Equal("flipped-meanwhile", written.HomedAppClientId);
Assert.True(written.PayingCustomer);
Assert.Equal("plan", source);
Assert.Equal("why", reason);
// The cached view is gone: the next read goes to storage.
await service.GetConfigurationIfExistsAsync(TenantId);
_repo.Verify(r => r.GetTenantConfigurationAsync(TenantId), Times.Exactly(2));
}
[Fact]
public async Task UpdateAsync_LostRace_RereadsAndDecidesAgainOnTheNewRow()
{
var first = Row();
var second = Row(r => r.CompanyName = "Contoso after the concurrent write");
_freshReads.Enqueue((first, "etag-1"));
_freshReads.Enqueue((second, "etag-2"));
_replaceResults.Enqueue(false);
_replaceResults.Enqueue(true);
var seen = new List<TenantConfiguration>();
var update = await Service().UpdateAsync(TenantId, row => { seen.Add(row); return true; }, "plan", "why");
Assert.Equal(TenantConfigUpdateStatus.Updated, update.Status);
Assert.Equal(new[] { first, second }, seen);
Assert.Equal(new[] { "etag-1", "etag-2" }, _replaces.Select(r => r.ETag));
Assert.Same(second, update.Config);
}
[Fact]
public async Task UpdateAsync_EveryAttemptLosesTheRace_Conflict()
{
for (var i = 0; i < TenantConfigurationService.MaxUpdateAttempts; i++)
{
_freshReads.Enqueue((Row(), $"etag-{i}"));
_replaceResults.Enqueue(false);
}
var update = await Service().UpdateAsync(TenantId, _ => true, "plan", "why");
Assert.Equal(TenantConfigUpdateStatus.Conflict, update.Status);
Assert.Equal(TenantConfigurationService.MaxUpdateAttempts, _replaces.Count);
}
[Fact]
public async Task UpdateAsync_OffboardingTombstone_IsNeverWritten()
{
_freshReads.Enqueue((Tombstone(), "etag-1"));
var mutated = false;
var update = await Service().UpdateAsync(TenantId, _ => mutated = true, "plan", "why");
Assert.Equal(TenantConfigUpdateStatus.OffboardingInProgress, update.Status);
Assert.False(mutated);
Assert.Empty(_replaces);
}
[Fact]
public async Task UpdateAsync_MutationDeclines_NothingIsWritten()
{
var row = Row();
_freshReads.Enqueue((row, "etag-1"));
var update = await Service().UpdateAsync(TenantId, _ => false, "plan", "why");
Assert.Equal(TenantConfigUpdateStatus.Declined, update.Status);
Assert.Same(row, update.Config);
Assert.Empty(_replaces);
}
[Fact]
public async Task UpdateAsync_NoRow_NotFound()
{
var update = await Service().UpdateAsync(TenantId, _ => true, "plan", "why");
Assert.Equal(TenantConfigUpdateStatus.NotFound, update.Status);
Assert.Empty(_replaces);
}
[Fact]
public async Task UpdateAsync_TheMutationSeesTheConferredProProjection()
{
// EvaluateTrialStart's AlreadyPro verdict for a managed tenant depends on it.
_freshReads.Enqueue((Row(), "etag-1"));
string? seen = null;
await Service(new StubManagedTenantProIndex(_ => "managing-tenant"))
.UpdateAsync(TenantId, row => { seen = row.ManagedByProTenantId; return false; }, "plan", "why");
Assert.Equal("managing-tenant", seen);
}
// ── CreateOrUpdateAsync + the default row ───────────────────────────────
[Fact]
public async Task CreateOrUpdateAsync_NoRow_InsertsTheMutatedDefault()
{
var update = await Service().CreateOrUpdateAsync(TenantId, row => { row.DomainName = "contoso.com"; return true; }, "auth", "why");
Assert.Equal(TenantConfigUpdateStatus.Updated, update.Status);
var created = Assert.Single(_creates);
Assert.Equal("contoso.com", created.DomainName);
Assert.Equal(TenantId, created.TenantId);
Assert.Empty(_replaces);
}
[Fact]
public async Task CreateOrUpdateAsync_RowCreatedConcurrently_DecidesAgainOnTheStoredRow()
{
// Our read saw no row, the insert lost (409), the re-read finds the concurrent writer's row.
var stored = Row(r => r.DomainName = "seeded-by-the-other-login.com");
var reads = 0;
_repo.Setup(r => r.GetTenantConfigurationWithEtagAsync(TenantId))
.ReturnsAsync(() => reads++ == 0 ? null : (stored, "etag-2"));
_createResults.Enqueue(false);
var seen = new List<string>();
var update = await Service().CreateOrUpdateAsync(TenantId, row =>
{
seen.Add(row.DomainName);
return string.IsNullOrEmpty(row.DomainName);
}, "auth", "why");
Assert.Equal(TenantConfigUpdateStatus.Declined, update.Status);
Assert.Equal(new[] { "", "seeded-by-the-other-login.com" }, seen);
Assert.Single(_creates);
Assert.Empty(_replaces);
}
[Fact]
public async Task CreateOrUpdateAsync_Tombstone_IsNeverWritten()
{
_freshReads.Enqueue((Tombstone(), "etag-1"));
var update = await Service().CreateOrUpdateAsync(TenantId, _ => true, "offboard", "why");
Assert.Equal(TenantConfigUpdateStatus.OffboardingInProgress, update.Status);
_repo.VerifyNoTenantConfigWrite();
}
[Fact]
public async Task GetConfigurationAsync_NoRow_CreatesTheDefaultConditionally()
{
_repo.Setup(r => r.GetTenantConfigurationAsync(TenantId)).ReturnsAsync((TenantConfiguration?)null);
var config = await Service().GetConfigurationAsync(TenantId);
Assert.Same(Assert.Single(_creates), config);
_repo.Verify(r => r.TryReplaceTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), It.IsAny<string>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
}
[Fact]
public async Task GetConfigurationAsync_DefaultLostTheRace_ReturnsTheRowTheOtherWriterCreated()
{
// The first-login seed landed between our read and our insert: never overwrite it.
var seeded = Row(r => r.OnboardedBy = "first@contoso.com");
var reads = 0;
_repo.Setup(r => r.GetTenantConfigurationAsync(TenantId))
.ReturnsAsync(() => reads++ == 0 ? null : seeded);
_createResults.Enqueue(false);
var config = await Service().GetConfigurationAsync(TenantId);
Assert.Same(seeded, config);
Assert.Single(_creates);
_repo.Verify(r => r.TryReplaceTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), It.IsAny<string>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
}
// ── Endpoints ───────────────────────────────────────────────────────────
private (PlanManagementFunction Sut, Mock<IOpsEventRepository> OpsRepo, Mock<IMaintenanceRepository> Audit) Plan(
TenantConfigurationService service)
{
var adminConfig = new Mock<AdminConfigurationService>(
_repo.Object, NullLogger<AdminConfigurationService>.Instance, new MemoryCache(new MemoryCacheOptions()))
{ CallBase = false };
var opsRepo = new Mock<IOpsEventRepository>();
var audit = new Mock<IMaintenanceRepository>();
var sut = new PlanManagementFunction(
NullLogger<PlanManagementFunction>.Instance, service, adminConfig.Object, audit.Object,
new OpsEventService(opsRepo.Object, NullLogger<OpsEventService>.Instance,
TestNotifications.InertOpsAlertDispatch(adminConfig.Object)),
TestProConferral.Inert());
return (sut, opsRepo, audit);
}
[Fact]
public async Task StartTrial_ConcurrentStartWonTheRace_SecondGets409TrialAlreadyConsumed()
{
_freshReads.Enqueue((Row(), "etag-1"));
_freshReads.Enqueue((Row(r => r.TrialConsumed = true), "etag-2"));
_replaceResults.Enqueue(false);
var (sut, opsRepo, audit) = Plan(Service());
var (req, _) = EndpointHarness.Request(TenantId);
var response = await sut.StartTrial(req, TenantId);
Assert.Equal(HttpStatusCode.Conflict, response.StatusCode);
Assert.Equal(Constants.ApiErrorCodes.TrialAlreadyConsumed, EndpointHarness.ErrorCode(response));
Assert.Single(_replaces);
opsRepo.Verify(r => r.SaveOpsEventAsync(It.IsAny<OpsEventEntry>()), Times.Never);
audit.Verify(a => a.LogAuditEntryAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(),
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<Dictionary<string, string>>()), Times.Never);
}
[Fact]
public async Task StartTrial_WarmCacheDuringTheDrainBarrier_CannotLiftTheTombstone()
{
var service = Service();
await PrimeCacheAsync(service, Row()); // this instance still sees the tenant enabled
_freshReads.Enqueue((Tombstone(), "etag-1"));
var (sut, _, _) = Plan(service);
var (req, _) = EndpointHarness.Request(TenantId);
var response = await sut.StartTrial(req, TenantId);
Assert.Equal(HttpStatusCode.Forbidden, response.StatusCode);
Assert.Equal(Constants.ApiErrorCodes.TenantSuspended, EndpointHarness.ErrorCode(response));
Assert.Empty(_replaces);
_repo.VerifyNoTenantConfigWrite();
}
[Fact]
public async Task StartTrial_Allowed_WritesTheTrialOntoTheFreshRow()
{
var service = Service();
await PrimeCacheAsync(service, Row(r => r.CompanyName = "stale"));
_freshReads.Enqueue((Row(r => r.CompanyName = "Contoso"), "etag-1"));
var (sut, opsRepo, _) = Plan(service);
var (req, _) = EndpointHarness.Request(TenantId, upn: "admin@contoso.com");
var response = await sut.StartTrial(req, TenantId);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
var (written, etag, source, reason) = Assert.Single(_replaces);
Assert.Equal("etag-1", etag);
Assert.Equal("Contoso", written.CompanyName);
Assert.True(written.TrialConsumed);
Assert.Equal("admin@contoso.com", written.TrialGrantedBy);
Assert.Equal(written.TrialStartedUtc!.Value.AddDays(PlanManagementFunction.SelfServiceTrialDays), written.TrialExpiresUtc);
Assert.Equal("plan", source);
Assert.Equal("self-service trial start", reason);
opsRepo.Verify(r => r.SaveOpsEventAsync(It.Is<OpsEventEntry>(e => e.EventType == OpsEventTypes.TenantTrialStarted)), Times.Once);
}
[Fact]
public async Task SetPlanTier_WritesOnTheFreshRowUnderItsETag()
{
var service = Service();
await PrimeCacheAsync(service, Row(r => r.CompanyName = "stale"));
_freshReads.Enqueue((Row(r => r.CompanyName = "Contoso"), "etag-1"));
var (sut, _, audit) = Plan(service);
var (req, _) = EndpointHarness.Request(TenantId,
jsonBody: $"{{\"{PlanManagementFunction.PlanPatchKeys.PayingCustomer}\":true}}");
var response = await sut.SetPlanTier(req, TenantId);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
var (written, etag, _, _) = Assert.Single(_replaces);
Assert.Equal("etag-1", etag);
Assert.Equal("Contoso", written.CompanyName);
Assert.True(written.PayingCustomer);
audit.Verify(a => a.LogAuditEntryAsync(TenantId, "UPDATE", "TenantPlan", TenantId, It.IsAny<string>(),
It.Is<Dictionary<string, string>>(d => d.ContainsKey("PayingCustomer"))), Times.Once);
}
[Fact]
public async Task SetPlanTier_NothingChangesOnTheFreshRow_200WithoutAWrite()
{
_freshReads.Enqueue((Row(r => r.PayingCustomer = true), "etag-1"));
var (sut, _, audit) = Plan(Service());
var (req, _) = EndpointHarness.Request(TenantId,
jsonBody: $"{{\"{PlanManagementFunction.PlanPatchKeys.PayingCustomer}\":true}}");
var response = await sut.SetPlanTier(req, TenantId);
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
Assert.Empty(_replaces);
audit.Verify(a => a.LogAuditEntryAsync(It.IsAny<string>(), It.IsAny<string>(), It.IsAny<string>(),
It.IsAny<string>(), It.IsAny<string>(), It.IsAny<Dictionary<string, string>>()), Times.Never);
}
}
@@ -100,7 +100,7 @@ public sealed class TenantConfigurationServiceReadSemanticsTests
// row can never be Pro, whatever the index says (fail-closed).
var (service, repo, _) = BuildProjecting(Owner);
repo.Setup(r => r.GetTenantConfigurationAsync(TenantId)).ReturnsAsync((TenantConfiguration?)null);
repo.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>())).ReturnsAsync(true);
repo.Setup(r => r.TryCreateTenantConfigurationAsync(It.IsAny<TenantConfiguration>())).ReturnsAsync(true);
var config = await service.GetConfigurationAsync(TenantId);
@@ -135,7 +135,7 @@ public sealed class TenantConfigurationServiceReadSemanticsTests
.ReturnsAsync((TenantConfiguration?)null);
Assert.Null(await service.GetConfigurationIfExistsAsync(TenantId));
// MockBehavior.Strict: any SaveTenantConfigurationAsync call would have thrown.
// MockBehavior.Strict: any write (create or replace) would have thrown.
}
[Fact]
@@ -186,56 +186,60 @@ public sealed class TenantConfigurationServiceReadSemanticsTests
Assert.Equal(TenantId, config.TenantId);
}
// ── Save semantics (Codex finding 2026-07-07): the repository swallows storage exceptions
// and reports failure via its bool return — the service must THROW on false so callers
// (plan/trial endpoints, config PUT) can never audit + 200 a write that never persisted.
// ── Write semantics: UpdateAsync is the write path (D-290). A storage failure propagates, and
// the cache is dropped on every attempt — success or failure — so no instance that was read
// before the write is served afterwards as if it were current.
[Fact]
public async Task SaveConfigurationAsync_RepoReportsFalse_Throws()
private static Mock<IConfigRepository> WithStoredRow(Mock<IConfigRepository> repo)
{
var (service, repo) = Build();
repo.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.ReturnsAsync(false);
await Assert.ThrowsAsync<InvalidOperationException>(
() => service.SaveConfigurationAsync(new TenantConfiguration { TenantId = TenantId, UpdatedBy = "alice@contoso.com" }));
repo.Setup(r => r.GetTenantConfigurationAsync(TenantId))
.ReturnsAsync(new TenantConfiguration { TenantId = TenantId, PlanTier = "community", UpdatedBy = "x" });
repo.Setup(r => r.GetTenantConfigurationWithEtagAsync(TenantId))
.ReturnsAsync((new TenantConfiguration { TenantId = TenantId, PlanTier = "community", UpdatedBy = "x" }, "etag"));
return repo;
}
[Fact]
public async Task SaveConfigurationAsync_FailedSave_InvalidatesCachedInstance()
public async Task UpdateAsync_StorageError_Propagates()
{
// The plan/trial endpoints mutate the CACHED instance in place before saving. A failed
// save must drop that instance from the cache — otherwise the unsaved mutation is served
// as if persisted for up to 5 minutes.
var (service, repo) = Build();
var stored = new TenantConfiguration { TenantId = TenantId, PlanTier = "free", UpdatedBy = "x" };
repo.Setup(r => r.GetTenantConfigurationAsync(TenantId)).ReturnsAsync(stored);
repo.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.ReturnsAsync(false);
WithStoredRow(repo).Setup(r => r.TryReplaceTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), "etag", It.IsAny<string?>(), It.IsAny<string?>()))
.ThrowsAsync(new InvalidOperationException("throttled"));
var cached = await service.GetConfigurationIfExistsAsync(TenantId);
cached!.PlanTier = "enterprise"; // in-place mutation, as the plan endpoint does
await Assert.ThrowsAsync<InvalidOperationException>(
() => service.UpdateAsync(TenantId, row => { row.PlanTier = "pro"; return true; }, "plan", "test"));
}
await Assert.ThrowsAsync<InvalidOperationException>(() => service.SaveConfigurationAsync(cached));
[Fact]
public async Task UpdateAsync_FailedWrite_InvalidatesTheCache()
{
var (service, repo) = Build();
WithStoredRow(repo).Setup(r => r.TryReplaceTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), "etag", It.IsAny<string?>(), It.IsAny<string?>()))
.ThrowsAsync(new InvalidOperationException("throttled"));
await service.GetConfigurationIfExistsAsync(TenantId); // prime cache
await Assert.ThrowsAsync<InvalidOperationException>(
() => service.UpdateAsync(TenantId, row => { row.PlanTier = "pro"; return true; }, "plan", "test"));
await service.GetConfigurationIfExistsAsync(TenantId); // must go back to storage
// Next read must go back to storage (cache dropped), not serve the mutated instance.
await service.GetConfigurationIfExistsAsync(TenantId);
repo.Verify(r => r.GetTenantConfigurationAsync(TenantId), Times.Exactly(2));
}
[Fact]
public async Task SaveConfigurationAsync_Success_InvalidatesCache()
public async Task UpdateAsync_Success_InvalidatesTheCache_AndNeverTouchesTheCachedInstance()
{
var (service, repo) = Build();
repo.Setup(r => r.GetTenantConfigurationAsync(TenantId))
.ReturnsAsync(new TenantConfiguration { TenantId = TenantId, UpdatedBy = "x" });
repo.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
WithStoredRow(repo).Setup(r => r.TryReplaceTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), "etag", It.IsAny<string?>(), It.IsAny<string?>()))
.ReturnsAsync(true);
await service.GetConfigurationIfExistsAsync(TenantId); // prime cache
await service.SaveConfigurationAsync(new TenantConfiguration { TenantId = TenantId, UpdatedBy = "x" });
await service.GetConfigurationIfExistsAsync(TenantId); // must re-read after save
var cached = await service.GetConfigurationIfExistsAsync(TenantId); // prime cache
await service.UpdateAsync(TenantId, row => { row.PlanTier = "pro"; return true; }, "plan", "test");
await service.GetConfigurationIfExistsAsync(TenantId); // must re-read after the write
Assert.Equal("community", cached!.PlanTier);
repo.Verify(r => r.GetTenantConfigurationAsync(TenantId), Times.Exactly(2));
}
}
@@ -43,7 +43,7 @@ public class TenantContactEmailTests
}
// ------------------------------------------------------------------
// Server-side validation (UpdateTenantConfigurationFunction)
// Server-side validation (TenantConfigValidation)
// ------------------------------------------------------------------
[Theory]
@@ -51,7 +51,7 @@ public class TenantContactEmailTests
[InlineData("first.last+autopilot@sub.fabrikam.co.uk")]
[InlineData("UPPER@CONTOSO.COM")]
public void ValidateContactEmail_accepts_real_addresses(string email)
=> Assert.Null(UpdateTenantConfigurationFunction.ValidateContactEmail(email));
=> Assert.Null(TenantConfigValidation.ValidateContactEmail(email));
[Theory]
[InlineData(null)]
@@ -60,7 +60,7 @@ public class TenantContactEmailTests
public void ValidateContactEmail_treats_absence_as_valid(string? email)
{
// No contact address is a legitimate state — it means we cannot reach this tenant.
Assert.Null(UpdateTenantConfigurationFunction.ValidateContactEmail(email));
Assert.Null(TenantConfigValidation.ValidateContactEmail(email));
}
[Theory]
@@ -76,7 +76,7 @@ public class TenantContactEmailTests
[InlineData("Ops Team <ops@contoso.com>", "single address")] // display-name form
public void ValidateContactEmail_rejects_values_that_are_not_a_single_address(string email, string expectedHint)
{
var error = UpdateTenantConfigurationFunction.ValidateContactEmail(email);
var error = TenantConfigValidation.ValidateContactEmail(email);
Assert.NotNull(error);
Assert.Contains(expectedHint, error!, StringComparison.OrdinalIgnoreCase);
}
@@ -85,7 +85,7 @@ public class TenantContactEmailTests
public void ValidateContactEmail_rejects_header_injection()
{
// Once this address is actually mailed, an embedded CR/LF would forge mail headers.
var error = UpdateTenantConfigurationFunction.ValidateContactEmail(
var error = TenantConfigValidation.ValidateContactEmail(
$"ops@contoso.com{(char)13}{(char)10}Bcc: attacker@evil.test");
Assert.NotNull(error);
@@ -94,8 +94,8 @@ public class TenantContactEmailTests
[Fact]
public void ValidateContactEmail_rejects_an_over_long_value()
{
var local = new string('a', UpdateTenantConfigurationFunction.MaxContactEmailLength);
var error = UpdateTenantConfigurationFunction.ValidateContactEmail($"{local}@contoso.com");
var local = new string('a', TenantConfigValidation.MaxContactEmailLength);
var error = TenantConfigValidation.ValidateContactEmail($"{local}@contoso.com");
Assert.NotNull(error);
Assert.Contains("at most", error!);
@@ -189,9 +189,7 @@ public class TenantContactEmailTests
await service.TrySeedContactEmailAsync(TenantId, "ops@contoso.com");
repo.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>()), Times.Never);
repo.Verify(r => r.SaveTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
repo.VerifyNoTenantConfigWrite();
}
// ------------------------------------------------------------------
@@ -86,9 +86,7 @@ public class TenantEntitlementServiceTests
// auto-create + persist a default config row.
var (svc, repo) = Build(config: null);
await svc.GetEditionAsync(TenantId);
repo.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>()), Times.Never);
repo.Verify(r => r.SaveTenantConfigurationAsync(
It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
repo.VerifyNoTenantConfigWrite();
}
[Fact]
@@ -52,9 +52,7 @@ public sealed class TenantOffboardFunctionTests
alreadyDisabled.Disabled = true;
alreadyDisabled.DisabledReason = "Offboarding in progress";
alreadyDisabled.LastUpdated = DateTime.UtcNow.AddDays(-30);
configRepoMock
.Setup(r => r.GetTenantConfigurationAsync(It.IsAny<string>()))
.ReturnsAsync(alreadyDisabled);
configRepoMock.BackWithRow(alreadyDisabled);
var cache = new Microsoft.Extensions.Caching.Memory.MemoryCache(
new Microsoft.Extensions.Caching.Memory.MemoryCacheOptions());
@@ -196,7 +194,7 @@ public sealed class TenantOffboardFunctionTests
var sut = Build(repo);
var historyRowKey = $"{DateTime.UtcNow:yyyyMMddHHmmssfff}_{TenantId}";
await sut.UpsertPointerWithCasAsync(TenantId, historyRowKey, DateTime.UtcNow);
await sut.UpsertPointerWithCasAsync(TenantId, historyRowKey, DateTime.UtcNow, trialConsumed: false);
Assert.True(repo.Pointers.TryGetValue(TenantId, out var entry));
Assert.Equal(historyRowKey, entry.Pointer.LatestHistoryRowKey);
@@ -222,13 +220,108 @@ public sealed class TenantOffboardFunctionTests
var sut = Build(repo);
var newHistoryRowKey = $"{DateTime.UtcNow:yyyyMMddHHmmssfff}_{TenantId}";
await sut.UpsertPointerWithCasAsync(TenantId, newHistoryRowKey, DateTime.UtcNow);
await sut.UpsertPointerWithCasAsync(TenantId, newHistoryRowKey, DateTime.UtcNow, trialConsumed: false);
Assert.Equal(newHistoryRowKey, repo.Pointers[TenantId].Pointer.LatestHistoryRowKey);
Assert.Equal("Initiated", repo.Pointers[TenantId].Pointer.LatestStatus);
Assert.Equal(3, repo.Pointers[TenantId].Pointer.OffboardCount);
}
[Theory]
[InlineData(false, false, false)]
[InlineData(false, true, true)]
[InlineData(true, false, true)] // a later round without a trial never clears it
[InlineData(true, true, true)]
public async Task UpsertPointer_TrialConsumed_IsStickyAcrossRounds(bool stored, bool thisRound, bool expected)
{
var repo = new FakeOffboardingAuditRepository();
await repo.InsertByTenantPointerAsync(new OffboardingByTenantPointer
{
PartitionKey = Constants.OffboardingPartitionKeys.ByTenant,
RowKey = TenantId,
TenantId = TenantId,
LatestHistoryRowKey = "20260101000000000_" + TenantId,
LatestStatus = "Completed",
OffboardCount = 1,
TrialConsumed = stored,
});
await Build(repo).UpsertPointerWithCasAsync(TenantId, "20260201000000000_" + TenantId, DateTime.UtcNow, thisRound);
Assert.Equal(expected, repo.Pointers[TenantId].Pointer.TrialConsumed);
Assert.Equal(2, repo.Pointers[TenantId].Pointer.OffboardCount);
}
[Theory]
[InlineData(true)]
[InlineData(false)]
public async Task UpsertPointer_FreshTenant_StampsTrialConsumed(bool trialConsumed)
{
var repo = new FakeOffboardingAuditRepository();
await Build(repo).UpsertPointerWithCasAsync(TenantId, "20260201000000000_" + TenantId, DateTime.UtcNow, trialConsumed);
Assert.Equal(trialConsumed, repo.Pointers[TenantId].Pointer.TrialConsumed);
}
// ── OffboardTenant (Phase 1): the consumed trial must reach the pointer before anything is deleted ──
private static (TenantOffboardFunction Sut, FakeOffboardingAuditRepository Repo, RecordingEnqueuer Enqueuer, Mock<IConfigRepository> ConfigRepo)
BuildForOffboard(TenantConfiguration? liveConfig, Exception? readFailure = null)
{
var configRepo = new Mock<IConfigRepository>();
configRepo.BackWithRow(liveConfig).ThrowOnRead = readFailure;
var repo = new FakeOffboardingAuditRepository();
var enqueuer = new RecordingEnqueuer();
var sut = new TenantOffboardFunction(
logger: NullLogger<TenantOffboardFunction>.Instance,
configRepo: configRepo.Object,
tenantConfigService: new TenantConfigurationService(
configRepo.Object, NullLogger<TenantConfigurationService>.Instance,
new Microsoft.Extensions.Caching.Memory.MemoryCache(new Microsoft.Extensions.Caching.Memory.MemoryCacheOptions())),
maintenanceRepo: Mock.Of<IMaintenanceRepository>(),
offboardingRepo: repo,
offboardingEnqueuer: enqueuer,
previewWhitelistService: BuildPreviewWhitelistService(configRepo));
return (sut, repo, enqueuer, configRepo);
}
[Theory]
[InlineData(true)]
[InlineData(false)]
public async Task OffboardTenant_CarriesTheLiveTrialConsumedOntoThePointer(bool trialConsumed)
{
var live = TenantConfiguration.CreateDefault(TenantId);
live.DomainName = "contoso.com";
live.TrialConsumed = trialConsumed;
var (sut, repo, enqueuer, _) = BuildForOffboard(live);
var (req, context) = EndpointHarness.Request(TenantId);
var response = await sut.OffboardTenant(req, TenantId, context);
Assert.Equal(System.Net.HttpStatusCode.Accepted, response.StatusCode);
Assert.Equal(trialConsumed, repo.Pointers[TenantId].Pointer.TrialConsumed);
Assert.Single(enqueuer.Sent);
}
[Fact]
public async Task OffboardTenant_ConfigReadFails_RefusesBeforeAnyStateIsWritten()
{
// Proceeding without the read would reset the one-trial rule for the returning tenant.
var (sut, repo, enqueuer, configRepo) = BuildForOffboard(null, new RequestFailedException(503, "Server busy"));
var (req, context) = EndpointHarness.Request(TenantId);
var response = await sut.OffboardTenant(req, TenantId, context);
Assert.Equal(System.Net.HttpStatusCode.InternalServerError, response.StatusCode);
Assert.Empty(repo.History);
Assert.Empty(repo.Pointers);
Assert.Empty(repo.Markers);
Assert.Empty(enqueuer.Sent);
configRepo.VerifyNoTenantConfigWrite();
}
[Fact]
public async Task UpsertPointer_409OnInsert_FallsBackToEtagUpdate()
{
@@ -273,7 +366,7 @@ public sealed class TenantOffboardFunctionTests
var sut = Build(repoMock.Object);
var newHistoryRowKey = $"{DateTime.UtcNow:yyyyMMddHHmmssfff}_{TenantId}";
await sut.UpsertPointerWithCasAsync(TenantId, newHistoryRowKey, DateTime.UtcNow);
await sut.UpsertPointerWithCasAsync(TenantId, newHistoryRowKey, DateTime.UtcNow, trialConsumed: false);
Assert.NotNull(updatedPointer);
Assert.Equal(newHistoryRowKey, updatedPointer!.LatestHistoryRowKey);
@@ -325,7 +418,7 @@ public sealed class TenantOffboardFunctionTests
var sut = Build(repoMock.Object);
var newHistoryRowKey = $"{DateTime.UtcNow:yyyyMMddHHmmssfff}_{TenantId}";
await sut.UpsertPointerWithCasAsync(TenantId, newHistoryRowKey, DateTime.UtcNow);
await sut.UpsertPointerWithCasAsync(TenantId, newHistoryRowKey, DateTime.UtcNow, trialConsumed: false);
Assert.Equal(2, updateCalls);
Assert.Equal(2, readIdx);
@@ -484,7 +577,7 @@ public sealed class TenantOffboardFunctionTests
var historyRowKey = $"{DateTime.UtcNow:yyyyMMddHHmmssfff}_{TenantId}";
await Assert.ThrowsAsync<InvalidOperationException>(() =>
sut.UpsertPointerWithCasAsync(TenantId, historyRowKey, DateTime.UtcNow));
sut.UpsertPointerWithCasAsync(TenantId, historyRowKey, DateTime.UtcNow, trialConsumed: false));
}
// ── Cache-Drain-Barrier (plan v2 §2.2 / §2.3) ─────────────────────────────
@@ -638,27 +731,42 @@ public sealed class TenantOffboardFunctionTests
var (sut, configRepo, cache, _) = BuildWithConfigService();
var existing = TenantConfiguration.CreateDefault(TenantId);
existing.Disabled = false;
configRepo.Setup(r => r.GetTenantConfigurationAsync(TenantId)).ReturnsAsync(existing);
TenantConfiguration? saved = null;
configRepo.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.Callback<TenantConfiguration, string?, string?>((c, _, _) => saved = c)
.ReturnsAsync(true);
existing.DomainName = "contoso.com";
var store = configRepo.BackWithRow(existing);
// Seed cache with stale "Disabled=false" entry so we can prove invalidation.
cache.Set($"tenant-config:{TenantId}", existing);
await sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com");
Assert.NotNull(saved);
Assert.True(saved!.Disabled);
var (saved, source, reason) = Assert.Single(store.Writes);
Assert.True(saved.Disabled);
Assert.Equal("Offboarding in progress", saved.DisabledReason);
Assert.Null(saved.DisabledUntil);
Assert.Equal("alice@contoso.com", saved.UpdatedBy);
Assert.Equal("contoso.com", saved.DomainName); // the rest of the row survives
Assert.Equal(("offboard", "Disabled-gate before cascade"), (source, reason));
Assert.False(cache.TryGetValue($"tenant-config:{TenantId}", out _),
"cache entry must be invalidated so warm reads pick up the new Disabled=true on next access");
}
[Fact]
public async Task EnsureTenantDisabled_DecidesOnTheStoredRow_NotAStaleCachedView()
{
// A cached copy still says "not disabled"; the stored row is already the tombstone.
var (sut, configRepo, cache, _) = BuildWithConfigService();
var tombstone = TenantConfiguration.CreateDefault(TenantId);
tombstone.Disabled = true;
tombstone.DisabledReason = "Offboarding in progress";
var store = configRepo.BackWithRow(tombstone);
cache.Set($"tenant-config:{TenantId}", TenantConfiguration.CreateDefault(TenantId));
var result = await sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com");
Assert.Equal(TenantOffboardFunction.EnsureDisabledOutcome.AlreadyTombstone, result.Outcome);
Assert.Empty(store.Writes);
}
[Fact]
public async Task EnsureTenantDisabled_AlreadyDisabledWithSameReason_DoesNotRewrite()
{
@@ -666,12 +774,12 @@ public sealed class TenantOffboardFunctionTests
var existing = TenantConfiguration.CreateDefault(TenantId);
existing.Disabled = true;
existing.DisabledReason = "Offboarding in progress";
configRepo.Setup(r => r.GetTenantConfigurationAsync(TenantId)).ReturnsAsync(existing);
var store = configRepo.BackWithRow(existing);
await sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com");
// Idempotent: no Save thrash on resume-clicks while the marker is already in flight.
configRepo.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
// Idempotent: no write thrash on resume-clicks while the marker is already in flight.
Assert.Empty(store.Writes);
}
[Fact]
@@ -681,21 +789,19 @@ public sealed class TenantOffboardFunctionTests
// skip the enqueue. We just assert the exception propagates — the caller's
// try/catch + 500-return is exercised by integration tests.
var (sut, configRepo, _, _) = BuildWithConfigService();
configRepo.Setup(r => r.GetTenantConfigurationAsync(TenantId))
.ThrowsAsync(new InvalidOperationException("storage unavailable"));
configRepo.BackWithRow(TenantConfiguration.CreateDefault(TenantId)).ThrowOnRead =
new InvalidOperationException("storage unavailable");
await Assert.ThrowsAsync<InvalidOperationException>(() =>
sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com"));
}
[Fact]
public async Task EnsureTenantDisabled_StorageThrowsOnSave_PropagatesAsExpected()
public async Task EnsureTenantDisabled_StorageThrowsOnWrite_PropagatesAsExpected()
{
var (sut, configRepo, _, _) = BuildWithConfigService();
configRepo.Setup(r => r.GetTenantConfigurationAsync(TenantId))
.ReturnsAsync(TenantConfiguration.CreateDefault(TenantId));
configRepo.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.ThrowsAsync(new InvalidOperationException("storage save failed"));
configRepo.BackWithRow(TenantConfiguration.CreateDefault(TenantId)).ThrowOnWrite =
new InvalidOperationException("storage write failed");
await Assert.ThrowsAsync<InvalidOperationException>(() =>
sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com"));
@@ -709,35 +815,25 @@ public sealed class TenantOffboardFunctionTests
// The fix creates a fresh CreateDefault row with Disabled=true so the auth-gate is
// committed even in this edge case.
var (sut, configRepo, _, _) = BuildWithConfigService();
configRepo.Setup(r => r.GetTenantConfigurationAsync(TenantId))
.ReturnsAsync((TenantConfiguration?)null);
var store = configRepo.BackWithRow(null);
TenantConfiguration? saved = null;
configRepo.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.Callback<TenantConfiguration, string?, string?>((c, _, _) => saved = c)
.ReturnsAsync(true);
var result = await sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com");
await sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com");
Assert.NotNull(saved);
Assert.Equal(TenantId, saved!.TenantId);
var saved = Assert.Single(store.Writes).Row;
Assert.Equal(TenantId, saved.TenantId);
Assert.True(saved.Disabled, "fresh tombstone must carry Disabled=true");
Assert.Equal("Offboarding in progress", saved.DisabledReason);
Assert.Equal("alice@contoso.com", saved.UpdatedBy);
Assert.Equal(TenantOffboardFunction.EnsureDisabledOutcome.WroteNewTombstone, result.Outcome);
}
[Fact]
public async Task EnsureTenantDisabled_SaveReturnsFalse_ThrowsAsHardFail()
public async Task EnsureTenantDisabled_EveryWriteLost_ThrowsAsHardFail()
{
// PR3.B-revised Finding 2: TableConfigRepository.SaveTenantConfigurationAsync
// returns false on transient storage failure (it does NOT throw). Ignoring that
// return-value would let the caller proceed to enqueue without the auth-gate
// committed. Turn it into a hard exception so the offboard is NOT enqueued.
// Without Disabled=true the caller would proceed to enqueue without the auth-gate
// committed: every lost conditional write is a hard exception, never a silent skip.
var (sut, configRepo, _, _) = BuildWithConfigService();
configRepo.Setup(r => r.GetTenantConfigurationAsync(TenantId))
.ReturnsAsync(TenantConfiguration.CreateDefault(TenantId));
configRepo.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.ReturnsAsync(false); // transient storage rejection — NOT a throw.
configRepo.BackWithRow(TenantConfiguration.CreateDefault(TenantId)).RejectWrites = true;
var ex = await Assert.ThrowsAsync<InvalidOperationException>(() =>
sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com"));
@@ -745,15 +841,10 @@ public sealed class TenantOffboardFunctionTests
}
[Fact]
public async Task EnsureTenantDisabled_NoConfigRow_SaveReturnsFalse_ThrowsAsHardFail()
public async Task EnsureTenantDisabled_NoConfigRow_EveryWriteLost_ThrowsAsHardFail()
{
// Combined edge case: missing row → we'd create a fresh tombstone → Save fails →
// hard fail (otherwise the caller would proceed without an auth-gate).
var (sut, configRepo, _, _) = BuildWithConfigService();
configRepo.Setup(r => r.GetTenantConfigurationAsync(TenantId))
.ReturnsAsync((TenantConfiguration?)null);
configRepo.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.ReturnsAsync(false);
configRepo.BackWithRow(null).RejectWrites = true;
await Assert.ThrowsAsync<InvalidOperationException>(() =>
sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com"));
@@ -766,12 +857,12 @@ public sealed class TenantOffboardFunctionTests
var tombstone = TenantConfiguration.CreateDefault(TenantId);
tombstone.Disabled = true;
tombstone.DisabledReason = "Offboarding in progress";
configRepo.Setup(r => r.GetTenantConfigurationAsync(TenantId)).ReturnsAsync(tombstone);
var store = configRepo.BackWithRow(tombstone);
var result = await sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com");
Assert.Equal(TenantOffboardFunction.EnsureDisabledOutcome.AlreadyTombstone, result.Outcome);
configRepo.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
Assert.Empty(store.Writes);
}
[Fact]
@@ -780,13 +871,13 @@ public sealed class TenantOffboardFunctionTests
var (sut, configRepo, _, _) = BuildWithConfigService();
var existing = TenantConfiguration.CreateDefault(TenantId);
existing.Disabled = false;
configRepo.Setup(r => r.GetTenantConfigurationAsync(TenantId)).ReturnsAsync(existing);
configRepo.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.ReturnsAsync(true);
var store = configRepo.BackWithRow(existing);
var result = await sut.EnsureTenantDisabledAsync(TenantId, "alice@contoso.com");
Assert.Equal(TenantOffboardFunction.EnsureDisabledOutcome.WroteNewTombstone, result.Outcome);
// The anchor of the drain-deadline math is the stored write time.
Assert.Equal(Assert.Single(store.Writes).Row.LastUpdated, result.TombstoneLastUpdated);
}
// ── Resume-path cache-drain bypass fix (Codex review) ────────────────────
@@ -830,10 +921,7 @@ public sealed class TenantOffboardFunctionTests
var configRepoMock = new Mock<IConfigRepository>();
var notYetDisabled = TenantConfiguration.CreateDefault(TenantId);
notYetDisabled.Disabled = false;
configRepoMock.Setup(r => r.GetTenantConfigurationAsync(It.IsAny<string>()))
.ReturnsAsync(notYetDisabled);
configRepoMock.Setup(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()))
.ReturnsAsync(true);
configRepoMock.BackWithRow(notYetDisabled);
var cache = new MemoryCache(new MemoryCacheOptions());
var tenantConfigService = new TenantConfigurationService(
@@ -925,8 +1013,7 @@ public sealed class TenantOffboardFunctionTests
tombstone.Disabled = true;
tombstone.DisabledReason = "Offboarding in progress";
tombstone.LastUpdated = DateTime.UtcNow.AddSeconds(-1); // freshly written ~1s ago
configRepoMock.Setup(r => r.GetTenantConfigurationAsync(It.IsAny<string>()))
.ReturnsAsync(tombstone);
configRepoMock.BackWithRow(tombstone);
var cache = new MemoryCache(new MemoryCacheOptions());
var tenantConfigService = new TenantConfigurationService(
@@ -1000,8 +1087,7 @@ public sealed class TenantOffboardFunctionTests
tombstone.Disabled = true;
tombstone.DisabledReason = "Offboarding in progress";
tombstone.LastUpdated = initiatedAt;
configRepoMock.Setup(r => r.GetTenantConfigurationAsync(It.IsAny<string>()))
.ReturnsAsync(tombstone);
configRepoMock.BackWithRow(tombstone);
var cache = new MemoryCache(new MemoryCacheOptions());
var tenantConfigService = new TenantConfigurationService(
@@ -1043,8 +1129,8 @@ public sealed class TenantOffboardFunctionTests
var historyAfter = repo.History[historyRowKey];
Assert.Equal(originalEarliestProcessingAt, historyAfter.EarliestProcessingAt);
// No Save call (the tombstone was already correct).
configRepoMock.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string?>(), It.IsAny<string?>()), Times.Never);
// No write (the tombstone was already correct).
configRepoMock.VerifyNoTenantConfigWrite();
}
// ── RetryFailedOffboardingAsync (operator retry of a Failed offboarding) ──────────
@@ -1208,7 +1294,7 @@ public sealed class TenantOffboardFunctionTests
tombstone.Disabled = true;
tombstone.DisabledReason = "Offboarding in progress";
tombstone.LastUpdated = DateTime.UtcNow.AddMinutes(-2);
configRepoMock.Setup(r => r.GetTenantConfigurationAsync(It.IsAny<string>())).ReturnsAsync(tombstone);
configRepoMock.BackWithRow(tombstone);
var cache = new MemoryCache(new MemoryCacheOptions());
var service = new TenantConfigurationService(configRepoMock.Object, NullLogger<TenantConfigurationService>.Instance, cache);
var enqueuer = new RecordingEnqueuer();
@@ -1227,7 +1313,7 @@ public sealed class TenantOffboardFunctionTests
var delay = Assert.Single(enqueuer.VisibilityDelays);
Assert.NotNull(delay);
Assert.InRange(delay!.Value, TimeSpan.FromMinutes(3.5), TimeSpan.FromMinutes(4.1));
configRepoMock.Verify(r => r.SaveTenantConfigurationAsync(It.IsAny<TenantConfiguration>(), It.IsAny<string>(), It.IsAny<string>()), Times.Never);
configRepoMock.VerifyNoTenantConfigWrite();
}
[Fact]
@@ -191,6 +191,7 @@ public class TypedContractCompletionWireParityTests
new
{
appName = "Contoso App",
source = "ime",
totalInstalls = 4,
succeeded = 2,
skipped = 1,
@@ -234,6 +235,7 @@ public class TypedContractCompletionWireParityTests
new AppMetricsAppGroup
{
AppName = "Contoso App",
Source = AppInstallSources.Ime,
TotalInstalls = 4,
Succeeded = 2,
Skipped = 1,
@@ -593,6 +595,7 @@ public class TypedContractCompletionWireParityTests
var item = new AppsListItem
{
AppName = "Contoso App",
Source = AppInstallSources.Ime,
AppType = "Win32",
TotalInstalls = 5,
Succeeded = 3,
@@ -610,6 +613,7 @@ public class TypedContractCompletionWireParityTests
var anonymousItem = new
{
appName = "Contoso App",
source = "ime",
appType = "Win32",
totalInstalls = 5,
succeeded = 3,
@@ -677,6 +681,7 @@ public class TypedContractCompletionWireParityTests
{
success = true,
appName = "Contoso App",
source = "ime",
appType = string.Empty,
windowDays = 30,
collisionExcluded = 0,
@@ -709,6 +714,7 @@ public class TypedContractCompletionWireParityTests
{
Success = true,
AppName = "Contoso App",
Source = AppInstallSources.Ime,
AppType = string.Empty,
WindowDays = 30,
CollisionExcluded = 0,
@@ -61,8 +61,7 @@ public class TypedRequestGuardTests
private static readonly Dictionary<string, int> NewtonsoftBaseline = new(StringComparer.OrdinalIgnoreCase)
{
["Functions/Config/PatchTenantConfigurationFieldsFunction.cs"] = 1,
["Functions/Config/UpdateAdminConfigurationFunction.cs"] = 1,
["Functions/Config/UpdateTenantConfigurationFunction.cs"] = 1,
["Functions/Config/PatchAdminConfigurationFunction.cs"] = 1,
["Functions/Rules/AnalyzeRulesFunction.cs"] = 2,
["Functions/Rules/DryRunAnalyzeRuleFunction.cs"] = 1,
["Functions/Rules/GatherRulesFunction.cs"] = 2,
@@ -157,13 +156,21 @@ public class TypedRequestGuardTests
var isObjectMap = slot.IsGenericType && slot.GetGenericTypeDefinition() == typeof(Dictionary<,>) && slot.GetGenericArguments()[1] == typeof(object);
if (slot == typeof(object))
offenders.Add($"{t.Name}.{p.Name}: object slot (TypeScript gets unknown)");
else if (isObjectMap && !(t == typeof(PatchTenantConfigurationFieldsRequest) && p.Name == nameof(PatchTenantConfigurationFieldsRequest.Fields)))
offenders.Add($"{t.Name}.{p.Name}: untyped map (only the tenant-config field patch carries one by design)");
else if (isObjectMap && !IsFieldPatchMap(t, p))
offenders.Add($"{t.Name}.{p.Name}: untyped map (only the config field patches carry one by design)");
}
}
Assert.True(offenders.Count == 0, "Request DTO shape violations:\n " + string.Join("\n ", offenders));
}
/// <summary>
/// The two field patches (tenant and admin configuration) carry a { field: value } map on purpose:
/// each value takes its field's own JSON type and the server gates the keys against the model.
/// </summary>
private static bool IsFieldPatchMap(Type type, PropertyInfo property)
=> (type == typeof(PatchTenantConfigurationFieldsRequest) && property.Name == nameof(PatchTenantConfigurationFieldsRequest.Fields))
|| (type == typeof(PatchAdminConfigurationRequest) && property.Name == nameof(PatchAdminConfigurationRequest.Fields));
private static IEnumerable<(string Relative, string Text)> HandlerSources()
{
var functionsRoot = Path.Combine(FindRepoRoot(), "src", "Backend", "AutopilotMonitor.Functions");
@@ -0,0 +1,247 @@
using System.Net;
using System.Net.Http;
using System.Net.Security;
using System.Net.Sockets;
using System.Security.Cryptography;
using System.Security.Cryptography.X509Certificates;
using System.Text;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Functions.Security;
using AutopilotMonitor.Functions.Services.Notifications;
using AutopilotMonitor.Shared.Models.Notifications;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Logging.Abstractions;
namespace AutopilotMonitor.Functions.Tests;
/// <summary>
/// The webhook SSRF gate on real loopback sockets. DNS is injected: <c>rebind.example.test</c> does
/// not exist, so a socket reaching a listener proves it used the gate's answer rather than a
/// second resolution.
/// </summary>
public class WebhookConnectGateTests
{
private const string Host = "rebind.example.test";
private sealed class CountingResolver
{
private readonly IPAddress[] _answer;
public List<string> Calls { get; } = new();
public CountingResolver(params string[] answer) => _answer = answer.Select(IPAddress.Parse).ToArray();
public Task<IPAddress[]> Resolve(string host, CancellationToken _)
{
Calls.Add(host);
return Task.FromResult(_answer);
}
}
private static HttpClient ClientThrough(Func<SocketsHttpConnectionContext, CancellationToken, ValueTask<Stream>> connect,
X509Certificate2? trustedServerCertificate = null)
{
var handler = new SocketsHttpHandler { AllowAutoRedirect = false, ConnectCallback = connect };
if (trustedServerCertificate != null)
{
// Pinned to the test's own self-signed certificate — validation stays on, only the trust anchor differs.
var pinned = trustedServerCertificate.GetCertHashString();
handler.SslOptions.RemoteCertificateValidationCallback = (_, cert, _, _) => cert?.GetCertHashString() == pinned;
}
// Short timeout: with a broken gate the TLS handshake waits on a listener that never accepts.
return new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(10) };
}
private static TcpListener StartListener()
{
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
return listener;
}
private static int PortOf(TcpListener listener) => ((IPEndPoint)listener.LocalEndpoint).Port;
[Fact]
public async Task PrivateAnswer_FailsClosed_WithoutOpeningASocket()
{
var listener = StartListener();
try
{
var resolver = new CountingResolver("127.0.0.1");
using var client = ClientThrough(SsrfGuard.CreateConnectCallback(resolver.Resolve));
var ex = await Assert.ThrowsAsync<HttpRequestException>(() => client.GetAsync($"https://{Host}:{PortOf(listener)}/hook"));
var refused = Assert.IsType<SsrfException>(ex.InnerException);
Assert.Contains("private or reserved", refused.Message);
Assert.Equal(new[] { Host }, resolver.Calls);
Assert.False(listener.Pending());
}
finally
{
listener.Stop();
}
}
[Fact]
public async Task MixedAnswer_WithOnePrivateAddress_FailsClosed()
{
var resolver = new CountingResolver("1.1.1.1", "169.254.169.254");
using var client = ClientThrough(SsrfGuard.CreateConnectCallback(resolver.Resolve));
var ex = await Assert.ThrowsAsync<HttpRequestException>(() => client.GetAsync($"https://{Host}/hook"));
Assert.IsType<SsrfException>(ex.InnerException);
}
[Fact]
public async Task PrivateIpLiteral_FailsClosed_WithoutResolving()
{
var listener = StartListener();
try
{
var resolver = new CountingResolver("1.1.1.1");
using var client = ClientThrough(SsrfGuard.CreateConnectCallback(resolver.Resolve));
var ex = await Assert.ThrowsAsync<HttpRequestException>(() => client.GetAsync($"https://127.0.0.1:{PortOf(listener)}/hook"));
Assert.IsType<SsrfException>(ex.InnerException);
Assert.Empty(resolver.Calls);
Assert.False(listener.Pending());
}
finally
{
listener.Stop();
}
}
[Fact]
public async Task PlainHttp_IsRefused_BeforeResolving()
{
var resolver = new CountingResolver("1.1.1.1");
using var client = ClientThrough(SsrfGuard.CreateConnectCallback(resolver.Resolve));
var ex = await Assert.ThrowsAsync<HttpRequestException>(() => client.GetAsync($"http://{Host}/hook"));
Assert.Contains("HTTPS", Assert.IsType<SsrfException>(ex.InnerException).Message);
Assert.Empty(resolver.Calls);
}
[Fact]
public async Task UnresolvableOrEmptyAnswer_FailsClosed()
{
Task<IPAddress[]> NotFound(string _, CancellationToken __) => throw new SocketException((int)SocketError.HostNotFound);
using (var client = ClientThrough(SsrfGuard.CreateConnectCallback(NotFound)))
{
var ex = await Assert.ThrowsAsync<HttpRequestException>(() => client.GetAsync($"https://{Host}/hook"));
Assert.Contains("resolve", Assert.IsType<SsrfException>(ex.InnerException).Message);
}
using (var client = ClientThrough(SsrfGuard.CreateConnectCallback(new CountingResolver().Resolve)))
{
var ex = await Assert.ThrowsAsync<HttpRequestException>(() => client.GetAsync($"https://{Host}/hook"));
Assert.Contains("no addresses", Assert.IsType<SsrfException>(ex.InnerException).Message);
}
}
[Fact]
public async Task AllowedAnswer_ConnectsToExactlyThatAddress_AndTlsStillTargetsTheRequestHost()
{
using var certificate = CreateServerCertificate(Host);
var listener = StartListener();
try
{
string? serverName = null;
var server = Task.Run(async () =>
{
using var tcp = await listener.AcceptTcpClientAsync();
using var tls = new SslStream(tcp.GetStream());
await tls.AuthenticateAsServerAsync(new SslServerAuthenticationOptions { ServerCertificate = certificate });
serverName = tls.TargetHostName;
var buffer = new byte[4096];
var request = new StringBuilder();
while (!request.ToString().Contains("\r\n\r\n"))
{
var read = await tls.ReadAsync(buffer);
if (read == 0) break;
request.Append(Encoding.ASCII.GetString(buffer, 0, read));
}
await tls.WriteAsync(Encoding.ASCII.GetBytes("HTTP/1.1 200 OK\r\nContent-Length: 2\r\nConnection: close\r\n\r\nok"));
});
// The classifier is widened to reach the loopback listener; the socket path is the production one.
var resolver = new CountingResolver("127.0.0.1");
using var client = ClientThrough(SsrfGuard.CreateConnectCallback(resolver.Resolve, _ => false), certificate);
var response = await client.GetAsync($"https://{Host}:{PortOf(listener)}/hook");
await server;
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
Assert.Equal("ok", await response.Content.ReadAsStringAsync());
Assert.Equal(Host, serverName);
Assert.Equal(new[] { Host }, resolver.Calls);
}
finally
{
listener.Stop();
}
}
[Fact]
public void WebhookClient_PrimaryHandler_ConnectsOnlyThroughTheGate()
{
var services = new ServiceCollection();
services.AddLogging();
services.AddSingleton<ResiliencePolicies>();
services.AddWebhookNotificationHttpClient();
using var provider = services.BuildServiceProvider();
HttpMessageHandler handler = provider.GetRequiredService<IHttpMessageHandlerFactory>().CreateHandler(nameof(WebhookNotificationService));
while (handler is DelegatingHandler delegating)
handler = delegating.InnerHandler!;
var primary = Assert.IsType<SocketsHttpHandler>(handler);
Assert.Same(SsrfGuard.ConnectToPublicAddress, primary.ConnectCallback);
Assert.False(primary.AllowAutoRedirect);
}
[Fact]
public async Task NotificationPolicy_DoesNotRetryARefusedDestination()
{
var policies = new ResiliencePolicies(NullLoggerFactory.Instance);
var attempts = 0;
await Assert.ThrowsAsync<HttpRequestException>(() => policies.Notification.ExecuteAsync(() =>
{
attempts++;
throw new HttpRequestException("refused", new SsrfException("Webhook URL targets a private or reserved network."));
}));
Assert.Equal(1, attempts);
}
[Fact]
public async Task TestNotification_ReportsTheRefusal()
{
var resolver = new CountingResolver("10.0.0.8");
var webhook = new WebhookNotificationService(ClientThrough(SsrfGuard.CreateConnectCallback(resolver.Resolve)),
NullLogger<WebhookNotificationService>.Instance);
var result = await webhook.SendNotificationWithResultAsync($"https://{Host}/hook", WebhookProviderType.GenericJson,
new NotificationAlert { Title = "T", Summary = "S" });
Assert.False(result.Success);
Assert.Equal("Webhook URL targets a private or reserved network.", result.Message);
}
private static X509Certificate2 CreateServerCertificate(string host)
{
using var key = ECDsa.Create(ECCurve.NamedCurves.nistP256);
var request = new CertificateRequest($"CN={host}", key, HashAlgorithmName.SHA256);
var san = new SubjectAlternativeNameBuilder();
san.AddDnsName(host);
request.CertificateExtensions.Add(san.Build());
request.CertificateExtensions.Add(new X509EnhancedKeyUsageExtension(new OidCollection { new Oid("1.3.6.1.5.5.7.3.1") }, false));
using var ephemeral = request.CreateSelfSigned(DateTimeOffset.UtcNow.AddMinutes(-5), DateTimeOffset.UtcNow.AddHours(1));
// Round-trip through PKCS#12: SChannel cannot serve a certificate whose key is ephemeral.
return X509CertificateLoader.LoadPkcs12(ephemeral.Export(X509ContentType.Pfx), null);
}
}
@@ -3,159 +3,158 @@ using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Text.Json;
using System.Threading;
using System.Threading.Tasks;
using AutopilotMonitor.Functions.Helpers;
using AutopilotMonitor.Shared.Pagination;
namespace AutopilotMonitor.Functions.DataAccess.TableStorage
{
/// <summary>
/// Generic helpers driving cross-partition merge-pagination for global views
/// where Azure-Tables' native (PK asc, RK asc) ordering would otherwise
/// surface partitions one-at-a-time instead of globally newest-first.
/// Used by ops-events (categories as partitions) and global audit logs
/// (tenants as partitions).
/// Newest-first paging across the partitions of one table: global ops events (categories as
/// partitions) and the global audit log (tenants as partitions). Azure Tables pages a
/// cross-partition query in (PK asc, RK asc) order, which would surface one partition at a time,
/// so every page runs one PartitionKey query per partition and merges the results.
/// </summary>
/// <remarks>
/// <para>
/// Per-partition continuation anchored on RowKey (not Azure's opaque token):
/// <c>LastRowKey</c> = the oldest RowKey we have already returned for this
/// partition. Next fetch filters <c>RowKey gt LastRowKey</c> — natural
/// ordering within the partition (revtick → newest-first) means this skips
/// exactly what we showed and re-considers everything older.
/// Invariant: every partition uses the same reverse-tick RowKey scheme, so the global order is
/// (RowKey asc, Partition asc) and that order is the display order. The merge sorts by exactly
/// that key and never by a resolved timestamp column — a merge key that disagrees with the
/// RowKey inside one partition would let the watermark skip rows.
/// </para>
/// <para>
/// Why not Azure's continuation token? Cross-partition merge can drop items
/// that were fetched but didn't survive the top-pageSize cut. If we advanced
/// an Azure cursor past those leftovers, we'd lose them forever. With a
/// RowKey bound we simply don't advance the continuation for partitions whose
/// fetched items lost the merge — they get re-fetched next call and compete
/// again at a lower floor.
/// The continuation is the (RowKey, Partition) of the last returned row, so its size does not
/// depend on the number of partitions. The earlier per-partition continuation map kept one entry
/// for every partition with rows and outgrew the host's URL limit (HTTP 414) on the tenant
/// fan-out.
/// </para>
/// <para>
/// Continuation shape: only <em>active</em> partitions are encoded. Missing
/// from the map = exhausted (or never had any data). For 200-tenant audit
/// fan-outs that previously kept exhausted entries around, the encoded
/// token would balloon past 30 KB and trip URL-length limits before
/// reaching the function (browser <c>net::ERR_FAILED</c> on Next click).
/// Now the token shrinks page-over-page as partitions exhaust.
/// Each partition reads at most <c>pageSize + 1</c> rows after the watermark. More rows exist
/// exactly when more than <c>pageSize</c> were read in total: if no partition reached its limit,
/// every partition is exhausted.
/// </para>
/// </remarks>
internal static class PerPartitionFanOutMerge
{
internal sealed record PartitionContinuation(string? LastRowKey);
/// <summary>The (RowKey, Partition) of the last row a page returned.</summary>
internal sealed record Watermark(string RowKey, string Partition);
internal sealed record PartitionFetchResult<T>(
string Partition,
IReadOnlyList<(string RowKey, T Item)> Items);
/// <summary>
/// Pure merge-and-advance step. Given pre-fetched per-partition items
/// (already newest-first within each list) and the previous per-partition
/// continuation map, returns the top-pageSize merged items plus the
/// next continuation map. Partitions with nothing left to return are
/// dropped from the map entirely — consumers treat "missing" as
/// exhausted on subsequent calls (see class remarks).
/// </summary>
public static (List<T> Items, Dictionary<string, PartitionContinuation> NextContinuations)
MergeAndAdvance<T>(
IReadOnlyList<PartitionFetchResult<T>> fetched,
IReadOnlyDictionary<string, PartitionContinuation> priorContinuations,
int pageSize,
Func<T, DateTime> timestampSelector)
/// <summary>Lower RowKey bound of one partition's read.</summary>
internal sealed record RowKeyBound(string RowKey, bool Inclusive)
{
var allFetched = fetched
.SelectMany(r => r.Items.Select(t => (part: r.Partition, rk: t.RowKey, item: t.Item)))
.OrderByDescending(t => timestampSelector(t.item))
.ToList();
var merged = allFetched.Take(pageSize).ToList();
var returnedItems = merged.Select(t => t.item).ToList();
var newContinuations = new Dictionary<string, PartitionContinuation>(StringComparer.Ordinal);
foreach (var r in fetched)
{
var returnedFromPart = merged.Where(m => m.part == r.Partition).ToList();
if (returnedFromPart.Count > 0)
{
// Advance to the oldest RowKey we returned for this partition.
// Revtick scheme: larger RowKey string = older timestamp, so
// OrdinalCompare-max gives us the oldest of the returned set.
var oldestRk = returnedFromPart
.Select(m => m.rk)
.OrderBy(rk => rk, StringComparer.Ordinal)
.Last();
newContinuations[r.Partition] = new PartitionContinuation(oldestRk);
}
else if (r.Items.Count == 0)
{
// No items returned from this partition's query → exhausted.
// Drop from map entirely; the consumer's "missing = exhausted"
// rule (when continuation was provided) does the right thing
// and keeps the encoded token small.
}
else
{
// Fetched some but they all lost the merge cut to other partitions.
// Keep continuation where it was — they'll be re-fetched and likely
// win on the next page when the floor drops. Wasted bandwidth, but
// correct (no item silently dropped).
priorContinuations.TryGetValue(r.Partition, out var prior);
newContinuations[r.Partition] = prior ?? new PartitionContinuation(null);
}
}
return (returnedItems, newContinuations);
public string ToODataClause() => $"RowKey {(Inclusive ? "ge" : "gt")} '{RowKey.Replace("'", "''")}'";
}
// Wire format v1 (legacy): every partition encoded with `{rk, x:bool}`.
// Wire format v2 (current): only ACTIVE partitions, `{rk}` only. Decoder
// accepts both and silently drops v1 `x:true` entries so old tokens
// emitted by previous deploys still page through cleanly.
private const int WireFormatVersion = 2;
/// <summary>
/// Delegate that reads up to <c>limit</c> rows of one partition, newest first (RowKey
/// ascending), restricted by <c>bound</c> when it is set. It must return every matching row
/// up to the limit — a short read is taken as "partition exhausted".
/// </summary>
internal delegate Task<List<(string RowKey, T Item)>> PartitionReader<T>(
string partition, RowKeyBound? bound, int limit, CancellationToken cancellationToken);
public static string EncodeMultiContinuation(IReadOnlyDictionary<string, PartitionContinuation> continuations)
/// <summary>
/// The rows of <paramref name="partition"/> that come strictly after the watermark in
/// (RowKey, Partition) order. A partition that sorts after the watermark's partition may still
/// hold rows with the watermark's own RowKey; one that sorts before (or is the same) may not.
/// </summary>
internal static RowKeyBound? BoundAfter(Watermark? watermark, string partition)
{
var doc = new Dictionary<string, object?>
if (watermark == null) return null;
var inclusive = string.CompareOrdinal(partition, watermark.Partition) > 0;
return new RowKeyBound(watermark.RowKey, inclusive);
}
public static async Task<RawPage<T>> FetchPageAsync<T>(
IEnumerable<string> partitions,
int pageSize,
string? continuation,
PartitionReader<T> readPartition,
CancellationToken cancellationToken = default)
{
if (pageSize < 1) throw new ArgumentOutOfRangeException(nameof(pageSize));
Watermark? watermark = null;
if (!string.IsNullOrEmpty(continuation))
{
watermark = DecodeWatermark(continuation);
// The HMAC wrapper already rejected foreign tokens; one that passes it and still does
// not decode was minted by an older deploy. End the list rather than restart it.
if (watermark == null) return RawPage<T>.Empty;
}
var limit = pageSize + 1;
var rows = await BoundedFanOut.RunAsync(partitions, BoundedFanOut.CrossTenantConcurrency, async (partition, ct) =>
{
var read = await readPartition(partition, BoundAfter(watermark, partition), limit, ct).ConfigureAwait(false);
return read.Select(r => (Partition: partition, r.RowKey, r.Item)).ToList();
}, cancellationToken).ConfigureAwait(false);
return Merge(rows, pageSize);
}
/// <summary>
/// Pure merge step: orders the rows read from all partitions by (RowKey, Partition), returns
/// the first <paramref name="pageSize"/> and a continuation when rows are left over.
/// </summary>
internal static RawPage<T> Merge<T>(IEnumerable<(string Partition, string RowKey, T Item)> rows, int pageSize)
{
var ordered = rows
.OrderBy(r => r.RowKey, StringComparer.Ordinal)
.ThenBy(r => r.Partition, StringComparer.Ordinal)
.ToList();
var page = ordered.Take(pageSize).ToList();
string? next = null;
if (ordered.Count > pageSize)
{
var last = page[page.Count - 1];
next = EncodeWatermark(new Watermark(last.RowKey, last.Partition));
}
return new RawPage<T>(page.Select(r => r.Item).ToList(), next);
}
private const int WireFormatVersion = 3;
public static string EncodeWatermark(Watermark watermark)
{
var json = JsonSerializer.Serialize(new Dictionary<string, object>
{
["v"] = WireFormatVersion,
["c"] = continuations.ToDictionary(
kv => kv.Key,
kv => (object?)new Dictionary<string, object?>
{
["rk"] = kv.Value.LastRowKey,
}),
};
var json = JsonSerializer.Serialize(doc);
["rk"] = watermark.RowKey,
["p"] = watermark.Partition,
});
return Convert.ToBase64String(Encoding.UTF8.GetBytes(json));
}
public static Dictionary<string, PartitionContinuation> DecodeMultiContinuation(string? raw)
/// <summary>Decodes a watermark; null for anything that is not a v3 watermark.</summary>
public static Watermark? DecodeWatermark(string? raw)
{
var result = new Dictionary<string, PartitionContinuation>(StringComparer.Ordinal);
if (string.IsNullOrEmpty(raw)) return result;
if (string.IsNullOrEmpty(raw)) return null;
try
{
var bytes = Convert.FromBase64String(raw);
using var doc = JsonDocument.Parse(bytes);
if (!doc.RootElement.TryGetProperty("c", out var entries)) return result;
foreach (var prop in entries.EnumerateObject())
{
// v1 back-compat: skip entries explicitly marked exhausted.
// v2 never writes them, but old in-flight tokens may.
if (prop.Value.TryGetProperty("x", out var xEl) && xEl.ValueKind == JsonValueKind.True)
continue;
using var doc = JsonDocument.Parse(Convert.FromBase64String(raw));
var root = doc.RootElement;
if (root.ValueKind != JsonValueKind.Object) return null;
if (!root.TryGetProperty("v", out var v) || v.ValueKind != JsonValueKind.Number
|| !v.TryGetInt32(out var version) || version != WireFormatVersion)
return null;
if (!root.TryGetProperty("rk", out var rk) || rk.ValueKind != JsonValueKind.String) return null;
if (!root.TryGetProperty("p", out var p) || p.ValueKind != JsonValueKind.String) return null;
string? rk = prop.Value.TryGetProperty("rk", out var rkEl) && rkEl.ValueKind == JsonValueKind.String
? rkEl.GetString()
: null;
result[prop.Name] = new PartitionContinuation(rk);
}
var rowKey = rk.GetString();
var partition = p.GetString();
return string.IsNullOrEmpty(rowKey) || partition == null ? null : new Watermark(rowKey, partition);
}
catch
catch (FormatException)
{
// Malformed continuation → treat as empty (start over). The wire-
// layer fingerprint already rejects tampered tokens before reaching here.
return null;
}
catch (JsonException)
{
return null;
}
return result;
}
}
}
@@ -498,7 +498,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
return groupIds;
}
public async Task<bool> RenameTenantGroupAsync(string groupId, string name)
public async Task<bool> UpdateTenantGroupAsync(string groupId, string? name, string? customerLabel)
{
if (string.IsNullOrWhiteSpace(groupId))
return false;
@@ -510,7 +510,10 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
var entity = result.Value;
if (entity == null) return false;
entity.Name = name?.Trim() ?? string.Empty;
if (name != null)
entity.Name = name.Trim();
if (customerLabel != null)
entity.CustomerLabel = customerLabel.Trim();
await _tenantGroupsTableClient.UpdateEntityAsync(entity, ETag.All);
return true;
}
@@ -1022,6 +1025,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
group.CreatedBy = entity.CreatedBy;
group.CreatedAt = entity.CreatedDate ?? default;
group.OwnerTenantId = string.IsNullOrEmpty(entity.OwnerTenantId) ? null : entity.OwnerTenantId;
group.CustomerLabel = string.IsNullOrEmpty(entity.CustomerLabel) ? null : entity.CustomerLabel;
return true;
}
@@ -78,27 +78,16 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
}
}
public Task<bool> SaveTenantConfigurationAsync(TenantConfiguration config)
=> SaveTenantConfigurationAsync(config, backupSource: null, backupReason: null);
public async Task<bool> SaveTenantConfigurationAsync(
TenantConfiguration config, string? backupSource, string? backupReason)
public async Task<bool> TryCreateTenantConfigurationAsync(TenantConfiguration config)
{
await TrySnapshotBeforeSaveAsync(
_tenantConfigTableClient, config.TenantId, "config",
ConvertFromTenantTableEntity, config, TenantBackupNoiseProperties,
config.UpdatedBy, backupSource, backupReason);
try
{
var entity = ConvertToTenantTableEntity(config);
await _tenantConfigTableClient.UpsertEntityAsync(entity, TableUpdateMode.Replace);
await _tenantConfigTableClient.AddEntityAsync(ConvertToTenantTableEntity(config));
return true;
}
catch (Exception ex)
catch (RequestFailedException ex) when (ex.Status == 409)
{
_logger.LogError(ex, "Error saving tenant configuration for {TenantId}", config.TenantId);
return false;
return false; // created concurrently — the caller re-reads and updates
}
}
@@ -134,6 +123,17 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
}
}
public async Task<bool> TryReplaceTenantConfigurationAsync(
TenantConfiguration config, string etag, string? backupSource, string? backupReason)
{
await TrySnapshotBeforeSaveAsync(
_tenantConfigTableClient, config.TenantId, "config",
ConvertFromTenantTableEntity, config, TenantBackupNoiseProperties,
config.UpdatedBy, backupSource, backupReason);
return await TryReplaceTenantConfigurationAsync(config, etag);
}
/// <summary>
/// Pre-write snapshot hook shared by the tenant- and admin-config save paths.
/// Fail-SOFT by design: this is a safety net around long-standing writers
@@ -327,23 +327,126 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
}
}
public async Task<bool> SaveAdminConfigurationAsync(AdminConfiguration config)
{
await TrySnapshotBeforeSaveAsync(
_adminConfigTableClient, "GlobalConfig", "config",
ConvertFromAdminTableEntity, config, AdminBackupNoiseProperties,
config.UpdatedBy, "admin-config", reason: null);
internal const int AdminUpdateMaxAttempts = 3;
/// <summary>Server stamps: written on every accepted change, never counted as one.</summary>
private static readonly HashSet<string> AdminStampColumns = new(StringComparer.Ordinal)
{
"PartitionKey", "RowKey", "LastUpdated", "UpdatedBy",
};
/// <summary>
/// The one write path of the admin row (D-285). Each attempt reads the raw row fresh with
/// its ETag and materializes it twice — <c>before</c> and the <c>draft</c> the mutation edits —
/// so the copy is exact for every persisted column. The changed columns are computed from
/// the two converter projections and laid onto the freshly read raw entity (a null value
/// drops the column); the conditional replace then writes exactly those plus the stamps.
/// Every other column keeps its stored value: the release pipeline's <c>LatestAgentV2*</c>,
/// columns of a newer build, retired columns. The former full-model replace wrote every
/// column from whatever copy the caller held, so a portal page loaded before an agent
/// release reverted the release's hashes on save (2026-09-25).
/// </summary>
public async Task<AdminConfigurationUpdateResult> UpdateAdminConfigurationAsync(
Func<AdminConfiguration, string?> mutate, string updatedBy, string? source = null)
{
if (mutate == null) throw new ArgumentNullException(nameof(mutate));
for (var attempt = 1; attempt <= AdminUpdateMaxAttempts; attempt++)
{
TableEntity? stored = null;
try
{
stored = (await _adminConfigTableClient.GetEntityAsync<TableEntity>("GlobalConfig", "config")).Value;
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
// No row yet: the first accepted change creates it from the defaults.
}
var before = stored != null ? ConvertFromAdminTableEntity(stored) : AdminConfiguration.CreateDefault();
var draft = stored != null ? ConvertFromAdminTableEntity(stored) : AdminConfiguration.CreateDefault();
var error = mutate(draft);
if (error != null)
return new AdminConfigurationUpdateResult { Error = error, Before = before, After = before };
var beforeColumns = ConvertToAdminTableEntity(before);
var draftColumns = ConvertToAdminTableEntity(draft);
var changed = ChangedAdminColumns(beforeColumns, draftColumns);
if (changed.Count == 0)
return new AdminConfigurationUpdateResult { Before = before, After = before };
draft.UpdatedBy = updatedBy;
draft.LastUpdated = DateTime.UtcNow;
await TrySnapshotBeforeSaveAsync(
_adminConfigTableClient, "GlobalConfig", "config",
ConvertFromAdminTableEntity, draft, AdminBackupNoiseProperties,
updatedBy, source ?? "admin-config", reason: null);
try
{
if (stored == null)
{
await _adminConfigTableClient.AddEntityAsync(ConvertToAdminTableEntity(draft));
}
else
{
foreach (var column in changed)
{
var value = draftColumns[column];
if (value == null) stored.Remove(column);
else stored[column] = value;
}
stored["LastUpdated"] = draft.LastUpdated;
stored["UpdatedBy"] = draft.UpdatedBy;
await _adminConfigTableClient.UpdateEntityAsync(stored, stored.ETag, TableUpdateMode.Replace);
}
return new AdminConfigurationUpdateResult { Before = before, After = draft, ChangedColumns = changed };
}
catch (RequestFailedException ex) when (ex.Status == 412 || ex.Status == 409)
{
// Someone wrote (or created) the row since this attempt's read: re-read and
// re-apply the mutation onto their state instead of overwriting it.
_logger.LogInformation(
"Admin configuration changed concurrently (attempt {Attempt}/{Max}) — re-reading",
attempt, AdminUpdateMaxAttempts);
await Task.Delay(TimeSpan.FromMilliseconds(150));
}
}
throw new InvalidOperationException(
$"Admin configuration update lost {AdminUpdateMaxAttempts} conditional-write races — retry shortly.");
}
/// <summary>
/// Columns whose projected value differs between the two converter outputs. Both sides come
/// from <see cref="ConvertToAdminTableEntity"/>, so value types match and plain equality holds.
/// </summary>
internal static List<string> ChangedAdminColumns(TableEntity before, TableEntity after)
{
var changed = new List<string>();
foreach (var column in after.Keys.Union(before.Keys, StringComparer.Ordinal))
{
if (AdminStampColumns.Contains(column)) continue;
before.TryGetValue(column, out var b);
after.TryGetValue(column, out var a);
if (!Equals(b, a)) changed.Add(column);
}
return changed;
}
public async Task<bool> CreateAdminConfigurationIfMissingAsync(AdminConfiguration config)
{
try
{
var entity = ConvertToAdminTableEntity(config);
await _adminConfigTableClient.UpsertEntityAsync(entity, TableUpdateMode.Replace);
await _adminConfigTableClient.AddEntityAsync(ConvertToAdminTableEntity(config));
return true;
}
catch (Exception ex)
catch (RequestFailedException ex) when (ex.Status == 409)
{
_logger.LogError(ex, "Error saving admin configuration");
return false;
return false; // A row exists — never replace it with defaults.
}
}
@@ -846,7 +949,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
EnrollmentSummaryTimeoutSeconds = entity.GetInt32("EnrollmentSummaryTimeoutSeconds"),
EnrollmentSummaryBrandingImageUrl = entity.GetString("EnrollmentSummaryBrandingImageUrl"),
EnrollmentSummaryLaunchRetrySeconds = entity.GetInt32("EnrollmentSummaryLaunchRetrySeconds"),
HelloWaitTimeoutSeconds = entity.GetInt32("HelloWaitTimeoutSeconds") ?? 30,
HelloWaitTimeoutSeconds = entity.GetInt32("HelloWaitTimeoutSeconds") ?? HelloWaitTimeout.DefaultSeconds,
AgentMaxLifetimeMinutes = entity.GetInt32("AgentMaxLifetimeMinutes"),
SendTraceEvents = entity.GetBoolean("SendTraceEvents") ?? true,
EnableLocalAdminAnalyzer = entity.GetBoolean("EnableLocalAdminAnalyzer"),
@@ -931,10 +1034,11 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
{ "OpsAlertSlackEnabled", config.OpsAlertSlackEnabled },
{ "OpsAlertSlackWebhookUrl", config.OpsAlertSlackWebhookUrl ?? string.Empty },
{ "OpsNotificationChannelsJson", config.OpsNotificationChannelsJson ?? string.Empty },
// Per-line agent binary integrity (written by build scripts via Merge).
// Per-line agent binary integrity: written by the release pipeline via Merge.
// The backend never changes them (the PATCH gate rejects them), so the
// changed-columns write (UpdateAdminConfigurationAsync) never touches them.
// V2 is the only wired line; future V3 = add field set here. Retired columns
// (V1-suffix and the even older unsuffixed "LatestAgent*") are never written,
// so the next Save evicts them implicitly on overwrite.
// (V1-suffix and the even older unsuffixed "LatestAgent*") are not projected.
{ "AllowAgentDowngrade", config.AllowAgentDowngrade },
{ "LatestAgentV2Version", config.LatestAgentV2Version ?? string.Empty },
{ "LatestAgentV2Sha256", config.LatestAgentV2Sha256 ?? string.Empty },
@@ -18,6 +18,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
/// - F3 rule regression: "ruleregression|{ruleId-lower}" (trimmed; payload-carrying rows
/// that are refreshed while the episode is active and deleted on re-arm)
/// - app-version duration regression: "appversionregression|{app-lower}|{version-lower}"
/// (non-IME channels: "appversionregression|{source}:{app-lower}|{version-lower}")
/// (trimmed + table-key-sanitized; same payload-carrying episode semantics — raw
/// AppName/CurrentVersion live in columns, the mapper never parses the RowKey)
/// Race-safe via AddEntityAsync: Azure Table Storage returns 409 Conflict if the entity already exists.
@@ -309,7 +310,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
}
}
public async Task DeleteAppVersionRegressionAsync(string tenantId, string appName, string currentVersion)
public async Task DeleteAppVersionRegressionAsync(string tenantId, string source, string appName, string currentVersion)
{
if (string.IsNullOrWhiteSpace(tenantId)
|| string.IsNullOrWhiteSpace(appName)
@@ -320,7 +321,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
try
{
await _table.DeleteEntityAsync(
tenantId.ToLowerInvariant(), BuildAppVersionRegressionRowKey(appName, currentVersion));
tenantId.ToLowerInvariant(), BuildAppVersionRegressionRowKey(source, appName, currentVersion));
}
catch (RequestFailedException ex) when (ex.Status == 404)
{
@@ -576,10 +577,14 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
/// two apps share an episode row — tolerable); the raw values live in columns and the
/// mapper reads them, never the RowKey.
/// </summary>
internal static string BuildAppVersionRegressionRowKey(string appName, string version)
internal static string BuildAppVersionRegressionRowKey(string? source, string appName, string version)
{
var app = SanitizeTableKey((appName ?? string.Empty).Trim().ToLowerInvariant());
var ver = SanitizeTableKey((version ?? string.Empty).Trim().ToLowerInvariant());
// IME keys predate channels and keep their shape; other channels prefix the app part.
var channel = AppInstallSources.Normalize(source);
if (channel != AppInstallSources.Ime)
app = $"{SanitizeTableKey(channel)}:{app}";
return $"{AppVersionRegressionRowKeyPrefix}{app}|{ver}";
}
@@ -598,10 +603,11 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
internal static TableEntity BuildAppVersionRegressionEntity(string tenantId, AppVersionRegressionAlert alert)
{
return new TableEntity(
tenantId.ToLowerInvariant(), BuildAppVersionRegressionRowKey(alert.AppName, alert.CurrentVersion))
tenantId.ToLowerInvariant(), BuildAppVersionRegressionRowKey(alert.Source, alert.AppName, alert.CurrentVersion))
{
["TenantId"] = tenantId,
["AppName"] = alert.AppName,
["Source"] = AppInstallSources.Normalize(alert.Source),
["CurrentVersion"] = alert.CurrentVersion,
["PreviousVersion"] = alert.PreviousVersion ?? string.Empty,
["CurrentMedianSeconds"] = alert.CurrentMedianSeconds,
@@ -620,6 +626,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
{
TenantId = entity.GetString("TenantId") ?? entity.PartitionKey,
AppName = entity.GetString("AppName") ?? string.Empty,
// Rows written before channels existed are IME episodes.
Source = AppInstallSources.Normalize(entity.GetString("Source")),
CurrentVersion = entity.GetString("CurrentVersion") ?? string.Empty,
PreviousVersion = entity.GetString("PreviousVersion") ?? string.Empty,
CurrentMedianSeconds = entity.GetInt32("CurrentMedianSeconds") ?? 0,
@@ -282,6 +282,7 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
["LatestStatus"] = p.LatestStatus,
["LatestUpdatedAt"] = p.LatestUpdatedAt,
["OffboardCount"] = p.OffboardCount,
["TrialConsumed"] = p.TrialConsumed,
};
private static OffboardingByTenantPointer MapByTenantPointer(TableEntity e) => new()
@@ -293,6 +294,8 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
LatestStatus = e.GetString("LatestStatus") ?? "Initiated",
LatestUpdatedAt = e.GetDateTime("LatestUpdatedAt") ?? default,
OffboardCount = e.GetInt32("OffboardCount") ?? 1,
// Absent on pointers from before the column; false is correct for all of them (D-288).
TrialConsumed = e.GetBoolean("TrialConsumed") ?? false,
};
}
}
@@ -115,60 +115,35 @@ namespace AutopilotMonitor.Functions.DataAccess.TableStorage
// partitions would just trade certainty for cost — the coverage test guards the list instead.
internal static readonly string[] AllCategories = OpsEventCategory.All;
private async Task<RawPage<OpsEventEntry>> FanOutAcrossCategoriesAsync(
private Task<RawPage<OpsEventEntry>> FanOutAcrossCategoriesAsync(
DateTime? dateFrom, DateTime? dateTo, int pageSize, string? continuation,
OpsEventQueryFilters? filters = null)
{
var continuations = PerPartitionFanOutMerge.DecodeMultiContinuation(continuation);
// First-page request fans out across every category; subsequent
// pages restrict to whatever is still active in the continuation
// map. With only 6 fixed categories this never grew to a problem
// size, but the convention now matches the audit fan-out so the
// shared fan-out helper has one rule.
var isFirstPage = string.IsNullOrEmpty(continuation);
var activeCats = AllCategories
.Where(cat => isFirstPage || continuations.ContainsKey(cat))
.ToList();
if (activeCats.Count == 0)
return new RawPage<OpsEventEntry>(new List<OpsEventEntry>(), null);
var fetchTasks = activeCats.Select(async cat =>
{
continuations.TryGetValue(cat, out var catContinuation);
var filter = BuildFilterWithRowKeyBound(cat, dateFrom, dateTo, catContinuation?.LastRowKey, filters);
var fetched = new List<(string RowKey, OpsEventEntry Item)>();
await foreach (var e in _table.QueryAsync<TableEntity>(filter: filter, maxPerPage: pageSize))
return PerPartitionFanOutMerge.FetchPageAsync<OpsEventEntry>(
AllCategories, pageSize, continuation,
async (category, bound, limit, ct) =>
{
fetched.Add((e.RowKey, MapToEntry(e)));
if (fetched.Count >= pageSize) break;
}
return new PerPartitionFanOutMerge.PartitionFetchResult<OpsEventEntry>(cat, fetched);
}).ToList();
var results = await Task.WhenAll(fetchTasks);
var (items, nextContinuations) = PerPartitionFanOutMerge.MergeAndAdvance(
results, continuations, pageSize, e => e.Timestamp);
// Map only carries active partitions; empty = pagination complete.
string? nextRawToken = nextContinuations.Count > 0
? PerPartitionFanOutMerge.EncodeMultiContinuation(nextContinuations)
: null;
return new RawPage<OpsEventEntry>(items, nextRawToken);
var filter = BuildFilterWithRowKeyBound(category, dateFrom, dateTo, bound, filters);
var read = new List<(string RowKey, OpsEventEntry Item)>();
await foreach (var e in _table.QueryAsync<TableEntity>(filter: filter, maxPerPage: limit, cancellationToken: ct))
{
read.Add((e.RowKey, MapToEntry(e)));
if (read.Count >= limit) break;
}
return read;
});
}
internal static string BuildFilterWithRowKeyBound(
string category, DateTime? dateFrom, DateTime? dateTo, string? lastRowKey,
string category, DateTime? dateFrom, DateTime? dateTo, PerPartitionFanOutMerge.RowKeyBound? rowKeyBound,
OpsEventQueryFilters? filters = null)
{
var clauses = new List<string>
{
$"PartitionKey eq '{category.Replace("'", "''")}'",
};
if (!string.IsNullOrEmpty(lastRowKey))
clauses.Add($"RowKey gt '{lastRowKey!.Replace("'", "''")}'");
if (rowKeyBound != null)
clauses.Add(rowKeyBound.ToODataClause());
if (dateFrom.HasValue)
clauses.Add(BusinessTimestamp.OpsDateFromClause(ToUtc(dateFrom.Value)));
if (dateTo.HasValue)
@@ -23,8 +23,9 @@ namespace AutopilotMonitor.Functions.Functions.Admin;
/// repository directly) so the delegated-scope cache is invalidated in lockstep.
///
/// Audit: access-affecting mutations are logged under the <b>affected managed tenant(s)</b>' trail (so the
/// customer sees "who can read my tenant" / "access removed"). Group create/rename carry no tenant context
/// and are not customer-visible access changes — they are logged operationally only (no AuditLogs partition).
/// customer sees "who can read my tenant" / "access removed"); the group appears there under its customer-facing
/// name (<see cref="DelegationSelfService.CustomerNameOf"/>), never its internal name. Group create/rename/label
/// carry no access change — they are logged operationally only (no AuditLogs partition).
/// </summary>
public class TenantGroupManagementFunction
{
@@ -100,8 +101,11 @@ public class TenantGroupManagementFunction
}
/// <summary>
/// PATCH /api/global/tenant-groups/{groupId} — rename a group. GlobalAdminOnly.
/// Body: { "name": "..." }. A rename has no scope effect.
/// PATCH /api/global/tenant-groups/{groupId} — rename a group and/or set the customer label of an operator
/// group. GlobalAdminOnly. Body: { "name"?: "...", "customerLabel"?: "..." } (only the fields to change;
/// an empty label clears it). Neither has a scope effect; the label only changes how the access is named to
/// the member tenants (<see cref="DelegationSelfService.CustomerNameOf"/>), so it is logged operationally
/// like a rename, not audited as an access change.
/// </summary>
[Function("UpdateTenantGroup")]
[Authorize]
@@ -114,14 +118,55 @@ public class TenantGroupManagementFunction
var read = await req.ReadAsync<UpdateTenantGroupRequest>();
if (read.Error != null) return read.Error;
var body = read.Value!;
if (string.IsNullOrWhiteSpace(body.Name))
return await Bad(req, "name is required");
var validationError = ValidateUpdateRequest(body);
if (validationError != null)
return await Bad(req, validationError);
if (!await _delegatedAdminService.RenameGroupAsync(groupId, body.Name))
return await NotFound(req);
_logger.LogInformation("Tenant group renamed: {GroupId} -> '{Name}' by {By}", groupId, body.Name, currentUpn);
return await UpdateGroupCoreAsync(groupId, body, currentUpn) switch
{
GroupUpdateOutcome.NotFound => await NotFound(req),
GroupUpdateOutcome.LabelOnOwnedGroup => await req.ConflictAsync(
"A self-service group has no customer label: its customers see the managing organization's name."),
_ => await req.OkAsync(new MessageResponse { Message = "Group updated" }),
};
}
return await req.OkAsync(new MessageResponse { Message = "Group updated" });
/// <summary>Outcome of <see cref="UpdateGroupCoreAsync"/>.</summary>
internal enum GroupUpdateOutcome { Updated, NotFound, LabelOnOwnedGroup }
/// <summary>Shape check of a PATCH body (400 message, or null when valid): at least one field, no blank name, a bounded label.</summary>
internal static string? ValidateUpdateRequest(UpdateTenantGroupRequest body)
{
if (body.Name == null && body.CustomerLabel == null)
return "name or customerLabel is required";
if (body.Name != null && string.IsNullOrWhiteSpace(body.Name))
return "name must not be empty";
if (body.CustomerLabel != null && body.CustomerLabel.Trim().Length > DelegationSelfService.CustomerLabelMaxLength)
return $"customerLabel must be at most {DelegationSelfService.CustomerLabelMaxLength} characters";
return null;
}
/// <summary>
/// Testable core of the group update (body already validated). A customer label is refused on an OWNED
/// (self-service) group — its customers see the managing tenant's name, so a label there would be dead data.
/// </summary>
internal async Task<GroupUpdateOutcome> UpdateGroupCoreAsync(string groupId, UpdateTenantGroupRequest body, string? currentUpn)
{
if (body.CustomerLabel != null)
{
var group = await _delegatedAdminService.GetGroupAsync(groupId);
if (group == null)
return GroupUpdateOutcome.NotFound;
if (!string.IsNullOrWhiteSpace(group.OwnerTenantId))
return GroupUpdateOutcome.LabelOnOwnedGroup;
}
if (!await _delegatedAdminService.UpdateGroupAsync(groupId, body.Name, body.CustomerLabel))
return GroupUpdateOutcome.NotFound;
_logger.LogInformation("Tenant group updated: {GroupId} name='{Name}' customerLabel='{CustomerLabel}' by {By}",
groupId, body.Name, body.CustomerLabel, currentUpn);
return GroupUpdateOutcome.Updated;
}
/// <summary>DELETE /api/global/tenant-groups/{groupId} — delete group + all its assignments. GlobalAdminOnly.</summary>
@@ -173,7 +218,7 @@ public class TenantGroupManagementFunction
await AuditPerTenantAsync(group.TenantIds, "DELETE", "*", currentUpn,
new Dictionary<string, string>
{
{ "Group", group.Name },
{ "Group", DelegationSelfService.CustomerNameOf(group) },
{ "GroupId", groupId },
{ "Reason", "group-deleted" },
{ "AssigneeCount", group.AssigneeCount.ToString() },
@@ -335,7 +380,7 @@ public class TenantGroupManagementFunction
await AuditPerTenantAsync(group.TenantIds, "CREATE", upn, currentUpn,
new Dictionary<string, string>
{
{ "Group", group.Name },
{ "Group", DelegationSelfService.CustomerNameOf(group) },
{ "GroupId", groupId },
{ "Role", role },
{ "Reason", "group-assigned" },
@@ -384,7 +429,7 @@ public class TenantGroupManagementFunction
await AuditPerTenantAsync(group.TenantIds, "DELETE", normalizedUpn, currentUpn,
new Dictionary<string, string>
{
{ "Group", group.Name },
{ "Group", DelegationSelfService.CustomerNameOf(group) },
{ "GroupId", groupId },
{ "Reason", "group-unassigned" },
});
@@ -100,19 +100,22 @@ public class TenantOffboardFunction
"TENANT OFFBOARD initiated for tenant {TenantId} by {Upn}",
normalizedTenantId, upn);
// 2. Capture DomainName from TenantConfiguration so the worker's Completed/Failed
// OpsEvents can render "{domain} ({tenantId})" instead of the bare GUID.
string? domainName = null;
// 2. Capture what must outlive the TenantConfiguration row the cascade deletes: DomainName
// for the worker's Completed/Failed OpsEvents ("{domain} ({tenantId})"), and TrialConsumed
// for the pointer, which carries it to the returning tenant's first login (D-288).
// Fail-loud: without the read the one-trial-per-tenant rule would silently reset.
string? domainName;
bool trialConsumed;
try
{
var existingConfig = await _configRepo.GetTenantConfigurationAsync(targetTenantId);
domainName = existingConfig?.DomainName;
trialConsumed = existingConfig?.TrialConsumed ?? false;
}
catch (Exception ex)
{
_logger.LogWarning(ex,
"Failed to read tenant configuration for {TenantId} before offboard; proceeding without domain name",
normalizedTenantId);
_logger.LogError(ex, "Failed to read tenant configuration for {TenantId} before offboard", normalizedTenantId);
return await Build500Async(req, "Failed to read tenant configuration");
}
// 2.b — Capture the Preview-Notification-Email before Phase 2.D wipes the
@@ -184,14 +187,14 @@ public class TenantOffboardFunction
// 5. Upsert ByTenant pointer with ETag-CAS so OffboardCount increments race-safely.
try
{
await UpsertPointerWithCasAsync(normalizedTenantId, historyRowKey, now);
await UpsertPointerWithCasAsync(normalizedTenantId, historyRowKey, now, trialConsumed);
}
catch (Exception ex)
{
// History row is already committed — operator can find it under
// PartitionKey="OffboardingHistory" and retry manually. Graceful degradation
// (plan §4.4): pointer is the re-onboarding index; without it PR4's auto-wipe
// is no-op but the offboarding itself still proceeds.
// PartitionKey="OffboardingHistory". Nothing destructive has started: without the
// pointer the returning tenant would get its trial back, so refuse and let the
// admin re-click.
_logger.LogError(ex,
"Failed to upsert OffboardingByTenant pointer for {TenantId} after history row inserted",
normalizedTenantId);
@@ -722,7 +725,7 @@ public class TenantOffboardFunction
/// <para>
/// Drain-deadline math the caller applies:
/// <c>effectiveDeadline = max(history.EarliestProcessingAt, TombstoneLastUpdated + DrainBarrier)</c>.
/// Because the tombstone's <c>LastUpdated</c> is part of the same atomic Save as the
/// Because the tombstone's <c>LastUpdated</c> is part of the same atomic write as the
/// Disabled-flip, it is always present once the auth-gate is committed — even if
/// the History-row patch later fails on a fast re-click.
/// </para>
@@ -734,7 +737,7 @@ public class TenantOffboardFunction
internal enum EnsureDisabledOutcome
{
/// <summary>Row was already a Disabled=true tombstone with the offboarding reason —
/// no Save was issued. Existing <c>LastUpdated</c> is returned.</summary>
/// nothing was written. Existing <c>LastUpdated</c> is returned.</summary>
AlreadyTombstone,
/// <summary>We actively wrote the Disabled=true gate (either flipped an existing
/// row or created a fresh one). <c>LastUpdated</c> is the timestamp of this write.</summary>
@@ -759,70 +762,54 @@ public class TenantOffboardFunction
/// <para>
/// Behaviour:
/// <list type="bullet">
/// <item>Existing TenantConfiguration row → read-modify-write so signup data + custom
/// rate limits etc. are preserved. Idempotent: if already
/// <c>Disabled=true</c> with the offboarding reason, no Save is issued and
/// <item>Existing TenantConfiguration row → conditional write on the fresh row
/// (<see cref="TenantConfigurationService.CreateOrUpdateAsync"/>) so signup data + custom
/// rate limits etc. are preserved and no concurrent write is rewound. Idempotent: if already
/// <c>Disabled=true</c> with the offboarding reason, nothing is written and
/// <see cref="EnsureDisabledResult.AlreadyTombstone"/> is returned.</item>
/// <item>Missing TenantConfiguration row (edge case: very old tenant / manual delete)
/// → create a fresh <c>CreateDefault</c> row with <c>Disabled=true</c>. Necessary
/// because <c>AuthFunction</c> would otherwise auto-create a <c>Disabled=false</c>
/// → create a fresh <c>CreateDefault</c> row with <c>Disabled=true</c> (conditional insert).
/// Necessary because <c>AuthFunction</c> would otherwise auto-create a <c>Disabled=false</c>
/// default on the next <c>/api/auth/me</c> and slip the offboarded tenant back through.</item>
/// <item><c>SaveTenantConfigurationAsync</c> returns <c>false</c> on transient storage
/// failure (the production repo does NOT throw; see Codex-finding TableConfigRepository).
/// We turn that into <see cref="InvalidOperationException"/> so the caller treats it
/// as a hard fail.</item>
/// <item>Every conditional write lost to concurrent changes, or storage failed →
/// throws, so the caller treats it as a hard fail.</item>
/// </list>
/// </para>
/// </summary>
internal async Task<EnsureDisabledResult> EnsureTenantDisabledAsync(string targetTenantId, string upn)
{
var tenantConfig = await _configRepo.GetTenantConfigurationAsync(targetTenantId);
if (tenantConfig != null)
var update = await _tenantConfigService.CreateOrUpdateAsync(targetTenantId, row =>
{
// Idempotent: if Disabled is already true with our reason, no-op the write so
// resume-clicks do not thrash the LastUpdated timestamp on the row.
var alreadyDisabled = tenantConfig.Disabled
&& string.Equals(tenantConfig.DisabledReason, OffboardingDisabledReason, StringComparison.Ordinal);
if (alreadyDisabled)
if (string.IsNullOrEmpty(row.DomainName))
{
_tenantConfigService.InvalidateCache(targetTenantId);
return new EnsureDisabledResult(EnsureDisabledOutcome.AlreadyTombstone, tenantConfig.LastUpdated);
// Missing-row edge case: the default is written as the tombstone so AuthFunction's
// auto-create-default cannot bring the tenant back enabled.
_logger.LogWarning(
"TenantOffboard: tenant configuration for {TenantId} has no domain (missing or never onboarded) — writing the Disabled=true tombstone onto it",
targetTenantId);
}
}
else
{
// Missing-row edge case: create a fresh Disabled=true row so AuthFunction's
// auto-create-default cannot bring the tenant back enabled.
tenantConfig = TenantConfiguration.CreateDefault(targetTenantId);
_logger.LogWarning(
"TenantOffboard: no TenantConfiguration row found for {TenantId} — creating fresh Disabled=true tombstone",
targetTenantId);
}
row.Disabled = true;
row.DisabledReason = OffboardingDisabledReason;
row.DisabledUntil = null;
row.UpdatedBy = upn;
return true;
}, "offboard", "Disabled-gate before cascade");
tenantConfig.Disabled = true;
tenantConfig.DisabledReason = OffboardingDisabledReason;
tenantConfig.DisabledUntil = null;
tenantConfig.UpdatedBy = upn;
// Explicit LastUpdated stamp — we bypass TenantConfigurationService.SaveConfigurationAsync
// (which would set this) and write to the repo directly, so we own the bookkeeping.
// Critical for the resume-path drain-deadline math: this timestamp is the
// authoritative "when did the cache-drain barrier start" anchor.
var writtenAt = DateTime.UtcNow;
tenantConfig.LastUpdated = writtenAt;
var saved = await _configRepo.SaveTenantConfigurationAsync(tenantConfig, "offboard", "Disabled-gate before cascade");
if (!saved)
switch (update.Status)
{
// TableConfigRepository.SaveTenantConfigurationAsync returns false on transient
// storage failure instead of throwing. Without throwing here, the caller would
// proceed to enqueue without the auth-gate committed — defeating the whole
// cache-drain barrier. Surface as a hard failure.
throw new InvalidOperationException(
$"SaveTenantConfigurationAsync returned false for tenant {targetTenantId}; Disabled-gate NOT committed");
case TenantConfigUpdateStatus.OffboardingInProgress:
// Idempotent: the tombstone is already in place — nothing written, so LastUpdated
// stays the authoritative "when did the cache-drain barrier start" anchor.
_tenantConfigService.InvalidateCache(targetTenantId);
return new EnsureDisabledResult(EnsureDisabledOutcome.AlreadyTombstone, update.Config!.LastUpdated);
case TenantConfigUpdateStatus.Updated:
// LastUpdated is stamped by the write itself — the anchor of the drain-deadline math.
return new EnsureDisabledResult(EnsureDisabledOutcome.WroteNewTombstone, update.Config!.LastUpdated);
default:
// Without Disabled=true the cache-drain barrier offers no protection: hard fail.
throw new InvalidOperationException(
$"Disabled-gate NOT committed for tenant {targetTenantId} ({update.Status})");
}
_tenantConfigService.InvalidateCache(targetTenantId);
return new EnsureDisabledResult(EnsureDisabledOutcome.WroteNewTombstone, writtenAt);
}
/// <summary>
@@ -869,11 +856,10 @@ public class TenantOffboardFunction
}
// Plan §4.4 read-modify-write CAS-loop. Bounded retry; throws on exhaustion so the
// caller turns it into a 500. The pointer is the O(1) re-onboarding index — failure
// here is recoverable (graceful degradation in PR4) but we still surface it so the
// operator can investigate.
// caller turns it into a 500. The pointer is the O(1) re-onboarding index; TrialConsumed
// is sticky across rounds (OR), never reset by an offboarding without a trial.
internal async Task UpsertPointerWithCasAsync(
string normalizedTenantId, string historyRowKey, DateTime now)
string normalizedTenantId, string historyRowKey, DateTime now, bool trialConsumed)
{
for (var attempt = 1; attempt <= PointerCasMaxAttempts; attempt++)
{
@@ -890,6 +876,7 @@ public class TenantOffboardFunction
LatestStatus = "Initiated",
LatestUpdatedAt = now,
OffboardCount = 1,
TrialConsumed = trialConsumed,
};
try
@@ -910,6 +897,7 @@ public class TenantOffboardFunction
existing.LatestStatus = "Initiated";
existing.LatestUpdatedAt = now;
existing.OffboardCount = existing.OffboardCount + 1;
existing.TrialConsumed |= trialConsumed;
try
{

Some files were not shown because too many files have changed in this diff Show More