Wire the V2 agent runtime that feeds the DecisionCore RealmJoin gate from Phase A.
On every Autopilot session the agent now observes the RJ deployment-state registry
keys and dual-emits DecisionSignals + InformationalEvents so the engine extends the
completion AND-gate while RJ is in flight.
Agent.V2.Core additions:
- RealmJoinInfo: registry constants + DeploymentPhase/Package readers
- UserSidResolver: advapi32 LookupAccountName + ConvertSidToStringSid P/Invoke
- RealmJoinWatcher: three RegistryWatchers (HKLM Parameters, HKLM Packages subtree,
HKU\<sid> Packages subtree). Lazy attach with 2 s retry, 500 ms debounce, catch-up
scan on Start() so pre-existing RJ state emits the matching one-shot events
immediately. Strict dedup on Detected/Resolved/Timeout + (scope, packageId).
- RealmJoinWatcherAdapter: maps each watcher event to (1) a new DecisionSignalKind
post + (2) a SharedConstants.EventTypes.RealmJoin* InformationalEvent. Package
DisplayName truncated to 256 chars before posting.
- RealmJoinHost: ICollectorHost with compile-time kill-switch RealmJoinTrackingEnabled.
- DesktopArrivalDetector: new RealUserOwnerObserved event raised before DesktopArrived
so the composition root can resolve the user's SID for the HKU watcher arm.
- DesktopArrivalHost: new onRealUserOwnerObserved callback param.
- DefaultComponentFactory: instantiates RealmJoinHost after AadJoinHost and routes the
DesktopArrival owner through UserSidResolver -> RealmJoinHost.ArmHkuWatcher.
Shared additions:
- SharedConstants.EventTypes adds realmjoin_{detected,phase_changed,resolved,timeout,
package_started,package_completed} so the engine + adapter + future tests all
reference the same strings.
- DecisionEngine.RealmJoinPayloadKeys promoted from internal to public so the
cross-assembly adapter can assemble signal payloads with the canonical key names.
Verified: full solution build clean; V2.Core tests 1124/1125 (the lone failure is the
documented WhiteGlove RecoveryPath flake — re-runs green in isolation, no Phase B link).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add a new completion-gate to the V2 enrollment AND-gate so the session keeps
running while a RealmJoin (RJ) deployment is in flight. The agent detects RJ
via HKLM\SYSTEM\CurrentControlSet\Services\realmjoin\Parameters and tracks
per-package installs from HKLM\SOFTWARE\RealmJoin\Packages\<id> + the
HKEY_USERS\<sid>\... counterpart; runtime wiring lands in Phase B.
Decision-layer changes:
- 5 new DecisionSignalKinds (RealmJoinDetected/Resolved/Timeout/PackageStarted/
PackageCompleted) routed through the existing reducer dispatch.
- New DeadlineNames.RealmJoinTimeout (60-min hard cap, hardcoded).
- New immutable RealmJoinFacts + RealmJoinPackageFact on DecisionState; schema
bumped v2 -> v3 (old snapshots load with RealmJoinFacts.Empty default).
- Classic Hello+Desktop AND-gate (HandleHelloResolvedV1, HandleDesktopArrivedV1
Hello-disabled fast-path + helloAlreadyResolved branch, HelloSafety deadline)
and SelfDeploying terminal path now AND with RealmJoinGateOpen(). When closed,
the prerequisite facts are still recorded but TransitionToFinalizing /
WithStage(Completed) is deferred until the RJ resolved/timeout handler
releases the gate via CompleteIfDeferredOrBookkeep.
- Per-package facts capped at 200 rows; DisplayName truncated to 256 chars.
- Audit-trail attaches realmjoinDetectedUtc/Outcome/LastPhase/ResolvedUtc/Packages
on terminal events when RJ was active.
Verified: DecisionCore tests 313/313, Agent V2.Core tests 1125/1125.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Session 9d052230 dropped enrollment_failed with reason esp_terminal_failure
but the underlying HRESULT 0x87d1041c (Application not detected after install)
sat unparsed in the ESP subcategory statusText, and the agent terminated before
IME could emit the matching app_install_failed. Two-part fix:
- ProvisioningStatusTracker arms a 30s settle window between ESP-subcategory
failure detection and EspFailureDetected fire. Gives ImeLogTracker a chance
to surface the app-level failure on the timeline. Emits esp_failure_settle_started
for observability.
- HRESULT extracted via language-invariant regex from the failed subcategory
statusText, surfaced top-level on esp_provisioning_status (failedSubcategoryErrorCode)
and propagated through new EspFailureDetectedEventArgs to the EspTerminalFailure
signal payload. DecisionEngine.Edge writes failureType/errorCode/failedSubcategory/
category onto the enrollment_failed event parameters. UI renders an HRESULT badge
with description from errorCodeMap.ts (analogous to app_install_failed).
V2-only; V1 (Agent.Core) untouched. ConfigVersion unchanged.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reads BlockInStatusPage (bitmask: AllowReset / AllowTryAgain /
AllowContinueAnyway) and SyncFailureTimeout from
HKLM\SOFTWARE\Microsoft\Enrollments\{guid}\FirstSync alongside the
existing skip flags. Two new SignalFacts in EnrollmentScenarioObservations
(EspSyncFailureTimeoutMinutes, EspAllowContinueAnyway) carry the values
into the reducer so the DecisionEngine can emit substantiated
terminal-failure verdicts:
- app_install_failed message now reports the actual ESP timeout
("ESP timed out (60 min) while still installing.") when the
FirstSync value is known; falls back to the generic wording.
- enrollment_failed event payload carries espSyncFailureTimeoutMinutes
+ espAllowContinueAnyway. When ContinueAnyway is true a
mayHaveContinuedAnyway flag + continueAnywayHint string are added
so the timeline shows the user may have dismissed ESP via the
"Continue anyway" button and reached the desktop even though the
agent saw a terminal failure.
- esp_config_detected event DataJson now also exposes the raw
blockInStatusPage value, decoded allowReset / allowTryAgain /
allowContinueAnyway flags, and syncFailureTimeoutMinutes for audit.
Bitmask semantics follow Microsoft's Get-AutopilotDiagnostics.ps1
(bit 1 = AllowReset, bit 2 = AllowTryAgain, bit 4 = ContinueAnyway).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Agent V2: when RecoveryPath loads a snapshot already at Stage=Completed/Failed,
no reducer transition fires so OnDecisionTerminalStage was never invoked — the
agent idled until AgentMaxLifetime (default 6h), emitting periodic perf/metrics
events the whole time. Fire it explicitly at the end of Start() so the
EnrollmentTerminationHandler runs (marker write + diagnostics + self-destruct).
The WhiteGloveSealed archive-and-reset at step 0 already neutralises the
Part-2 case; non-terminal stages take the normal recovery path.
Web: gate the SignalR eventStream refetch and the 30s fallback polling on
non-terminal session status. Once Succeeded/Failed, late perf/metrics arrivals
no longer trigger a full-page refetch + re-render that scroll-jumps the
timeline while the user is reading.
Observed in session 17ef0e0e-... (2026-05-19): 4x enrollment_complete, only 1x
agent_shutting_down; reboot mid-shutdown left subsequent agent instances
running for hours emitting noise.
Tests: 3 new in RecoveryPathTests (Completed/Failed -> Terminated, non-terminal
-> no Terminated), 4 in sessionStatus.test.ts. Suites grun: Web 384/384,
Agent V2.Core 1095/1095, Backend 1844/1844.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- docs/architecture.md Diagnostics section rewritten: both destinations
(CustomerSas default + Hosted opt-in), new download routing on shape,
cascade-delete step §5b with SAS-delete-permission gating, and the V2
remote-config retry + observability events.
- Submodule .claude/commands bumped to 17a6f5b for the dual-destination
fetch-diag-zip.sh refactor + skill error-handling note updates.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Motivating incident — session 8f2bef72 (2026-05-19, OK's VM test): a Function
App cold-start triggered by a fresh deploy stalled the agent's single-shot
GetAgentConfig call for ~60 s; the client gave up, RemoteConfigService
silently fell back to built-in defaults (ConfigVersion=0,
SelfDestructOnComplete=true, DiagnosticsUploadMode=Off), and the entire
enrollment ran the wrong policy with no wire-visible trace of the degraded
mode. ReportAgentError (the emergency channel) was also racing the cold
start and lost to 499.
Three fixes:
1. Retry FetchConfigAsync on transient errors (3 attempts; 10/30/60 s linear
backoff). BackendAuthException still bails immediately — 401/403 won't
change with a retry and an operator needs the AuthCertificateRejected /
DeviceNotRegistered distress signal fast. EmergencyReporter only fires on
the final attempt to give the function time to warm. Live ServerAction
rotate_config keeps single-shot so it doesn't block the action handler
for ~100 s.
2. agent_started DataJson now carries configVersion + remoteConfigFetched +
remoteConfigOutcome (Succeeded / FromCache / UsedDefaults / NotAttempted).
Operators can tell at a glance whether the agent is running on fresh
tenant config vs. defaults vs. stale cache without inferring from
absence of expected tenant-controlled knobs.
3. New remote_config_fetch_failed Warning event emitted right after
agent_started when the fallback path was taken. Carries attempts,
failureType, failureMessage (truncated 512), and authStatusCode when
relevant. Closes the blind spot — the only prior trace was a Warning
in the local agent log which is gone after a SelfDestruct cleanup.
V2 Agent suite: 1092/1092 (+16: 7 retry behaviour + 9 emitter shape).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The red color treatment + warning-triangle icon on the Hosted-storage
option implied misconfiguration. Hosted is a deliberately good choice
(blob-scoped per-upload SAS, isolated per-tenant prefix, automated
retention via cascade-delete) — the only trade-off is data leaving the
customer's Azure tenant. Swap red for sky-blue + info-circle icon, drop
the "data leaves your tenant" red badge in favour of "managed for you",
and rephrase the disclosure as a friendly heads-up instead of a warning.
The substance is unchanged; the visual cue just no longer reads as a
fault state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an opt-in second destination for agent diagnostics uploads:
- CustomerSas (default, unchanged): tenant's own SAS-backed container
- Hosted (new, opt-in via explicit admin click): backend's storage account
under {tenantId}/AgentDiagnostics-...zip, with per-upload 15-min
blob-scoped Write+Create SAS minted via the Function App's MI
Closes the long-standing gap where Session.DiagnosticsBlobName was captured
into the deletion-manifest but the blob was never actually cleaned up by
the cascade. Hosted blobs are always deleted (we own the storage); customer
blobs are deleted only when the SAS includes the 'd' permission, otherwise
logged + skipped.
Existing tenants change nothing: the new TenantConfiguration.DiagnosticsUploadDestination
field defaults to CustomerSas, no silent flip. Hosted requires an explicit
admin click + Save with a clearly marked "data leaves your tenant" disclosure.
Tests: Backend 1844, V2 1076, V1 538, vitest 380 (+78 new across all suites).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Persist tenant Preview-Notification-Email on OffboardingHistory row at Phase 1 so it
survives Phase 2 wipe; ResendEmailService.SendAsync wired but disarmed by a const-false
arm-flag with a [DRAFT] placeholder template — explicit flip required to ship.
- Move user feedback storage out of PreviewConfig discriminator into a dedicated Feedback
table (PK=InApp / Offboarding) that is NOT in any tenant-offboarding wipe list — feedback
from departing tenants is exactly the signal we want to keep. New IFeedbackRepository +
TableFeedbackRepository; FeedbackFunction migrated; (PreviewConfig, "Feedback")
removed from TenantOffboardingHandler.DiscriminatorTables.
- POST /api/tenants/{tenantId}/offboard/feedback collects a free-form farewell comment
during the drain-barrier countdown (marker-gated, fail-soft OpsEvent emit). Reports >
User Feedback page gains an Offboarding-Feedback sub-tab next to In-App Feedback.
- OffboardingFeedbackReceived OpsEventType registered (info-severity) + catalog entry
in OpsAlertRulesSection so admins can wire Telegram in two clicks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`tenantNavSections.ts` (settings-page routing) was wired but the sidebar reads
from `globalNavConfig.tsx`, which is a separate source of truth for the left-rail
nav. Adding the matching entry between Bootstrap Sessions and Submit Logs so the
section becomes reachable without typing the URL manually.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
So we can answer "which tenants actively have this on" and "how many people are
using it" later — without bolting on a separate metrics store.
Backend events
- `GraphAddOnPermissionsDetected` (GraphFeatureDetector, fresh-acquire path):
fires at most every ~55min per tenant (cache TTL). Props: TenantId,
GrantedRoleCount, HasScriptDisplayNames, TokenCacheTtlMinutes. → distinct
count over time = tenants who have the optional permission turned on.
- `ScriptDisplayNamesResolved` (ScriptDisplayNameResolver, only when permission
is active and the resolver actually did work): per session-view with scripts.
Props: TenantId, RefsRequested, RefsResolved, CacheHits, TriggeredFullPull.
→ real usage pulse.
- `GraphAddOnStatusChecked` (GET status endpoint): admin engagement with the
capability page. Props: TenantId, UserId, IsTransient, ScriptDisplayNamesGranted.
- `GraphAddOnRefreshTriggered` (POST refresh endpoint): high-signal "tenant just
granted" event. Props: TenantId, UserId.
Web events (via `trackEvent` -- tenantId auto-attached by the telemetry initializer)
- `GraphAddOnStatusViewed` on Settings → Optional Graph capabilities mount.
- `GraphAddOnGrantCommandCopied` on Copy command / Copy permissions click.
Notes
- All emits wrapped in try/catch with debug-level logging — telemetry failures
must never poison the acquire / resolve hot paths.
- Tests pass a no-op `TelemetryClient(new TelemetryConfiguration())` (same
pattern as SlaTenantStatusLifecycleTests). 1718 backend / 364 web tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds opt-in resolution of Intune Platform Script + Remediation display names in
session timelines via a tenant-side appRoleAssignment grant. No app manifest
change, no impact on customers who don't enable it. Architecture is reusable for
future optional Graph permissions: catalog -> detector -> resolver -> endpoint.
Backend
- GraphFeatureDetector parses the `roles` claim from the SP access token and
caches the (token, roles) bundle per tenant. GetSnapshotAsync surfaces a
transient flag so admin-UI callers can render "unknown" instead of a
misleading "0 of 0 granted" when token acquire times out.
- GraphTokenService now accepts a CancellationToken so optional-feature callers
can apply tight budgets instead of inheriting the 5+15+30 s retry chain.
- ScriptDisplayNameResolver: cache-first, list-pull-on-cold-tenant, per-ID
fallback. Throttle-aware abort: skip per-ID for kinds whose full-pull hit a
transient failure; break the per-ID loop on the first HttpRequestException.
Total budget 5 s, with a tight 1.5 s sub-budget on the token acquire.
- Azure Table `ScriptNameCache` + direct-TableClient fail-loud repository
(mirrors TableOffboardingAuditRepository pattern). Daily
ScriptNameCacheCleanupFunction: 7 d positive / 1 h NotFound TTL.
- POST /api/tenants/{id}/scripts/display-names (body-carried refs to dodge URL
length limits; backend cap 200 refs/request).
- GET /api/tenants/{id}/graph-permissions/status (4 s budget; includes app
ClientId so the UI can render an exact grant command).
- POST /api/tenants/{id}/graph-permissions/refresh (invalidates token cache).
- All three routes registered in EndpointAccessPolicyCatalog (MemberRead for the
resolve, TenantAdminOrGA for the admin-UI surfaces).
Web
- `useScriptDisplayNames` hook with stable refset fingerprint (no spurious
refetches on parent re-renders); clears stale map on tenant/refset change so
a session switch never leaks names across tenants.
- Chunked POST loop: 200-ref batches with partial-success semantics. Preserves
earlier chunks when later ones fail (non-2xx OR transport throw); only
TokenExpiredError propagates so auth flow can react.
- ScriptExecutions renders " . {DisplayName}" alongside existing label/short-id,
keyed by Kind + policyId so Platform/Remediation under the same id never
collide.
- New Settings section "Optional Graph capabilities": status table with granted
/ not granted / unknown, copy-paste PS one-liner with ClientId baked in,
refresh button gated on response.ok.
Customer-facing
- scripts/CustomerSetup/Grant-AutopilotMonitorAddOn.ps1: idempotent
grant / verify / revoke script. -Features ScriptDisplayNames (high level) or
-Permissions ... (admin-UI generated). Pure ASCII for PS 5.1 compat.
- docs/customer-graph-add-on-permissions.md: end-to-end customer guide
covering prerequisites, the appRoleAssignment pattern, and revocation.
Tests
- 1718 backend xUnit / 364 web vitest. Highlights: JWT roles parsing edge
cases; full-pull-429 suppresses per-ID; per-ID-429 breaks loop; transient
detection via GetSnapshotAsync; chunked partial success (HTTP failure + thrown
transport); TokenExpiredError propagation; EndpointPolicyCatalogCompleteness
covers the three new routes by reflection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Session 330f73f3 completed via DeadlineFired:finalizing_grace (helloOutcome=
Timeout) while ESP AccountSetup was still at "4 of 5 subcategories completed"
and 3 of 8 apps still pending. ShouldTransitionToAwaitingHello promoted on
AccountSetupEnteredUtc != null alone, but Windows CXH event 62407 fires at
every ESP-page transition — the first occurrence (Device→Account handoff,
9 s after AccountSetup entered) was treated as the genuine final exit,
armed HelloSafety against an in-progress AccountSetup, fired its 5-min
synthetic timeout, and FinalizingGrace declared the session terminal
mid-flight.
Introduce a strong "AccountSetup truly succeeded" fact sourced from
ProvisioningStatusTracker (categorySucceeded=true OR fallback after 30 s
of all subcategories at succeeded/notRequired — same shape as DeviceSetup).
Forwarded by EspAndHelloTracker, posted as new DecisionSignalKind.
AccountSetupProvisioningComplete by the V2 adapter, lands as DecisionState.
AccountSetupProvisioningSucceededUtc. The reducer's
ShouldTransitionToAwaitingHello now requires this fact (or SkipUserEsp);
the old AccountSetupEnteredUtc gate becomes an entry-time marker only.
HandleAccountSetupProvisioningCompleteV1 has a deferred-promotion path:
when the strong fact arrives AFTER a guard-blocked EspExiting
(EspFinalExitUtc set) OR after a guard-blocked EspPhaseChanged(Finalizing-
Setup) (FinalizingEnteredUtc set, ShellCoreTracker adapter fire-once-
dedupes), the handler performs the deferred promotion + arms HelloSafety
anchored at the AccountSetup-succeeded instant.
The new fact is registered in DecisionStateSignalCensus so enrollment_*
audit trails and final-status.json show whether the strong gate was
satisfied at termination — load-bearing for diagnosing future
false-negative timeouts under the new gate.
Side cleanup: ProvisioningStatusTracker's fallback timer callbacks (new
AccountSetup AND pre-existing DeviceSetup parallel path) now reset the
timer field on every abort path AND in the outer exception catch. Without
this, a transient subcategory flicker between fallback arm and timer fire
could permanently prevent re-arming even when the state re-stabilises.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `sidebarState` ("full" | "icons" | "hidden") as a global telemetry
property on every App Insights event/pageView, analogous to `theme`.
Pre-seeded from localStorage at init and kept in sync via the
useSidebarState hook on every state change.
Fires a `sidebar_state_changed` customEvent with `{from, to}` only on
user-initiated cycle clicks (guarded by previous != next, so SSR
hydration and idempotent setter calls do not produce noise).
Together these answer "how many users live in the collapsed sidebar?"
(persistent preference per Auth-User via global prop) and "how often do
people toggle it?" (active interaction via customEvent). Bumps the
.claude/commands submodule for the matching ux-insights skill catalog
update.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls in submodule commit e27a9d3 which adds queries for the new AgentCertRejected
+ AgentRequestRejected backend warnings and the V2 distress cert-context fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend emits AgentCertRejected (with reason enum) at every cert-validation
failure path and AgentRequestRejected (with request-context) at every 401/403
return so operators can diagnose why mTLS auth was denied. V2 agent now
attaches a Thumbprint/Subject/Issuer/NotBefore/NotAfter snapshot plus a
DistressCertSourceState classifier to each distress report; correlates 1:1
with the backend warnings via tenant + serial + thumbprint.
Distress endpoint stays unauthenticated and security-restrictive: rate-limit
(3-layer) + sanitize + truncate + enum-validation unchanged; new fields are
hex-regex / IsDefined / hard-capped at 96 chars; MaxContentLength bumped from
1024 to 1536 to fit the cert-context payload. V1 agent untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a "Only tenants with restore data" checkbox to the Restore Browser
tenant dropdown so a Global Admin can hide tenants that have nothing to
recover. Default off (full AdminConfig list); flipping it on fetches a
cheap hierarchy listing once (cached for page lifetime) and intersects
the dropdown with the tenants that own ≥1 snapshot blob.
Backend: new GlobalAdminOnly endpoint
GET /api/global/tenants-with-deletion-manifests backed by
BlobStorageService.ListTenantsWithDeletionManifestsAsync, which uses
GetBlobsByHierarchyAsync with delimiter="/" so Azure returns just the
top-level prefixes (= tenant IDs) without enumerating blobs underneath.
Cost scales with tenant count, not manifest count.
Web: checkbox under the tenant dropdown shows a "(X of Y)" counter when
active, dropdown placeholder reflects the loading state, and the
currently-selected tenant clears if the filter excludes it. Fetch
failures snap the checkbox back to off so the operator never gets
stranded with an empty list.
Build: 0 warnings/errors. Backend 1526/1526 pass. tsc clean.
Vitest 346/346 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cascade-delete V2 is the only path now. Drops the per-tenant
EnableCascadeDeleteV2 flag, the legacy ETag-CAS-guarded DeleteSessionAsync
helper, the AppInstallSummaries cleanup helper (only legacy caller),
and the SessionsLegacyDeleted counter chain (fanout → maintenance → OpsEvent).
DeleteSessionFunction collapses to a single kill-switch gate; the producer
owns 404 / 409 / Poisoned mapping AND the intentional Queued/Preparing+Snapshot
→ 202 recovery resume — gating those at HTTP would strand crashed admin
cascades. Fanout now aborts both the inner loop AND outer loop on producer
KillSwitchActive instead of leaking calls into the next tenant.
Test coverage:
- DeleteSessionCasGuardTests removed (helper gone)
- AdminConfigurationSessionDeletionKillSwitchTests trimmed to kill-switch only
- DeleteSessionFunctionTests pinned to the kill-switch short-circuit + null
- SessionRetentionFanoutServiceTests: legacy path tests dropped, two new
abort regressions for producer KillSwitchActive (inner + outer loop)
Build: 0 warnings, 0 errors. Backend: 1526 passed / 0 failed.
Web: tsc clean, vitest 346 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review of ef530fb8 surfaced three issues; all addressed.
P1 - Auth-failure bypassed the new shutdown gate. CreateAuthThresholdHandler
emitted agent_shutting_down directly without claiming the cross-path slot,
so the finally-block runtime_host_exit fallback then also emitted, putting
two events on the wire. The handler now accepts tryClaimShutdownEvent +
releaseShutdownEventClaim Funcs and participates in the same gate as the
other paths. AgentRuntimeHost wires it in via TryClaimAgentShuttingDownEmit
plus the new ReleaseShutdownEventClaim helper.
P2 - The gate was claimed before the emit had any chance to succeed, so a
Ctrl+C that fired before onIngressReady (lifecyclePost == null) consumed
the slot permanently and the later finally fallback could no longer surface
the event. EmitAgentShuttingDownGapPath now returns the new
AgentShuttingDownEmitResult enum (Success / NoPost / EmitFailed). The
AgentRuntimeHost wraps claim + emit in TryEmitAgentShuttingDownGap which
releases the slot on NoPost so a later path can retry, while keeping it
claimed on Success and EmitFailed. Auth-failure path symmetric: releases
on null post.
P3 - GetProcessOwner swallowed its WMI exceptions and returned null without
bumping _wmiErrorCountSinceStart. desktop_detector_first_poll and
_no_candidate therefore reported wmiErrorCount=0 even when WMI was broken,
masking the very failure mode the liveness events were designed to detect.
The counter is now incremented inside GetProcessOwner's catch AND on
non-zero InvokeMethod results, so the per-poll PID-owner failure rate
flows through to telemetry.
Tests: 3777 green (1538 backend + 1068 V2.Core, +3 for new auth_failure
gate tests + 530 V1 + 294 DecisionCore + 347 vitest). TypeScript noEmit
clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DAD liveness (3 new state-change-only events, NOT periodic, max 3 per
detector lifetime): desktop_detector_started / _first_poll /
_no_candidate. Distinguishes "agent dead post-reboot" from "DAD wiring
broken" from "user never logged in" in sessions missing desktop_arrived
(the ca9e3c59 timeout pattern). Threshold via global AdminConfiguration
.DesktopDetectorNoCandidateTimeoutMinutes (default 10, 0 = disabled),
round-trip wired through TableConfigRepository, GetAgentConfigFunction
(ConfigVersion 27 -> 28), CollectorConfiguration, DefaultComponentFactory,
DesktopArrivalHost + admin UI. Also fixes a peripheral-loss bug: V2's
DesktopArrivalDetector.OnTraceEvent callback was never wired, so the
existing desktop_excluded_user and desktop_real_user_detected events
were dead code in V2. Now routed via InformationalEventPost in the host.
Shutdown gap closure (4 new agent_shutting_down emit paths): ctrl_c,
process_exit, unhandled_exception (with exception type + 500-char-
truncated message), runtime_host_exit (defensive finally fallback).
Cross-path Interlocked idempotency gate shared with EnrollmentTermination
Handler via new tryClaimShutdownEvent ctor param prevents double-emit
when Ctrl+C races a real Terminated transition.
Event-type unification (V2 only): CreateAuthThresholdHandler now emits
agent_shutting_down with data.reason=auth_failure (was agent_shutdown).
EnrollmentTerminationHandler reason vocabulary unified to lowercased
tags (decision_terminal, max_lifetime). V1 keeps legacy agent_shutdown
(phasing out). Web/MCP consumers (eventHelpers, whiteGloveSplit test
fixtures, gather-rules eventTypes catalog, MCP semantic-event list,
docs) treat both as synonyms.
Tests: 3411 green (1538 backend + 1065 V2 core + 530 V1 core + 294
DecisionCore + 347 vitest). TypeScript noEmit clean. 5 new DAD-liveness
tests, 5 new gap-path tests, 2 new idempotency tests, 3 reason-vocab
updates.
Not in this PR: cross-process death detection for the actual ca9e3c59
silent-post-reboot-death pattern (Logon-Trigger / external watchdog) -
separate follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /api/admin/* prefix collides with the Azure Functions runtime's built-in
admin paths, which sit behind mTLS client-certificate auth. Browser preflights
to /api/admin/* trigger TLS renegotiation against the Function App's mTLS
boundary and die before our PolicyEnforcementMiddleware ever runs — the
operator sees CORS-shaped errors that aren't actually CORS-shaped.
Repo convention all along has been /api/global/* for cross-tenant GA-only
HTTP endpoints (e.g. global/session-deletions, global/session-reports). The
cascade-delete endpoints landed under /api/admin/* in the original 7-PR
backend branch and went undetected until the admin UI started calling them
from the browser.
Routes renamed:
admin/sessions/{sessionId}/delete/preview → global/sessions/{sessionId}/delete/preview
admin/sessions/{sessionId}/restore → global/sessions/{sessionId}/restore
admin/sessions/{sessionId}/deletion-manifest → global/sessions/{sessionId}/deletion-manifest
admin/tenants/{tenantId}/deletion-manifests → global/tenants/{tenantId}/deletion-manifests
Touched:
- 4 Function classes: Route attribute + XML-doc header
- EndpointAccessPolicyCatalog: 4 entries (TenantScoping.RouteParam preserved
on the tenants route), with a comment explaining why /api/global/* is the
load-bearing prefix for new GA-only HTTP routes
- DeleteSessionFunction: 3 user-facing "Cascade is poisoned; recover via
POST /api/.../restore" error messages updated
- SessionDeletion.cs: XML-doc reference to the restore route
- lib/api.ts: 4 frontend URL builders (storedManifest, preview, restore,
tenantManifests)
- RestoreSessionFunctionTests: 3 EndpointAccessPolicyCatalog.FindPolicy
lookups
- deleteSessionResponse.test.ts: test fixture message string
No semantic change — only the URL prefix moves. CORS config + JWT middleware
+ policy gating all work the same. The Restore Browser tab + the per-row
Preview / Restore buttons in the Session Cleanup admin page now actually
reach the worker from a browser.
Tests: backend xUnit 1538/1538 pass, web vitest 347/347 pass, dotnet build +
tsc clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-ups on the Session Cleanup admin page after first hands-on review:
an entry point for restoring already-completed cascades (Sessions row gone,
manifest still in the 33-day TTL window), a proper outcome panel that surfaces
what the backend already returned, and a sidebar fix.
== New Restore Browser tab ==
- Backend: GET /api/admin/tenants/{tenantId}/deletion-manifests enumerates
every persisted snapshot blob under <tenantId>/, parses (sessionId,
manifestId, sizeBytes, lastModifiedUtc) from the blob name, groups by
sessionId and sorts both levels by recency. GA-only + TenantScoping.RouteParam
in EndpointAccessPolicyCatalog (pinned by EndpointPolicyCatalogCompletenessTests).
- BlobStorageService.EnumerateDeletionManifestsByTenantAsync — sibling to
the existing maintenance-sweep enumerator but tenant-prefixed and yielding
size + timestamp so the UI can rank without an extra HEAD per blob.
- Web: new tab "Restore Browser" between Stranded and Activity. Tenant
dropdown (sourced from AdminConfigContext.tenants) → collapsible
session/manifest tree on the left, manifest summary on the right with a
Restore… button that opens the dialog with IDs pre-filled. Useful when
the cascade completed cleanly and the row no longer appears on the
In-Flight / Poisoned / Stranded tabs.
== Reusable summary view ==
DeletionManifestSummaryView extracted from the modal so the same fetch +
field block + worker-progress + preflight-counts + last-failure panel renders
both inside the modal (Preview button on the other tabs) and inline on the
Restore Browser's right-hand panel. Modal becomes a thin wrapper.
== Dry-run / real-run side-by-side ==
RestoreConfirmDialog now keeps both outcomes visible:
- Two slots (dryRunResult, realRunResult), each rendered as a full result
section with predicted/actual rows-by-table, inventory re-increments,
duration, backend message, current state, pending manifest id.
- "Start over" resets both slots so the operator can re-run cleanly.
- Submit buttons gate the workflow: pre-submit shows "Run dry-run" + an
explicit "Skip dry-run · Restore now"; after dry-run the second button
becomes "Apply restore" (clear-intent); after real-run all submit
buttons disappear so accidental double-applies aren't possible.
Previously the dialog discarded everything but Outcome + Mode + dryRun
flag, hiding the actual point of dry-run (the wouldRestoreByTable map).
== Codex review follow-ups (this round) ==
- P2: Restore Browser cleared the displayed sessions only after the new
fetch returned, so a fast A→B switch (or out-of-order responses) could
leave the operator looking at tenant A's manifests with the dropdown
saying B. Fix: clear sessions/expandedSessions/selection synchronously
on tenant change, abort the in-flight request via AbortController,
pair the response with a loadedTenantId state, and gate rendering on
loadedTenantId === selectedTenantId. Three-layer protection so a late
response that sneaks past the abort can't paint stale data.
- P3: PoisonedTab's onRestored callback closed the dialog immediately
after a real restore, defeating the new dry-run vs real-run side-by-side
display. Fix: drop setRestoring(null) from the callback so the dialog
stays open until the operator dismisses it via Close / Start over.
Background refresh of the Poisoned list still happens via setRefreshKey.
Now consistent with the Restore Browser tab.
== Sidebar highlight fix ==
GlobalSidebar marked BOTH ga-ops sub-items active when on the longer
nested route, because each sub used pathname.startsWith(sub.href + "/")
independently. Visible bug: navigating to /admin/ops/session-cleanup lit
up both "Maintenance" (/admin/ops) and "Session Cleanup" in the sidebar.
Fix: compute the longest-matching sub.href once per expanded group, mark
only that one active.
Tests: backend xUnit 1538/1538 pass (one new endpoint registered in the
catalog, picked up by the existing completeness test). Web vitest 347/347
pass. TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit for GetTenantDeletionManifestsFunction (same HttpRequestData
mock-harness gap as the other read endpoints).
- vitest for RestoreBrowserTab + the new dialog flow (defer until a
real field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operationalises the 7-PR backend cascade-delete pipeline that landed via commits
c8658b9f..921994e8: per-tenant V2 opt-in switch, audit-trail consolidation, and
a Global-Admin-only Session Cleanup page to track in-flight / poisoned / stranded
cascades and trigger restores.
== PR-A: per-tenant V2 enrollment ==
- UpdateTenantConfigurationFunction extends the GA-only gate list with
EnableCascadeDeleteV2 so tenant admins cannot self-opt-in / out of the
snapshot+restore pathway.
- TenantManagementSection.tsx adds a purple-bordered "Enable V2 Cascade Delete"
checkbox in the GA edit modal and a "Cascade V2" badge on the tenant card so
rollout state is visible at a glance.
== PR-B: audit consolidation ==
- SessionDeletionHandler drops the 15-per-cascade deletion_step_completed /
deletion_step_failed / deletion_verification_failed audit emissions. Step
telemetry lives in DeletionProgress + structured logs; tenant admins now see
just deletion_started + deletion_completed + deletion_restored.
- SessionDeletionWorker no longer writes a deletion_poisoned tenant audit;
the signal moved to a new SessionDeletionPoisoned OpsEvent (operator-bound,
Telegram-routable, dual-registered in OpsAlertRulesSection.tsx catalog).
IMaintenanceRepository dependency removed from the worker constructor.
- SessionRestoreService stops emitting the deletion_manifest_downloaded
breadcrumb; deletion_restored is the operator-visible outcome.
== PR-C: Session Cleanup admin page ==
- New /admin/ops/session-cleanup route (gated by app/admin/layout.tsx's
ProtectedRoute requireGlobalAdmin). Sidebar entry under ga-ops alongside
Maintenance. Four URL-state tabs: In-Flight / Poisoned / Stranded Queued /
Activity (links to existing /audit page).
- Backend reads: GetSessionDeletionsListFunction
(GET /api/global/session-deletions?state=&strandedSinceMinutes=) +
GetSessionDeletionManifestFunction
(GET /api/admin/sessions/{id}/deletion-manifest?tenantId=&manifestId=&mode=).
Both registered as GlobalAdminOnly in EndpointAccessPolicyCatalog.
- DeletionPreviewModal reads the persisted snapshot via the stored-manifest
endpoint (NOT a fresh dry-run via the existing /delete/preview, which would
enumerate live-but-partially-deleted data for in-flight cascades). Renders
manifest counts, snapshot SHA, worker progress (CompletedSteps,
VerificationDone, TombstoneStarted, aggregate decrements), and the last
verification failure with residual sample. Download button uses
authenticatedFetch + Blob + Object URL so the Bearer token is sent.
- RestoreConfirmDialog drops the Full/Partial radio (mode is auto-selected
by SessionRestoreService based on cascade state — forcing the wrong mode
would corrupt state). Reason field threads through to deletion_restored
audit details. Dry-run default = true and previews the auto-selected mode.
== Codex review follow-ups (across four rounds) ==
- High: tenantId is explicitly accepted on /delete/preview (query) and
/restore (body); GA cross-tenant resolution prefers the explicit value over
JWT TargetTenantId, falling back to SessionsIndex lookup. Without this, a
GA acting on a session in another tenant got their JWT home tenant.
- High: new stored-manifest endpoint replaces the dry-run preview for
in-flight / poisoned rows; previous fix built a fresh manifest against
already-partially-deleted data and misrepresented the cascade.
- Medium: SessionDeletionPoisoned OpsEvent now carries failureType +
failureMessage + observedResidualCount + a small residual preview, persisted
by the handler into DeletionProgress (LastFailureType, LastFailureMessage,
LastObservedResidualCount, LastResidualSampleJson) before the throw so the
worker can include them.
- Medium: dry-run restore reports the AUTO-SELECTED mode (full/partial) so
the dialog's preview reflects what the real run will do, instead of the
literal string "dryRun".
- Medium: residual count field renamed to observedResidualCount with an
honest XML doc — CascadeVerificationService caps at 50 per table AND
short-circuits after the first failing table, so the number is a lower
bound. UI surfaces "N of 50+ observed rows" when N hits the cap.
- Medium: three-layer cap defense on the OpsEvent residual sample to stay
under the 4096-char TableOpsEventRepository truncation: max 5 entries
(OpsEventResidualSamplePreviewSize), per-field max 96 chars with `…` marker
(OpsEventResidualKeyMaxChars), total preview budget 1200 chars
(OpsEventResidualPreviewBudgetChars) with trailing-entry drop. Full sample
stays in the progress blob for the admin UI.
- Low: estimatedSnapshotBytes field name aligned end-to-end; activity tab
text clarifies audit-action vs OpsEvent distinction.
Tests:
- Backend xUnit 1538/1538 pass (up from 1510 baseline; +28 across the new
Session Cleanup endpoints, OpsEvent enrichment, restore-mode reporting,
cap-defense unit + integration tests).
- Web vitest 347/347 pass.
- TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit tests for GetSessionDeletionsListFunction +
GetSessionDeletionManifestFunction (no HttpRequestData mock harness in repo;
would be a separate refactor).
- vitest for DeletionPreviewModal + RestoreConfirmDialog (defer until first
field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The PR4 coordinator-pipeline tests drove adapter.TriggerEspExitingFromTest(),
which only proves the adapter emit — it bypasses the coordinator's OnEspExited
handler (where LastEventOccurredAtUtc mirroring + public EspExited re-raise
live). A future regression that breaks the coordinator subscription, the
LastEventOccurredAtUtc mirror, or the public EspExited fire would slip past.
This commit adds a coordinator-level test seam mirroring the existing
TriggerFinalizingSetupPhaseForTest pattern, and reroutes the wire-through tests
through it.
EspAndHelloTracker.TriggerEspExitedForTest:
- internal seam (InternalsVisibleTo) that calls OnEspExited(this, args)
- exact signature match to the live ShellCoreTracker.EspExited delegate
- exercises LastEventOccurredAtUtc mirror + public EspExited invoke + finally
reset — i.e. the full coordinator-side contract
ShellCoreToEngineEspExitingTests:
- Coordinator_EspExited_handler_re_raises_to_adapter_as_EspExiting_signal:
asserts source-time preservation through the coordinator → adapter chain
and that no clock-fallback "derivedTimestamp" evidence tag was added.
- Coordinator_EspExited_handler_clears_LastEventOccurredAtUtc_after_re_raise:
new defensive test for the finally-block reset; protects against a stale
ShellCore timestamp bleeding into a subsequent Hello/Provisioning forward.
- Coordinator_to_reducer_arms_HelloSafety_when_AccountSetup_observed:
rewritten to use the coordinator seam; now asserts HelloSafety dueAtUtc ==
sourceTime + 300s (not Clock.UtcNow + 300s) so source-time threading is
proven end-to-end (ShellCoreTracker → Coordinator → Adapter → Engine), not
just adapter → engine.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Oliver Kieselbach