Agent V2: when RecoveryPath loads a snapshot already at Stage=Completed/Failed,
no reducer transition fires so OnDecisionTerminalStage was never invoked — the
agent idled until AgentMaxLifetime (default 6h), emitting periodic perf/metrics
events the whole time. Fire it explicitly at the end of Start() so the
EnrollmentTerminationHandler runs (marker write + diagnostics + self-destruct).
The WhiteGloveSealed archive-and-reset at step 0 already neutralises the
Part-2 case; non-terminal stages take the normal recovery path.
Web: gate the SignalR eventStream refetch and the 30s fallback polling on
non-terminal session status. Once Succeeded/Failed, late perf/metrics arrivals
no longer trigger a full-page refetch + re-render that scroll-jumps the
timeline while the user is reading.
Observed in session 17ef0e0e-... (2026-05-19): 4x enrollment_complete, only 1x
agent_shutting_down; reboot mid-shutdown left subsequent agent instances
running for hours emitting noise.
Tests: 3 new in RecoveryPathTests (Completed/Failed -> Terminated, non-terminal
-> no Terminated), 4 in sessionStatus.test.ts. Suites grun: Web 384/384,
Agent V2.Core 1095/1095, Backend 1844/1844.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- docs/architecture.md Diagnostics section rewritten: both destinations
(CustomerSas default + Hosted opt-in), new download routing on shape,
cascade-delete step §5b with SAS-delete-permission gating, and the V2
remote-config retry + observability events.
- Submodule .claude/commands bumped to 17a6f5b for the dual-destination
fetch-diag-zip.sh refactor + skill error-handling note updates.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Motivating incident — session 8f2bef72 (2026-05-19, OK's VM test): a Function
App cold-start triggered by a fresh deploy stalled the agent's single-shot
GetAgentConfig call for ~60 s; the client gave up, RemoteConfigService
silently fell back to built-in defaults (ConfigVersion=0,
SelfDestructOnComplete=true, DiagnosticsUploadMode=Off), and the entire
enrollment ran the wrong policy with no wire-visible trace of the degraded
mode. ReportAgentError (the emergency channel) was also racing the cold
start and lost to 499.
Three fixes:
1. Retry FetchConfigAsync on transient errors (3 attempts; 10/30/60 s linear
backoff). BackendAuthException still bails immediately — 401/403 won't
change with a retry and an operator needs the AuthCertificateRejected /
DeviceNotRegistered distress signal fast. EmergencyReporter only fires on
the final attempt to give the function time to warm. Live ServerAction
rotate_config keeps single-shot so it doesn't block the action handler
for ~100 s.
2. agent_started DataJson now carries configVersion + remoteConfigFetched +
remoteConfigOutcome (Succeeded / FromCache / UsedDefaults / NotAttempted).
Operators can tell at a glance whether the agent is running on fresh
tenant config vs. defaults vs. stale cache without inferring from
absence of expected tenant-controlled knobs.
3. New remote_config_fetch_failed Warning event emitted right after
agent_started when the fallback path was taken. Carries attempts,
failureType, failureMessage (truncated 512), and authStatusCode when
relevant. Closes the blind spot — the only prior trace was a Warning
in the local agent log which is gone after a SelfDestruct cleanup.
V2 Agent suite: 1092/1092 (+16: 7 retry behaviour + 9 emitter shape).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The red color treatment + warning-triangle icon on the Hosted-storage
option implied misconfiguration. Hosted is a deliberately good choice
(blob-scoped per-upload SAS, isolated per-tenant prefix, automated
retention via cascade-delete) — the only trade-off is data leaving the
customer's Azure tenant. Swap red for sky-blue + info-circle icon, drop
the "data leaves your tenant" red badge in favour of "managed for you",
and rephrase the disclosure as a friendly heads-up instead of a warning.
The substance is unchanged; the visual cue just no longer reads as a
fault state.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds an opt-in second destination for agent diagnostics uploads:
- CustomerSas (default, unchanged): tenant's own SAS-backed container
- Hosted (new, opt-in via explicit admin click): backend's storage account
under {tenantId}/AgentDiagnostics-...zip, with per-upload 15-min
blob-scoped Write+Create SAS minted via the Function App's MI
Closes the long-standing gap where Session.DiagnosticsBlobName was captured
into the deletion-manifest but the blob was never actually cleaned up by
the cascade. Hosted blobs are always deleted (we own the storage); customer
blobs are deleted only when the SAS includes the 'd' permission, otherwise
logged + skipped.
Existing tenants change nothing: the new TenantConfiguration.DiagnosticsUploadDestination
field defaults to CustomerSas, no silent flip. Hosted requires an explicit
admin click + Save with a clearly marked "data leaves your tenant" disclosure.
Tests: Backend 1844, V2 1076, V1 538, vitest 380 (+78 new across all suites).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Persist tenant Preview-Notification-Email on OffboardingHistory row at Phase 1 so it
survives Phase 2 wipe; ResendEmailService.SendAsync wired but disarmed by a const-false
arm-flag with a [DRAFT] placeholder template — explicit flip required to ship.
- Move user feedback storage out of PreviewConfig discriminator into a dedicated Feedback
table (PK=InApp / Offboarding) that is NOT in any tenant-offboarding wipe list — feedback
from departing tenants is exactly the signal we want to keep. New IFeedbackRepository +
TableFeedbackRepository; FeedbackFunction migrated; (PreviewConfig, "Feedback")
removed from TenantOffboardingHandler.DiscriminatorTables.
- POST /api/tenants/{tenantId}/offboard/feedback collects a free-form farewell comment
during the drain-barrier countdown (marker-gated, fail-soft OpsEvent emit). Reports >
User Feedback page gains an Offboarding-Feedback sub-tab next to In-App Feedback.
- OffboardingFeedbackReceived OpsEventType registered (info-severity) + catalog entry
in OpsAlertRulesSection so admins can wire Telegram in two clicks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
`tenantNavSections.ts` (settings-page routing) was wired but the sidebar reads
from `globalNavConfig.tsx`, which is a separate source of truth for the left-rail
nav. Adding the matching entry between Bootstrap Sessions and Submit Logs so the
section becomes reachable without typing the URL manually.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
So we can answer "which tenants actively have this on" and "how many people are
using it" later — without bolting on a separate metrics store.
Backend events
- `GraphAddOnPermissionsDetected` (GraphFeatureDetector, fresh-acquire path):
fires at most every ~55min per tenant (cache TTL). Props: TenantId,
GrantedRoleCount, HasScriptDisplayNames, TokenCacheTtlMinutes. → distinct
count over time = tenants who have the optional permission turned on.
- `ScriptDisplayNamesResolved` (ScriptDisplayNameResolver, only when permission
is active and the resolver actually did work): per session-view with scripts.
Props: TenantId, RefsRequested, RefsResolved, CacheHits, TriggeredFullPull.
→ real usage pulse.
- `GraphAddOnStatusChecked` (GET status endpoint): admin engagement with the
capability page. Props: TenantId, UserId, IsTransient, ScriptDisplayNamesGranted.
- `GraphAddOnRefreshTriggered` (POST refresh endpoint): high-signal "tenant just
granted" event. Props: TenantId, UserId.
Web events (via `trackEvent` -- tenantId auto-attached by the telemetry initializer)
- `GraphAddOnStatusViewed` on Settings → Optional Graph capabilities mount.
- `GraphAddOnGrantCommandCopied` on Copy command / Copy permissions click.
Notes
- All emits wrapped in try/catch with debug-level logging — telemetry failures
must never poison the acquire / resolve hot paths.
- Tests pass a no-op `TelemetryClient(new TelemetryConfiguration())` (same
pattern as SlaTenantStatusLifecycleTests). 1718 backend / 364 web tests green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds opt-in resolution of Intune Platform Script + Remediation display names in
session timelines via a tenant-side appRoleAssignment grant. No app manifest
change, no impact on customers who don't enable it. Architecture is reusable for
future optional Graph permissions: catalog -> detector -> resolver -> endpoint.
Backend
- GraphFeatureDetector parses the `roles` claim from the SP access token and
caches the (token, roles) bundle per tenant. GetSnapshotAsync surfaces a
transient flag so admin-UI callers can render "unknown" instead of a
misleading "0 of 0 granted" when token acquire times out.
- GraphTokenService now accepts a CancellationToken so optional-feature callers
can apply tight budgets instead of inheriting the 5+15+30 s retry chain.
- ScriptDisplayNameResolver: cache-first, list-pull-on-cold-tenant, per-ID
fallback. Throttle-aware abort: skip per-ID for kinds whose full-pull hit a
transient failure; break the per-ID loop on the first HttpRequestException.
Total budget 5 s, with a tight 1.5 s sub-budget on the token acquire.
- Azure Table `ScriptNameCache` + direct-TableClient fail-loud repository
(mirrors TableOffboardingAuditRepository pattern). Daily
ScriptNameCacheCleanupFunction: 7 d positive / 1 h NotFound TTL.
- POST /api/tenants/{id}/scripts/display-names (body-carried refs to dodge URL
length limits; backend cap 200 refs/request).
- GET /api/tenants/{id}/graph-permissions/status (4 s budget; includes app
ClientId so the UI can render an exact grant command).
- POST /api/tenants/{id}/graph-permissions/refresh (invalidates token cache).
- All three routes registered in EndpointAccessPolicyCatalog (MemberRead for the
resolve, TenantAdminOrGA for the admin-UI surfaces).
Web
- `useScriptDisplayNames` hook with stable refset fingerprint (no spurious
refetches on parent re-renders); clears stale map on tenant/refset change so
a session switch never leaks names across tenants.
- Chunked POST loop: 200-ref batches with partial-success semantics. Preserves
earlier chunks when later ones fail (non-2xx OR transport throw); only
TokenExpiredError propagates so auth flow can react.
- ScriptExecutions renders " . {DisplayName}" alongside existing label/short-id,
keyed by Kind + policyId so Platform/Remediation under the same id never
collide.
- New Settings section "Optional Graph capabilities": status table with granted
/ not granted / unknown, copy-paste PS one-liner with ClientId baked in,
refresh button gated on response.ok.
Customer-facing
- scripts/CustomerSetup/Grant-AutopilotMonitorAddOn.ps1: idempotent
grant / verify / revoke script. -Features ScriptDisplayNames (high level) or
-Permissions ... (admin-UI generated). Pure ASCII for PS 5.1 compat.
- docs/customer-graph-add-on-permissions.md: end-to-end customer guide
covering prerequisites, the appRoleAssignment pattern, and revocation.
Tests
- 1718 backend xUnit / 364 web vitest. Highlights: JWT roles parsing edge
cases; full-pull-429 suppresses per-ID; per-ID-429 breaks loop; transient
detection via GetSnapshotAsync; chunked partial success (HTTP failure + thrown
transport); TokenExpiredError propagation; EndpointPolicyCatalogCompleteness
covers the three new routes by reflection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Session 330f73f3 completed via DeadlineFired:finalizing_grace (helloOutcome=
Timeout) while ESP AccountSetup was still at "4 of 5 subcategories completed"
and 3 of 8 apps still pending. ShouldTransitionToAwaitingHello promoted on
AccountSetupEnteredUtc != null alone, but Windows CXH event 62407 fires at
every ESP-page transition — the first occurrence (Device→Account handoff,
9 s after AccountSetup entered) was treated as the genuine final exit,
armed HelloSafety against an in-progress AccountSetup, fired its 5-min
synthetic timeout, and FinalizingGrace declared the session terminal
mid-flight.
Introduce a strong "AccountSetup truly succeeded" fact sourced from
ProvisioningStatusTracker (categorySucceeded=true OR fallback after 30 s
of all subcategories at succeeded/notRequired — same shape as DeviceSetup).
Forwarded by EspAndHelloTracker, posted as new DecisionSignalKind.
AccountSetupProvisioningComplete by the V2 adapter, lands as DecisionState.
AccountSetupProvisioningSucceededUtc. The reducer's
ShouldTransitionToAwaitingHello now requires this fact (or SkipUserEsp);
the old AccountSetupEnteredUtc gate becomes an entry-time marker only.
HandleAccountSetupProvisioningCompleteV1 has a deferred-promotion path:
when the strong fact arrives AFTER a guard-blocked EspExiting
(EspFinalExitUtc set) OR after a guard-blocked EspPhaseChanged(Finalizing-
Setup) (FinalizingEnteredUtc set, ShellCoreTracker adapter fire-once-
dedupes), the handler performs the deferred promotion + arms HelloSafety
anchored at the AccountSetup-succeeded instant.
The new fact is registered in DecisionStateSignalCensus so enrollment_*
audit trails and final-status.json show whether the strong gate was
satisfied at termination — load-bearing for diagnosing future
false-negative timeouts under the new gate.
Side cleanup: ProvisioningStatusTracker's fallback timer callbacks (new
AccountSetup AND pre-existing DeviceSetup parallel path) now reset the
timer field on every abort path AND in the outer exception catch. Without
this, a transient subcategory flicker between fallback arm and timer fire
could permanently prevent re-arming even when the state re-stabilises.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds `sidebarState` ("full" | "icons" | "hidden") as a global telemetry
property on every App Insights event/pageView, analogous to `theme`.
Pre-seeded from localStorage at init and kept in sync via the
useSidebarState hook on every state change.
Fires a `sidebar_state_changed` customEvent with `{from, to}` only on
user-initiated cycle clicks (guarded by previous != next, so SSR
hydration and idempotent setter calls do not produce noise).
Together these answer "how many users live in the collapsed sidebar?"
(persistent preference per Auth-User via global prop) and "how often do
people toggle it?" (active interaction via customEvent). Bumps the
.claude/commands submodule for the matching ux-insights skill catalog
update.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Pulls in submodule commit e27a9d3 which adds queries for the new AgentCertRejected
+ AgentRequestRejected backend warnings and the V2 distress cert-context fields.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Backend emits AgentCertRejected (with reason enum) at every cert-validation
failure path and AgentRequestRejected (with request-context) at every 401/403
return so operators can diagnose why mTLS auth was denied. V2 agent now
attaches a Thumbprint/Subject/Issuer/NotBefore/NotAfter snapshot plus a
DistressCertSourceState classifier to each distress report; correlates 1:1
with the backend warnings via tenant + serial + thumbprint.
Distress endpoint stays unauthenticated and security-restrictive: rate-limit
(3-layer) + sanitize + truncate + enum-validation unchanged; new fields are
hex-regex / IsDefined / hard-capped at 96 chars; MaxContentLength bumped from
1024 to 1536 to fit the cert-context payload. V1 agent untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds a "Only tenants with restore data" checkbox to the Restore Browser
tenant dropdown so a Global Admin can hide tenants that have nothing to
recover. Default off (full AdminConfig list); flipping it on fetches a
cheap hierarchy listing once (cached for page lifetime) and intersects
the dropdown with the tenants that own ≥1 snapshot blob.
Backend: new GlobalAdminOnly endpoint
GET /api/global/tenants-with-deletion-manifests backed by
BlobStorageService.ListTenantsWithDeletionManifestsAsync, which uses
GetBlobsByHierarchyAsync with delimiter="/" so Azure returns just the
top-level prefixes (= tenant IDs) without enumerating blobs underneath.
Cost scales with tenant count, not manifest count.
Web: checkbox under the tenant dropdown shows a "(X of Y)" counter when
active, dropdown placeholder reflects the loading state, and the
currently-selected tenant clears if the filter excludes it. Fetch
failures snap the checkbox back to off so the operator never gets
stranded with an empty list.
Build: 0 warnings/errors. Backend 1526/1526 pass. tsc clean.
Vitest 346/346 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Cascade-delete V2 is the only path now. Drops the per-tenant
EnableCascadeDeleteV2 flag, the legacy ETag-CAS-guarded DeleteSessionAsync
helper, the AppInstallSummaries cleanup helper (only legacy caller),
and the SessionsLegacyDeleted counter chain (fanout → maintenance → OpsEvent).
DeleteSessionFunction collapses to a single kill-switch gate; the producer
owns 404 / 409 / Poisoned mapping AND the intentional Queued/Preparing+Snapshot
→ 202 recovery resume — gating those at HTTP would strand crashed admin
cascades. Fanout now aborts both the inner loop AND outer loop on producer
KillSwitchActive instead of leaking calls into the next tenant.
Test coverage:
- DeleteSessionCasGuardTests removed (helper gone)
- AdminConfigurationSessionDeletionKillSwitchTests trimmed to kill-switch only
- DeleteSessionFunctionTests pinned to the kill-switch short-circuit + null
- SessionRetentionFanoutServiceTests: legacy path tests dropped, two new
abort regressions for producer KillSwitchActive (inner + outer loop)
Build: 0 warnings, 0 errors. Backend: 1526 passed / 0 failed.
Web: tsc clean, vitest 346 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Codex review of ef530fb8 surfaced three issues; all addressed.
P1 - Auth-failure bypassed the new shutdown gate. CreateAuthThresholdHandler
emitted agent_shutting_down directly without claiming the cross-path slot,
so the finally-block runtime_host_exit fallback then also emitted, putting
two events on the wire. The handler now accepts tryClaimShutdownEvent +
releaseShutdownEventClaim Funcs and participates in the same gate as the
other paths. AgentRuntimeHost wires it in via TryClaimAgentShuttingDownEmit
plus the new ReleaseShutdownEventClaim helper.
P2 - The gate was claimed before the emit had any chance to succeed, so a
Ctrl+C that fired before onIngressReady (lifecyclePost == null) consumed
the slot permanently and the later finally fallback could no longer surface
the event. EmitAgentShuttingDownGapPath now returns the new
AgentShuttingDownEmitResult enum (Success / NoPost / EmitFailed). The
AgentRuntimeHost wraps claim + emit in TryEmitAgentShuttingDownGap which
releases the slot on NoPost so a later path can retry, while keeping it
claimed on Success and EmitFailed. Auth-failure path symmetric: releases
on null post.
P3 - GetProcessOwner swallowed its WMI exceptions and returned null without
bumping _wmiErrorCountSinceStart. desktop_detector_first_poll and
_no_candidate therefore reported wmiErrorCount=0 even when WMI was broken,
masking the very failure mode the liveness events were designed to detect.
The counter is now incremented inside GetProcessOwner's catch AND on
non-zero InvokeMethod results, so the per-poll PID-owner failure rate
flows through to telemetry.
Tests: 3777 green (1538 backend + 1068 V2.Core, +3 for new auth_failure
gate tests + 530 V1 + 294 DecisionCore + 347 vitest). TypeScript noEmit
clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
DAD liveness (3 new state-change-only events, NOT periodic, max 3 per
detector lifetime): desktop_detector_started / _first_poll /
_no_candidate. Distinguishes "agent dead post-reboot" from "DAD wiring
broken" from "user never logged in" in sessions missing desktop_arrived
(the ca9e3c59 timeout pattern). Threshold via global AdminConfiguration
.DesktopDetectorNoCandidateTimeoutMinutes (default 10, 0 = disabled),
round-trip wired through TableConfigRepository, GetAgentConfigFunction
(ConfigVersion 27 -> 28), CollectorConfiguration, DefaultComponentFactory,
DesktopArrivalHost + admin UI. Also fixes a peripheral-loss bug: V2's
DesktopArrivalDetector.OnTraceEvent callback was never wired, so the
existing desktop_excluded_user and desktop_real_user_detected events
were dead code in V2. Now routed via InformationalEventPost in the host.
Shutdown gap closure (4 new agent_shutting_down emit paths): ctrl_c,
process_exit, unhandled_exception (with exception type + 500-char-
truncated message), runtime_host_exit (defensive finally fallback).
Cross-path Interlocked idempotency gate shared with EnrollmentTermination
Handler via new tryClaimShutdownEvent ctor param prevents double-emit
when Ctrl+C races a real Terminated transition.
Event-type unification (V2 only): CreateAuthThresholdHandler now emits
agent_shutting_down with data.reason=auth_failure (was agent_shutdown).
EnrollmentTerminationHandler reason vocabulary unified to lowercased
tags (decision_terminal, max_lifetime). V1 keeps legacy agent_shutdown
(phasing out). Web/MCP consumers (eventHelpers, whiteGloveSplit test
fixtures, gather-rules eventTypes catalog, MCP semantic-event list,
docs) treat both as synonyms.
Tests: 3411 green (1538 backend + 1065 V2 core + 530 V1 core + 294
DecisionCore + 347 vitest). TypeScript noEmit clean. 5 new DAD-liveness
tests, 5 new gap-path tests, 2 new idempotency tests, 3 reason-vocab
updates.
Not in this PR: cross-process death detection for the actual ca9e3c59
silent-post-reboot-death pattern (Logon-Trigger / external watchdog) -
separate follow-up.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The /api/admin/* prefix collides with the Azure Functions runtime's built-in
admin paths, which sit behind mTLS client-certificate auth. Browser preflights
to /api/admin/* trigger TLS renegotiation against the Function App's mTLS
boundary and die before our PolicyEnforcementMiddleware ever runs — the
operator sees CORS-shaped errors that aren't actually CORS-shaped.
Repo convention all along has been /api/global/* for cross-tenant GA-only
HTTP endpoints (e.g. global/session-deletions, global/session-reports). The
cascade-delete endpoints landed under /api/admin/* in the original 7-PR
backend branch and went undetected until the admin UI started calling them
from the browser.
Routes renamed:
admin/sessions/{sessionId}/delete/preview → global/sessions/{sessionId}/delete/preview
admin/sessions/{sessionId}/restore → global/sessions/{sessionId}/restore
admin/sessions/{sessionId}/deletion-manifest → global/sessions/{sessionId}/deletion-manifest
admin/tenants/{tenantId}/deletion-manifests → global/tenants/{tenantId}/deletion-manifests
Touched:
- 4 Function classes: Route attribute + XML-doc header
- EndpointAccessPolicyCatalog: 4 entries (TenantScoping.RouteParam preserved
on the tenants route), with a comment explaining why /api/global/* is the
load-bearing prefix for new GA-only HTTP routes
- DeleteSessionFunction: 3 user-facing "Cascade is poisoned; recover via
POST /api/.../restore" error messages updated
- SessionDeletion.cs: XML-doc reference to the restore route
- lib/api.ts: 4 frontend URL builders (storedManifest, preview, restore,
tenantManifests)
- RestoreSessionFunctionTests: 3 EndpointAccessPolicyCatalog.FindPolicy
lookups
- deleteSessionResponse.test.ts: test fixture message string
No semantic change — only the URL prefix moves. CORS config + JWT middleware
+ policy gating all work the same. The Restore Browser tab + the per-row
Preview / Restore buttons in the Session Cleanup admin page now actually
reach the worker from a browser.
Tests: backend xUnit 1538/1538 pass, web vitest 347/347 pass, dotnet build +
tsc clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Follow-ups on the Session Cleanup admin page after first hands-on review:
an entry point for restoring already-completed cascades (Sessions row gone,
manifest still in the 33-day TTL window), a proper outcome panel that surfaces
what the backend already returned, and a sidebar fix.
== New Restore Browser tab ==
- Backend: GET /api/admin/tenants/{tenantId}/deletion-manifests enumerates
every persisted snapshot blob under <tenantId>/, parses (sessionId,
manifestId, sizeBytes, lastModifiedUtc) from the blob name, groups by
sessionId and sorts both levels by recency. GA-only + TenantScoping.RouteParam
in EndpointAccessPolicyCatalog (pinned by EndpointPolicyCatalogCompletenessTests).
- BlobStorageService.EnumerateDeletionManifestsByTenantAsync — sibling to
the existing maintenance-sweep enumerator but tenant-prefixed and yielding
size + timestamp so the UI can rank without an extra HEAD per blob.
- Web: new tab "Restore Browser" between Stranded and Activity. Tenant
dropdown (sourced from AdminConfigContext.tenants) → collapsible
session/manifest tree on the left, manifest summary on the right with a
Restore… button that opens the dialog with IDs pre-filled. Useful when
the cascade completed cleanly and the row no longer appears on the
In-Flight / Poisoned / Stranded tabs.
== Reusable summary view ==
DeletionManifestSummaryView extracted from the modal so the same fetch +
field block + worker-progress + preflight-counts + last-failure panel renders
both inside the modal (Preview button on the other tabs) and inline on the
Restore Browser's right-hand panel. Modal becomes a thin wrapper.
== Dry-run / real-run side-by-side ==
RestoreConfirmDialog now keeps both outcomes visible:
- Two slots (dryRunResult, realRunResult), each rendered as a full result
section with predicted/actual rows-by-table, inventory re-increments,
duration, backend message, current state, pending manifest id.
- "Start over" resets both slots so the operator can re-run cleanly.
- Submit buttons gate the workflow: pre-submit shows "Run dry-run" + an
explicit "Skip dry-run · Restore now"; after dry-run the second button
becomes "Apply restore" (clear-intent); after real-run all submit
buttons disappear so accidental double-applies aren't possible.
Previously the dialog discarded everything but Outcome + Mode + dryRun
flag, hiding the actual point of dry-run (the wouldRestoreByTable map).
== Codex review follow-ups (this round) ==
- P2: Restore Browser cleared the displayed sessions only after the new
fetch returned, so a fast A→B switch (or out-of-order responses) could
leave the operator looking at tenant A's manifests with the dropdown
saying B. Fix: clear sessions/expandedSessions/selection synchronously
on tenant change, abort the in-flight request via AbortController,
pair the response with a loadedTenantId state, and gate rendering on
loadedTenantId === selectedTenantId. Three-layer protection so a late
response that sneaks past the abort can't paint stale data.
- P3: PoisonedTab's onRestored callback closed the dialog immediately
after a real restore, defeating the new dry-run vs real-run side-by-side
display. Fix: drop setRestoring(null) from the callback so the dialog
stays open until the operator dismisses it via Close / Start over.
Background refresh of the Poisoned list still happens via setRefreshKey.
Now consistent with the Restore Browser tab.
== Sidebar highlight fix ==
GlobalSidebar marked BOTH ga-ops sub-items active when on the longer
nested route, because each sub used pathname.startsWith(sub.href + "/")
independently. Visible bug: navigating to /admin/ops/session-cleanup lit
up both "Maintenance" (/admin/ops) and "Session Cleanup" in the sidebar.
Fix: compute the longest-matching sub.href once per expanded group, mark
only that one active.
Tests: backend xUnit 1538/1538 pass (one new endpoint registered in the
catalog, picked up by the existing completeness test). Web vitest 347/347
pass. TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit for GetTenantDeletionManifestsFunction (same HttpRequestData
mock-harness gap as the other read endpoints).
- vitest for RestoreBrowserTab + the new dialog flow (defer until a
real field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operationalises the 7-PR backend cascade-delete pipeline that landed via commits
c8658b9f..921994e8: per-tenant V2 opt-in switch, audit-trail consolidation, and
a Global-Admin-only Session Cleanup page to track in-flight / poisoned / stranded
cascades and trigger restores.
== PR-A: per-tenant V2 enrollment ==
- UpdateTenantConfigurationFunction extends the GA-only gate list with
EnableCascadeDeleteV2 so tenant admins cannot self-opt-in / out of the
snapshot+restore pathway.
- TenantManagementSection.tsx adds a purple-bordered "Enable V2 Cascade Delete"
checkbox in the GA edit modal and a "Cascade V2" badge on the tenant card so
rollout state is visible at a glance.
== PR-B: audit consolidation ==
- SessionDeletionHandler drops the 15-per-cascade deletion_step_completed /
deletion_step_failed / deletion_verification_failed audit emissions. Step
telemetry lives in DeletionProgress + structured logs; tenant admins now see
just deletion_started + deletion_completed + deletion_restored.
- SessionDeletionWorker no longer writes a deletion_poisoned tenant audit;
the signal moved to a new SessionDeletionPoisoned OpsEvent (operator-bound,
Telegram-routable, dual-registered in OpsAlertRulesSection.tsx catalog).
IMaintenanceRepository dependency removed from the worker constructor.
- SessionRestoreService stops emitting the deletion_manifest_downloaded
breadcrumb; deletion_restored is the operator-visible outcome.
== PR-C: Session Cleanup admin page ==
- New /admin/ops/session-cleanup route (gated by app/admin/layout.tsx's
ProtectedRoute requireGlobalAdmin). Sidebar entry under ga-ops alongside
Maintenance. Four URL-state tabs: In-Flight / Poisoned / Stranded Queued /
Activity (links to existing /audit page).
- Backend reads: GetSessionDeletionsListFunction
(GET /api/global/session-deletions?state=&strandedSinceMinutes=) +
GetSessionDeletionManifestFunction
(GET /api/admin/sessions/{id}/deletion-manifest?tenantId=&manifestId=&mode=).
Both registered as GlobalAdminOnly in EndpointAccessPolicyCatalog.
- DeletionPreviewModal reads the persisted snapshot via the stored-manifest
endpoint (NOT a fresh dry-run via the existing /delete/preview, which would
enumerate live-but-partially-deleted data for in-flight cascades). Renders
manifest counts, snapshot SHA, worker progress (CompletedSteps,
VerificationDone, TombstoneStarted, aggregate decrements), and the last
verification failure with residual sample. Download button uses
authenticatedFetch + Blob + Object URL so the Bearer token is sent.
- RestoreConfirmDialog drops the Full/Partial radio (mode is auto-selected
by SessionRestoreService based on cascade state — forcing the wrong mode
would corrupt state). Reason field threads through to deletion_restored
audit details. Dry-run default = true and previews the auto-selected mode.
== Codex review follow-ups (across four rounds) ==
- High: tenantId is explicitly accepted on /delete/preview (query) and
/restore (body); GA cross-tenant resolution prefers the explicit value over
JWT TargetTenantId, falling back to SessionsIndex lookup. Without this, a
GA acting on a session in another tenant got their JWT home tenant.
- High: new stored-manifest endpoint replaces the dry-run preview for
in-flight / poisoned rows; previous fix built a fresh manifest against
already-partially-deleted data and misrepresented the cascade.
- Medium: SessionDeletionPoisoned OpsEvent now carries failureType +
failureMessage + observedResidualCount + a small residual preview, persisted
by the handler into DeletionProgress (LastFailureType, LastFailureMessage,
LastObservedResidualCount, LastResidualSampleJson) before the throw so the
worker can include them.
- Medium: dry-run restore reports the AUTO-SELECTED mode (full/partial) so
the dialog's preview reflects what the real run will do, instead of the
literal string "dryRun".
- Medium: residual count field renamed to observedResidualCount with an
honest XML doc — CascadeVerificationService caps at 50 per table AND
short-circuits after the first failing table, so the number is a lower
bound. UI surfaces "N of 50+ observed rows" when N hits the cap.
- Medium: three-layer cap defense on the OpsEvent residual sample to stay
under the 4096-char TableOpsEventRepository truncation: max 5 entries
(OpsEventResidualSamplePreviewSize), per-field max 96 chars with `…` marker
(OpsEventResidualKeyMaxChars), total preview budget 1200 chars
(OpsEventResidualPreviewBudgetChars) with trailing-entry drop. Full sample
stays in the progress blob for the admin UI.
- Low: estimatedSnapshotBytes field name aligned end-to-end; activity tab
text clarifies audit-action vs OpsEvent distinction.
Tests:
- Backend xUnit 1538/1538 pass (up from 1510 baseline; +28 across the new
Session Cleanup endpoints, OpsEvent enrichment, restore-mode reporting,
cap-defense unit + integration tests).
- Web vitest 347/347 pass.
- TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit tests for GetSessionDeletionsListFunction +
GetSessionDeletionManifestFunction (no HttpRequestData mock harness in repo;
would be a separate refactor).
- vitest for DeletionPreviewModal + RestoreConfirmDialog (defer until first
field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The PR4 coordinator-pipeline tests drove adapter.TriggerEspExitingFromTest(),
which only proves the adapter emit — it bypasses the coordinator's OnEspExited
handler (where LastEventOccurredAtUtc mirroring + public EspExited re-raise
live). A future regression that breaks the coordinator subscription, the
LastEventOccurredAtUtc mirror, or the public EspExited fire would slip past.
This commit adds a coordinator-level test seam mirroring the existing
TriggerFinalizingSetupPhaseForTest pattern, and reroutes the wire-through tests
through it.
EspAndHelloTracker.TriggerEspExitedForTest:
- internal seam (InternalsVisibleTo) that calls OnEspExited(this, args)
- exact signature match to the live ShellCoreTracker.EspExited delegate
- exercises LastEventOccurredAtUtc mirror + public EspExited invoke + finally
reset — i.e. the full coordinator-side contract
ShellCoreToEngineEspExitingTests:
- Coordinator_EspExited_handler_re_raises_to_adapter_as_EspExiting_signal:
asserts source-time preservation through the coordinator → adapter chain
and that no clock-fallback "derivedTimestamp" evidence tag was added.
- Coordinator_EspExited_handler_clears_LastEventOccurredAtUtc_after_re_raise:
new defensive test for the finally-block reset; protects against a stale
ShellCore timestamp bleeding into a subsequent Hello/Provisioning forward.
- Coordinator_to_reducer_arms_HelloSafety_when_AccountSetup_observed:
rewritten to use the coordinator seam; now asserts HelloSafety dueAtUtc ==
sourceTime + 300s (not Clock.UtcNow + 300s) so source-time threading is
proven end-to-end (ShellCoreTracker → Coordinator → Adapter → Engine), not
just adapter → engine.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The Hello-disabled fast-path called builder.CancelDeadline(HelloSafety) but did
not emit a corresponding DecisionEffectKind.CancelDeadline. The reducer state
no longer carries the deadline, but the external scheduler
(DefaultComponentFactory) keeps the timer armed and fires DeadlineFired:HelloSafety
after the 300s window — which then dead-ends in the reducer because the deadline
name is no longer in state.
This commit adds the missing scheduler-visible cancel:
- TransitionToFinalizing gains an optional extraLeadingEffects parameter so the
fast-path can prepend a CancelDeadline effect ahead of the normal
ScheduleDeadline(FinalizingGrace) + phase_transition. Existing call-sites
(HandleHelloResolvedV1, HandleDesktopArrivedV1's non-fast-path branch,
HandleHelloSafetyDeadlineFired) keep their current behaviour by passing null.
- The fast-path checks state.Deadlines for HelloSafety BEFORE building the
effect — only emits the cancel when the deadline was actually armed. Avoids
no-op cancellations on the common case where the fast-path is the sole
completion path.
Tests:
- HelloDisabled_FastPath_cancels_armed_HelloSafety_deadline_in_state_and_effects:
expanded the existing state-only assertion to also assert the
CancelDeadline(HelloSafety) effect alongside ScheduleDeadline(FinalizingGrace)
and phase_transition.
- HelloDisabled_FastPath_without_armed_HelloSafety_does_not_emit_redundant_CancelDeadline_effect:
new negative test guarding against audit-trail noise.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR1 added EspExited on ShellCoreTracker + a mapping on ShellCoreTrackerAdapter,
but ShellCoreTrackerAdapter is only used in single-tracker test wiring. In
production EspAndHelloHost instantiates the EspAndHelloTracker coordinator, and
EspAndHelloTrackerAdapter subscribes to coordinator-forwarded events — the new
EspExited never reached the engine on a real device.
This commit reroutes EspExited through the coordinator (the same path the other
ShellCore-originated events take):
- EspAndHelloTracker re-raises EspExited from its inner ShellCoreTracker, with
LastEventOccurredAtUtc mirrored from the EventArgs timestamp so the adapter's
ResolveOccurredAt picks up the source-event time (matching the existing
WhiteGloveCompleted / EspFailureDetected forwarding contract).
- EspAndHelloTrackerAdapter subscribes to coordinator.EspExited and posts
DecisionSignalKind.EspExiting with no per-kind dedup — Shell-Core 62407 fires
on every ESP phase transition, and the reducer's ShouldTransitionToAwaitingHello
guard decides which occurrence arms HelloSafety. The coordinator's existing
IsIntermediateDeviceEspExit filter only governs the legacy
FinalizingSetupPhaseTriggered path; the new EspExiting signal bypasses it
deliberately so the engine remains the single source of truth.
- EspExitedEventArgs becomes public (was internal) because the coordinator's
public event surface needs an at-least-as-accessible args type.
Tests:
- EspAndHelloTrackerAdapterTests: +2 tests covering the coordinator-forwarded
mapping and the no-dedup contract.
- ShellCoreToEngineEspExitingTests: +2 tests exercising the full production
topology (coordinator → adapter → ingress → engine.Reduce → HelloSafety arm).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three new test suites verifying both halves of the V2 Classic-v1 + Hello-disabled
deadlock fix:
ShellCoreTrackerAdapterTests (+3 tests, +1 expanded):
- EspExited → DecisionSignalKind.EspExiting mapping (kind, source, evidence)
- No-dedup contract: every 62407 occurrence forwarded individually so the
reducer's ShouldTransitionToAwaitingHello guard can disambiguate the genuine
post-AccountSetup exit from intermediate Device→Account exits.
- Source-event timestamp preserved (not _clock.UtcNow) — critical for backfill
so EffectiveDeadlineBase floors HelloSafety at the actual ESP-exit instant.
- Existing All_three_events_can_fire_in_one_session expanded to All_four_*.
ClassicHelloDisabledCompletionTests (6 tests):
- Fast-path happy path: HelloPolicyDisabled + AccountSetup + DesktopArrived →
Finalizing → FinalizingGrace deadline → enrollment_complete.
- Three guard tests: fast-path skipped when policy not detected, before
AccountSetup, and when policy is enabled.
- HelloSafety reducer path (Fix 1 belt-and-suspenders): EspExiting →
HelloSafety 300s → Timeout → DesktopArrived → Finalizing.
- Fast-path cancels an armed HelloSafety deadline so it cannot fire
post-Completion.
ShellCoreToEngineEspExitingTests (2 wire-through tests):
- Real ShellCoreTracker.HandleBackfillRecord(esp_exiting_description) drives
the .NET event → adapter subscription → ingress.Posted, with source
timestamp preserved.
- End-to-end: same backfill → adapter signal → engine.Reduce on a seed state
with AccountSetup entered → HelloSafety deadline armed at sourceTime+300s.
Full DecisionCore.Tests: 293/293 pass. Full Agent.V2.Core.Tests: 1048/1048 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When HelloPolicyEnabled is explicitly false AND AccountSetup has been reached,
synthesise HelloOutcome="Skipped" on DesktopArrived and route directly through
Finalizing instead of waiting out the full HelloSafety 300s window. This is the
belt-and-suspenders companion to PR1's EspExiting reducer path: whichever fires
first wins, and the dual-path design survives a future regression in either
half (missing EspExiting adapter or missing HelloPolicyDetected adapter).
Both guards are required:
- HelloPolicyEnabled?.Value == false (policy explicitly disabled)
- AccountSetupEnteredUtc != null (mirrors ShouldTransitionToAwaitingHello)
This prevents a premature mid-DeviceSetup completion if HelloPolicyDetected
happens to fire before AccountSetup is observed. Unknown policy
(HelloPolicyEnabled == null) keeps the prior pessimistic behaviour.
The fast-path cancels HelloSafety on its way through so a deadline armed by an
earlier EspExiting cannot fire post-Completion and dead-end in the reducer.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ShellCoreTracker observes the Shell-Core 62407 OOBE_ESP*Exiting event but only
emitted it as a wire-event + FinalizingSetupPhaseTriggered notification. No V2
SignalAdapter mapped it to DecisionSignalKind.EspExiting, so HandleEspExitingV1
(which arms the HelloSafety 300s deadline) was never invoked. On Classic-v1
enrollments with Hello policy disabled this leaves the reducer stuck in
EspAccountSetup until the agent max-lifetime watchdog fires after 360 min.
This commit adds a dedicated EspExited .NET event on ShellCoreTracker (raised
both on live event log records and on backfill) and a matching subscription in
ShellCoreTrackerAdapter that posts a DecisionSignalKind.EspExiting signal.
OccurredAtUtc is taken from the source event timestamp (not _clock.UtcNow) so
HandleEspExitingV1's EffectiveDeadlineBase floors HelloSafety at the actual
ESP-exit instant rather than wall-clock-now on a backfilled run.
The adapter does NOT dedup EspExiting because Shell-Core 62407 fires at every
ESP phase transition (Device→Account, Account→End); the reducer's
ShouldTransitionToAwaitingHello guard already distinguishes the genuine
post-AccountSetup exit from intermediate exits.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
F1 — DELETE /api/sessions/{sessionId} bypassed cross-tenant validation
because the catalog entry had no TenantScoping. A Global Admin clicking
"delete" on a session in tenant B from the portal silently routed the
delete into their own JWT tenant A, never finding the row. Catalog
entry gains TenantScoping.QueryParam; DeleteSessionFunction switches
from TenantHelper.GetTenantId (JWT-only) to RequestContext.TargetTenantId
(middleware-validated).
F2 — RegisterSessionFunction wrote sessions without any deletion-state
check; a mid-cascade register could mutate a Preparing/Queued/Running
row, and a fresh register past the tombstone could re-Add a row the
manifest no longer covers. Guard now runs at the entry of the shared
ProcessRegisterAsync core — covering BOTH /agent/register-session
(cert-auth) AND BootstrapRegisterSessionFunction (bootstrap-code-auth),
which previously bypassed the cert-auth entry's check. 410 Gone on
lock-state or tombstone hit.
F3 — SessionDeletionGuard's "Sessions row missing -> silent pass"
contract let late ingests / registers slip past a completed cascade
and orphan rows the manifest couldn't describe. New short-lived
SessionTombstones table (PK=tenantId, RK=sessionId) carries a marker
written by SessionDeletionHandler.ExecuteTombstoneAsync BEFORE the
Sessions-row delete, with a 7-day default ExpiresAt. Guard now reads
the marker on a missing Sessions row; live and present -> throws with
sentinel state "Tombstoned" -> 410 Gone. Full-Restore deletes the
marker after re-inserting the Sessions row; partial-restore leaves it
alone (never written). SessionDeletionMaintenanceFunction pruning
step runs even when the kill-switch is active (markers are race
shields, not policy artefacts).
F4 — CrossTenantAccessTests had no coverage for DELETE
sessions/{sessionId}. Theory entries lifted into the same three asserts
that already cover GET — same-tenant allowed, cross-tenant non-GA
blocked, cross-tenant GA allowed — and into the QueryParam-scoping
catalog theory so the regression that produced F1 surfaces in CI.
Bootstrap follow-up — RegisterSessionFunction.Run had the guard inline
but BootstrapRegisterSessionFunction calls _inner.ProcessRegisterAsync
directly, bypassing it. Guard moved into ProcessRegisterAsync so both
entry points hit it before StoreSessionAsync. Added structural wiring
tests (DI field, internal accessibility, bootstrap delegation type)
that fail in CI if the architecture regresses to inline copy.
Tests:
- SessionDeletionGuardTests +3: marker-active blocks, no-marker passes
on missing row, sessions-row-exists skips tombstone probe.
- SessionDeletionHandlerTests +1: marker is written BEFORE the
Sessions-row delete (Moq Callback ordering assertion).
- SessionRestoreServiceTests +2: full-restore deletes marker,
partial-restore does not.
- SessionDeletionMaintenanceFunctionTests +2: prunes expired markers,
pruning runs even with kill-switch active.
- CrossTenantAccessTests +3 inline DELETE cases.
- RegisterSessionGuardWiringTests (NEW): asserts guard DI field,
ProcessRegisterAsync internal accessibility, BootstrapRegisterSession
delegation type.
- CascadeVerificationServiceTests.FakeReader gains the new
GetActiveSessionTombstoneAsync interface member.
Suites: Backend xUnit 1524, Agent V1 530, Agent V2 1042 — all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
F1 — V2 ingest writes were landing past the cascade lock. Wired
SessionDeletionGuard.EnsureWritableAsync into IngestTelemetryFunction
before PersistItemsAsync so Event / Signal / Transition writes are
refused with 410 Gone when DeletionState is Preparing / Queued /
Running / Poisoned. Switched UpdateSessionImeAgentVersionAsync from
UpsertEntityAsync to UpdateEntityAsync(ETag.All, Merge) with 404
swallow so a tombstoned Sessions row can no longer be silently
resurrected by an in-flight ingest.
F2 — Partial-restore could leave SoftwareInventory at -N drift when
the cascade poisoned mid-AGGREGATE-loop: per-key decrement was
persisted into AggregateDecrementsApplied BEFORE the underlying
decrement, but the gate used CompletedSteps.Contains(aggregate.Order)
— which is only set AFTER the whole loop. Switched the gate to
AggregateDecrementsApplied as the authoritative ledger and pass a
filtered DeletionStep into RunInventoryReIncrementWithIdempotencyAsync
so only the keys that were actually decremented are re-incremented.
F3 — Legacy direct-delete path deleted Events / RuleResults /
AppInstallSummaries before the ETag-CAS-guarded DeleteSessionAsync,
leaving a TOCTOU window where a V2 producer could CAS-claim
DeletionState=Preparing after the pre-read but before our side-
table mutations — landing the producer's manifest snapshot on top
of already-half-purged state. Reordered both admin (DeleteSessionFunction)
and retention (SessionRetentionFanoutService) paths to Sessions-CAS
first; side-tables only after the Sessions row is gone (no producer
can claim a missing row).
F4 — Restore per-row Add fallback only caught HTTP 409 for duplicate-
row conflicts; Azurite and some legacy Azure deployments surface
duplicate-row as 400 with ErrorCode=EntityAlreadyExists, breaking
idempotent restore-retry after partial-failed restore. Centralised
the check via IsAlreadyExistsStatus(rfe) helper accepting 409 OR 400
+ ErrorCode=EntityAlreadyExists; non-duplicate 400s still propagate.
Tests:
- SessionRestoreServiceTests: 2 partial-restore tests refit to the new
AggregateDecrementsApplied contract + new explicit Codex-scenario
test (only-decremented-keys re-incremented, untouched key skipped).
- SessionDeletionPr4cCodexFixesTests F4: harness setup adds full
AggregateDecrementsApplied so the per-key idempotency assertion
reflects the new gate.
- SessionRetentionFanoutServiceTests: pre-read tests replaced by two
tests covering "DeleteSessionAsync refuses → no side tables" and
"DeleteSessionAsync succeeds → side tables run after".
- TableStorageRestoreBatchTests: two new F4 tests — 400
EntityAlreadyExists treated as Skipped, 400 with non-duplicate
ErrorCode still propagates.
- UpdateSessionImeAgentVersionTests (NEW): asserts UpdateEntityAsync
is used (not Upsert), 404 swallowed silently, other storage errors
non-fatal.
Suites: Backend xUnit 1510, Agent V1 530, Agent V2 1042 — all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three Round-1 review findings from PR6:
F1 (High) — Legacy retention path mutated side-tables before the
CAS-guarded DeleteSessionAsync. Plan PR6 step 7 says "legacy must skip
locked sessions", but our LegacyDeleteAsync deleted Events / RuleResults
/ AppInstallSummaries first and only discovered the V2 cascade lock at
the tombstone. Fix: pre-read the Sessions row in LegacyDeleteAsync; if
DeletionState != None, return false WITHOUT touching any side table.
The original CAS-guard stays as a backstop for the T1-T3 race after the
pre-read.
F2 (High) — Kill-switch checked once per run; a flip-ON mid-fanout kept
legacy deletes running. SessionRetentionFanoutService now takes
AdminConfigurationService, calls IsSessionDeletionKillSwitchActiveAsync
(uncached, fail-closed per PR5 F1) at both the per-tenant boundary and
inside the per-session loop. FanoutResult gains AbortedByKillSwitch so
the per-tenant audit + the global OpsEvent surface the early halt.
F3 (Medium) — Global-scope lifecycle audits used LogAuditEntryAsync with
tenantId: null!, which the table-storage helper silently dropped because
the schema requires a non-null PartitionKey. Three event types
(Completed / FanoutSkipped / Failed) moved to OpsEvents (TenantId is
nullable on OpsEventEntry). Two new helpers in OpsEventService —
RecordSessionDeletionMaintenance{Completed,FanoutSkipped}Async — also
dual-registered in the OpsAlertRulesSection.tsx catalog per memory
feedback_ops_event_types_dual_register. Per-tenant audits inside the
fanout (which already had a real tenantId) are unchanged.
Tests: 4 new xUnit cases — F1 skip-without-mutation, F1 row-not-found
skip; F2 mid-tenant abort, F2 between-tenants abort. Existing 17 PR6
tests updated to assert OpsEvents instead of audit calls for the global
lifecycle events. Total: 1504/1504 backend pass, 347/347 web pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Oliver Kieselbach