Classic UD fleets held at the Device-ESP timeout wall by one slow blocking
app showed ~100% Failed (tenant c9787ba2: 948/1000 esp_terminal_failure)
although the profile allows "Continue anyway" and users demonstrably reach
the desktop — the classic advisory defang requires AccountSetupEnteredUtc
and never applied there.
New operator-set tenant opt-in EnableEspContinueAnywayObservation
(ConfigVersion 36 -> 37, default off):
- DecisionCore: a Device-phase EspTerminalFailure with ContinueAnyway
allowed + opt-in observed (EspConfigDetected payload fact — kernel stays
config-free) + Classic/Unknown non-pre-prov scenario defangs into the
advisory rail with a 60-min AdvisoryCompletion window. Real end via the
DAD-validated real-user desktop + Hello gate (IME user-session gate is
structurally unavailable without the AccountSetup anchor): eager
completion on DesktopArrived, category recovery via the 4910a5a5 hook,
expiry un-defangs to the original esp_terminal_failure — every observed
session still gets a terminal verdict.
- Honest outcome: enrollment_complete under an unresolved advisory carries
espSoftFailure/completionSource (also on the classic post-AccountSetup
advisory path); backend stamps EspSoftFailure/CompletionSource on the
Sessions row (+ index mirror, terminal-tuple preservation); web renders
an amber "with issues" badge plus a session-detail banner. No new
terminal status — Succeeded + marker.
- Agent: remote-config field merged into AgentConfiguration; stamped into
both esp_config_detected emitters (orchestrator bootstrap via live
Func<bool> seam, DeviceInfoCollector via host/factory pass-through).
- Admin UI: GA-only toggle in the tenant management modal (next to
Bootstrap Token) + stored/runtime rows in the tenant config report;
GA-gated on both write paths (PUT silently reverts for tenant admins,
field patch rejects with 400).
Tests: 13 new ContinueAnywayObservationTests; DecisionCore 513, Agent V2
2162, Functions 3948, SummaryDialog 44, vitest 632, tsc clean.
Docs: docs/agent/continue-anyway-observation.md (OKF) + index/log.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
While a tenant offboarding is queued or draining, the TenantConfiguration
tombstone (Disabled=true, 'Offboarding in progress') is the cascade's only
auth gate. A GA lifting Disabled mid-cascade re-opened the tenant against a
wipe that was still coming: data written afterwards was destroyed, and
sessions registered after enumeration survived as orphans while their event
partitions were wiped.
Three layers close it:
- PUT config/{tenantId} returns 409 while the row is the offboarding
tombstone — for all callers, GA included
- TenantConfigPatchService refuses patch AND revert against the tombstone
(a revert restores a pre-offboarding snapshot with Disabled=false — the
exact race)
- TenantOffboardingHandler re-reads the config before the destructive
post-drain phases and fail-closes with 'unoffboard_race' when the
tombstone is gone but the wipe has not run (last line of defense)
Shared predicate: TenantOffboardFunction.IsOffboardingTombstone.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(backend): pre-write config backup net - every tenant/admin config save snapshots first
New ConfigurationBackups table (PK=tenantId or GlobalConfig, RK=reverse-ticks+guid,
newest-first) holds the raw stored row as EntityJson before any overwriting save of
TenantConfiguration or AdminConfiguration. Hook lives in TableConfigRepository, the
single choke point all writers funnel through - fail-soft (a backup hiccup never
breaks a save), skips row creations and noise-only changes (LastAuthClientId flips),
prunes to the newest 2 per partition. Source/reason tagging via a new
SaveTenantConfigurationAsync overload (not optional params - Moq expression trees,
CS0854); portal-put/plan/auth/app-homing/offboard call sites tagged.
Offboarding: backup partition wiped in the bulk phase (TenantPartitionTables) and by
the marker-cleanup tombstone sweep; the GlobalConfig partition never matches a tenant
GUID and survives. Table registered in TableNames.All + CriticalBackupTables.All.
Groundwork for the transactional MCP config-write flow (patch/revert endpoints follow).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(backend): transactional tenant-config field patch + revert with verified writes
New GA-only surface (policy catalog: GlobalAdminOnly + RouteParam):
- PATCH config/{tenantId}/fields - field-level patch through TenantConfigPatchService:
fresh ETag read -> field gate (deny-list incl. plan/trial, HomedAppClientId,
auth provenance; unknown fields 400 with the name) -> patch onto a clone ->
shared validation -> fail-CLOSED backup -> CAS replace -> fresh re-read ->
verify that EXACTLY the intended fields changed -> automatic ETag-conditional
rollback on drift (catches Store/Map serialization asymmetry).
- POST config/{tenantId}/revert - restore from a ConfigurationBackups snapshot
(latest by default); snapshots current state first (revert is revertible),
preserves plan/homing/provenance fields unless includeProtectedFields.
- GET config/{tenantId}/backups - metadata + masked diff only, never EntityJson.
Supporting changes: IConfigRepository gains GetTenantConfigurationWithEtagAsync +
TryReplaceTenantConfigurationAsync (412 -> false, no implicit backup hook);
validators moved from the PUT into shared TenantConfigValidation (PUT re-pointed,
forwarding shims keep test call sites); GET config/{tenantId} honors ?view=redacted
so a GA's MCP session never pulls clear-text secrets into model context.
Caller-tier deny-lists (TenantAdmin/DelegatedAdmin) ship now for the phase-2 flip.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat(mcp): tenant-config write tools - update/revert with snapshot-backed verified writes
First mutating surface on the MCP server, strictGa (real Global Admin) only:
- get_tenant_config: full config read, ALWAYS ?view=redacted (secrets never
reach model context; pinned by config-write-tools tests)
- update_tenant_config: field-level patch via PATCH config/{tenantId}/fields;
required reason; response = applied fields + masked diff + backupId
- list_tenant_config_backups: snapshot metadata + masked diff only
- revert_tenant_config: restore latest/named snapshot; protected fields stay
current unless includeProtectedFields; revert is itself revertible
New MUTATING annotation + TenantGuidSchema (path-traversal guard) in shared.ts;
role-aware instructions headline (non-GA keeps the exact READ-ONLY wording);
catalog snapshot test extends GA_FULL + new CONFIG_WRITE_GA_STRICT difference
list (Global Reader/tenant/delegated see none of the four).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* test(backend): de-flake session-deletion heartbeat test - condition-gated handler instead of wall-clock window
The heartbeat test held the handler for a FIXED 350ms and expected the 50ms
heartbeat timer to tick >=3 times inside that window. On loaded CI runners the
heartbeat Task.Delay stretches disproportionately to the handler delay, the
handler completes with <3 heartbeats fired, and the awaited condition can never
become true afterwards (CI failure on PR #120). The handler now holds the
message in-flight UNTIL the test observed 3 heartbeats (with ct + own 15s
deadline as termination guards), so the assertion is reached deterministically
regardless of scheduler jitter. Verified: 12/12 local runs green, full suite green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Two findings from the re-run public-availability review:
- UpdateTenantConfigurationFunction preserved HomedAppClientId, LastAuth*
and plan fields against round-trip clobber but not OnboardedBy, despite
AuthFunction documenting it as write-once: any own-tenant admin PUT could
overwrite or null it, corrupting audit provenance and letting a
revoke -> re-approve cycle auto-promote an attacker-chosen UPN. Now
preserved like the other system-owned fields.
- TenantAutoApproveHandler checked the suspension gate through the cached
TryGetConfigurationAsync (5-min per-instance TTL): a tenant suspended
right after signup could still be auto-approved from a stale view. Now
reads via GetConfigurationFreshAsync; a storage error propagates so the
message retries (flag re-checked per attempt) instead of silently
downgrading the tenant to manual approval.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prod incident 2026-07-31 18:03Z (tenant b54dc1af): the consent-driven homing
flip landed and 600ms later the frontend persisted the validation toggle via
the generic config PUT - round-tripping a config loaded BEFORE the flip. The
GA branch treated the stale null HomedAppClientId as an intentional revert to
legacy, silently undoing the flip (App Insights: AppHomingFlipped 18:03:00.44,
PUT revert warning 18:03:01.03).
- HomedAppClientId is now IMMUTABLE via the generic PUT for ALL callers, GA
included: a full-model round-trip can never carry flip intent. The only
writers are the consent auto-flip and POST config/{tenantId}/app-homing
(which the GA tenant-management panel already uses).
- HandleAuthClientIdTrackingAsync (same failure class, cross-instance): the
last-seen tracking write now mutates a cache-BYPASSING fresh read via new
TenantConfigurationService.GetConfigurationFreshAsync instead of blind-saving
the up-to-5-min-stale cached view - the login right after a flip is exactly
the one that carries a changed audience and triggers this write.
3 regression tests (fresh-entity mutation preserves a concurrent flip,
fresh-already-current suppresses the write, steady state stays zero-I/O).
Backend xUnit 3767/3767.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
True parallel operation, no forced re-consent: existing tenants keep signing
in via the legacy app (1a400946), new tenants land on the C4A8 app (886ab5e2).
Model "default legacy, funnel primary, learn in the background":
- Web: am_auth_app flag -> NEXT_PUBLIC_ENTRA_DEFAULT_APP -> primary; get-started
CTA hands over with ?authapp=primary; auth/me homedApp updates the flag so an
operator tenant flip takes effect on each user's NEXT login; AADSTS90094/65001
one-shot cross-app retry, AADSTS65004 -> failed screen with previous-app link;
am_login_attempt pins redirect returns to the app that started them.
- Backend: TenantConfiguration.HomedAppClientId (null = legacy, GA-only flip via
PUT config) + LastAuthClientId/Since observability; EntraAppRegistry resolves
the acting app per tenant (no-op until the legacy config pair is set); Graph
client-credentials, admin-consent URLs and the surfaced client id follow the
homed app (app-scoped cache keys); JwtBearer [Authorize] pipeline now shares
the middleware audience builders (was single-audience - would 401 legacy
tokens after the swap); middleware trusts EntraId:LegacyClientId implicitly.
- well-known association file now lists both app ids (publisher domain);
deploy-web passes NEXT_PUBLIC_ENTRA_LEGACY_CLIENT_ID / _DEFAULT_APP;
mcp-server.bicep default client id -> C4A8 app.
Backend xUnit 3730/3730, vitest 625/625 (13 new), tsc/ESLint clean, next build ok.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The tenant contact address had no server-side validation, so a direct API
caller could store any string in the field service notices are sent to.
UpdateTenantConfiguration now rejects non-addresses, recipient lists (a comma
would silently widen who receives those notices), display-name forms and
control characters (mail-header injection), and normalizes what it stores.
Empty stays valid - having no contact address is a legitimate state.
The seed path had a worse problem than missing validation. It was a
read-modify-write of the whole model through SaveTenantConfigurationAsync,
which upserts with TableUpdateMode.Replace and no ETag: a concurrent portal
save was clobbered by the seeder's stale snapshot of EVERY field, not just
ContactEmail. It also wrote straight through IConfigRepository, bypassing the
TenantConfigurationService cache, so a seeded address stayed invisible for the
full 5-minute TTL.
TrySeedTenantContactEmailAsync now writes ETag-conditionally in Merge mode -
one property, and a 412 means someone else wrote the row so the seed abandons
rather than overwriting the winner. A lost seed is recoverable on the next
maintenance run; a lost admin edit is not.
The same seed logic existed twice (preview notification-email save and the
maintenance backfill). Both now call a single owner on
TenantConfigurationService, which holds the never-overwrite invariant, the
cache eviction and the fail-soft guarantee. Fail-soft lives in the service
rather than the repository so it holds for any implementation.
Tests cover the validator, the service contract, and the repository against a
mocked TableClient: Merge mode, the ETag actually passed, only ContactEmail in
the patch, the 412 race, 404, storage faults. Plus a regression pin that
SaveTenantConfigurationAsync is never called from the seed path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Phase 1 - named notification channels (1..N per tenant):
- NotificationChannel model (provider/url/headers/enabled + per-event
opt-ins incl. single NotifyOnSlaEvents); stored as JSON column
NotificationChannelsJson, cap 10, fail-soft ParseList
- No data migration: GetNotificationChannels() synthesizes one channel
from the legacy single-webhook fields under stable id "legacy";
web materializes on first save and clears legacy fields
- All dispatch sites (ingest success/failure/white-glove/ESP, SLA breach/
resolved/consecutive, enrollment-start, hardware-rejected, test) route
via SendToChannelsAsync with per-channel toggle filtering
- Per-channel secret redaction (structure stays visible to readers) with
restore-by-id; save-time validation (dup ids, provider, https, headers)
- Settings UI: channel card list with add/remove, event pills, per-channel
test button ({channelId} body, legacy-compatible without body)
Phase 2 - analyze-rule channel notifications:
- NotifyDefault (definition, false for all built-ins, test-guarded) +
Notify/NotifyChannelIds tenant override in RuleStates (two-tier mirror
of MarkSessionAsFailed); custom rules fold into the tenant-owned row
- Dispatch in AnalyzeOnEnrollmentEndHandler only: outcome.Results is
new-findings-only (engine dedup) so no re-notification; manual
"Analyze Now" reanalyze never dispatches
- BuildRuleFiredAlert (eventType analyze_rule_fired): device, serial,
rule, severity, category, confidence, explanation + session deep link
- Rule card: notify toggle + channel target pills, bell header badge
2858/2858 Functions tests (19 new), web tsc + vitest clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fail-closed entitlement layer (FeatureEntitlementCatalog + TenantEntitlementService): edition resolves at read time (PlanTier == 'enterprise' OR active trial; legacy free/pro => Community, no migration).
- Plan/trial endpoints: PATCH config/{tid}/plan (GA: tier + grant/extend/end trial, audited, cache-safe) and POST config/{tid}/trial (tenant-admin self-service 30d, once per tenant, 409 after). Generic PUT preserves plan/trial fields for all callers. Fixes SetPlanTier writing past the config cache without audit/UpdatedBy.
- Rate limits: Enterprise floor 150/min for BOTH user (portal/JWT) and device (agent/cert) paths - floor raises the global default, an explicit per-tenant override still wins; MSP users follow their home tenant.
- Retention cap: Community 90d / Enterprise 365d, clamped in the retention fanout (warn log) + non-GA write validation 7..cap (only on changed values; 0=infinite stays GA-only).
- Delegated admin (MSP): Enterprise gate on the delegated admin's HOME tenant (JWT tid, fail-closed on unknown) - managed target tenants may be any edition. Grant-time target check removed; inert grants are suppressed at resolve time (single choke point in GetScopeAsync, applied after the UPN-keyed cache).
- MCP quota: McpQuotaService + McpQuotaEnforcementMiddleware (X-Client-Source: mcp only, GA exempt) enforce daily/monthly budgets - per-user plan override > tenant edition plan; limits from SectionUsagePlans with catalog fallbacks (community 100/3000, enterprise 1000/20000); 429 with X-MCP-Quota-* headers + Retry-After; fail-open on counter reads, fail-closed on plan resolution. /api/metrics/mcp-usage/me reports effective plan + counters.
- Trial visibility: TrialExpirySweepFunction (daily 03:30 UTC, informational) emits TenantTrialExpired (Warning, 24h look-back) + TenantTrialExpiring (Info, <=3d heads-up); dual-registered in web OPS_EVENT_TYPES.
- Web: EditionBadge + one-time trial CTA in settings header, Plan & Trial card in the GA tenant edit modal (own PATCH save), retention cap hints from entitlements, pure lib/edition.ts (vitest), feature-flags surface edition/trial/entitlements; MCP TENANT_SAFE_FIELDS + trial fields.
Verified: solution build clean; backend 2653/2653 tests; web tsc + 516/516 vitest; MCP server tsc + tests green.
Replace the denormalized per-tenant RateLimitRequestsPerMinute mirror (and its
SaveConfigurationAsync sync job) with compute-at-read resolution: effective limit =
per-tenant override ?? global default. Fixes the clobbering bug where a per-tenant
edit to the base field was overwritten on the next global config save.
- Model: drop TenantConfiguration.RateLimitRequestsPerMinute; add nullable
CustomUserRateLimitRequestsPerMinute (device override CustomRateLimit... kept).
- RateLimitResolver centralizes precedence (device + user); both paths call it.
- SecurityValidator reads the global default (AdminConfigurationService threaded
through ValidateSecurityAsync + all device-auth callers); 5-min cached.
- UserRateLimitMiddleware: per-tenant user override for tenant-scoped callers only
(skipped for HasGlobalScope = GlobalAdmin|GlobalReader); GA -> GA budget, others ->
user default. Uses GetConfigurationIfExistsAsync so an unregistered-tenant JWT can
no longer auto-create a config row via read endpoints.
- Validation: reject rate limits < 1 at the write functions; RateLimitService clamps
< 1 to 1 as a safety net (avoids Min()-on-empty / throttle-everything).
- Web: Global Settings exposes User + GlobalAdmin limits; tenant modal writes the
override fields (blank = inherit); report + usage-plans updated.
Legacy RateLimitRequestsPerMinute Table column is ignored on read and dropped on the
next Replace save (self-cleaning; no migration).
Backend 2577 xUnit green, tsc clean, web 506 vitest green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GlobalReader = cross-tenant READ visibility like a Global Admin, but no mutating
operations and no platform settings. Provisioned exactly like a GA: a GlobalAdmins
table row with Role=GlobalReader (empty/legacy Role = GlobalAdmin, back-compat).
Semantics are ADDITIVE — a user who is also their own tenant's Admin keeps edit
rights on that tenant; a pure GlobalReader is read-only everywhere.
Backend:
- GlobalAdmins.Role column + GlobalAdminService.GetGlobalRoleAsync; IsGlobalAdminAsync
(incl. TableAdminRepository) delegates so a GlobalReader row never reads back as admin
- RequestContext.IsGlobalReader + HasGlobalScope
- Split GlobalAdminOnly into GlobalReadOrAdmin (cross-tenant reads) vs GlobalAdminOnly
(writes/settings) + TenantAdminOrGlobalReader for config-read
- PolicyEnforcementMiddleware: extracted testable DecideAsync seam; cross-tenant gate keys
off HasGlobalScope; additive own-tenant role carried so IsTenantAdmin stays correct
- Config secret redaction for readers (RedactedCopyForReader) + save-time placeholder
restore (RestoreRedactedSecretsFrom) so a redacted view can't blank real secrets
- raw table/log tools stay GlobalAdminOnly; SignalR global-admins read group admits HasGlobalScope
- /auth/me emits isGlobalReader (+ preview-gate bypass, no auto-admin); /auth/mcp emits globalRole
MCP: hasGlobalScope (GA or Reader) drives routing + tool catalog; raw secret-bearing tools
(query_table/list_tables/query_backend_logs) gated strict-GA only.
Web: AuthContext isGlobalReader + hasGlobalScope; reader reaches /dashboard + cross-tenant
views; platform Settings/Ops/Backups/Software GA-only (nav-hidden + requireGlobalAdmin route
layouts); mutating controls on mixed data pages disabled via useCanMutatePlatform; rules pages
read-only on foreign tenants; additive own-tenant edit preserved.
Tests: backend 2369, MCP 252, web 469 green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds WebhookProviderType.GenericJson (20) so a tenant can POST a stable,
channel-agnostic JSON payload to any HTTPS endpoint - ticketing systems,
automation, or an SMTP gateway like Postal - instead of only Teams/Slack.
Backend:
- GenericJsonRenderer emits schemaVersion 1.0 with eventType, severity,
title/summary/themeColor, primaryUrl (first openUrl action), facts[],
sections[], actions[]; registered in WebhookNotificationService.
- NotificationAlert.EventType set on all 8 builders (enrollment_succeeded/
_failed/_started, preprovisioning_*, hardware_rejected, sla_breach/_resolved,
consecutive_failures, test) for consumer-side routing.
- Per-tenant WebhookCustomHeadersJson + GetGenericWebhookHeaders() (generic-only,
restricted-header denylist, fail-soft parse) for API-key auth; dispatcher
gained optional customHeaders applied via HttpRequestMessage; threaded into
all 12 tenant call sites (Ops/admin-config sites unchanged).
- TableConfigRepository persists the field; UpdateTenantConfigurationFunction
validates it (object, string values, RFC-7230 token names, CRLF-injection
guard, 8 KB / 25-header caps).
Web:
- Generic provider option + key->value WebhookHeaderEditor (rendered only for
provider 20); leaving the generic provider clears local headers so a later
switch back cannot revive stale auth headers; context/types wiring sends
headers only when provider===20; report shows Generic JSON provider name.
ConfigVersion unchanged (backend-only, no agent payload). No new
Constants.EventTypes, so no MCP catalog sync. Still fire-and-forget (no SMTP
client / retry by design). New GenericWebhookTests cover renderer, header
parsing/denylist, and save-time validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Grant tenant member roles via Entra ID app-role assignments on the
Enterprise App (the token's "roles" claim), in addition to the
TenantAdmins table. Per-tenant opt-in via the new GA-only
EntraAppRolesEnabled flag (backend-only, no ConfigVersion bump).
Resolution is table-first and tri-state (NotPresent/Disabled/Enabled):
an enabled table row wins, a disabled row is an explicit deny that
suppresses the claim entirely (a disabled member can never be
re-authorized through their app-role assignment), and only a missing
row falls back to the claim. Only Admin and Operator are claim-mappable
(Admin outranks Operator); Viewer and the platform GlobalAdmin role are
never claim-derived.
All effective-role consumers share one pure decision authority
(EntraAppRoleResolver.Resolve): PolicyEnforcementMiddleware, auth/me,
and feedback/status — the latter previously used a table-only check, so
Entra-only members did not see the feedback bubble.
GA toggle wired through the BootstrapToken pattern: TableConfigRepository
persistence, UpdateTenantConfigurationFunction GA-only protection, and a
checkbox in the admin tenant editor.
Tests: EntraAppRoleResolverTests (incl. disabled-row deny) + an
auto-promote regression guard; 2308/2308 backend green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Cascade-delete V2 is the only path now. Drops the per-tenant
EnableCascadeDeleteV2 flag, the legacy ETag-CAS-guarded DeleteSessionAsync
helper, the AppInstallSummaries cleanup helper (only legacy caller),
and the SessionsLegacyDeleted counter chain (fanout → maintenance → OpsEvent).
DeleteSessionFunction collapses to a single kill-switch gate; the producer
owns 404 / 409 / Poisoned mapping AND the intentional Queued/Preparing+Snapshot
→ 202 recovery resume — gating those at HTTP would strand crashed admin
cascades. Fanout now aborts both the inner loop AND outer loop on producer
KillSwitchActive instead of leaking calls into the next tenant.
Test coverage:
- DeleteSessionCasGuardTests removed (helper gone)
- AdminConfigurationSessionDeletionKillSwitchTests trimmed to kill-switch only
- DeleteSessionFunctionTests pinned to the kill-switch short-circuit + null
- SessionRetentionFanoutServiceTests: legacy path tests dropped, two new
abort regressions for producer KillSwitchActive (inner + outer loop)
Build: 0 warnings, 0 errors. Backend: 1526 passed / 0 failed.
Web: tsc clean, vitest 346 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Operationalises the 7-PR backend cascade-delete pipeline that landed via commits
c8658b9f..921994e8: per-tenant V2 opt-in switch, audit-trail consolidation, and
a Global-Admin-only Session Cleanup page to track in-flight / poisoned / stranded
cascades and trigger restores.
== PR-A: per-tenant V2 enrollment ==
- UpdateTenantConfigurationFunction extends the GA-only gate list with
EnableCascadeDeleteV2 so tenant admins cannot self-opt-in / out of the
snapshot+restore pathway.
- TenantManagementSection.tsx adds a purple-bordered "Enable V2 Cascade Delete"
checkbox in the GA edit modal and a "Cascade V2" badge on the tenant card so
rollout state is visible at a glance.
== PR-B: audit consolidation ==
- SessionDeletionHandler drops the 15-per-cascade deletion_step_completed /
deletion_step_failed / deletion_verification_failed audit emissions. Step
telemetry lives in DeletionProgress + structured logs; tenant admins now see
just deletion_started + deletion_completed + deletion_restored.
- SessionDeletionWorker no longer writes a deletion_poisoned tenant audit;
the signal moved to a new SessionDeletionPoisoned OpsEvent (operator-bound,
Telegram-routable, dual-registered in OpsAlertRulesSection.tsx catalog).
IMaintenanceRepository dependency removed from the worker constructor.
- SessionRestoreService stops emitting the deletion_manifest_downloaded
breadcrumb; deletion_restored is the operator-visible outcome.
== PR-C: Session Cleanup admin page ==
- New /admin/ops/session-cleanup route (gated by app/admin/layout.tsx's
ProtectedRoute requireGlobalAdmin). Sidebar entry under ga-ops alongside
Maintenance. Four URL-state tabs: In-Flight / Poisoned / Stranded Queued /
Activity (links to existing /audit page).
- Backend reads: GetSessionDeletionsListFunction
(GET /api/global/session-deletions?state=&strandedSinceMinutes=) +
GetSessionDeletionManifestFunction
(GET /api/admin/sessions/{id}/deletion-manifest?tenantId=&manifestId=&mode=).
Both registered as GlobalAdminOnly in EndpointAccessPolicyCatalog.
- DeletionPreviewModal reads the persisted snapshot via the stored-manifest
endpoint (NOT a fresh dry-run via the existing /delete/preview, which would
enumerate live-but-partially-deleted data for in-flight cascades). Renders
manifest counts, snapshot SHA, worker progress (CompletedSteps,
VerificationDone, TombstoneStarted, aggregate decrements), and the last
verification failure with residual sample. Download button uses
authenticatedFetch + Blob + Object URL so the Bearer token is sent.
- RestoreConfirmDialog drops the Full/Partial radio (mode is auto-selected
by SessionRestoreService based on cascade state — forcing the wrong mode
would corrupt state). Reason field threads through to deletion_restored
audit details. Dry-run default = true and previews the auto-selected mode.
== Codex review follow-ups (across four rounds) ==
- High: tenantId is explicitly accepted on /delete/preview (query) and
/restore (body); GA cross-tenant resolution prefers the explicit value over
JWT TargetTenantId, falling back to SessionsIndex lookup. Without this, a
GA acting on a session in another tenant got their JWT home tenant.
- High: new stored-manifest endpoint replaces the dry-run preview for
in-flight / poisoned rows; previous fix built a fresh manifest against
already-partially-deleted data and misrepresented the cascade.
- Medium: SessionDeletionPoisoned OpsEvent now carries failureType +
failureMessage + observedResidualCount + a small residual preview, persisted
by the handler into DeletionProgress (LastFailureType, LastFailureMessage,
LastObservedResidualCount, LastResidualSampleJson) before the throw so the
worker can include them.
- Medium: dry-run restore reports the AUTO-SELECTED mode (full/partial) so
the dialog's preview reflects what the real run will do, instead of the
literal string "dryRun".
- Medium: residual count field renamed to observedResidualCount with an
honest XML doc — CascadeVerificationService caps at 50 per table AND
short-circuits after the first failing table, so the number is a lower
bound. UI surfaces "N of 50+ observed rows" when N hits the cap.
- Medium: three-layer cap defense on the OpsEvent residual sample to stay
under the 4096-char TableOpsEventRepository truncation: max 5 entries
(OpsEventResidualSamplePreviewSize), per-field max 96 chars with `…` marker
(OpsEventResidualKeyMaxChars), total preview budget 1200 chars
(OpsEventResidualPreviewBudgetChars) with trailing-entry drop. Full sample
stays in the progress blob for the admin UI.
- Low: estimatedSnapshotBytes field name aligned end-to-end; activity tab
text clarifies audit-action vs OpsEvent distinction.
Tests:
- Backend xUnit 1538/1538 pass (up from 1510 baseline; +28 across the new
Session Cleanup endpoints, OpsEvent enrichment, restore-mode reporting,
cap-defense unit + integration tests).
- Web vitest 347/347 pass.
- TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit tests for GetSessionDeletionsListFunction +
GetSessionDeletionManifestFunction (no HttpRequestData mock harness in repo;
would be a separate refactor).
- vitest for DeletionPreviewModal + RestoreConfirmDialog (defer until first
field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Oliver Kieselbach