mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-25 21:19:38 +00:00
d697e02f04331b1191407dec7b86c8b8ec897b06
538 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oliver Kieselbach
|
d697e02f04 | chore(backend,web): bump feedback comment limit 500 -> 4096 chars | ||
|
Oliver Kieselbach
|
c26b25b81b |
feat(backend,web): offboarding farewell-email capture + dedicated feedback table + reports tabs
- Persist tenant Preview-Notification-Email on OffboardingHistory row at Phase 1 so it
survives Phase 2 wipe; ResendEmailService.SendAsync wired but disarmed by a const-false
arm-flag with a [DRAFT] placeholder template — explicit flip required to ship.
- Move user feedback storage out of PreviewConfig discriminator into a dedicated Feedback
table (PK=InApp / Offboarding) that is NOT in any tenant-offboarding wipe list — feedback
from departing tenants is exactly the signal we want to keep. New IFeedbackRepository +
TableFeedbackRepository; FeedbackFunction migrated; (PreviewConfig, "Feedback")
removed from TenantOffboardingHandler.DiscriminatorTables.
- POST /api/tenants/{tenantId}/offboard/feedback collects a free-form farewell comment
during the drain-barrier countdown (marker-gated, fail-soft OpsEvent emit). Reports >
User Feedback page gains an Offboarding-Feedback sub-tab next to In-App Feedback.
- OffboardingFeedbackReceived OpsEventType registered (info-severity) + catalog entry
in OpsAlertRulesSection so admins can wire Telegram in two clicks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
bd9fc6f6f4 | feat(backend): ScriptNameCache cache negative verdict | ||
|
Oliver Kieselbach
|
863291ff7b |
feat(backend,web): App Insights telemetry for optional Graph add-on feature
So we can answer "which tenants actively have this on" and "how many people are using it" later — without bolting on a separate metrics store. Backend events - `GraphAddOnPermissionsDetected` (GraphFeatureDetector, fresh-acquire path): fires at most every ~55min per tenant (cache TTL). Props: TenantId, GrantedRoleCount, HasScriptDisplayNames, TokenCacheTtlMinutes. → distinct count over time = tenants who have the optional permission turned on. - `ScriptDisplayNamesResolved` (ScriptDisplayNameResolver, only when permission is active and the resolver actually did work): per session-view with scripts. Props: TenantId, RefsRequested, RefsResolved, CacheHits, TriggeredFullPull. → real usage pulse. - `GraphAddOnStatusChecked` (GET status endpoint): admin engagement with the capability page. Props: TenantId, UserId, IsTransient, ScriptDisplayNamesGranted. - `GraphAddOnRefreshTriggered` (POST refresh endpoint): high-signal "tenant just granted" event. Props: TenantId, UserId. Web events (via `trackEvent` -- tenantId auto-attached by the telemetry initializer) - `GraphAddOnStatusViewed` on Settings → Optional Graph capabilities mount. - `GraphAddOnGrantCommandCopied` on Copy command / Copy permissions click. Notes - All emits wrapped in try/catch with debug-level logging — telemetry failures must never poison the acquire / resolve hot paths. - Tests pass a no-op `TelemetryClient(new TelemetryConfiguration())` (same pattern as SlaTenantStatusLifecycleTests). 1718 backend / 364 web tests green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
e4c4dd57db |
feat(backend,web): optional Graph add-on permissions for Intune script display names
Adds opt-in resolution of Intune Platform Script + Remediation display names in
session timelines via a tenant-side appRoleAssignment grant. No app manifest
change, no impact on customers who don't enable it. Architecture is reusable for
future optional Graph permissions: catalog -> detector -> resolver -> endpoint.
Backend
- GraphFeatureDetector parses the `roles` claim from the SP access token and
caches the (token, roles) bundle per tenant. GetSnapshotAsync surfaces a
transient flag so admin-UI callers can render "unknown" instead of a
misleading "0 of 0 granted" when token acquire times out.
- GraphTokenService now accepts a CancellationToken so optional-feature callers
can apply tight budgets instead of inheriting the 5+15+30 s retry chain.
- ScriptDisplayNameResolver: cache-first, list-pull-on-cold-tenant, per-ID
fallback. Throttle-aware abort: skip per-ID for kinds whose full-pull hit a
transient failure; break the per-ID loop on the first HttpRequestException.
Total budget 5 s, with a tight 1.5 s sub-budget on the token acquire.
- Azure Table `ScriptNameCache` + direct-TableClient fail-loud repository
(mirrors TableOffboardingAuditRepository pattern). Daily
ScriptNameCacheCleanupFunction: 7 d positive / 1 h NotFound TTL.
- POST /api/tenants/{id}/scripts/display-names (body-carried refs to dodge URL
length limits; backend cap 200 refs/request).
- GET /api/tenants/{id}/graph-permissions/status (4 s budget; includes app
ClientId so the UI can render an exact grant command).
- POST /api/tenants/{id}/graph-permissions/refresh (invalidates token cache).
- All three routes registered in EndpointAccessPolicyCatalog (MemberRead for the
resolve, TenantAdminOrGA for the admin-UI surfaces).
Web
- `useScriptDisplayNames` hook with stable refset fingerprint (no spurious
refetches on parent re-renders); clears stale map on tenant/refset change so
a session switch never leaks names across tenants.
- Chunked POST loop: 200-ref batches with partial-success semantics. Preserves
earlier chunks when later ones fail (non-2xx OR transport throw); only
TokenExpiredError propagates so auth flow can react.
- ScriptExecutions renders " . {DisplayName}" alongside existing label/short-id,
keyed by Kind + policyId so Platform/Remediation under the same id never
collide.
- New Settings section "Optional Graph capabilities": status table with granted
/ not granted / unknown, copy-paste PS one-liner with ClientId baked in,
refresh button gated on response.ok.
Customer-facing
- scripts/CustomerSetup/Grant-AutopilotMonitorAddOn.ps1: idempotent
grant / verify / revoke script. -Features ScriptDisplayNames (high level) or
-Permissions ... (admin-UI generated). Pure ASCII for PS 5.1 compat.
- docs/customer-graph-add-on-permissions.md: end-to-end customer guide
covering prerequisites, the appRoleAssignment pattern, and revocation.
Tests
- 1718 backend xUnit / 364 web vitest. Highlights: JWT roles parsing edge
cases; full-pull-429 suppresses per-ID; per-ID-429 breaks loop; transient
detection via GetSnapshotAsync; chunked partial success (HTTP failure + thrown
transport); TokenExpiredError propagation; EndpointPolicyCatalogCompleteness
covers the three new routes by reflection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
206388da99 |
feat(backend,web): tenant offboarding async cascade with customs archive
Replace synchronous DeleteAllTenantDataAsync with a queued async cascade
that wipes all 24 tenant-scoped tables safely, archives custom rules,
and survives crashes at every step.
Phase 1 (TenantOffboardFunction, ~200ms sync):
- Write History / ByTenantPointer / Marker rows (OffboardingAudit table)
- Flip TenantConfiguration.Disabled=true (the active auth gate)
- Enqueue with 6 min visibility-delay (cache-drain barrier =
TenantConfigurationService.CacheDuration 5 min + 1 min safety)
- Return 202 with EarliestProcessingAt for the UI countdown
Phase 2 (TenantOffboardingWorker + Handler, async after drain):
- Enumerate sessions, per-session cascade-enqueue, persist
Expectations blob (separate offboarding-state container)
- Drain probe against per-session progress blobs; self re-enqueue
up to 60x at 2 min visibility-delay
- 2.D-archive: GatherRules / AnalyzeRules / ImeLogPatterns moved
to TenantOffboardingCustomsArchive table (3-iter cap + verify)
- 2.D-wipe: SafeWipe over all tenant tables (4 variants: exact-PK,
composite-PK range, discriminator+TenantId, property-only) with
fetch+verify+delete pattern + SafeWipeVerificationException
- 2.E: blob-prefix wipe deletion-manifests/{tenantId}/
- 2.D-pass-2: re-wipe AuditLogs (defense-in-depth)
- 2.G: side-effects FIRST (Marker, Pointer, blob, audit, OpsEvent)
then History=Completed LAST (DrainCompletedAt as resume anchor)
- 2.F-final: delete TenantConfiguration row LAST so self-service
re-onboarding (next /api/auth/me auto-creates default) works
Defense-in-depth:
- OffboardingMarkerCleanupFunction (2h timer): sweeps Completed
markers after 15 min grace, conditional TenantConfiguration sweep
(only when row is still the offboarding tombstone — preserves
user-driven self-service re-onboards)
- Worker poison path: durable Failed state MUST land before queue
moves; transient failure leaves message visible for retry
- SessionDeletionHandler reorder: Audit + SignalR notify BEFORE
CompletedAt write so the drain predicate is correct
Web UI:
- /settings danger zone: typed "OFFBOARD" confirmation dialog,
post-202 drain-barrier banner with minute-based countdown, hint
"you can sign out and close this tab — deletion runs in
background", silent auto-logout when countdown elapses
- /admin/customs-archive (GA only): list runs + inspect entries +
typed "DELETE" confirmation modal for run/entry deletion
- OpsAlertRulesSection: TenantOffboardingFailed event registered
- lib/api.ts: shared CustomsArchive* response types
Tests: 100+ new tests covering drain contract, handler ordering,
worker poison path, ETag-CAS, SafeWipe variants, OffboardingFilters
prefix-bleed safety, MarkerCleanup conditional sweep, Bootstrap
auth-disabled-tenant gate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
884f29b019 | update offboard function | ||
|
Oliver Kieselbach
|
0a8a6eabd7 |
feat(backend,agent): cert-reject visibility — structured backend warnings + V2 distress cert-context
Backend emits AgentCertRejected (with reason enum) at every cert-validation failure path and AgentRequestRejected (with request-context) at every 401/403 return so operators can diagnose why mTLS auth was denied. V2 agent now attaches a Thumbprint/Subject/Issuer/NotBefore/NotAfter snapshot plus a DistressCertSourceState classifier to each distress report; correlates 1:1 with the backend warnings via tenant + serial + thumbprint. Distress endpoint stays unauthenticated and security-restrictive: rate-limit (3-layer) + sanitize + truncate + enum-validation unchanged; new fields are hex-regex / IsDefined / hard-capped at 96 chars; MaxContentLength bumped from 1024 to 1536 to fit the cert-context payload. V1 agent untouched. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
1041d44831 |
feat(backend,web): restore-browser tenant filter — only tenants with snapshot blobs
Adds a "Only tenants with restore data" checkbox to the Restore Browser tenant dropdown so a Global Admin can hide tenants that have nothing to recover. Default off (full AdminConfig list); flipping it on fetches a cheap hierarchy listing once (cached for page lifetime) and intersects the dropdown with the tenants that own ≥1 snapshot blob. Backend: new GlobalAdminOnly endpoint GET /api/global/tenants-with-deletion-manifests backed by BlobStorageService.ListTenantsWithDeletionManifestsAsync, which uses GetBlobsByHierarchyAsync with delimiter="/" so Azure returns just the top-level prefixes (= tenant IDs) without enumerating blobs underneath. Cost scales with tenant count, not manifest count. Web: checkbox under the tenant dropdown shows a "(X of Y)" counter when active, dropdown placeholder reflects the loading state, and the currently-selected tenant clears if the filter excludes it. Fetch failures snap the checkbox back to off so the operator never gets stranded with an empty list. Build: 0 warnings/errors. Backend 1526/1526 pass. tsc clean. Vitest 346/346 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
c99ca0e461 | update audit entry to more specified perfomed by | ||
|
Oliver Kieselbach
|
2211399f23 |
chore(backend,web): remove legacy session-delete path (PR7)
Cascade-delete V2 is the only path now. Drops the per-tenant EnableCascadeDeleteV2 flag, the legacy ETag-CAS-guarded DeleteSessionAsync helper, the AppInstallSummaries cleanup helper (only legacy caller), and the SessionsLegacyDeleted counter chain (fanout → maintenance → OpsEvent). DeleteSessionFunction collapses to a single kill-switch gate; the producer owns 404 / 409 / Poisoned mapping AND the intentional Queued/Preparing+Snapshot → 202 recovery resume — gating those at HTTP would strand crashed admin cascades. Fanout now aborts both the inner loop AND outer loop on producer KillSwitchActive instead of leaking calls into the next tenant. Test coverage: - DeleteSessionCasGuardTests removed (helper gone) - AdminConfigurationSessionDeletionKillSwitchTests trimmed to kill-switch only - DeleteSessionFunctionTests pinned to the kill-switch short-circuit + null - SessionRetentionFanoutServiceTests: legacy path tests dropped, two new abort regressions for producer KillSwitchActive (inner + outer loop) Build: 0 warnings, 0 errors. Backend: 1526 passed / 0 failed. Web: tsc clean, vitest 346 passed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
ef530fb890 |
feat(agent/v2): DAD liveness + agent_shutting_down gap closure
DAD liveness (3 new state-change-only events, NOT periodic, max 3 per detector lifetime): desktop_detector_started / _first_poll / _no_candidate. Distinguishes "agent dead post-reboot" from "DAD wiring broken" from "user never logged in" in sessions missing desktop_arrived (the ca9e3c59 timeout pattern). Threshold via global AdminConfiguration .DesktopDetectorNoCandidateTimeoutMinutes (default 10, 0 = disabled), round-trip wired through TableConfigRepository, GetAgentConfigFunction (ConfigVersion 27 -> 28), CollectorConfiguration, DefaultComponentFactory, DesktopArrivalHost + admin UI. Also fixes a peripheral-loss bug: V2's DesktopArrivalDetector.OnTraceEvent callback was never wired, so the existing desktop_excluded_user and desktop_real_user_detected events were dead code in V2. Now routed via InformationalEventPost in the host. Shutdown gap closure (4 new agent_shutting_down emit paths): ctrl_c, process_exit, unhandled_exception (with exception type + 500-char- truncated message), runtime_host_exit (defensive finally fallback). Cross-path Interlocked idempotency gate shared with EnrollmentTermination Handler via new tryClaimShutdownEvent ctor param prevents double-emit when Ctrl+C races a real Terminated transition. Event-type unification (V2 only): CreateAuthThresholdHandler now emits agent_shutting_down with data.reason=auth_failure (was agent_shutdown). EnrollmentTerminationHandler reason vocabulary unified to lowercased tags (decision_terminal, max_lifetime). V1 keeps legacy agent_shutdown (phasing out). Web/MCP consumers (eventHelpers, whiteGloveSplit test fixtures, gather-rules eventTypes catalog, MCP semantic-event list, docs) treat both as synonyms. Tests: 3411 green (1538 backend + 1065 V2 core + 530 V1 core + 294 DecisionCore + 347 vitest). TypeScript noEmit clean. 5 new DAD-liveness tests, 5 new gap-path tests, 2 new idempotency tests, 3 reason-vocab updates. Not in this PR: cross-process death detection for the actual ca9e3c59 silent-post-reboot-death pattern (Logon-Trigger / external watchdog) - separate follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
f1583fc586 |
fix(backend,web): move cascade-delete admin routes from /api/admin/* to /api/global/*
The /api/admin/* prefix collides with the Azure Functions runtime's built-in
admin paths, which sit behind mTLS client-certificate auth. Browser preflights
to /api/admin/* trigger TLS renegotiation against the Function App's mTLS
boundary and die before our PolicyEnforcementMiddleware ever runs — the
operator sees CORS-shaped errors that aren't actually CORS-shaped.
Repo convention all along has been /api/global/* for cross-tenant GA-only
HTTP endpoints (e.g. global/session-deletions, global/session-reports). The
cascade-delete endpoints landed under /api/admin/* in the original 7-PR
backend branch and went undetected until the admin UI started calling them
from the browser.
Routes renamed:
admin/sessions/{sessionId}/delete/preview → global/sessions/{sessionId}/delete/preview
admin/sessions/{sessionId}/restore → global/sessions/{sessionId}/restore
admin/sessions/{sessionId}/deletion-manifest → global/sessions/{sessionId}/deletion-manifest
admin/tenants/{tenantId}/deletion-manifests → global/tenants/{tenantId}/deletion-manifests
Touched:
- 4 Function classes: Route attribute + XML-doc header
- EndpointAccessPolicyCatalog: 4 entries (TenantScoping.RouteParam preserved
on the tenants route), with a comment explaining why /api/global/* is the
load-bearing prefix for new GA-only HTTP routes
- DeleteSessionFunction: 3 user-facing "Cascade is poisoned; recover via
POST /api/.../restore" error messages updated
- SessionDeletion.cs: XML-doc reference to the restore route
- lib/api.ts: 4 frontend URL builders (storedManifest, preview, restore,
tenantManifests)
- RestoreSessionFunctionTests: 3 EndpointAccessPolicyCatalog.FindPolicy
lookups
- deleteSessionResponse.test.ts: test fixture message string
No semantic change — only the URL prefix moves. CORS config + JWT middleware
+ policy gating all work the same. The Restore Browser tab + the per-row
Preview / Restore buttons in the Session Cleanup admin page now actually
reach the worker from a browser.
Tests: backend xUnit 1538/1538 pass, web vitest 347/347 pass, dotnet build +
tsc clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
44076686b6 |
feat(backend,web): restore-browser tab + detailed dry-run/real-run dialog
Follow-ups on the Session Cleanup admin page after first hands-on review:
an entry point for restoring already-completed cascades (Sessions row gone,
manifest still in the 33-day TTL window), a proper outcome panel that surfaces
what the backend already returned, and a sidebar fix.
== New Restore Browser tab ==
- Backend: GET /api/admin/tenants/{tenantId}/deletion-manifests enumerates
every persisted snapshot blob under <tenantId>/, parses (sessionId,
manifestId, sizeBytes, lastModifiedUtc) from the blob name, groups by
sessionId and sorts both levels by recency. GA-only + TenantScoping.RouteParam
in EndpointAccessPolicyCatalog (pinned by EndpointPolicyCatalogCompletenessTests).
- BlobStorageService.EnumerateDeletionManifestsByTenantAsync — sibling to
the existing maintenance-sweep enumerator but tenant-prefixed and yielding
size + timestamp so the UI can rank without an extra HEAD per blob.
- Web: new tab "Restore Browser" between Stranded and Activity. Tenant
dropdown (sourced from AdminConfigContext.tenants) → collapsible
session/manifest tree on the left, manifest summary on the right with a
Restore… button that opens the dialog with IDs pre-filled. Useful when
the cascade completed cleanly and the row no longer appears on the
In-Flight / Poisoned / Stranded tabs.
== Reusable summary view ==
DeletionManifestSummaryView extracted from the modal so the same fetch +
field block + worker-progress + preflight-counts + last-failure panel renders
both inside the modal (Preview button on the other tabs) and inline on the
Restore Browser's right-hand panel. Modal becomes a thin wrapper.
== Dry-run / real-run side-by-side ==
RestoreConfirmDialog now keeps both outcomes visible:
- Two slots (dryRunResult, realRunResult), each rendered as a full result
section with predicted/actual rows-by-table, inventory re-increments,
duration, backend message, current state, pending manifest id.
- "Start over" resets both slots so the operator can re-run cleanly.
- Submit buttons gate the workflow: pre-submit shows "Run dry-run" + an
explicit "Skip dry-run · Restore now"; after dry-run the second button
becomes "Apply restore" (clear-intent); after real-run all submit
buttons disappear so accidental double-applies aren't possible.
Previously the dialog discarded everything but Outcome + Mode + dryRun
flag, hiding the actual point of dry-run (the wouldRestoreByTable map).
== Codex review follow-ups (this round) ==
- P2: Restore Browser cleared the displayed sessions only after the new
fetch returned, so a fast A→B switch (or out-of-order responses) could
leave the operator looking at tenant A's manifests with the dropdown
saying B. Fix: clear sessions/expandedSessions/selection synchronously
on tenant change, abort the in-flight request via AbortController,
pair the response with a loadedTenantId state, and gate rendering on
loadedTenantId === selectedTenantId. Three-layer protection so a late
response that sneaks past the abort can't paint stale data.
- P3: PoisonedTab's onRestored callback closed the dialog immediately
after a real restore, defeating the new dry-run vs real-run side-by-side
display. Fix: drop setRestoring(null) from the callback so the dialog
stays open until the operator dismisses it via Close / Start over.
Background refresh of the Poisoned list still happens via setRefreshKey.
Now consistent with the Restore Browser tab.
== Sidebar highlight fix ==
GlobalSidebar marked BOTH ga-ops sub-items active when on the longer
nested route, because each sub used pathname.startsWith(sub.href + "/")
independently. Visible bug: navigating to /admin/ops/session-cleanup lit
up both "Maintenance" (/admin/ops) and "Session Cleanup" in the sidebar.
Fix: compute the longest-matching sub.href once per expanded group, mark
only that one active.
Tests: backend xUnit 1538/1538 pass (one new endpoint registered in the
catalog, picked up by the existing completeness test). Web vitest 347/347
pass. TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit for GetTenantDeletionManifestsFunction (same HttpRequestData
mock-harness gap as the other read endpoints).
- vitest for RestoreBrowserTab + the new dialog flow (defer until a
real field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
09bfe7a1bc |
feat(backend,web): admin session-cleanup UI for V2 cascade-delete + codex follow-ups
Operationalises the 7-PR backend cascade-delete pipeline that landed via commits
c8658b9f..921994e8: per-tenant V2 opt-in switch, audit-trail consolidation, and
a Global-Admin-only Session Cleanup page to track in-flight / poisoned / stranded
cascades and trigger restores.
== PR-A: per-tenant V2 enrollment ==
- UpdateTenantConfigurationFunction extends the GA-only gate list with
EnableCascadeDeleteV2 so tenant admins cannot self-opt-in / out of the
snapshot+restore pathway.
- TenantManagementSection.tsx adds a purple-bordered "Enable V2 Cascade Delete"
checkbox in the GA edit modal and a "Cascade V2" badge on the tenant card so
rollout state is visible at a glance.
== PR-B: audit consolidation ==
- SessionDeletionHandler drops the 15-per-cascade deletion_step_completed /
deletion_step_failed / deletion_verification_failed audit emissions. Step
telemetry lives in DeletionProgress + structured logs; tenant admins now see
just deletion_started + deletion_completed + deletion_restored.
- SessionDeletionWorker no longer writes a deletion_poisoned tenant audit;
the signal moved to a new SessionDeletionPoisoned OpsEvent (operator-bound,
Telegram-routable, dual-registered in OpsAlertRulesSection.tsx catalog).
IMaintenanceRepository dependency removed from the worker constructor.
- SessionRestoreService stops emitting the deletion_manifest_downloaded
breadcrumb; deletion_restored is the operator-visible outcome.
== PR-C: Session Cleanup admin page ==
- New /admin/ops/session-cleanup route (gated by app/admin/layout.tsx's
ProtectedRoute requireGlobalAdmin). Sidebar entry under ga-ops alongside
Maintenance. Four URL-state tabs: In-Flight / Poisoned / Stranded Queued /
Activity (links to existing /audit page).
- Backend reads: GetSessionDeletionsListFunction
(GET /api/global/session-deletions?state=&strandedSinceMinutes=) +
GetSessionDeletionManifestFunction
(GET /api/admin/sessions/{id}/deletion-manifest?tenantId=&manifestId=&mode=).
Both registered as GlobalAdminOnly in EndpointAccessPolicyCatalog.
- DeletionPreviewModal reads the persisted snapshot via the stored-manifest
endpoint (NOT a fresh dry-run via the existing /delete/preview, which would
enumerate live-but-partially-deleted data for in-flight cascades). Renders
manifest counts, snapshot SHA, worker progress (CompletedSteps,
VerificationDone, TombstoneStarted, aggregate decrements), and the last
verification failure with residual sample. Download button uses
authenticatedFetch + Blob + Object URL so the Bearer token is sent.
- RestoreConfirmDialog drops the Full/Partial radio (mode is auto-selected
by SessionRestoreService based on cascade state — forcing the wrong mode
would corrupt state). Reason field threads through to deletion_restored
audit details. Dry-run default = true and previews the auto-selected mode.
== Codex review follow-ups (across four rounds) ==
- High: tenantId is explicitly accepted on /delete/preview (query) and
/restore (body); GA cross-tenant resolution prefers the explicit value over
JWT TargetTenantId, falling back to SessionsIndex lookup. Without this, a
GA acting on a session in another tenant got their JWT home tenant.
- High: new stored-manifest endpoint replaces the dry-run preview for
in-flight / poisoned rows; previous fix built a fresh manifest against
already-partially-deleted data and misrepresented the cascade.
- Medium: SessionDeletionPoisoned OpsEvent now carries failureType +
failureMessage + observedResidualCount + a small residual preview, persisted
by the handler into DeletionProgress (LastFailureType, LastFailureMessage,
LastObservedResidualCount, LastResidualSampleJson) before the throw so the
worker can include them.
- Medium: dry-run restore reports the AUTO-SELECTED mode (full/partial) so
the dialog's preview reflects what the real run will do, instead of the
literal string "dryRun".
- Medium: residual count field renamed to observedResidualCount with an
honest XML doc — CascadeVerificationService caps at 50 per table AND
short-circuits after the first failing table, so the number is a lower
bound. UI surfaces "N of 50+ observed rows" when N hits the cap.
- Medium: three-layer cap defense on the OpsEvent residual sample to stay
under the 4096-char TableOpsEventRepository truncation: max 5 entries
(OpsEventResidualSamplePreviewSize), per-field max 96 chars with `…` marker
(OpsEventResidualKeyMaxChars), total preview budget 1200 chars
(OpsEventResidualPreviewBudgetChars) with trailing-entry drop. Full sample
stays in the progress blob for the admin UI.
- Low: estimatedSnapshotBytes field name aligned end-to-end; activity tab
text clarifies audit-action vs OpsEvent distinction.
Tests:
- Backend xUnit 1538/1538 pass (up from 1510 baseline; +28 across the new
Session Cleanup endpoints, OpsEvent enrichment, restore-mode reporting,
cap-defense unit + integration tests).
- Web vitest 347/347 pass.
- TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit tests for GetSessionDeletionsListFunction +
GetSessionDeletionManifestFunction (no HttpRequestData mock harness in repo;
would be a separate refactor).
- vitest for DeletionPreviewModal + RestoreConfirmDialog (defer until first
field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
c237140d59 |
fix(backend): cascade-delete codex follow-ups #2 (F1 DELETE cross-tenant, F2 register guard, F3 tombstone marker, F4 cross-tenant test, bootstrap-bypass)
F1 — DELETE /api/sessions/{sessionId} bypassed cross-tenant validation
because the catalog entry had no TenantScoping. A Global Admin clicking
"delete" on a session in tenant B from the portal silently routed the
delete into their own JWT tenant A, never finding the row. Catalog
entry gains TenantScoping.QueryParam; DeleteSessionFunction switches
from TenantHelper.GetTenantId (JWT-only) to RequestContext.TargetTenantId
(middleware-validated).
F2 — RegisterSessionFunction wrote sessions without any deletion-state
check; a mid-cascade register could mutate a Preparing/Queued/Running
row, and a fresh register past the tombstone could re-Add a row the
manifest no longer covers. Guard now runs at the entry of the shared
ProcessRegisterAsync core — covering BOTH /agent/register-session
(cert-auth) AND BootstrapRegisterSessionFunction (bootstrap-code-auth),
which previously bypassed the cert-auth entry's check. 410 Gone on
lock-state or tombstone hit.
F3 — SessionDeletionGuard's "Sessions row missing -> silent pass"
contract let late ingests / registers slip past a completed cascade
and orphan rows the manifest couldn't describe. New short-lived
SessionTombstones table (PK=tenantId, RK=sessionId) carries a marker
written by SessionDeletionHandler.ExecuteTombstoneAsync BEFORE the
Sessions-row delete, with a 7-day default ExpiresAt. Guard now reads
the marker on a missing Sessions row; live and present -> throws with
sentinel state "Tombstoned" -> 410 Gone. Full-Restore deletes the
marker after re-inserting the Sessions row; partial-restore leaves it
alone (never written). SessionDeletionMaintenanceFunction pruning
step runs even when the kill-switch is active (markers are race
shields, not policy artefacts).
F4 — CrossTenantAccessTests had no coverage for DELETE
sessions/{sessionId}. Theory entries lifted into the same three asserts
that already cover GET — same-tenant allowed, cross-tenant non-GA
blocked, cross-tenant GA allowed — and into the QueryParam-scoping
catalog theory so the regression that produced F1 surfaces in CI.
Bootstrap follow-up — RegisterSessionFunction.Run had the guard inline
but BootstrapRegisterSessionFunction calls _inner.ProcessRegisterAsync
directly, bypassing it. Guard moved into ProcessRegisterAsync so both
entry points hit it before StoreSessionAsync. Added structural wiring
tests (DI field, internal accessibility, bootstrap delegation type)
that fail in CI if the architecture regresses to inline copy.
Tests:
- SessionDeletionGuardTests +3: marker-active blocks, no-marker passes
on missing row, sessions-row-exists skips tombstone probe.
- SessionDeletionHandlerTests +1: marker is written BEFORE the
Sessions-row delete (Moq Callback ordering assertion).
- SessionRestoreServiceTests +2: full-restore deletes marker,
partial-restore does not.
- SessionDeletionMaintenanceFunctionTests +2: prunes expired markers,
pruning runs even with kill-switch active.
- CrossTenantAccessTests +3 inline DELETE cases.
- RegisterSessionGuardWiringTests (NEW): asserts guard DI field,
ProcessRegisterAsync internal accessibility, BootstrapRegisterSession
delegation type.
- CascadeVerificationServiceTests.FakeReader gains the new
GetActiveSessionTombstoneAsync interface member.
Suites: Backend xUnit 1524, Agent V1 530, Agent V2 1042 — all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
921994e845 |
fix(backend): cascade-delete codex follow-ups (F1 V2 guard + upsert-revival, F2 partial-restore key-level re-increment, F3 legacy delete sessions-first, F4 restore 400 EntityAlreadyExists)
F1 — V2 ingest writes were landing past the cascade lock. Wired SessionDeletionGuard.EnsureWritableAsync into IngestTelemetryFunction before PersistItemsAsync so Event / Signal / Transition writes are refused with 410 Gone when DeletionState is Preparing / Queued / Running / Poisoned. Switched UpdateSessionImeAgentVersionAsync from UpsertEntityAsync to UpdateEntityAsync(ETag.All, Merge) with 404 swallow so a tombstoned Sessions row can no longer be silently resurrected by an in-flight ingest. F2 — Partial-restore could leave SoftwareInventory at -N drift when the cascade poisoned mid-AGGREGATE-loop: per-key decrement was persisted into AggregateDecrementsApplied BEFORE the underlying decrement, but the gate used CompletedSteps.Contains(aggregate.Order) — which is only set AFTER the whole loop. Switched the gate to AggregateDecrementsApplied as the authoritative ledger and pass a filtered DeletionStep into RunInventoryReIncrementWithIdempotencyAsync so only the keys that were actually decremented are re-incremented. F3 — Legacy direct-delete path deleted Events / RuleResults / AppInstallSummaries before the ETag-CAS-guarded DeleteSessionAsync, leaving a TOCTOU window where a V2 producer could CAS-claim DeletionState=Preparing after the pre-read but before our side- table mutations — landing the producer's manifest snapshot on top of already-half-purged state. Reordered both admin (DeleteSessionFunction) and retention (SessionRetentionFanoutService) paths to Sessions-CAS first; side-tables only after the Sessions row is gone (no producer can claim a missing row). F4 — Restore per-row Add fallback only caught HTTP 409 for duplicate- row conflicts; Azurite and some legacy Azure deployments surface duplicate-row as 400 with ErrorCode=EntityAlreadyExists, breaking idempotent restore-retry after partial-failed restore. Centralised the check via IsAlreadyExistsStatus(rfe) helper accepting 409 OR 400 + ErrorCode=EntityAlreadyExists; non-duplicate 400s still propagate. Tests: - SessionRestoreServiceTests: 2 partial-restore tests refit to the new AggregateDecrementsApplied contract + new explicit Codex-scenario test (only-decremented-keys re-incremented, untouched key skipped). - SessionDeletionPr4cCodexFixesTests F4: harness setup adds full AggregateDecrementsApplied so the per-key idempotency assertion reflects the new gate. - SessionRetentionFanoutServiceTests: pre-read tests replaced by two tests covering "DeleteSessionAsync refuses → no side tables" and "DeleteSessionAsync succeeds → side tables run after". - TableStorageRestoreBatchTests: two new F4 tests — 400 EntityAlreadyExists treated as Skipped, 400 with non-duplicate ErrorCode still propagates. - UpdateSessionImeAgentVersionTests (NEW): asserts UpdateEntityAsync is used (not Upsert), 404 swallowed silently, other storage errors non-fatal. Suites: Backend xUnit 1510, Agent V1 530, Agent V2 1042 — all green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
4c676d076a |
fix(backend,web): PR6 codex-review follow-ups (F1 legacy pre-check, F2 mid-run kill-switch, F3 OpsEvent audit shift)
Three Round-1 review findings from PR6:
F1 (High) — Legacy retention path mutated side-tables before the
CAS-guarded DeleteSessionAsync. Plan PR6 step 7 says "legacy must skip
locked sessions", but our LegacyDeleteAsync deleted Events / RuleResults
/ AppInstallSummaries first and only discovered the V2 cascade lock at
the tombstone. Fix: pre-read the Sessions row in LegacyDeleteAsync; if
DeletionState != None, return false WITHOUT touching any side table.
The original CAS-guard stays as a backstop for the T1-T3 race after the
pre-read.
F2 (High) — Kill-switch checked once per run; a flip-ON mid-fanout kept
legacy deletes running. SessionRetentionFanoutService now takes
AdminConfigurationService, calls IsSessionDeletionKillSwitchActiveAsync
(uncached, fail-closed per PR5 F1) at both the per-tenant boundary and
inside the per-session loop. FanoutResult gains AbortedByKillSwitch so
the per-tenant audit + the global OpsEvent surface the early halt.
F3 (Medium) — Global-scope lifecycle audits used LogAuditEntryAsync with
tenantId: null!, which the table-storage helper silently dropped because
the schema requires a non-null PartitionKey. Three event types
(Completed / FanoutSkipped / Failed) moved to OpsEvents (TenantId is
nullable on OpsEventEntry). Two new helpers in OpsEventService —
RecordSessionDeletionMaintenance{Completed,FanoutSkipped}Async — also
dual-registered in the OpsAlertRulesSection.tsx catalog per memory
feedback_ops_event_types_dual_register. Per-tenant audits inside the
fanout (which already had a real tenantId) are unchanged.
Tests: 4 new xUnit cases — F1 skip-without-mutation, F1 row-not-found
skip; F2 mid-tenant abort, F2 between-tenants abort. Existing 17 PR6
tests updated to assert OpsEvents instead of audit calls for the global
lifecycle events. Total: 1504/1504 backend pass, 347/347 web pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
50071c5e42 |
feat(backend,web): dedicated cascade-delete maintenance function (PR6/7)
Plan §5 PR6 / §16 R14: extracts cascade-lifecycle maintenance out of the
2h Maintenance timer into a dedicated 12h SessionDeletionMaintenanceFunction
with independent cadence, kill-switch, and OpsEvent watchdogs. Adds four
work-blocks (Plan §10):
1. Manifest-blob TTL sweep — defence-in-depth on top of Azure Blob
Lifecycle Management. EnumerateOldDeletionManifestsAsync +
DeleteDeletionManifestPairAsync (33-day effective window via the
existing soft-delete tombstone, not permanent deletes).
2. Stale-Preparing GC — reverts DeletionState=Preparing rows older than
1h that have NO progress blob (producer crashed before upload). Skips
when a progress blob exists; producer's own Preparing-resume path
handles those.
3. Stranded-Queued detection — alert-only OpsEvent per row in
DeletionState=Queued for > 30min (no auto-clear; operator inspects).
4. Retention fanout via new SessionRetentionFanoutService — per-tenant
DataRetentionDays sweep, dispatched per EnableCascadeDeleteV2 flag:
V2 → producer.EnqueueAsync, legacy → ISessionRepository.DeleteSessionAsync
(now CAS-guarded since PR5 finding 2). Rate-limited to 100 enqueues
per tenant per run with 50ms throttle.
Operational alerting:
- 30min watchdog → OpsEventType SessionDeletionMaintenanceLongRunning (Warning)
- 60min watchdog → SessionDeletionMaintenanceLongRunningSevere (Error)
- Exception path → SessionDeletionMaintenanceFailed (Error) + re-throw
- Per stranded row → SessionDeletionStrandedQueued (Warning)
All four registered both in OpsEventService.cs and the OpsAlertRulesSection.tsx
catalog so admin can wire Telegram per memory feedback_ops_event_types_dual_register.
Reuses PR5 finding 1's IsSessionDeletionKillSwitchActiveAsync so the
fanout-skip check honors flip-ON across scaled-out instances within seconds.
host.json: functionTimeout=01:00:00 (Flex Consumption max, matches the
60min severe watchdog so operator gets alerted before Azure hard-aborts).
Cron: const "0 0 */12 * * *" — matches the repo's IndexReconcileTimer
convention (no AdminConfiguration field; redeploy to change).
Removed: session retention loop from MaintenanceService.CleanupOldDataAsync
(both RunAllAsync and RunManualAsync paths now call CleanupOldUsageDataAsync
directly). Non-session housekeeping (rule-stats, cert-expiry, SignalR
quota, poison-queue watcher) untouched.
EnableCascadeDeleteV2 + SessionDeletionKillSwitch defaults stay false —
PR6 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation.
Backend xUnit: 1500/1500 pass (1483 + 17 new). Web vitest: 347/347 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
8134adf3e4 |
feat(backend,web): wire admin-delete + worker + UI for V2 cascade (PR5/7)
Plan §5 PR5 (session-deletion-cascade-safe.md): flag-gated dispatch in
DeleteSessionFunction between the legacy direct-delete path and the V2
cascade producer; SessionDeletionWorker registered as a hosted service;
UI shows a "deletion queued" toast + per-row spinner while the worker
drains, then removes the row on the SignalR sessionDeleted event.
Includes the codex review follow-ups:
F1 — IsSessionDeletionKillSwitchActiveAsync bypasses the 5-min admin
config cache; fail-CLOSED on storage error so a flip-ON is honored
across scaled-out Function-host instances within seconds (instead
of minutes). Wired into DeleteSessionFunction, Producer, Worker.
F2 — DeleteSessionAsync now reads DeletionState + ETag together with
IndexRowKey, refuses on locked state, and ETag-conditionals the
delete; closes the T2-T4 race where a concurrent V2 producer
could orphan side-table rows. Caller in the legacy path
disambiguates the bool via a re-read into 404 / 409 / 500.
F3 — pendingDeletions threaded from useDeleteSession through page.tsx
→ SessionTable → SessionCell; action cell renders a disabled
spinner instead of the delete button. Hook subscribes to
sessionDeleted SignalR + polls GET /api/sessions/{id} every 60s
as a fallback for missed events; only 404 clears the pending row.
EnableCascadeDeleteV2 + SessionDeletionKillSwitch both default false —
PR5 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation. PR6 wires the producer into the dedicated
SessionDeletionMaintenanceFunction next.
Backend xUnit: 1483/1483 pass. Web vitest: 347/347 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
5dbc44c4ec |
feat(backend): cascade-delete worker + restore + codex-fixes (PR4/7)
PR4 ships the cascade-delete executor for the safe-cascade-delete plan: the
SessionDeletionWorker (BackgroundService queue poll-loop) plus the
SessionDeletionHandler (per-step execution, live verification, tombstone).
Worker is NOT registered in Program.cs — that lights up in PR5.
Three logical phases land together as one commit per Oliver's call:
PR4 — Worker + Handler + CascadeVerificationService
- SessionDeletionWorker: BatchSize=1, kill-switch entry guard, heartbeat task
extending visibility every 60s during long handler runs, poison after
max-dequeue
- SessionDeletionHandler: download manifest+progress, CAS Queued→Running
(with Running→Running resume), per-step dispatch by Class, AGGREGATE
decrement, live verification, ordered tombstone (SessionsIndex first,
Sessions last)
- CascadeVerificationService: per-Class live residual scans (CveIndex is
manifest-key-only per §12-Q8; AGGREGATE+FINAL skipped)
- NotifySessionDeletedAsync added to ISignalRNotificationService
- BlobStorageService: DownloadDeletionProgressAsync + ETag-CAS
UpdateDeletionProgressAsync
PR4b — Restore endpoint
- POST /api/admin/sessions/{sessionId}/restore (GA-only via Policy Catalog)
- SessionRestoreService: dispatch table per §13.4, full + partial-poisoned-
recovery modes
- TableStorageService.Restore.cs: RestoreRowsByExactKeysInBatchesAsync +
ConvertFromPropValue + ConvertDumpToEntity (Sessions-row override forces
DeletionState=None on restore so the row isn't permanently locked)
- RestoreSoftwareInventoryEntryByKeyAsync helper
- Worker MoveToPoisonAsync now CAS-transitions Running→Poisoned (E1 fix)
so restore can dispatch into partial-restore mode
PR4c — Codex review fixes (6 findings)
- F1: AGGREGATE per-key idempotency via DeletionProgress.AggregateDecrementsApplied
(persist-first/decrement-second bounds drift to +1 per crash vs unbounded -N)
- F2: TombstoneStarted flag closes the gap where sessions=null+completedAt=null
blocked restore after a tombstone crash
- F3a: Sessions row inserted LAST in full restore (was first → trapped retry
with RejectAlreadyAtOriginalState)
- F3b: RestoreMode.Full now 409-ignores (was throw-on-409); converges with
Partial mode, makes retry-after-partial-failure idempotent
- F4: Partial-restore per-key re-increment idempotency via
DeletionProgress.RestoreReIncrementsApplied
- F5: Poison transition pre-checks PendingDeletionManifestId so stale
envelopes can't flip a fresh active cascade to Poisoned
- F6: DownloadDeletionManifestWithShaAsync exposes snapshot SHA; Handler +
Restore both verify against progress.SnapshotSha256 (snapshot↔progress
binding per plan §3)
DeletionProgress schema gains 3 additive optional fields (AggregateDecrementsApplied,
RestoreReIncrementsApplied, TombstoneStarted) with safe defaults so PR1-PR4
progress blobs deserialize cleanly.
Tests: 1457/1457 backend xUnit facts green (1391 pre-PR4 + 66 new across PR4,
PR4b, PR4c). Full solution build green. Program.cs has zero
SessionDeletionWorker references (PR5 territory stays untouched).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
522f0ae048 |
fix(backend): PR3 codex round-2 follow-ups (F2 ETag-CAS, F4 Preparing-resume)
Round 2 of codex feedback flagged that the first follow-up wasn't tight
enough:
== F2 round 2 (HIGH): TOCTOU race still possible on Sessions row ==
The first fix only preserved DeletionState if it was non-empty on the
INITIAL read. The race remained: T1 StoreSessionAsync reads row →
DeletionState=None; T2 producer CAS-es None → Preparing; T3
StoreSessionAsync writes Replace → silently clobbers the lock. Codex
specifically asked for ETag/Retry oder Merge.
Fix: replace the unconditional UpsertEntityAsync with an ETag-bound CAS
write loop (bounded retry = 5 attempts). Each iteration re-reads the
CURRENT row to capture (a) the fresh ETag and (b) the live cascade-lock
columns, stamps the cascade-lock columns from that FRESH read onto our
entity (so the producer's intervening write isn't clobbered), then tries
UpdateEntity(etag, Replace). On 412/409 conflict: retry. Falls back to
AddEntity when the row was deleted between reads (tombstone race). The
preserved-fields extract from the initial read still runs as a
best-effort hint; the per-iteration fresh read is what actually wins
the race.
Replace mode is kept (not Merge) so registration can still clear stale
fields like FailureReason on a successful re-register. Only the
cascade-lock columns get the cross-iteration preservation treatment.
== F4 round 2 (MEDIUM): stranded Preparing with progress blob ==
If the prior producer call CAS'd None → Preparing AND uploaded snapshot
+ progress blobs AND audited deletion_started, then crashed before the
CAS Preparing → Queued step — the row stays in Preparing forever. Plan
§10 GC only clears Preparing-WITHOUT-progress-blob (intentional, to
avoid racing in-flight producers); the resume path I added before only
covered the Queued case. So this would sit until manual intervention.
Fix: in the WrongState=Preparing branch with non-empty
PendingDeletionManifestId, probe the snapshot blob via the new
BlobStorageService.DeletionSnapshotExistsAsync helper. If it exists,
the prior crash happened POST-upload — safe to finish the CAS
Preparing → Queued + send the queue message (resume from step 6). If
absent, the prior crash was PRE-upload — bail out as AlreadyInFlight
and let the §10 GC clean up after 1h.
Probe failure (e.g. transient blob storage outage) also falls back to
AlreadyInFlight rather than risk a half-completed recovery. Extracted
the queue-message-resume call into a shared helper
(ResumeBySendingQueueMessageAsync) so both Queued-resume and
Preparing-resume go through one audit-marker code path
(Reason="resume").
== Tests ==
- SessionDeletionPr3CodexFixesRound2Tests (4 new):
- Stranded Preparing-with-blob → resumes (CAS Preparing→Queued runs,
queue message sent, returns Enqueued + Reason=resume)
- Preparing without blob → AlreadyInFlight, no CAS attempt
- Snapshot existence probe throws → AlreadyInFlight, no CAS attempt
- Resume CAS loses to concurrent writer → AlreadyInFlight, no queue
message
- Backend suite: 1349/1349 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
dc56c8a73a |
fix(backend): PR3 codex-review follow-ups (F2 lock-clobber, F3 SchemaHash, F4 resume, F5 preview shape)
Round-2 codex review on PR3 surfaced four real issues. Finding 1 (the
guard not being wired at the 9 writer sites) was the deferred-by-design
decision and stays scheduled for PR5.
== F2 (HIGH): StoreSessionAsync silently clobbered the cascade lock ==
Re-registration reads the existing Sessions row but built a fresh
TableEntity for the UpsertEntity (Replace) without carrying over
DeletionState + PendingDeletionManifestId. An agent restart during
Preparing/Queued/Running would have cleared the lock and let the
agent's subsequent writes race the in-flight cascade. Fix: extract
both columns alongside the other preserved fields and re-stamp on
the new entity before the Upsert. Comment marks the silent-clobber
risk so a future refactor can't quietly remove the preservation.
== F3 (HIGH): Producer invalidated the manifest SchemaHash ==
Builder generated a ManifestId, hashed the manifest content, then the
producer overwrote manifest.ManifestId. SchemaHash was now stale
(computed against the old ID); a future hash-verification path on the
worker would treat legitimate manifests as tampered/corrupt. Fix:
DeletionManifestBuilder.BuildAsync accepts an optional
preAllocatedManifestId so the producer's CAS-stamped ID lands on the
manifest BEFORE the hash computation. Producer no longer overwrites.
Preview path passes null (transient ID).
== F4 (MEDIUM): No resume path for stranded Queued state ==
If CAS Preparing→Queued succeeded but SendMessageAsync failed
(network blip, queue throttling), the Sessions row sat in Queued state
with no live queue message. A retry just returned AlreadyInFlight and
left the cascade stranded. Fix: producer's WrongState=Queued branch
re-sends the queue envelope with the EXISTING manifestId (stamped
with reason ":resume" for audit traceability), returns Enqueued.
Idempotent — duplicate envelopes are tolerated by PR4's worker via
the manifest-bound progress blob. Preparing + Running unchanged
(let GC handle Preparing; trust the worker for Running).
== F5 (MEDIUM): mode=full preview wrapped manifest in an envelope ==
Plan §5 PR3 promises mode=full returns "the entire manifest JSON
snapshot inline ... identical structure to what would be uploaded
to deletion-manifests blob container" — but the response wrapped the
manifest in { success, mode, manifest } so byte-comparison against
a real cascade's snapshot was impossible. Fix: mode=full now returns
the RAW manifest JSON as the body. Side-channel data (in-flight hint,
builder duration, schema hash, manifest id) moves to response headers
(X-Deletion-Preview-*). Only mode=summary still uses the envelope
shape (it's a summary, not a snapshot).
== Tests ==
- SessionDeletionPr3CodexFixesTests (5 new): pin F3 hash consistency
with + without preAllocatedManifestId, F4 resume re-sends queue
message with existing ManifestId, F4 Running + Preparing do NOT
re-send.
- Updated 1 existing producer test: AlreadyInFlight assertion now
uses Running state (Queued is the resume path post-F4).
- Backend suite: 1345/1345 green.
== Finding 1 (guard wiring) ==
Confirmed deferred to PR5 per the original PR3 scope decision. PR5
wires the producer call into DeleteSessionFunction AND the 9 guard
sites in the same PR. Cascade can't fire until then, so the guard
being dormant in PR3 has zero runtime impact.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
257c5c4354 |
feat(backend,web): cascade-delete state-machine + guard + producer + preview (PR3/7)
PR3 of the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). Substantial PR per
plan §16 — adds the concurrency state-machine, the single-chokepoint
SessionDeletionGuard, the producer that builds manifest + uploads
blobs + enqueues the worker envelope, and the read-only preview
endpoint Global Admins use to verify a cascade plan before flipping
EnableCascadeDeleteV2 on. Wiring the guard at the 9 writer sites is
intentionally deferred to PR5 — no cascade actually fires until PR5
calls the producer from DeleteSessionFunction, so PR3 ships as
infrastructure-only.
== State machine + guard infrastructure ==
SessionDeletionState (Shared, Plan §1 P7 / §16-R2):
None | Preparing | Queued | Running | Poisoned. Completed deliberately
omitted — the cascade FINAL step removes the row, so "completed" is
observable via row-absence + progress.completedAt, not via a stable
state value. IsValidTransition(from, to) enforces the legal edges
(None→Preparing, Preparing→{Queued,None,Poisoned}, Queued→{Running,
Poisoned}, Running→Poisoned, Poisoned→None) and treats null/empty
legacy rows as None.
SessionDeletionLockedException carries (TenantId, SessionId,
CallerContext, CurrentState, ManifestId) for audit + log lines.
SessionDeletionGuard (Plan §1 P7 + §16-R13):
- ThrowIfLocked(TableEntity sessionRow, callerContext) — piggyback
variant for hot paths that already loaded the Sessions row; zero
added I/O. Caller must include DeletionState +
PendingDeletionManifestId in its select projection.
- EnsureWritableAsync(tenantId, sessionId, callerContext) —
standalone variant for paths with no existing read; one Get
round-trip via the PR1 ISessionDeletionInventoryReader seam.
Both paths throw the same exception. Per §16-R13 framing the guard
is "best-effort fast-fail, not safety" — correctness lives in §1 P4
live verification + poison + §13 restore-from-poisoned (PR4).
== Sessions row state-machine columns ==
SessionSummary.DeletionState (string) +
SessionSummary.PendingDeletionManifestId (string?) — added to the DTO
+ MapToSessionSummary read path. Empty/null means no cascade in flight
(legacy rows; treated as None per the guard contract). The producer
writes both columns via direct CAS via the new helper; existing
session-write paths don't need to know about them in PR3.
TableStorageService.CasSetSessionDeletionStateAsync — new partial
helper. Read-then-CAS on Sessions row, returns a structured
SessionDeletionStateCasResult (Updated | SessionMissing | WrongState |
EtagConflict) so the producer can branch correctly. Stamps newManifestId
on transition to Preparing; clears it on transition back to None
(poisoned-recovery restore).
== Producer + preview endpoint ==
SessionDeletionProducer.EnqueueAsync(tenantId, sessionId, reason,
actor) — Plan §5 PR3 step sequence: kill-switch check → CAS None→
Preparing with new ManifestId → DeletionManifestBuilder.BuildAsync →
BlobStorageService.UploadDeletionManifestAsync → UploadInitial-
DeletionProgressAsync (new helper for the mutable .progress.json
sibling) → audit deletion_started → CAS Preparing→Queued → enqueue
SessionDeletionEnvelope to Constants.QueueNames.SessionDeletion.
Returns SessionDeletionEnqueueOutcome enum so PR5/PR6 callers
translate per their UX (admin → 409 on AlreadyInFlight, maintenance
→ skip+audit). Crash recovery contract: row stuck in Preparing with
no progress blob → maintenance GC clears after 1h (PR6); Queued/
Running/Poisoned NEVER auto-cleared.
GET /api/admin/sessions/{sessionId}/delete/preview?mode={summary|full|
download} — GlobalAdminOnly per the EndpointAccessPolicyCatalog (per
memory feedback_route_policy_catalog). Three modes:
- summary: preflightCounts + sample (PK, RK) per step + estimated
snapshot bytes + builderDurationMs + schemaHash.
- full: entire manifest JSON inline (every row dump, every decrement
key) — byte-level review of "what would be deleted".
- download: gzipped *.snapshot.json.gz with Content-Disposition
attachment, identical bytes to what the real cascade would upload.
NEVER mutates state, NEVER uploads a blob, NEVER enqueues. Calls the
guard only as a hint (in-flight cascades surface in the response, no
lock acquired). Audits deletion_preview per Plan §7.
== Configuration changes (with web round-trip) ==
AdminConfiguration.SessionDeletionKillSwitch (bool, default false,
Plan §1 P8 / §9) — global emergency stop. When true, V2 producer +
legacy direct-delete return 503; PR4 worker pauses on entry. Round-
tripped through TableConfigRepository Store + Map AND the 4-file web
chain (AdminConfigContext.tsx, AdminConfigSettingsSection.tsx,
SectionGlobalSettings.tsx, types/adminConfig.ts) per memory
feedback_admin_config_ui_roundtrip — without it, every admin Save
silently clobbers the field back to false. UI surfaces a red checkbox
with explanatory copy.
TenantConfiguration.EnableCascadeDeleteV2 (bool, default false) —
per-tenant rollout flag. When true, PR5 admin-delete + PR6 retention
fanout build a manifest + enqueue; when false, legacy direct-delete
runs. Per-tenant flag selects PATH; global kill-switch halts BOTH.
Constants.QueueNames.SessionDeletion = "session-deletion" — worker
queue name.
Shared envelope SessionDeletionEnvelope { TenantId, SessionId,
ManifestId, Reason, EnqueuedAt } — wire shape between producer and
PR4 worker.
== DI ==
Program.cs registers SessionDeletionGuard + SessionDeletionProducer
as singletons next to the existing DeletionManifestBuilder.
== Tests (50 new, backend suite 1340/1340 green) ==
- SessionDeletionStateTests (10) — IsLocked over every state value
including legacy null/empty, IsValidTransition over every legal
edge, explicit rejection of illegal edges, "Completed is not a
state" guarantee.
- SessionDeletionGuardTests (8) — null row no-op, every lock state
throws with correct exception fields, missing-session no-op,
context propagation in the exception.
- SessionDeletionProducerTests (7) — kill-switch returns
KillSwitchActive without downstream calls; SessionMissing /
AlreadyInFlight / Poisoned outcomes from CAS; happy path verifies
all 7 steps in order via Mock.Verify; CAS-2 failure leaves snapshot
uploaded but no queue message; producer's pre-allocated ManifestId
flows to both the snapshot blob and the CAS marker (consistency
invariant the worker depends on).
- AdminConfigurationSessionDeletionKillSwitchTests (6) — both flags
default false on new config + on CreateDefault, true persists.
Wiring 9 sites with the guard (telemetry V1+V2 ingest, mark-success/
failed, server actions, maintenance marker, re-analyze handler,
vulnerability handler, manual rescan, index reconcile) deferred to
PR5 — no cascade fires in PR3, so the guard is dormant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
66fc23b609 |
feat(backend): SoftwareInventory contributions side-row + cascade delete helpers (PR2/7)
PR2 of the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). Adds the per-table
delete primitives the cascade worker will use in PR4 plus the §17
SessionInventoryContributions side-row that makes per-session inventory
counter increments at-most-once and exactly reversible. No production
caller invokes the cascade primitives yet; the inventory rewrite IS a
call-site change in VulnerabilityCorrelationService and ships in this PR.
== Cascade delete helpers (TableStorageService.Deletion.cs) ==
DeleteByExactKeysInBatchesAsync(tableName, keys) is the load-bearing
helper: groups by PartitionKey (Azure batch transactions are
partition-scoped), chunks at 100 actions per batch, falls back to
per-row DeleteEntityAsync with 404-ignore on whole-batch rollback.
Returns (Attempted, DeletedNow, AlreadyMissing) for accurate progress
accounting on idempotent re-runs of partial cascades. Plus 9 thin
wrappers per the §3 taxonomy: 7 keyed (EventTypeIndex, CveIndex,
SessionsByTerminal, SessionsByStage, DeadEndsByReason,
ClassifierVerdictsByIdLevel, SignalsByKind), 1 single-row
(DeviceSnapshot), and 2 enumerate-then-delete (Signals,
DecisionTransitions for the PK_BY_SESSION class). All fail-loud per
memory feedback_storage_helpers_fail_soft.
== SoftwareInventory side-row (TableStorageService.Inventory.cs) ==
Plan §17 design: each inventory-correlated session contributes ONE
SessionInventoryContributions row (PK=tenantId, RK=sessionId) with the
normalized software keys it added. Increments to SoftwareInventory
counters are now driven by deltas against this row, so:
- WG sessions that fire software_inventory_analysis 4× no longer
over-count by 4× per software per session
- Cascade decrement is exact: the side-row is the source of truth
for which counters this session contributed to
- Pre-side-row sessions (created before this deploy) have no
contributions row → cascade skips inventory entirely for them
(drift accepted on the historical backlog, ages out via 90d
retention).
Helpers (all fail-loud, all bounded retry per §12-Q10 = 10 attempts /
60s wall-clock cap, all ETag-CAS):
- GetSessionInventoryContributionsAsync: returns prior key list or
null on 404
- UpsertSessionInventoryContributionsAsync: side-row CAS that
computes (added, removed) delta against the new key set
- DeleteSessionInventoryContributionsAsync: 404-ignore tombstone
- IncrementSoftwareInventoryEntryAsync: ETag-CAS +1 with first-time
AddEntity that writes full metadata
- DecrementSoftwareInventoryEntryAsync: ETag-CAS -1, clamp >= 0,
deletes the row when count hits zero
Codec: SoftwareKeysJsonCodec encodes the key list as raw JSON below
30 KB, gzip+Base64 above. Same shape DeletionManifestBuilder (PR1)
already reads at preflight, so the cascade decrement reads back what
the increment wrote with no schema gap. Plus deterministic ComputeDelta
(case-insensitive composite key) so concurrent correlations on the
same session collapse to a single counter update.
== VulnerabilityCorrelationService rewrite ==
The Task.Run fire-and-forget at the old line 246 now calls the new
ApplyInventoryContributionsAsync helper instead of
UpsertSoftwareInventoryAsync. Each Increment / Decrement is wrapped
in its OWN try/catch so one counter failure does NOT abort the rest
of the loop (Plan §17.3 + §17.8): without that, the side-row would
already promise the new key set while subsequent runs see an empty
delta and never retry the under-counted keys. Each individual failure
emits an `inventory_counter_write_failed` audit entry with operation
+ vendor + name + version + exception type/message via the existing
LogAuditEntryAsync. IMaintenanceRepository was added as an OPTIONAL
constructor param (positional default null) so existing 8-arg test
factories keep compiling unchanged.
ParseSoftwareInventoryItems exposed as internal static for testability;
preserves the same dedup + cpe-mapping behaviour as the old
UpsertSoftwareInventoryAsync path. The legacy method is left in place
on TableStorageService.Rules.cs for now (no remaining callers; PR7
cleanup can remove if desired).
== Tests (39 new) ==
- SoftwareKeysJsonCodecTests (12): raw vs gzip+Base64 paths,
determinism, malformed-input rejection, delta cases (added /
removed / identity / case-insensitive), composite-key shape.
- TableStorageDeletionBatchTests (8, SDK-mocked via internal test
ctor): empty input, PK grouping (one batch per PK), chunking at
100, batch 404 + all per-row 404, mixed batch 404 (2 succeed +
3 missing) with accurate accounting, non-404 propagates, no
over-deletion, ETag.All on every action.
- TableStorageInventoryCasTests (9): Increment first-create with
full metadata, CAS +1 on existing row, 412 retry, 409 first-create
race fall back to Update, empty-triple skip; Decrement 404 no-op,
clamp-at-zero deletes the row, count>1 subtract, 0-count clamp,
412 retry.
- VulnerabilityCorrelationServiceContributionsTests (10): Parser
correctness, end-to-end add/remove flow with Mock.Verify counts,
empty-input no-op, side-row exhaustion swallowed, per-counter
failure does not abort + audits the failed key (both increment
and decrement variants).
Backend suite: 1290/1290 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
c8658b9feb |
feat(backend): scaffold cascade-deletion manifest format + builder (PR1/7)
Read-only foundation for the safe cascade-delete plan (.claude/plans/session-deletion-cascade-safe.md). PR1 ships the manifest shape, the read-only DeletionManifestBuilder enumerator, and the gzip+SHA-256 blob upload/download plumbing. No production caller invokes any of this yet; no existing endpoint behaviour changes. PR2-PR7 wire the producer, worker, and call sites on top. Manifest captures all 15 cascade tables per the §3 taxonomy (4 PK_BY_SESSION, 1 PROP_TENANT_PK, 3 PK_RK_EXACT, 1 DISCRIMINATOR_PK_RK_SUFFIX, 1 DISCRIMINATOR_PK_RK_EXACT, 5 DISCRIMINATOR_PK_PROP), the optional SoftwareInventory side-row (omitted for pre-side-row sessions per §17.5), and the FINAL tombstone step ordered SessionsIndex-then-Sessions. Row dumps tag every column with its EDM type (String / Int32 / Int64 / Double / Boolean / DateTime / Guid / Binary) so DateTime / Guid / byte[] / long round-trip restore-faithfully through the JSON snapshot. Falls back to a SessionsIndex partition scan when Sessions.IndexRowKey is empty or stale, so a partially-corrupted Sessions row can't strand a SessionsIndex orphan after cascade. BlobStorageService.UploadDeletionManifestAsync compresses with gzip, pins SHA-256 of the uncompressed payload to blob metadata, and uses IfNoneMatch=* to enforce snapshot-write-once. DownloadDeletionManifest re-verifies the hash and refuses on mismatch (corruption signal that the §13 restore semantics rely on). Two protected-internal-virtual seams isolate the Azure SDK calls so tests bypass BlobsModelFactory entirely. New SessionInventoryContributions table constant added to TableNames.All so the builder can Try-Read it (404 = pre-side-row session = inventory steps omitted). PR2 adds the side-row write path via VulnerabilityCorrelationService. Tests: 25 new xUnit cases (DeletionManifestSerializationTests, DeletionManifestBuilderTests, BlobStorageServiceDeletionManifestTests) cover counts, dumps with EdmType per Azure Tables type, gzip round-trip, hash-mismatch + missing-metadata corruption signals, IfNoneMatch overwrite refusal, SessionsIndex fallback (both empty-key and stale-key paths), determinism across runs, and pre-side-row inventory skip. Backend suite: 1251/1251 green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
7020a2e5a6 |
fix(backend): propagate device/version block+kill across Function App instances
BlockedDeviceService and BlockedVersionService kept their cache per Function App
instance. Once an instance had lazy-loaded a tenant's block list, a cache miss
short-circuited to "not blocked" without ever re-consulting storage — so a Kill
(or Block) added on instance A stayed invisible to instance B for the lifetime
of the process. Live repro: device was killed at 13:11:46Z and kept posting
telemetry for 11+ minutes afterwards on every cadence tick.
BlockedDeviceService.IsBlockedAsync now has two safety nets to bridge other
instances to storage:
- Cache miss after tenant was loaded -> storage point-read on the spot, then
promote the result into the cache. Lets a freshly-set block from another
instance reach this instance immediately.
- Positive cache hit older than EntryRevalidateAfter (30s) -> storage
point-read to re-confirm. Lets manual Unblock, Action upgrade (Block->Kill),
and UnblockAt changes from another instance propagate within seconds.
BlockedVersionService CacheRefreshInterval lowered from 5min to 30s for the same
reason — a version-Kill should not take up to five minutes per instance.
IDeviceSecurityRepository.IsDeviceBlockedAsync gains a BlockedSessionIds tuple
slot so the cache-miss fallback preserves session-aware semantics instead of
silently degrading session-scoped blocks to whole-device.
Tests cover the cache-miss fallback for both Block and Kill actions, the
stale-positive Block->Kill upgrade path, and the stale-positive Unblock path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
9826b0a321 |
fix(agent/v2,backend,web): flag in-flight installs as "likely stuck" on terminal ESP-Apps failure
When the ESP times out on the Apps subcategory, V2 used to terminate the session with the last in-flight app silently frozen in the Installing bucket; the user saw a Failed enrollment but no app name and no app_install_failed event. The agent has no way to tell whether the app eventually succeeded, failed, or hung — so promote it as hedged "likely stuck" with confidence "presumed" rather than a confirmed failure. The promotion fires only on the explicit EspTerminalFailure pathway: a new LastFailureTrigger fact on DecisionState lets EnrollmentTerminationHandler discriminate against MaxLifetimeExceeded (watchdog notbremse), SessionAborted (admin-kill) and EffectInfrastructureFailure (effect-runner gave up) — none of those carry a meaningful claim about app outcome. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
76284cfcb2 |
fix(agent/v2,rules): two-stage health-script emission to close short-session timing gap
The post-PR-HS1 single-source design (HS-NEW-RESULT JSON only) had a real timing gap: IME emits the [HS] new result = JSON line only AFTER its batch send to the Microsoft service, ~30-90 s after the actual script ran (verified in session c1e714e6: 34-74 s gap per policy). Short Autopilot enrollments often end inside that window, so health- script rows were entirely missing from completed sessions even though IME had executed the scripts. Two-stage approach (matches original V1 timing for the early signal, while keeping HS-NEW-RESULT as the canonical truth): - Re-add HS-COMPLIANCE pattern matching BOTH `pre-remdiation` AND `post-remdiation` detection compliance lines → HandleHealthScriptDetectionResult emits a script_completed event immediately. Single regex (`(?<part>pre|post)-remdiation`) covers both phases. - Re-add HS-RUN-CONTEXT / HS-EXITCODE / HS-STDOUT / HS-STDERR patterns feeding a single-slot `_pendingHealthScript` accumulator (IME runs health scripts sequentially within a session — verified). HS-COMPLIANCE merges the slot data onto the early-signal event so exit / stdout / stderr / runContext are populated from the start, not just the inferred compliance verdict. - HS-NEW-RESULT continues to fire later with the full JSON (RemediationStatus, TargetType, ErrorCode, plus the actual remediation phase's data). UI reducer's dataCompleteness() scoring keeps the more complete entry — early-signal row gets enriched in place, no duplicate cards. Remediation phase itself has no early-signal source in IME logs (only the late JSON exposes Info.RemediationExitCode and RemediationScriptOutputDetails). A follow-up based on the agentexecutor invocation line's _RemediationScript.output suffix is possible once we have a real non-compliant cycle log to validate against. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
594b3906ce |
feat(agent/v2,web): capture full health-script lifecycle with phase-aware semantics + live indicator
Replaces the line-by-line HS-COMPLIANCE / HS-EXITCODE / HS-STDOUT / HS-STDERR /
HS-RUN-CONTEXT pattern set (only ever captured the pre-detection phase, reset state
mid-cycle) with a single atomic source: HS-NEW-RESULT matches IME's "[HS] new result =
{JSON}" line and emits up to three script_completed events per policy run (pre-detection
/ remediation / post-detection). HS-SCRIPT-START now also fires a live script_started
event so the UI shows a "running" indicator immediately, before the consolidated final
result lands ~30s-3min later.
Phase-aware failure semantics: detection / post-detection scripts use exit code as a
compliance verdict (not a crash signal), so a non-compliant detection that gets
remediated no longer counts as a "failed" event in metrics — the row renders with
amber styling via isNonCompliantReport(). Only the remediation phase + platform
scripts route non-zero exit to script_failed.
Defensive against wire-format drift: regex uses [\s\S]* with end-anchor, JSON parsed
with try/catch + counter, frontend toNumber() accepts both numeric and string fields
(backend serializes ints as strings via Dictionary<string,string>). Reducer dedupes
with timestamp included so repeated runs of the same policy in long sessions don't
collapse into one row.
V2 only — V1 (Agent.Core) phasing out.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
eacef9c38a |
feat(notifications): opt-in "enrollment started" webhook on session registration
Fires Teams/Slack alert at backend session registration. Default off so existing tenants don't get a surprise notification storm. Triggers on fresh session OR WhiteGlove Part 2 resume; agent restart and re-registration of terminal sessions are skipped via pre-read of the session row before StoreSessionAsync. NotifyOnWhiteGloveResume constant in RegisterSessionFunction lets us disable the resume signal with a 1-line patch if it lands badly with end users, without removing the fresh-session signal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
fce4a03d23 |
feat(sla): persistent SlaTenantStatus row replaces in-memory throttle, fixes 2h breach-notification spam
Background
----------
The in-memory SlaNotificationThrottleService kept its 4h cooldown in a process-
local ConcurrentDictionary. On Flex Consumption the Function host recycles
between 2h timer fires, wiping the dictionary, so the throttle was effectively
a no-op — observed in production as tenant b54dc1af receiving an SLA breach
notification every 2 hours for days on end (1 failed of 11-13 sessions, target
95%, success rate 91-92%).
Design
------
One row per tenant in a new SlaTenantStatus table (PartitionKey = tenantId,
RowKey = "status"). Flat schema with {Type}_* property prefixes for the four
breach kinds (SuccessRate, Duration, AppInstall, ConsecutiveFailures); one row,
one Upsert per evaluation cycle. Rationale documented inline: per-type rows
would be over-normalised for the access pattern (always all-types-per-tenant),
and ~50 tenants x 4 types = max 200 rows ever.
Throttle correctness
--------------------
The throttle decision (now - LastNotifiedAt >= cooldown) and the LastNotifiedAt
write happen in the same TryUpsertAsync(status, ifMatchETag) call. Two parallel
inline calls cannot both succeed in sending the same notification — the loser
gets a 412 from Table Storage, refetches, sees the updated watermark, and
either skips or schedules a new claim attempt. Up to 4 retries; on exhausted
retries we drop the notification rather than risk double-fire (at-most-once).
Cooldown is configurable: AdminConfiguration.SlaNotificationCooldownHours
(default 24, range 1-168). UI round-trip wired through the four required
files (AdminConfigContext, AdminConfigSettingsSection, types/adminConfig.ts,
SectionGlobalSettings).
Resolve path
------------
When a previously-active breach is no longer detected, the row transitions to
IsActive=false + ResolvedAt=now and a single sla_resolved notification fires.
Orphan sweep: tenants whose toggles get turned off while a breach is active
fall out of the qualifying loop, so the timer now also scans every status row
and silently clears any IsActive that no longer has a toggle backing it
(no resolved notification — the tenant disabled SLA, they don't care).
Per-type notes
--------------
- SuccessRate breach notifies with the configured warn-threshold, not the SLA
target — a 94% rate against target=99% / threshold=95% reads "below 95%",
not "below 99%".
- AppInstall now actually evaluates (the toggle was wired but evaluation was
missing). Aligned to the same ISO-week window the dashboard uses
(SlaMetricsService.GetIsoWeekKey), so notifications cannot diverge from the
/sla page. Sample-size floor of 5 apps per week, same as the dashboard.
- AppInstall has its own notification template (own title/summary/facts plus
TopFailingApp fact) — the builder used to silently fall through to the
Duration template for everything not SuccessRate.
- Telemetry Period tag derived from breachType: CurrentMonth for SuccessRate
and Duration, CurrentWeek for AppInstall — so AppInsights / ops dashboards
don't group AppInstall under the wrong period.
- AppInstall ops events now carry the actual installer totals (10 evaluated,
4 failed) instead of 0/0.
- ConsecutiveFailures: inline (ingest) path detects breaches, timer path
detects resolve. Both paths write to the same row through the same CAS
retry loop.
Clock injection
---------------
SlaBreachEvaluationService takes an optional Func<DateTime> nowProvider
(default () => DateTime.UtcNow) so the week-window regression test runs
deterministically on every test day instead of silently skipping when today
happens to fall in the first ISO week of the month.
Removed
-------
- SlaNotificationThrottleService and its tests. The in-memory model is gone;
there is no scenario where it would be wanted alongside the persistent row.
Tests
-----
- 21 new lifecycle + mapping + telemetry tests (CAS conflict + retry, CAS
exhausted-retries drops notification, threshold-not-target in notification,
orphan clearing, AppInstall template / week-window / totals, ConsecutiveFailures
inline persistence, multi-type race-probe, period-mapping theory).
- Backend xUnit: 1218 passed.
- Web vitest: 266 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
7d05a25c90 |
fix(dashboard/stats): move stats routes out from under sessions/ to dodge Azure Functions route collision
Azure Functions' attribute router does not honour literal-vs-parametric
specificity the way ASP.NET attribute routing does. A literal route of
"sessions/stats" was being eaten by the sibling GetSessionFunction whose
template is "sessions/{sessionId}" — the runtime bound /api/sessions/stats
to GetSession with sessionId="stats", which then 404'd because "stats"
fails GUID validation.
Diagnosed live via App Insights:
/api/global/sessions/stats -> 200 OK -> Functions.GetAllSessionStats
/api/sessions/stats -> 404 -> Functions.GetSession (!)
Per-tenant + GA stats routes moved out from under sessions/ for symmetry:
/api/sessions/stats -> /api/stats/sessions (MemberRead)
/api/global/sessions/stats -> /api/global/stats/sessions (GlobalAdminOnly)
EndpointAccessPolicyCatalog, api.ts, and the URL-builder tests follow.
Function-side + api.ts both carry a comment pinning the rationale so a
future "let's tidy these routes" refactor doesn't regress it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
c8902de7c9 |
feat(backend/maintenance): poison-queue backlog watcher with Telegram alerts
Extends the SignalR/Cert pattern to async-worker poison queues. The previous
PR made backlogs visible on /health/detailed - this one makes them actively
page the operator via the existing OpsAlertDispatch -> Telegram path.
New partial MaintenanceService.PoisonQueue.cs runs on the 2 h timer (and on
manual maintenance triggers) and emits two new EventTypes:
- PoisonQueueBacklogHigh (Severity=Warning, threshold >= 1)
- PoisonQueueBacklogCritical (Severity=Error, threshold >= 10)
Dedup mirrors the cert watcher: composite key {EventType}|{queueName} built
from today's Security OpsEvents -> one alert per queue per tier per UTC day.
A tier escalation (warning -> critical) fires a new alert because the
EventType changes; a stable backlog re-pings once per UTC day so it stays in
the operator's attention.
Constants + classifier + watch-list moved from HealthCheckService to the new
MaintenanceService partial - the live health card and the timer-driven
Telegram alert now classify identically (same canonical source). Boundary
and JSON-extractor tests follow the move; HealthCheck tests are reduced to
the card-form assertions they actually exercise.
Operator follow-up (manual, not in code): add two OpsAlertRules in the admin
UI - one per new EventType - to wire Telegram. Until then the watcher writes
OpsEvents but no message is sent, matching how SignalR/Cert alerts were
introduced.
Backend: 1199/1199 (1193 + 6 net from test resplit). Build 0/0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
bf974c4a57 |
fix(dashboard/stats): Active = InProgress only, drop Pending + Stalled
Live data showed cross-tenant 7d breakdown of non-terminal sessions: InProgress 11, Pending 43, Stalled 6 = 60 total. The Pending bucket dominates for any tenant with regular WhiteGlove provisioning — those sessions have completed pre-prov and are waiting for the end user to power on the device, which can take days or weeks. Counting them under "Currently enrolling" implies live activity that isn't happening. Active now matches the literal in-flight status. Pending and Stalled remain visible via the sessions list filters; if we want a richer breakdown on the card row later, it can be split into separate cards without further changing the Active semantics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
2284100915 |
feat(backend/health): surface async-worker poison queue backlog in /health/detailed
Three async workers (analyze-on-enrollment-end, vulnerability-correlate, telemetry-index-reconcile) park 5x-failed messages on <name>-poison. Until now there was no operator-visible signal when this happened: backlogs could grow silently and sessions stayed un-analyzed / vulnerability reports were lost. Adds a "Poison Queues" entry to the detailed health-check next to "SignalR Quota" with the same shape: parallel probe via IPoisonQueueProbe, tri-tier classifier (0=healthy, >=1 warning, >=10 critical, thresholds App-Setting configurable via PoisonQueueWarningThreshold / PoisonQueueCriticalThreshold), GA-only visibility. Per-queue 404 counts as zero because poison queues are created lazily on first poison-move. Per-queue exceptions surface in details as "error: <msg>" and force at least warning so a flaky Storage call cannot mask a real backlog. Web UI needs no change - the generic checks[] render loop on /health-check picks up the new card automatically. Backend: 1193/1193 (1178 + 15 new). Build 0 warnings, 0 errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
068321baaa |
fix(dashboard/stats): reset cards on scope change, pin "in flight" semantics
Follow-up to
|
||
|
Oliver Kieselbach
|
b76a6ccaf0 |
fix(dashboard/stats): server-aggregate cards so they stop drifting with pagination
The five dashboard cards (Active Sessions, Success Rate, Avg. Duration, Total Today, Failed Today) were derived from whatever sessions the client had paginated into memory. With default pageSize=10 and SignalR-driven appends, "Total Today" would jump 40 -> 22 -> 20 within a single visit as the list grew/shrank. "Last 7 days" was a printed label only - the client compute had no 7-day filter at all. Now: - GET /api/sessions/stats?days=7 (MemberRead, JWT tenant) - GET /api/global/sessions/stats?days=7&tenantId= (GlobalAdminOnly, optional tenant filter; absent = cross-tenant sum) - TableStorageService.AggregateSessionStats drains SessionsIndex once over the window: UTC-midnight boundary for today, success rate over terminal sessions only (InProgress no longer drags it down), avg duration over Succeeded with DurationSeconds > 0 only. - useDashboardStats hook owns the cards: initial fetch + debounced refetch on SignalR newSession/newevents in the active scope + forced refetch on reconnect to recover messages missed during an outage. - Client compute removed from useDashboardFilters. New submittedTenantIdFilter state isolates typing-in-filter from submitted-filter so keystrokes never trigger backend round-trips. Tests: 17 xUnit (aggregation invariants + days parsing) + 7 vitest (URL builder). Suites: backend 1177/1178 (unrelated RawTablePaginationTests flake), web 266/266. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
5f6de2c02c | fix health dashboard | ||
|
Oliver Kieselbach
|
c101eda796 | health check fix | ||
|
Oliver Kieselbach
|
871ec31bf6 |
feat(backend/health): surface SignalR free-tier quota in /health/detailed
HealthCheckService now runs a fourth check (SignalR Quota) alongside Storage, Processing, and Agent Binaries. Live read of ConnectionCount (max/1h) and MessageCount (total since 00:00 UTC) via the same IAzureMonitorMetricsReader that backs the maintenance-time Ops watcher - no cache; the data is already 3-5min stale from Azure Monitor's side, and a /health/detailed re-check is expected to give fresh values during incident response. Reuses MaintenanceService's internal classifier (CalculatePercent, ClassifySignalRQuotaTier) so the watcher and the dashboard agree on the 80%/95% boundaries down to the floor-vs-ceil rounding. Tier mapping: None -> healthy, Warning -> warning, Critical -> unhealthy. Worst-of-both-metrics wins. PerformAllChecksAsync now uses a tri-state combiner (any unhealthy / any warning / else healthy) so a Warning tier correctly tints the OverallStatus bar yellow instead of red. Fail-soft: - SignalRResourceId unset -> Status="unknown" with "watcher not configured" (excluded from OverallStatus combiner so dev/local stays healthy) - Both metric reads null -> Status="unhealthy" pointing at the most likely cause (Function App MI missing 'Monitoring Reader' on the SignalR resource) - One read null -> "no data" for that row, normal classification for the other Frontend unchanged: the existing /health-check Backend Services card loop renders the new check with Connections, Messages, and Resource as Details. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
d51bde6017 |
refactor(release): parametric per-line agent build pipeline
Replaces V1/V2-hardcoded build + bootstrap with a single parametric
pipeline that scales to V3+ without rewrites. Full architecture in
.claude/plans/v2-cutover.md.
Stable namespace (Bootstrap-side, never renamed): generic Bootstrap
script, agent owns its --install defaults so it works for any line.
Versioned namespace (per-line, SelfUpdater-side): isolated *-v{N}.*
blobs; SelfUpdater never crosses lines or touches stable.
Backend GetAgentConfig dispatches per X-Agent-Version-Major to
AdminConfiguration.GetAgentLine(int) -> LatestAgentV{N}* fields.
Future V3 = add field set + switch arm. TableConfigRepository.Map
falls back to legacy unsuffixed columns for transparent migration.
V1 SelfUpdater now reads version-v1.json (would have been clobbered
by the V2 cutover otherwise). V2 agent owns its own 600s TenantId-wait
default so the Bootstrap calls --install plain.
Tests: 3289 across 6 suites, all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
0b80f09a74 |
refactor(notifications): bell on pure SignalR push, kill 60s polling
The two notification-bell React contexts polled GET /api/notifications and
GET /api/global/notifications every 60s regardless of activity, generating
3017+618 = 3635 calls/24h with near-zero hit rate (notifications are emitted
sparsely). SignalR was already used app-wide for live session updates; the
bell was the lone hold-out on a pull model.
Switch to the same push model. Steady-state traffic drops ~86% and the bell
updates sub-second instead of within 60s.
Architecture (Option C+):
- Mount -> 1x HTTP fetch (cold-start hydration via existing GET endpoint)
- SignalR onreconnected -> re-fetch (covers the disconnect window)
- SignalR push -> live append/dismiss
- No periodic polling, no setInterval
Audience-tier security: TenantNotificationAudienceCatalog still drives
visibility (Member vs Admin), but enforcement moves into SignalR group
membership: tenant-{id}-notify-member is joined by every tenant member,
tenant-{id}-notify-admin only by Tenant Admins / Global Admins. The
AddToGroup function fail-closes with 403 on Member-tier joins to the
admin group, so Admin-tier notifications cannot leak via push.
Backend
- ISignalRNotificationService extracted; SignalRNotificationService
implements it. Resolves the long-standing test-mocking blocker
(the real service's constructor validates AzureSignalRConnectionString
and BuildServiceManager(); previously you could not unit-test
consumers). Tenant + Global NotificationService consume the
interface; the three legacy callers (Analyze, Ingest, VulnCorrelate)
still bind to the concrete class -- migrate per test need.
- Six new push methods: SendTenantNotification{,Dismissed,DismissedAll}
and SendGlobalNotification{,Dismissed,DismissedAll}. Tenant variant
routes to the audience-correct group.
- Tenant + Global service push after a successful repo write
(fire-and-forget; never throws into the caller).
- SignalRGroupHelper: parsing handles the two new -notify-{member,admin}
suffixes; new IsTenantNotifyAdminGroup predicate; group-name builders.
- SignalRAddToGroupFunction: explicit 403 when a non-admin attempts
to join the admin-tier group.
Frontend
- TenantNotificationContext: setInterval gone. useSignalR() to
join member-group always, admin-group iff isTenantAdmin || isGlobalAdmin.
Listens for tenantNotification (append + dedupe), tenantNotificationDismissed
(filter), tenantNotificationsDismissedAll (clear). onreconnected
triggers re-fetch.
- GlobalNotificationContext: same pattern against global-admins group.
- app/layout.tsx: SignalRProvider hoisted above the notification
providers so useSignalR() resolves inside them.
Tests
- SignalRGroupHelperTests (8): parsing for all four group formats
incl. new -notify suffixes, IsTenantNotifyAdminGroup predicate.
- TenantNotificationServiceSignalRPushTests (8): audience-correct push
on Create (Admin / Member / Unknown -> Admin fail-closed), no-push
on repo failure, no-push on DismissAll(count=0), no-push on empty
tenantId.
- FakeSignalRNotificationService: in-memory spy implementing the
new interface; replaces the unmockable real service in tests.
- AuthFunctionSideEffectTests + TenantNotificationServiceTests:
constructor signatures updated for the new SignalR dep.
Suites
Backend xUnit: 1161/1161 (was 1146, +15 net).
Web vitest: 259/259 (unchanged; existing notification-API +
gate tests still cover).
Solution build: 0 warnings, 0 errors.
Expected post-deploy reduction (24h):
GET GetTenantNotifications: 3017 -> ~50
GET GetGlobalNotifications: 618 -> ~10
POST AddToGroup: ~246 -> ~350-450 (two joins per tab)
Net bell load: -86%, sub-second live UX.
Disconnect tradeoff (accepted): if SignalR drops without reconnect
(rare -- withAutomaticReconnect retries 1/2/4/8/16/30s ... forever),
the bell shows the snapshot from the last fetch but no live updates
until next reload. The initial-3-retry give-up path of the SignalR
client still gets the user a hydrated bell because the HTTP fetch
runs independently on mount.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
bc3cefa790 |
feat(backend/observability): SignalR free-quota watcher with tiered OpsEvents
Hourly TimerFunction queries Azure Monitor for SignalR ConnectionCount (max over last hour) and MessageCount (sum since UTC midnight) and emits Warning (>=80%) / Error (>=95%) OpsEvents when the free-tier limits (20 connections, 20k messages/day) are approached. Mirror of the cert-expiry pattern - global TenantId=null events, dedup'd to one emission per EventType per UTC day. Fail-soft: skips silently when SignalRResourceId app setting is unset, logs warning on Azure Monitor auth/query errors. Function App's managed identity needs 'Monitoring Reader' on the SignalR resource. Also called from the manual maintenance trigger so 'Run Maintenance Now' exercises every health check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
5a4309dc9a |
fix(backend/maintenance): also run cert-expiry watcher in manual trigger path
|
||
|
Oliver Kieselbach
|
f2257a4b02 |
fix(backend/security): cert-expiry watcher uses bucket-level logic, not per-cert
77fde56f's per-cert alarm fires for every expiring cert independently. That is wrong: when a Root or Intermediate has a fresh successor in the bundle (e.g. intune-intermediate-mdm-2026.pem alongside intune-intermediate-mdm-2028 .pem), the older cert is just a chain-continuity helper for in-flight device certs. Alarming on it would page the operator to "rotate" something that is already rotated. Refactor to bucket-level: per role (Root, Intermediate), pick the cert with the MAXIMUM NotAfter and classify only that. Older expiring/expired siblings are silent. Operator only gets paged when the freshest itself nears expiry, i.e. when there is genuinely no successor yet. Concrete behaviour change for current bundle (today 2026-05-08): Roots: freshest is intune-root-2030.pem (1591d) -> silent Intermediates: freshest is intune-intermediate-mdm-2028.pem (859d) -> silent The expired intune-intermediate-mdm-2026.pem (-3d) no longer triggers a spurious Critical because mdm-2028 has it covered. Tests: 7 new EvaluateBucket cases (empty, single fresh, single soon, old+ fresh successor stays silent, old+soon successor fires on successor, all expired falls back to freshest-expired, two future successors). Total 38 cert-related tests, full suite 1105/1105. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
77fde56f30 |
feat(backend/security): cert-expiry watcher in maintenance with tiered OpsEvents
Adds proactive monitoring of the embedded Intune CA bundle so the operator
gets a Telegram ping months before agent mTLS would break. Bundle rotation
(re-exporting fresh PEMs from a corporate-enrolled device) is the only fix
and needs lead time - today's outage from
|
||
|
Oliver Kieselbach
|
0d9ef12f90 |
chore(backend/security): drop legacy duplicate Intune CA PEMs
intune-root-ca.pem and intune-intermediate-ca.pem are bit-for-bit duplicates of intune-root-2030.pem and intune-intermediate-mdm-2026.pem (matching SHA1 fingerprints A1:97:D6:... and B7:E4:96:...). The post-9af3de15 loader globs intune-root-*.pem and intune-intermediate-*.pem and dedupes by thumbprint, so the duplicates were silently de-duped at load. Remove them now to make the bundle self-explanatory: one PEM per real CA, name encodes its NotAfter. Verified by running CertificateValidatorTests (8/8 incl. the real-device-cert happy-path) which loads the bundle via the same glob path the runtime uses. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
794800c0cf |
chore(backend/security): include leaf cert Subject/Issuer/Thumbprint in chain-failure log
Post-deploy of
|
||
|
Oliver Kieselbach
|
9af3de15d4 |
fix(backend/security): pin client cert chain trust to embedded Intune root (v2)
Re-applies
|