mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 20:53:06 +00:00
ef530fb890deb4efad712d25974e100c4fd64fb8
1482 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oliver Kieselbach
|
ef530fb890 |
feat(agent/v2): DAD liveness + agent_shutting_down gap closure
DAD liveness (3 new state-change-only events, NOT periodic, max 3 per detector lifetime): desktop_detector_started / _first_poll / _no_candidate. Distinguishes "agent dead post-reboot" from "DAD wiring broken" from "user never logged in" in sessions missing desktop_arrived (the ca9e3c59 timeout pattern). Threshold via global AdminConfiguration .DesktopDetectorNoCandidateTimeoutMinutes (default 10, 0 = disabled), round-trip wired through TableConfigRepository, GetAgentConfigFunction (ConfigVersion 27 -> 28), CollectorConfiguration, DefaultComponentFactory, DesktopArrivalHost + admin UI. Also fixes a peripheral-loss bug: V2's DesktopArrivalDetector.OnTraceEvent callback was never wired, so the existing desktop_excluded_user and desktop_real_user_detected events were dead code in V2. Now routed via InformationalEventPost in the host. Shutdown gap closure (4 new agent_shutting_down emit paths): ctrl_c, process_exit, unhandled_exception (with exception type + 500-char- truncated message), runtime_host_exit (defensive finally fallback). Cross-path Interlocked idempotency gate shared with EnrollmentTermination Handler via new tryClaimShutdownEvent ctor param prevents double-emit when Ctrl+C races a real Terminated transition. Event-type unification (V2 only): CreateAuthThresholdHandler now emits agent_shutting_down with data.reason=auth_failure (was agent_shutdown). EnrollmentTerminationHandler reason vocabulary unified to lowercased tags (decision_terminal, max_lifetime). V1 keeps legacy agent_shutdown (phasing out). Web/MCP consumers (eventHelpers, whiteGloveSplit test fixtures, gather-rules eventTypes catalog, MCP semantic-event list, docs) treat both as synonyms. Tests: 3411 green (1538 backend + 1065 V2 core + 530 V1 core + 294 DecisionCore + 347 vitest). TypeScript noEmit clean. 5 new DAD-liveness tests, 5 new gap-path tests, 2 new idempotency tests, 3 reason-vocab updates. Not in this PR: cross-process death detection for the actual ca9e3c59 silent-post-reboot-death pattern (Logon-Trigger / external watchdog) - separate follow-up. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
f1583fc586 |
fix(backend,web): move cascade-delete admin routes from /api/admin/* to /api/global/*
The /api/admin/* prefix collides with the Azure Functions runtime's built-in
admin paths, which sit behind mTLS client-certificate auth. Browser preflights
to /api/admin/* trigger TLS renegotiation against the Function App's mTLS
boundary and die before our PolicyEnforcementMiddleware ever runs — the
operator sees CORS-shaped errors that aren't actually CORS-shaped.
Repo convention all along has been /api/global/* for cross-tenant GA-only
HTTP endpoints (e.g. global/session-deletions, global/session-reports). The
cascade-delete endpoints landed under /api/admin/* in the original 7-PR
backend branch and went undetected until the admin UI started calling them
from the browser.
Routes renamed:
admin/sessions/{sessionId}/delete/preview → global/sessions/{sessionId}/delete/preview
admin/sessions/{sessionId}/restore → global/sessions/{sessionId}/restore
admin/sessions/{sessionId}/deletion-manifest → global/sessions/{sessionId}/deletion-manifest
admin/tenants/{tenantId}/deletion-manifests → global/tenants/{tenantId}/deletion-manifests
Touched:
- 4 Function classes: Route attribute + XML-doc header
- EndpointAccessPolicyCatalog: 4 entries (TenantScoping.RouteParam preserved
on the tenants route), with a comment explaining why /api/global/* is the
load-bearing prefix for new GA-only HTTP routes
- DeleteSessionFunction: 3 user-facing "Cascade is poisoned; recover via
POST /api/.../restore" error messages updated
- SessionDeletion.cs: XML-doc reference to the restore route
- lib/api.ts: 4 frontend URL builders (storedManifest, preview, restore,
tenantManifests)
- RestoreSessionFunctionTests: 3 EndpointAccessPolicyCatalog.FindPolicy
lookups
- deleteSessionResponse.test.ts: test fixture message string
No semantic change — only the URL prefix moves. CORS config + JWT middleware
+ policy gating all work the same. The Restore Browser tab + the per-row
Preview / Restore buttons in the Session Cleanup admin page now actually
reach the worker from a browser.
Tests: backend xUnit 1538/1538 pass, web vitest 347/347 pass, dotnet build +
tsc clean.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
44076686b6 |
feat(backend,web): restore-browser tab + detailed dry-run/real-run dialog
Follow-ups on the Session Cleanup admin page after first hands-on review:
an entry point for restoring already-completed cascades (Sessions row gone,
manifest still in the 33-day TTL window), a proper outcome panel that surfaces
what the backend already returned, and a sidebar fix.
== New Restore Browser tab ==
- Backend: GET /api/admin/tenants/{tenantId}/deletion-manifests enumerates
every persisted snapshot blob under <tenantId>/, parses (sessionId,
manifestId, sizeBytes, lastModifiedUtc) from the blob name, groups by
sessionId and sorts both levels by recency. GA-only + TenantScoping.RouteParam
in EndpointAccessPolicyCatalog (pinned by EndpointPolicyCatalogCompletenessTests).
- BlobStorageService.EnumerateDeletionManifestsByTenantAsync — sibling to
the existing maintenance-sweep enumerator but tenant-prefixed and yielding
size + timestamp so the UI can rank without an extra HEAD per blob.
- Web: new tab "Restore Browser" between Stranded and Activity. Tenant
dropdown (sourced from AdminConfigContext.tenants) → collapsible
session/manifest tree on the left, manifest summary on the right with a
Restore… button that opens the dialog with IDs pre-filled. Useful when
the cascade completed cleanly and the row no longer appears on the
In-Flight / Poisoned / Stranded tabs.
== Reusable summary view ==
DeletionManifestSummaryView extracted from the modal so the same fetch +
field block + worker-progress + preflight-counts + last-failure panel renders
both inside the modal (Preview button on the other tabs) and inline on the
Restore Browser's right-hand panel. Modal becomes a thin wrapper.
== Dry-run / real-run side-by-side ==
RestoreConfirmDialog now keeps both outcomes visible:
- Two slots (dryRunResult, realRunResult), each rendered as a full result
section with predicted/actual rows-by-table, inventory re-increments,
duration, backend message, current state, pending manifest id.
- "Start over" resets both slots so the operator can re-run cleanly.
- Submit buttons gate the workflow: pre-submit shows "Run dry-run" + an
explicit "Skip dry-run · Restore now"; after dry-run the second button
becomes "Apply restore" (clear-intent); after real-run all submit
buttons disappear so accidental double-applies aren't possible.
Previously the dialog discarded everything but Outcome + Mode + dryRun
flag, hiding the actual point of dry-run (the wouldRestoreByTable map).
== Codex review follow-ups (this round) ==
- P2: Restore Browser cleared the displayed sessions only after the new
fetch returned, so a fast A→B switch (or out-of-order responses) could
leave the operator looking at tenant A's manifests with the dropdown
saying B. Fix: clear sessions/expandedSessions/selection synchronously
on tenant change, abort the in-flight request via AbortController,
pair the response with a loadedTenantId state, and gate rendering on
loadedTenantId === selectedTenantId. Three-layer protection so a late
response that sneaks past the abort can't paint stale data.
- P3: PoisonedTab's onRestored callback closed the dialog immediately
after a real restore, defeating the new dry-run vs real-run side-by-side
display. Fix: drop setRestoring(null) from the callback so the dialog
stays open until the operator dismisses it via Close / Start over.
Background refresh of the Poisoned list still happens via setRefreshKey.
Now consistent with the Restore Browser tab.
== Sidebar highlight fix ==
GlobalSidebar marked BOTH ga-ops sub-items active when on the longer
nested route, because each sub used pathname.startsWith(sub.href + "/")
independently. Visible bug: navigating to /admin/ops/session-cleanup lit
up both "Maintenance" (/admin/ops) and "Session Cleanup" in the sidebar.
Fix: compute the longest-matching sub.href once per expanded group, mark
only that one active.
Tests: backend xUnit 1538/1538 pass (one new endpoint registered in the
catalog, picked up by the existing completeness test). Web vitest 347/347
pass. TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit for GetTenantDeletionManifestsFunction (same HttpRequestData
mock-harness gap as the other read endpoints).
- vitest for RestoreBrowserTab + the new dialog flow (defer until a
real field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
fa8ad38ed1 | change bell notification to always visible | ||
|
Oliver Kieselbach
|
09bfe7a1bc |
feat(backend,web): admin session-cleanup UI for V2 cascade-delete + codex follow-ups
Operationalises the 7-PR backend cascade-delete pipeline that landed via commits
c8658b9f..921994e8: per-tenant V2 opt-in switch, audit-trail consolidation, and
a Global-Admin-only Session Cleanup page to track in-flight / poisoned / stranded
cascades and trigger restores.
== PR-A: per-tenant V2 enrollment ==
- UpdateTenantConfigurationFunction extends the GA-only gate list with
EnableCascadeDeleteV2 so tenant admins cannot self-opt-in / out of the
snapshot+restore pathway.
- TenantManagementSection.tsx adds a purple-bordered "Enable V2 Cascade Delete"
checkbox in the GA edit modal and a "Cascade V2" badge on the tenant card so
rollout state is visible at a glance.
== PR-B: audit consolidation ==
- SessionDeletionHandler drops the 15-per-cascade deletion_step_completed /
deletion_step_failed / deletion_verification_failed audit emissions. Step
telemetry lives in DeletionProgress + structured logs; tenant admins now see
just deletion_started + deletion_completed + deletion_restored.
- SessionDeletionWorker no longer writes a deletion_poisoned tenant audit;
the signal moved to a new SessionDeletionPoisoned OpsEvent (operator-bound,
Telegram-routable, dual-registered in OpsAlertRulesSection.tsx catalog).
IMaintenanceRepository dependency removed from the worker constructor.
- SessionRestoreService stops emitting the deletion_manifest_downloaded
breadcrumb; deletion_restored is the operator-visible outcome.
== PR-C: Session Cleanup admin page ==
- New /admin/ops/session-cleanup route (gated by app/admin/layout.tsx's
ProtectedRoute requireGlobalAdmin). Sidebar entry under ga-ops alongside
Maintenance. Four URL-state tabs: In-Flight / Poisoned / Stranded Queued /
Activity (links to existing /audit page).
- Backend reads: GetSessionDeletionsListFunction
(GET /api/global/session-deletions?state=&strandedSinceMinutes=) +
GetSessionDeletionManifestFunction
(GET /api/admin/sessions/{id}/deletion-manifest?tenantId=&manifestId=&mode=).
Both registered as GlobalAdminOnly in EndpointAccessPolicyCatalog.
- DeletionPreviewModal reads the persisted snapshot via the stored-manifest
endpoint (NOT a fresh dry-run via the existing /delete/preview, which would
enumerate live-but-partially-deleted data for in-flight cascades). Renders
manifest counts, snapshot SHA, worker progress (CompletedSteps,
VerificationDone, TombstoneStarted, aggregate decrements), and the last
verification failure with residual sample. Download button uses
authenticatedFetch + Blob + Object URL so the Bearer token is sent.
- RestoreConfirmDialog drops the Full/Partial radio (mode is auto-selected
by SessionRestoreService based on cascade state — forcing the wrong mode
would corrupt state). Reason field threads through to deletion_restored
audit details. Dry-run default = true and previews the auto-selected mode.
== Codex review follow-ups (across four rounds) ==
- High: tenantId is explicitly accepted on /delete/preview (query) and
/restore (body); GA cross-tenant resolution prefers the explicit value over
JWT TargetTenantId, falling back to SessionsIndex lookup. Without this, a
GA acting on a session in another tenant got their JWT home tenant.
- High: new stored-manifest endpoint replaces the dry-run preview for
in-flight / poisoned rows; previous fix built a fresh manifest against
already-partially-deleted data and misrepresented the cascade.
- Medium: SessionDeletionPoisoned OpsEvent now carries failureType +
failureMessage + observedResidualCount + a small residual preview, persisted
by the handler into DeletionProgress (LastFailureType, LastFailureMessage,
LastObservedResidualCount, LastResidualSampleJson) before the throw so the
worker can include them.
- Medium: dry-run restore reports the AUTO-SELECTED mode (full/partial) so
the dialog's preview reflects what the real run will do, instead of the
literal string "dryRun".
- Medium: residual count field renamed to observedResidualCount with an
honest XML doc — CascadeVerificationService caps at 50 per table AND
short-circuits after the first failing table, so the number is a lower
bound. UI surfaces "N of 50+ observed rows" when N hits the cap.
- Medium: three-layer cap defense on the OpsEvent residual sample to stay
under the 4096-char TableOpsEventRepository truncation: max 5 entries
(OpsEventResidualSamplePreviewSize), per-field max 96 chars with `…` marker
(OpsEventResidualKeyMaxChars), total preview budget 1200 chars
(OpsEventResidualPreviewBudgetChars) with trailing-entry drop. Full sample
stays in the progress blob for the admin UI.
- Low: estimatedSnapshotBytes field name aligned end-to-end; activity tab
text clarifies audit-action vs OpsEvent distinction.
Tests:
- Backend xUnit 1538/1538 pass (up from 1510 baseline; +28 across the new
Session Cleanup endpoints, OpsEvent enrichment, restore-mode reporting,
cap-defense unit + integration tests).
- Web vitest 347/347 pass.
- TypeScript strict + dotnet build clean.
Deliberately deferred:
- xUnit tests for GetSessionDeletionsListFunction +
GetSessionDeletionManifestFunction (no HttpRequestData mock harness in repo;
would be a separate refactor).
- vitest for DeletionPreviewModal + RestoreConfirmDialog (defer until first
field issue).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
5846b11775 |
fix(mcp-deps): bump transitive protobufjs to 7.5.8 to resolve Dependabot alerts
Patch-level bump within onnxruntime-web's ^7.2.4 constraint; closes 8 open Dependabot alerts (CVE family: DoS, code injection, prototype pollution). Lockfile-only change, no manifest update needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
1c00f44709 |
Merge pull request #63 from okieselbach/fix/v2-classic-hello-disabled-deadlock AND parts of the deletion cascade feature
fix(agent-v2): Classic-v1 + Hello=disabled completion deadlock AND parts of the deletion cascade feature |
||
|
Oliver Kieselbach
|
2e7172b6ab |
test(agent-v2): pin coordinator re-raise path via OnEspExited test seam (PR6/6)
The PR4 coordinator-pipeline tests drove adapter.TriggerEspExitingFromTest(), which only proves the adapter emit — it bypasses the coordinator's OnEspExited handler (where LastEventOccurredAtUtc mirroring + public EspExited re-raise live). A future regression that breaks the coordinator subscription, the LastEventOccurredAtUtc mirror, or the public EspExited fire would slip past. This commit adds a coordinator-level test seam mirroring the existing TriggerFinalizingSetupPhaseForTest pattern, and reroutes the wire-through tests through it. EspAndHelloTracker.TriggerEspExitedForTest: - internal seam (InternalsVisibleTo) that calls OnEspExited(this, args) - exact signature match to the live ShellCoreTracker.EspExited delegate - exercises LastEventOccurredAtUtc mirror + public EspExited invoke + finally reset — i.e. the full coordinator-side contract ShellCoreToEngineEspExitingTests: - Coordinator_EspExited_handler_re_raises_to_adapter_as_EspExiting_signal: asserts source-time preservation through the coordinator → adapter chain and that no clock-fallback "derivedTimestamp" evidence tag was added. - Coordinator_EspExited_handler_clears_LastEventOccurredAtUtc_after_re_raise: new defensive test for the finally-block reset; protects against a stale ShellCore timestamp bleeding into a subsequent Hello/Provisioning forward. - Coordinator_to_reducer_arms_HelloSafety_when_AccountSetup_observed: rewritten to use the coordinator seam; now asserts HelloSafety dueAtUtc == sourceTime + 300s (not Clock.UtcNow + 300s) so source-time threading is proven end-to-end (ShellCoreTracker → Coordinator → Adapter → Engine), not just adapter → engine. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
35c7e4cfc9 |
fix(decision-core): emit CancelDeadline effect for HelloSafety in fast-path (PR5/5)
The Hello-disabled fast-path called builder.CancelDeadline(HelloSafety) but did not emit a corresponding DecisionEffectKind.CancelDeadline. The reducer state no longer carries the deadline, but the external scheduler (DefaultComponentFactory) keeps the timer armed and fires DeadlineFired:HelloSafety after the 300s window — which then dead-ends in the reducer because the deadline name is no longer in state. This commit adds the missing scheduler-visible cancel: - TransitionToFinalizing gains an optional extraLeadingEffects parameter so the fast-path can prepend a CancelDeadline effect ahead of the normal ScheduleDeadline(FinalizingGrace) + phase_transition. Existing call-sites (HandleHelloResolvedV1, HandleDesktopArrivedV1's non-fast-path branch, HandleHelloSafetyDeadlineFired) keep their current behaviour by passing null. - The fast-path checks state.Deadlines for HelloSafety BEFORE building the effect — only emits the cancel when the deadline was actually armed. Avoids no-op cancellations on the common case where the fast-path is the sole completion path. Tests: - HelloDisabled_FastPath_cancels_armed_HelloSafety_deadline_in_state_and_effects: expanded the existing state-only assertion to also assert the CancelDeadline(HelloSafety) effect alongside ScheduleDeadline(FinalizingGrace) and phase_transition. - HelloDisabled_FastPath_without_armed_HelloSafety_does_not_emit_redundant_CancelDeadline_effect: new negative test guarding against audit-trail noise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
4a6bc656df |
fix(agent-v2): wire EspExited through coordinator to production adapter (PR4/5)
PR1 added EspExited on ShellCoreTracker + a mapping on ShellCoreTrackerAdapter, but ShellCoreTrackerAdapter is only used in single-tracker test wiring. In production EspAndHelloHost instantiates the EspAndHelloTracker coordinator, and EspAndHelloTrackerAdapter subscribes to coordinator-forwarded events — the new EspExited never reached the engine on a real device. This commit reroutes EspExited through the coordinator (the same path the other ShellCore-originated events take): - EspAndHelloTracker re-raises EspExited from its inner ShellCoreTracker, with LastEventOccurredAtUtc mirrored from the EventArgs timestamp so the adapter's ResolveOccurredAt picks up the source-event time (matching the existing WhiteGloveCompleted / EspFailureDetected forwarding contract). - EspAndHelloTrackerAdapter subscribes to coordinator.EspExited and posts DecisionSignalKind.EspExiting with no per-kind dedup — Shell-Core 62407 fires on every ESP phase transition, and the reducer's ShouldTransitionToAwaitingHello guard decides which occurrence arms HelloSafety. The coordinator's existing IsIntermediateDeviceEspExit filter only governs the legacy FinalizingSetupPhaseTriggered path; the new EspExiting signal bypasses it deliberately so the engine remains the single source of truth. - EspExitedEventArgs becomes public (was internal) because the coordinator's public event surface needs an at-least-as-accessible args type. Tests: - EspAndHelloTrackerAdapterTests: +2 tests covering the coordinator-forwarded mapping and the no-dedup contract. - ShellCoreToEngineEspExitingTests: +2 tests exercising the full production topology (coordinator → adapter → ingress → engine.Reduce → HelloSafety arm). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
4853f8c6d1 |
test: cover EspExiting adapter + Hello-disabled fast-path + wire-through (PR3/3)
Three new test suites verifying both halves of the V2 Classic-v1 + Hello-disabled deadlock fix: ShellCoreTrackerAdapterTests (+3 tests, +1 expanded): - EspExited → DecisionSignalKind.EspExiting mapping (kind, source, evidence) - No-dedup contract: every 62407 occurrence forwarded individually so the reducer's ShouldTransitionToAwaitingHello guard can disambiguate the genuine post-AccountSetup exit from intermediate Device→Account exits. - Source-event timestamp preserved (not _clock.UtcNow) — critical for backfill so EffectiveDeadlineBase floors HelloSafety at the actual ESP-exit instant. - Existing All_three_events_can_fire_in_one_session expanded to All_four_*. ClassicHelloDisabledCompletionTests (6 tests): - Fast-path happy path: HelloPolicyDisabled + AccountSetup + DesktopArrived → Finalizing → FinalizingGrace deadline → enrollment_complete. - Three guard tests: fast-path skipped when policy not detected, before AccountSetup, and when policy is enabled. - HelloSafety reducer path (Fix 1 belt-and-suspenders): EspExiting → HelloSafety 300s → Timeout → DesktopArrived → Finalizing. - Fast-path cancels an armed HelloSafety deadline so it cannot fire post-Completion. ShellCoreToEngineEspExitingTests (2 wire-through tests): - Real ShellCoreTracker.HandleBackfillRecord(esp_exiting_description) drives the .NET event → adapter subscription → ingress.Posted, with source timestamp preserved. - End-to-end: same backfill → adapter signal → engine.Reduce on a seed state with AccountSetup entered → HelloSafety deadline armed at sourceTime+300s. Full DecisionCore.Tests: 293/293 pass. Full Agent.V2.Core.Tests: 1048/1048 pass. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
a32706dee8 |
fix(decision-core): Hello-disabled fast-path in HandleDesktopArrivedV1 (PR2/3)
When HelloPolicyEnabled is explicitly false AND AccountSetup has been reached, synthesise HelloOutcome="Skipped" on DesktopArrived and route directly through Finalizing instead of waiting out the full HelloSafety 300s window. This is the belt-and-suspenders companion to PR1's EspExiting reducer path: whichever fires first wins, and the dual-path design survives a future regression in either half (missing EspExiting adapter or missing HelloPolicyDetected adapter). Both guards are required: - HelloPolicyEnabled?.Value == false (policy explicitly disabled) - AccountSetupEnteredUtc != null (mirrors ShouldTransitionToAwaitingHello) This prevents a premature mid-DeviceSetup completion if HelloPolicyDetected happens to fire before AccountSetup is observed. Unknown policy (HelloPolicyEnabled == null) keeps the prior pessimistic behaviour. The fast-path cancels HelloSafety on its way through so a deadline armed by an earlier EspExiting cannot fire post-Completion and dead-end in the reducer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
879b89d711 |
fix(agent-v2): emit DecisionSignalKind.EspExiting from ShellCoreTracker (PR1/3)
ShellCoreTracker observes the Shell-Core 62407 OOBE_ESP*Exiting event but only emitted it as a wire-event + FinalizingSetupPhaseTriggered notification. No V2 SignalAdapter mapped it to DecisionSignalKind.EspExiting, so HandleEspExitingV1 (which arms the HelloSafety 300s deadline) was never invoked. On Classic-v1 enrollments with Hello policy disabled this leaves the reducer stuck in EspAccountSetup until the agent max-lifetime watchdog fires after 360 min. This commit adds a dedicated EspExited .NET event on ShellCoreTracker (raised both on live event log records and on backfill) and a matching subscription in ShellCoreTrackerAdapter that posts a DecisionSignalKind.EspExiting signal. OccurredAtUtc is taken from the source event timestamp (not _clock.UtcNow) so HandleEspExitingV1's EffectiveDeadlineBase floors HelloSafety at the actual ESP-exit instant rather than wall-clock-now on a backfilled run. The adapter does NOT dedup EspExiting because Shell-Core 62407 fires at every ESP phase transition (Device→Account, Account→End); the reducer's ShouldTransitionToAwaitingHello guard already distinguishes the genuine post-AccountSetup exit from intermediate exits. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
c237140d59 |
fix(backend): cascade-delete codex follow-ups #2 (F1 DELETE cross-tenant, F2 register guard, F3 tombstone marker, F4 cross-tenant test, bootstrap-bypass)
F1 — DELETE /api/sessions/{sessionId} bypassed cross-tenant validation
because the catalog entry had no TenantScoping. A Global Admin clicking
"delete" on a session in tenant B from the portal silently routed the
delete into their own JWT tenant A, never finding the row. Catalog
entry gains TenantScoping.QueryParam; DeleteSessionFunction switches
from TenantHelper.GetTenantId (JWT-only) to RequestContext.TargetTenantId
(middleware-validated).
F2 — RegisterSessionFunction wrote sessions without any deletion-state
check; a mid-cascade register could mutate a Preparing/Queued/Running
row, and a fresh register past the tombstone could re-Add a row the
manifest no longer covers. Guard now runs at the entry of the shared
ProcessRegisterAsync core — covering BOTH /agent/register-session
(cert-auth) AND BootstrapRegisterSessionFunction (bootstrap-code-auth),
which previously bypassed the cert-auth entry's check. 410 Gone on
lock-state or tombstone hit.
F3 — SessionDeletionGuard's "Sessions row missing -> silent pass"
contract let late ingests / registers slip past a completed cascade
and orphan rows the manifest couldn't describe. New short-lived
SessionTombstones table (PK=tenantId, RK=sessionId) carries a marker
written by SessionDeletionHandler.ExecuteTombstoneAsync BEFORE the
Sessions-row delete, with a 7-day default ExpiresAt. Guard now reads
the marker on a missing Sessions row; live and present -> throws with
sentinel state "Tombstoned" -> 410 Gone. Full-Restore deletes the
marker after re-inserting the Sessions row; partial-restore leaves it
alone (never written). SessionDeletionMaintenanceFunction pruning
step runs even when the kill-switch is active (markers are race
shields, not policy artefacts).
F4 — CrossTenantAccessTests had no coverage for DELETE
sessions/{sessionId}. Theory entries lifted into the same three asserts
that already cover GET — same-tenant allowed, cross-tenant non-GA
blocked, cross-tenant GA allowed — and into the QueryParam-scoping
catalog theory so the regression that produced F1 surfaces in CI.
Bootstrap follow-up — RegisterSessionFunction.Run had the guard inline
but BootstrapRegisterSessionFunction calls _inner.ProcessRegisterAsync
directly, bypassing it. Guard moved into ProcessRegisterAsync so both
entry points hit it before StoreSessionAsync. Added structural wiring
tests (DI field, internal accessibility, bootstrap delegation type)
that fail in CI if the architecture regresses to inline copy.
Tests:
- SessionDeletionGuardTests +3: marker-active blocks, no-marker passes
on missing row, sessions-row-exists skips tombstone probe.
- SessionDeletionHandlerTests +1: marker is written BEFORE the
Sessions-row delete (Moq Callback ordering assertion).
- SessionRestoreServiceTests +2: full-restore deletes marker,
partial-restore does not.
- SessionDeletionMaintenanceFunctionTests +2: prunes expired markers,
pruning runs even with kill-switch active.
- CrossTenantAccessTests +3 inline DELETE cases.
- RegisterSessionGuardWiringTests (NEW): asserts guard DI field,
ProcessRegisterAsync internal accessibility, BootstrapRegisterSession
delegation type.
- CascadeVerificationServiceTests.FakeReader gains the new
GetActiveSessionTombstoneAsync interface member.
Suites: Backend xUnit 1524, Agent V1 530, Agent V2 1042 — all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
921994e845 |
fix(backend): cascade-delete codex follow-ups (F1 V2 guard + upsert-revival, F2 partial-restore key-level re-increment, F3 legacy delete sessions-first, F4 restore 400 EntityAlreadyExists)
F1 — V2 ingest writes were landing past the cascade lock. Wired SessionDeletionGuard.EnsureWritableAsync into IngestTelemetryFunction before PersistItemsAsync so Event / Signal / Transition writes are refused with 410 Gone when DeletionState is Preparing / Queued / Running / Poisoned. Switched UpdateSessionImeAgentVersionAsync from UpsertEntityAsync to UpdateEntityAsync(ETag.All, Merge) with 404 swallow so a tombstoned Sessions row can no longer be silently resurrected by an in-flight ingest. F2 — Partial-restore could leave SoftwareInventory at -N drift when the cascade poisoned mid-AGGREGATE-loop: per-key decrement was persisted into AggregateDecrementsApplied BEFORE the underlying decrement, but the gate used CompletedSteps.Contains(aggregate.Order) — which is only set AFTER the whole loop. Switched the gate to AggregateDecrementsApplied as the authoritative ledger and pass a filtered DeletionStep into RunInventoryReIncrementWithIdempotencyAsync so only the keys that were actually decremented are re-incremented. F3 — Legacy direct-delete path deleted Events / RuleResults / AppInstallSummaries before the ETag-CAS-guarded DeleteSessionAsync, leaving a TOCTOU window where a V2 producer could CAS-claim DeletionState=Preparing after the pre-read but before our side- table mutations — landing the producer's manifest snapshot on top of already-half-purged state. Reordered both admin (DeleteSessionFunction) and retention (SessionRetentionFanoutService) paths to Sessions-CAS first; side-tables only after the Sessions row is gone (no producer can claim a missing row). F4 — Restore per-row Add fallback only caught HTTP 409 for duplicate- row conflicts; Azurite and some legacy Azure deployments surface duplicate-row as 400 with ErrorCode=EntityAlreadyExists, breaking idempotent restore-retry after partial-failed restore. Centralised the check via IsAlreadyExistsStatus(rfe) helper accepting 409 OR 400 + ErrorCode=EntityAlreadyExists; non-duplicate 400s still propagate. Tests: - SessionRestoreServiceTests: 2 partial-restore tests refit to the new AggregateDecrementsApplied contract + new explicit Codex-scenario test (only-decremented-keys re-incremented, untouched key skipped). - SessionDeletionPr4cCodexFixesTests F4: harness setup adds full AggregateDecrementsApplied so the per-key idempotency assertion reflects the new gate. - SessionRetentionFanoutServiceTests: pre-read tests replaced by two tests covering "DeleteSessionAsync refuses → no side tables" and "DeleteSessionAsync succeeds → side tables run after". - TableStorageRestoreBatchTests: two new F4 tests — 400 EntityAlreadyExists treated as Skipped, 400 with non-duplicate ErrorCode still propagates. - UpdateSessionImeAgentVersionTests (NEW): asserts UpdateEntityAsync is used (not Upsert), 404 swallowed silently, other storage errors non-fatal. Suites: Backend xUnit 1510, Agent V1 530, Agent V2 1042 — all green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
4c676d076a |
fix(backend,web): PR6 codex-review follow-ups (F1 legacy pre-check, F2 mid-run kill-switch, F3 OpsEvent audit shift)
Three Round-1 review findings from PR6:
F1 (High) — Legacy retention path mutated side-tables before the
CAS-guarded DeleteSessionAsync. Plan PR6 step 7 says "legacy must skip
locked sessions", but our LegacyDeleteAsync deleted Events / RuleResults
/ AppInstallSummaries first and only discovered the V2 cascade lock at
the tombstone. Fix: pre-read the Sessions row in LegacyDeleteAsync; if
DeletionState != None, return false WITHOUT touching any side table.
The original CAS-guard stays as a backstop for the T1-T3 race after the
pre-read.
F2 (High) — Kill-switch checked once per run; a flip-ON mid-fanout kept
legacy deletes running. SessionRetentionFanoutService now takes
AdminConfigurationService, calls IsSessionDeletionKillSwitchActiveAsync
(uncached, fail-closed per PR5 F1) at both the per-tenant boundary and
inside the per-session loop. FanoutResult gains AbortedByKillSwitch so
the per-tenant audit + the global OpsEvent surface the early halt.
F3 (Medium) — Global-scope lifecycle audits used LogAuditEntryAsync with
tenantId: null!, which the table-storage helper silently dropped because
the schema requires a non-null PartitionKey. Three event types
(Completed / FanoutSkipped / Failed) moved to OpsEvents (TenantId is
nullable on OpsEventEntry). Two new helpers in OpsEventService —
RecordSessionDeletionMaintenance{Completed,FanoutSkipped}Async — also
dual-registered in the OpsAlertRulesSection.tsx catalog per memory
feedback_ops_event_types_dual_register. Per-tenant audits inside the
fanout (which already had a real tenantId) are unchanged.
Tests: 4 new xUnit cases — F1 skip-without-mutation, F1 row-not-found
skip; F2 mid-tenant abort, F2 between-tenants abort. Existing 17 PR6
tests updated to assert OpsEvents instead of audit calls for the global
lifecycle events. Total: 1504/1504 backend pass, 347/347 web pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
50071c5e42 |
feat(backend,web): dedicated cascade-delete maintenance function (PR6/7)
Plan §5 PR6 / §16 R14: extracts cascade-lifecycle maintenance out of the
2h Maintenance timer into a dedicated 12h SessionDeletionMaintenanceFunction
with independent cadence, kill-switch, and OpsEvent watchdogs. Adds four
work-blocks (Plan §10):
1. Manifest-blob TTL sweep — defence-in-depth on top of Azure Blob
Lifecycle Management. EnumerateOldDeletionManifestsAsync +
DeleteDeletionManifestPairAsync (33-day effective window via the
existing soft-delete tombstone, not permanent deletes).
2. Stale-Preparing GC — reverts DeletionState=Preparing rows older than
1h that have NO progress blob (producer crashed before upload). Skips
when a progress blob exists; producer's own Preparing-resume path
handles those.
3. Stranded-Queued detection — alert-only OpsEvent per row in
DeletionState=Queued for > 30min (no auto-clear; operator inspects).
4. Retention fanout via new SessionRetentionFanoutService — per-tenant
DataRetentionDays sweep, dispatched per EnableCascadeDeleteV2 flag:
V2 → producer.EnqueueAsync, legacy → ISessionRepository.DeleteSessionAsync
(now CAS-guarded since PR5 finding 2). Rate-limited to 100 enqueues
per tenant per run with 50ms throttle.
Operational alerting:
- 30min watchdog → OpsEventType SessionDeletionMaintenanceLongRunning (Warning)
- 60min watchdog → SessionDeletionMaintenanceLongRunningSevere (Error)
- Exception path → SessionDeletionMaintenanceFailed (Error) + re-throw
- Per stranded row → SessionDeletionStrandedQueued (Warning)
All four registered both in OpsEventService.cs and the OpsAlertRulesSection.tsx
catalog so admin can wire Telegram per memory feedback_ops_event_types_dual_register.
Reuses PR5 finding 1's IsSessionDeletionKillSwitchActiveAsync so the
fanout-skip check honors flip-ON across scaled-out instances within seconds.
host.json: functionTimeout=01:00:00 (Flex Consumption max, matches the
60min severe watchdog so operator gets alerted before Azure hard-aborts).
Cron: const "0 0 */12 * * *" — matches the repo's IndexReconcileTimer
convention (no AdminConfiguration field; redeploy to change).
Removed: session retention loop from MaintenanceService.CleanupOldDataAsync
(both RunAllAsync and RunManualAsync paths now call CleanupOldUsageDataAsync
directly). Non-session housekeeping (rule-stats, cert-expiry, SignalR
quota, poison-queue watcher) untouched.
EnableCascadeDeleteV2 + SessionDeletionKillSwitch defaults stay false —
PR6 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation.
Backend xUnit: 1500/1500 pass (1483 + 17 new). Web vitest: 347/347 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
8134adf3e4 |
feat(backend,web): wire admin-delete + worker + UI for V2 cascade (PR5/7)
Plan §5 PR5 (session-deletion-cascade-safe.md): flag-gated dispatch in
DeleteSessionFunction between the legacy direct-delete path and the V2
cascade producer; SessionDeletionWorker registered as a hosted service;
UI shows a "deletion queued" toast + per-row spinner while the worker
drains, then removes the row on the SignalR sessionDeleted event.
Includes the codex review follow-ups:
F1 — IsSessionDeletionKillSwitchActiveAsync bypasses the 5-min admin
config cache; fail-CLOSED on storage error so a flip-ON is honored
across scaled-out Function-host instances within seconds (instead
of minutes). Wired into DeleteSessionFunction, Producer, Worker.
F2 — DeleteSessionAsync now reads DeletionState + ETag together with
IndexRowKey, refuses on locked state, and ETag-conditionals the
delete; closes the T2-T4 race where a concurrent V2 producer
could orphan side-table rows. Caller in the legacy path
disambiguates the bool via a re-read into 404 / 409 / 500.
F3 — pendingDeletions threaded from useDeleteSession through page.tsx
→ SessionTable → SessionCell; action cell renders a disabled
spinner instead of the delete button. Hook subscribes to
sessionDeleted SignalR + polls GET /api/sessions/{id} every 60s
as a fallback for missed events; only 404 clears the pending row.
EnableCascadeDeleteV2 + SessionDeletionKillSwitch both default false —
PR5 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation. PR6 wires the producer into the dedicated
SessionDeletionMaintenanceFunction next.
Backend xUnit: 1483/1483 pass. Web vitest: 347/347 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
5dbc44c4ec |
feat(backend): cascade-delete worker + restore + codex-fixes (PR4/7)
PR4 ships the cascade-delete executor for the safe-cascade-delete plan: the
SessionDeletionWorker (BackgroundService queue poll-loop) plus the
SessionDeletionHandler (per-step execution, live verification, tombstone).
Worker is NOT registered in Program.cs — that lights up in PR5.
Three logical phases land together as one commit per Oliver's call:
PR4 — Worker + Handler + CascadeVerificationService
- SessionDeletionWorker: BatchSize=1, kill-switch entry guard, heartbeat task
extending visibility every 60s during long handler runs, poison after
max-dequeue
- SessionDeletionHandler: download manifest+progress, CAS Queued→Running
(with Running→Running resume), per-step dispatch by Class, AGGREGATE
decrement, live verification, ordered tombstone (SessionsIndex first,
Sessions last)
- CascadeVerificationService: per-Class live residual scans (CveIndex is
manifest-key-only per §12-Q8; AGGREGATE+FINAL skipped)
- NotifySessionDeletedAsync added to ISignalRNotificationService
- BlobStorageService: DownloadDeletionProgressAsync + ETag-CAS
UpdateDeletionProgressAsync
PR4b — Restore endpoint
- POST /api/admin/sessions/{sessionId}/restore (GA-only via Policy Catalog)
- SessionRestoreService: dispatch table per §13.4, full + partial-poisoned-
recovery modes
- TableStorageService.Restore.cs: RestoreRowsByExactKeysInBatchesAsync +
ConvertFromPropValue + ConvertDumpToEntity (Sessions-row override forces
DeletionState=None on restore so the row isn't permanently locked)
- RestoreSoftwareInventoryEntryByKeyAsync helper
- Worker MoveToPoisonAsync now CAS-transitions Running→Poisoned (E1 fix)
so restore can dispatch into partial-restore mode
PR4c — Codex review fixes (6 findings)
- F1: AGGREGATE per-key idempotency via DeletionProgress.AggregateDecrementsApplied
(persist-first/decrement-second bounds drift to +1 per crash vs unbounded -N)
- F2: TombstoneStarted flag closes the gap where sessions=null+completedAt=null
blocked restore after a tombstone crash
- F3a: Sessions row inserted LAST in full restore (was first → trapped retry
with RejectAlreadyAtOriginalState)
- F3b: RestoreMode.Full now 409-ignores (was throw-on-409); converges with
Partial mode, makes retry-after-partial-failure idempotent
- F4: Partial-restore per-key re-increment idempotency via
DeletionProgress.RestoreReIncrementsApplied
- F5: Poison transition pre-checks PendingDeletionManifestId so stale
envelopes can't flip a fresh active cascade to Poisoned
- F6: DownloadDeletionManifestWithShaAsync exposes snapshot SHA; Handler +
Restore both verify against progress.SnapshotSha256 (snapshot↔progress
binding per plan §3)
DeletionProgress schema gains 3 additive optional fields (AggregateDecrementsApplied,
RestoreReIncrementsApplied, TombstoneStarted) with safe defaults so PR1-PR4
progress blobs deserialize cleanly.
Tests: 1457/1457 backend xUnit facts green (1391 pre-PR4 + 66 new across PR4,
PR4b, PR4c). Full solution build green. Program.cs has zero
SessionDeletionWorker references (PR5 territory stays untouched).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
522f0ae048 |
fix(backend): PR3 codex round-2 follow-ups (F2 ETag-CAS, F4 Preparing-resume)
Round 2 of codex feedback flagged that the first follow-up wasn't tight
enough:
== F2 round 2 (HIGH): TOCTOU race still possible on Sessions row ==
The first fix only preserved DeletionState if it was non-empty on the
INITIAL read. The race remained: T1 StoreSessionAsync reads row →
DeletionState=None; T2 producer CAS-es None → Preparing; T3
StoreSessionAsync writes Replace → silently clobbers the lock. Codex
specifically asked for ETag/Retry oder Merge.
Fix: replace the unconditional UpsertEntityAsync with an ETag-bound CAS
write loop (bounded retry = 5 attempts). Each iteration re-reads the
CURRENT row to capture (a) the fresh ETag and (b) the live cascade-lock
columns, stamps the cascade-lock columns from that FRESH read onto our
entity (so the producer's intervening write isn't clobbered), then tries
UpdateEntity(etag, Replace). On 412/409 conflict: retry. Falls back to
AddEntity when the row was deleted between reads (tombstone race). The
preserved-fields extract from the initial read still runs as a
best-effort hint; the per-iteration fresh read is what actually wins
the race.
Replace mode is kept (not Merge) so registration can still clear stale
fields like FailureReason on a successful re-register. Only the
cascade-lock columns get the cross-iteration preservation treatment.
== F4 round 2 (MEDIUM): stranded Preparing with progress blob ==
If the prior producer call CAS'd None → Preparing AND uploaded snapshot
+ progress blobs AND audited deletion_started, then crashed before the
CAS Preparing → Queued step — the row stays in Preparing forever. Plan
§10 GC only clears Preparing-WITHOUT-progress-blob (intentional, to
avoid racing in-flight producers); the resume path I added before only
covered the Queued case. So this would sit until manual intervention.
Fix: in the WrongState=Preparing branch with non-empty
PendingDeletionManifestId, probe the snapshot blob via the new
BlobStorageService.DeletionSnapshotExistsAsync helper. If it exists,
the prior crash happened POST-upload — safe to finish the CAS
Preparing → Queued + send the queue message (resume from step 6). If
absent, the prior crash was PRE-upload — bail out as AlreadyInFlight
and let the §10 GC clean up after 1h.
Probe failure (e.g. transient blob storage outage) also falls back to
AlreadyInFlight rather than risk a half-completed recovery. Extracted
the queue-message-resume call into a shared helper
(ResumeBySendingQueueMessageAsync) so both Queued-resume and
Preparing-resume go through one audit-marker code path
(Reason="resume").
== Tests ==
- SessionDeletionPr3CodexFixesRound2Tests (4 new):
- Stranded Preparing-with-blob → resumes (CAS Preparing→Queued runs,
queue message sent, returns Enqueued + Reason=resume)
- Preparing without blob → AlreadyInFlight, no CAS attempt
- Snapshot existence probe throws → AlreadyInFlight, no CAS attempt
- Resume CAS loses to concurrent writer → AlreadyInFlight, no queue
message
- Backend suite: 1349/1349 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
dc56c8a73a |
fix(backend): PR3 codex-review follow-ups (F2 lock-clobber, F3 SchemaHash, F4 resume, F5 preview shape)
Round-2 codex review on PR3 surfaced four real issues. Finding 1 (the
guard not being wired at the 9 writer sites) was the deferred-by-design
decision and stays scheduled for PR5.
== F2 (HIGH): StoreSessionAsync silently clobbered the cascade lock ==
Re-registration reads the existing Sessions row but built a fresh
TableEntity for the UpsertEntity (Replace) without carrying over
DeletionState + PendingDeletionManifestId. An agent restart during
Preparing/Queued/Running would have cleared the lock and let the
agent's subsequent writes race the in-flight cascade. Fix: extract
both columns alongside the other preserved fields and re-stamp on
the new entity before the Upsert. Comment marks the silent-clobber
risk so a future refactor can't quietly remove the preservation.
== F3 (HIGH): Producer invalidated the manifest SchemaHash ==
Builder generated a ManifestId, hashed the manifest content, then the
producer overwrote manifest.ManifestId. SchemaHash was now stale
(computed against the old ID); a future hash-verification path on the
worker would treat legitimate manifests as tampered/corrupt. Fix:
DeletionManifestBuilder.BuildAsync accepts an optional
preAllocatedManifestId so the producer's CAS-stamped ID lands on the
manifest BEFORE the hash computation. Producer no longer overwrites.
Preview path passes null (transient ID).
== F4 (MEDIUM): No resume path for stranded Queued state ==
If CAS Preparing→Queued succeeded but SendMessageAsync failed
(network blip, queue throttling), the Sessions row sat in Queued state
with no live queue message. A retry just returned AlreadyInFlight and
left the cascade stranded. Fix: producer's WrongState=Queued branch
re-sends the queue envelope with the EXISTING manifestId (stamped
with reason ":resume" for audit traceability), returns Enqueued.
Idempotent — duplicate envelopes are tolerated by PR4's worker via
the manifest-bound progress blob. Preparing + Running unchanged
(let GC handle Preparing; trust the worker for Running).
== F5 (MEDIUM): mode=full preview wrapped manifest in an envelope ==
Plan §5 PR3 promises mode=full returns "the entire manifest JSON
snapshot inline ... identical structure to what would be uploaded
to deletion-manifests blob container" — but the response wrapped the
manifest in { success, mode, manifest } so byte-comparison against
a real cascade's snapshot was impossible. Fix: mode=full now returns
the RAW manifest JSON as the body. Side-channel data (in-flight hint,
builder duration, schema hash, manifest id) moves to response headers
(X-Deletion-Preview-*). Only mode=summary still uses the envelope
shape (it's a summary, not a snapshot).
== Tests ==
- SessionDeletionPr3CodexFixesTests (5 new): pin F3 hash consistency
with + without preAllocatedManifestId, F4 resume re-sends queue
message with existing ManifestId, F4 Running + Preparing do NOT
re-send.
- Updated 1 existing producer test: AlreadyInFlight assertion now
uses Running state (Queued is the resume path post-F4).
- Backend suite: 1345/1345 green.
== Finding 1 (guard wiring) ==
Confirmed deferred to PR5 per the original PR3 scope decision. PR5
wires the producer call into DeleteSessionFunction AND the 9 guard
sites in the same PR. Cascade can't fire until then, so the guard
being dormant in PR3 has zero runtime impact.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
257c5c4354 |
feat(backend,web): cascade-delete state-machine + guard + producer + preview (PR3/7)
PR3 of the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). Substantial PR per
plan §16 — adds the concurrency state-machine, the single-chokepoint
SessionDeletionGuard, the producer that builds manifest + uploads
blobs + enqueues the worker envelope, and the read-only preview
endpoint Global Admins use to verify a cascade plan before flipping
EnableCascadeDeleteV2 on. Wiring the guard at the 9 writer sites is
intentionally deferred to PR5 — no cascade actually fires until PR5
calls the producer from DeleteSessionFunction, so PR3 ships as
infrastructure-only.
== State machine + guard infrastructure ==
SessionDeletionState (Shared, Plan §1 P7 / §16-R2):
None | Preparing | Queued | Running | Poisoned. Completed deliberately
omitted — the cascade FINAL step removes the row, so "completed" is
observable via row-absence + progress.completedAt, not via a stable
state value. IsValidTransition(from, to) enforces the legal edges
(None→Preparing, Preparing→{Queued,None,Poisoned}, Queued→{Running,
Poisoned}, Running→Poisoned, Poisoned→None) and treats null/empty
legacy rows as None.
SessionDeletionLockedException carries (TenantId, SessionId,
CallerContext, CurrentState, ManifestId) for audit + log lines.
SessionDeletionGuard (Plan §1 P7 + §16-R13):
- ThrowIfLocked(TableEntity sessionRow, callerContext) — piggyback
variant for hot paths that already loaded the Sessions row; zero
added I/O. Caller must include DeletionState +
PendingDeletionManifestId in its select projection.
- EnsureWritableAsync(tenantId, sessionId, callerContext) —
standalone variant for paths with no existing read; one Get
round-trip via the PR1 ISessionDeletionInventoryReader seam.
Both paths throw the same exception. Per §16-R13 framing the guard
is "best-effort fast-fail, not safety" — correctness lives in §1 P4
live verification + poison + §13 restore-from-poisoned (PR4).
== Sessions row state-machine columns ==
SessionSummary.DeletionState (string) +
SessionSummary.PendingDeletionManifestId (string?) — added to the DTO
+ MapToSessionSummary read path. Empty/null means no cascade in flight
(legacy rows; treated as None per the guard contract). The producer
writes both columns via direct CAS via the new helper; existing
session-write paths don't need to know about them in PR3.
TableStorageService.CasSetSessionDeletionStateAsync — new partial
helper. Read-then-CAS on Sessions row, returns a structured
SessionDeletionStateCasResult (Updated | SessionMissing | WrongState |
EtagConflict) so the producer can branch correctly. Stamps newManifestId
on transition to Preparing; clears it on transition back to None
(poisoned-recovery restore).
== Producer + preview endpoint ==
SessionDeletionProducer.EnqueueAsync(tenantId, sessionId, reason,
actor) — Plan §5 PR3 step sequence: kill-switch check → CAS None→
Preparing with new ManifestId → DeletionManifestBuilder.BuildAsync →
BlobStorageService.UploadDeletionManifestAsync → UploadInitial-
DeletionProgressAsync (new helper for the mutable .progress.json
sibling) → audit deletion_started → CAS Preparing→Queued → enqueue
SessionDeletionEnvelope to Constants.QueueNames.SessionDeletion.
Returns SessionDeletionEnqueueOutcome enum so PR5/PR6 callers
translate per their UX (admin → 409 on AlreadyInFlight, maintenance
→ skip+audit). Crash recovery contract: row stuck in Preparing with
no progress blob → maintenance GC clears after 1h (PR6); Queued/
Running/Poisoned NEVER auto-cleared.
GET /api/admin/sessions/{sessionId}/delete/preview?mode={summary|full|
download} — GlobalAdminOnly per the EndpointAccessPolicyCatalog (per
memory feedback_route_policy_catalog). Three modes:
- summary: preflightCounts + sample (PK, RK) per step + estimated
snapshot bytes + builderDurationMs + schemaHash.
- full: entire manifest JSON inline (every row dump, every decrement
key) — byte-level review of "what would be deleted".
- download: gzipped *.snapshot.json.gz with Content-Disposition
attachment, identical bytes to what the real cascade would upload.
NEVER mutates state, NEVER uploads a blob, NEVER enqueues. Calls the
guard only as a hint (in-flight cascades surface in the response, no
lock acquired). Audits deletion_preview per Plan §7.
== Configuration changes (with web round-trip) ==
AdminConfiguration.SessionDeletionKillSwitch (bool, default false,
Plan §1 P8 / §9) — global emergency stop. When true, V2 producer +
legacy direct-delete return 503; PR4 worker pauses on entry. Round-
tripped through TableConfigRepository Store + Map AND the 4-file web
chain (AdminConfigContext.tsx, AdminConfigSettingsSection.tsx,
SectionGlobalSettings.tsx, types/adminConfig.ts) per memory
feedback_admin_config_ui_roundtrip — without it, every admin Save
silently clobbers the field back to false. UI surfaces a red checkbox
with explanatory copy.
TenantConfiguration.EnableCascadeDeleteV2 (bool, default false) —
per-tenant rollout flag. When true, PR5 admin-delete + PR6 retention
fanout build a manifest + enqueue; when false, legacy direct-delete
runs. Per-tenant flag selects PATH; global kill-switch halts BOTH.
Constants.QueueNames.SessionDeletion = "session-deletion" — worker
queue name.
Shared envelope SessionDeletionEnvelope { TenantId, SessionId,
ManifestId, Reason, EnqueuedAt } — wire shape between producer and
PR4 worker.
== DI ==
Program.cs registers SessionDeletionGuard + SessionDeletionProducer
as singletons next to the existing DeletionManifestBuilder.
== Tests (50 new, backend suite 1340/1340 green) ==
- SessionDeletionStateTests (10) — IsLocked over every state value
including legacy null/empty, IsValidTransition over every legal
edge, explicit rejection of illegal edges, "Completed is not a
state" guarantee.
- SessionDeletionGuardTests (8) — null row no-op, every lock state
throws with correct exception fields, missing-session no-op,
context propagation in the exception.
- SessionDeletionProducerTests (7) — kill-switch returns
KillSwitchActive without downstream calls; SessionMissing /
AlreadyInFlight / Poisoned outcomes from CAS; happy path verifies
all 7 steps in order via Mock.Verify; CAS-2 failure leaves snapshot
uploaded but no queue message; producer's pre-allocated ManifestId
flows to both the snapshot blob and the CAS marker (consistency
invariant the worker depends on).
- AdminConfigurationSessionDeletionKillSwitchTests (6) — both flags
default false on new config + on CreateDefault, true persists.
Wiring 9 sites with the guard (telemetry V1+V2 ingest, mark-success/
failed, server actions, maintenance marker, re-analyze handler,
vulnerability handler, manual rescan, index reconcile) deferred to
PR5 — no cascade fires in PR3, so the guard is dormant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
66fc23b609 |
feat(backend): SoftwareInventory contributions side-row + cascade delete helpers (PR2/7)
PR2 of the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). Adds the per-table
delete primitives the cascade worker will use in PR4 plus the §17
SessionInventoryContributions side-row that makes per-session inventory
counter increments at-most-once and exactly reversible. No production
caller invokes the cascade primitives yet; the inventory rewrite IS a
call-site change in VulnerabilityCorrelationService and ships in this PR.
== Cascade delete helpers (TableStorageService.Deletion.cs) ==
DeleteByExactKeysInBatchesAsync(tableName, keys) is the load-bearing
helper: groups by PartitionKey (Azure batch transactions are
partition-scoped), chunks at 100 actions per batch, falls back to
per-row DeleteEntityAsync with 404-ignore on whole-batch rollback.
Returns (Attempted, DeletedNow, AlreadyMissing) for accurate progress
accounting on idempotent re-runs of partial cascades. Plus 9 thin
wrappers per the §3 taxonomy: 7 keyed (EventTypeIndex, CveIndex,
SessionsByTerminal, SessionsByStage, DeadEndsByReason,
ClassifierVerdictsByIdLevel, SignalsByKind), 1 single-row
(DeviceSnapshot), and 2 enumerate-then-delete (Signals,
DecisionTransitions for the PK_BY_SESSION class). All fail-loud per
memory feedback_storage_helpers_fail_soft.
== SoftwareInventory side-row (TableStorageService.Inventory.cs) ==
Plan §17 design: each inventory-correlated session contributes ONE
SessionInventoryContributions row (PK=tenantId, RK=sessionId) with the
normalized software keys it added. Increments to SoftwareInventory
counters are now driven by deltas against this row, so:
- WG sessions that fire software_inventory_analysis 4× no longer
over-count by 4× per software per session
- Cascade decrement is exact: the side-row is the source of truth
for which counters this session contributed to
- Pre-side-row sessions (created before this deploy) have no
contributions row → cascade skips inventory entirely for them
(drift accepted on the historical backlog, ages out via 90d
retention).
Helpers (all fail-loud, all bounded retry per §12-Q10 = 10 attempts /
60s wall-clock cap, all ETag-CAS):
- GetSessionInventoryContributionsAsync: returns prior key list or
null on 404
- UpsertSessionInventoryContributionsAsync: side-row CAS that
computes (added, removed) delta against the new key set
- DeleteSessionInventoryContributionsAsync: 404-ignore tombstone
- IncrementSoftwareInventoryEntryAsync: ETag-CAS +1 with first-time
AddEntity that writes full metadata
- DecrementSoftwareInventoryEntryAsync: ETag-CAS -1, clamp >= 0,
deletes the row when count hits zero
Codec: SoftwareKeysJsonCodec encodes the key list as raw JSON below
30 KB, gzip+Base64 above. Same shape DeletionManifestBuilder (PR1)
already reads at preflight, so the cascade decrement reads back what
the increment wrote with no schema gap. Plus deterministic ComputeDelta
(case-insensitive composite key) so concurrent correlations on the
same session collapse to a single counter update.
== VulnerabilityCorrelationService rewrite ==
The Task.Run fire-and-forget at the old line 246 now calls the new
ApplyInventoryContributionsAsync helper instead of
UpsertSoftwareInventoryAsync. Each Increment / Decrement is wrapped
in its OWN try/catch so one counter failure does NOT abort the rest
of the loop (Plan §17.3 + §17.8): without that, the side-row would
already promise the new key set while subsequent runs see an empty
delta and never retry the under-counted keys. Each individual failure
emits an `inventory_counter_write_failed` audit entry with operation
+ vendor + name + version + exception type/message via the existing
LogAuditEntryAsync. IMaintenanceRepository was added as an OPTIONAL
constructor param (positional default null) so existing 8-arg test
factories keep compiling unchanged.
ParseSoftwareInventoryItems exposed as internal static for testability;
preserves the same dedup + cpe-mapping behaviour as the old
UpsertSoftwareInventoryAsync path. The legacy method is left in place
on TableStorageService.Rules.cs for now (no remaining callers; PR7
cleanup can remove if desired).
== Tests (39 new) ==
- SoftwareKeysJsonCodecTests (12): raw vs gzip+Base64 paths,
determinism, malformed-input rejection, delta cases (added /
removed / identity / case-insensitive), composite-key shape.
- TableStorageDeletionBatchTests (8, SDK-mocked via internal test
ctor): empty input, PK grouping (one batch per PK), chunking at
100, batch 404 + all per-row 404, mixed batch 404 (2 succeed +
3 missing) with accurate accounting, non-404 propagates, no
over-deletion, ETag.All on every action.
- TableStorageInventoryCasTests (9): Increment first-create with
full metadata, CAS +1 on existing row, 412 retry, 409 first-create
race fall back to Update, empty-triple skip; Decrement 404 no-op,
clamp-at-zero deletes the row, count>1 subtract, 0-count clamp,
412 retry.
- VulnerabilityCorrelationServiceContributionsTests (10): Parser
correctness, end-to-end add/remove flow with Mock.Verify counts,
empty-input no-op, side-row exhaustion swallowed, per-counter
failure does not abort + audits the failed key (both increment
and decrement variants).
Backend suite: 1290/1290 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
c8658b9feb |
feat(backend): scaffold cascade-deletion manifest format + builder (PR1/7)
Read-only foundation for the safe cascade-delete plan (.claude/plans/session-deletion-cascade-safe.md). PR1 ships the manifest shape, the read-only DeletionManifestBuilder enumerator, and the gzip+SHA-256 blob upload/download plumbing. No production caller invokes any of this yet; no existing endpoint behaviour changes. PR2-PR7 wire the producer, worker, and call sites on top. Manifest captures all 15 cascade tables per the §3 taxonomy (4 PK_BY_SESSION, 1 PROP_TENANT_PK, 3 PK_RK_EXACT, 1 DISCRIMINATOR_PK_RK_SUFFIX, 1 DISCRIMINATOR_PK_RK_EXACT, 5 DISCRIMINATOR_PK_PROP), the optional SoftwareInventory side-row (omitted for pre-side-row sessions per §17.5), and the FINAL tombstone step ordered SessionsIndex-then-Sessions. Row dumps tag every column with its EDM type (String / Int32 / Int64 / Double / Boolean / DateTime / Guid / Binary) so DateTime / Guid / byte[] / long round-trip restore-faithfully through the JSON snapshot. Falls back to a SessionsIndex partition scan when Sessions.IndexRowKey is empty or stale, so a partially-corrupted Sessions row can't strand a SessionsIndex orphan after cascade. BlobStorageService.UploadDeletionManifestAsync compresses with gzip, pins SHA-256 of the uncompressed payload to blob metadata, and uses IfNoneMatch=* to enforce snapshot-write-once. DownloadDeletionManifest re-verifies the hash and refuses on mismatch (corruption signal that the §13 restore semantics rely on). Two protected-internal-virtual seams isolate the Azure SDK calls so tests bypass BlobsModelFactory entirely. New SessionInventoryContributions table constant added to TableNames.All so the builder can Try-Read it (404 = pre-side-row session = inventory steps omitted). PR2 adds the side-row write path via VulnerabilityCorrelationService. Tests: 25 new xUnit cases (DeletionManifestSerializationTests, DeletionManifestBuilderTests, BlobStorageServiceDeletionManifestTests) cover counts, dumps with EdmType per Azure Tables type, gzip round-trip, hash-mismatch + missing-metadata corruption signals, IfNoneMatch overwrite refusal, SessionsIndex fallback (both empty-key and stale-key paths), determinism across runs, and pre-side-row inventory skip. Backend suite: 1251/1251 green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
7020a2e5a6 |
fix(backend): propagate device/version block+kill across Function App instances
BlockedDeviceService and BlockedVersionService kept their cache per Function App
instance. Once an instance had lazy-loaded a tenant's block list, a cache miss
short-circuited to "not blocked" without ever re-consulting storage — so a Kill
(or Block) added on instance A stayed invisible to instance B for the lifetime
of the process. Live repro: device was killed at 13:11:46Z and kept posting
telemetry for 11+ minutes afterwards on every cadence tick.
BlockedDeviceService.IsBlockedAsync now has two safety nets to bridge other
instances to storage:
- Cache miss after tenant was loaded -> storage point-read on the spot, then
promote the result into the cache. Lets a freshly-set block from another
instance reach this instance immediately.
- Positive cache hit older than EntryRevalidateAfter (30s) -> storage
point-read to re-confirm. Lets manual Unblock, Action upgrade (Block->Kill),
and UnblockAt changes from another instance propagate within seconds.
BlockedVersionService CacheRefreshInterval lowered from 5min to 30s for the same
reason — a version-Kill should not take up to five minutes per instance.
IDeviceSecurityRepository.IsDeviceBlockedAsync gains a BlockedSessionIds tuple
slot so the cache-miss fallback preserves session-aware semantics instead of
silently degrading session-scoped blocks to whole-device.
Tests cover the cache-miss fallback for both Block and Kill actions, the
stale-positive Block->Kill upgrade path, and the stale-positive Unblock path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
9826b0a321 |
fix(agent/v2,backend,web): flag in-flight installs as "likely stuck" on terminal ESP-Apps failure
When the ESP times out on the Apps subcategory, V2 used to terminate the session with the last in-flight app silently frozen in the Installing bucket; the user saw a Failed enrollment but no app name and no app_install_failed event. The agent has no way to tell whether the app eventually succeeded, failed, or hung — so promote it as hedged "likely stuck" with confidence "presumed" rather than a confirmed failure. The promotion fires only on the explicit EspTerminalFailure pathway: a new LastFailureTrigger fact on DecisionState lets EnrollmentTerminationHandler discriminate against MaxLifetimeExceeded (watchdog notbremse), SessionAborted (admin-kill) and EffectInfrastructureFailure (effect-runner gave up) — none of those carry a meaningful claim about app outcome. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
9a45690062 |
fix(web/scripts): default-collapse remediation cycle cards so long lists stay scannable
User reported the all-expanded default produced very long scrolly lists when a session has many remediation policies — the header (Status: Compliant / Remediated successfully / Remediation script failed) is the at-a-glance signal they actually want first; per-phase details are a drill-in. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
7e67a70756 |
fix(web/scripts): group remediation cycle phases under one parent card (Intune-cycle UX)
Confirmed via real non-compliant cycle data from session 8810cf81 / policy 99e1274b: detection (non-compliant) + remediation (script crashed exit 1) + post-detection (still non-compliant) — three separate sibling cards for the same Remediation policy was visually fragmenting and obscured that they all belong to one cycle. New grouping: - groupScriptItems() folds phases for the same (policyId, scriptType=remediation) into a single ScriptCard with phases[] sorted detection → remediation → post-detection. Platform scripts and detect-only / running remediations stay as single-phase cards. - deriveRemediationHeader() computes the cycle-level outcome label + state with priority: Running → Failed (e.g. remediation script crash) → Non-compliant after remediation → Remediated successfully → Non-compliant (detect-only) → Compliant (detect-only). - ScriptCardView wraps multi-phase cycles in a parent header (icon + "Remediation" + shortId + cycle badge + System/Target/Status meta + collapsible chevron) with the individual phase rows nested inside on a lighter inner background. Single-phase cards reuse the existing ScriptItemRow layout 1:1 (no parent wrapper) so the detect-only majority case stays unchanged. - Header counter "succeeded / non-compliant / failed" now counts CARDS not phases, so a remediated cycle reads as "1 succeeded" rather than "1 succeeded + 2 non-compliant" from its detection / post-detection en route. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
76284cfcb2 |
fix(agent/v2,rules): two-stage health-script emission to close short-session timing gap
The post-PR-HS1 single-source design (HS-NEW-RESULT JSON only) had a real timing gap: IME emits the [HS] new result = JSON line only AFTER its batch send to the Microsoft service, ~30-90 s after the actual script ran (verified in session c1e714e6: 34-74 s gap per policy). Short Autopilot enrollments often end inside that window, so health- script rows were entirely missing from completed sessions even though IME had executed the scripts. Two-stage approach (matches original V1 timing for the early signal, while keeping HS-NEW-RESULT as the canonical truth): - Re-add HS-COMPLIANCE pattern matching BOTH `pre-remdiation` AND `post-remdiation` detection compliance lines → HandleHealthScriptDetectionResult emits a script_completed event immediately. Single regex (`(?<part>pre|post)-remdiation`) covers both phases. - Re-add HS-RUN-CONTEXT / HS-EXITCODE / HS-STDOUT / HS-STDERR patterns feeding a single-slot `_pendingHealthScript` accumulator (IME runs health scripts sequentially within a session — verified). HS-COMPLIANCE merges the slot data onto the early-signal event so exit / stdout / stderr / runContext are populated from the start, not just the inferred compliance verdict. - HS-NEW-RESULT continues to fire later with the full JSON (RemediationStatus, TargetType, ErrorCode, plus the actual remediation phase's data). UI reducer's dataCompleteness() scoring keeps the more complete entry — early-signal row gets enriched in place, no duplicate cards. Remediation phase itself has no early-signal source in IME logs (only the late JSON exposes Info.RemediationExitCode and RemediationScriptOutputDetails). A follow-up based on the agentexecutor invocation line's _RemediationScript.output suffix is possible once we have a real non-compliant cycle log to validate against. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
33023ecfc8 |
fix(agent/v2,web/scripts): UX polish + dedupe duplicates + stderr-as-failure rule
UX fallout from session e2929c97 debrief:
- Dedupe: same (policyId, scriptType, scriptPart) re-emissions collapse into one row
keeping the most-complete entry via dataCompleteness() score. Real-world Autopilot
pattern is IME re-emitting the same script across ESP-phase transitions, sometimes
with degraded data on the second emission. Trade-off vs. Codex finding 3 (preserving
distinct runs of scheduled scripts in long sessions) — out of scope for this UI.
- Labels: use Intune Admin Center terminology — all health-script rows now read
"Remediation" with phase shown via separate badge ("detection" / "remediation" /
"post-detection"). The customer-facing word for these policies in Intune is
"Remediation", not "Health Detection".
- Flicker fix: React key included firstSeenIndex which was reassigned on every
reducer run, causing ScriptItemRow to remount on upstream prop changes (live
polling / SignalR) and collapse expanded detail panels. New scriptItemKey() uses
only identity-shaped fields (policyId, scriptType, scriptPart). firstSeenIndex
field removed from ScriptItem entirely.
- Animation polish: removed animate-pulse from Running container — only the inline
spinner SVG conveys motion now. Container is neutral gray. With multiple concurrent
Running cards the full-box pulse made the page feel unstable.
- stderr-as-failure rule (consistent across script types): stderr present →
script_failed regardless of exit code or compliance verdict. Eliminates the
inconsistency where a platform script with stderr showed red but a remediation
detection with stderr (and exit 0 + compliant) showed green for the same data
shape. Implemented in BOTH the Agent adapter and the Web reducer.
Agent V2 only — V1 phasing out.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
527e36b957 |
fix(rules/schema): allow healthScriptResult action in ime-log-pattern enum
CI ajv validation rejected HS-NEW-RESULT.json because the new healthScriptResult action wasn't in the schema's enum allowlist. Add it next to the other script* actions. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
594b3906ce |
feat(agent/v2,web): capture full health-script lifecycle with phase-aware semantics + live indicator
Replaces the line-by-line HS-COMPLIANCE / HS-EXITCODE / HS-STDOUT / HS-STDERR /
HS-RUN-CONTEXT pattern set (only ever captured the pre-detection phase, reset state
mid-cycle) with a single atomic source: HS-NEW-RESULT matches IME's "[HS] new result =
{JSON}" line and emits up to three script_completed events per policy run (pre-detection
/ remediation / post-detection). HS-SCRIPT-START now also fires a live script_started
event so the UI shows a "running" indicator immediately, before the consolidated final
result lands ~30s-3min later.
Phase-aware failure semantics: detection / post-detection scripts use exit code as a
compliance verdict (not a crash signal), so a non-compliant detection that gets
remediated no longer counts as a "failed" event in metrics — the row renders with
amber styling via isNonCompliantReport(). Only the remediation phase + platform
scripts route non-zero exit to script_failed.
Defensive against wire-format drift: regex uses [\s\S]* with end-anchor, JSON parsed
with try/catch + counter, frontend toNumber() accepts both numeric and string fields
(backend serializes ints as strings via Dictionary<string,string>). Reducer dedupes
with timestamp included so repeated runs of the same policy in long sessions don't
collapse into one row.
V2 only — V1 (Agent.Core) phasing out.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
eacef9c38a |
feat(notifications): opt-in "enrollment started" webhook on session registration
Fires Teams/Slack alert at backend session registration. Default off so existing tenants don't get a surprise notification storm. Triggers on fresh session OR WhiteGlove Part 2 resume; agent restart and re-registration of terminal sessions are skipped via pre-read of the session row before StoreSessionAsync. NotifyOnWhiteGloveResume constant in RegisterSessionFunction lets us disable the resume signal with a 1-line patch if it lands badly with end users, without removing the fresh-session signal. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
fce4a03d23 |
feat(sla): persistent SlaTenantStatus row replaces in-memory throttle, fixes 2h breach-notification spam
Background
----------
The in-memory SlaNotificationThrottleService kept its 4h cooldown in a process-
local ConcurrentDictionary. On Flex Consumption the Function host recycles
between 2h timer fires, wiping the dictionary, so the throttle was effectively
a no-op — observed in production as tenant b54dc1af receiving an SLA breach
notification every 2 hours for days on end (1 failed of 11-13 sessions, target
95%, success rate 91-92%).
Design
------
One row per tenant in a new SlaTenantStatus table (PartitionKey = tenantId,
RowKey = "status"). Flat schema with {Type}_* property prefixes for the four
breach kinds (SuccessRate, Duration, AppInstall, ConsecutiveFailures); one row,
one Upsert per evaluation cycle. Rationale documented inline: per-type rows
would be over-normalised for the access pattern (always all-types-per-tenant),
and ~50 tenants x 4 types = max 200 rows ever.
Throttle correctness
--------------------
The throttle decision (now - LastNotifiedAt >= cooldown) and the LastNotifiedAt
write happen in the same TryUpsertAsync(status, ifMatchETag) call. Two parallel
inline calls cannot both succeed in sending the same notification — the loser
gets a 412 from Table Storage, refetches, sees the updated watermark, and
either skips or schedules a new claim attempt. Up to 4 retries; on exhausted
retries we drop the notification rather than risk double-fire (at-most-once).
Cooldown is configurable: AdminConfiguration.SlaNotificationCooldownHours
(default 24, range 1-168). UI round-trip wired through the four required
files (AdminConfigContext, AdminConfigSettingsSection, types/adminConfig.ts,
SectionGlobalSettings).
Resolve path
------------
When a previously-active breach is no longer detected, the row transitions to
IsActive=false + ResolvedAt=now and a single sla_resolved notification fires.
Orphan sweep: tenants whose toggles get turned off while a breach is active
fall out of the qualifying loop, so the timer now also scans every status row
and silently clears any IsActive that no longer has a toggle backing it
(no resolved notification — the tenant disabled SLA, they don't care).
Per-type notes
--------------
- SuccessRate breach notifies with the configured warn-threshold, not the SLA
target — a 94% rate against target=99% / threshold=95% reads "below 95%",
not "below 99%".
- AppInstall now actually evaluates (the toggle was wired but evaluation was
missing). Aligned to the same ISO-week window the dashboard uses
(SlaMetricsService.GetIsoWeekKey), so notifications cannot diverge from the
/sla page. Sample-size floor of 5 apps per week, same as the dashboard.
- AppInstall has its own notification template (own title/summary/facts plus
TopFailingApp fact) — the builder used to silently fall through to the
Duration template for everything not SuccessRate.
- Telemetry Period tag derived from breachType: CurrentMonth for SuccessRate
and Duration, CurrentWeek for AppInstall — so AppInsights / ops dashboards
don't group AppInstall under the wrong period.
- AppInstall ops events now carry the actual installer totals (10 evaluated,
4 failed) instead of 0/0.
- ConsecutiveFailures: inline (ingest) path detects breaches, timer path
detects resolve. Both paths write to the same row through the same CAS
retry loop.
Clock injection
---------------
SlaBreachEvaluationService takes an optional Func<DateTime> nowProvider
(default () => DateTime.UtcNow) so the week-window regression test runs
deterministically on every test day instead of silently skipping when today
happens to fall in the first ISO week of the month.
Removed
-------
- SlaNotificationThrottleService and its tests. The in-memory model is gone;
there is no scenario where it would be wanted alongside the persistent row.
Tests
-----
- 21 new lifecycle + mapping + telemetry tests (CAS conflict + retry, CAS
exhausted-retries drops notification, threshold-not-target in notification,
orphan clearing, AppInstall template / week-window / totals, ConsecutiveFailures
inline persistence, multi-type race-probe, period-mapping theory).
- Backend xUnit: 1218 passed.
- Web vitest: 266 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
7d05a25c90 |
fix(dashboard/stats): move stats routes out from under sessions/ to dodge Azure Functions route collision
Azure Functions' attribute router does not honour literal-vs-parametric
specificity the way ASP.NET attribute routing does. A literal route of
"sessions/stats" was being eaten by the sibling GetSessionFunction whose
template is "sessions/{sessionId}" — the runtime bound /api/sessions/stats
to GetSession with sessionId="stats", which then 404'd because "stats"
fails GUID validation.
Diagnosed live via App Insights:
/api/global/sessions/stats -> 200 OK -> Functions.GetAllSessionStats
/api/sessions/stats -> 404 -> Functions.GetSession (!)
Per-tenant + GA stats routes moved out from under sessions/ for symmetry:
/api/sessions/stats -> /api/stats/sessions (MemberRead)
/api/global/sessions/stats -> /api/global/stats/sessions (GlobalAdminOnly)
EndpointAccessPolicyCatalog, api.ts, and the URL-builder tests follow.
Function-side + api.ts both carry a comment pinning the rationale so a
future "let's tidy these routes" refactor doesn't regress it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
af46160a65 |
fix(agent/v2): fail-soft runtime handoff so ASR-blocked installs don't strand devices
A customer device 2026-05-11 failed bootstrap at the WMI runtime spawn: returnValue=2 (Access Denied -- Defender ASR rule d1e49aac-8f56-4280-b9ba-993a6d77406c "Block process creations originating from PSExec and WMI commands"). The throw propagated to --install exit 1 -> bootstrap throw -> next IME run SKIPped via pre-flight, leaving the device with payload deployed + BootTrigger task registered but no live runtime until manual intervention. --install now runs a two-tier fallback: try WMI Win32_Process.Create -> try schtasks /Run on the BootTrigger task -> defer to BootTrigger on next reboot. Exit code is 0 whenever deployment + Scheduled Task creation succeeded; runtime-spawn failure is a WARNING, not fatal. The bootstrap script's Get-Process probe (Install-AutopilotMonitor.ps1:280) is now the canonical "no runtime came up" signal. Orchestration extracted to pure function DecideRuntimeLaunchOutcome, unit-tested with 8 cases pinning the contract (incl. ASR-rule GUID surfaced in diagnostic only when returnValue=2). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
6c643068c7 |
fix(web/admin): expose PoisonQueueBacklog event types in OpsAlertRules UI
The previous commit added the two new EventTypes server-side (PoisonQueueBacklogHigh / PoisonQueueBacklogCritical) but the admin alert rules section hardcodes its known-event-types catalog in the component itself - so the dropdown didn't list them and operators couldn't wire Telegram to the new alerts. Adds both strings to the Security category, matching how the SignalR/Cert event types were registered when those watchers landed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
c8902de7c9 |
feat(backend/maintenance): poison-queue backlog watcher with Telegram alerts
Extends the SignalR/Cert pattern to async-worker poison queues. The previous
PR made backlogs visible on /health/detailed - this one makes them actively
page the operator via the existing OpsAlertDispatch -> Telegram path.
New partial MaintenanceService.PoisonQueue.cs runs on the 2 h timer (and on
manual maintenance triggers) and emits two new EventTypes:
- PoisonQueueBacklogHigh (Severity=Warning, threshold >= 1)
- PoisonQueueBacklogCritical (Severity=Error, threshold >= 10)
Dedup mirrors the cert watcher: composite key {EventType}|{queueName} built
from today's Security OpsEvents -> one alert per queue per tier per UTC day.
A tier escalation (warning -> critical) fires a new alert because the
EventType changes; a stable backlog re-pings once per UTC day so it stays in
the operator's attention.
Constants + classifier + watch-list moved from HealthCheckService to the new
MaintenanceService partial - the live health card and the timer-driven
Telegram alert now classify identically (same canonical source). Boundary
and JSON-extractor tests follow the move; HealthCheck tests are reduced to
the card-form assertions they actually exercise.
Operator follow-up (manual, not in code): add two OpsAlertRules in the admin
UI - one per new EventType - to wire Telegram. Until then the watcher writes
OpsEvents but no message is sent, matching how SignalR/Cert alerts were
introduced.
Backend: 1199/1199 (1193 + 6 net from test resplit). Build 0/0.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
bf974c4a57 |
fix(dashboard/stats): Active = InProgress only, drop Pending + Stalled
Live data showed cross-tenant 7d breakdown of non-terminal sessions: InProgress 11, Pending 43, Stalled 6 = 60 total. The Pending bucket dominates for any tenant with regular WhiteGlove provisioning — those sessions have completed pre-prov and are waiting for the end user to power on the device, which can take days or weeks. Counting them under "Currently enrolling" implies live activity that isn't happening. Active now matches the literal in-flight status. Pending and Stalled remain visible via the sessions list filters; if we want a richer breakdown on the card row later, it can be split into separate cards without further changing the Active semantics. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
2284100915 |
feat(backend/health): surface async-worker poison queue backlog in /health/detailed
Three async workers (analyze-on-enrollment-end, vulnerability-correlate, telemetry-index-reconcile) park 5x-failed messages on <name>-poison. Until now there was no operator-visible signal when this happened: backlogs could grow silently and sessions stayed un-analyzed / vulnerability reports were lost. Adds a "Poison Queues" entry to the detailed health-check next to "SignalR Quota" with the same shape: parallel probe via IPoisonQueueProbe, tri-tier classifier (0=healthy, >=1 warning, >=10 critical, thresholds App-Setting configurable via PoisonQueueWarningThreshold / PoisonQueueCriticalThreshold), GA-only visibility. Per-queue 404 counts as zero because poison queues are created lazily on first poison-move. Per-queue exceptions surface in details as "error: <msg>" and force at least warning so a flaky Storage call cannot mask a real backlog. Web UI needs no change - the generic checks[] render loop on /health-check picks up the new card automatically. Backend: 1193/1193 (1178 + 15 new). Build 0 warnings, 0 errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
068321baaa |
fix(dashboard/stats): reset cards on scope change, pin "in flight" semantics
Follow-up to
|
||
|
Oliver Kieselbach
|
b76a6ccaf0 |
fix(dashboard/stats): server-aggregate cards so they stop drifting with pagination
The five dashboard cards (Active Sessions, Success Rate, Avg. Duration, Total Today, Failed Today) were derived from whatever sessions the client had paginated into memory. With default pageSize=10 and SignalR-driven appends, "Total Today" would jump 40 -> 22 -> 20 within a single visit as the list grew/shrank. "Last 7 days" was a printed label only - the client compute had no 7-day filter at all. Now: - GET /api/sessions/stats?days=7 (MemberRead, JWT tenant) - GET /api/global/sessions/stats?days=7&tenantId= (GlobalAdminOnly, optional tenant filter; absent = cross-tenant sum) - TableStorageService.AggregateSessionStats drains SessionsIndex once over the window: UTC-midnight boundary for today, success rate over terminal sessions only (InProgress no longer drags it down), avg duration over Succeeded with DurationSeconds > 0 only. - useDashboardStats hook owns the cards: initial fetch + debounced refetch on SignalR newSession/newevents in the active scope + forced refetch on reconnect to recover messages missed during an outage. - Client compute removed from useDashboardFilters. New submittedTenantIdFilter state isolates typing-in-filter from submitted-filter so keystrokes never trigger backend round-trips. Tests: 17 xUnit (aggregation invariants + days parsing) + 7 vitest (URL builder). Suites: backend 1177/1178 (unrelated RawTablePaginationTests flake), web 266/266. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
5e2cdf0aad |
perf(agent/v2): pre-size hot-path event dictionaries
Skip the default 0->3->7->17 Dictionary resize cascade on hot-path event/audit-trail allocations by setting initial capacity + Ordinal comparer. ~3k Gen0 allocs saved per session for an agent that runs continuously on endpoints. Behaviour unchanged, 1295 tests green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
5f6de2c02c | fix health dashboard | ||
|
Oliver Kieselbach
|
c101eda796 | health check fix | ||
|
Oliver Kieselbach
|
7da5d482f1 |
fix(mcp/deps): bump fast-uri 3.1.0->3.1.2 to patch 2 high-sev advisories
Resolves GHSA-q3j6-qgpj-74h6 (path traversal via percent-encoded dot segments) and GHSA-v39h-62p7-jpjc (host confusion via percent-encoded authority delimiters). Transitive via @modelcontextprotocol/sdk -> ajv. Lockfile-only change, no API surface impact. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
871ec31bf6 |
feat(backend/health): surface SignalR free-tier quota in /health/detailed
HealthCheckService now runs a fourth check (SignalR Quota) alongside Storage, Processing, and Agent Binaries. Live read of ConnectionCount (max/1h) and MessageCount (total since 00:00 UTC) via the same IAzureMonitorMetricsReader that backs the maintenance-time Ops watcher - no cache; the data is already 3-5min stale from Azure Monitor's side, and a /health/detailed re-check is expected to give fresh values during incident response. Reuses MaintenanceService's internal classifier (CalculatePercent, ClassifySignalRQuotaTier) so the watcher and the dashboard agree on the 80%/95% boundaries down to the floor-vs-ceil rounding. Tier mapping: None -> healthy, Warning -> warning, Critical -> unhealthy. Worst-of-both-metrics wins. PerformAllChecksAsync now uses a tri-state combiner (any unhealthy / any warning / else healthy) so a Warning tier correctly tints the OverallStatus bar yellow instead of red. Fail-soft: - SignalRResourceId unset -> Status="unknown" with "watcher not configured" (excluded from OverallStatus combiner so dev/local stays healthy) - Both metric reads null -> Status="unhealthy" pointing at the most likely cause (Function App MI missing 'Monitoring Reader' on the SignalR resource) - One read null -> "no data" for that row, normal classification for the other Frontend unchanged: the existing /health-check Backend Services card loop renders the new check with Connections, Messages, and Resource as Details. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
5cb7480bb4 | updated Changelog | ||
|
Oliver Kieselbach
|
4af28fefbe | added V2 rollout notify banner to dashboard | ||
|
Oliver Kieselbach
|
d51bde6017 |
refactor(release): parametric per-line agent build pipeline
Replaces V1/V2-hardcoded build + bootstrap with a single parametric
pipeline that scales to V3+ without rewrites. Full architecture in
.claude/plans/v2-cutover.md.
Stable namespace (Bootstrap-side, never renamed): generic Bootstrap
script, agent owns its --install defaults so it works for any line.
Versioned namespace (per-line, SelfUpdater-side): isolated *-v{N}.*
blobs; SelfUpdater never crosses lines or touches stable.
Backend GetAgentConfig dispatches per X-Agent-Version-Major to
AdminConfiguration.GetAgentLine(int) -> LatestAgentV{N}* fields.
Future V3 = add field set + switch arm. TableConfigRepository.Map
falls back to legacy unsuffixed columns for transparent migration.
V1 SelfUpdater now reads version-v1.json (would have been clobbered
by the V2 cutover otherwise). V2 agent owns its own 600s TenantId-wait
default so the Bootstrap calls --install plain.
Tests: 3289 across 6 suites, all green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|