gsadmin/AutopilotMonitor
1
0
Fork 0
mirror of https://github.com/okieselbach/AutopilotMonitor.git synced 2026-10-25 05:00:07 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,485 Commits 1 Branch 128 Tags
51920dbf4e8ec61bae9ca3a7cffb95cd6a4007aa
Commit Graph

1485 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Oliver Kieselbach 51920dbf4e added session id to restore browser 2026-05-15 11:03:39 +02:00
Oliver Kieselbach 6f900a412a fix(agent/v2): tighten shutdown-gate semantics + WMI error counting 
Codex review of ef530fb8 surfaced three issues; all addressed.

P1 - Auth-failure bypassed the new shutdown gate. CreateAuthThresholdHandler
emitted agent_shutting_down directly without claiming the cross-path slot,
so the finally-block runtime_host_exit fallback then also emitted, putting
two events on the wire. The handler now accepts tryClaimShutdownEvent +
releaseShutdownEventClaim Funcs and participates in the same gate as the
other paths. AgentRuntimeHost wires it in via TryClaimAgentShuttingDownEmit
plus the new ReleaseShutdownEventClaim helper.

P2 - The gate was claimed before the emit had any chance to succeed, so a
Ctrl+C that fired before onIngressReady (lifecyclePost == null) consumed
the slot permanently and the later finally fallback could no longer surface
the event. EmitAgentShuttingDownGapPath now returns the new
AgentShuttingDownEmitResult enum (Success / NoPost / EmitFailed). The
AgentRuntimeHost wraps claim + emit in TryEmitAgentShuttingDownGap which
releases the slot on NoPost so a later path can retry, while keeping it
claimed on Success and EmitFailed. Auth-failure path symmetric: releases
on null post.

P3 - GetProcessOwner swallowed its WMI exceptions and returned null without
bumping _wmiErrorCountSinceStart. desktop_detector_first_poll and
_no_candidate therefore reported wmiErrorCount=0 even when WMI was broken,
masking the very failure mode the liveness events were designed to detect.
The counter is now incremented inside GetProcessOwner's catch AND on
non-zero InvokeMethod results, so the per-poll PID-owner failure rate
flows through to telemetry.

Tests: 3777 green (1538 backend + 1068 V2.Core, +3 for new auth_failure
gate tests + 530 V1 + 294 DecisionCore + 347 vitest). TypeScript noEmit
clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 09:56:16 +02:00
Oliver Kieselbach 8dd80d7232 update gitignore 2026-05-15 09:39:37 +02:00
Oliver Kieselbach ef530fb890 feat(agent/v2): DAD liveness + agent_shutting_down gap closure 
DAD liveness (3 new state-change-only events, NOT periodic, max 3 per
detector lifetime): desktop_detector_started / _first_poll /
_no_candidate. Distinguishes "agent dead post-reboot" from "DAD wiring
broken" from "user never logged in" in sessions missing desktop_arrived
(the ca9e3c59 timeout pattern). Threshold via global AdminConfiguration
.DesktopDetectorNoCandidateTimeoutMinutes (default 10, 0 = disabled),
round-trip wired through TableConfigRepository, GetAgentConfigFunction
(ConfigVersion 27 -> 28), CollectorConfiguration, DefaultComponentFactory,
DesktopArrivalHost + admin UI. Also fixes a peripheral-loss bug: V2's
DesktopArrivalDetector.OnTraceEvent callback was never wired, so the
existing desktop_excluded_user and desktop_real_user_detected events
were dead code in V2. Now routed via InformationalEventPost in the host.

Shutdown gap closure (4 new agent_shutting_down emit paths): ctrl_c,
process_exit, unhandled_exception (with exception type + 500-char-
truncated message), runtime_host_exit (defensive finally fallback).
Cross-path Interlocked idempotency gate shared with EnrollmentTermination
Handler via new tryClaimShutdownEvent ctor param prevents double-emit
when Ctrl+C races a real Terminated transition.

Event-type unification (V2 only): CreateAuthThresholdHandler now emits
agent_shutting_down with data.reason=auth_failure (was agent_shutdown).
EnrollmentTerminationHandler reason vocabulary unified to lowercased
tags (decision_terminal, max_lifetime). V1 keeps legacy agent_shutdown
(phasing out). Web/MCP consumers (eventHelpers, whiteGloveSplit test
fixtures, gather-rules eventTypes catalog, MCP semantic-event list,
docs) treat both as synonyms.

Tests: 3411 green (1538 backend + 1065 V2 core + 530 V1 core + 294
DecisionCore + 347 vitest). TypeScript noEmit clean. 5 new DAD-liveness
tests, 5 new gap-path tests, 2 new idempotency tests, 3 reason-vocab
updates.

Not in this PR: cross-process death detection for the actual ca9e3c59
silent-post-reboot-death pattern (Logon-Trigger / external watchdog) -
separate follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-15 09:39:07 +02:00
Oliver Kieselbach f1583fc586 fix(backend,web): move cascade-delete admin routes from /api/admin/* to /api/global/* 
The /api/admin/* prefix collides with the Azure Functions runtime's built-in
admin paths, which sit behind mTLS client-certificate auth. Browser preflights
to /api/admin/* trigger TLS renegotiation against the Function App's mTLS
boundary and die before our PolicyEnforcementMiddleware ever runs — the
operator sees CORS-shaped errors that aren't actually CORS-shaped.

Repo convention all along has been /api/global/* for cross-tenant GA-only
HTTP endpoints (e.g. global/session-deletions, global/session-reports). The
cascade-delete endpoints landed under /api/admin/* in the original 7-PR
backend branch and went undetected until the admin UI started calling them
from the browser.

Routes renamed:
  admin/sessions/{sessionId}/delete/preview     → global/sessions/{sessionId}/delete/preview
  admin/sessions/{sessionId}/restore            → global/sessions/{sessionId}/restore
  admin/sessions/{sessionId}/deletion-manifest  → global/sessions/{sessionId}/deletion-manifest
  admin/tenants/{tenantId}/deletion-manifests   → global/tenants/{tenantId}/deletion-manifests

Touched:
- 4 Function classes: Route attribute + XML-doc header
- EndpointAccessPolicyCatalog: 4 entries (TenantScoping.RouteParam preserved
  on the tenants route), with a comment explaining why /api/global/* is the
  load-bearing prefix for new GA-only HTTP routes
- DeleteSessionFunction: 3 user-facing "Cascade is poisoned; recover via
  POST /api/.../restore" error messages updated
- SessionDeletion.cs: XML-doc reference to the restore route
- lib/api.ts: 4 frontend URL builders (storedManifest, preview, restore,
  tenantManifests)
- RestoreSessionFunctionTests: 3 EndpointAccessPolicyCatalog.FindPolicy
  lookups
- deleteSessionResponse.test.ts: test fixture message string

No semantic change — only the URL prefix moves. CORS config + JWT middleware
+ policy gating all work the same. The Restore Browser tab + the per-row
Preview / Restore buttons in the Session Cleanup admin page now actually
reach the worker from a browser.

Tests: backend xUnit 1538/1538 pass, web vitest 347/347 pass, dotnet build +
tsc clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-14 11:38:34 +02:00
Oliver Kieselbach 44076686b6 feat(backend,web): restore-browser tab + detailed dry-run/real-run dialog 
Follow-ups on the Session Cleanup admin page after first hands-on review:
an entry point for restoring already-completed cascades (Sessions row gone,
manifest still in the 33-day TTL window), a proper outcome panel that surfaces
what the backend already returned, and a sidebar fix.

== New Restore Browser tab ==
- Backend: GET /api/admin/tenants/{tenantId}/deletion-manifests enumerates
  every persisted snapshot blob under <tenantId>/, parses (sessionId,
  manifestId, sizeBytes, lastModifiedUtc) from the blob name, groups by
  sessionId and sorts both levels by recency. GA-only + TenantScoping.RouteParam
  in EndpointAccessPolicyCatalog (pinned by EndpointPolicyCatalogCompletenessTests).
- BlobStorageService.EnumerateDeletionManifestsByTenantAsync — sibling to
  the existing maintenance-sweep enumerator but tenant-prefixed and yielding
  size + timestamp so the UI can rank without an extra HEAD per blob.
- Web: new tab "Restore Browser" between Stranded and Activity. Tenant
  dropdown (sourced from AdminConfigContext.tenants) → collapsible
  session/manifest tree on the left, manifest summary on the right with a
  Restore… button that opens the dialog with IDs pre-filled. Useful when
  the cascade completed cleanly and the row no longer appears on the
  In-Flight / Poisoned / Stranded tabs.

== Reusable summary view ==
DeletionManifestSummaryView extracted from the modal so the same fetch +
field block + worker-progress + preflight-counts + last-failure panel renders
both inside the modal (Preview button on the other tabs) and inline on the
Restore Browser's right-hand panel. Modal becomes a thin wrapper.

== Dry-run / real-run side-by-side ==
RestoreConfirmDialog now keeps both outcomes visible:
- Two slots (dryRunResult, realRunResult), each rendered as a full result
  section with predicted/actual rows-by-table, inventory re-increments,
  duration, backend message, current state, pending manifest id.
- "Start over" resets both slots so the operator can re-run cleanly.
- Submit buttons gate the workflow: pre-submit shows "Run dry-run" + an
  explicit "Skip dry-run · Restore now"; after dry-run the second button
  becomes "Apply restore" (clear-intent); after real-run all submit
  buttons disappear so accidental double-applies aren't possible.
Previously the dialog discarded everything but Outcome + Mode + dryRun
flag, hiding the actual point of dry-run (the wouldRestoreByTable map).

== Codex review follow-ups (this round) ==
- P2: Restore Browser cleared the displayed sessions only after the new
  fetch returned, so a fast A→B switch (or out-of-order responses) could
  leave the operator looking at tenant A's manifests with the dropdown
  saying B. Fix: clear sessions/expandedSessions/selection synchronously
  on tenant change, abort the in-flight request via AbortController,
  pair the response with a loadedTenantId state, and gate rendering on
  loadedTenantId === selectedTenantId. Three-layer protection so a late
  response that sneaks past the abort can't paint stale data.
- P3: PoisonedTab's onRestored callback closed the dialog immediately
  after a real restore, defeating the new dry-run vs real-run side-by-side
  display. Fix: drop setRestoring(null) from the callback so the dialog
  stays open until the operator dismisses it via Close / Start over.
  Background refresh of the Poisoned list still happens via setRefreshKey.
  Now consistent with the Restore Browser tab.

== Sidebar highlight fix ==
GlobalSidebar marked BOTH ga-ops sub-items active when on the longer
nested route, because each sub used pathname.startsWith(sub.href + "/")
independently. Visible bug: navigating to /admin/ops/session-cleanup lit
up both "Maintenance" (/admin/ops) and "Session Cleanup" in the sidebar.
Fix: compute the longest-matching sub.href once per expanded group, mark
only that one active.

Tests: backend xUnit 1538/1538 pass (one new endpoint registered in the
catalog, picked up by the existing completeness test). Web vitest 347/347
pass. TypeScript strict + dotnet build clean.

Deliberately deferred:
- xUnit for GetTenantDeletionManifestsFunction (same HttpRequestData
  mock-harness gap as the other read endpoints).
- vitest for RestoreBrowserTab + the new dialog flow (defer until a
  real field issue).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 21:48:30 +02:00
Oliver Kieselbach fa8ad38ed1 change bell notification to always visible 2026-05-13 20:19:40 +02:00
Oliver Kieselbach 09bfe7a1bc feat(backend,web): admin session-cleanup UI for V2 cascade-delete + codex follow-ups 
Operationalises the 7-PR backend cascade-delete pipeline that landed via commits
c8658b9f..921994e8: per-tenant V2 opt-in switch, audit-trail consolidation, and
a Global-Admin-only Session Cleanup page to track in-flight / poisoned / stranded
cascades and trigger restores.

== PR-A: per-tenant V2 enrollment ==
- UpdateTenantConfigurationFunction extends the GA-only gate list with
  EnableCascadeDeleteV2 so tenant admins cannot self-opt-in / out of the
  snapshot+restore pathway.
- TenantManagementSection.tsx adds a purple-bordered "Enable V2 Cascade Delete"
  checkbox in the GA edit modal and a "Cascade V2" badge on the tenant card so
  rollout state is visible at a glance.

== PR-B: audit consolidation ==
- SessionDeletionHandler drops the 15-per-cascade deletion_step_completed /
  deletion_step_failed / deletion_verification_failed audit emissions. Step
  telemetry lives in DeletionProgress + structured logs; tenant admins now see
  just deletion_started + deletion_completed + deletion_restored.
- SessionDeletionWorker no longer writes a deletion_poisoned tenant audit;
  the signal moved to a new SessionDeletionPoisoned OpsEvent (operator-bound,
  Telegram-routable, dual-registered in OpsAlertRulesSection.tsx catalog).
  IMaintenanceRepository dependency removed from the worker constructor.
- SessionRestoreService stops emitting the deletion_manifest_downloaded
  breadcrumb; deletion_restored is the operator-visible outcome.

== PR-C: Session Cleanup admin page ==
- New /admin/ops/session-cleanup route (gated by app/admin/layout.tsx's
  ProtectedRoute requireGlobalAdmin). Sidebar entry under ga-ops alongside
  Maintenance. Four URL-state tabs: In-Flight / Poisoned / Stranded Queued /
  Activity (links to existing /audit page).
- Backend reads: GetSessionDeletionsListFunction
  (GET /api/global/session-deletions?state=&strandedSinceMinutes=) +
  GetSessionDeletionManifestFunction
  (GET /api/admin/sessions/{id}/deletion-manifest?tenantId=&manifestId=&mode=).
  Both registered as GlobalAdminOnly in EndpointAccessPolicyCatalog.
- DeletionPreviewModal reads the persisted snapshot via the stored-manifest
  endpoint (NOT a fresh dry-run via the existing /delete/preview, which would
  enumerate live-but-partially-deleted data for in-flight cascades). Renders
  manifest counts, snapshot SHA, worker progress (CompletedSteps,
  VerificationDone, TombstoneStarted, aggregate decrements), and the last
  verification failure with residual sample. Download button uses
  authenticatedFetch + Blob + Object URL so the Bearer token is sent.
- RestoreConfirmDialog drops the Full/Partial radio (mode is auto-selected
  by SessionRestoreService based on cascade state — forcing the wrong mode
  would corrupt state). Reason field threads through to deletion_restored
  audit details. Dry-run default = true and previews the auto-selected mode.

== Codex review follow-ups (across four rounds) ==
- High: tenantId is explicitly accepted on /delete/preview (query) and
  /restore (body); GA cross-tenant resolution prefers the explicit value over
  JWT TargetTenantId, falling back to SessionsIndex lookup. Without this, a
  GA acting on a session in another tenant got their JWT home tenant.
- High: new stored-manifest endpoint replaces the dry-run preview for
  in-flight / poisoned rows; previous fix built a fresh manifest against
  already-partially-deleted data and misrepresented the cascade.
- Medium: SessionDeletionPoisoned OpsEvent now carries failureType +
  failureMessage + observedResidualCount + a small residual preview, persisted
  by the handler into DeletionProgress (LastFailureType, LastFailureMessage,
  LastObservedResidualCount, LastResidualSampleJson) before the throw so the
  worker can include them.
- Medium: dry-run restore reports the AUTO-SELECTED mode (full/partial) so
  the dialog's preview reflects what the real run will do, instead of the
  literal string "dryRun".
- Medium: residual count field renamed to observedResidualCount with an
  honest XML doc — CascadeVerificationService caps at 50 per table AND
  short-circuits after the first failing table, so the number is a lower
  bound. UI surfaces "N of 50+ observed rows" when N hits the cap.
- Medium: three-layer cap defense on the OpsEvent residual sample to stay
  under the 4096-char TableOpsEventRepository truncation: max 5 entries
  (OpsEventResidualSamplePreviewSize), per-field max 96 chars with `…` marker
  (OpsEventResidualKeyMaxChars), total preview budget 1200 chars
  (OpsEventResidualPreviewBudgetChars) with trailing-entry drop. Full sample
  stays in the progress blob for the admin UI.
- Low: estimatedSnapshotBytes field name aligned end-to-end; activity tab
  text clarifies audit-action vs OpsEvent distinction.

Tests:
- Backend xUnit 1538/1538 pass (up from 1510 baseline; +28 across the new
  Session Cleanup endpoints, OpsEvent enrichment, restore-mode reporting,
  cap-defense unit + integration tests).
- Web vitest 347/347 pass.
- TypeScript strict + dotnet build clean.

Deliberately deferred:
- xUnit tests for GetSessionDeletionsListFunction +
  GetSessionDeletionManifestFunction (no HttpRequestData mock harness in repo;
  would be a separate refactor).
- vitest for DeletionPreviewModal + RestoreConfirmDialog (defer until first
  field issue).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 20:09:50 +02:00
Oliver Kieselbach 5846b11775 fix(mcp-deps): bump transitive protobufjs to 7.5.8 to resolve Dependabot alerts 
Patch-level bump within onnxruntime-web's ^7.2.4 constraint; closes 8 open
Dependabot alerts (CVE family: DoS, code injection, prototype pollution).
Lockfile-only change, no manifest update needed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 16:54:31 +02:00
Oliver Kieselbach 1c00f44709 Merge pull request #63 from okieselbach/fix/v2-classic-hello-disabled-deadlock AND parts of the deletion cascade feature 
fix(agent-v2): Classic-v1 + Hello=disabled completion deadlock AND parts of the deletion cascade feature
 2026-05-13 15:10:39 +02:00
Oliver Kieselbach 2e7172b6ab test(agent-v2): pin coordinator re-raise path via OnEspExited test seam (PR6/6) 
The PR4 coordinator-pipeline tests drove adapter.TriggerEspExitingFromTest(),
which only proves the adapter emit — it bypasses the coordinator's OnEspExited
handler (where LastEventOccurredAtUtc mirroring + public EspExited re-raise
live). A future regression that breaks the coordinator subscription, the
LastEventOccurredAtUtc mirror, or the public EspExited fire would slip past.

This commit adds a coordinator-level test seam mirroring the existing
TriggerFinalizingSetupPhaseForTest pattern, and reroutes the wire-through tests
through it.

EspAndHelloTracker.TriggerEspExitedForTest:
- internal seam (InternalsVisibleTo) that calls OnEspExited(this, args)
- exact signature match to the live ShellCoreTracker.EspExited delegate
- exercises LastEventOccurredAtUtc mirror + public EspExited invoke + finally
  reset — i.e. the full coordinator-side contract

ShellCoreToEngineEspExitingTests:
- Coordinator_EspExited_handler_re_raises_to_adapter_as_EspExiting_signal:
  asserts source-time preservation through the coordinator → adapter chain
  and that no clock-fallback "derivedTimestamp" evidence tag was added.
- Coordinator_EspExited_handler_clears_LastEventOccurredAtUtc_after_re_raise:
  new defensive test for the finally-block reset; protects against a stale
  ShellCore timestamp bleeding into a subsequent Hello/Provisioning forward.
- Coordinator_to_reducer_arms_HelloSafety_when_AccountSetup_observed:
  rewritten to use the coordinator seam; now asserts HelloSafety dueAtUtc ==
  sourceTime + 300s (not Clock.UtcNow + 300s) so source-time threading is
  proven end-to-end (ShellCoreTracker → Coordinator → Adapter → Engine), not
  just adapter → engine.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 13:29:27 +02:00
Oliver Kieselbach 35c7e4cfc9 fix(decision-core): emit CancelDeadline effect for HelloSafety in fast-path (PR5/5) 
The Hello-disabled fast-path called builder.CancelDeadline(HelloSafety) but did
not emit a corresponding DecisionEffectKind.CancelDeadline. The reducer state
no longer carries the deadline, but the external scheduler
(DefaultComponentFactory) keeps the timer armed and fires DeadlineFired:HelloSafety
after the 300s window — which then dead-ends in the reducer because the deadline
name is no longer in state.

This commit adds the missing scheduler-visible cancel:

- TransitionToFinalizing gains an optional extraLeadingEffects parameter so the
  fast-path can prepend a CancelDeadline effect ahead of the normal
  ScheduleDeadline(FinalizingGrace) + phase_transition. Existing call-sites
  (HandleHelloResolvedV1, HandleDesktopArrivedV1's non-fast-path branch,
  HandleHelloSafetyDeadlineFired) keep their current behaviour by passing null.
- The fast-path checks state.Deadlines for HelloSafety BEFORE building the
  effect — only emits the cancel when the deadline was actually armed. Avoids
  no-op cancellations on the common case where the fast-path is the sole
  completion path.

Tests:
- HelloDisabled_FastPath_cancels_armed_HelloSafety_deadline_in_state_and_effects:
  expanded the existing state-only assertion to also assert the
  CancelDeadline(HelloSafety) effect alongside ScheduleDeadline(FinalizingGrace)
  and phase_transition.
- HelloDisabled_FastPath_without_armed_HelloSafety_does_not_emit_redundant_CancelDeadline_effect:
  new negative test guarding against audit-trail noise.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 13:20:46 +02:00
Oliver Kieselbach 4a6bc656df fix(agent-v2): wire EspExited through coordinator to production adapter (PR4/5) 
PR1 added EspExited on ShellCoreTracker + a mapping on ShellCoreTrackerAdapter,
but ShellCoreTrackerAdapter is only used in single-tracker test wiring. In
production EspAndHelloHost instantiates the EspAndHelloTracker coordinator, and
EspAndHelloTrackerAdapter subscribes to coordinator-forwarded events — the new
EspExited never reached the engine on a real device.

This commit reroutes EspExited through the coordinator (the same path the other
ShellCore-originated events take):

- EspAndHelloTracker re-raises EspExited from its inner ShellCoreTracker, with
  LastEventOccurredAtUtc mirrored from the EventArgs timestamp so the adapter's
  ResolveOccurredAt picks up the source-event time (matching the existing
  WhiteGloveCompleted / EspFailureDetected forwarding contract).
- EspAndHelloTrackerAdapter subscribes to coordinator.EspExited and posts
  DecisionSignalKind.EspExiting with no per-kind dedup — Shell-Core 62407 fires
  on every ESP phase transition, and the reducer's ShouldTransitionToAwaitingHello
  guard decides which occurrence arms HelloSafety. The coordinator's existing
  IsIntermediateDeviceEspExit filter only governs the legacy
  FinalizingSetupPhaseTriggered path; the new EspExiting signal bypasses it
  deliberately so the engine remains the single source of truth.
- EspExitedEventArgs becomes public (was internal) because the coordinator's
  public event surface needs an at-least-as-accessible args type.

Tests:
- EspAndHelloTrackerAdapterTests: +2 tests covering the coordinator-forwarded
  mapping and the no-dedup contract.
- ShellCoreToEngineEspExitingTests: +2 tests exercising the full production
  topology (coordinator → adapter → ingress → engine.Reduce → HelloSafety arm).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 13:20:34 +02:00
Oliver Kieselbach 4853f8c6d1 test: cover EspExiting adapter + Hello-disabled fast-path + wire-through (PR3/3) 
Three new test suites verifying both halves of the V2 Classic-v1 + Hello-disabled
deadlock fix:

ShellCoreTrackerAdapterTests (+3 tests, +1 expanded):
- EspExited → DecisionSignalKind.EspExiting mapping (kind, source, evidence)
- No-dedup contract: every 62407 occurrence forwarded individually so the
  reducer's ShouldTransitionToAwaitingHello guard can disambiguate the genuine
  post-AccountSetup exit from intermediate Device→Account exits.
- Source-event timestamp preserved (not _clock.UtcNow) — critical for backfill
  so EffectiveDeadlineBase floors HelloSafety at the actual ESP-exit instant.
- Existing All_three_events_can_fire_in_one_session expanded to All_four_*.

ClassicHelloDisabledCompletionTests (6 tests):
- Fast-path happy path: HelloPolicyDisabled + AccountSetup + DesktopArrived →
  Finalizing → FinalizingGrace deadline → enrollment_complete.
- Three guard tests: fast-path skipped when policy not detected, before
  AccountSetup, and when policy is enabled.
- HelloSafety reducer path (Fix 1 belt-and-suspenders): EspExiting →
  HelloSafety 300s → Timeout → DesktopArrived → Finalizing.
- Fast-path cancels an armed HelloSafety deadline so it cannot fire
  post-Completion.

ShellCoreToEngineEspExitingTests (2 wire-through tests):
- Real ShellCoreTracker.HandleBackfillRecord(esp_exiting_description) drives
  the .NET event → adapter subscription → ingress.Posted, with source
  timestamp preserved.
- End-to-end: same backfill → adapter signal → engine.Reduce on a seed state
  with AccountSetup entered → HelloSafety deadline armed at sourceTime+300s.

Full DecisionCore.Tests: 293/293 pass. Full Agent.V2.Core.Tests: 1048/1048 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 12:57:49 +02:00
Oliver Kieselbach a32706dee8 fix(decision-core): Hello-disabled fast-path in HandleDesktopArrivedV1 (PR2/3) 
When HelloPolicyEnabled is explicitly false AND AccountSetup has been reached,
synthesise HelloOutcome="Skipped" on DesktopArrived and route directly through
Finalizing instead of waiting out the full HelloSafety 300s window. This is the
belt-and-suspenders companion to PR1's EspExiting reducer path: whichever fires
first wins, and the dual-path design survives a future regression in either
half (missing EspExiting adapter or missing HelloPolicyDetected adapter).

Both guards are required:
- HelloPolicyEnabled?.Value == false (policy explicitly disabled)
- AccountSetupEnteredUtc != null (mirrors ShouldTransitionToAwaitingHello)

This prevents a premature mid-DeviceSetup completion if HelloPolicyDetected
happens to fire before AccountSetup is observed. Unknown policy
(HelloPolicyEnabled == null) keeps the prior pessimistic behaviour.

The fast-path cancels HelloSafety on its way through so a deadline armed by an
earlier EspExiting cannot fire post-Completion and dead-end in the reducer.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 12:50:03 +02:00
Oliver Kieselbach 879b89d711 fix(agent-v2): emit DecisionSignalKind.EspExiting from ShellCoreTracker (PR1/3) 
ShellCoreTracker observes the Shell-Core 62407 OOBE_ESP*Exiting event but only
emitted it as a wire-event + FinalizingSetupPhaseTriggered notification. No V2
SignalAdapter mapped it to DecisionSignalKind.EspExiting, so HandleEspExitingV1
(which arms the HelloSafety 300s deadline) was never invoked. On Classic-v1
enrollments with Hello policy disabled this leaves the reducer stuck in
EspAccountSetup until the agent max-lifetime watchdog fires after 360 min.

This commit adds a dedicated EspExited .NET event on ShellCoreTracker (raised
both on live event log records and on backfill) and a matching subscription in
ShellCoreTrackerAdapter that posts a DecisionSignalKind.EspExiting signal.

OccurredAtUtc is taken from the source event timestamp (not _clock.UtcNow) so
HandleEspExitingV1's EffectiveDeadlineBase floors HelloSafety at the actual
ESP-exit instant rather than wall-clock-now on a backfilled run.

The adapter does NOT dedup EspExiting because Shell-Core 62407 fires at every
ESP phase transition (Device→Account, Account→End); the reducer's
ShouldTransitionToAwaitingHello guard already distinguishes the genuine
post-AccountSetup exit from intermediate exits.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 12:49:03 +02:00
Oliver Kieselbach c237140d59 fix(backend): cascade-delete codex follow-ups #2 (F1 DELETE cross-tenant, F2 register guard, F3 tombstone marker, F4 cross-tenant test, bootstrap-bypass) 
F1 — DELETE /api/sessions/{sessionId} bypassed cross-tenant validation
because the catalog entry had no TenantScoping. A Global Admin clicking
"delete" on a session in tenant B from the portal silently routed the
delete into their own JWT tenant A, never finding the row. Catalog
entry gains TenantScoping.QueryParam; DeleteSessionFunction switches
from TenantHelper.GetTenantId (JWT-only) to RequestContext.TargetTenantId
(middleware-validated).

F2 — RegisterSessionFunction wrote sessions without any deletion-state
check; a mid-cascade register could mutate a Preparing/Queued/Running
row, and a fresh register past the tombstone could re-Add a row the
manifest no longer covers. Guard now runs at the entry of the shared
ProcessRegisterAsync core — covering BOTH /agent/register-session
(cert-auth) AND BootstrapRegisterSessionFunction (bootstrap-code-auth),
which previously bypassed the cert-auth entry's check. 410 Gone on
lock-state or tombstone hit.

F3 — SessionDeletionGuard's "Sessions row missing -> silent pass"
contract let late ingests / registers slip past a completed cascade
and orphan rows the manifest couldn't describe. New short-lived
SessionTombstones table (PK=tenantId, RK=sessionId) carries a marker
written by SessionDeletionHandler.ExecuteTombstoneAsync BEFORE the
Sessions-row delete, with a 7-day default ExpiresAt. Guard now reads
the marker on a missing Sessions row; live and present -> throws with
sentinel state "Tombstoned" -> 410 Gone. Full-Restore deletes the
marker after re-inserting the Sessions row; partial-restore leaves it
alone (never written). SessionDeletionMaintenanceFunction pruning
step runs even when the kill-switch is active (markers are race
shields, not policy artefacts).

F4 — CrossTenantAccessTests had no coverage for DELETE
sessions/{sessionId}. Theory entries lifted into the same three asserts
that already cover GET — same-tenant allowed, cross-tenant non-GA
blocked, cross-tenant GA allowed — and into the QueryParam-scoping
catalog theory so the regression that produced F1 surfaces in CI.

Bootstrap follow-up — RegisterSessionFunction.Run had the guard inline
but BootstrapRegisterSessionFunction calls _inner.ProcessRegisterAsync
directly, bypassing it. Guard moved into ProcessRegisterAsync so both
entry points hit it before StoreSessionAsync. Added structural wiring
tests (DI field, internal accessibility, bootstrap delegation type)
that fail in CI if the architecture regresses to inline copy.

Tests:
- SessionDeletionGuardTests +3: marker-active blocks, no-marker passes
  on missing row, sessions-row-exists skips tombstone probe.
- SessionDeletionHandlerTests +1: marker is written BEFORE the
  Sessions-row delete (Moq Callback ordering assertion).
- SessionRestoreServiceTests +2: full-restore deletes marker,
  partial-restore does not.
- SessionDeletionMaintenanceFunctionTests +2: prunes expired markers,
  pruning runs even with kill-switch active.
- CrossTenantAccessTests +3 inline DELETE cases.
- RegisterSessionGuardWiringTests (NEW): asserts guard DI field,
  ProcessRegisterAsync internal accessibility, BootstrapRegisterSession
  delegation type.
- CascadeVerificationServiceTests.FakeReader gains the new
  GetActiveSessionTombstoneAsync interface member.

Suites: Backend xUnit 1524, Agent V1 530, Agent V2 1042 — all green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 12:08:28 +02:00
Oliver Kieselbach 921994e845 fix(backend): cascade-delete codex follow-ups (F1 V2 guard + upsert-revival, F2 partial-restore key-level re-increment, F3 legacy delete sessions-first, F4 restore 400 EntityAlreadyExists) 
F1 — V2 ingest writes were landing past the cascade lock. Wired
SessionDeletionGuard.EnsureWritableAsync into IngestTelemetryFunction
before PersistItemsAsync so Event / Signal / Transition writes are
refused with 410 Gone when DeletionState is Preparing / Queued /
Running / Poisoned. Switched UpdateSessionImeAgentVersionAsync from
UpsertEntityAsync to UpdateEntityAsync(ETag.All, Merge) with 404
swallow so a tombstoned Sessions row can no longer be silently
resurrected by an in-flight ingest.

F2 — Partial-restore could leave SoftwareInventory at -N drift when
the cascade poisoned mid-AGGREGATE-loop: per-key decrement was
persisted into AggregateDecrementsApplied BEFORE the underlying
decrement, but the gate used CompletedSteps.Contains(aggregate.Order)
— which is only set AFTER the whole loop. Switched the gate to
AggregateDecrementsApplied as the authoritative ledger and pass a
filtered DeletionStep into RunInventoryReIncrementWithIdempotencyAsync
so only the keys that were actually decremented are re-incremented.

F3 — Legacy direct-delete path deleted Events / RuleResults /
AppInstallSummaries before the ETag-CAS-guarded DeleteSessionAsync,
leaving a TOCTOU window where a V2 producer could CAS-claim
DeletionState=Preparing after the pre-read but before our side-
table mutations — landing the producer's manifest snapshot on top
of already-half-purged state. Reordered both admin (DeleteSessionFunction)
and retention (SessionRetentionFanoutService) paths to Sessions-CAS
first; side-tables only after the Sessions row is gone (no producer
can claim a missing row).

F4 — Restore per-row Add fallback only caught HTTP 409 for duplicate-
row conflicts; Azurite and some legacy Azure deployments surface
duplicate-row as 400 with ErrorCode=EntityAlreadyExists, breaking
idempotent restore-retry after partial-failed restore. Centralised
the check via IsAlreadyExistsStatus(rfe) helper accepting 409 OR 400
+ ErrorCode=EntityAlreadyExists; non-duplicate 400s still propagate.

Tests:
- SessionRestoreServiceTests: 2 partial-restore tests refit to the new
  AggregateDecrementsApplied contract + new explicit Codex-scenario
  test (only-decremented-keys re-incremented, untouched key skipped).
- SessionDeletionPr4cCodexFixesTests F4: harness setup adds full
  AggregateDecrementsApplied so the per-key idempotency assertion
  reflects the new gate.
- SessionRetentionFanoutServiceTests: pre-read tests replaced by two
  tests covering "DeleteSessionAsync refuses → no side tables" and
  "DeleteSessionAsync succeeds → side tables run after".
- TableStorageRestoreBatchTests: two new F4 tests — 400
  EntityAlreadyExists treated as Skipped, 400 with non-duplicate
  ErrorCode still propagates.
- UpdateSessionImeAgentVersionTests (NEW): asserts UpdateEntityAsync
  is used (not Upsert), 404 swallowed silently, other storage errors
  non-fatal.

Suites: Backend xUnit 1510, Agent V1 530, Agent V2 1042 — all green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 10:44:54 +02:00
Oliver Kieselbach 4c676d076a fix(backend,web): PR6 codex-review follow-ups (F1 legacy pre-check, F2 mid-run kill-switch, F3 OpsEvent audit shift) 
Three Round-1 review findings from PR6:

  F1 (High) — Legacy retention path mutated side-tables before the
  CAS-guarded DeleteSessionAsync. Plan PR6 step 7 says "legacy must skip
  locked sessions", but our LegacyDeleteAsync deleted Events / RuleResults
  / AppInstallSummaries first and only discovered the V2 cascade lock at
  the tombstone. Fix: pre-read the Sessions row in LegacyDeleteAsync; if
  DeletionState != None, return false WITHOUT touching any side table.
  The original CAS-guard stays as a backstop for the T1-T3 race after the
  pre-read.

  F2 (High) — Kill-switch checked once per run; a flip-ON mid-fanout kept
  legacy deletes running. SessionRetentionFanoutService now takes
  AdminConfigurationService, calls IsSessionDeletionKillSwitchActiveAsync
  (uncached, fail-closed per PR5 F1) at both the per-tenant boundary and
  inside the per-session loop. FanoutResult gains AbortedByKillSwitch so
  the per-tenant audit + the global OpsEvent surface the early halt.

  F3 (Medium) — Global-scope lifecycle audits used LogAuditEntryAsync with
  tenantId: null!, which the table-storage helper silently dropped because
  the schema requires a non-null PartitionKey. Three event types
  (Completed / FanoutSkipped / Failed) moved to OpsEvents (TenantId is
  nullable on OpsEventEntry). Two new helpers in OpsEventService —
  RecordSessionDeletionMaintenance{Completed,FanoutSkipped}Async — also
  dual-registered in the OpsAlertRulesSection.tsx catalog per memory
  feedback_ops_event_types_dual_register. Per-tenant audits inside the
  fanout (which already had a real tenantId) are unchanged.

Tests: 4 new xUnit cases — F1 skip-without-mutation, F1 row-not-found
skip; F2 mid-tenant abort, F2 between-tenants abort. Existing 17 PR6
tests updated to assert OpsEvents instead of audit calls for the global
lifecycle events. Total: 1504/1504 backend pass, 347/347 web pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 02:41:45 +02:00
Oliver Kieselbach 50071c5e42 feat(backend,web): dedicated cascade-delete maintenance function (PR6/7) 
Plan §5 PR6 / §16 R14: extracts cascade-lifecycle maintenance out of the
2h Maintenance timer into a dedicated 12h SessionDeletionMaintenanceFunction
with independent cadence, kill-switch, and OpsEvent watchdogs. Adds four
work-blocks (Plan §10):

  1. Manifest-blob TTL sweep — defence-in-depth on top of Azure Blob
     Lifecycle Management. EnumerateOldDeletionManifestsAsync +
     DeleteDeletionManifestPairAsync (33-day effective window via the
     existing soft-delete tombstone, not permanent deletes).
  2. Stale-Preparing GC — reverts DeletionState=Preparing rows older than
     1h that have NO progress blob (producer crashed before upload). Skips
     when a progress blob exists; producer's own Preparing-resume path
     handles those.
  3. Stranded-Queued detection — alert-only OpsEvent per row in
     DeletionState=Queued for > 30min (no auto-clear; operator inspects).
  4. Retention fanout via new SessionRetentionFanoutService — per-tenant
     DataRetentionDays sweep, dispatched per EnableCascadeDeleteV2 flag:
     V2 → producer.EnqueueAsync, legacy → ISessionRepository.DeleteSessionAsync
     (now CAS-guarded since PR5 finding 2). Rate-limited to 100 enqueues
     per tenant per run with 50ms throttle.

Operational alerting:
  - 30min watchdog → OpsEventType SessionDeletionMaintenanceLongRunning (Warning)
  - 60min watchdog → SessionDeletionMaintenanceLongRunningSevere (Error)
  - Exception path → SessionDeletionMaintenanceFailed (Error) + re-throw
  - Per stranded row → SessionDeletionStrandedQueued (Warning)
  All four registered both in OpsEventService.cs and the OpsAlertRulesSection.tsx
  catalog so admin can wire Telegram per memory feedback_ops_event_types_dual_register.

Reuses PR5 finding 1's IsSessionDeletionKillSwitchActiveAsync so the
fanout-skip check honors flip-ON across scaled-out instances within seconds.

host.json: functionTimeout=01:00:00 (Flex Consumption max, matches the
60min severe watchdog so operator gets alerted before Azure hard-aborts).

Cron: const "0 0 */12 * * *" — matches the repo's IndexReconcileTimer
convention (no AdminConfiguration field; redeploy to change).

Removed: session retention loop from MaintenanceService.CleanupOldDataAsync
(both RunAllAsync and RunManualAsync paths now call CleanupOldUsageDataAsync
directly). Non-session housekeeping (rule-stats, cert-expiry, SignalR
quota, poison-queue watcher) untouched.

EnableCascadeDeleteV2 + SessionDeletionKillSwitch defaults stay false —
PR6 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation.

Backend xUnit: 1500/1500 pass (1483 + 17 new). Web vitest: 347/347 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 02:24:03 +02:00
Oliver Kieselbach 8134adf3e4 feat(backend,web): wire admin-delete + worker + UI for V2 cascade (PR5/7) 
Plan §5 PR5 (session-deletion-cascade-safe.md): flag-gated dispatch in
DeleteSessionFunction between the legacy direct-delete path and the V2
cascade producer; SessionDeletionWorker registered as a hosted service;
UI shows a "deletion queued" toast + per-row spinner while the worker
drains, then removes the row on the SignalR sessionDeleted event.

Includes the codex review follow-ups:
  F1 — IsSessionDeletionKillSwitchActiveAsync bypasses the 5-min admin
       config cache; fail-CLOSED on storage error so a flip-ON is honored
       across scaled-out Function-host instances within seconds (instead
       of minutes). Wired into DeleteSessionFunction, Producer, Worker.
  F2 — DeleteSessionAsync now reads DeletionState + ETag together with
       IndexRowKey, refuses on locked state, and ETag-conditionals the
       delete; closes the T2-T4 race where a concurrent V2 producer
       could orphan side-table rows. Caller in the legacy path
       disambiguates the bool via a re-read into 404 / 409 / 500.
  F3 — pendingDeletions threaded from useDeleteSession through page.tsx
       → SessionTable → SessionCell; action cell renders a disabled
       spinner instead of the delete button. Hook subscribes to
       sessionDeleted SignalR + polls GET /api/sessions/{id} every 60s
       as a fallback for missed events; only 404 clears the pending row.

EnableCascadeDeleteV2 + SessionDeletionKillSwitch both default false —
PR5 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation. PR6 wires the producer into the dedicated
SessionDeletionMaintenanceFunction next.

Backend xUnit: 1483/1483 pass. Web vitest: 347/347 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-13 01:08:04 +02:00
Oliver Kieselbach 5dbc44c4ec feat(backend): cascade-delete worker + restore + codex-fixes (PR4/7) 
PR4 ships the cascade-delete executor for the safe-cascade-delete plan: the
SessionDeletionWorker (BackgroundService queue poll-loop) plus the
SessionDeletionHandler (per-step execution, live verification, tombstone).
Worker is NOT registered in Program.cs — that lights up in PR5.

Three logical phases land together as one commit per Oliver's call:

PR4 — Worker + Handler + CascadeVerificationService
- SessionDeletionWorker: BatchSize=1, kill-switch entry guard, heartbeat task
  extending visibility every 60s during long handler runs, poison after
  max-dequeue
- SessionDeletionHandler: download manifest+progress, CAS Queued→Running
  (with Running→Running resume), per-step dispatch by Class, AGGREGATE
  decrement, live verification, ordered tombstone (SessionsIndex first,
  Sessions last)
- CascadeVerificationService: per-Class live residual scans (CveIndex is
  manifest-key-only per §12-Q8; AGGREGATE+FINAL skipped)
- NotifySessionDeletedAsync added to ISignalRNotificationService
- BlobStorageService: DownloadDeletionProgressAsync + ETag-CAS
  UpdateDeletionProgressAsync

PR4b — Restore endpoint
- POST /api/admin/sessions/{sessionId}/restore (GA-only via Policy Catalog)
- SessionRestoreService: dispatch table per §13.4, full + partial-poisoned-
  recovery modes
- TableStorageService.Restore.cs: RestoreRowsByExactKeysInBatchesAsync +
  ConvertFromPropValue + ConvertDumpToEntity (Sessions-row override forces
  DeletionState=None on restore so the row isn't permanently locked)
- RestoreSoftwareInventoryEntryByKeyAsync helper
- Worker MoveToPoisonAsync now CAS-transitions Running→Poisoned (E1 fix)
  so restore can dispatch into partial-restore mode

PR4c — Codex review fixes (6 findings)
- F1: AGGREGATE per-key idempotency via DeletionProgress.AggregateDecrementsApplied
  (persist-first/decrement-second bounds drift to +1 per crash vs unbounded -N)
- F2: TombstoneStarted flag closes the gap where sessions=null+completedAt=null
  blocked restore after a tombstone crash
- F3a: Sessions row inserted LAST in full restore (was first → trapped retry
  with RejectAlreadyAtOriginalState)
- F3b: RestoreMode.Full now 409-ignores (was throw-on-409); converges with
  Partial mode, makes retry-after-partial-failure idempotent
- F4: Partial-restore per-key re-increment idempotency via
  DeletionProgress.RestoreReIncrementsApplied
- F5: Poison transition pre-checks PendingDeletionManifestId so stale
  envelopes can't flip a fresh active cascade to Poisoned
- F6: DownloadDeletionManifestWithShaAsync exposes snapshot SHA; Handler +
  Restore both verify against progress.SnapshotSha256 (snapshot↔progress
  binding per plan §3)

DeletionProgress schema gains 3 additive optional fields (AggregateDecrementsApplied,
RestoreReIncrementsApplied, TombstoneStarted) with safe defaults so PR1-PR4
progress blobs deserialize cleanly.

Tests: 1457/1457 backend xUnit facts green (1391 pre-PR4 + 66 new across PR4,
PR4b, PR4c). Full solution build green. Program.cs has zero
SessionDeletionWorker references (PR5 territory stays untouched).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 23:44:20 +02:00
Oliver Kieselbach 522f0ae048 fix(backend): PR3 codex round-2 follow-ups (F2 ETag-CAS, F4 Preparing-resume) 
Round 2 of codex feedback flagged that the first follow-up wasn't tight
enough:

== F2 round 2 (HIGH): TOCTOU race still possible on Sessions row ==
The first fix only preserved DeletionState if it was non-empty on the
INITIAL read. The race remained: T1 StoreSessionAsync reads row →
DeletionState=None; T2 producer CAS-es None → Preparing; T3
StoreSessionAsync writes Replace → silently clobbers the lock. Codex
specifically asked for ETag/Retry oder Merge.

Fix: replace the unconditional UpsertEntityAsync with an ETag-bound CAS
write loop (bounded retry = 5 attempts). Each iteration re-reads the
CURRENT row to capture (a) the fresh ETag and (b) the live cascade-lock
columns, stamps the cascade-lock columns from that FRESH read onto our
entity (so the producer's intervening write isn't clobbered), then tries
UpdateEntity(etag, Replace). On 412/409 conflict: retry. Falls back to
AddEntity when the row was deleted between reads (tombstone race). The
preserved-fields extract from the initial read still runs as a
best-effort hint; the per-iteration fresh read is what actually wins
the race.

Replace mode is kept (not Merge) so registration can still clear stale
fields like FailureReason on a successful re-register. Only the
cascade-lock columns get the cross-iteration preservation treatment.

== F4 round 2 (MEDIUM): stranded Preparing with progress blob ==
If the prior producer call CAS'd None → Preparing AND uploaded snapshot
+ progress blobs AND audited deletion_started, then crashed before the
CAS Preparing → Queued step — the row stays in Preparing forever. Plan
§10 GC only clears Preparing-WITHOUT-progress-blob (intentional, to
avoid racing in-flight producers); the resume path I added before only
covered the Queued case. So this would sit until manual intervention.

Fix: in the WrongState=Preparing branch with non-empty
PendingDeletionManifestId, probe the snapshot blob via the new
BlobStorageService.DeletionSnapshotExistsAsync helper. If it exists,
the prior crash happened POST-upload — safe to finish the CAS
Preparing → Queued + send the queue message (resume from step 6). If
absent, the prior crash was PRE-upload — bail out as AlreadyInFlight
and let the §10 GC clean up after 1h.

Probe failure (e.g. transient blob storage outage) also falls back to
AlreadyInFlight rather than risk a half-completed recovery. Extracted
the queue-message-resume call into a shared helper
(ResumeBySendingQueueMessageAsync) so both Queued-resume and
Preparing-resume go through one audit-marker code path
(Reason="resume").

== Tests ==
- SessionDeletionPr3CodexFixesRound2Tests (4 new):
  - Stranded Preparing-with-blob → resumes (CAS Preparing→Queued runs,
    queue message sent, returns Enqueued + Reason=resume)
  - Preparing without blob → AlreadyInFlight, no CAS attempt
  - Snapshot existence probe throws → AlreadyInFlight, no CAS attempt
  - Resume CAS loses to concurrent writer → AlreadyInFlight, no queue
    message
- Backend suite: 1349/1349 green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 21:34:42 +02:00
Oliver Kieselbach dc56c8a73a fix(backend): PR3 codex-review follow-ups (F2 lock-clobber, F3 SchemaHash, F4 resume, F5 preview shape) 
Round-2 codex review on PR3 surfaced four real issues. Finding 1 (the
guard not being wired at the 9 writer sites) was the deferred-by-design
decision and stays scheduled for PR5.

== F2 (HIGH): StoreSessionAsync silently clobbered the cascade lock ==
Re-registration reads the existing Sessions row but built a fresh
TableEntity for the UpsertEntity (Replace) without carrying over
DeletionState + PendingDeletionManifestId. An agent restart during
Preparing/Queued/Running would have cleared the lock and let the
agent's subsequent writes race the in-flight cascade. Fix: extract
both columns alongside the other preserved fields and re-stamp on
the new entity before the Upsert. Comment marks the silent-clobber
risk so a future refactor can't quietly remove the preservation.

== F3 (HIGH): Producer invalidated the manifest SchemaHash ==
Builder generated a ManifestId, hashed the manifest content, then the
producer overwrote manifest.ManifestId. SchemaHash was now stale
(computed against the old ID); a future hash-verification path on the
worker would treat legitimate manifests as tampered/corrupt. Fix:
DeletionManifestBuilder.BuildAsync accepts an optional
preAllocatedManifestId so the producer's CAS-stamped ID lands on the
manifest BEFORE the hash computation. Producer no longer overwrites.
Preview path passes null (transient ID).

== F4 (MEDIUM): No resume path for stranded Queued state ==
If CAS Preparing→Queued succeeded but SendMessageAsync failed
(network blip, queue throttling), the Sessions row sat in Queued state
with no live queue message. A retry just returned AlreadyInFlight and
left the cascade stranded. Fix: producer's WrongState=Queued branch
re-sends the queue envelope with the EXISTING manifestId (stamped
with reason ":resume" for audit traceability), returns Enqueued.
Idempotent — duplicate envelopes are tolerated by PR4's worker via
the manifest-bound progress blob. Preparing + Running unchanged
(let GC handle Preparing; trust the worker for Running).

== F5 (MEDIUM): mode=full preview wrapped manifest in an envelope ==
Plan §5 PR3 promises mode=full returns "the entire manifest JSON
snapshot inline ... identical structure to what would be uploaded
to deletion-manifests blob container" — but the response wrapped the
manifest in { success, mode, manifest } so byte-comparison against
a real cascade's snapshot was impossible. Fix: mode=full now returns
the RAW manifest JSON as the body. Side-channel data (in-flight hint,
builder duration, schema hash, manifest id) moves to response headers
(X-Deletion-Preview-*). Only mode=summary still uses the envelope
shape (it's a summary, not a snapshot).

== Tests ==
- SessionDeletionPr3CodexFixesTests (5 new): pin F3 hash consistency
  with + without preAllocatedManifestId, F4 resume re-sends queue
  message with existing ManifestId, F4 Running + Preparing do NOT
  re-send.
- Updated 1 existing producer test: AlreadyInFlight assertion now
  uses Running state (Queued is the resume path post-F4).
- Backend suite: 1345/1345 green.

== Finding 1 (guard wiring) ==
Confirmed deferred to PR5 per the original PR3 scope decision. PR5
wires the producer call into DeleteSessionFunction AND the 9 guard
sites in the same PR. Cascade can't fire until then, so the guard
being dormant in PR3 has zero runtime impact.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 21:18:40 +02:00
Oliver Kieselbach 257c5c4354 feat(backend,web): cascade-delete state-machine + guard + producer + preview (PR3/7) 
PR3 of the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). Substantial PR per
plan §16 — adds the concurrency state-machine, the single-chokepoint
SessionDeletionGuard, the producer that builds manifest + uploads
blobs + enqueues the worker envelope, and the read-only preview
endpoint Global Admins use to verify a cascade plan before flipping
EnableCascadeDeleteV2 on. Wiring the guard at the 9 writer sites is
intentionally deferred to PR5 — no cascade actually fires until PR5
calls the producer from DeleteSessionFunction, so PR3 ships as
infrastructure-only.

== State machine + guard infrastructure ==

SessionDeletionState (Shared, Plan §1 P7 / §16-R2):
  None | Preparing | Queued | Running | Poisoned. Completed deliberately
  omitted — the cascade FINAL step removes the row, so "completed" is
  observable via row-absence + progress.completedAt, not via a stable
  state value. IsValidTransition(from, to) enforces the legal edges
  (None→Preparing, Preparing→{Queued,None,Poisoned}, Queued→{Running,
  Poisoned}, Running→Poisoned, Poisoned→None) and treats null/empty
  legacy rows as None.

SessionDeletionLockedException carries (TenantId, SessionId,
CallerContext, CurrentState, ManifestId) for audit + log lines.

SessionDeletionGuard (Plan §1 P7 + §16-R13):
  - ThrowIfLocked(TableEntity sessionRow, callerContext) — piggyback
    variant for hot paths that already loaded the Sessions row; zero
    added I/O. Caller must include DeletionState +
    PendingDeletionManifestId in its select projection.
  - EnsureWritableAsync(tenantId, sessionId, callerContext) —
    standalone variant for paths with no existing read; one Get
    round-trip via the PR1 ISessionDeletionInventoryReader seam.
  Both paths throw the same exception. Per §16-R13 framing the guard
  is "best-effort fast-fail, not safety" — correctness lives in §1 P4
  live verification + poison + §13 restore-from-poisoned (PR4).

== Sessions row state-machine columns ==

SessionSummary.DeletionState (string) +
SessionSummary.PendingDeletionManifestId (string?) — added to the DTO
+ MapToSessionSummary read path. Empty/null means no cascade in flight
(legacy rows; treated as None per the guard contract). The producer
writes both columns via direct CAS via the new helper; existing
session-write paths don't need to know about them in PR3.

TableStorageService.CasSetSessionDeletionStateAsync — new partial
helper. Read-then-CAS on Sessions row, returns a structured
SessionDeletionStateCasResult (Updated | SessionMissing | WrongState |
EtagConflict) so the producer can branch correctly. Stamps newManifestId
on transition to Preparing; clears it on transition back to None
(poisoned-recovery restore).

== Producer + preview endpoint ==

SessionDeletionProducer.EnqueueAsync(tenantId, sessionId, reason,
actor) — Plan §5 PR3 step sequence: kill-switch check → CAS None→
Preparing with new ManifestId → DeletionManifestBuilder.BuildAsync →
BlobStorageService.UploadDeletionManifestAsync → UploadInitial-
DeletionProgressAsync (new helper for the mutable .progress.json
sibling) → audit deletion_started → CAS Preparing→Queued → enqueue
SessionDeletionEnvelope to Constants.QueueNames.SessionDeletion.
Returns SessionDeletionEnqueueOutcome enum so PR5/PR6 callers
translate per their UX (admin → 409 on AlreadyInFlight, maintenance
→ skip+audit). Crash recovery contract: row stuck in Preparing with
no progress blob → maintenance GC clears after 1h (PR6); Queued/
Running/Poisoned NEVER auto-cleared.

GET /api/admin/sessions/{sessionId}/delete/preview?mode={summary|full|
download} — GlobalAdminOnly per the EndpointAccessPolicyCatalog (per
memory feedback_route_policy_catalog). Three modes:
  - summary: preflightCounts + sample (PK, RK) per step + estimated
    snapshot bytes + builderDurationMs + schemaHash.
  - full: entire manifest JSON inline (every row dump, every decrement
    key) — byte-level review of "what would be deleted".
  - download: gzipped *.snapshot.json.gz with Content-Disposition
    attachment, identical bytes to what the real cascade would upload.
NEVER mutates state, NEVER uploads a blob, NEVER enqueues. Calls the
guard only as a hint (in-flight cascades surface in the response, no
lock acquired). Audits deletion_preview per Plan §7.

== Configuration changes (with web round-trip) ==

AdminConfiguration.SessionDeletionKillSwitch (bool, default false,
Plan §1 P8 / §9) — global emergency stop. When true, V2 producer +
legacy direct-delete return 503; PR4 worker pauses on entry. Round-
tripped through TableConfigRepository Store + Map AND the 4-file web
chain (AdminConfigContext.tsx, AdminConfigSettingsSection.tsx,
SectionGlobalSettings.tsx, types/adminConfig.ts) per memory
feedback_admin_config_ui_roundtrip — without it, every admin Save
silently clobbers the field back to false. UI surfaces a red checkbox
with explanatory copy.

TenantConfiguration.EnableCascadeDeleteV2 (bool, default false) —
per-tenant rollout flag. When true, PR5 admin-delete + PR6 retention
fanout build a manifest + enqueue; when false, legacy direct-delete
runs. Per-tenant flag selects PATH; global kill-switch halts BOTH.

Constants.QueueNames.SessionDeletion = "session-deletion" — worker
queue name.

Shared envelope SessionDeletionEnvelope { TenantId, SessionId,
ManifestId, Reason, EnqueuedAt } — wire shape between producer and
PR4 worker.

== DI ==

Program.cs registers SessionDeletionGuard + SessionDeletionProducer
as singletons next to the existing DeletionManifestBuilder.

== Tests (50 new, backend suite 1340/1340 green) ==

- SessionDeletionStateTests (10) — IsLocked over every state value
  including legacy null/empty, IsValidTransition over every legal
  edge, explicit rejection of illegal edges, "Completed is not a
  state" guarantee.
- SessionDeletionGuardTests (8) — null row no-op, every lock state
  throws with correct exception fields, missing-session no-op,
  context propagation in the exception.
- SessionDeletionProducerTests (7) — kill-switch returns
  KillSwitchActive without downstream calls; SessionMissing /
  AlreadyInFlight / Poisoned outcomes from CAS; happy path verifies
  all 7 steps in order via Mock.Verify; CAS-2 failure leaves snapshot
  uploaded but no queue message; producer's pre-allocated ManifestId
  flows to both the snapshot blob and the CAS marker (consistency
  invariant the worker depends on).
- AdminConfigurationSessionDeletionKillSwitchTests (6) — both flags
  default false on new config + on CreateDefault, true persists.

Wiring 9 sites with the guard (telemetry V1+V2 ingest, mark-success/
failed, server actions, maintenance marker, re-analyze handler,
vulnerability handler, manual rescan, index reconcile) deferred to
PR5 — no cascade fires in PR3, so the guard is dormant.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 20:41:32 +02:00
Oliver Kieselbach 66fc23b609 feat(backend): SoftwareInventory contributions side-row + cascade delete helpers (PR2/7) 
PR2 of the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). Adds the per-table
delete primitives the cascade worker will use in PR4 plus the §17
SessionInventoryContributions side-row that makes per-session inventory
counter increments at-most-once and exactly reversible. No production
caller invokes the cascade primitives yet; the inventory rewrite IS a
call-site change in VulnerabilityCorrelationService and ships in this PR.

== Cascade delete helpers (TableStorageService.Deletion.cs) ==
DeleteByExactKeysInBatchesAsync(tableName, keys) is the load-bearing
helper: groups by PartitionKey (Azure batch transactions are
partition-scoped), chunks at 100 actions per batch, falls back to
per-row DeleteEntityAsync with 404-ignore on whole-batch rollback.
Returns (Attempted, DeletedNow, AlreadyMissing) for accurate progress
accounting on idempotent re-runs of partial cascades. Plus 9 thin
wrappers per the §3 taxonomy: 7 keyed (EventTypeIndex, CveIndex,
SessionsByTerminal, SessionsByStage, DeadEndsByReason,
ClassifierVerdictsByIdLevel, SignalsByKind), 1 single-row
(DeviceSnapshot), and 2 enumerate-then-delete (Signals,
DecisionTransitions for the PK_BY_SESSION class). All fail-loud per
memory feedback_storage_helpers_fail_soft.

== SoftwareInventory side-row (TableStorageService.Inventory.cs) ==
Plan §17 design: each inventory-correlated session contributes ONE
SessionInventoryContributions row (PK=tenantId, RK=sessionId) with the
normalized software keys it added. Increments to SoftwareInventory
counters are now driven by deltas against this row, so:
  - WG sessions that fire software_inventory_analysis 4× no longer
    over-count by 4× per software per session
  - Cascade decrement is exact: the side-row is the source of truth
    for which counters this session contributed to
  - Pre-side-row sessions (created before this deploy) have no
    contributions row → cascade skips inventory entirely for them
    (drift accepted on the historical backlog, ages out via 90d
    retention).

Helpers (all fail-loud, all bounded retry per §12-Q10 = 10 attempts /
60s wall-clock cap, all ETag-CAS):
  - GetSessionInventoryContributionsAsync: returns prior key list or
    null on 404
  - UpsertSessionInventoryContributionsAsync: side-row CAS that
    computes (added, removed) delta against the new key set
  - DeleteSessionInventoryContributionsAsync: 404-ignore tombstone
  - IncrementSoftwareInventoryEntryAsync: ETag-CAS +1 with first-time
    AddEntity that writes full metadata
  - DecrementSoftwareInventoryEntryAsync: ETag-CAS -1, clamp >= 0,
    deletes the row when count hits zero

Codec: SoftwareKeysJsonCodec encodes the key list as raw JSON below
30 KB, gzip+Base64 above. Same shape DeletionManifestBuilder (PR1)
already reads at preflight, so the cascade decrement reads back what
the increment wrote with no schema gap. Plus deterministic ComputeDelta
(case-insensitive composite key) so concurrent correlations on the
same session collapse to a single counter update.

== VulnerabilityCorrelationService rewrite ==
The Task.Run fire-and-forget at the old line 246 now calls the new
ApplyInventoryContributionsAsync helper instead of
UpsertSoftwareInventoryAsync. Each Increment / Decrement is wrapped
in its OWN try/catch so one counter failure does NOT abort the rest
of the loop (Plan §17.3 + §17.8): without that, the side-row would
already promise the new key set while subsequent runs see an empty
delta and never retry the under-counted keys. Each individual failure
emits an `inventory_counter_write_failed` audit entry with operation
+ vendor + name + version + exception type/message via the existing
LogAuditEntryAsync. IMaintenanceRepository was added as an OPTIONAL
constructor param (positional default null) so existing 8-arg test
factories keep compiling unchanged.

ParseSoftwareInventoryItems exposed as internal static for testability;
preserves the same dedup + cpe-mapping behaviour as the old
UpsertSoftwareInventoryAsync path. The legacy method is left in place
on TableStorageService.Rules.cs for now (no remaining callers; PR7
cleanup can remove if desired).

== Tests (39 new) ==
- SoftwareKeysJsonCodecTests (12): raw vs gzip+Base64 paths,
  determinism, malformed-input rejection, delta cases (added /
  removed / identity / case-insensitive), composite-key shape.
- TableStorageDeletionBatchTests (8, SDK-mocked via internal test
  ctor): empty input, PK grouping (one batch per PK), chunking at
  100, batch 404 + all per-row 404, mixed batch 404 (2 succeed +
  3 missing) with accurate accounting, non-404 propagates, no
  over-deletion, ETag.All on every action.
- TableStorageInventoryCasTests (9): Increment first-create with
  full metadata, CAS +1 on existing row, 412 retry, 409 first-create
  race fall back to Update, empty-triple skip; Decrement 404 no-op,
  clamp-at-zero deletes the row, count>1 subtract, 0-count clamp,
  412 retry.
- VulnerabilityCorrelationServiceContributionsTests (10): Parser
  correctness, end-to-end add/remove flow with Mock.Verify counts,
  empty-input no-op, side-row exhaustion swallowed, per-counter
  failure does not abort + audits the failed key (both increment
  and decrement variants).

Backend suite: 1290/1290 green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 19:59:28 +02:00
Oliver Kieselbach c8658b9feb feat(backend): scaffold cascade-deletion manifest format + builder (PR1/7) 
Read-only foundation for the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). PR1 ships the manifest
shape, the read-only DeletionManifestBuilder enumerator, and the
gzip+SHA-256 blob upload/download plumbing. No production caller
invokes any of this yet; no existing endpoint behaviour changes.
PR2-PR7 wire the producer, worker, and call sites on top.

Manifest captures all 15 cascade tables per the §3 taxonomy (4
PK_BY_SESSION, 1 PROP_TENANT_PK, 3 PK_RK_EXACT, 1
DISCRIMINATOR_PK_RK_SUFFIX, 1 DISCRIMINATOR_PK_RK_EXACT, 5
DISCRIMINATOR_PK_PROP), the optional SoftwareInventory side-row
(omitted for pre-side-row sessions per §17.5), and the FINAL tombstone
step ordered SessionsIndex-then-Sessions. Row dumps tag every column
with its EDM type (String / Int32 / Int64 / Double / Boolean /
DateTime / Guid / Binary) so DateTime / Guid / byte[] / long round-trip
restore-faithfully through the JSON snapshot. Falls back to a
SessionsIndex partition scan when Sessions.IndexRowKey is empty or
stale, so a partially-corrupted Sessions row can't strand a
SessionsIndex orphan after cascade.

BlobStorageService.UploadDeletionManifestAsync compresses with gzip,
pins SHA-256 of the uncompressed payload to blob metadata, and uses
IfNoneMatch=* to enforce snapshot-write-once. DownloadDeletionManifest
re-verifies the hash and refuses on mismatch (corruption signal that
the §13 restore semantics rely on). Two protected-internal-virtual
seams isolate the Azure SDK calls so tests bypass BlobsModelFactory
entirely.

New SessionInventoryContributions table constant added to TableNames.All
so the builder can Try-Read it (404 = pre-side-row session = inventory
steps omitted). PR2 adds the side-row write path via
VulnerabilityCorrelationService.

Tests: 25 new xUnit cases (DeletionManifestSerializationTests,
DeletionManifestBuilderTests, BlobStorageServiceDeletionManifestTests)
cover counts, dumps with EdmType per Azure Tables type, gzip round-trip,
hash-mismatch + missing-metadata corruption signals, IfNoneMatch
overwrite refusal, SessionsIndex fallback (both empty-key and stale-key
paths), determinism across runs, and pre-side-row inventory skip.
Backend suite: 1251/1251 green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 19:13:21 +02:00
Oliver Kieselbach 7020a2e5a6 fix(backend): propagate device/version block+kill across Function App instances 
BlockedDeviceService and BlockedVersionService kept their cache per Function App
instance. Once an instance had lazy-loaded a tenant's block list, a cache miss
short-circuited to "not blocked" without ever re-consulting storage — so a Kill
(or Block) added on instance A stayed invisible to instance B for the lifetime
of the process. Live repro: device was killed at 13:11:46Z and kept posting
telemetry for 11+ minutes afterwards on every cadence tick.

BlockedDeviceService.IsBlockedAsync now has two safety nets to bridge other
instances to storage:
  - Cache miss after tenant was loaded -> storage point-read on the spot, then
    promote the result into the cache. Lets a freshly-set block from another
    instance reach this instance immediately.
  - Positive cache hit older than EntryRevalidateAfter (30s) -> storage
    point-read to re-confirm. Lets manual Unblock, Action upgrade (Block->Kill),
    and UnblockAt changes from another instance propagate within seconds.

BlockedVersionService CacheRefreshInterval lowered from 5min to 30s for the same
reason — a version-Kill should not take up to five minutes per instance.

IDeviceSecurityRepository.IsDeviceBlockedAsync gains a BlockedSessionIds tuple
slot so the cache-miss fallback preserves session-aware semantics instead of
silently degrading session-scoped blocks to whole-device.

Tests cover the cache-miss fallback for both Block and Kill actions, the
stale-positive Block->Kill upgrade path, and the stale-positive Unblock path.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 17:00:00 +02:00
Oliver Kieselbach 9826b0a321 fix(agent/v2,backend,web): flag in-flight installs as "likely stuck" on terminal ESP-Apps failure 
When the ESP times out on the Apps subcategory, V2 used to terminate the
session with the last in-flight app silently frozen in the Installing bucket;
the user saw a Failed enrollment but no app name and no app_install_failed
event. The agent has no way to tell whether the app eventually succeeded,
failed, or hung — so promote it as hedged "likely stuck" with confidence
"presumed" rather than a confirmed failure.

The promotion fires only on the explicit EspTerminalFailure pathway: a new
LastFailureTrigger fact on DecisionState lets EnrollmentTerminationHandler
discriminate against MaxLifetimeExceeded (watchdog notbremse), SessionAborted
(admin-kill) and EffectInfrastructureFailure (effect-runner gave up) —
none of those carry a meaningful claim about app outcome.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 13:04:39 +02:00
Oliver Kieselbach 9a45690062 fix(web/scripts): default-collapse remediation cycle cards so long lists stay scannable 
User reported the all-expanded default produced very long scrolly lists when a session
has many remediation policies — the header (Status: Compliant / Remediated successfully /
Remediation script failed) is the at-a-glance signal they actually want first; per-phase
details are a drill-in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 11:24:55 +02:00
Oliver Kieselbach 7e67a70756 fix(web/scripts): group remediation cycle phases under one parent card (Intune-cycle UX) 
Confirmed via real non-compliant cycle data from session 8810cf81 / policy 99e1274b:
detection (non-compliant) + remediation (script crashed exit 1) + post-detection (still
non-compliant) — three separate sibling cards for the same Remediation policy was
visually fragmenting and obscured that they all belong to one cycle.

New grouping:

- groupScriptItems() folds phases for the same (policyId, scriptType=remediation) into
  a single ScriptCard with phases[] sorted detection → remediation → post-detection.
  Platform scripts and detect-only / running remediations stay as single-phase cards.

- deriveRemediationHeader() computes the cycle-level outcome label + state with priority:
  Running → Failed (e.g. remediation script crash) → Non-compliant after remediation →
  Remediated successfully → Non-compliant (detect-only) → Compliant (detect-only).

- ScriptCardView wraps multi-phase cycles in a parent header (icon + "Remediation" +
  shortId + cycle badge + System/Target/Status meta + collapsible chevron) with the
  individual phase rows nested inside on a lighter inner background. Single-phase
  cards reuse the existing ScriptItemRow layout 1:1 (no parent wrapper) so the
  detect-only majority case stays unchanged.

- Header counter "succeeded / non-compliant / failed" now counts CARDS not phases, so
  a remediated cycle reads as "1 succeeded" rather than "1 succeeded + 2 non-compliant"
  from its detection / post-detection en route.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 11:09:37 +02:00
Oliver Kieselbach 76284cfcb2 fix(agent/v2,rules): two-stage health-script emission to close short-session timing gap 
The post-PR-HS1 single-source design (HS-NEW-RESULT JSON only) had a real timing gap:
IME emits the [HS] new result = JSON line only AFTER its batch send to the Microsoft
service, ~30-90 s after the actual script ran (verified in session c1e714e6: 34-74 s
gap per policy). Short Autopilot enrollments often end inside that window, so health-
script rows were entirely missing from completed sessions even though IME had executed
the scripts.

Two-stage approach (matches original V1 timing for the early signal, while keeping
HS-NEW-RESULT as the canonical truth):

- Re-add HS-COMPLIANCE pattern matching BOTH `pre-remdiation` AND `post-remdiation`
  detection compliance lines → HandleHealthScriptDetectionResult emits a script_completed
  event immediately. Single regex (`(?<part>pre|post)-remdiation`) covers both phases.

- Re-add HS-RUN-CONTEXT / HS-EXITCODE / HS-STDOUT / HS-STDERR patterns feeding a
  single-slot `_pendingHealthScript` accumulator (IME runs health scripts sequentially
  within a session — verified). HS-COMPLIANCE merges the slot data onto the early-signal
  event so exit / stdout / stderr / runContext are populated from the start, not just
  the inferred compliance verdict.

- HS-NEW-RESULT continues to fire later with the full JSON (RemediationStatus,
  TargetType, ErrorCode, plus the actual remediation phase's data). UI reducer's
  dataCompleteness() scoring keeps the more complete entry — early-signal row gets
  enriched in place, no duplicate cards.

Remediation phase itself has no early-signal source in IME logs (only the late JSON
exposes Info.RemediationExitCode and RemediationScriptOutputDetails). A follow-up
based on the agentexecutor invocation line's _RemediationScript.output suffix is
possible once we have a real non-compliant cycle log to validate against.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-12 00:58:21 +02:00
Oliver Kieselbach 33023ecfc8 fix(agent/v2,web/scripts): UX polish + dedupe duplicates + stderr-as-failure rule 
UX fallout from session e2929c97 debrief:

- Dedupe: same (policyId, scriptType, scriptPart) re-emissions collapse into one row
  keeping the most-complete entry via dataCompleteness() score. Real-world Autopilot
  pattern is IME re-emitting the same script across ESP-phase transitions, sometimes
  with degraded data on the second emission. Trade-off vs. Codex finding 3 (preserving
  distinct runs of scheduled scripts in long sessions) — out of scope for this UI.

- Labels: use Intune Admin Center terminology — all health-script rows now read
  "Remediation" with phase shown via separate badge ("detection" / "remediation" /
  "post-detection"). The customer-facing word for these policies in Intune is
  "Remediation", not "Health Detection".

- Flicker fix: React key included firstSeenIndex which was reassigned on every
  reducer run, causing ScriptItemRow to remount on upstream prop changes (live
  polling / SignalR) and collapse expanded detail panels. New scriptItemKey() uses
  only identity-shaped fields (policyId, scriptType, scriptPart). firstSeenIndex
  field removed from ScriptItem entirely.

- Animation polish: removed animate-pulse from Running container — only the inline
  spinner SVG conveys motion now. Container is neutral gray. With multiple concurrent
  Running cards the full-box pulse made the page feel unstable.

- stderr-as-failure rule (consistent across script types): stderr present →
  script_failed regardless of exit code or compliance verdict. Eliminates the
  inconsistency where a platform script with stderr showed red but a remediation
  detection with stderr (and exit 0 + compliant) showed green for the same data
  shape. Implemented in BOTH the Agent adapter and the Web reducer.

Agent V2 only — V1 phasing out.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 23:26:44 +02:00
Oliver Kieselbach 527e36b957 fix(rules/schema): allow healthScriptResult action in ime-log-pattern enum 
CI ajv validation rejected HS-NEW-RESULT.json because the new healthScriptResult
action wasn't in the schema's enum allowlist. Add it next to the other script*
actions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 22:41:19 +02:00
Oliver Kieselbach 594b3906ce feat(agent/v2,web): capture full health-script lifecycle with phase-aware semantics + live indicator 
Replaces the line-by-line HS-COMPLIANCE / HS-EXITCODE / HS-STDOUT / HS-STDERR /
HS-RUN-CONTEXT pattern set (only ever captured the pre-detection phase, reset state
mid-cycle) with a single atomic source: HS-NEW-RESULT matches IME's "[HS] new result =
{JSON}" line and emits up to three script_completed events per policy run (pre-detection
/ remediation / post-detection). HS-SCRIPT-START now also fires a live script_started
event so the UI shows a "running" indicator immediately, before the consolidated final
result lands ~30s-3min later.

Phase-aware failure semantics: detection / post-detection scripts use exit code as a
compliance verdict (not a crash signal), so a non-compliant detection that gets
remediated no longer counts as a "failed" event in metrics — the row renders with
amber styling via isNonCompliantReport(). Only the remediation phase + platform
scripts route non-zero exit to script_failed.

Defensive against wire-format drift: regex uses [\s\S]* with end-anchor, JSON parsed
with try/catch + counter, frontend toNumber() accepts both numeric and string fields
(backend serializes ints as strings via Dictionary<string,string>). Reducer dedupes
with timestamp included so repeated runs of the same policy in long sessions don't
collapse into one row.

V2 only — V1 (Agent.Core) phasing out.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 22:36:49 +02:00
Oliver Kieselbach eacef9c38a feat(notifications): opt-in "enrollment started" webhook on session registration 
Fires Teams/Slack alert at backend session registration. Default off so
existing tenants don't get a surprise notification storm. Triggers on
fresh session OR WhiteGlove Part 2 resume; agent restart and
re-registration of terminal sessions are skipped via pre-read of the
session row before StoreSessionAsync.

NotifyOnWhiteGloveResume constant in RegisterSessionFunction lets us
disable the resume signal with a 1-line patch if it lands badly with
end users, without removing the fresh-session signal.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 19:02:59 +02:00
Oliver Kieselbach fce4a03d23 feat(sla): persistent SlaTenantStatus row replaces in-memory throttle, fixes 2h breach-notification spam 
Background
----------
The in-memory SlaNotificationThrottleService kept its 4h cooldown in a process-
local ConcurrentDictionary. On Flex Consumption the Function host recycles
between 2h timer fires, wiping the dictionary, so the throttle was effectively
a no-op — observed in production as tenant b54dc1af receiving an SLA breach
notification every 2 hours for days on end (1 failed of 11-13 sessions, target
95%, success rate 91-92%).

Design
------
One row per tenant in a new SlaTenantStatus table (PartitionKey = tenantId,
RowKey = "status"). Flat schema with {Type}_* property prefixes for the four
breach kinds (SuccessRate, Duration, AppInstall, ConsecutiveFailures); one row,
one Upsert per evaluation cycle. Rationale documented inline: per-type rows
would be over-normalised for the access pattern (always all-types-per-tenant),
and ~50 tenants x 4 types = max 200 rows ever.

Throttle correctness
--------------------
The throttle decision (now - LastNotifiedAt >= cooldown) and the LastNotifiedAt
write happen in the same TryUpsertAsync(status, ifMatchETag) call. Two parallel
inline calls cannot both succeed in sending the same notification — the loser
gets a 412 from Table Storage, refetches, sees the updated watermark, and
either skips or schedules a new claim attempt. Up to 4 retries; on exhausted
retries we drop the notification rather than risk double-fire (at-most-once).

Cooldown is configurable: AdminConfiguration.SlaNotificationCooldownHours
(default 24, range 1-168). UI round-trip wired through the four required
files (AdminConfigContext, AdminConfigSettingsSection, types/adminConfig.ts,
SectionGlobalSettings).

Resolve path
------------
When a previously-active breach is no longer detected, the row transitions to
IsActive=false + ResolvedAt=now and a single sla_resolved notification fires.
Orphan sweep: tenants whose toggles get turned off while a breach is active
fall out of the qualifying loop, so the timer now also scans every status row
and silently clears any IsActive that no longer has a toggle backing it
(no resolved notification — the tenant disabled SLA, they don't care).

Per-type notes
--------------
- SuccessRate breach notifies with the configured warn-threshold, not the SLA
  target — a 94% rate against target=99% / threshold=95% reads "below 95%",
  not "below 99%".
- AppInstall now actually evaluates (the toggle was wired but evaluation was
  missing). Aligned to the same ISO-week window the dashboard uses
  (SlaMetricsService.GetIsoWeekKey), so notifications cannot diverge from the
  /sla page. Sample-size floor of 5 apps per week, same as the dashboard.
- AppInstall has its own notification template (own title/summary/facts plus
  TopFailingApp fact) — the builder used to silently fall through to the
  Duration template for everything not SuccessRate.
- Telemetry Period tag derived from breachType: CurrentMonth for SuccessRate
  and Duration, CurrentWeek for AppInstall — so AppInsights / ops dashboards
  don't group AppInstall under the wrong period.
- AppInstall ops events now carry the actual installer totals (10 evaluated,
  4 failed) instead of 0/0.
- ConsecutiveFailures: inline (ingest) path detects breaches, timer path
  detects resolve. Both paths write to the same row through the same CAS
  retry loop.

Clock injection
---------------
SlaBreachEvaluationService takes an optional Func<DateTime> nowProvider
(default () => DateTime.UtcNow) so the week-window regression test runs
deterministically on every test day instead of silently skipping when today
happens to fall in the first ISO week of the month.

Removed
-------
- SlaNotificationThrottleService and its tests. The in-memory model is gone;
  there is no scenario where it would be wanted alongside the persistent row.

Tests
-----
- 21 new lifecycle + mapping + telemetry tests (CAS conflict + retry, CAS
  exhausted-retries drops notification, threshold-not-target in notification,
  orphan clearing, AppInstall template / week-window / totals, ConsecutiveFailures
  inline persistence, multi-type race-probe, period-mapping theory).
- Backend xUnit: 1218 passed.
- Web vitest: 266 passed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 18:47:58 +02:00
Oliver Kieselbach 7d05a25c90 fix(dashboard/stats): move stats routes out from under sessions/ to dodge Azure Functions route collision 
Azure Functions' attribute router does not honour literal-vs-parametric
specificity the way ASP.NET attribute routing does. A literal route of
"sessions/stats" was being eaten by the sibling GetSessionFunction whose
template is "sessions/{sessionId}" — the runtime bound /api/sessions/stats
to GetSession with sessionId="stats", which then 404'd because "stats"
fails GUID validation.

Diagnosed live via App Insights:
  /api/global/sessions/stats -> 200 OK   -> Functions.GetAllSessionStats
  /api/sessions/stats        -> 404      -> Functions.GetSession  (!)

Per-tenant + GA stats routes moved out from under sessions/ for symmetry:
  /api/sessions/stats        -> /api/stats/sessions        (MemberRead)
  /api/global/sessions/stats -> /api/global/stats/sessions (GlobalAdminOnly)

EndpointAccessPolicyCatalog, api.ts, and the URL-builder tests follow.
Function-side + api.ts both carry a comment pinning the rationale so a
future "let's tidy these routes" refactor doesn't regress it.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 15:45:41 +02:00
Oliver Kieselbach af46160a65 fix(agent/v2): fail-soft runtime handoff so ASR-blocked installs don't strand devices 
A customer device 2026-05-11 failed bootstrap at the WMI runtime spawn:
returnValue=2 (Access Denied -- Defender ASR rule
d1e49aac-8f56-4280-b9ba-993a6d77406c "Block process creations originating
from PSExec and WMI commands"). The throw propagated to --install exit 1
-> bootstrap throw -> next IME run SKIPped via pre-flight, leaving the
device with payload deployed + BootTrigger task registered but no live
runtime until manual intervention.

--install now runs a two-tier fallback: try WMI Win32_Process.Create ->
try schtasks /Run on the BootTrigger task -> defer to BootTrigger on next
reboot. Exit code is 0 whenever deployment + Scheduled Task creation
succeeded; runtime-spawn failure is a WARNING, not fatal. The bootstrap
script's Get-Process probe (Install-AutopilotMonitor.ps1:280) is now the
canonical "no runtime came up" signal. Orchestration extracted to pure
function DecideRuntimeLaunchOutcome, unit-tested with 8 cases pinning
the contract (incl. ASR-rule GUID surfaced in diagnostic only when
returnValue=2).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 15:17:30 +02:00
Oliver Kieselbach 6c643068c7 fix(web/admin): expose PoisonQueueBacklog event types in OpsAlertRules UI 
The previous commit added the two new EventTypes server-side
(PoisonQueueBacklogHigh / PoisonQueueBacklogCritical) but the admin alert
rules section hardcodes its known-event-types catalog in the component
itself - so the dropdown didn't list them and operators couldn't wire
Telegram to the new alerts.

Adds both strings to the Security category, matching how the SignalR/Cert
event types were registered when those watchers landed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 14:40:56 +02:00
Oliver Kieselbach c8902de7c9 feat(backend/maintenance): poison-queue backlog watcher with Telegram alerts 
Extends the SignalR/Cert pattern to async-worker poison queues. The previous
PR made backlogs visible on /health/detailed - this one makes them actively
page the operator via the existing OpsAlertDispatch -> Telegram path.

New partial MaintenanceService.PoisonQueue.cs runs on the 2 h timer (and on
manual maintenance triggers) and emits two new EventTypes:
  - PoisonQueueBacklogHigh    (Severity=Warning, threshold >= 1)
  - PoisonQueueBacklogCritical (Severity=Error,   threshold >= 10)

Dedup mirrors the cert watcher: composite key {EventType}|{queueName} built
from today's Security OpsEvents -> one alert per queue per tier per UTC day.
A tier escalation (warning -> critical) fires a new alert because the
EventType changes; a stable backlog re-pings once per UTC day so it stays in
the operator's attention.

Constants + classifier + watch-list moved from HealthCheckService to the new
MaintenanceService partial - the live health card and the timer-driven
Telegram alert now classify identically (same canonical source). Boundary
and JSON-extractor tests follow the move; HealthCheck tests are reduced to
the card-form assertions they actually exercise.

Operator follow-up (manual, not in code): add two OpsAlertRules in the admin
UI - one per new EventType - to wire Telegram. Until then the watcher writes
OpsEvents but no message is sent, matching how SignalR/Cert alerts were
introduced.

Backend: 1199/1199 (1193 + 6 net from test resplit). Build 0/0.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 14:32:38 +02:00
Oliver Kieselbach bf974c4a57 fix(dashboard/stats): Active = InProgress only, drop Pending + Stalled 
Live data showed cross-tenant 7d breakdown of non-terminal sessions:
InProgress 11, Pending 43, Stalled 6 = 60 total. The Pending bucket
dominates for any tenant with regular WhiteGlove provisioning — those
sessions have completed pre-prov and are waiting for the end user to
power on the device, which can take days or weeks. Counting them under
"Currently enrolling" implies live activity that isn't happening.

Active now matches the literal in-flight status. Pending and Stalled
remain visible via the sessions list filters; if we want a richer
breakdown on the card row later, it can be split into separate cards
without further changing the Active semantics.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 14:22:37 +02:00
Oliver Kieselbach 2284100915 feat(backend/health): surface async-worker poison queue backlog in /health/detailed 
Three async workers (analyze-on-enrollment-end, vulnerability-correlate,
telemetry-index-reconcile) park 5x-failed messages on <name>-poison. Until now
there was no operator-visible signal when this happened: backlogs could grow
silently and sessions stayed un-analyzed / vulnerability reports were lost.

Adds a "Poison Queues" entry to the detailed health-check next to "SignalR
Quota" with the same shape: parallel probe via IPoisonQueueProbe, tri-tier
classifier (0=healthy, >=1 warning, >=10 critical, thresholds App-Setting
configurable via PoisonQueueWarningThreshold / PoisonQueueCriticalThreshold),
GA-only visibility. Per-queue 404 counts as zero because poison queues are
created lazily on first poison-move. Per-queue exceptions surface in details
as "error: <msg>" and force at least warning so a flaky Storage call cannot
mask a real backlog.

Web UI needs no change - the generic checks[] render loop on /health-check
picks up the new card automatically.

Backend: 1193/1193 (1178 + 15 new). Build 0 warnings, 0 errors.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 14:18:38 +02:00
Oliver Kieselbach 068321baaa fix(dashboard/stats): reset cards on scope change, pin "in flight" semantics 
Follow-up to b76a6cca. Two issues surfaced after deploy:

1. Toggling GA mode off left the cross-tenant numbers stuck on the cards.
   Hook now setStats(null) synchronously before refetching on every scope
   change, and the card render falls back to "..." whenever stats is null
   (covers initial load, post-toggle reset, and fetch errors — previously
   an error would silently show 0).

2. "Active Sessions" intent: the user wants in-flight sessions in the
   broader sense, which matches the original code. AggregateSessionStats
   counts InProgress + Pending (WhiteGlove waiting on user) + Stalled
   (>60min no progress, non-terminal, can heal back to InProgress).
   Test renamed + comment pins the rationale so a future "tighten to
   InProgress only" refactor knows what it's changing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 14:00:02 +02:00
Oliver Kieselbach b76a6ccaf0 fix(dashboard/stats): server-aggregate cards so they stop drifting with pagination 
The five dashboard cards (Active Sessions, Success Rate, Avg. Duration,
Total Today, Failed Today) were derived from whatever sessions the client
had paginated into memory. With default pageSize=10 and SignalR-driven
appends, "Total Today" would jump 40 -> 22 -> 20 within a single visit
as the list grew/shrank. "Last 7 days" was a printed label only - the
client compute had no 7-day filter at all.

Now:
- GET /api/sessions/stats?days=7 (MemberRead, JWT tenant)
- GET /api/global/sessions/stats?days=7&tenantId= (GlobalAdminOnly,
  optional tenant filter; absent = cross-tenant sum)
- TableStorageService.AggregateSessionStats drains SessionsIndex once
  over the window: UTC-midnight boundary for today, success rate over
  terminal sessions only (InProgress no longer drags it down), avg
  duration over Succeeded with DurationSeconds > 0 only.
- useDashboardStats hook owns the cards: initial fetch + debounced
  refetch on SignalR newSession/newevents in the active scope + forced
  refetch on reconnect to recover messages missed during an outage.
- Client compute removed from useDashboardFilters. New
  submittedTenantIdFilter state isolates typing-in-filter from
  submitted-filter so keystrokes never trigger backend round-trips.

Tests: 17 xUnit (aggregation invariants + days parsing) +
7 vitest (URL builder). Suites: backend 1177/1178 (unrelated
RawTablePaginationTests flake), web 266/266.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 13:40:40 +02:00
Oliver Kieselbach 5e2cdf0aad perf(agent/v2): pre-size hot-path event dictionaries 
Skip the default 0->3->7->17 Dictionary resize cascade on hot-path
event/audit-trail allocations by setting initial capacity + Ordinal
comparer. ~3k Gen0 allocs saved per session for an agent that runs
continuously on endpoints. Behaviour unchanged, 1295 tests green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-11 11:36:00 +02:00
Oliver Kieselbach 5f6de2c02c fix health dashboard 2026-05-10 23:56:23 +02:00
Oliver Kieselbach c101eda796 health check fix 2026-05-10 23:16:30 +02:00
Oliver Kieselbach 7da5d482f1 fix(mcp/deps): bump fast-uri 3.1.0->3.1.2 to patch 2 high-sev advisories 
Resolves GHSA-q3j6-qgpj-74h6 (path traversal via percent-encoded dot
segments) and GHSA-v39h-62p7-jpjc (host confusion via percent-encoded
authority delimiters). Transitive via @modelcontextprotocol/sdk -> ajv.
Lockfile-only change, no API surface impact.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-10 22:57:07 +02:00
Oliver Kieselbach 871ec31bf6 feat(backend/health): surface SignalR free-tier quota in /health/detailed 
HealthCheckService now runs a fourth check (SignalR Quota) alongside Storage,
Processing, and Agent Binaries. Live read of ConnectionCount (max/1h) and
MessageCount (total since 00:00 UTC) via the same IAzureMonitorMetricsReader
that backs the maintenance-time Ops watcher - no cache; the data is already
3-5min stale from Azure Monitor's side, and a /health/detailed re-check is
expected to give fresh values during incident response.

Reuses MaintenanceService's internal classifier (CalculatePercent,
ClassifySignalRQuotaTier) so the watcher and the dashboard agree on the
80%/95% boundaries down to the floor-vs-ceil rounding.

Tier mapping: None -> healthy, Warning -> warning, Critical -> unhealthy.
Worst-of-both-metrics wins. PerformAllChecksAsync now uses a tri-state
combiner (any unhealthy / any warning / else healthy) so a Warning tier
correctly tints the OverallStatus bar yellow instead of red.

Fail-soft:
- SignalRResourceId unset -> Status="unknown" with "watcher not configured"
  (excluded from OverallStatus combiner so dev/local stays healthy)
- Both metric reads null -> Status="unhealthy" pointing at the most likely
  cause (Function App MI missing 'Monitoring Reader' on the SignalR resource)
- One read null -> "no data" for that row, normal classification for the other

Frontend unchanged: the existing /health-check Backend Services card loop
renders the new check with Connections, Messages, and Resource as Details.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-10 22:55:42 +02:00