Plan §5 PR5 (session-deletion-cascade-safe.md): flag-gated dispatch in
DeleteSessionFunction between the legacy direct-delete path and the V2
cascade producer; SessionDeletionWorker registered as a hosted service;
UI shows a "deletion queued" toast + per-row spinner while the worker
drains, then removes the row on the SignalR sessionDeleted event.
Includes the codex review follow-ups:
F1 — IsSessionDeletionKillSwitchActiveAsync bypasses the 5-min admin
config cache; fail-CLOSED on storage error so a flip-ON is honored
across scaled-out Function-host instances within seconds (instead
of minutes). Wired into DeleteSessionFunction, Producer, Worker.
F2 — DeleteSessionAsync now reads DeletionState + ETag together with
IndexRowKey, refuses on locked state, and ETag-conditionals the
delete; closes the T2-T4 race where a concurrent V2 producer
could orphan side-table rows. Caller in the legacy path
disambiguates the bool via a re-read into 404 / 409 / 500.
F3 — pendingDeletions threaded from useDeleteSession through page.tsx
→ SessionTable → SessionCell; action cell renders a disabled
spinner instead of the delete button. Hook subscribes to
sessionDeleted SignalR + polls GET /api/sessions/{id} every 60s
as a fallback for missed events; only 404 clears the pending row.
EnableCascadeDeleteV2 + SessionDeletionKillSwitch both default false —
PR5 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation. PR6 wires the producer into the dedicated
SessionDeletionMaintenanceFunction next.
Backend xUnit: 1483/1483 pass. Web vitest: 347/347 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR4 ships the cascade-delete executor for the safe-cascade-delete plan: the
SessionDeletionWorker (BackgroundService queue poll-loop) plus the
SessionDeletionHandler (per-step execution, live verification, tombstone).
Worker is NOT registered in Program.cs — that lights up in PR5.
Three logical phases land together as one commit per Oliver's call:
PR4 — Worker + Handler + CascadeVerificationService
- SessionDeletionWorker: BatchSize=1, kill-switch entry guard, heartbeat task
extending visibility every 60s during long handler runs, poison after
max-dequeue
- SessionDeletionHandler: download manifest+progress, CAS Queued→Running
(with Running→Running resume), per-step dispatch by Class, AGGREGATE
decrement, live verification, ordered tombstone (SessionsIndex first,
Sessions last)
- CascadeVerificationService: per-Class live residual scans (CveIndex is
manifest-key-only per §12-Q8; AGGREGATE+FINAL skipped)
- NotifySessionDeletedAsync added to ISignalRNotificationService
- BlobStorageService: DownloadDeletionProgressAsync + ETag-CAS
UpdateDeletionProgressAsync
PR4b — Restore endpoint
- POST /api/admin/sessions/{sessionId}/restore (GA-only via Policy Catalog)
- SessionRestoreService: dispatch table per §13.4, full + partial-poisoned-
recovery modes
- TableStorageService.Restore.cs: RestoreRowsByExactKeysInBatchesAsync +
ConvertFromPropValue + ConvertDumpToEntity (Sessions-row override forces
DeletionState=None on restore so the row isn't permanently locked)
- RestoreSoftwareInventoryEntryByKeyAsync helper
- Worker MoveToPoisonAsync now CAS-transitions Running→Poisoned (E1 fix)
so restore can dispatch into partial-restore mode
PR4c — Codex review fixes (6 findings)
- F1: AGGREGATE per-key idempotency via DeletionProgress.AggregateDecrementsApplied
(persist-first/decrement-second bounds drift to +1 per crash vs unbounded -N)
- F2: TombstoneStarted flag closes the gap where sessions=null+completedAt=null
blocked restore after a tombstone crash
- F3a: Sessions row inserted LAST in full restore (was first → trapped retry
with RejectAlreadyAtOriginalState)
- F3b: RestoreMode.Full now 409-ignores (was throw-on-409); converges with
Partial mode, makes retry-after-partial-failure idempotent
- F4: Partial-restore per-key re-increment idempotency via
DeletionProgress.RestoreReIncrementsApplied
- F5: Poison transition pre-checks PendingDeletionManifestId so stale
envelopes can't flip a fresh active cascade to Poisoned
- F6: DownloadDeletionManifestWithShaAsync exposes snapshot SHA; Handler +
Restore both verify against progress.SnapshotSha256 (snapshot↔progress
binding per plan §3)
DeletionProgress schema gains 3 additive optional fields (AggregateDecrementsApplied,
RestoreReIncrementsApplied, TombstoneStarted) with safe defaults so PR1-PR4
progress blobs deserialize cleanly.
Tests: 1457/1457 backend xUnit facts green (1391 pre-PR4 + 66 new across PR4,
PR4b, PR4c). Full solution build green. Program.cs has zero
SessionDeletionWorker references (PR5 territory stays untouched).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR3 of the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). Substantial PR per
plan §16 — adds the concurrency state-machine, the single-chokepoint
SessionDeletionGuard, the producer that builds manifest + uploads
blobs + enqueues the worker envelope, and the read-only preview
endpoint Global Admins use to verify a cascade plan before flipping
EnableCascadeDeleteV2 on. Wiring the guard at the 9 writer sites is
intentionally deferred to PR5 — no cascade actually fires until PR5
calls the producer from DeleteSessionFunction, so PR3 ships as
infrastructure-only.
== State machine + guard infrastructure ==
SessionDeletionState (Shared, Plan §1 P7 / §16-R2):
None | Preparing | Queued | Running | Poisoned. Completed deliberately
omitted — the cascade FINAL step removes the row, so "completed" is
observable via row-absence + progress.completedAt, not via a stable
state value. IsValidTransition(from, to) enforces the legal edges
(None→Preparing, Preparing→{Queued,None,Poisoned}, Queued→{Running,
Poisoned}, Running→Poisoned, Poisoned→None) and treats null/empty
legacy rows as None.
SessionDeletionLockedException carries (TenantId, SessionId,
CallerContext, CurrentState, ManifestId) for audit + log lines.
SessionDeletionGuard (Plan §1 P7 + §16-R13):
- ThrowIfLocked(TableEntity sessionRow, callerContext) — piggyback
variant for hot paths that already loaded the Sessions row; zero
added I/O. Caller must include DeletionState +
PendingDeletionManifestId in its select projection.
- EnsureWritableAsync(tenantId, sessionId, callerContext) —
standalone variant for paths with no existing read; one Get
round-trip via the PR1 ISessionDeletionInventoryReader seam.
Both paths throw the same exception. Per §16-R13 framing the guard
is "best-effort fast-fail, not safety" — correctness lives in §1 P4
live verification + poison + §13 restore-from-poisoned (PR4).
== Sessions row state-machine columns ==
SessionSummary.DeletionState (string) +
SessionSummary.PendingDeletionManifestId (string?) — added to the DTO
+ MapToSessionSummary read path. Empty/null means no cascade in flight
(legacy rows; treated as None per the guard contract). The producer
writes both columns via direct CAS via the new helper; existing
session-write paths don't need to know about them in PR3.
TableStorageService.CasSetSessionDeletionStateAsync — new partial
helper. Read-then-CAS on Sessions row, returns a structured
SessionDeletionStateCasResult (Updated | SessionMissing | WrongState |
EtagConflict) so the producer can branch correctly. Stamps newManifestId
on transition to Preparing; clears it on transition back to None
(poisoned-recovery restore).
== Producer + preview endpoint ==
SessionDeletionProducer.EnqueueAsync(tenantId, sessionId, reason,
actor) — Plan §5 PR3 step sequence: kill-switch check → CAS None→
Preparing with new ManifestId → DeletionManifestBuilder.BuildAsync →
BlobStorageService.UploadDeletionManifestAsync → UploadInitial-
DeletionProgressAsync (new helper for the mutable .progress.json
sibling) → audit deletion_started → CAS Preparing→Queued → enqueue
SessionDeletionEnvelope to Constants.QueueNames.SessionDeletion.
Returns SessionDeletionEnqueueOutcome enum so PR5/PR6 callers
translate per their UX (admin → 409 on AlreadyInFlight, maintenance
→ skip+audit). Crash recovery contract: row stuck in Preparing with
no progress blob → maintenance GC clears after 1h (PR6); Queued/
Running/Poisoned NEVER auto-cleared.
GET /api/admin/sessions/{sessionId}/delete/preview?mode={summary|full|
download} — GlobalAdminOnly per the EndpointAccessPolicyCatalog (per
memory feedback_route_policy_catalog). Three modes:
- summary: preflightCounts + sample (PK, RK) per step + estimated
snapshot bytes + builderDurationMs + schemaHash.
- full: entire manifest JSON inline (every row dump, every decrement
key) — byte-level review of "what would be deleted".
- download: gzipped *.snapshot.json.gz with Content-Disposition
attachment, identical bytes to what the real cascade would upload.
NEVER mutates state, NEVER uploads a blob, NEVER enqueues. Calls the
guard only as a hint (in-flight cascades surface in the response, no
lock acquired). Audits deletion_preview per Plan §7.
== Configuration changes (with web round-trip) ==
AdminConfiguration.SessionDeletionKillSwitch (bool, default false,
Plan §1 P8 / §9) — global emergency stop. When true, V2 producer +
legacy direct-delete return 503; PR4 worker pauses on entry. Round-
tripped through TableConfigRepository Store + Map AND the 4-file web
chain (AdminConfigContext.tsx, AdminConfigSettingsSection.tsx,
SectionGlobalSettings.tsx, types/adminConfig.ts) per memory
feedback_admin_config_ui_roundtrip — without it, every admin Save
silently clobbers the field back to false. UI surfaces a red checkbox
with explanatory copy.
TenantConfiguration.EnableCascadeDeleteV2 (bool, default false) —
per-tenant rollout flag. When true, PR5 admin-delete + PR6 retention
fanout build a manifest + enqueue; when false, legacy direct-delete
runs. Per-tenant flag selects PATH; global kill-switch halts BOTH.
Constants.QueueNames.SessionDeletion = "session-deletion" — worker
queue name.
Shared envelope SessionDeletionEnvelope { TenantId, SessionId,
ManifestId, Reason, EnqueuedAt } — wire shape between producer and
PR4 worker.
== DI ==
Program.cs registers SessionDeletionGuard + SessionDeletionProducer
as singletons next to the existing DeletionManifestBuilder.
== Tests (50 new, backend suite 1340/1340 green) ==
- SessionDeletionStateTests (10) — IsLocked over every state value
including legacy null/empty, IsValidTransition over every legal
edge, explicit rejection of illegal edges, "Completed is not a
state" guarantee.
- SessionDeletionGuardTests (8) — null row no-op, every lock state
throws with correct exception fields, missing-session no-op,
context propagation in the exception.
- SessionDeletionProducerTests (7) — kill-switch returns
KillSwitchActive without downstream calls; SessionMissing /
AlreadyInFlight / Poisoned outcomes from CAS; happy path verifies
all 7 steps in order via Mock.Verify; CAS-2 failure leaves snapshot
uploaded but no queue message; producer's pre-allocated ManifestId
flows to both the snapshot blob and the CAS marker (consistency
invariant the worker depends on).
- AdminConfigurationSessionDeletionKillSwitchTests (6) — both flags
default false on new config + on CreateDefault, true persists.
Wiring 9 sites with the guard (telemetry V1+V2 ingest, mark-success/
failed, server actions, maintenance marker, re-analyze handler,
vulnerability handler, manual rescan, index reconcile) deferred to
PR5 — no cascade fires in PR3, so the guard is dormant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Read-only foundation for the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). PR1 ships the manifest
shape, the read-only DeletionManifestBuilder enumerator, and the
gzip+SHA-256 blob upload/download plumbing. No production caller
invokes any of this yet; no existing endpoint behaviour changes.
PR2-PR7 wire the producer, worker, and call sites on top.
Manifest captures all 15 cascade tables per the §3 taxonomy (4
PK_BY_SESSION, 1 PROP_TENANT_PK, 3 PK_RK_EXACT, 1
DISCRIMINATOR_PK_RK_SUFFIX, 1 DISCRIMINATOR_PK_RK_EXACT, 5
DISCRIMINATOR_PK_PROP), the optional SoftwareInventory side-row
(omitted for pre-side-row sessions per §17.5), and the FINAL tombstone
step ordered SessionsIndex-then-Sessions. Row dumps tag every column
with its EDM type (String / Int32 / Int64 / Double / Boolean /
DateTime / Guid / Binary) so DateTime / Guid / byte[] / long round-trip
restore-faithfully through the JSON snapshot. Falls back to a
SessionsIndex partition scan when Sessions.IndexRowKey is empty or
stale, so a partially-corrupted Sessions row can't strand a
SessionsIndex orphan after cascade.
BlobStorageService.UploadDeletionManifestAsync compresses with gzip,
pins SHA-256 of the uncompressed payload to blob metadata, and uses
IfNoneMatch=* to enforce snapshot-write-once. DownloadDeletionManifest
re-verifies the hash and refuses on mismatch (corruption signal that
the §13 restore semantics rely on). Two protected-internal-virtual
seams isolate the Azure SDK calls so tests bypass BlobsModelFactory
entirely.
New SessionInventoryContributions table constant added to TableNames.All
so the builder can Try-Read it (404 = pre-side-row session = inventory
steps omitted). PR2 adds the side-row write path via
VulnerabilityCorrelationService.
Tests: 25 new xUnit cases (DeletionManifestSerializationTests,
DeletionManifestBuilderTests, BlobStorageServiceDeletionManifestTests)
cover counts, dumps with EdmType per Azure Tables type, gzip round-trip,
hash-mismatch + missing-metadata corruption signals, IfNoneMatch
overwrite refusal, SessionsIndex fallback (both empty-key and stale-key
paths), determinism across runs, and pre-side-row inventory skip.
Backend suite: 1251/1251 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Background
----------
The in-memory SlaNotificationThrottleService kept its 4h cooldown in a process-
local ConcurrentDictionary. On Flex Consumption the Function host recycles
between 2h timer fires, wiping the dictionary, so the throttle was effectively
a no-op — observed in production as tenant b54dc1af receiving an SLA breach
notification every 2 hours for days on end (1 failed of 11-13 sessions, target
95%, success rate 91-92%).
Design
------
One row per tenant in a new SlaTenantStatus table (PartitionKey = tenantId,
RowKey = "status"). Flat schema with {Type}_* property prefixes for the four
breach kinds (SuccessRate, Duration, AppInstall, ConsecutiveFailures); one row,
one Upsert per evaluation cycle. Rationale documented inline: per-type rows
would be over-normalised for the access pattern (always all-types-per-tenant),
and ~50 tenants x 4 types = max 200 rows ever.
Throttle correctness
--------------------
The throttle decision (now - LastNotifiedAt >= cooldown) and the LastNotifiedAt
write happen in the same TryUpsertAsync(status, ifMatchETag) call. Two parallel
inline calls cannot both succeed in sending the same notification — the loser
gets a 412 from Table Storage, refetches, sees the updated watermark, and
either skips or schedules a new claim attempt. Up to 4 retries; on exhausted
retries we drop the notification rather than risk double-fire (at-most-once).
Cooldown is configurable: AdminConfiguration.SlaNotificationCooldownHours
(default 24, range 1-168). UI round-trip wired through the four required
files (AdminConfigContext, AdminConfigSettingsSection, types/adminConfig.ts,
SectionGlobalSettings).
Resolve path
------------
When a previously-active breach is no longer detected, the row transitions to
IsActive=false + ResolvedAt=now and a single sla_resolved notification fires.
Orphan sweep: tenants whose toggles get turned off while a breach is active
fall out of the qualifying loop, so the timer now also scans every status row
and silently clears any IsActive that no longer has a toggle backing it
(no resolved notification — the tenant disabled SLA, they don't care).
Per-type notes
--------------
- SuccessRate breach notifies with the configured warn-threshold, not the SLA
target — a 94% rate against target=99% / threshold=95% reads "below 95%",
not "below 99%".
- AppInstall now actually evaluates (the toggle was wired but evaluation was
missing). Aligned to the same ISO-week window the dashboard uses
(SlaMetricsService.GetIsoWeekKey), so notifications cannot diverge from the
/sla page. Sample-size floor of 5 apps per week, same as the dashboard.
- AppInstall has its own notification template (own title/summary/facts plus
TopFailingApp fact) — the builder used to silently fall through to the
Duration template for everything not SuccessRate.
- Telemetry Period tag derived from breachType: CurrentMonth for SuccessRate
and Duration, CurrentWeek for AppInstall — so AppInsights / ops dashboards
don't group AppInstall under the wrong period.
- AppInstall ops events now carry the actual installer totals (10 evaluated,
4 failed) instead of 0/0.
- ConsecutiveFailures: inline (ingest) path detects breaches, timer path
detects resolve. Both paths write to the same row through the same CAS
retry loop.
Clock injection
---------------
SlaBreachEvaluationService takes an optional Func<DateTime> nowProvider
(default () => DateTime.UtcNow) so the week-window regression test runs
deterministically on every test day instead of silently skipping when today
happens to fall in the first ISO week of the month.
Removed
-------
- SlaNotificationThrottleService and its tests. The in-memory model is gone;
there is no scenario where it would be wanted alongside the persistent row.
Tests
-----
- 21 new lifecycle + mapping + telemetry tests (CAS conflict + retry, CAS
exhausted-retries drops notification, threshold-not-target in notification,
orphan clearing, AppInstall template / week-window / totals, ConsecutiveFailures
inline persistence, multi-type race-probe, period-mapping theory).
- Backend xUnit: 1218 passed.
- Web vitest: 266 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three async workers (analyze-on-enrollment-end, vulnerability-correlate,
telemetry-index-reconcile) park 5x-failed messages on <name>-poison. Until now
there was no operator-visible signal when this happened: backlogs could grow
silently and sessions stayed un-analyzed / vulnerability reports were lost.
Adds a "Poison Queues" entry to the detailed health-check next to "SignalR
Quota" with the same shape: parallel probe via IPoisonQueueProbe, tri-tier
classifier (0=healthy, >=1 warning, >=10 critical, thresholds App-Setting
configurable via PoisonQueueWarningThreshold / PoisonQueueCriticalThreshold),
GA-only visibility. Per-queue 404 counts as zero because poison queues are
created lazily on first poison-move. Per-queue exceptions surface in details
as "error: <msg>" and force at least warning so a flaky Storage call cannot
mask a real backlog.
Web UI needs no change - the generic checks[] render loop on /health-check
picks up the new card automatically.
Backend: 1193/1193 (1178 + 15 new). Build 0 warnings, 0 errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two notification-bell React contexts polled GET /api/notifications and
GET /api/global/notifications every 60s regardless of activity, generating
3017+618 = 3635 calls/24h with near-zero hit rate (notifications are emitted
sparsely). SignalR was already used app-wide for live session updates; the
bell was the lone hold-out on a pull model.
Switch to the same push model. Steady-state traffic drops ~86% and the bell
updates sub-second instead of within 60s.
Architecture (Option C+):
- Mount -> 1x HTTP fetch (cold-start hydration via existing GET endpoint)
- SignalR onreconnected -> re-fetch (covers the disconnect window)
- SignalR push -> live append/dismiss
- No periodic polling, no setInterval
Audience-tier security: TenantNotificationAudienceCatalog still drives
visibility (Member vs Admin), but enforcement moves into SignalR group
membership: tenant-{id}-notify-member is joined by every tenant member,
tenant-{id}-notify-admin only by Tenant Admins / Global Admins. The
AddToGroup function fail-closes with 403 on Member-tier joins to the
admin group, so Admin-tier notifications cannot leak via push.
Backend
- ISignalRNotificationService extracted; SignalRNotificationService
implements it. Resolves the long-standing test-mocking blocker
(the real service's constructor validates AzureSignalRConnectionString
and BuildServiceManager(); previously you could not unit-test
consumers). Tenant + Global NotificationService consume the
interface; the three legacy callers (Analyze, Ingest, VulnCorrelate)
still bind to the concrete class -- migrate per test need.
- Six new push methods: SendTenantNotification{,Dismissed,DismissedAll}
and SendGlobalNotification{,Dismissed,DismissedAll}. Tenant variant
routes to the audience-correct group.
- Tenant + Global service push after a successful repo write
(fire-and-forget; never throws into the caller).
- SignalRGroupHelper: parsing handles the two new -notify-{member,admin}
suffixes; new IsTenantNotifyAdminGroup predicate; group-name builders.
- SignalRAddToGroupFunction: explicit 403 when a non-admin attempts
to join the admin-tier group.
Frontend
- TenantNotificationContext: setInterval gone. useSignalR() to
join member-group always, admin-group iff isTenantAdmin || isGlobalAdmin.
Listens for tenantNotification (append + dedupe), tenantNotificationDismissed
(filter), tenantNotificationsDismissedAll (clear). onreconnected
triggers re-fetch.
- GlobalNotificationContext: same pattern against global-admins group.
- app/layout.tsx: SignalRProvider hoisted above the notification
providers so useSignalR() resolves inside them.
Tests
- SignalRGroupHelperTests (8): parsing for all four group formats
incl. new -notify suffixes, IsTenantNotifyAdminGroup predicate.
- TenantNotificationServiceSignalRPushTests (8): audience-correct push
on Create (Admin / Member / Unknown -> Admin fail-closed), no-push
on repo failure, no-push on DismissAll(count=0), no-push on empty
tenantId.
- FakeSignalRNotificationService: in-memory spy implementing the
new interface; replaces the unmockable real service in tests.
- AuthFunctionSideEffectTests + TenantNotificationServiceTests:
constructor signatures updated for the new SignalR dep.
Suites
Backend xUnit: 1161/1161 (was 1146, +15 net).
Web vitest: 259/259 (unchanged; existing notification-API +
gate tests still cover).
Solution build: 0 warnings, 0 errors.
Expected post-deploy reduction (24h):
GET GetTenantNotifications: 3017 -> ~50
GET GetGlobalNotifications: 618 -> ~10
POST AddToGroup: ~246 -> ~350-450 (two joins per tab)
Net bell load: -86%, sub-second live UX.
Disconnect tradeoff (accepted): if SignalR drops without reconnect
(rare -- withAutomaticReconnect retries 1/2/4/8/16/30s ... forever),
the bell shows the snapshot from the last fetch but no live updates
until next reload. The initial-3-retry give-up path of the SignalR
client still gets the user a hydrated bell because the HTTP fetch
runs independently on mount.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hourly TimerFunction queries Azure Monitor for SignalR ConnectionCount (max
over last hour) and MessageCount (sum since UTC midnight) and emits Warning
(>=80%) / Error (>=95%) OpsEvents when the free-tier limits (20 connections,
20k messages/day) are approached. Mirror of the cert-expiry pattern - global
TenantId=null events, dedup'd to one emission per EventType per UTC day.
Fail-soft: skips silently when SignalRResourceId app setting is unset, logs
warning on Azure Monitor auth/query errors. Function App's managed identity
needs 'Monitoring Reader' on the SignalR resource.
Also called from the manual maintenance trigger so 'Run Maintenance Now'
exercises every health check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Browser RUM on portal/dashboard surfaced ~1.7s time-to-data driven by
sequential backend reads, two stray duplicate fetches, and a 2s SignalR
defer. Five focused changes:
- AuthContext: store the auth/me prefetch as a Promise instead of its
resolved value so fetchUserInfo awaits the in-flight request rather
than racing a second fetch when the prefetch hasn't settled yet.
- useTenantSecurityConfig: depend on user.isTenantAdmin/isGlobalAdmin/
role primitives instead of the whole user object so the effect stops
refiring when AuthContext swaps prefetched user for fresh user (the
duplicate /config/<t>/feature-flags call observed on /dashboard).
- SignalRContext: drop the hardcoded 2000ms setTimeout before
startConnection. The original "avoid competing with critical API
calls" rationale doesn't apply across origins (web is HTTP/2, API is
a different host); the delay only pushed live updates ~2s into the
load for no benefit.
- FeedbackFunction: keep the kill-switch check sequential, then run the
remaining four Table reads (role, tenant config, sessions existence,
feedback cooldown) in parallel via Task.WhenAll. Endpoint dropped
from ~1285ms (6 sequential reads on Flex Consumption) toward the
slowest single read.
- TimingAllowOriginMiddleware: emit Timing-Allow-Origin: * on every
function response so the portal's Resource Timing API can read API
payload sizes (transferSize / nextHopProtocol) for our own RUM —
cross-origin entries currently report 0/unknown.
Backend tests 1058/1058, Web tsc clean, vitest 249/249.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the fire-and-forget Task.Run inside EventIngestProcessor — which
Azure Functions scale-in could kill mid-flight, leaving sessions with no
vulnerability_report and forcing users to click "Analyze Now" — with a
dedicated queue + BackgroundService worker (vulnerability-correlate),
mirroring the analyze-on-enrollment-end pattern. Adds a V2-agent trigger
that emits a Part-1 software-inventory snapshot the moment Windows Event
62407 fires (WhiteGlove sealing success), so Part-1 vuln correlation
happens BEFORE the admin clicks Reseal — closing the gap that existed
because reboot_kill termination at sealing left no shutdown inventory
for the backend to correlate.
Backend
* IVulnerabilityCorrelateProducer + AzureQueueVulnerabilityCorrelate
Producer (fail-soft on agent's hot path).
* VulnerabilityCorrelateQueueWorker (BackgroundService, 5-attempt
visibility-timeout retry, poison-queue on exhaustion).
* VulnerabilityCorrelateHandler ports the full pipeline (correlate +
store + synth event + re-analyze enqueue + SignalR notify).
* IVulnerabilityInventoryLoader extracted from
GetVulnerabilityReportFunction so worker + manual rescan share one
loader; phase-aware filter via whiteGlovePart parameter +
includeAllPhases escape-hatch for the rescan UI.
* EventIngestProcessor cutover: drops three concrete-class deps
(_vulnerabilityCorrelation, _vulnRepo, _signalRNotification) and
two now-orphaned private methods. 561 → 388 LOC.
Agent V2
* WhiteGloveInventoryTrigger subscribes to EspAndHelloHost.WhiteGlove
Completed and calls AgentAnalyzerManager.RunWhiteGlovePart1Inventory
Snapshot via injected Action (no concrete-class coupling, trivially
mockable). Single-fire guarded by Interlocked.Exchange.
* IWhiteGloveCompletedSource interface keeps EspAndHelloHost internal
sealed while letting the trigger be unit-tested with a 4-line stub.
* AgentAnalyzerManager.RunWhiteGlovePart1InventorySnapshot runs ONLY
SoftwareInventoryAnalyzer (LocalAdmin/IntegrityBypass deliberately
skipped at WG hand-off — still SYSTEM context).
* EnrollmentTerminationHandler now also tags Part 2 (WhiteGloveCompleted
Part2 → wgPart=2); was previously untagged so the new phase filter
couldn't see it.
* SoftwareInventoryAnalyzer.AnalyzeAtShutdown gets per-WG-phase atomic
idempotency via Interlocked.CompareExchange, with reset-on-failure
so the EnrollmentTerminationHandler stays a working fallback if the
Trigger crashes.
Cross-phase contamination & out-of-order resilience
* Inventory loader filters by whiteGlovePart so a Part-2 correlation
cannot see Part-1 inventory items (and vice versa).
* Handler checks the existing report on BOTH Part-1 and Part-2 paths:
Part-2 merges with existing Part-1, Part-1 SKIPS its store when the
existing report already contains user_enrollment / both findings or
has scan_summary.phase_contributions.user_enrollment > 0 — protects
against late-arriving Part-1 retries overwriting the richer Part-2
or merged report after at-least-once redelivery.
* VulnerabilityCorrelationService.MergeReports gains NormalizeForMerge
that recursively unwraps Newtonsoft JObject/JArray/JValue to native
Dictionary/List/primitive shapes BEFORE the cast-based extract
helpers run. Without this the normal-order Part-2 merge silently
dropped Part-1 findings, because GetVulnerabilityReportAsync round-
trips through JsonConvert and the existing helpers' OfType / "as
Dictionary" casts produced empty results on JTokens.
Tests (+34 new across Backend & Agent V2.Core)
* VulnerabilityCorrelateEnvelopeSerializationTests — wire-format pin.
* VulnerabilityInventoryLoaderTests — phase filtering, includeAllPhases,
lenient long-parsing.
* VulnerabilityCorrelateHandlerSkipTests — out-of-order skip helper
against native, JArray, JObject, both phase tag, phase_contributions
counter.
* VulnerabilityCorrelationServiceTests — round-trip safety of
MergeReports against JsonConvert-deserialized existing reports.
* WhiteGloveInventoryTriggerTests — fire-once, dedup, dispose-then-late,
idempotent dispose, action-exception isolation, null-arg validation.
* SoftwareInventoryAnalyzerIdempotencyTests — Part-1 dedup, Part-2
dedup, Part-1 ≠ Part-2 independence.
Build green, 0 errors, 0 warnings. Backend 844/844 tests pass, Agent
V2.Core 825/825 tests pass (parallel-flaky tests pass isolated, see
feedback_v2_recoverypath_corrupt_snapshot_flake).
V1 ingest path (/api/agent/ingest) deliberately untouched per
project_v1_vuln_pipeline_gap — V1 is being phased out; fixing it
forward would inflate scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Until now, hardware-whitelist rejections were only visible to tenant admins
who actively navigated to the settings page or had a webhook configured.
Admins missed new rejected models if neither was the case.
This adds a parallel tenant-scoped notification pipeline (separate from the
existing GlobalAdmin-only feed): on the first-ever HardwareNotAllowed
distress per (tenantId, manufacturer-lower, model-lower), a persistent
bell notification is created with a deep link to the hardware-whitelist
page. Lifetime dedup via a dedicated tracker table makes it a one-shot per
model. The webhook path is unchanged and fires in parallel.
Routes are gated TenantAdminOrGA + TenantScoping.Jwt for now; an Operator
follow-up (new TenantAdminOrOperator policy) is tracked separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous design ran RuleEngine in Task.Run after HTTP 200 — Functions scale-in
could kill it mid-flight, leaving rule results un-persisted and forcing manual
"Analyze Now" clicks. New design enqueues onto analyze-on-enrollment-end and
runs the engine in a BackgroundService worker with visibility-timeout retry +
poison move after 5 attempts (mirrors telemetry-index-reconcile pattern).
Three trigger sites cut over: V1 ingest, V2 EventIngestProcessor primary, V2
post-vulnerability re-run. Manual "Analyze Now" remains as fallback.
Plus retry-semantics fix: RuleEngine.AnalyzeSessionAsync no longer swallows
storage exceptions in an outer try/catch (per-rule inner catch stays), and the
handler now throws when StoreRuleResultAsync returns false. Without this the
worker would have deleted messages on transient Table Storage failures and never
retried.
Tests: +4 failure-propagation, +6 envelope round-trip, 762 backend pass.
Follow-ups noted: TableStorageService helpers still swallow exceptions one
layer below; SignalRNotificationService needs interface for handler-level
unit tests; V1 ingest path missing vulnerability_report synth event.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Container App scale-to-zero clears the in-memory MSRC and KEV indexes on
every cold start, leaving a window where session correlation runs without
those data sources until the next refresh. Re-fetching ~1800 MSRC CVEs
plus the KEV catalog from upstream on every cold start is wasteful and
slow; persisting the snapshots to blob solves both problems.
New BlobBackedReferenceCache<T> generic helper handles single-blob JSON
snapshots in the autopilotmonitor / vulnerability-data container:
- msrc-index.json — full MSRC CVRF index (~350 KB for last 6 Patch Tuesdays)
- kev-catalog.json — full CISA KEV catalog (~500 KB)
Both services gained:
- EnsureHydratedAsync — idempotent, semaphore-guarded blob load
- Blob-write at the end of RefreshAsync (manual trigger + daily timer)
- Hydrate guard set after refresh so the no-op path is taken next time
VulnerabilityCacheWarmer (IHostedService) kicks off both EnsureHydratedAsync
calls fire-and-forget at app startup so re-deploys do not pay a blob-read
latency cost in their cold-start window. The correlation hot path also
calls EnsureHydratedAsync as a defensive backstop — idempotent and
semaphore-protected, so a request arriving mid-warmup awaits the same
in-flight load instead of duplicating it.
NVD's per-CPE cache stays in Table Storage — random-access lookup by CPE
key plus per-key 7d stale-refresh logic is the right shape for that
workload and would degrade if forced into a single-blob model.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MsrcApiClient was injected into VulnerabilityCorrelationService for months
but never invoked — every report shipped with source="NVD" only. New
MsrcCorrelationService owns refresh + per-CVE lookup against an in-memory
index built from the last 6 Patch Tuesdays; daily VulnerabilityDataSync
populates it; CorrelateAsync enriches each finding with msrcDocumentId,
msrcReleaseDate, msrcSeverity, msrcBaseScore, msrcVector and flips source
to "NVD+MSRC" when the CVE was published in the indexed window.
Health check also surfaced that all worker-side traces of the daily sync
and CPE auto-resolve were sampled out of App Insights despite running.
Adds Event to host.json excludedTypes and emits TelemetryClient events
for VulnerabilityDataSyncCompleted, VulnerabilityCorrelationCompleted,
CpeAutoResolveHit/Miss, CpeMappingPersisted so customEvents queries can
prove freshness without depending on sampled traces.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The QueueTrigger binding required a Functions-host-specific app setting
(`<Connection>__queueServiceUri` with `__credential=managedidentity`) that
diverged from the project's storage-access pattern. Tables, blobs and the
index-reconcile producer all use a single `AzureStorageAccountName` (Managed
Identity) or `AzureTableStorageConnectionString` (full conn-string) and build
their clients with `DefaultAzureCredential` directly via DI — no host-side
binding glue.
In production the trigger silently failed to bind on every Function-App
worker start ("Error indexing method 'Functions.IndexReconcile'. Storage
account connection string 'AzureWebJobsAzureTableStorageConnectionString'
does not exist"), so messages enqueued by the producer would have piled up
unconsumed once `EnableIndexDualWrite` was flipped on.
Replaces:
- IndexReconcileFunction.cs (40 LoC, QueueTrigger) → DELETED
- New IndexReconcileQueueWorker.cs (BackgroundService) — same DI-based
config resolution as AzureQueueIndexReconcileProducer, full
QueueTrigger-platform parity (visibility-timeout retries, poison-queue
move after 5 failed attempts, malformed-JSON drop, idle 10s sleep).
Also pre-existing TableInitializerService is registered as IHostedService
in the same project, so the pattern is established.
Build: 0 warnings, 0 errors. Backend tests: 736/736 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consumes the telemetry-index-reconcile queue (M5.d.2) and fans each
envelope out into 0–3 index-table rows via IIndexTableRepository
(M5.d.1). Closes the producer→consumer loop; once the
EnableIndexDualWrite flag is flipped, primary commits drive
eventual-consistent index upserts asynchronously.
Components:
- Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues 5.5.4
package for the isolated-worker QueueTrigger binding.
- IndexReconcileHandler (plain class — not interface-abstracted per
Plan §M5.d design: DAL abstraction lives at IIndexTableRepository,
not here). Routing:
Signal → SignalsByKind (always, 1 row).
DecisionTransition:
IsTerminal=true → SessionsByTerminal
Taken=true → SessionsByStage
Taken=false + reason → DeadEndsByReason
ClassifierVerdictId!=null → ClassifierVerdictsByIdLevel
Up to 3 rows per DecisionTransition envelope (a Taken+terminal+
classifier transition fans to SessionsByTerminal + SessionsByStage
+ ClassifierVerdictsByIdLevel).
- IndexReconcileFunction thin QueueTrigger wrapper
(Connection = AzureTableStorageConnectionString). Malformed JSON
logged + dropped to avoid poison-queue retry loops; transient
repo errors propagate so the platform retries.
- DI: handler registered as Singleton in Program.cs.
Failure semantics:
- Required-field-missing envelopes (Signal without SessionSignalOrdinal
or SignalKind; DecisionTransition without StepIndex) → log warn,
drop. These won't become valid on retry — avoiding the retry loop.
- Transient Table Storage errors → rethrow. QueueTrigger platform
retries up to 5 times by default, then moves to
telemetry-index-reconcile-poison for manual investigation.
Deferred to M5.d.4:
- SessionsByStage stale-row pruning: when a session moves from
Stage-X→Stage-Y, the old Stage-X row stays. Either query-side
dedupe (per SessionId, max-Ticks RK) or reconcile-timer pruning
handles this. Deliberately not solved in the handler to avoid
a query+delete round-trip per Taken transition.
- End-to-end dual-write verification tests (Azurite harness) are
the M5.d.4 release gate before the flag is flipped.
Tests (+10 in Functions) via capture-only FakeIndexTableRepository:
- Signal happy path (1 SignalsByKind row).
- Signal missing SessionSignalOrdinal / SignalKind → 0 rows (x2).
- DecisionTransition Terminal+Taken+Classifier → 3 rows.
- DecisionTransition Taken non-terminal no-classifier → 1 row
(SessionsByStage only).
- DecisionTransition dead-end with classifier → 2 rows
(DeadEndsByReason + ClassifierVerdictsByIdLevel).
- DecisionTransition dead-end without reason → 0 rows (PK-collision
defense).
- DecisionTransition missing StepIndex → 0 rows (all paths gated).
- Unknown SourceKind → 0 rows.
- Null envelope → 0 rows.
Full Functions suite: 682 green (prev 672 + 10 new). Full solution
build green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the async fan-out producer path from Plan §2.8 / §M5.d so primary
Signals + DecisionTransitions rows can be projected onto their index
tables without blocking the hot ingest loop.
Components:
- Constants.QueueNames.TelemetryIndexReconcile (azure-queue-name-safe,
all lowercase + hyphens).
- IndexReconcileEnvelope (Shared DTO, flat shape with nullable fields
per SourceKind="Signal"/"DecisionTransition"). Avoids polymorphic
Newtonsoft deserialization — see M5.c.3 finding.
- IndexReconcileEnvelopeFactory pure static helper (FromSignal /
FromDecisionTransition / BuildBatch) — state-less projection.
- IIndexReconcileProducer (Shared interface) contract: no-op when
the feature flag is off; never rethrows underlying-queue failures.
- AzureQueueIndexReconcileProducer (Functions) — Base64-encoded JSON
messages, lazy CreateIfNotExistsAsync guarded by Interlocked flag,
Managed-Identity-or-connection-string construction mirroring
BlobStorageService / TableStorageService.
- IngestTelemetryFunction ctor gains IIndexReconcileProducer; after
primary Signals + DecisionTransitions commit it builds envelopes
and enqueues them. Event path is untouched (events don't feed the
index tables per §2.8).
- Program.cs registers the producer as a singleton.
Feature gate (AdminConfiguration.EnableIndexDualWrite, default false):
- New bool field in AdminConfiguration.
- TableConfigRepository Store + Map extension (per
feedback_table_storage_serialization).
- Producer checks config each call via AdminConfigurationService's
5-min memory cache. Flag off → primary behaviour is bit-exact.
Fault tolerance: queue-side exceptions (network, throttling, missing
queue before first CreateIfNotExists) are logged per envelope and
swallowed. The primary row is the source of truth; the 2h reconcile
timer (M5.d.4) is the safety-net recovery path.
Tests (+11 in Functions):
- 5 IndexReconcileEnvelopeFactoryTests: Signal/DecisionTransition
projection, dead-end preservation, batch ordering, null/empty.
- 3 IndexReconcileEnvelopeSerializationTests: Newtonsoft round-trip
(Signal, full DecisionTransition, minimal-envelope tolerance).
- 3 AdminConfigurationEnableIndexDualWriteTests: default-false on
new config + CreateDefault, persists when set.
Deferred to later sub-milestones:
- M5.d.3: queue-triggered consumer that maps envelopes to 0–3 index
rows using M5.d.1 TableIndexRepository.
- M5.d.4: 2h reconcile timer + release-gate dual-write verification
tests, then flip EnableIndexDualWrite=true in prod admin config.
Full Functions suite: 672 green (prev 661 + 11 new). Full solution
build green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extract the post-parse event-processing tail of IngestEventsFunction into a
standalone EventIngestProcessor service so /api/agent/telemetry runs the same
downstream logic as /api/agent/ingest. Deliberate copy-duplication, not a
shared refactor:
- Legacy /api/agent/ingest is production-hot — it serves live traffic from
every deployed V1 agent and is NOT touched by this commit. Zero regression
risk on the existing path.
- EventIngestProcessor is a near-verbatim copy of the legacy pipeline's tail
(~1000 LoC). Duplication is accepted because legacy /api/agent/ingest is
scheduled for decommission post-v11 GA (see tasks/todo.md → Follow-Ups →
Legacy-Agent-Ausmusterung). Until then: bugs fixed in legacy must be ported
here manually.
Copied pipeline steps (all 1:1 with legacy ProcessIngestAsync lines 150-566):
StampServerFields → SanitizeEventTimestamps → StoreEventsBatchAsync →
Index updates fire-and-forget → IME agent version history → ClassifyEvents
→ App install summaries → Geo-location update → UpdateSessionStatusAsync
(all 7 status branches) → IncrementSessionEventCountAsync → Rule engine
analysis fire-and-forget + SignalR + stats → Vulnerability correlation with
WhiteGlove Part-2 merge → IncrementPlatformStatAsync TotalEvents +
SuccessfulEnrollments → RecordGatherRuleStatsAsync → Diagnostics blob
update → GetSession for updatedSession → Session age 4h warning → WhiteGlove
IsPreProvisioned error log → SendWebhookNotificationsAsync (all 3 alert
paths) → SLA breach evaluation → Admin override detection → ServerAction
fetch-and-clear with TrackEvent telemetry → BuildSignalRMessages.
Shared (not duplicated, to avoid drift in pure helpers):
IngestEventsFunction.StampServerFields, SanitizeEventTimestamps,
RecalculateAppDurations (all internal static, state-less, stable). The
internal DTOs EventClassification + AppInstallAggregationState are also
reused — they carry no behaviour, only fields.
IngestTelemetryFunction wiring:
- Return type HttpResponseData → IngestEventsOutput so the V2 endpoint can
emit SignalR messages for the real-time UI push (matches legacy binding).
- PersistItemsAsync partitions the batch: Events go through the processor
(full pipeline + AdminAction + pending ServerActions + SignalR); Signal +
Transition batches skip the processor and read control signals directly
via a lean ReadControlSignalsAsync helper on ISessionRepository — no point
firing the full pipeline for a batch that carries no events.
- EventIngestProcessor registered in Program.cs DI.
Known gap (explicitly NOT in this commit): the V2 endpoint lacks a payload
size guard (legacy reads MaxNdjsonPayloadSizeMB from tenant config in
ParseNdjsonRequest). Different payload shape (JSON array vs NDJSON), deferred
to a dedicated follow-up.
All 596 backend tests green (unchanged count — copy path has no new tests
yet; the V2 endpoint's behaviour is validated by the legacy test coverage
through the shared-helper surface + the TelemetryPayloadParser tests from
M5.b.1). Integration coverage lands with M7 test-VM smoke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduces a clean abstraction layer to decouple domain services from Azure Table Storage,
enabling future migration to Cosmos DB and Event Streaming without touching business logic.
Architecture:
- 10 repository interfaces in Shared/DataAccess/ (ISessionRepository, IRuleRepository, etc.)
- IDataEventPublisher interface with NullDataEventPublisher for future event streaming
- IStorageInitializer for provider-agnostic initialization
- 5 Table Storage implementations that delegate to existing TableStorageService
- DI extension method AddTableStorageDataAccess() with AddEventStreaming<T>() hook
- Migration plan in tasks/todo.md for incremental adoption
https://claude.ai/code/session_01NHWJfPaa8TQHuPzMD7gG63
Oliver Kieselbach