Foundation for "MSP mode" — a scoped-global authorization tier between a
single-tenant member and a platform GlobalAdmin. A delegated admin may access a
SUBSET of tenants (its assignment rows), resolved UPN-based and tid-agnostic,
exactly like GlobalAdmins.
Phase 0 is purely additive: the service is registered but no auth path consults
it yet (the cross-tenant gate change lands in Phase 1), so behavior is unchanged.
- Constants: DelegatedRoles/DelegatedStatus/DelegatedSource; new DelegatedAdmins
table added to TableNames.All (auto-create), CriticalBackupTables.All and
AuthTablesFullRestoreForbidden (identity table — full-table restore blocked).
- IAdminRepository + TableAdminRepository: DelegatedAdminEntry DTO + CRUD
(PK=UPN, RK=TenantId). Hot-path GetDelegatedTenantsAsync (partition scan) and
reverse GetDelegatedAssigneesAsync. Queries use the typed predicate overload
(no string-interpolated OData filter — avoids filter injection).
- DelegatedAdminService: resolves DelegatedScope (Covers/RoleFor/CanWrite) with a
5-min cache + write-through invalidation. Fail-closed: only Active+enabled rows
with a recognized role confer scope; empty role -> DelegatedReader; unknown -> dropped.
- 16 unit tests (fake repo); full Functions suite 2434/2434 green. Backup
catalog-count assertion bumped 15 -> 16.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Global-Admin cross-tenant view of web users active "now" (made an authenticated
request within a window, default 5min), backed by a new self-maintaining
UserPresence table. Not App Insights (sampling) / not SignalR.
Backend:
- UserPresence table (one row/user, PK=tenantId, RK=SHA-256(lowercase UPN));
upserted fire-and-forget by UserPresenceMiddleware (after PolicyEnforcement,
60s per-process throttle, UPN-gated so agent cert calls are skipped).
- GET /api/global/presence (GlobalReadOrAdmin) via GetActiveUsersFunction; read
path propagates storage errors (5xx) instead of masking them as "0 active".
- 1-day presence retention + 90-day UserActivity retention in maintenance;
PlatformStats.TotalUsers clamped monotonic so the public "Users Seen" figure
cannot regress after a prune.
Web:
- /admin/presence page (live counter + table), nav entry under GA > Metrics.
- "Total Users" relabeled "Unique Users (90d)" to match the windowed metric.
Backend+web only, no ConfigVersion bump, inert until redeploy.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Lets tenant-admins and GAs run end-to-end session debugging over MCP: backend
telemetry plus the agent diagnostics ZIP, downloaded and analyzed locally by the
AI client. The backend never unzips/parses the archive (no parse CPU); it only
mints a short-lived signed download ticket and proxy-streams bytes.
Backend (no ConfigVersion bump, no agent change):
- DiagnosticsDownloadTicket: HMAC codec (reuses PaginationTokenSigningKey,
purpose-tagged diag-dl-v1, 10-min TTL) so a no-JWT client can download.
- DiagnosticsBlobStreamer: shared proxy-stream engine; DiagnosticsDownloadFunction
(web/JWT) refactored onto it. ClassifyBlobName/BlobDestination stay put (xUnit).
- POST diagnostics/download-ticket (MemberRead, QueryParam, GA-bypass) +
GET diagnostics/download?t= (anonymous; tenantId/blobName read only from the
signed ticket -> no retargeting). Both added to NoStoreCacheMiddleware.
MCP:
- get_session_diagnostics tool (all roles) -> absolute downloadUrl + static zipMap
+ instructions; unwraps the {success,session} envelope.
- diag_zip_layout catalog (get_resource + protocol resource).
- debug-session prompt (tenant + GA variants).
- Tool descriptions flag the client requirement (needs local file/shell tools;
pure chat clients cannot unzip).
Tests: backend 2380/2380 (ticket codec, NoStore, classify, catalog completeness);
MCP 261 passed (incl. envelope-unwrap regression). Inert until redeploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deploy-verified on f4d51ea4: the worker-side adaptive-sampling exclusion
(EnableAdaptiveSampling=false + manual Configure<TelemetryConfiguration>
UseAdaptiveSampling(excludedTypes:"Request;Event")) does NOT take effect in the
Functions isolated worker. Live data showed the enriched worker request copy
(Source=WorkerMiddleware) sampled to ~63% while the redundant host request item
(Source="") stayed fully billed — strictly worse (lossy enriched copy, host
duplicate unchanged). host.json "Host.Results": "Error" likewise did not drop
the host request item (emitted by the host process, not governed by that level).
Restores known-good: default worker sampling (enriched copy complete at normal
traffic, burst-capped) + host.json Host.Results back to Information. L1 storage
dependency filter and L2 Function-trace suppression are unaffected and still
verified live. L4 deferred pending a mechanism verified against the isolated-
worker pipeline. H2/H3/O1/O2 from f4d51ea4 verified healthy (0 exceptions).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Telemetry cost (L4): worker-side adaptive sampling now excludes Request;Event
so the enriched RequestTelemetryMiddleware copy is never thinned; host.json
Host.Results=Error drops the duplicate unenriched host request item. One
complete request row per call instead of two (one unsampled+unenriched, one
sampled+enriched).
Ingest hot path (H2/H3): BlockedDeviceService caches negative answers (30s
revalidation, symmetric with positive entries) so healthy devices no longer
pay a BlockedDevices point-read per request; SessionDeletionGuard returns its
already-loaded row and IncrementSessionEventCountAsync returns the post-merge
snapshot, collapsing ~4 Sessions reads/batch to ~2.
Queue-worker correctness (O1/O2): new GetSessionEventsStrictAsync propagates
storage failures on the analyze + vulnerability-correlate paths so a transient
read fault retries/poisons instead of silently deleting the message after
"analyzing" an empty stream. Fail-soft variant kept for UI/metrics surfaces.
Review fixes: parse prefetched Status as string (the actual wire format, was
int -> optimization never fired); session-aware block over a cached negative
entry keeps its session scope instead of widening to whole-device.
2274/2274 backend tests green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Eliminates the copy-paste queue scaffolding flagged as finding D2: identical
Managed-Identity/connection-string QueueClient bootstrap hand-rolled across 13
sites, plus a poll-loop/poison/safe-delete skeleton duplicated in 5 workers.
- QueueClientFactory: resolves the MI/connstring model once; consumed by all 5
producers, the offboarding enqueuer, the backup producer and the poison probe.
- QueuePollingWorkerBase: shared bootstrap + poll loop + poison-move + safe-delete
+ queue-create (ExecuteAsync/ProcessOneAsync sealed to prevent drift).
- QueuePollingWorker<TEnvelope>: deserialize -> validate -> handler flow with an
opt-in in-flight heartbeat.
- 3 vanilla workers (Index/Analyze/Vuln) collapse to ctor + HandleAsync.
- SessionDeletion/Offboarding use BatchSize/UseHeartbeat/ShouldPauseAsync +
Before/AfterPoisonMoveAsync hooks (kill-switch, CAS state transition, OpsEvent
enrichment, Failed-state-before-poison all preserved bit-for-bit).
- Backup derives from the non-generic base and overrides ProcessOneAsync (its
lease + CAS state machine is intentionally bespoke).
Producers get the factory only -- no shared base, their business logic is
genuinely divergent. Test seams unchanged; the static SessionDeletionWorker.
MaxDequeueCount reference moves to QueuePollingWorkerBase.DefaultMaxDequeueCount.
2168/2168 backend tests green, 0 warnings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /api/global/backups/{backupId}/restore-row (GA-only) with preview|commit
modes. Preview scans the ETag-pinned NDJSON blob for the requested (pk, rk),
diffs against the live row, and returns rowSha256 + currentETag for echo.
Commit re-validates everything under a short maintenance lease and writes via
UpdateEntity(IfMatch=etag, Replace) or AddEntity; 412/404/409 → CurrentRowChanged.
Backend:
- RestoreTablePreflightValidator (no I/O) + BackupRestoreInputValidator (manifest
parse, entry check, BlobName canonical, ETag-pin + SHA in one stream read)
- CriticalTableRestoreService with PreviewRowAsync (no lease) + CommitRowAsync
(MaintenanceLeaseHolder, 45 s renewal, deterministic finally release)
- BlobBackupStore.OpenNdjsonReadAsync — GetProperties + OpenRead(IfMatch=etag)
- BackupTerminalException carries a stable Code for HTTP envelope mapping
- OpsEventService.RecordBackupRowRestoredAsync (Warning; payload is
container + blobName only, never a SAS URL)
- 27 new xUnit tests; full backend suite 1979/1979 green
Web:
- /admin/backups list page with newest-first list + "Run backup now" trigger
and async job-status poller
- /admin/backups/{backupId} detail page with per-table breakdown + single-row
restore trigger
- RestoreRowDiffModal with Added/Removed/Changed/Unchanged pills and
auth-table security banner (GlobalAdmins/TenantAdmins/McpUsers)
- api.backups namespace + matching DTO types
- globalNavConfig: ga-backups under Ops
- BackupRowRestored registered in OPS_EVENT_TYPES (dual-register policy)
- 6 new vitest cases; full web suite 390/390 green
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the full backup pipeline that snapshots 15 critical tables (Admin/Tenant
config, rules, identities, audit, preview) into a logically-isolated blob
container, daily at 04:00 UTC and on operator demand. Restore endpoints follow
in PR2/PR3.
Architecture:
* Daily TimerTrigger + manual POST /api/global/backups/trigger (GA-only,
fail-hard producer). HTTP returns 202 + jobId on enqueue, 500 if the queue
write fails (and the BackupJobs row is rolled to Failed).
* BackgroundService queue worker (not QueueTrigger — matches the Analyze/
IndexReconcile pattern that avoids the Functions-host connection-binding
app-setting). Worker holds the maintenance lease for the whole run; service
is intentionally lease-unaware so it stays unit-testable.
* BlobBackupStore wraps the new 'critical-table-backups' container with a
shared maintenance-lease sentinel at '_lock/maintenance.lock' that
serialises every backup or restore op across worker/timer/single-row paths.
* CriticalTableBackupService streams each table through CryptoStream(SHA256)
into per-table NDJSON blobs, then writes manifest.json LAST as the
durability anchor. Per-table failures degrade Outcome to Partial without
aborting; per-run budget (50 min) → Skipped status.
* BackupJobsRepository persists the six-state machine
(Queued/Running/Completed/Failed/Skipped/BlockedTerminal) with ETag-CAS
updates. Enum→string mapping rejects unknown values on read so refactors
surface loudly instead of silently shifting state.
* BackupJobWatchdog (30 min timer) heals stuck rows via a probe lease on the
shared sentinel: Queued > 60 min → Failed, Running with stale heartbeat
AND free lease → Failed (lease-busy means a live worker — leave alone).
* OpsEvents: distinct CriticalTableBackupPartial (Warning) vs Failed (Error)
so operators can wire softer Telegram rules for partial captures.
* MaintenanceService.MonitoredPoisonQueues + OPS_EVENT_TYPES catalog
registered (dual-register memory).
Test coverage (15 new tests, 1952/1952 backend xUnit green):
* BackupManifestJsonTests: enum-as-string (allowIntegerValues=false rejects
numeric values), DictionaryKeyPolicy unset (Azure column names preserved
case-faithfully), manifest field round-trip.
* BackupJobsRepositoryMappingTests: full entity↔domain round-trip, unknown
Kind/State/BackupOutcome rejected, legacy BackupOutcome="Failed" rejected.
* CriticalTableBackupServiceTests: happy/partial/empty/budget-exceeded paths
exercise the per-table loop with in-memory table + blob fakes.
Setup: scripts/Backup/setup-backup-container.ps1 — idempotent az-CLI bootstrap
for container + lifecycle rule (90 d delete on prefix). Soft-Delete and
Versioning are NOT applied automatically (storage-account-wide blast radius
— operator must validate per separate doc note).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds an opt-in second destination for agent diagnostics uploads:
- CustomerSas (default, unchanged): tenant's own SAS-backed container
- Hosted (new, opt-in via explicit admin click): backend's storage account
under {tenantId}/AgentDiagnostics-...zip, with per-upload 15-min
blob-scoped Write+Create SAS minted via the Function App's MI
Closes the long-standing gap where Session.DiagnosticsBlobName was captured
into the deletion-manifest but the blob was never actually cleaned up by
the cascade. Hosted blobs are always deleted (we own the storage); customer
blobs are deleted only when the SAS includes the 'd' permission, otherwise
logged + skipped.
Existing tenants change nothing: the new TenantConfiguration.DiagnosticsUploadDestination
field defaults to CustomerSas, no silent flip. Hosted requires an explicit
admin click + Save with a clearly marked "data leaves your tenant" disclosure.
Tests: Backend 1844, V2 1076, V1 538, vitest 380 (+78 new across all suites).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Persist tenant Preview-Notification-Email on OffboardingHistory row at Phase 1 so it
survives Phase 2 wipe; ResendEmailService.SendAsync wired but disarmed by a const-false
arm-flag with a [DRAFT] placeholder template — explicit flip required to ship.
- Move user feedback storage out of PreviewConfig discriminator into a dedicated Feedback
table (PK=InApp / Offboarding) that is NOT in any tenant-offboarding wipe list — feedback
from departing tenants is exactly the signal we want to keep. New IFeedbackRepository +
TableFeedbackRepository; FeedbackFunction migrated; (PreviewConfig, "Feedback")
removed from TenantOffboardingHandler.DiscriminatorTables.
- POST /api/tenants/{tenantId}/offboard/feedback collects a free-form farewell comment
during the drain-barrier countdown (marker-gated, fail-soft OpsEvent emit). Reports >
User Feedback page gains an Offboarding-Feedback sub-tab next to In-App Feedback.
- OffboardingFeedbackReceived OpsEventType registered (info-severity) + catalog entry
in OpsAlertRulesSection so admins can wire Telegram in two clicks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds opt-in resolution of Intune Platform Script + Remediation display names in
session timelines via a tenant-side appRoleAssignment grant. No app manifest
change, no impact on customers who don't enable it. Architecture is reusable for
future optional Graph permissions: catalog -> detector -> resolver -> endpoint.
Backend
- GraphFeatureDetector parses the `roles` claim from the SP access token and
caches the (token, roles) bundle per tenant. GetSnapshotAsync surfaces a
transient flag so admin-UI callers can render "unknown" instead of a
misleading "0 of 0 granted" when token acquire times out.
- GraphTokenService now accepts a CancellationToken so optional-feature callers
can apply tight budgets instead of inheriting the 5+15+30 s retry chain.
- ScriptDisplayNameResolver: cache-first, list-pull-on-cold-tenant, per-ID
fallback. Throttle-aware abort: skip per-ID for kinds whose full-pull hit a
transient failure; break the per-ID loop on the first HttpRequestException.
Total budget 5 s, with a tight 1.5 s sub-budget on the token acquire.
- Azure Table `ScriptNameCache` + direct-TableClient fail-loud repository
(mirrors TableOffboardingAuditRepository pattern). Daily
ScriptNameCacheCleanupFunction: 7 d positive / 1 h NotFound TTL.
- POST /api/tenants/{id}/scripts/display-names (body-carried refs to dodge URL
length limits; backend cap 200 refs/request).
- GET /api/tenants/{id}/graph-permissions/status (4 s budget; includes app
ClientId so the UI can render an exact grant command).
- POST /api/tenants/{id}/graph-permissions/refresh (invalidates token cache).
- All three routes registered in EndpointAccessPolicyCatalog (MemberRead for the
resolve, TenantAdminOrGA for the admin-UI surfaces).
Web
- `useScriptDisplayNames` hook with stable refset fingerprint (no spurious
refetches on parent re-renders); clears stale map on tenant/refset change so
a session switch never leaks names across tenants.
- Chunked POST loop: 200-ref batches with partial-success semantics. Preserves
earlier chunks when later ones fail (non-2xx OR transport throw); only
TokenExpiredError propagates so auth flow can react.
- ScriptExecutions renders " . {DisplayName}" alongside existing label/short-id,
keyed by Kind + policyId so Platform/Remediation under the same id never
collide.
- New Settings section "Optional Graph capabilities": status table with granted
/ not granted / unknown, copy-paste PS one-liner with ClientId baked in,
refresh button gated on response.ok.
Customer-facing
- scripts/CustomerSetup/Grant-AutopilotMonitorAddOn.ps1: idempotent
grant / verify / revoke script. -Features ScriptDisplayNames (high level) or
-Permissions ... (admin-UI generated). Pure ASCII for PS 5.1 compat.
- docs/customer-graph-add-on-permissions.md: end-to-end customer guide
covering prerequisites, the appRoleAssignment pattern, and revocation.
Tests
- 1718 backend xUnit / 364 web vitest. Highlights: JWT roles parsing edge
cases; full-pull-429 suppresses per-ID; per-ID-429 breaks loop; transient
detection via GetSnapshotAsync; chunked partial success (HTTP failure + thrown
transport); TokenExpiredError propagation; EndpointPolicyCatalogCompleteness
covers the three new routes by reflection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan §5 PR6 / §16 R14: extracts cascade-lifecycle maintenance out of the
2h Maintenance timer into a dedicated 12h SessionDeletionMaintenanceFunction
with independent cadence, kill-switch, and OpsEvent watchdogs. Adds four
work-blocks (Plan §10):
1. Manifest-blob TTL sweep — defence-in-depth on top of Azure Blob
Lifecycle Management. EnumerateOldDeletionManifestsAsync +
DeleteDeletionManifestPairAsync (33-day effective window via the
existing soft-delete tombstone, not permanent deletes).
2. Stale-Preparing GC — reverts DeletionState=Preparing rows older than
1h that have NO progress blob (producer crashed before upload). Skips
when a progress blob exists; producer's own Preparing-resume path
handles those.
3. Stranded-Queued detection — alert-only OpsEvent per row in
DeletionState=Queued for > 30min (no auto-clear; operator inspects).
4. Retention fanout via new SessionRetentionFanoutService — per-tenant
DataRetentionDays sweep, dispatched per EnableCascadeDeleteV2 flag:
V2 → producer.EnqueueAsync, legacy → ISessionRepository.DeleteSessionAsync
(now CAS-guarded since PR5 finding 2). Rate-limited to 100 enqueues
per tenant per run with 50ms throttle.
Operational alerting:
- 30min watchdog → OpsEventType SessionDeletionMaintenanceLongRunning (Warning)
- 60min watchdog → SessionDeletionMaintenanceLongRunningSevere (Error)
- Exception path → SessionDeletionMaintenanceFailed (Error) + re-throw
- Per stranded row → SessionDeletionStrandedQueued (Warning)
All four registered both in OpsEventService.cs and the OpsAlertRulesSection.tsx
catalog so admin can wire Telegram per memory feedback_ops_event_types_dual_register.
Reuses PR5 finding 1's IsSessionDeletionKillSwitchActiveAsync so the
fanout-skip check honors flip-ON across scaled-out instances within seconds.
host.json: functionTimeout=01:00:00 (Flex Consumption max, matches the
60min severe watchdog so operator gets alerted before Azure hard-aborts).
Cron: const "0 0 */12 * * *" — matches the repo's IndexReconcileTimer
convention (no AdminConfiguration field; redeploy to change).
Removed: session retention loop from MaintenanceService.CleanupOldDataAsync
(both RunAllAsync and RunManualAsync paths now call CleanupOldUsageDataAsync
directly). Non-session housekeeping (rule-stats, cert-expiry, SignalR
quota, poison-queue watcher) untouched.
EnableCascadeDeleteV2 + SessionDeletionKillSwitch defaults stay false —
PR6 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation.
Backend xUnit: 1500/1500 pass (1483 + 17 new). Web vitest: 347/347 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Plan §5 PR5 (session-deletion-cascade-safe.md): flag-gated dispatch in
DeleteSessionFunction between the legacy direct-delete path and the V2
cascade producer; SessionDeletionWorker registered as a hosted service;
UI shows a "deletion queued" toast + per-row spinner while the worker
drains, then removes the row on the SignalR sessionDeleted event.
Includes the codex review follow-ups:
F1 — IsSessionDeletionKillSwitchActiveAsync bypasses the 5-min admin
config cache; fail-CLOSED on storage error so a flip-ON is honored
across scaled-out Function-host instances within seconds (instead
of minutes). Wired into DeleteSessionFunction, Producer, Worker.
F2 — DeleteSessionAsync now reads DeletionState + ETag together with
IndexRowKey, refuses on locked state, and ETag-conditionals the
delete; closes the T2-T4 race where a concurrent V2 producer
could orphan side-table rows. Caller in the legacy path
disambiguates the bool via a re-read into 404 / 409 / 500.
F3 — pendingDeletions threaded from useDeleteSession through page.tsx
→ SessionTable → SessionCell; action cell renders a disabled
spinner instead of the delete button. Hook subscribes to
sessionDeleted SignalR + polls GET /api/sessions/{id} every 60s
as a fallback for missed events; only 404 clears the pending row.
EnableCascadeDeleteV2 + SessionDeletionKillSwitch both default false —
PR5 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation. PR6 wires the producer into the dedicated
SessionDeletionMaintenanceFunction next.
Backend xUnit: 1483/1483 pass. Web vitest: 347/347 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR4 ships the cascade-delete executor for the safe-cascade-delete plan: the
SessionDeletionWorker (BackgroundService queue poll-loop) plus the
SessionDeletionHandler (per-step execution, live verification, tombstone).
Worker is NOT registered in Program.cs — that lights up in PR5.
Three logical phases land together as one commit per Oliver's call:
PR4 — Worker + Handler + CascadeVerificationService
- SessionDeletionWorker: BatchSize=1, kill-switch entry guard, heartbeat task
extending visibility every 60s during long handler runs, poison after
max-dequeue
- SessionDeletionHandler: download manifest+progress, CAS Queued→Running
(with Running→Running resume), per-step dispatch by Class, AGGREGATE
decrement, live verification, ordered tombstone (SessionsIndex first,
Sessions last)
- CascadeVerificationService: per-Class live residual scans (CveIndex is
manifest-key-only per §12-Q8; AGGREGATE+FINAL skipped)
- NotifySessionDeletedAsync added to ISignalRNotificationService
- BlobStorageService: DownloadDeletionProgressAsync + ETag-CAS
UpdateDeletionProgressAsync
PR4b — Restore endpoint
- POST /api/admin/sessions/{sessionId}/restore (GA-only via Policy Catalog)
- SessionRestoreService: dispatch table per §13.4, full + partial-poisoned-
recovery modes
- TableStorageService.Restore.cs: RestoreRowsByExactKeysInBatchesAsync +
ConvertFromPropValue + ConvertDumpToEntity (Sessions-row override forces
DeletionState=None on restore so the row isn't permanently locked)
- RestoreSoftwareInventoryEntryByKeyAsync helper
- Worker MoveToPoisonAsync now CAS-transitions Running→Poisoned (E1 fix)
so restore can dispatch into partial-restore mode
PR4c — Codex review fixes (6 findings)
- F1: AGGREGATE per-key idempotency via DeletionProgress.AggregateDecrementsApplied
(persist-first/decrement-second bounds drift to +1 per crash vs unbounded -N)
- F2: TombstoneStarted flag closes the gap where sessions=null+completedAt=null
blocked restore after a tombstone crash
- F3a: Sessions row inserted LAST in full restore (was first → trapped retry
with RejectAlreadyAtOriginalState)
- F3b: RestoreMode.Full now 409-ignores (was throw-on-409); converges with
Partial mode, makes retry-after-partial-failure idempotent
- F4: Partial-restore per-key re-increment idempotency via
DeletionProgress.RestoreReIncrementsApplied
- F5: Poison transition pre-checks PendingDeletionManifestId so stale
envelopes can't flip a fresh active cascade to Poisoned
- F6: DownloadDeletionManifestWithShaAsync exposes snapshot SHA; Handler +
Restore both verify against progress.SnapshotSha256 (snapshot↔progress
binding per plan §3)
DeletionProgress schema gains 3 additive optional fields (AggregateDecrementsApplied,
RestoreReIncrementsApplied, TombstoneStarted) with safe defaults so PR1-PR4
progress blobs deserialize cleanly.
Tests: 1457/1457 backend xUnit facts green (1391 pre-PR4 + 66 new across PR4,
PR4b, PR4c). Full solution build green. Program.cs has zero
SessionDeletionWorker references (PR5 territory stays untouched).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
PR3 of the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). Substantial PR per
plan §16 — adds the concurrency state-machine, the single-chokepoint
SessionDeletionGuard, the producer that builds manifest + uploads
blobs + enqueues the worker envelope, and the read-only preview
endpoint Global Admins use to verify a cascade plan before flipping
EnableCascadeDeleteV2 on. Wiring the guard at the 9 writer sites is
intentionally deferred to PR5 — no cascade actually fires until PR5
calls the producer from DeleteSessionFunction, so PR3 ships as
infrastructure-only.
== State machine + guard infrastructure ==
SessionDeletionState (Shared, Plan §1 P7 / §16-R2):
None | Preparing | Queued | Running | Poisoned. Completed deliberately
omitted — the cascade FINAL step removes the row, so "completed" is
observable via row-absence + progress.completedAt, not via a stable
state value. IsValidTransition(from, to) enforces the legal edges
(None→Preparing, Preparing→{Queued,None,Poisoned}, Queued→{Running,
Poisoned}, Running→Poisoned, Poisoned→None) and treats null/empty
legacy rows as None.
SessionDeletionLockedException carries (TenantId, SessionId,
CallerContext, CurrentState, ManifestId) for audit + log lines.
SessionDeletionGuard (Plan §1 P7 + §16-R13):
- ThrowIfLocked(TableEntity sessionRow, callerContext) — piggyback
variant for hot paths that already loaded the Sessions row; zero
added I/O. Caller must include DeletionState +
PendingDeletionManifestId in its select projection.
- EnsureWritableAsync(tenantId, sessionId, callerContext) —
standalone variant for paths with no existing read; one Get
round-trip via the PR1 ISessionDeletionInventoryReader seam.
Both paths throw the same exception. Per §16-R13 framing the guard
is "best-effort fast-fail, not safety" — correctness lives in §1 P4
live verification + poison + §13 restore-from-poisoned (PR4).
== Sessions row state-machine columns ==
SessionSummary.DeletionState (string) +
SessionSummary.PendingDeletionManifestId (string?) — added to the DTO
+ MapToSessionSummary read path. Empty/null means no cascade in flight
(legacy rows; treated as None per the guard contract). The producer
writes both columns via direct CAS via the new helper; existing
session-write paths don't need to know about them in PR3.
TableStorageService.CasSetSessionDeletionStateAsync — new partial
helper. Read-then-CAS on Sessions row, returns a structured
SessionDeletionStateCasResult (Updated | SessionMissing | WrongState |
EtagConflict) so the producer can branch correctly. Stamps newManifestId
on transition to Preparing; clears it on transition back to None
(poisoned-recovery restore).
== Producer + preview endpoint ==
SessionDeletionProducer.EnqueueAsync(tenantId, sessionId, reason,
actor) — Plan §5 PR3 step sequence: kill-switch check → CAS None→
Preparing with new ManifestId → DeletionManifestBuilder.BuildAsync →
BlobStorageService.UploadDeletionManifestAsync → UploadInitial-
DeletionProgressAsync (new helper for the mutable .progress.json
sibling) → audit deletion_started → CAS Preparing→Queued → enqueue
SessionDeletionEnvelope to Constants.QueueNames.SessionDeletion.
Returns SessionDeletionEnqueueOutcome enum so PR5/PR6 callers
translate per their UX (admin → 409 on AlreadyInFlight, maintenance
→ skip+audit). Crash recovery contract: row stuck in Preparing with
no progress blob → maintenance GC clears after 1h (PR6); Queued/
Running/Poisoned NEVER auto-cleared.
GET /api/admin/sessions/{sessionId}/delete/preview?mode={summary|full|
download} — GlobalAdminOnly per the EndpointAccessPolicyCatalog (per
memory feedback_route_policy_catalog). Three modes:
- summary: preflightCounts + sample (PK, RK) per step + estimated
snapshot bytes + builderDurationMs + schemaHash.
- full: entire manifest JSON inline (every row dump, every decrement
key) — byte-level review of "what would be deleted".
- download: gzipped *.snapshot.json.gz with Content-Disposition
attachment, identical bytes to what the real cascade would upload.
NEVER mutates state, NEVER uploads a blob, NEVER enqueues. Calls the
guard only as a hint (in-flight cascades surface in the response, no
lock acquired). Audits deletion_preview per Plan §7.
== Configuration changes (with web round-trip) ==
AdminConfiguration.SessionDeletionKillSwitch (bool, default false,
Plan §1 P8 / §9) — global emergency stop. When true, V2 producer +
legacy direct-delete return 503; PR4 worker pauses on entry. Round-
tripped through TableConfigRepository Store + Map AND the 4-file web
chain (AdminConfigContext.tsx, AdminConfigSettingsSection.tsx,
SectionGlobalSettings.tsx, types/adminConfig.ts) per memory
feedback_admin_config_ui_roundtrip — without it, every admin Save
silently clobbers the field back to false. UI surfaces a red checkbox
with explanatory copy.
TenantConfiguration.EnableCascadeDeleteV2 (bool, default false) —
per-tenant rollout flag. When true, PR5 admin-delete + PR6 retention
fanout build a manifest + enqueue; when false, legacy direct-delete
runs. Per-tenant flag selects PATH; global kill-switch halts BOTH.
Constants.QueueNames.SessionDeletion = "session-deletion" — worker
queue name.
Shared envelope SessionDeletionEnvelope { TenantId, SessionId,
ManifestId, Reason, EnqueuedAt } — wire shape between producer and
PR4 worker.
== DI ==
Program.cs registers SessionDeletionGuard + SessionDeletionProducer
as singletons next to the existing DeletionManifestBuilder.
== Tests (50 new, backend suite 1340/1340 green) ==
- SessionDeletionStateTests (10) — IsLocked over every state value
including legacy null/empty, IsValidTransition over every legal
edge, explicit rejection of illegal edges, "Completed is not a
state" guarantee.
- SessionDeletionGuardTests (8) — null row no-op, every lock state
throws with correct exception fields, missing-session no-op,
context propagation in the exception.
- SessionDeletionProducerTests (7) — kill-switch returns
KillSwitchActive without downstream calls; SessionMissing /
AlreadyInFlight / Poisoned outcomes from CAS; happy path verifies
all 7 steps in order via Mock.Verify; CAS-2 failure leaves snapshot
uploaded but no queue message; producer's pre-allocated ManifestId
flows to both the snapshot blob and the CAS marker (consistency
invariant the worker depends on).
- AdminConfigurationSessionDeletionKillSwitchTests (6) — both flags
default false on new config + on CreateDefault, true persists.
Wiring 9 sites with the guard (telemetry V1+V2 ingest, mark-success/
failed, server actions, maintenance marker, re-analyze handler,
vulnerability handler, manual rescan, index reconcile) deferred to
PR5 — no cascade fires in PR3, so the guard is dormant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Read-only foundation for the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). PR1 ships the manifest
shape, the read-only DeletionManifestBuilder enumerator, and the
gzip+SHA-256 blob upload/download plumbing. No production caller
invokes any of this yet; no existing endpoint behaviour changes.
PR2-PR7 wire the producer, worker, and call sites on top.
Manifest captures all 15 cascade tables per the §3 taxonomy (4
PK_BY_SESSION, 1 PROP_TENANT_PK, 3 PK_RK_EXACT, 1
DISCRIMINATOR_PK_RK_SUFFIX, 1 DISCRIMINATOR_PK_RK_EXACT, 5
DISCRIMINATOR_PK_PROP), the optional SoftwareInventory side-row
(omitted for pre-side-row sessions per §17.5), and the FINAL tombstone
step ordered SessionsIndex-then-Sessions. Row dumps tag every column
with its EDM type (String / Int32 / Int64 / Double / Boolean /
DateTime / Guid / Binary) so DateTime / Guid / byte[] / long round-trip
restore-faithfully through the JSON snapshot. Falls back to a
SessionsIndex partition scan when Sessions.IndexRowKey is empty or
stale, so a partially-corrupted Sessions row can't strand a
SessionsIndex orphan after cascade.
BlobStorageService.UploadDeletionManifestAsync compresses with gzip,
pins SHA-256 of the uncompressed payload to blob metadata, and uses
IfNoneMatch=* to enforce snapshot-write-once. DownloadDeletionManifest
re-verifies the hash and refuses on mismatch (corruption signal that
the §13 restore semantics rely on). Two protected-internal-virtual
seams isolate the Azure SDK calls so tests bypass BlobsModelFactory
entirely.
New SessionInventoryContributions table constant added to TableNames.All
so the builder can Try-Read it (404 = pre-side-row session = inventory
steps omitted). PR2 adds the side-row write path via
VulnerabilityCorrelationService.
Tests: 25 new xUnit cases (DeletionManifestSerializationTests,
DeletionManifestBuilderTests, BlobStorageServiceDeletionManifestTests)
cover counts, dumps with EdmType per Azure Tables type, gzip round-trip,
hash-mismatch + missing-metadata corruption signals, IfNoneMatch
overwrite refusal, SessionsIndex fallback (both empty-key and stale-key
paths), determinism across runs, and pre-side-row inventory skip.
Backend suite: 1251/1251 green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Background
----------
The in-memory SlaNotificationThrottleService kept its 4h cooldown in a process-
local ConcurrentDictionary. On Flex Consumption the Function host recycles
between 2h timer fires, wiping the dictionary, so the throttle was effectively
a no-op — observed in production as tenant b54dc1af receiving an SLA breach
notification every 2 hours for days on end (1 failed of 11-13 sessions, target
95%, success rate 91-92%).
Design
------
One row per tenant in a new SlaTenantStatus table (PartitionKey = tenantId,
RowKey = "status"). Flat schema with {Type}_* property prefixes for the four
breach kinds (SuccessRate, Duration, AppInstall, ConsecutiveFailures); one row,
one Upsert per evaluation cycle. Rationale documented inline: per-type rows
would be over-normalised for the access pattern (always all-types-per-tenant),
and ~50 tenants x 4 types = max 200 rows ever.
Throttle correctness
--------------------
The throttle decision (now - LastNotifiedAt >= cooldown) and the LastNotifiedAt
write happen in the same TryUpsertAsync(status, ifMatchETag) call. Two parallel
inline calls cannot both succeed in sending the same notification — the loser
gets a 412 from Table Storage, refetches, sees the updated watermark, and
either skips or schedules a new claim attempt. Up to 4 retries; on exhausted
retries we drop the notification rather than risk double-fire (at-most-once).
Cooldown is configurable: AdminConfiguration.SlaNotificationCooldownHours
(default 24, range 1-168). UI round-trip wired through the four required
files (AdminConfigContext, AdminConfigSettingsSection, types/adminConfig.ts,
SectionGlobalSettings).
Resolve path
------------
When a previously-active breach is no longer detected, the row transitions to
IsActive=false + ResolvedAt=now and a single sla_resolved notification fires.
Orphan sweep: tenants whose toggles get turned off while a breach is active
fall out of the qualifying loop, so the timer now also scans every status row
and silently clears any IsActive that no longer has a toggle backing it
(no resolved notification — the tenant disabled SLA, they don't care).
Per-type notes
--------------
- SuccessRate breach notifies with the configured warn-threshold, not the SLA
target — a 94% rate against target=99% / threshold=95% reads "below 95%",
not "below 99%".
- AppInstall now actually evaluates (the toggle was wired but evaluation was
missing). Aligned to the same ISO-week window the dashboard uses
(SlaMetricsService.GetIsoWeekKey), so notifications cannot diverge from the
/sla page. Sample-size floor of 5 apps per week, same as the dashboard.
- AppInstall has its own notification template (own title/summary/facts plus
TopFailingApp fact) — the builder used to silently fall through to the
Duration template for everything not SuccessRate.
- Telemetry Period tag derived from breachType: CurrentMonth for SuccessRate
and Duration, CurrentWeek for AppInstall — so AppInsights / ops dashboards
don't group AppInstall under the wrong period.
- AppInstall ops events now carry the actual installer totals (10 evaluated,
4 failed) instead of 0/0.
- ConsecutiveFailures: inline (ingest) path detects breaches, timer path
detects resolve. Both paths write to the same row through the same CAS
retry loop.
Clock injection
---------------
SlaBreachEvaluationService takes an optional Func<DateTime> nowProvider
(default () => DateTime.UtcNow) so the week-window regression test runs
deterministically on every test day instead of silently skipping when today
happens to fall in the first ISO week of the month.
Removed
-------
- SlaNotificationThrottleService and its tests. The in-memory model is gone;
there is no scenario where it would be wanted alongside the persistent row.
Tests
-----
- 21 new lifecycle + mapping + telemetry tests (CAS conflict + retry, CAS
exhausted-retries drops notification, threshold-not-target in notification,
orphan clearing, AppInstall template / week-window / totals, ConsecutiveFailures
inline persistence, multi-type race-probe, period-mapping theory).
- Backend xUnit: 1218 passed.
- Web vitest: 266 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three async workers (analyze-on-enrollment-end, vulnerability-correlate,
telemetry-index-reconcile) park 5x-failed messages on <name>-poison. Until now
there was no operator-visible signal when this happened: backlogs could grow
silently and sessions stayed un-analyzed / vulnerability reports were lost.
Adds a "Poison Queues" entry to the detailed health-check next to "SignalR
Quota" with the same shape: parallel probe via IPoisonQueueProbe, tri-tier
classifier (0=healthy, >=1 warning, >=10 critical, thresholds App-Setting
configurable via PoisonQueueWarningThreshold / PoisonQueueCriticalThreshold),
GA-only visibility. Per-queue 404 counts as zero because poison queues are
created lazily on first poison-move. Per-queue exceptions surface in details
as "error: <msg>" and force at least warning so a flaky Storage call cannot
mask a real backlog.
Web UI needs no change - the generic checks[] render loop on /health-check
picks up the new card automatically.
Backend: 1193/1193 (1178 + 15 new). Build 0 warnings, 0 errors.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two notification-bell React contexts polled GET /api/notifications and
GET /api/global/notifications every 60s regardless of activity, generating
3017+618 = 3635 calls/24h with near-zero hit rate (notifications are emitted
sparsely). SignalR was already used app-wide for live session updates; the
bell was the lone hold-out on a pull model.
Switch to the same push model. Steady-state traffic drops ~86% and the bell
updates sub-second instead of within 60s.
Architecture (Option C+):
- Mount -> 1x HTTP fetch (cold-start hydration via existing GET endpoint)
- SignalR onreconnected -> re-fetch (covers the disconnect window)
- SignalR push -> live append/dismiss
- No periodic polling, no setInterval
Audience-tier security: TenantNotificationAudienceCatalog still drives
visibility (Member vs Admin), but enforcement moves into SignalR group
membership: tenant-{id}-notify-member is joined by every tenant member,
tenant-{id}-notify-admin only by Tenant Admins / Global Admins. The
AddToGroup function fail-closes with 403 on Member-tier joins to the
admin group, so Admin-tier notifications cannot leak via push.
Backend
- ISignalRNotificationService extracted; SignalRNotificationService
implements it. Resolves the long-standing test-mocking blocker
(the real service's constructor validates AzureSignalRConnectionString
and BuildServiceManager(); previously you could not unit-test
consumers). Tenant + Global NotificationService consume the
interface; the three legacy callers (Analyze, Ingest, VulnCorrelate)
still bind to the concrete class -- migrate per test need.
- Six new push methods: SendTenantNotification{,Dismissed,DismissedAll}
and SendGlobalNotification{,Dismissed,DismissedAll}. Tenant variant
routes to the audience-correct group.
- Tenant + Global service push after a successful repo write
(fire-and-forget; never throws into the caller).
- SignalRGroupHelper: parsing handles the two new -notify-{member,admin}
suffixes; new IsTenantNotifyAdminGroup predicate; group-name builders.
- SignalRAddToGroupFunction: explicit 403 when a non-admin attempts
to join the admin-tier group.
Frontend
- TenantNotificationContext: setInterval gone. useSignalR() to
join member-group always, admin-group iff isTenantAdmin || isGlobalAdmin.
Listens for tenantNotification (append + dedupe), tenantNotificationDismissed
(filter), tenantNotificationsDismissedAll (clear). onreconnected
triggers re-fetch.
- GlobalNotificationContext: same pattern against global-admins group.
- app/layout.tsx: SignalRProvider hoisted above the notification
providers so useSignalR() resolves inside them.
Tests
- SignalRGroupHelperTests (8): parsing for all four group formats
incl. new -notify suffixes, IsTenantNotifyAdminGroup predicate.
- TenantNotificationServiceSignalRPushTests (8): audience-correct push
on Create (Admin / Member / Unknown -> Admin fail-closed), no-push
on repo failure, no-push on DismissAll(count=0), no-push on empty
tenantId.
- FakeSignalRNotificationService: in-memory spy implementing the
new interface; replaces the unmockable real service in tests.
- AuthFunctionSideEffectTests + TenantNotificationServiceTests:
constructor signatures updated for the new SignalR dep.
Suites
Backend xUnit: 1161/1161 (was 1146, +15 net).
Web vitest: 259/259 (unchanged; existing notification-API +
gate tests still cover).
Solution build: 0 warnings, 0 errors.
Expected post-deploy reduction (24h):
GET GetTenantNotifications: 3017 -> ~50
GET GetGlobalNotifications: 618 -> ~10
POST AddToGroup: ~246 -> ~350-450 (two joins per tab)
Net bell load: -86%, sub-second live UX.
Disconnect tradeoff (accepted): if SignalR drops without reconnect
(rare -- withAutomaticReconnect retries 1/2/4/8/16/30s ... forever),
the bell shows the snapshot from the last fetch but no live updates
until next reload. The initial-3-retry give-up path of the SignalR
client still gets the user a hydrated bell because the HTTP fetch
runs independently on mount.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hourly TimerFunction queries Azure Monitor for SignalR ConnectionCount (max
over last hour) and MessageCount (sum since UTC midnight) and emits Warning
(>=80%) / Error (>=95%) OpsEvents when the free-tier limits (20 connections,
20k messages/day) are approached. Mirror of the cert-expiry pattern - global
TenantId=null events, dedup'd to one emission per EventType per UTC day.
Fail-soft: skips silently when SignalRResourceId app setting is unset, logs
warning on Azure Monitor auth/query errors. Function App's managed identity
needs 'Monitoring Reader' on the SignalR resource.
Also called from the manual maintenance trigger so 'Run Maintenance Now'
exercises every health check.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Browser RUM on portal/dashboard surfaced ~1.7s time-to-data driven by
sequential backend reads, two stray duplicate fetches, and a 2s SignalR
defer. Five focused changes:
- AuthContext: store the auth/me prefetch as a Promise instead of its
resolved value so fetchUserInfo awaits the in-flight request rather
than racing a second fetch when the prefetch hasn't settled yet.
- useTenantSecurityConfig: depend on user.isTenantAdmin/isGlobalAdmin/
role primitives instead of the whole user object so the effect stops
refiring when AuthContext swaps prefetched user for fresh user (the
duplicate /config/<t>/feature-flags call observed on /dashboard).
- SignalRContext: drop the hardcoded 2000ms setTimeout before
startConnection. The original "avoid competing with critical API
calls" rationale doesn't apply across origins (web is HTTP/2, API is
a different host); the delay only pushed live updates ~2s into the
load for no benefit.
- FeedbackFunction: keep the kill-switch check sequential, then run the
remaining four Table reads (role, tenant config, sessions existence,
feedback cooldown) in parallel via Task.WhenAll. Endpoint dropped
from ~1285ms (6 sequential reads on Flex Consumption) toward the
slowest single read.
- TimingAllowOriginMiddleware: emit Timing-Allow-Origin: * on every
function response so the portal's Resource Timing API can read API
payload sizes (transferSize / nextHopProtocol) for our own RUM —
cross-origin entries currently report 0/unknown.
Backend tests 1058/1058, Web tsc clean, vitest 249/249.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replaces the fire-and-forget Task.Run inside EventIngestProcessor — which
Azure Functions scale-in could kill mid-flight, leaving sessions with no
vulnerability_report and forcing users to click "Analyze Now" — with a
dedicated queue + BackgroundService worker (vulnerability-correlate),
mirroring the analyze-on-enrollment-end pattern. Adds a V2-agent trigger
that emits a Part-1 software-inventory snapshot the moment Windows Event
62407 fires (WhiteGlove sealing success), so Part-1 vuln correlation
happens BEFORE the admin clicks Reseal — closing the gap that existed
because reboot_kill termination at sealing left no shutdown inventory
for the backend to correlate.
Backend
* IVulnerabilityCorrelateProducer + AzureQueueVulnerabilityCorrelate
Producer (fail-soft on agent's hot path).
* VulnerabilityCorrelateQueueWorker (BackgroundService, 5-attempt
visibility-timeout retry, poison-queue on exhaustion).
* VulnerabilityCorrelateHandler ports the full pipeline (correlate +
store + synth event + re-analyze enqueue + SignalR notify).
* IVulnerabilityInventoryLoader extracted from
GetVulnerabilityReportFunction so worker + manual rescan share one
loader; phase-aware filter via whiteGlovePart parameter +
includeAllPhases escape-hatch for the rescan UI.
* EventIngestProcessor cutover: drops three concrete-class deps
(_vulnerabilityCorrelation, _vulnRepo, _signalRNotification) and
two now-orphaned private methods. 561 → 388 LOC.
Agent V2
* WhiteGloveInventoryTrigger subscribes to EspAndHelloHost.WhiteGlove
Completed and calls AgentAnalyzerManager.RunWhiteGlovePart1Inventory
Snapshot via injected Action (no concrete-class coupling, trivially
mockable). Single-fire guarded by Interlocked.Exchange.
* IWhiteGloveCompletedSource interface keeps EspAndHelloHost internal
sealed while letting the trigger be unit-tested with a 4-line stub.
* AgentAnalyzerManager.RunWhiteGlovePart1InventorySnapshot runs ONLY
SoftwareInventoryAnalyzer (LocalAdmin/IntegrityBypass deliberately
skipped at WG hand-off — still SYSTEM context).
* EnrollmentTerminationHandler now also tags Part 2 (WhiteGloveCompleted
Part2 → wgPart=2); was previously untagged so the new phase filter
couldn't see it.
* SoftwareInventoryAnalyzer.AnalyzeAtShutdown gets per-WG-phase atomic
idempotency via Interlocked.CompareExchange, with reset-on-failure
so the EnrollmentTerminationHandler stays a working fallback if the
Trigger crashes.
Cross-phase contamination & out-of-order resilience
* Inventory loader filters by whiteGlovePart so a Part-2 correlation
cannot see Part-1 inventory items (and vice versa).
* Handler checks the existing report on BOTH Part-1 and Part-2 paths:
Part-2 merges with existing Part-1, Part-1 SKIPS its store when the
existing report already contains user_enrollment / both findings or
has scan_summary.phase_contributions.user_enrollment > 0 — protects
against late-arriving Part-1 retries overwriting the richer Part-2
or merged report after at-least-once redelivery.
* VulnerabilityCorrelationService.MergeReports gains NormalizeForMerge
that recursively unwraps Newtonsoft JObject/JArray/JValue to native
Dictionary/List/primitive shapes BEFORE the cast-based extract
helpers run. Without this the normal-order Part-2 merge silently
dropped Part-1 findings, because GetVulnerabilityReportAsync round-
trips through JsonConvert and the existing helpers' OfType / "as
Dictionary" casts produced empty results on JTokens.
Tests (+34 new across Backend & Agent V2.Core)
* VulnerabilityCorrelateEnvelopeSerializationTests — wire-format pin.
* VulnerabilityInventoryLoaderTests — phase filtering, includeAllPhases,
lenient long-parsing.
* VulnerabilityCorrelateHandlerSkipTests — out-of-order skip helper
against native, JArray, JObject, both phase tag, phase_contributions
counter.
* VulnerabilityCorrelationServiceTests — round-trip safety of
MergeReports against JsonConvert-deserialized existing reports.
* WhiteGloveInventoryTriggerTests — fire-once, dedup, dispose-then-late,
idempotent dispose, action-exception isolation, null-arg validation.
* SoftwareInventoryAnalyzerIdempotencyTests — Part-1 dedup, Part-2
dedup, Part-1 ≠ Part-2 independence.
Build green, 0 errors, 0 warnings. Backend 844/844 tests pass, Agent
V2.Core 825/825 tests pass (parallel-flaky tests pass isolated, see
feedback_v2_recoverypath_corrupt_snapshot_flake).
V1 ingest path (/api/agent/ingest) deliberately untouched per
project_v1_vuln_pipeline_gap — V1 is being phased out; fixing it
forward would inflate scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Until now, hardware-whitelist rejections were only visible to tenant admins
who actively navigated to the settings page or had a webhook configured.
Admins missed new rejected models if neither was the case.
This adds a parallel tenant-scoped notification pipeline (separate from the
existing GlobalAdmin-only feed): on the first-ever HardwareNotAllowed
distress per (tenantId, manufacturer-lower, model-lower), a persistent
bell notification is created with a deep link to the hardware-whitelist
page. Lifetime dedup via a dedicated tracker table makes it a one-shot per
model. The webhook path is unchanged and fires in parallel.
Routes are gated TenantAdminOrGA + TenantScoping.Jwt for now; an Operator
follow-up (new TenantAdminOrOperator policy) is tracked separately.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous design ran RuleEngine in Task.Run after HTTP 200 — Functions scale-in
could kill it mid-flight, leaving rule results un-persisted and forcing manual
"Analyze Now" clicks. New design enqueues onto analyze-on-enrollment-end and
runs the engine in a BackgroundService worker with visibility-timeout retry +
poison move after 5 attempts (mirrors telemetry-index-reconcile pattern).
Three trigger sites cut over: V1 ingest, V2 EventIngestProcessor primary, V2
post-vulnerability re-run. Manual "Analyze Now" remains as fallback.
Plus retry-semantics fix: RuleEngine.AnalyzeSessionAsync no longer swallows
storage exceptions in an outer try/catch (per-rule inner catch stays), and the
handler now throws when StoreRuleResultAsync returns false. Without this the
worker would have deleted messages on transient Table Storage failures and never
retried.
Tests: +4 failure-propagation, +6 envelope round-trip, 762 backend pass.
Follow-ups noted: TableStorageService helpers still swallow exceptions one
layer below; SignalRNotificationService needs interface for handler-level
unit tests; V1 ingest path missing vulnerability_report synth event.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Container App scale-to-zero clears the in-memory MSRC and KEV indexes on
every cold start, leaving a window where session correlation runs without
those data sources until the next refresh. Re-fetching ~1800 MSRC CVEs
plus the KEV catalog from upstream on every cold start is wasteful and
slow; persisting the snapshots to blob solves both problems.
New BlobBackedReferenceCache<T> generic helper handles single-blob JSON
snapshots in the autopilotmonitor / vulnerability-data container:
- msrc-index.json — full MSRC CVRF index (~350 KB for last 6 Patch Tuesdays)
- kev-catalog.json — full CISA KEV catalog (~500 KB)
Both services gained:
- EnsureHydratedAsync — idempotent, semaphore-guarded blob load
- Blob-write at the end of RefreshAsync (manual trigger + daily timer)
- Hydrate guard set after refresh so the no-op path is taken next time
VulnerabilityCacheWarmer (IHostedService) kicks off both EnsureHydratedAsync
calls fire-and-forget at app startup so re-deploys do not pay a blob-read
latency cost in their cold-start window. The correlation hot path also
calls EnsureHydratedAsync as a defensive backstop — idempotent and
semaphore-protected, so a request arriving mid-warmup awaits the same
in-flight load instead of duplicating it.
NVD's per-CPE cache stays in Table Storage — random-access lookup by CPE
key plus per-key 7d stale-refresh logic is the right shape for that
workload and would degrade if forced into a single-blob model.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
MsrcApiClient was injected into VulnerabilityCorrelationService for months
but never invoked — every report shipped with source="NVD" only. New
MsrcCorrelationService owns refresh + per-CVE lookup against an in-memory
index built from the last 6 Patch Tuesdays; daily VulnerabilityDataSync
populates it; CorrelateAsync enriches each finding with msrcDocumentId,
msrcReleaseDate, msrcSeverity, msrcBaseScore, msrcVector and flips source
to "NVD+MSRC" when the CVE was published in the indexed window.
Health check also surfaced that all worker-side traces of the daily sync
and CPE auto-resolve were sampled out of App Insights despite running.
Adds Event to host.json excludedTypes and emits TelemetryClient events
for VulnerabilityDataSyncCompleted, VulnerabilityCorrelationCompleted,
CpeAutoResolveHit/Miss, CpeMappingPersisted so customEvents queries can
prove freshness without depending on sampled traces.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The QueueTrigger binding required a Functions-host-specific app setting
(`<Connection>__queueServiceUri` with `__credential=managedidentity`) that
diverged from the project's storage-access pattern. Tables, blobs and the
index-reconcile producer all use a single `AzureStorageAccountName` (Managed
Identity) or `AzureTableStorageConnectionString` (full conn-string) and build
their clients with `DefaultAzureCredential` directly via DI — no host-side
binding glue.
In production the trigger silently failed to bind on every Function-App
worker start ("Error indexing method 'Functions.IndexReconcile'. Storage
account connection string 'AzureWebJobsAzureTableStorageConnectionString'
does not exist"), so messages enqueued by the producer would have piled up
unconsumed once `EnableIndexDualWrite` was flipped on.
Replaces:
- IndexReconcileFunction.cs (40 LoC, QueueTrigger) → DELETED
- New IndexReconcileQueueWorker.cs (BackgroundService) — same DI-based
config resolution as AzureQueueIndexReconcileProducer, full
QueueTrigger-platform parity (visibility-timeout retries, poison-queue
move after 5 failed attempts, malformed-JSON drop, idle 10s sleep).
Also pre-existing TableInitializerService is registered as IHostedService
in the same project, so the pattern is established.
Build: 0 warnings, 0 errors. Backend tests: 736/736 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consumes the telemetry-index-reconcile queue (M5.d.2) and fans each
envelope out into 0–3 index-table rows via IIndexTableRepository
(M5.d.1). Closes the producer→consumer loop; once the
EnableIndexDualWrite flag is flipped, primary commits drive
eventual-consistent index upserts asynchronously.
Components:
- Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues 5.5.4
package for the isolated-worker QueueTrigger binding.
- IndexReconcileHandler (plain class — not interface-abstracted per
Plan §M5.d design: DAL abstraction lives at IIndexTableRepository,
not here). Routing:
Signal → SignalsByKind (always, 1 row).
DecisionTransition:
IsTerminal=true → SessionsByTerminal
Taken=true → SessionsByStage
Taken=false + reason → DeadEndsByReason
ClassifierVerdictId!=null → ClassifierVerdictsByIdLevel
Up to 3 rows per DecisionTransition envelope (a Taken+terminal+
classifier transition fans to SessionsByTerminal + SessionsByStage
+ ClassifierVerdictsByIdLevel).
- IndexReconcileFunction thin QueueTrigger wrapper
(Connection = AzureTableStorageConnectionString). Malformed JSON
logged + dropped to avoid poison-queue retry loops; transient
repo errors propagate so the platform retries.
- DI: handler registered as Singleton in Program.cs.
Failure semantics:
- Required-field-missing envelopes (Signal without SessionSignalOrdinal
or SignalKind; DecisionTransition without StepIndex) → log warn,
drop. These won't become valid on retry — avoiding the retry loop.
- Transient Table Storage errors → rethrow. QueueTrigger platform
retries up to 5 times by default, then moves to
telemetry-index-reconcile-poison for manual investigation.
Deferred to M5.d.4:
- SessionsByStage stale-row pruning: when a session moves from
Stage-X→Stage-Y, the old Stage-X row stays. Either query-side
dedupe (per SessionId, max-Ticks RK) or reconcile-timer pruning
handles this. Deliberately not solved in the handler to avoid
a query+delete round-trip per Taken transition.
- End-to-end dual-write verification tests (Azurite harness) are
the M5.d.4 release gate before the flag is flipped.
Tests (+10 in Functions) via capture-only FakeIndexTableRepository:
- Signal happy path (1 SignalsByKind row).
- Signal missing SessionSignalOrdinal / SignalKind → 0 rows (x2).
- DecisionTransition Terminal+Taken+Classifier → 3 rows.
- DecisionTransition Taken non-terminal no-classifier → 1 row
(SessionsByStage only).
- DecisionTransition dead-end with classifier → 2 rows
(DeadEndsByReason + ClassifierVerdictsByIdLevel).
- DecisionTransition dead-end without reason → 0 rows (PK-collision
defense).
- DecisionTransition missing StepIndex → 0 rows (all paths gated).
- Unknown SourceKind → 0 rows.
- Null envelope → 0 rows.
Full Functions suite: 682 green (prev 672 + 10 new). Full solution
build green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds the async fan-out producer path from Plan §2.8 / §M5.d so primary
Signals + DecisionTransitions rows can be projected onto their index
tables without blocking the hot ingest loop.
Components:
- Constants.QueueNames.TelemetryIndexReconcile (azure-queue-name-safe,
all lowercase + hyphens).
- IndexReconcileEnvelope (Shared DTO, flat shape with nullable fields
per SourceKind="Signal"/"DecisionTransition"). Avoids polymorphic
Newtonsoft deserialization — see M5.c.3 finding.
- IndexReconcileEnvelopeFactory pure static helper (FromSignal /
FromDecisionTransition / BuildBatch) — state-less projection.
- IIndexReconcileProducer (Shared interface) contract: no-op when
the feature flag is off; never rethrows underlying-queue failures.
- AzureQueueIndexReconcileProducer (Functions) — Base64-encoded JSON
messages, lazy CreateIfNotExistsAsync guarded by Interlocked flag,
Managed-Identity-or-connection-string construction mirroring
BlobStorageService / TableStorageService.
- IngestTelemetryFunction ctor gains IIndexReconcileProducer; after
primary Signals + DecisionTransitions commit it builds envelopes
and enqueues them. Event path is untouched (events don't feed the
index tables per §2.8).
- Program.cs registers the producer as a singleton.
Feature gate (AdminConfiguration.EnableIndexDualWrite, default false):
- New bool field in AdminConfiguration.
- TableConfigRepository Store + Map extension (per
feedback_table_storage_serialization).
- Producer checks config each call via AdminConfigurationService's
5-min memory cache. Flag off → primary behaviour is bit-exact.
Fault tolerance: queue-side exceptions (network, throttling, missing
queue before first CreateIfNotExists) are logged per envelope and
swallowed. The primary row is the source of truth; the 2h reconcile
timer (M5.d.4) is the safety-net recovery path.
Tests (+11 in Functions):
- 5 IndexReconcileEnvelopeFactoryTests: Signal/DecisionTransition
projection, dead-end preservation, batch ordering, null/empty.
- 3 IndexReconcileEnvelopeSerializationTests: Newtonsoft round-trip
(Signal, full DecisionTransition, minimal-envelope tolerance).
- 3 AdminConfigurationEnableIndexDualWriteTests: default-false on
new config + CreateDefault, persists when set.
Deferred to later sub-milestones:
- M5.d.3: queue-triggered consumer that maps envelopes to 0–3 index
rows using M5.d.1 TableIndexRepository.
- M5.d.4: 2h reconcile timer + release-gate dual-write verification
tests, then flip EnableIndexDualWrite=true in prod admin config.
Full Functions suite: 672 green (prev 661 + 11 new). Full solution
build green.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extract the post-parse event-processing tail of IngestEventsFunction into a
standalone EventIngestProcessor service so /api/agent/telemetry runs the same
downstream logic as /api/agent/ingest. Deliberate copy-duplication, not a
shared refactor:
- Legacy /api/agent/ingest is production-hot — it serves live traffic from
every deployed V1 agent and is NOT touched by this commit. Zero regression
risk on the existing path.
- EventIngestProcessor is a near-verbatim copy of the legacy pipeline's tail
(~1000 LoC). Duplication is accepted because legacy /api/agent/ingest is
scheduled for decommission post-v11 GA (see tasks/todo.md → Follow-Ups →
Legacy-Agent-Ausmusterung). Until then: bugs fixed in legacy must be ported
here manually.
Copied pipeline steps (all 1:1 with legacy ProcessIngestAsync lines 150-566):
StampServerFields → SanitizeEventTimestamps → StoreEventsBatchAsync →
Index updates fire-and-forget → IME agent version history → ClassifyEvents
→ App install summaries → Geo-location update → UpdateSessionStatusAsync
(all 7 status branches) → IncrementSessionEventCountAsync → Rule engine
analysis fire-and-forget + SignalR + stats → Vulnerability correlation with
WhiteGlove Part-2 merge → IncrementPlatformStatAsync TotalEvents +
SuccessfulEnrollments → RecordGatherRuleStatsAsync → Diagnostics blob
update → GetSession for updatedSession → Session age 4h warning → WhiteGlove
IsPreProvisioned error log → SendWebhookNotificationsAsync (all 3 alert
paths) → SLA breach evaluation → Admin override detection → ServerAction
fetch-and-clear with TrackEvent telemetry → BuildSignalRMessages.
Shared (not duplicated, to avoid drift in pure helpers):
IngestEventsFunction.StampServerFields, SanitizeEventTimestamps,
RecalculateAppDurations (all internal static, state-less, stable). The
internal DTOs EventClassification + AppInstallAggregationState are also
reused — they carry no behaviour, only fields.
IngestTelemetryFunction wiring:
- Return type HttpResponseData → IngestEventsOutput so the V2 endpoint can
emit SignalR messages for the real-time UI push (matches legacy binding).
- PersistItemsAsync partitions the batch: Events go through the processor
(full pipeline + AdminAction + pending ServerActions + SignalR); Signal +
Transition batches skip the processor and read control signals directly
via a lean ReadControlSignalsAsync helper on ISessionRepository — no point
firing the full pipeline for a batch that carries no events.
- EventIngestProcessor registered in Program.cs DI.
Known gap (explicitly NOT in this commit): the V2 endpoint lacks a payload
size guard (legacy reads MaxNdjsonPayloadSizeMB from tenant config in
ParseNdjsonRequest). Different payload shape (JSON array vs NDJSON), deferred
to a dedicated follow-up.
All 596 backend tests green (unchanged count — copy path has no new tests
yet; the V2 endpoint's behaviour is validated by the legacy test coverage
through the shared-helper surface + the TelemetryPayloadParser tests from
M5.b.1). Integration coverage lands with M7 test-VM smoke.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Oliver Kieselbach