mirror of
https://github.com/okieselbach/AutopilotMonitor.git
synced 2026-10-24 20:53:06 +00:00
ea6a375c4b4d3fb3c21ee57244fe7399c07d0ab2
109 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Oliver Kieselbach
|
ea6a375c4b |
feat(security): cert-to-device binding, stage 3 GA-gated shadow
Tenant binding (stage 1) proves the client certificate was issued to this
tenant. It does not prove the certificate belongs to a device the tenant
still has: a cert lifted from a decommissioned machine keeps working.
IntuneDeviceBindingValidator resolves the Intune managedDevice id from the
certificate's Subject CN against the tenant's own managedDevices inventory.
Point-GET on managedDevices/{id} rather than an OData filter, so there is no
filter literal to inject into and a 404 is unambiguous. Cache/retry/transient
contract mirrors CloudPcDeviceValidator.
Shadow only, and visible only to Global Admins (same gate as the DevPrep
association preview) because the open question is a race, not an attack: a
device object can appear after the agent's first call. Every result carries
enrolledDateTime and the log line records enrolledAgeSeconds, so a device
created seconds before the request is distinguishable from a foreign cert.
Grep marker: CERT-DEVICE-BINDING-SHADOW.
The lookup is fire-and-forget like the DevPrep shadow so a cold Graph
round-trip never sits in the agent request path. That rules out stage 1's
request-row dimension (the row is written before the task finishes), hence
the Warning log; widening beyond preview should move to inline + dimension.
Needs DeviceManagementManagedDevices.Read.All, granted per tenant via the
customer script as feature IntuneDeviceBinding. The script's catalog,
ValidateSet and help block stay pinned to GraphFeatureCatalog by
GraphFeatureCatalogSyncTests (extended with the help-block case).
Build 0 warnings, backend 4629/4629, web 874/874, tsc clean, eslint 0.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
0d5e2572aa |
perf(telemetry): one canonical request record; drop expected storage outcomes and SignalR success deps
- RequestTelemetryMiddleware: SamplingPercentage=100 per item (worker copy was adaptively sampled, ItemCount 2-5 on weekdays, despite host.json excludedTypes) - StorageDependencyFilterProcessor: also drop 404/412/409 storage outcomes (InProc + HTTP shape) and successful *.service.signalr.net REST calls - infra/appinsights-workspace-transforms.bicep: DCR dropping the host HTTP request duplicate at ingestion (timer/queue rows kept) - docs/backend/telemetry-ingest-shaping.md; skills updated Measured baseline 7d: ~1.3 GB/week removed, requests now 100% with tenant context. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
3a89319472 |
feat(telemetry): startup duration metrics for backend and MCP
StartupTelemetryService emits BackendStartupMs (process start ->
ApplicationStarted) and BackendTableInitMs as App Insights custom
metrics with version/commit/tableInitFullPass/instance dimensions;
TableStorageService exposes LastInitialization. MCP server writes one
{"type":"boot","readyMs"} stderr line per start next to the
tool_call lines so cold-start time is queryable as a trend.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
7569061983 |
feat(admin): editable email templates — preview, test send and HTML override in Settings
- EmailTemplateService: override-or-built-in resolution per send (PreviewConfig
table, partition EmailTemplates, 5-min cache, {{domainName}} placeholder,
30k limit); send path fail-soft, admin read path strict
- GA endpoints global/email-templates/{kind} (GET/PUT/DELETE) and /test, which
mails the effective template or an unsaved draft to the caller's tenant
contact address
- portal: Settings → Email Templates with preview frame, test send, HTML editor
- mail header: brand mark drawn with CSS borders, no images left in the mails
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
8675e7ee06 |
feat(backend): provider-neutral EmailService on Mailchimp Transactional (Mandrill), replaces Resend
- EmailService + IEmailService: Mandrill messages/send via typed HttpClient with the Notification resilience policy; provider confined to one transport method and the Email:* settings (ApiKey, Endpoint, FromAddress, FromName). Resend NuGet dropped. - open/click tracking explicitly off; sent/queued/scheduled = ok, rejected/invalid = warn - callers (TenantApprovalService, PreviewWhitelistFunction) depend on IEmailService - EmailServiceTests pin wire format, result semantics and fail-soft contract - privacy page + OKF welcome-mail doc updated for the new sub-processor Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
935f0e8c71 |
feat(backend): agent-efficiency aggregate + honest scan count + filtered metrics fetch
- New GET /api/global/metrics/agent-efficiency (GlobalReadOrAdmin, ?tenantId=): per-agent-version percentiles (CPU, working set, private bytes, threads, handles, spool, API latency/requests), crash rates per version, top offender sessions per dimension, 5-min cache keyed (days, limit, tenant). First backend consumption of agent_thread_count/agent_handle_count. - PlatformAgentMetricsResponse.SessionsScanned: sessions the scan actually covered — the honest basis for the caller's truncated flag (Sessions.Count only counts sessions WITH snapshots). - Per-session fetch no longer drains the full Events partition: new GetSessionEventsByTypesAsync (multi-type OData filter + column projection + real maxResults enforcement); the 20 delivery-latency sample sessions keep the full fetch so latency deltas stay computed over all events. - Shared AgentMetricsAggregation (crash tally, payload getters, bounded fan-out, metrics event types) so the two services cannot drift. - GetAllSessionsPageAsync gains an optional select projection (default null, existing callers unchanged), pinned by an equivalence test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
ca26cb4f7c |
backend: auto-archive IME installer on first sighting of a new version
When RecordImeVersionAsync reports a genuinely new IME version (~monthly),
EventIngestProcessor now also enqueues an ime-msi-archive job. The worker
(QueuePollingWorker, heartbeat, pause on ImeMsiArchivingEnabled=false)
downloads the installer and stores it write-once (If-None-Match:*) in the
permanent ime-archive container ({version}/IntuneWindowsAgent.msi +
provenance.json), then merges MsiArchiveStatus/BlobPath/Sha256/Bytes/SourceUrl
onto the ImeVersionHistory row - so every fleet-observed IME build stays
available for decompilation/diffing after Microsoft's versionless CDN moves on.
The download URL comes from agent telemetry (msiDownloadUrl enrichment) and is
treated as untrusted: https-only *.manage.microsoft.com allowlist with exact
filename match, canonical versionless fallback, no redirects, size cap
(MaxImeMsiDownloadSizeMB, default 250, preflight + mid-stream) enforced by a
SHA-256-hashing pass-through stream. Archiver never throws; the worker retries
only transient statuses (download/timeout/error), permanent rejections
complete with their Failed:* status persisted.
Admin settings ImeMsiArchivingEnabled (default true) + MaxImeMsiDownloadSizeMB
via the full 4-file web chain; archive fields on the version-history API are
GA-only (non-GA projection unchanged, redaction tests extended). Shared
manifests regenerated. Tests: archiver status branches via recording subclass
(allowlist, caps, 409 idempotency, cancellation propagation), config defaults.
Backend suite 4262 green, web tsc + vitest 762 green, solution 0 warnings.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
72136f0273 |
backend+mcp+web: evaluateOn interim triggers for analyze rules
Implements docs/rules/analyze-rule-triggers.md (spec approved today): - AnalyzeRule.EvaluateOn (absent = [enrollment_end], today's behavior bit-for-bit) with the shared AnalyzeRuleTriggers grammar/matching helpers; grammar enforced in tenant rule CRUD and the analyze-rule JSON schema. - RuleEngine: AnalyzeRunContext (Terminal/Interim/Reanalyze), interim rule filter, KO suppression on interim runs, and RuleResult update-semantics replacing the permanent dedupe freeze — interim rows refresh, the terminal pass finalizes or resolves them (ResolvedAt, kept for audit), Reanalyze preserves lifecycle markers. - Notification dedupe moves onto RuleResult.NotifiedAt (one send per session+rule, stamped after a successful send; survives the manual reanalyze rebuild). - Handler: whiteglove_sealed / interim_trigger reasons — notify + SignalR yes, IssuesDetected + rule stats terminal-only (Rule Regression Radar baselines clean). - Ingest: InterimTriggerRegistry (per-tenant 5-min TTL cache, fail-soft) gates the interim_trigger enqueue (batch eventType intersection) and the whiteglove_sealed enqueue on the first genuine WhiteGlove seal. - Storage: EvaluateOnJson on AnalyzeRules; FirstDetectedAt/LastEvaluatedAt/ IsInterim/ResolvedAt/NotifiedAt on RuleResults (nullable — legacy rows read as final). - MCP: three authoring lints (terminal-precondition trap, unknown/high-frequency on_event types, KO-on-interim), authoring guide + baked schema updated; get_session_summary marks interim and excludes resolved findings. - Web: preliminary badge on interim findings, resolved findings behind a toggle, issue counts exclude resolved (analysis section + diagnosis flow). - ANALYZE-ID-004 v1.1.0: first interim-enabled rule (on_event:hybrid_login_pending), threshold raised to 75 so only repeated overdue-login signals fire mid-run. Suites: Backend 4152, MCP 548, Web 762 + tsc + lint clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
1c5f69a9fe |
w365: CloudPc device validator — cert-CN-bound fallback for Cloud PCs (#127)
* w365: CloudPc device validator — cert-CN-bound fallback for Cloud PCs Windows 365 Cloud PCs are never Autopilot-registered, so the serial lookup (and the Corporate Identifier fallback) structurally reject them and Cloud PC first-connect enrollment could not be monitored. New stage-3 fallback in the device-validation chain: extract the Intune device id from the chain-validated MDM client certificate's Subject CN (field-verified: CN == managedDevice id == cloudPC.managedDeviceId) and require a matching cloudPC object in the claimed tenant (Graph v1.0 virtualEndpoint/cloudPCs, $filter=managedDeviceId eq). Only service-provisioned Cloud PCs have such an object — a regular enrolled non-Autopilot device still ends in the 403, and local W365 markers are never consulted for auth. - Shared: ValidatorType.CloudPc = 5, TenantConfiguration.ValidateCloudPcDevice (default false, tenant-admin writable), GraphAppPermissions.CloudPCReadAll, GraphFeatureCatalog feature W365CloudPcValidation -> CloudPC.Read.All - Backend: CloudPcDeviceValidator (AutopilotDeviceValidator sibling contract: 30/5 min cache, 2 attempts, transient->503; GUID gate before the OData filter; exact-match guard; Graph 403 = definitive "grant missing", not transient), SecurityValidator CN extraction + fallback stage + hard gate, wiring through the 5 agent endpoints, TableConfigRepository persistence - Consent script: W365CloudPcValidation in catalog/ValidateSet/help; NEW GraphFeatureCatalogSyncTests pins C# catalog <-> grant script (previously comment-only lock-step) - Bootstrap dev script v2.3-dev.3: guard 4 (uptime window) exempted while the Cloud-PC relax is active — Cloud PCs run headless for days before first connect; the fresh-profile window is the time anchor there (+ Pester tests) - Web: validation toggle + context plumbing, ValidatedBy "CloudPc" label, privacy-page validation claim generalized (Last updated bumped) - Docs: OKF concept doc backend/cloudpc-device-validation.md + index + log Verified: web tsc clean, ESLint 0 errors (99 pre-existing warnings = ratchet cap, none added). .NET build/tests and Pester run in CI — SDK download and pwsh are unavailable in this sandbox's network policy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RqEFrcAHhWgCvgJgHGzMuv * w365: agent-side IsCloudPc flag + engine expectation "no Device-ESP phase" Companion to the CloudPc validator: the agent now classifies Windows 365 Cloud PCs locally with the bootstrap's field-verified marker AND (HKLM\SOFTWARE\Microsoft\Windows365 key + installed CloudManagedDesktopExtension service, probed via its Services registry key so net48 needs no System.ServiceProcess). SKIP-safe: any error resolves false. Never an auth input - backend admission stays exclusively the cert-CN-bound Graph lookup (ValidatedBy=CloudPc); the two signals are independent and may disagree. Two transport rails: - Session metadata: SessionRegistration.IsCloudPc -> Sessions table (sticky-true OR across re-registrations, IsSelfDeployingProfile pattern) -> SessionsIndex full mirror -> SessionSummary -> new isCloudPc search filter end to end (SearchSessions + QueryRawSessions + pagination fingerprint + all three in-memory predicates; MCP search_sessions / query_raw_sessions params + summary projection; web type, Cloud PC pill, SessionInfoCard enrollment label, CSV export appended column). - Decision engine: isCloudPc rides the EnrollmentFactsObserved payload, recorded set-once for BOTH values in EnrollmentScenarioObservations.CloudPc and annotates the scenario profile reason cloud_pc_first_connect:no_device_esp_expected - the engine-recorded expectation that Device-ESP already ran headless at provisioning and the session starts at Account Setup. Deliberately reason+observation only: no completion arm depends on DeviceSetup, Mode stays AccountSetup/IME- classified (Classic), EspConfig keeps configured-FirstSync semantics. Positive marker surfaces as cloud_pc_marker in the audit-trail census. Updater idempotency: the cloudPcAnnotates gate keeps the generic v1 reason branch from re-claiming the slot on repeat posts (would bump EvidenceOrdinal every post). Additive-nullable observation - no state schema bump (v4 convention), serializer roundtrip pinned. Tests: CloudPcDetector resolver (marker AND, throwing probe), registration body, facts-signal payload shape, updater annotation/idempotency/ self-deploying interplay, reducer observation both values + census fold, serializer roundtrip, index projection superset, sticky-true both directions. Suites: DecisionCore 500, Functions 3941, Agent V2 2154, vitest 632 - all green; tsc web+MCP clean; ESLint 0 errors (cap 99). Docs: cloudpc-device-validation.md follow-up section replaced with the shipped mechanism + log entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
395fdd49df |
Transactional tenant-config writes via MCP + ConfigurationBackups safety net (#120)
* feat(backend): pre-write config backup net - every tenant/admin config save snapshots first New ConfigurationBackups table (PK=tenantId or GlobalConfig, RK=reverse-ticks+guid, newest-first) holds the raw stored row as EntityJson before any overwriting save of TenantConfiguration or AdminConfiguration. Hook lives in TableConfigRepository, the single choke point all writers funnel through - fail-soft (a backup hiccup never breaks a save), skips row creations and noise-only changes (LastAuthClientId flips), prunes to the newest 2 per partition. Source/reason tagging via a new SaveTenantConfigurationAsync overload (not optional params - Moq expression trees, CS0854); portal-put/plan/auth/app-homing/offboard call sites tagged. Offboarding: backup partition wiped in the bulk phase (TenantPartitionTables) and by the marker-cleanup tombstone sweep; the GlobalConfig partition never matches a tenant GUID and survives. Table registered in TableNames.All + CriticalBackupTables.All. Groundwork for the transactional MCP config-write flow (patch/revert endpoints follow). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(backend): transactional tenant-config field patch + revert with verified writes New GA-only surface (policy catalog: GlobalAdminOnly + RouteParam): - PATCH config/{tenantId}/fields - field-level patch through TenantConfigPatchService: fresh ETag read -> field gate (deny-list incl. plan/trial, HomedAppClientId, auth provenance; unknown fields 400 with the name) -> patch onto a clone -> shared validation -> fail-CLOSED backup -> CAS replace -> fresh re-read -> verify that EXACTLY the intended fields changed -> automatic ETag-conditional rollback on drift (catches Store/Map serialization asymmetry). - POST config/{tenantId}/revert - restore from a ConfigurationBackups snapshot (latest by default); snapshots current state first (revert is revertible), preserves plan/homing/provenance fields unless includeProtectedFields. - GET config/{tenantId}/backups - metadata + masked diff only, never EntityJson. Supporting changes: IConfigRepository gains GetTenantConfigurationWithEtagAsync + TryReplaceTenantConfigurationAsync (412 -> false, no implicit backup hook); validators moved from the PUT into shared TenantConfigValidation (PUT re-pointed, forwarding shims keep test call sites); GET config/{tenantId} honors ?view=redacted so a GA's MCP session never pulls clear-text secrets into model context. Caller-tier deny-lists (TenantAdmin/DelegatedAdmin) ship now for the phase-2 flip. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat(mcp): tenant-config write tools - update/revert with snapshot-backed verified writes First mutating surface on the MCP server, strictGa (real Global Admin) only: - get_tenant_config: full config read, ALWAYS ?view=redacted (secrets never reach model context; pinned by config-write-tools tests) - update_tenant_config: field-level patch via PATCH config/{tenantId}/fields; required reason; response = applied fields + masked diff + backupId - list_tenant_config_backups: snapshot metadata + masked diff only - revert_tenant_config: restore latest/named snapshot; protected fields stay current unless includeProtectedFields; revert is itself revertible New MUTATING annotation + TenantGuidSchema (path-traversal guard) in shared.ts; role-aware instructions headline (non-GA keeps the exact READ-ONLY wording); catalog snapshot test extends GA_FULL + new CONFIG_WRITE_GA_STRICT difference list (Global Reader/tenant/delegated see none of the four). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * test(backend): de-flake session-deletion heartbeat test - condition-gated handler instead of wall-clock window The heartbeat test held the handler for a FIXED 350ms and expected the 50ms heartbeat timer to tick >=3 times inside that window. On loaded CI runners the heartbeat Task.Delay stretches disproportionately to the handler delay, the handler completes with <3 heartbeats fired, and the awaited condition can never become true afterwards (CI failure on PR #120). The handler now holds the message in-flight UNTIL the test observed 3 heartbeats (with ct + own 15s deadline as termination guards), so the assertion is reached deterministically regardless of scheduler jitter. Verified: 12/12 local runs green, full suite green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
0e2d620ad4 |
feat(auth): self-service app-homing flip via consent flow + GA lever
Re-consent, verify and flip collapse into the existing consent flow: the
consent URL funnels legacy-homed tenants to the primary app registration,
and consent-status/access-check verify admin consent via an app-only token
probe under explicit primary credentials, then auto-flip HomedAppClientId.
- AdminConfiguration.SelfServiceAppHomingEnabled: kill switch (default off,
cache-bypassing fail-closed read); gates funnel + auto-flip + tenant-admin
manual flip. GA path deliberately survives the kill switch.
- POST config/{tenantId}/app-homing (TenantAdminOrGA): tenant admins flip to
primary only, probe-gated; GA gets force + revert. Pure decision helper
EvaluateManualFlip shared by function and tests.
- GraphTokenService.GetAccessTokenForAppAsync: probe seam with explicit app
credentials, cache-safe via app-scoped token keys.
- EntraAppRolesEnabled does not block a flip; it emits a Warning ops event
(AppHomingFlippedWithEntraRoles) reminding to re-assign enterprise-app roles.
- Web: homing badge + "App Registration Homing" panel with confirm dialog in
GA tenant management (typed homedAppClientId hardens the PUT round-trip),
funnel/success banners in tenant settings with switchAuthApp re-login,
admin-config kill-switch round-trip, ops event types dual-registered.
- Browser telemetry (App Insights): pre-init event buffer so module-init auth
events survive; strategic events across the whole flow (app_homing_flipped,
consent_flow_*, consent_verify_failed, auth_app_fallback_*, auth_app_switched,
auth_app_homing_learned, app_homing_manual_flip*).
Backend must deploy before web (additive JSON fields); everything is inert
while EntraId:LegacyClientId is unset. Enable the kill switch in the admin
portal after the dual app-reg config swap.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
f795fbcb4e |
feat(auth): dual app-reg parallel operation - legacy gktatooine app alongside new C4A8 app
True parallel operation, no forced re-consent: existing tenants keep signing in via the legacy app (1a400946), new tenants land on the C4A8 app (886ab5e2). Model "default legacy, funnel primary, learn in the background": - Web: am_auth_app flag -> NEXT_PUBLIC_ENTRA_DEFAULT_APP -> primary; get-started CTA hands over with ?authapp=primary; auth/me homedApp updates the flag so an operator tenant flip takes effect on each user's NEXT login; AADSTS90094/65001 one-shot cross-app retry, AADSTS65004 -> failed screen with previous-app link; am_login_attempt pins redirect returns to the app that started them. - Backend: TenantConfiguration.HomedAppClientId (null = legacy, GA-only flip via PUT config) + LastAuthClientId/Since observability; EntraAppRegistry resolves the acting app per tenant (no-op until the legacy config pair is set); Graph client-credentials, admin-consent URLs and the surfaced client id follow the homed app (app-scoped cache keys); JwtBearer [Authorize] pipeline now shares the middleware audience builders (was single-audience - would 401 legacy tokens after the swap); middleware trusts EntraId:LegacyClientId implicitly. - well-known association file now lists both app ids (publisher domain); deploy-web passes NEXT_PUBLIC_ENTRA_LEGACY_CLIENT_ID / _DEFAULT_APP; mcp-server.bicep default client id -> C4A8 app. Backend xUnit 3730/3730, vitest 625/625 (13 new), tsc/ESLint clean, next build ok. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
6bd5728652 |
feat(platform): public availability — tenant activation replaces Private Preview, optional auto-approve
Backend: shared TenantApprovalService (whitelist add + auto-promote shape-guard + welcome email), AdminConfiguration.AutoApproveNewTenants (default off, uncached fail-closed accessor), tenant-auto-approve queue (1-min visibility delay, worker drops on flag-off/approved/suspended/missing config, approves as "System (auto-approve)" + TenantAutoApproved ops event), 30s negative whitelist cache, gate 2 error PendingActivation with mode-neutral message, signup strings and welcome email without preview wording. Web: /preview → /activation (SWA 301, robots, AuthGate), AuthContext accepts both error codes and clears pending state on successful auth/me, activation page polls every 30s and auto-redirects into the portal, dashboard banner reframed to "Actively developed" + feedback call, terms/privacy/get-started updated, feature badges → Beta, admin-config web chain + auto-approve checkbox, OPS_EVENT_TYPES. Deliberately unchanged: PreviewWhitelist/PreviewConfig tables, internal preview/* API routes, preview_signup notification type, Microsoft DevPrep preview mentions. Verified: solution build 0 errors, xUnit 3701/3701, tsc 0 errors, vitest 612/612, ESLint ratchet holds (0/115), next build exports /activation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
48308df4ff |
feat: opt-in preservation of session diagnostics archive with session reports
Session reports persist while sessions and their diagnostics blobs are
removed by user deletion or retention. A new checkbox in the Report
Session modal (shown only when a diagnostics ZIP was uploaded, default
checked) lets the submitter include the archive: the backend streams a
server-side copy into the durable session-reports container.
- SessionReportDiagnosticsArchiveCopier: fail-soft copy (never blocks the
report), reuses the Hosted/CustomerSas read paths, re-verifies the
hosted tenant prefix via ClassifyBlobName, honours the admin size cap
and download timeout; outcome persisted as DiagnosticsCopyStatus
- Report row/API gain DiagnosticsBlobName + DiagnosticsCopyStatus
(written only when requested, omitted for legacy rows)
- report-metadata.json records includedDiagnostics {requested, blobName, status}
- Admin reports UI: second Download Diagnostics button + failure status line
- Privacy page: explicit exception for the opt-in report copy (Last updated bump)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
e0ad51b965 |
refactor(urls): one guard-enforced URL registry per component, agent downloads move to the alias
Every well-known host -- own subdomains, both blob accounts, the API origin, Graph, Entra login -- now lives in exactly one registry per component: Constants.cs (*BaseUrl) for C#, utils/config.ts for the web app, src/config.ts for the MCP server. lib/hostRouting.ts derives its hostnames from the registry and middleware.ts imports from it, retiring what turned out to be a fourth independent copy of the www/portal host pair. Three new guard tests (backend xUnit, web vitest, MCP vitest) scan non-test source and fail on any enforced host outside its registry. This is the lesson of the EU cutover: the registry existed then too, but LatestVersionsService (feeds the portal's version badge) and ValidateBootstrapCodeFunction carried their own literal copies of the legacy blob host and were missed -- kept working only by the fail-soft legacy mirror in build-agent.yml, whose silent failure would have pinned badge and self-update to stale bits with a green release. Both guards proved themselves on their first run, flagging the middleware host copy and a literal in an MCP tool description. Test files stay exempt on purpose: literals there are independent oracles, and ValidateBootstrapCodeResponseShapeTests now pins the full alias URL against the Constants-built one so registry changes must be acknowledged consciously. The two legacy readers plus the agent SelfUpdater now use Constants.AgentDownloadBaseUrl (download.autopilotmonitor.com/agent); the alias was verified live for version.json, version-v2.json and both ZIPs before the switch. The portal's bootstrap allow-list accepts the alias AND the legacy host (AGENT_DOWNLOAD_HOSTNAMES) for already-deployed customer scripts, which imposes a deploy order: web before or with backend. The legacy keepalive probes and the release mirror stay untouched -- they exist for exactly this transition. Also: the bootstrap short-link now targets www directly instead of riding the registrar's apex 301, and the welcome-mail docs button points at docs. instead of bouncing through www./docs. Suites: backend 3194, agent core 1925, DecisionCore 474, web 582, MCP 449 -- all green; next build verified (it, not tsc, proves the next.config.ts registry import). Docs: docs/url-registry.md. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
da958b5398 |
fix(backend): business timestamps survive storage migrations (OccurredUtc + RowKey decode)
The 2026-07-18 storage migration reset every row's system Timestamp, exposing that AuditLogs, OpsEvents and Events displayed, filtered and retention-swept on it (custom "Timestamp" writes were silent no-ops — reserved system property). All migrated rows read as 2026-07-18. - New Helpers/BusinessTimestamp: RowKey decoders (audit !revtick_guid, ops bare revtick, event yyyyMMddHHmmssfff_seq), RowKey-range OData clause builders (tick-exact, with the mandatory RowKey lt '"' guard so no lower bound can ever match legacy bare-GUID audit rows), and dual DateTimeOffset/DateTime column reads. - Writes in all three tables now persist OccurredUtc (custom column, copied verbatim by future migrations); no-op ["Timestamp"] removed. - Read mappers resolve OccurredUtc -> RowKey decode -> system Timestamp; GetEarliestSessionEventTimestampAsync uses the same resolution so StartedAt/Duration reconciliation can no longer absorb the migration date on migrated sessions. - Date-window filters and retention moved off the system Timestamp onto RowKey ranges; ops retention loses its full-table scan and the post-migration mass-delete cliff; audit retention runs a temporary second pass for legacy GUID rows (TODO: remove after 2027-03-01). - New GlobalAdmin backfill endpoint POST /api/maintenance/ backfill-occurred-utc (dry-run by default, batched, idempotent MERGE of only OccurredUtc, table=audit|ops) — Events rows are deliberately never mutated; their read path decodes the RowKey instead. - Tests: decoders, clause boundary proofs incl. GUID-safety pins, mapper precedence, backfill row decisions (Functions.Tests 3078). - Docs: docs/backend/business-timestamps.md (system Timestamp is never authoritative; migration checklist rule). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
d02fa454a7 |
fix(status): honest verdicts across all misclassification audit findings
Misclassification audit 2026-07-16 (1000 non-succeeded sessions, 22 deep
dives) surfaced six systematic misdeclaration sources. This implements
all of them:
1. DecisionCore: HelloNeverObservedPromote in the esp-exit advisory
handler — when the Hello policy read never committed and no wizard
was ever seen, hello_resolution is structurally unresolvable; promote
to AwaitingHello + hello_safety instead of hard-failing a provably
finished enrollment (sessions 2dac8298/7635ab18). Placed after the
enforcement-progress re-arm so active installs still win; advisory-
anchor variant stays exempt.
2. Ingest: both max-lifetime shapes (enrollment_failed/agent_timeout and
agent_shutting_down/max_lifetime) now route through the
EnrollmentTimeoutClassifier instead of hard Failed (tenant a53e67ec:
fully provisioned WG Part 2, user never logged in => AwaitingUser).
agent_timeout no longer counts as HasExplicitFailure; failure
webhooks fire only on actual failure verdicts.
3. Maintenance sweep: dropped the too-broad "IsPreProvisioned ne true"
exclusion (sealed devices are protected by their Pending status);
resumed WG Part-2 runs sweep normally with the window anchored to
ResumedAt. New stage-2 silence guard also fixes a latent bug where an
actively-reporting >5h enrollment could be terminalized mid-run.
4. Registration supersede: a fresh registration resolves older
non-terminal sessions of the same device (serial, placeholder-guarded)
as Incomplete("Superseded by ...") with live SignalR updates — closes
the forever-Pending orphan gap when WG Part 2 runs under a new id.
5. Admin retro-reconcile: POST /api/maintenance/reclassify-legacy
(GlobalAdminOnly, DRY-RUN BY DEFAULT) re-runs the classifier over the
pre-classifier "Session timed out" Failed backlog (~495 rows, sample
projected ~87% misdeclared) and resolves Pending orphans. Requires the
new opt-in allowTerminalReclassification override in the status
writer (admin-marked rows stay untouchable).
6. Polish: "0 of N" AccountSetup rollups keep their total (no more
fabricated "0/0" reasons, "never observed" wording when absent);
late-telemetry reconcile heals Incomplete/AwaitingUser to Succeeded
when straggler evidence proves completion; SLA page knows
AwaitingUser/Incomplete ordinals; script_failed on Success/exit-0 now
names the stderr rule in its message.
Tests: backend 2965, DecisionCore 474 (+6 new), agent 1777, web 557 —
all green.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
b0105b123a |
feat(maintenance): session-deletion run budget, live watchdog counters, manual trigger
- Watchdog fix: FanoutResult is caller-allocated and mutated live per tenant/session, so 30/60min LongRunning warnings report real progress instead of the always-zero counters (today's warning showed 0/0 while 5 tenants / 54 sessions were already done). - 50min run budget: the retention fanout stops cleanly at the deadline (tenant boundary + per-session check) and reports AbortedByBudget — no more risk of the 60min Flex functionTimeout hard-aborting mid-run. New Warning OpsEvent SessionDeletionMaintenanceBudgetExceeded; Completed gains abortedByBudget. - Blob-lease serialization: dedicated sentinel _lock/session-deletion-maintenance.lock (deletion-manifests container) via SessionDeletionMaintenanceLockStore + reused MaintenanceLeaseHolder; contended runs emit SessionDeletionMaintenanceSkippedLocked and return. - Manual trigger: POST /api/global/session-deletions/maintenance/trigger (GlobalAdminOnly, policy catalog) -> fail-hard queue producer -> SessionDeletionMaintenanceQueueWorker runs the same RunCoreAsync body with triggeredBy = requesting admin; lease probe returns 409 while a run is active. New Started OpsEvent carries triggeredBy. - Web: MaintenanceStatusBanner on the Session Cleanup page derives active / budget-exceeded / failed / idle from the lifecycle ops events and offers a Run-now button (202/409 handling). New event types dual-registered in OPS_EVENT_TYPES. - Tests: 7 new (budget abort immediate/mid-tenant, live counter increments, watchdog live-progress regression, BudgetExceeded + Completed flag, SkippedLocked, Started triggeredBy); suites green (backend 2935, web 557, tsc clean). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
d8e8cd15fd |
feat(entitlements): Community/Enterprise editions with self-service trial
Fail-closed entitlement layer (FeatureEntitlementCatalog + TenantEntitlementService): edition resolves at read time (PlanTier == 'enterprise' OR active trial; legacy free/pro => Community, no migration).
- Plan/trial endpoints: PATCH config/{tid}/plan (GA: tier + grant/extend/end trial, audited, cache-safe) and POST config/{tid}/trial (tenant-admin self-service 30d, once per tenant, 409 after). Generic PUT preserves plan/trial fields for all callers. Fixes SetPlanTier writing past the config cache without audit/UpdatedBy.
- Rate limits: Enterprise floor 150/min for BOTH user (portal/JWT) and device (agent/cert) paths - floor raises the global default, an explicit per-tenant override still wins; MSP users follow their home tenant.
- Retention cap: Community 90d / Enterprise 365d, clamped in the retention fanout (warn log) + non-GA write validation 7..cap (only on changed values; 0=infinite stays GA-only).
- Delegated admin (MSP): Enterprise gate on the delegated admin's HOME tenant (JWT tid, fail-closed on unknown) - managed target tenants may be any edition. Grant-time target check removed; inert grants are suppressed at resolve time (single choke point in GetScopeAsync, applied after the UPN-keyed cache).
- MCP quota: McpQuotaService + McpQuotaEnforcementMiddleware (X-Client-Source: mcp only, GA exempt) enforce daily/monthly budgets - per-user plan override > tenant edition plan; limits from SectionUsagePlans with catalog fallbacks (community 100/3000, enterprise 1000/20000); 429 with X-MCP-Quota-* headers + Retry-After; fail-open on counter reads, fail-closed on plan resolution. /api/metrics/mcp-usage/me reports effective plan + counters.
- Trial visibility: TrialExpirySweepFunction (daily 03:30 UTC, informational) emits TenantTrialExpired (Warning, 24h look-back) + TenantTrialExpiring (Info, <=3d heads-up); dual-registered in web OPS_EVENT_TYPES.
- Web: EditionBadge + one-time trial CTA in settings header, Plan & Trial card in the GA tenant edit modal (own PATCH save), retention cap hints from entitlements, pure lib/edition.ts (vitest), feature-flags surface edition/trial/entitlements; MCP TENANT_SAFE_FIELDS + trial fields.
Verified: solution build clean; backend 2653/2653 tests; web tsc + 516/516 vitest; MCP server tsc + tests green.
|
||
|
Oliver Kieselbach
|
34582c0f0e |
feat(kill-switch): deliver kill on config channel + KillSignalDelivered ops event
Kill/Block rules were only served on the telemetry response, which cannot reach an agent whose upload loop is paused (indefinite block pins the drain until DateTime.MaxValue), whose spool is empty, or whose binary is kill-blind. Config is fetched at every process start (boot via Scheduled Task), making it the reliable control channel - no heartbeat added by design (48h emergency break + 6h max lifetime bound the remaining cases). Backend (effective on deploy): - KillSwitchEvaluator: shared device+version check for both channels; ingest wire response and check order unchanged (device first, short-circuits) - Every served Kill emits ops event KillSignalDelivered (Security/Warning), throttled 24h per tenant+serial+pattern; Blocks stay event-free - GET agent/config (incl. bootstrap wrapper) now returns DeviceBlocked/ DeviceKillSignal/UnblockAt alongside the full config; old binaries ignore the new fields - Web: KillSignalDelivered dual-registered in OPS_EVENT_TYPES Agent V2 (effective with next agent release): - CheckConfigKillSignal guard after the Phase-4 fetch: forced self-destruct (overrides SelfDestructOnComplete), enrollment-complete marker, session cleared, exit before mTLS/registration - Kill honoured ONLY from a live fetch: config disk cache strips the flags on write and read (UnrestrictedMode pattern), guard re-checks LastFetchOutcome Tests: 9 KillSwitchEvaluatorTests + 7 ProgramConfigKillSignalTests; Functions 2557/2557, Agent V2 1578/1578, web tsc clean. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
9f7b070e80 |
chore: remove legacy V1 agent and its backend ingest path
The V1 agent line is retired: no new V1 sessions since 2026-05-09, only orphaned straggler traffic remained on /api/agent/ingest (last seen 2026-06-29, events for long-cleaned-up sessions). Bootstrap/Intune/CI have deployed V2-only since the v2 cutover. Agent: - Delete AutopilotMonitor.Agent, .Agent.Core, .Agent.Core.Tests (projects + sln entries). V2 trio, SummaryDialog, DecisionCore.Tests and V2Isolation.targets stay. Backend: - Relocate the shared statics/DTOs V2 reused from the V1 partials (StampServerFields, SanitizeEventTimestamps, IsMaxLifetimeAgentShutdown, RecalculateAppDurations, EventClassification, AppInstallAggregationState) into EventIngestProcessor; IngestEventsOutput moves next to IngestTelemetryFunction (same namespace, zero ripple). - Delete IngestEventsFunction (route agent/ingest), BootstrapIngestEvents (route bootstrap/ingest), NdjsonParser and their DI registration. EventDataNormalizer stays (used by TelemetryPayloadParser). - Drop both routes from EndpointAccessPolicyCatalog and RequestTelemetryMiddleware.TenantHeaderTrustedPaths. - Retire the V1 update-hash oracle: LatestAgentV1*/LatestBootstrapV1* fields + GetAgentLine case 1 (default arm returns empty hashes, so stragglers gracefully skip the integrity check; ParseAgentMajor keeps its default of 1 on purpose — defaulting to 2 would feed V2 hashes to V1 binaries). TableConfigRepository Store+Map updated in lockstep. - Remove ApiEndpoints.IngestEvents/.BootstrapIngestEvents constants. Tests: NdjsonParserTests deleted; timestamp/critical-path/max-lifetime tests repointed to EventIngestProcessor; the two NDJSON pipeline tests removed (coverage retained by direct-list siblings); catalog/no-store InlineData rows updated. Docs: architecture.md (endpoint tables, ingest flow diagram, solution layout, test matrix), web SectionAgentInternals endpoint tables, build.ps1 help text. Azure: no config change needed — /api/agent is not in the client-cert exclusion paths (agent/ingest was always cert-enforced) and the /api/bootstrap prefix exclusion remains for the surviving bootstrap routes (verified via ARM on autopilotmonitor-api). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
05a591e1d4 |
feat(backend): delegated-admin (MSP mode) data model + scope service [Phase 0]
Foundation for "MSP mode" — a scoped-global authorization tier between a single-tenant member and a platform GlobalAdmin. A delegated admin may access a SUBSET of tenants (its assignment rows), resolved UPN-based and tid-agnostic, exactly like GlobalAdmins. Phase 0 is purely additive: the service is registered but no auth path consults it yet (the cross-tenant gate change lands in Phase 1), so behavior is unchanged. - Constants: DelegatedRoles/DelegatedStatus/DelegatedSource; new DelegatedAdmins table added to TableNames.All (auto-create), CriticalBackupTables.All and AuthTablesFullRestoreForbidden (identity table — full-table restore blocked). - IAdminRepository + TableAdminRepository: DelegatedAdminEntry DTO + CRUD (PK=UPN, RK=TenantId). Hot-path GetDelegatedTenantsAsync (partition scan) and reverse GetDelegatedAssigneesAsync. Queries use the typed predicate overload (no string-interpolated OData filter — avoids filter injection). - DelegatedAdminService: resolves DelegatedScope (Covers/RoleFor/CanWrite) with a 5-min cache + write-through invalidation. Fail-closed: only Active+enabled rows with a recognized role confer scope; empty role -> DelegatedReader; unknown -> dropped. - 16 unit tests (fake repo); full Functions suite 2434/2434 green. Backup catalog-count assertion bumped 15 -> 16. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
7c1f14c66b |
feat(backend+web): GA "Active Users" live presence dashboard + UserActivity retention
Global-Admin cross-tenant view of web users active "now" (made an authenticated request within a window, default 5min), backed by a new self-maintaining UserPresence table. Not App Insights (sampling) / not SignalR. Backend: - UserPresence table (one row/user, PK=tenantId, RK=SHA-256(lowercase UPN)); upserted fire-and-forget by UserPresenceMiddleware (after PolicyEnforcement, 60s per-process throttle, UPN-gated so agent cert calls are skipped). - GET /api/global/presence (GlobalReadOrAdmin) via GetActiveUsersFunction; read path propagates storage errors (5xx) instead of masking them as "0 active". - 1-day presence retention + 90-day UserActivity retention in maintenance; PlatformStats.TotalUsers clamped monotonic so the public "Users Seen" figure cannot regress after a prune. Web: - /admin/presence page (live counter + table), nav entry under GA > Metrics. - "Total Users" relabeled "Unique Users (90d)" to match the windowed metric. Backend+web only, no ConfigVersion bump, inert until redeploy. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
12d67380e9 |
feat(mcp+backend): session_debug — client-side diag-ZIP analysis via HMAC ticket
Lets tenant-admins and GAs run end-to-end session debugging over MCP: backend
telemetry plus the agent diagnostics ZIP, downloaded and analyzed locally by the
AI client. The backend never unzips/parses the archive (no parse CPU); it only
mints a short-lived signed download ticket and proxy-streams bytes.
Backend (no ConfigVersion bump, no agent change):
- DiagnosticsDownloadTicket: HMAC codec (reuses PaginationTokenSigningKey,
purpose-tagged diag-dl-v1, 10-min TTL) so a no-JWT client can download.
- DiagnosticsBlobStreamer: shared proxy-stream engine; DiagnosticsDownloadFunction
(web/JWT) refactored onto it. ClassifyBlobName/BlobDestination stay put (xUnit).
- POST diagnostics/download-ticket (MemberRead, QueryParam, GA-bypass) +
GET diagnostics/download?t= (anonymous; tenantId/blobName read only from the
signed ticket -> no retargeting). Both added to NoStoreCacheMiddleware.
MCP:
- get_session_diagnostics tool (all roles) -> absolute downloadUrl + static zipMap
+ instructions; unwraps the {success,session} envelope.
- diag_zip_layout catalog (get_resource + protocol resource).
- debug-session prompt (tenant + GA variants).
- Tool descriptions flag the client requirement (needs local file/shell tools;
pure chat clients cannot unzip).
Tests: backend 2380/2380 (ticket codec, NoStore, classify, catalog completeness);
MCP 261 passed (incl. envelope-unwrap regression). Inert until redeploy.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
bc1f11d2db |
revert(backend): undo L4 telemetry sampling — ineffective in isolated worker
Deploy-verified on |
||
|
Oliver Kieselbach
|
f4d51ea499 |
perf(backend): cut App Insights cost + ingest hot-path reads (review follow-through)
Telemetry cost (L4): worker-side adaptive sampling now excludes Request;Event so the enriched RequestTelemetryMiddleware copy is never thinned; host.json Host.Results=Error drops the duplicate unenriched host request item. One complete request row per call instead of two (one unsampled+unenriched, one sampled+enriched). Ingest hot path (H2/H3): BlockedDeviceService caches negative answers (30s revalidation, symmetric with positive entries) so healthy devices no longer pay a BlockedDevices point-read per request; SessionDeletionGuard returns its already-loaded row and IncrementSessionEventCountAsync returns the post-merge snapshot, collapsing ~4 Sessions reads/batch to ~2. Queue-worker correctness (O1/O2): new GetSessionEventsStrictAsync propagates storage failures on the analyze + vulnerability-correlate paths so a transient read fault retries/poisons instead of silently deleting the message after "analyzing" an empty stream. Fail-soft variant kept for UI/metrics surfaces. Review fixes: parse prefetched Status as string (the actual wire format, was int -> optimization never fired); session-aware block over a cached negative entry keeps its session scope instead of widening to whole-device. 2274/2274 backend tests green. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
c0552b7f9d | reduced AppInsights dependency telemetry | ||
|
Oliver Kieselbach
|
467ee07810 |
refactor(queue): extract QueueClientFactory + QueuePollingWorker base (audit D2)
Eliminates the copy-paste queue scaffolding flagged as finding D2: identical Managed-Identity/connection-string QueueClient bootstrap hand-rolled across 13 sites, plus a poll-loop/poison/safe-delete skeleton duplicated in 5 workers. - QueueClientFactory: resolves the MI/connstring model once; consumed by all 5 producers, the offboarding enqueuer, the backup producer and the poison probe. - QueuePollingWorkerBase: shared bootstrap + poll loop + poison-move + safe-delete + queue-create (ExecuteAsync/ProcessOneAsync sealed to prevent drift). - QueuePollingWorker<TEnvelope>: deserialize -> validate -> handler flow with an opt-in in-flight heartbeat. - 3 vanilla workers (Index/Analyze/Vuln) collapse to ctor + HandleAsync. - SessionDeletion/Offboarding use BatchSize/UseHeartbeat/ShouldPauseAsync + Before/AfterPoisonMoveAsync hooks (kill-switch, CAS state transition, OpsEvent enrichment, Failed-state-before-poison all preserved bit-for-bit). - Backup derives from the non-generic base and overrides ProcessOneAsync (its lease + CAS state machine is intentionally bespoke). Producers get the factory only -- no shared base, their business logic is genuinely divergent. Test seams unchanged; the static SessionDeletionWorker. MaxDequeueCount reference moves to QueuePollingWorkerBase.DefaultMaxDequeueCount. 2168/2168 backend tests green, 0 warnings. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
9e689dc9d8 |
feat(backend/backup): PR2 — single-row restore + admin UI
POST /api/global/backups/{backupId}/restore-row (GA-only) with preview|commit
modes. Preview scans the ETag-pinned NDJSON blob for the requested (pk, rk),
diffs against the live row, and returns rowSha256 + currentETag for echo.
Commit re-validates everything under a short maintenance lease and writes via
UpdateEntity(IfMatch=etag, Replace) or AddEntity; 412/404/409 → CurrentRowChanged.
Backend:
- RestoreTablePreflightValidator (no I/O) + BackupRestoreInputValidator (manifest
parse, entry check, BlobName canonical, ETag-pin + SHA in one stream read)
- CriticalTableRestoreService with PreviewRowAsync (no lease) + CommitRowAsync
(MaintenanceLeaseHolder, 45 s renewal, deterministic finally release)
- BlobBackupStore.OpenNdjsonReadAsync — GetProperties + OpenRead(IfMatch=etag)
- BackupTerminalException carries a stable Code for HTTP envelope mapping
- OpsEventService.RecordBackupRowRestoredAsync (Warning; payload is
container + blobName only, never a SAS URL)
- 27 new xUnit tests; full backend suite 1979/1979 green
Web:
- /admin/backups list page with newest-first list + "Run backup now" trigger
and async job-status poller
- /admin/backups/{backupId} detail page with per-table breakdown + single-row
restore trigger
- RestoreRowDiffModal with Added/Removed/Changed/Unchanged pills and
auth-table security banner (GlobalAdmins/TenantAdmins/McpUsers)
- api.backups namespace + matching DTO types
- globalNavConfig: ga-backups under Ops
- BackupRowRestored registered in OPS_EVENT_TYPES (dual-register policy)
- 6 new vitest cases; full web suite 390/390 green
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
d4034842ee |
feat(backend/backup): PR1 — critical-table backup feature (daily timer + manual GA trigger)
Adds the full backup pipeline that snapshots 15 critical tables (Admin/Tenant config, rules, identities, audit, preview) into a logically-isolated blob container, daily at 04:00 UTC and on operator demand. Restore endpoints follow in PR2/PR3. Architecture: * Daily TimerTrigger + manual POST /api/global/backups/trigger (GA-only, fail-hard producer). HTTP returns 202 + jobId on enqueue, 500 if the queue write fails (and the BackupJobs row is rolled to Failed). * BackgroundService queue worker (not QueueTrigger — matches the Analyze/ IndexReconcile pattern that avoids the Functions-host connection-binding app-setting). Worker holds the maintenance lease for the whole run; service is intentionally lease-unaware so it stays unit-testable. * BlobBackupStore wraps the new 'critical-table-backups' container with a shared maintenance-lease sentinel at '_lock/maintenance.lock' that serialises every backup or restore op across worker/timer/single-row paths. * CriticalTableBackupService streams each table through CryptoStream(SHA256) into per-table NDJSON blobs, then writes manifest.json LAST as the durability anchor. Per-table failures degrade Outcome to Partial without aborting; per-run budget (50 min) → Skipped status. * BackupJobsRepository persists the six-state machine (Queued/Running/Completed/Failed/Skipped/BlockedTerminal) with ETag-CAS updates. Enum→string mapping rejects unknown values on read so refactors surface loudly instead of silently shifting state. * BackupJobWatchdog (30 min timer) heals stuck rows via a probe lease on the shared sentinel: Queued > 60 min → Failed, Running with stale heartbeat AND free lease → Failed (lease-busy means a live worker — leave alone). * OpsEvents: distinct CriticalTableBackupPartial (Warning) vs Failed (Error) so operators can wire softer Telegram rules for partial captures. * MaintenanceService.MonitoredPoisonQueues + OPS_EVENT_TYPES catalog registered (dual-register memory). Test coverage (15 new tests, 1952/1952 backend xUnit green): * BackupManifestJsonTests: enum-as-string (allowIntegerValues=false rejects numeric values), DictionaryKeyPolicy unset (Azure column names preserved case-faithfully), manifest field round-trip. * BackupJobsRepositoryMappingTests: full entity↔domain round-trip, unknown Kind/State/BackupOutcome rejected, legacy BackupOutcome="Failed" rejected. * CriticalTableBackupServiceTests: happy/partial/empty/budget-exceeded paths exercise the per-table loop with in-memory table + blob fakes. Setup: scripts/Backup/setup-backup-container.ps1 — idempotent az-CLI bootstrap for container + lifecycle rule (90 d delete on prefix). Soft-Delete and Versioning are NOT applied automatically (storage-account-wide blast radius — operator must validate per separate doc note). Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
b2e17d39f6 | security hardnings - nosniff, x-forwaredded-for, ensure-tenantid | ||
|
Oliver Kieselbach
|
7a05268250 | security hardenings | ||
|
Oliver Kieselbach
|
feff123f11 |
feat(backend,agent,web): hosted diagnostics destination + cascade-delete cleanup
Adds an opt-in second destination for agent diagnostics uploads:
- CustomerSas (default, unchanged): tenant's own SAS-backed container
- Hosted (new, opt-in via explicit admin click): backend's storage account
under {tenantId}/AgentDiagnostics-...zip, with per-upload 15-min
blob-scoped Write+Create SAS minted via the Function App's MI
Closes the long-standing gap where Session.DiagnosticsBlobName was captured
into the deletion-manifest but the blob was never actually cleaned up by
the cascade. Hosted blobs are always deleted (we own the storage); customer
blobs are deleted only when the SAS includes the 'd' permission, otherwise
logged + skipped.
Existing tenants change nothing: the new TenantConfiguration.DiagnosticsUploadDestination
field defaults to CustomerSas, no silent flip. Hosted requires an explicit
admin click + Save with a clearly marked "data leaves your tenant" disclosure.
Tests: Backend 1844, V2 1076, V1 538, vitest 380 (+78 new across all suites).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
c26b25b81b |
feat(backend,web): offboarding farewell-email capture + dedicated feedback table + reports tabs
- Persist tenant Preview-Notification-Email on OffboardingHistory row at Phase 1 so it
survives Phase 2 wipe; ResendEmailService.SendAsync wired but disarmed by a const-false
arm-flag with a [DRAFT] placeholder template — explicit flip required to ship.
- Move user feedback storage out of PreviewConfig discriminator into a dedicated Feedback
table (PK=InApp / Offboarding) that is NOT in any tenant-offboarding wipe list — feedback
from departing tenants is exactly the signal we want to keep. New IFeedbackRepository +
TableFeedbackRepository; FeedbackFunction migrated; (PreviewConfig, "Feedback")
removed from TenantOffboardingHandler.DiscriminatorTables.
- POST /api/tenants/{tenantId}/offboard/feedback collects a free-form farewell comment
during the drain-barrier countdown (marker-gated, fail-soft OpsEvent emit). Reports >
User Feedback page gains an Offboarding-Feedback sub-tab next to In-App Feedback.
- OffboardingFeedbackReceived OpsEventType registered (info-severity) + catalog entry
in OpsAlertRulesSection so admins can wire Telegram in two clicks.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
e4c4dd57db |
feat(backend,web): optional Graph add-on permissions for Intune script display names
Adds opt-in resolution of Intune Platform Script + Remediation display names in
session timelines via a tenant-side appRoleAssignment grant. No app manifest
change, no impact on customers who don't enable it. Architecture is reusable for
future optional Graph permissions: catalog -> detector -> resolver -> endpoint.
Backend
- GraphFeatureDetector parses the `roles` claim from the SP access token and
caches the (token, roles) bundle per tenant. GetSnapshotAsync surfaces a
transient flag so admin-UI callers can render "unknown" instead of a
misleading "0 of 0 granted" when token acquire times out.
- GraphTokenService now accepts a CancellationToken so optional-feature callers
can apply tight budgets instead of inheriting the 5+15+30 s retry chain.
- ScriptDisplayNameResolver: cache-first, list-pull-on-cold-tenant, per-ID
fallback. Throttle-aware abort: skip per-ID for kinds whose full-pull hit a
transient failure; break the per-ID loop on the first HttpRequestException.
Total budget 5 s, with a tight 1.5 s sub-budget on the token acquire.
- Azure Table `ScriptNameCache` + direct-TableClient fail-loud repository
(mirrors TableOffboardingAuditRepository pattern). Daily
ScriptNameCacheCleanupFunction: 7 d positive / 1 h NotFound TTL.
- POST /api/tenants/{id}/scripts/display-names (body-carried refs to dodge URL
length limits; backend cap 200 refs/request).
- GET /api/tenants/{id}/graph-permissions/status (4 s budget; includes app
ClientId so the UI can render an exact grant command).
- POST /api/tenants/{id}/graph-permissions/refresh (invalidates token cache).
- All three routes registered in EndpointAccessPolicyCatalog (MemberRead for the
resolve, TenantAdminOrGA for the admin-UI surfaces).
Web
- `useScriptDisplayNames` hook with stable refset fingerprint (no spurious
refetches on parent re-renders); clears stale map on tenant/refset change so
a session switch never leaks names across tenants.
- Chunked POST loop: 200-ref batches with partial-success semantics. Preserves
earlier chunks when later ones fail (non-2xx OR transport throw); only
TokenExpiredError propagates so auth flow can react.
- ScriptExecutions renders " . {DisplayName}" alongside existing label/short-id,
keyed by Kind + policyId so Platform/Remediation under the same id never
collide.
- New Settings section "Optional Graph capabilities": status table with granted
/ not granted / unknown, copy-paste PS one-liner with ClientId baked in,
refresh button gated on response.ok.
Customer-facing
- scripts/CustomerSetup/Grant-AutopilotMonitorAddOn.ps1: idempotent
grant / verify / revoke script. -Features ScriptDisplayNames (high level) or
-Permissions ... (admin-UI generated). Pure ASCII for PS 5.1 compat.
- docs/customer-graph-add-on-permissions.md: end-to-end customer guide
covering prerequisites, the appRoleAssignment pattern, and revocation.
Tests
- 1718 backend xUnit / 364 web vitest. Highlights: JWT roles parsing edge
cases; full-pull-429 suppresses per-ID; per-ID-429 breaks loop; transient
detection via GetSnapshotAsync; chunked partial success (HTTP failure + thrown
transport); TokenExpiredError propagation; EndpointPolicyCatalogCompleteness
covers the three new routes by reflection.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
206388da99 |
feat(backend,web): tenant offboarding async cascade with customs archive
Replace synchronous DeleteAllTenantDataAsync with a queued async cascade
that wipes all 24 tenant-scoped tables safely, archives custom rules,
and survives crashes at every step.
Phase 1 (TenantOffboardFunction, ~200ms sync):
- Write History / ByTenantPointer / Marker rows (OffboardingAudit table)
- Flip TenantConfiguration.Disabled=true (the active auth gate)
- Enqueue with 6 min visibility-delay (cache-drain barrier =
TenantConfigurationService.CacheDuration 5 min + 1 min safety)
- Return 202 with EarliestProcessingAt for the UI countdown
Phase 2 (TenantOffboardingWorker + Handler, async after drain):
- Enumerate sessions, per-session cascade-enqueue, persist
Expectations blob (separate offboarding-state container)
- Drain probe against per-session progress blobs; self re-enqueue
up to 60x at 2 min visibility-delay
- 2.D-archive: GatherRules / AnalyzeRules / ImeLogPatterns moved
to TenantOffboardingCustomsArchive table (3-iter cap + verify)
- 2.D-wipe: SafeWipe over all tenant tables (4 variants: exact-PK,
composite-PK range, discriminator+TenantId, property-only) with
fetch+verify+delete pattern + SafeWipeVerificationException
- 2.E: blob-prefix wipe deletion-manifests/{tenantId}/
- 2.D-pass-2: re-wipe AuditLogs (defense-in-depth)
- 2.G: side-effects FIRST (Marker, Pointer, blob, audit, OpsEvent)
then History=Completed LAST (DrainCompletedAt as resume anchor)
- 2.F-final: delete TenantConfiguration row LAST so self-service
re-onboarding (next /api/auth/me auto-creates default) works
Defense-in-depth:
- OffboardingMarkerCleanupFunction (2h timer): sweeps Completed
markers after 15 min grace, conditional TenantConfiguration sweep
(only when row is still the offboarding tombstone — preserves
user-driven self-service re-onboards)
- Worker poison path: durable Failed state MUST land before queue
moves; transient failure leaves message visible for retry
- SessionDeletionHandler reorder: Audit + SignalR notify BEFORE
CompletedAt write so the drain predicate is correct
Web UI:
- /settings danger zone: typed "OFFBOARD" confirmation dialog,
post-202 drain-barrier banner with minute-based countdown, hint
"you can sign out and close this tab — deletion runs in
background", silent auto-logout when countdown elapses
- /admin/customs-archive (GA only): list runs + inspect entries +
typed "DELETE" confirmation modal for run/entry deletion
- OpsAlertRulesSection: TenantOffboardingFailed event registered
- lib/api.ts: shared CustomsArchive* response types
Tests: 100+ new tests covering drain contract, handler ordering,
worker poison path, ETag-CAS, SafeWipe variants, OffboardingFilters
prefix-bleed safety, MarkerCleanup conditional sweep, Bootstrap
auth-disabled-tenant gate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
50071c5e42 |
feat(backend,web): dedicated cascade-delete maintenance function (PR6/7)
Plan §5 PR6 / §16 R14: extracts cascade-lifecycle maintenance out of the
2h Maintenance timer into a dedicated 12h SessionDeletionMaintenanceFunction
with independent cadence, kill-switch, and OpsEvent watchdogs. Adds four
work-blocks (Plan §10):
1. Manifest-blob TTL sweep — defence-in-depth on top of Azure Blob
Lifecycle Management. EnumerateOldDeletionManifestsAsync +
DeleteDeletionManifestPairAsync (33-day effective window via the
existing soft-delete tombstone, not permanent deletes).
2. Stale-Preparing GC — reverts DeletionState=Preparing rows older than
1h that have NO progress blob (producer crashed before upload). Skips
when a progress blob exists; producer's own Preparing-resume path
handles those.
3. Stranded-Queued detection — alert-only OpsEvent per row in
DeletionState=Queued for > 30min (no auto-clear; operator inspects).
4. Retention fanout via new SessionRetentionFanoutService — per-tenant
DataRetentionDays sweep, dispatched per EnableCascadeDeleteV2 flag:
V2 → producer.EnqueueAsync, legacy → ISessionRepository.DeleteSessionAsync
(now CAS-guarded since PR5 finding 2). Rate-limited to 100 enqueues
per tenant per run with 50ms throttle.
Operational alerting:
- 30min watchdog → OpsEventType SessionDeletionMaintenanceLongRunning (Warning)
- 60min watchdog → SessionDeletionMaintenanceLongRunningSevere (Error)
- Exception path → SessionDeletionMaintenanceFailed (Error) + re-throw
- Per stranded row → SessionDeletionStrandedQueued (Warning)
All four registered both in OpsEventService.cs and the OpsAlertRulesSection.tsx
catalog so admin can wire Telegram per memory feedback_ops_event_types_dual_register.
Reuses PR5 finding 1's IsSessionDeletionKillSwitchActiveAsync so the
fanout-skip check honors flip-ON across scaled-out instances within seconds.
host.json: functionTimeout=01:00:00 (Flex Consumption max, matches the
60min severe watchdog so operator gets alerted before Azure hard-aborts).
Cron: const "0 0 */12 * * *" — matches the repo's IndexReconcileTimer
convention (no AdminConfiguration field; redeploy to change).
Removed: session retention loop from MaintenanceService.CleanupOldDataAsync
(both RunAllAsync and RunManualAsync paths now call CleanupOldUsageDataAsync
directly). Non-session housekeeping (rule-stats, cert-expiry, SignalR
quota, poison-queue watcher) untouched.
EnableCascadeDeleteV2 + SessionDeletionKillSwitch defaults stay false —
PR6 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation.
Backend xUnit: 1500/1500 pass (1483 + 17 new). Web vitest: 347/347 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
8134adf3e4 |
feat(backend,web): wire admin-delete + worker + UI for V2 cascade (PR5/7)
Plan §5 PR5 (session-deletion-cascade-safe.md): flag-gated dispatch in
DeleteSessionFunction between the legacy direct-delete path and the V2
cascade producer; SessionDeletionWorker registered as a hosted service;
UI shows a "deletion queued" toast + per-row spinner while the worker
drains, then removes the row on the SignalR sessionDeleted event.
Includes the codex review follow-ups:
F1 — IsSessionDeletionKillSwitchActiveAsync bypasses the 5-min admin
config cache; fail-CLOSED on storage error so a flip-ON is honored
across scaled-out Function-host instances within seconds (instead
of minutes). Wired into DeleteSessionFunction, Producer, Worker.
F2 — DeleteSessionAsync now reads DeletionState + ETag together with
IndexRowKey, refuses on locked state, and ETag-conditionals the
delete; closes the T2-T4 race where a concurrent V2 producer
could orphan side-table rows. Caller in the legacy path
disambiguates the bool via a re-read into 404 / 409 / 500.
F3 — pendingDeletions threaded from useDeleteSession through page.tsx
→ SessionTable → SessionCell; action cell renders a disabled
spinner instead of the delete button. Hook subscribes to
sessionDeleted SignalR + polls GET /api/sessions/{id} every 60s
as a fallback for missed events; only 404 clears the pending row.
EnableCascadeDeleteV2 + SessionDeletionKillSwitch both default false —
PR5 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation. PR6 wires the producer into the dedicated
SessionDeletionMaintenanceFunction next.
Backend xUnit: 1483/1483 pass. Web vitest: 347/347 pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
5dbc44c4ec |
feat(backend): cascade-delete worker + restore + codex-fixes (PR4/7)
PR4 ships the cascade-delete executor for the safe-cascade-delete plan: the
SessionDeletionWorker (BackgroundService queue poll-loop) plus the
SessionDeletionHandler (per-step execution, live verification, tombstone).
Worker is NOT registered in Program.cs — that lights up in PR5.
Three logical phases land together as one commit per Oliver's call:
PR4 — Worker + Handler + CascadeVerificationService
- SessionDeletionWorker: BatchSize=1, kill-switch entry guard, heartbeat task
extending visibility every 60s during long handler runs, poison after
max-dequeue
- SessionDeletionHandler: download manifest+progress, CAS Queued→Running
(with Running→Running resume), per-step dispatch by Class, AGGREGATE
decrement, live verification, ordered tombstone (SessionsIndex first,
Sessions last)
- CascadeVerificationService: per-Class live residual scans (CveIndex is
manifest-key-only per §12-Q8; AGGREGATE+FINAL skipped)
- NotifySessionDeletedAsync added to ISignalRNotificationService
- BlobStorageService: DownloadDeletionProgressAsync + ETag-CAS
UpdateDeletionProgressAsync
PR4b — Restore endpoint
- POST /api/admin/sessions/{sessionId}/restore (GA-only via Policy Catalog)
- SessionRestoreService: dispatch table per §13.4, full + partial-poisoned-
recovery modes
- TableStorageService.Restore.cs: RestoreRowsByExactKeysInBatchesAsync +
ConvertFromPropValue + ConvertDumpToEntity (Sessions-row override forces
DeletionState=None on restore so the row isn't permanently locked)
- RestoreSoftwareInventoryEntryByKeyAsync helper
- Worker MoveToPoisonAsync now CAS-transitions Running→Poisoned (E1 fix)
so restore can dispatch into partial-restore mode
PR4c — Codex review fixes (6 findings)
- F1: AGGREGATE per-key idempotency via DeletionProgress.AggregateDecrementsApplied
(persist-first/decrement-second bounds drift to +1 per crash vs unbounded -N)
- F2: TombstoneStarted flag closes the gap where sessions=null+completedAt=null
blocked restore after a tombstone crash
- F3a: Sessions row inserted LAST in full restore (was first → trapped retry
with RejectAlreadyAtOriginalState)
- F3b: RestoreMode.Full now 409-ignores (was throw-on-409); converges with
Partial mode, makes retry-after-partial-failure idempotent
- F4: Partial-restore per-key re-increment idempotency via
DeletionProgress.RestoreReIncrementsApplied
- F5: Poison transition pre-checks PendingDeletionManifestId so stale
envelopes can't flip a fresh active cascade to Poisoned
- F6: DownloadDeletionManifestWithShaAsync exposes snapshot SHA; Handler +
Restore both verify against progress.SnapshotSha256 (snapshot↔progress
binding per plan §3)
DeletionProgress schema gains 3 additive optional fields (AggregateDecrementsApplied,
RestoreReIncrementsApplied, TombstoneStarted) with safe defaults so PR1-PR4
progress blobs deserialize cleanly.
Tests: 1457/1457 backend xUnit facts green (1391 pre-PR4 + 66 new across PR4,
PR4b, PR4c). Full solution build green. Program.cs has zero
SessionDeletionWorker references (PR5 territory stays untouched).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
257c5c4354 |
feat(backend,web): cascade-delete state-machine + guard + producer + preview (PR3/7)
PR3 of the safe cascade-delete plan
(.claude/plans/session-deletion-cascade-safe.md). Substantial PR per
plan §16 — adds the concurrency state-machine, the single-chokepoint
SessionDeletionGuard, the producer that builds manifest + uploads
blobs + enqueues the worker envelope, and the read-only preview
endpoint Global Admins use to verify a cascade plan before flipping
EnableCascadeDeleteV2 on. Wiring the guard at the 9 writer sites is
intentionally deferred to PR5 — no cascade actually fires until PR5
calls the producer from DeleteSessionFunction, so PR3 ships as
infrastructure-only.
== State machine + guard infrastructure ==
SessionDeletionState (Shared, Plan §1 P7 / §16-R2):
None | Preparing | Queued | Running | Poisoned. Completed deliberately
omitted — the cascade FINAL step removes the row, so "completed" is
observable via row-absence + progress.completedAt, not via a stable
state value. IsValidTransition(from, to) enforces the legal edges
(None→Preparing, Preparing→{Queued,None,Poisoned}, Queued→{Running,
Poisoned}, Running→Poisoned, Poisoned→None) and treats null/empty
legacy rows as None.
SessionDeletionLockedException carries (TenantId, SessionId,
CallerContext, CurrentState, ManifestId) for audit + log lines.
SessionDeletionGuard (Plan §1 P7 + §16-R13):
- ThrowIfLocked(TableEntity sessionRow, callerContext) — piggyback
variant for hot paths that already loaded the Sessions row; zero
added I/O. Caller must include DeletionState +
PendingDeletionManifestId in its select projection.
- EnsureWritableAsync(tenantId, sessionId, callerContext) —
standalone variant for paths with no existing read; one Get
round-trip via the PR1 ISessionDeletionInventoryReader seam.
Both paths throw the same exception. Per §16-R13 framing the guard
is "best-effort fast-fail, not safety" — correctness lives in §1 P4
live verification + poison + §13 restore-from-poisoned (PR4).
== Sessions row state-machine columns ==
SessionSummary.DeletionState (string) +
SessionSummary.PendingDeletionManifestId (string?) — added to the DTO
+ MapToSessionSummary read path. Empty/null means no cascade in flight
(legacy rows; treated as None per the guard contract). The producer
writes both columns via direct CAS via the new helper; existing
session-write paths don't need to know about them in PR3.
TableStorageService.CasSetSessionDeletionStateAsync — new partial
helper. Read-then-CAS on Sessions row, returns a structured
SessionDeletionStateCasResult (Updated | SessionMissing | WrongState |
EtagConflict) so the producer can branch correctly. Stamps newManifestId
on transition to Preparing; clears it on transition back to None
(poisoned-recovery restore).
== Producer + preview endpoint ==
SessionDeletionProducer.EnqueueAsync(tenantId, sessionId, reason,
actor) — Plan §5 PR3 step sequence: kill-switch check → CAS None→
Preparing with new ManifestId → DeletionManifestBuilder.BuildAsync →
BlobStorageService.UploadDeletionManifestAsync → UploadInitial-
DeletionProgressAsync (new helper for the mutable .progress.json
sibling) → audit deletion_started → CAS Preparing→Queued → enqueue
SessionDeletionEnvelope to Constants.QueueNames.SessionDeletion.
Returns SessionDeletionEnqueueOutcome enum so PR5/PR6 callers
translate per their UX (admin → 409 on AlreadyInFlight, maintenance
→ skip+audit). Crash recovery contract: row stuck in Preparing with
no progress blob → maintenance GC clears after 1h (PR6); Queued/
Running/Poisoned NEVER auto-cleared.
GET /api/admin/sessions/{sessionId}/delete/preview?mode={summary|full|
download} — GlobalAdminOnly per the EndpointAccessPolicyCatalog (per
memory feedback_route_policy_catalog). Three modes:
- summary: preflightCounts + sample (PK, RK) per step + estimated
snapshot bytes + builderDurationMs + schemaHash.
- full: entire manifest JSON inline (every row dump, every decrement
key) — byte-level review of "what would be deleted".
- download: gzipped *.snapshot.json.gz with Content-Disposition
attachment, identical bytes to what the real cascade would upload.
NEVER mutates state, NEVER uploads a blob, NEVER enqueues. Calls the
guard only as a hint (in-flight cascades surface in the response, no
lock acquired). Audits deletion_preview per Plan §7.
== Configuration changes (with web round-trip) ==
AdminConfiguration.SessionDeletionKillSwitch (bool, default false,
Plan §1 P8 / §9) — global emergency stop. When true, V2 producer +
legacy direct-delete return 503; PR4 worker pauses on entry. Round-
tripped through TableConfigRepository Store + Map AND the 4-file web
chain (AdminConfigContext.tsx, AdminConfigSettingsSection.tsx,
SectionGlobalSettings.tsx, types/adminConfig.ts) per memory
feedback_admin_config_ui_roundtrip — without it, every admin Save
silently clobbers the field back to false. UI surfaces a red checkbox
with explanatory copy.
TenantConfiguration.EnableCascadeDeleteV2 (bool, default false) —
per-tenant rollout flag. When true, PR5 admin-delete + PR6 retention
fanout build a manifest + enqueue; when false, legacy direct-delete
runs. Per-tenant flag selects PATH; global kill-switch halts BOTH.
Constants.QueueNames.SessionDeletion = "session-deletion" — worker
queue name.
Shared envelope SessionDeletionEnvelope { TenantId, SessionId,
ManifestId, Reason, EnqueuedAt } — wire shape between producer and
PR4 worker.
== DI ==
Program.cs registers SessionDeletionGuard + SessionDeletionProducer
as singletons next to the existing DeletionManifestBuilder.
== Tests (50 new, backend suite 1340/1340 green) ==
- SessionDeletionStateTests (10) — IsLocked over every state value
including legacy null/empty, IsValidTransition over every legal
edge, explicit rejection of illegal edges, "Completed is not a
state" guarantee.
- SessionDeletionGuardTests (8) — null row no-op, every lock state
throws with correct exception fields, missing-session no-op,
context propagation in the exception.
- SessionDeletionProducerTests (7) — kill-switch returns
KillSwitchActive without downstream calls; SessionMissing /
AlreadyInFlight / Poisoned outcomes from CAS; happy path verifies
all 7 steps in order via Mock.Verify; CAS-2 failure leaves snapshot
uploaded but no queue message; producer's pre-allocated ManifestId
flows to both the snapshot blob and the CAS marker (consistency
invariant the worker depends on).
- AdminConfigurationSessionDeletionKillSwitchTests (6) — both flags
default false on new config + on CreateDefault, true persists.
Wiring 9 sites with the guard (telemetry V1+V2 ingest, mark-success/
failed, server actions, maintenance marker, re-analyze handler,
vulnerability handler, manual rescan, index reconcile) deferred to
PR5 — no cascade fires in PR3, so the guard is dormant.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
c8658b9feb |
feat(backend): scaffold cascade-deletion manifest format + builder (PR1/7)
Read-only foundation for the safe cascade-delete plan (.claude/plans/session-deletion-cascade-safe.md). PR1 ships the manifest shape, the read-only DeletionManifestBuilder enumerator, and the gzip+SHA-256 blob upload/download plumbing. No production caller invokes any of this yet; no existing endpoint behaviour changes. PR2-PR7 wire the producer, worker, and call sites on top. Manifest captures all 15 cascade tables per the §3 taxonomy (4 PK_BY_SESSION, 1 PROP_TENANT_PK, 3 PK_RK_EXACT, 1 DISCRIMINATOR_PK_RK_SUFFIX, 1 DISCRIMINATOR_PK_RK_EXACT, 5 DISCRIMINATOR_PK_PROP), the optional SoftwareInventory side-row (omitted for pre-side-row sessions per §17.5), and the FINAL tombstone step ordered SessionsIndex-then-Sessions. Row dumps tag every column with its EDM type (String / Int32 / Int64 / Double / Boolean / DateTime / Guid / Binary) so DateTime / Guid / byte[] / long round-trip restore-faithfully through the JSON snapshot. Falls back to a SessionsIndex partition scan when Sessions.IndexRowKey is empty or stale, so a partially-corrupted Sessions row can't strand a SessionsIndex orphan after cascade. BlobStorageService.UploadDeletionManifestAsync compresses with gzip, pins SHA-256 of the uncompressed payload to blob metadata, and uses IfNoneMatch=* to enforce snapshot-write-once. DownloadDeletionManifest re-verifies the hash and refuses on mismatch (corruption signal that the §13 restore semantics rely on). Two protected-internal-virtual seams isolate the Azure SDK calls so tests bypass BlobsModelFactory entirely. New SessionInventoryContributions table constant added to TableNames.All so the builder can Try-Read it (404 = pre-side-row session = inventory steps omitted). PR2 adds the side-row write path via VulnerabilityCorrelationService. Tests: 25 new xUnit cases (DeletionManifestSerializationTests, DeletionManifestBuilderTests, BlobStorageServiceDeletionManifestTests) cover counts, dumps with EdmType per Azure Tables type, gzip round-trip, hash-mismatch + missing-metadata corruption signals, IfNoneMatch overwrite refusal, SessionsIndex fallback (both empty-key and stale-key paths), determinism across runs, and pre-side-row inventory skip. Backend suite: 1251/1251 green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
fce4a03d23 |
feat(sla): persistent SlaTenantStatus row replaces in-memory throttle, fixes 2h breach-notification spam
Background
----------
The in-memory SlaNotificationThrottleService kept its 4h cooldown in a process-
local ConcurrentDictionary. On Flex Consumption the Function host recycles
between 2h timer fires, wiping the dictionary, so the throttle was effectively
a no-op — observed in production as tenant b54dc1af receiving an SLA breach
notification every 2 hours for days on end (1 failed of 11-13 sessions, target
95%, success rate 91-92%).
Design
------
One row per tenant in a new SlaTenantStatus table (PartitionKey = tenantId,
RowKey = "status"). Flat schema with {Type}_* property prefixes for the four
breach kinds (SuccessRate, Duration, AppInstall, ConsecutiveFailures); one row,
one Upsert per evaluation cycle. Rationale documented inline: per-type rows
would be over-normalised for the access pattern (always all-types-per-tenant),
and ~50 tenants x 4 types = max 200 rows ever.
Throttle correctness
--------------------
The throttle decision (now - LastNotifiedAt >= cooldown) and the LastNotifiedAt
write happen in the same TryUpsertAsync(status, ifMatchETag) call. Two parallel
inline calls cannot both succeed in sending the same notification — the loser
gets a 412 from Table Storage, refetches, sees the updated watermark, and
either skips or schedules a new claim attempt. Up to 4 retries; on exhausted
retries we drop the notification rather than risk double-fire (at-most-once).
Cooldown is configurable: AdminConfiguration.SlaNotificationCooldownHours
(default 24, range 1-168). UI round-trip wired through the four required
files (AdminConfigContext, AdminConfigSettingsSection, types/adminConfig.ts,
SectionGlobalSettings).
Resolve path
------------
When a previously-active breach is no longer detected, the row transitions to
IsActive=false + ResolvedAt=now and a single sla_resolved notification fires.
Orphan sweep: tenants whose toggles get turned off while a breach is active
fall out of the qualifying loop, so the timer now also scans every status row
and silently clears any IsActive that no longer has a toggle backing it
(no resolved notification — the tenant disabled SLA, they don't care).
Per-type notes
--------------
- SuccessRate breach notifies with the configured warn-threshold, not the SLA
target — a 94% rate against target=99% / threshold=95% reads "below 95%",
not "below 99%".
- AppInstall now actually evaluates (the toggle was wired but evaluation was
missing). Aligned to the same ISO-week window the dashboard uses
(SlaMetricsService.GetIsoWeekKey), so notifications cannot diverge from the
/sla page. Sample-size floor of 5 apps per week, same as the dashboard.
- AppInstall has its own notification template (own title/summary/facts plus
TopFailingApp fact) — the builder used to silently fall through to the
Duration template for everything not SuccessRate.
- Telemetry Period tag derived from breachType: CurrentMonth for SuccessRate
and Duration, CurrentWeek for AppInstall — so AppInsights / ops dashboards
don't group AppInstall under the wrong period.
- AppInstall ops events now carry the actual installer totals (10 evaluated,
4 failed) instead of 0/0.
- ConsecutiveFailures: inline (ingest) path detects breaches, timer path
detects resolve. Both paths write to the same row through the same CAS
retry loop.
Clock injection
---------------
SlaBreachEvaluationService takes an optional Func<DateTime> nowProvider
(default () => DateTime.UtcNow) so the week-window regression test runs
deterministically on every test day instead of silently skipping when today
happens to fall in the first ISO week of the month.
Removed
-------
- SlaNotificationThrottleService and its tests. The in-memory model is gone;
there is no scenario where it would be wanted alongside the persistent row.
Tests
-----
- 21 new lifecycle + mapping + telemetry tests (CAS conflict + retry, CAS
exhausted-retries drops notification, threshold-not-target in notification,
orphan clearing, AppInstall template / week-window / totals, ConsecutiveFailures
inline persistence, multi-type race-probe, period-mapping theory).
- Backend xUnit: 1218 passed.
- Web vitest: 266 passed.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
2284100915 |
feat(backend/health): surface async-worker poison queue backlog in /health/detailed
Three async workers (analyze-on-enrollment-end, vulnerability-correlate, telemetry-index-reconcile) park 5x-failed messages on <name>-poison. Until now there was no operator-visible signal when this happened: backlogs could grow silently and sessions stayed un-analyzed / vulnerability reports were lost. Adds a "Poison Queues" entry to the detailed health-check next to "SignalR Quota" with the same shape: parallel probe via IPoisonQueueProbe, tri-tier classifier (0=healthy, >=1 warning, >=10 critical, thresholds App-Setting configurable via PoisonQueueWarningThreshold / PoisonQueueCriticalThreshold), GA-only visibility. Per-queue 404 counts as zero because poison queues are created lazily on first poison-move. Per-queue exceptions surface in details as "error: <msg>" and force at least warning so a flaky Storage call cannot mask a real backlog. Web UI needs no change - the generic checks[] render loop on /health-check picks up the new card automatically. Backend: 1193/1193 (1178 + 15 new). Build 0 warnings, 0 errors. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
0b80f09a74 |
refactor(notifications): bell on pure SignalR push, kill 60s polling
The two notification-bell React contexts polled GET /api/notifications and
GET /api/global/notifications every 60s regardless of activity, generating
3017+618 = 3635 calls/24h with near-zero hit rate (notifications are emitted
sparsely). SignalR was already used app-wide for live session updates; the
bell was the lone hold-out on a pull model.
Switch to the same push model. Steady-state traffic drops ~86% and the bell
updates sub-second instead of within 60s.
Architecture (Option C+):
- Mount -> 1x HTTP fetch (cold-start hydration via existing GET endpoint)
- SignalR onreconnected -> re-fetch (covers the disconnect window)
- SignalR push -> live append/dismiss
- No periodic polling, no setInterval
Audience-tier security: TenantNotificationAudienceCatalog still drives
visibility (Member vs Admin), but enforcement moves into SignalR group
membership: tenant-{id}-notify-member is joined by every tenant member,
tenant-{id}-notify-admin only by Tenant Admins / Global Admins. The
AddToGroup function fail-closes with 403 on Member-tier joins to the
admin group, so Admin-tier notifications cannot leak via push.
Backend
- ISignalRNotificationService extracted; SignalRNotificationService
implements it. Resolves the long-standing test-mocking blocker
(the real service's constructor validates AzureSignalRConnectionString
and BuildServiceManager(); previously you could not unit-test
consumers). Tenant + Global NotificationService consume the
interface; the three legacy callers (Analyze, Ingest, VulnCorrelate)
still bind to the concrete class -- migrate per test need.
- Six new push methods: SendTenantNotification{,Dismissed,DismissedAll}
and SendGlobalNotification{,Dismissed,DismissedAll}. Tenant variant
routes to the audience-correct group.
- Tenant + Global service push after a successful repo write
(fire-and-forget; never throws into the caller).
- SignalRGroupHelper: parsing handles the two new -notify-{member,admin}
suffixes; new IsTenantNotifyAdminGroup predicate; group-name builders.
- SignalRAddToGroupFunction: explicit 403 when a non-admin attempts
to join the admin-tier group.
Frontend
- TenantNotificationContext: setInterval gone. useSignalR() to
join member-group always, admin-group iff isTenantAdmin || isGlobalAdmin.
Listens for tenantNotification (append + dedupe), tenantNotificationDismissed
(filter), tenantNotificationsDismissedAll (clear). onreconnected
triggers re-fetch.
- GlobalNotificationContext: same pattern against global-admins group.
- app/layout.tsx: SignalRProvider hoisted above the notification
providers so useSignalR() resolves inside them.
Tests
- SignalRGroupHelperTests (8): parsing for all four group formats
incl. new -notify suffixes, IsTenantNotifyAdminGroup predicate.
- TenantNotificationServiceSignalRPushTests (8): audience-correct push
on Create (Admin / Member / Unknown -> Admin fail-closed), no-push
on repo failure, no-push on DismissAll(count=0), no-push on empty
tenantId.
- FakeSignalRNotificationService: in-memory spy implementing the
new interface; replaces the unmockable real service in tests.
- AuthFunctionSideEffectTests + TenantNotificationServiceTests:
constructor signatures updated for the new SignalR dep.
Suites
Backend xUnit: 1161/1161 (was 1146, +15 net).
Web vitest: 259/259 (unchanged; existing notification-API +
gate tests still cover).
Solution build: 0 warnings, 0 errors.
Expected post-deploy reduction (24h):
GET GetTenantNotifications: 3017 -> ~50
GET GetGlobalNotifications: 618 -> ~10
POST AddToGroup: ~246 -> ~350-450 (two joins per tab)
Net bell load: -86%, sub-second live UX.
Disconnect tradeoff (accepted): if SignalR drops without reconnect
(rare -- withAutomaticReconnect retries 1/2/4/8/16/30s ... forever),
the bell shows the snapshot from the last fetch but no live updates
until next reload. The initial-3-retry give-up path of the SignalR
client still gets the user a hydrated bell because the HTTP fetch
runs independently on mount.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
bc3cefa790 |
feat(backend/observability): SignalR free-quota watcher with tiered OpsEvents
Hourly TimerFunction queries Azure Monitor for SignalR ConnectionCount (max over last hour) and MessageCount (sum since UTC midnight) and emits Warning (>=80%) / Error (>=95%) OpsEvents when the free-tier limits (20 connections, 20k messages/day) are approached. Mirror of the cert-expiry pattern - global TenantId=null events, dedup'd to one emission per EventType per UTC day. Fail-soft: skips silently when SignalRResourceId app setting is unset, logs warning on Azure Monitor auth/query errors. Function App's managed identity needs 'Monitoring Reader' on the SignalR resource. Also called from the manual maintenance trigger so 'Run Maintenance Now' exercises every health check. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
0fc81f99ed |
perf(web,backend): cut dashboard load time + fix duplicate fetches
Browser RUM on portal/dashboard surfaced ~1.7s time-to-data driven by sequential backend reads, two stray duplicate fetches, and a 2s SignalR defer. Five focused changes: - AuthContext: store the auth/me prefetch as a Promise instead of its resolved value so fetchUserInfo awaits the in-flight request rather than racing a second fetch when the prefetch hasn't settled yet. - useTenantSecurityConfig: depend on user.isTenantAdmin/isGlobalAdmin/ role primitives instead of the whole user object so the effect stops refiring when AuthContext swaps prefetched user for fresh user (the duplicate /config/<t>/feature-flags call observed on /dashboard). - SignalRContext: drop the hardcoded 2000ms setTimeout before startConnection. The original "avoid competing with critical API calls" rationale doesn't apply across origins (web is HTTP/2, API is a different host); the delay only pushed live updates ~2s into the load for no benefit. - FeedbackFunction: keep the kill-switch check sequential, then run the remaining four Table reads (role, tenant config, sessions existence, feedback cooldown) in parallel via Task.WhenAll. Endpoint dropped from ~1285ms (6 sequential reads on Flex Consumption) toward the slowest single read. - TimingAllowOriginMiddleware: emit Timing-Allow-Origin: * on every function response so the portal's Resource Timing API can read API payload sizes (transferSize / nextHopProtocol) for our own RUM — cross-origin entries currently report 0/unknown. Backend tests 1058/1058, Web tsc clean, vitest 249/249. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
dbef6921ae |
feat(vuln): queue-driven correlation + WhiteGlove Part-1 inventory trigger
Replaces the fire-and-forget Task.Run inside EventIngestProcessor — which
Azure Functions scale-in could kill mid-flight, leaving sessions with no
vulnerability_report and forcing users to click "Analyze Now" — with a
dedicated queue + BackgroundService worker (vulnerability-correlate),
mirroring the analyze-on-enrollment-end pattern. Adds a V2-agent trigger
that emits a Part-1 software-inventory snapshot the moment Windows Event
62407 fires (WhiteGlove sealing success), so Part-1 vuln correlation
happens BEFORE the admin clicks Reseal — closing the gap that existed
because reboot_kill termination at sealing left no shutdown inventory
for the backend to correlate.
Backend
* IVulnerabilityCorrelateProducer + AzureQueueVulnerabilityCorrelate
Producer (fail-soft on agent's hot path).
* VulnerabilityCorrelateQueueWorker (BackgroundService, 5-attempt
visibility-timeout retry, poison-queue on exhaustion).
* VulnerabilityCorrelateHandler ports the full pipeline (correlate +
store + synth event + re-analyze enqueue + SignalR notify).
* IVulnerabilityInventoryLoader extracted from
GetVulnerabilityReportFunction so worker + manual rescan share one
loader; phase-aware filter via whiteGlovePart parameter +
includeAllPhases escape-hatch for the rescan UI.
* EventIngestProcessor cutover: drops three concrete-class deps
(_vulnerabilityCorrelation, _vulnRepo, _signalRNotification) and
two now-orphaned private methods. 561 → 388 LOC.
Agent V2
* WhiteGloveInventoryTrigger subscribes to EspAndHelloHost.WhiteGlove
Completed and calls AgentAnalyzerManager.RunWhiteGlovePart1Inventory
Snapshot via injected Action (no concrete-class coupling, trivially
mockable). Single-fire guarded by Interlocked.Exchange.
* IWhiteGloveCompletedSource interface keeps EspAndHelloHost internal
sealed while letting the trigger be unit-tested with a 4-line stub.
* AgentAnalyzerManager.RunWhiteGlovePart1InventorySnapshot runs ONLY
SoftwareInventoryAnalyzer (LocalAdmin/IntegrityBypass deliberately
skipped at WG hand-off — still SYSTEM context).
* EnrollmentTerminationHandler now also tags Part 2 (WhiteGloveCompleted
Part2 → wgPart=2); was previously untagged so the new phase filter
couldn't see it.
* SoftwareInventoryAnalyzer.AnalyzeAtShutdown gets per-WG-phase atomic
idempotency via Interlocked.CompareExchange, with reset-on-failure
so the EnrollmentTerminationHandler stays a working fallback if the
Trigger crashes.
Cross-phase contamination & out-of-order resilience
* Inventory loader filters by whiteGlovePart so a Part-2 correlation
cannot see Part-1 inventory items (and vice versa).
* Handler checks the existing report on BOTH Part-1 and Part-2 paths:
Part-2 merges with existing Part-1, Part-1 SKIPS its store when the
existing report already contains user_enrollment / both findings or
has scan_summary.phase_contributions.user_enrollment > 0 — protects
against late-arriving Part-1 retries overwriting the richer Part-2
or merged report after at-least-once redelivery.
* VulnerabilityCorrelationService.MergeReports gains NormalizeForMerge
that recursively unwraps Newtonsoft JObject/JArray/JValue to native
Dictionary/List/primitive shapes BEFORE the cast-based extract
helpers run. Without this the normal-order Part-2 merge silently
dropped Part-1 findings, because GetVulnerabilityReportAsync round-
trips through JsonConvert and the existing helpers' OfType / "as
Dictionary" casts produced empty results on JTokens.
Tests (+34 new across Backend & Agent V2.Core)
* VulnerabilityCorrelateEnvelopeSerializationTests — wire-format pin.
* VulnerabilityInventoryLoaderTests — phase filtering, includeAllPhases,
lenient long-parsing.
* VulnerabilityCorrelateHandlerSkipTests — out-of-order skip helper
against native, JArray, JObject, both phase tag, phase_contributions
counter.
* VulnerabilityCorrelationServiceTests — round-trip safety of
MergeReports against JsonConvert-deserialized existing reports.
* WhiteGloveInventoryTriggerTests — fire-once, dedup, dispose-then-late,
idempotent dispose, action-exception isolation, null-arg validation.
* SoftwareInventoryAnalyzerIdempotencyTests — Part-1 dedup, Part-2
dedup, Part-1 ≠ Part-2 independence.
Build green, 0 errors, 0 warnings. Backend 844/844 tests pass, Agent
V2.Core 825/825 tests pass (parallel-flaky tests pass isolated, see
feedback_v2_recoverypath_corrupt_snapshot_flake).
V1 ingest path (/api/agent/ingest) deliberately untouched per
project_v1_vuln_pipeline_gap — V1 is being phased out; fixing it
forward would inflate scope.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
Oliver Kieselbach
|
f3b4d7571f |
feat(notifications): tenant-scoped bell notifications for hardware rejections
Until now, hardware-whitelist rejections were only visible to tenant admins who actively navigated to the settings page or had a webhook configured. Admins missed new rejected models if neither was the case. This adds a parallel tenant-scoped notification pipeline (separate from the existing GlobalAdmin-only feed): on the first-ever HardwareNotAllowed distress per (tenantId, manufacturer-lower, model-lower), a persistent bell notification is created with a deep link to the hardware-whitelist page. Lifetime dedup via a dedicated tracker table makes it a one-shot per model. The webhook path is unchanged and fires in parallel. Routes are gated TenantAdminOrGA + TenantScoping.Jwt for now; an Operator follow-up (new TenantAdminOrOperator policy) is tracked separately. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
e55ad3e22d |
feat(analyze): retry-safe queue for auto-analyze, drop fire-and-forget
The previous design ran RuleEngine in Task.Run after HTTP 200 — Functions scale-in could kill it mid-flight, leaving rule results un-persisted and forcing manual "Analyze Now" clicks. New design enqueues onto analyze-on-enrollment-end and runs the engine in a BackgroundService worker with visibility-timeout retry + poison move after 5 attempts (mirrors telemetry-index-reconcile pattern). Three trigger sites cut over: V1 ingest, V2 EventIngestProcessor primary, V2 post-vulnerability re-run. Manual "Analyze Now" remains as fallback. Plus retry-semantics fix: RuleEngine.AnalyzeSessionAsync no longer swallows storage exceptions in an outer try/catch (per-rule inner catch stays), and the handler now throws when StoreRuleResultAsync returns false. Without this the worker would have deleted messages on transient Table Storage failures and never retried. Tests: +4 failure-propagation, +6 envelope round-trip, 762 backend pass. Follow-ups noted: TableStorageService helpers still swallow exceptions one layer below; SignalRNotificationService needs interface for handler-level unit tests; V1 ingest path missing vulnerability_report synth event. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
Oliver Kieselbach
|
5b9410eb9c |
feat(vuln): persist MSRC + KEV reference caches as blob snapshots
Container App scale-to-zero clears the in-memory MSRC and KEV indexes on every cold start, leaving a window where session correlation runs without those data sources until the next refresh. Re-fetching ~1800 MSRC CVEs plus the KEV catalog from upstream on every cold start is wasteful and slow; persisting the snapshots to blob solves both problems. New BlobBackedReferenceCache<T> generic helper handles single-blob JSON snapshots in the autopilotmonitor / vulnerability-data container: - msrc-index.json — full MSRC CVRF index (~350 KB for last 6 Patch Tuesdays) - kev-catalog.json — full CISA KEV catalog (~500 KB) Both services gained: - EnsureHydratedAsync — idempotent, semaphore-guarded blob load - Blob-write at the end of RefreshAsync (manual trigger + daily timer) - Hydrate guard set after refresh so the no-op path is taken next time VulnerabilityCacheWarmer (IHostedService) kicks off both EnsureHydratedAsync calls fire-and-forget at app startup so re-deploys do not pay a blob-read latency cost in their cold-start window. The correlation hot path also calls EnsureHydratedAsync as a defensive backstop — idempotent and semaphore-protected, so a request arriving mid-warmup awaits the same in-flight load instead of duplicating it. NVD's per-CPE cache stays in Table Storage — random-access lookup by CPE key plus per-key 7d stale-refresh logic is the right shape for that workload and would degrade if forced into a single-blob model. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |