Commit Graph

10 Commits

Author SHA1 Message Date
Oliver Kieselbach bc1f11d2db revert(backend): undo L4 telemetry sampling — ineffective in isolated worker
Deploy-verified on f4d51ea4: the worker-side adaptive-sampling exclusion
(EnableAdaptiveSampling=false + manual Configure<TelemetryConfiguration>
UseAdaptiveSampling(excludedTypes:"Request;Event")) does NOT take effect in the
Functions isolated worker. Live data showed the enriched worker request copy
(Source=WorkerMiddleware) sampled to ~63% while the redundant host request item
(Source="") stayed fully billed — strictly worse (lossy enriched copy, host
duplicate unchanged). host.json "Host.Results": "Error" likewise did not drop
the host request item (emitted by the host process, not governed by that level).

Restores known-good: default worker sampling (enriched copy complete at normal
traffic, burst-capped) + host.json Host.Results back to Information. L1 storage
dependency filter and L2 Function-trace suppression are unaffected and still
verified live. L4 deferred pending a mechanism verified against the isolated-
worker pipeline. H2/H3/O1/O2 from f4d51ea4 verified healthy (0 exceptions).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-09 23:17:17 +02:00
Oliver Kieselbach f4d51ea499 perf(backend): cut App Insights cost + ingest hot-path reads (review follow-through)
Telemetry cost (L4): worker-side adaptive sampling now excludes Request;Event
so the enriched RequestTelemetryMiddleware copy is never thinned; host.json
Host.Results=Error drops the duplicate unenriched host request item. One
complete request row per call instead of two (one unsampled+unenriched, one
sampled+enriched).

Ingest hot path (H2/H3): BlockedDeviceService caches negative answers (30s
revalidation, symmetric with positive entries) so healthy devices no longer
pay a BlockedDevices point-read per request; SessionDeletionGuard returns its
already-loaded row and IncrementSessionEventCountAsync returns the post-merge
snapshot, collapsing ~4 Sessions reads/batch to ~2.

Queue-worker correctness (O1/O2): new GetSessionEventsStrictAsync propagates
storage failures on the analyze + vulnerability-correlate paths so a transient
read fault retries/poisons instead of silently deleting the message after
"analyzing" an empty stream. Fail-soft variant kept for UI/metrics surfaces.

Review fixes: parse prefetched Status as string (the actual wire format, was
int -> optimization never fired); session-aware block over a cached negative
entry keeps its session scope instead of widening to whole-device.

2274/2274 backend tests green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-09 22:43:39 +02:00
Oliver Kieselbach 5e6d3f303d adjusted AppInsights telemetry filter 2026-06-09 21:12:38 +02:00
Oliver Kieselbach 50071c5e42 feat(backend,web): dedicated cascade-delete maintenance function (PR6/7)
Plan §5 PR6 / §16 R14: extracts cascade-lifecycle maintenance out of the
2h Maintenance timer into a dedicated 12h SessionDeletionMaintenanceFunction
with independent cadence, kill-switch, and OpsEvent watchdogs. Adds four
work-blocks (Plan §10):

  1. Manifest-blob TTL sweep — defence-in-depth on top of Azure Blob
     Lifecycle Management. EnumerateOldDeletionManifestsAsync +
     DeleteDeletionManifestPairAsync (33-day effective window via the
     existing soft-delete tombstone, not permanent deletes).
  2. Stale-Preparing GC — reverts DeletionState=Preparing rows older than
     1h that have NO progress blob (producer crashed before upload). Skips
     when a progress blob exists; producer's own Preparing-resume path
     handles those.
  3. Stranded-Queued detection — alert-only OpsEvent per row in
     DeletionState=Queued for > 30min (no auto-clear; operator inspects).
  4. Retention fanout via new SessionRetentionFanoutService — per-tenant
     DataRetentionDays sweep, dispatched per EnableCascadeDeleteV2 flag:
     V2 → producer.EnqueueAsync, legacy → ISessionRepository.DeleteSessionAsync
     (now CAS-guarded since PR5 finding 2). Rate-limited to 100 enqueues
     per tenant per run with 50ms throttle.

Operational alerting:
  - 30min watchdog → OpsEventType SessionDeletionMaintenanceLongRunning (Warning)
  - 60min watchdog → SessionDeletionMaintenanceLongRunningSevere (Error)
  - Exception path → SessionDeletionMaintenanceFailed (Error) + re-throw
  - Per stranded row → SessionDeletionStrandedQueued (Warning)
  All four registered both in OpsEventService.cs and the OpsAlertRulesSection.tsx
  catalog so admin can wire Telegram per memory feedback_ops_event_types_dual_register.

Reuses PR5 finding 1's IsSessionDeletionKillSwitchActiveAsync so the
fanout-skip check honors flip-ON across scaled-out instances within seconds.

host.json: functionTimeout=01:00:00 (Flex Consumption max, matches the
60min severe watchdog so operator gets alerted before Azure hard-aborts).

Cron: const "0 0 */12 * * *" — matches the repo's IndexReconcileTimer
convention (no AdminConfiguration field; redeploy to change).

Removed: session retention loop from MaintenanceService.CleanupOldDataAsync
(both RunAllAsync and RunManualAsync paths now call CleanupOldUsageDataAsync
directly). Non-session housekeeping (rule-stats, cert-expiry, SignalR
quota, poison-queue watcher) untouched.

EnableCascadeDeleteV2 + SessionDeletionKillSwitch defaults stay false —
PR6 stop-condition keeps real tenants on the legacy path until §18.4
internal-tenant validation.

Backend xUnit: 1500/1500 pass (1483 + 17 new). Web vitest: 347/347 pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-13 02:24:03 +02:00
Oliver Kieselbach be63dad534 feat(backend): wire MSRC into correlation + sampling-bypass observability
MsrcApiClient was injected into VulnerabilityCorrelationService for months
but never invoked — every report shipped with source="NVD" only. New
MsrcCorrelationService owns refresh + per-CVE lookup against an in-memory
index built from the last 6 Patch Tuesdays; daily VulnerabilityDataSync
populates it; CorrelateAsync enriches each finding with msrcDocumentId,
msrcReleaseDate, msrcSeverity, msrcBaseScore, msrcVector and flips source
to "NVD+MSRC" when the CVE was published in the indexed window.

Health check also surfaced that all worker-side traces of the daily sync
and CPE auto-resolve were sampled out of App Insights despite running.
Adds Event to host.json excludedTypes and emits TelemetryClient events
for VulnerabilityDataSyncCompleted, VulnerabilityCorrelationCompleted,
CpeAutoResolveHit/Miss, CpeMappingPersisted so customEvents queries can
prove freshness without depending on sampled traces.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-04-28 13:16:22 +02:00
Oliver Kieselbach f20670db66 changed hosts logging 2026-03-29 13:45:22 +02:00
Oliver Kieselbach dd2ee928b0 mcp tool extented and meta, mcp usage data UI, better logging for backend 2026-03-29 12:41:09 +02:00
Oliver Kieselbach 4591d05e65 embedded Intune root + intermediate certs for chain validation 2026-02-11 11:51:30 +01:00
Oliver Kieselbach 2d82980702 config adjustments 2026-02-10 19:30:09 +01:00
Oliver Kieselbach 5690620058 Initial commit 2026-02-05 06:14:41 +01:00