5 Commits

Author	SHA1	Message	Date
Oliver Kieselbach	ee07059bc3	fix(backend): harden /api/agent/telemetry batch validation (Codex Finding 3) ... The ingest endpoint trusted the first item's PartitionKey for the whole batch and silently defaulted RowKey-driver fields to 0 when the agent envelope omitted them. Both paths can misroute data on the storage layer. 1. Cross-session guard: every TelemetryItemDto in a batch must share the same PartitionKey as items[0]. Without this, a mixed-session batch would be stamped with session-A's identity inside PersistItemsAsync and silently relocate session-B's signals/transitions into session-A's primary rows. Mismatch now returns 400 BadRequest with the offending index logged. Extracted as pure helper FindMismatchingPartitionKey so the guard is unit-testable without an HTTP harness. 2. RowKey-driver enforcement in TelemetryPayloadParser: - ParseSignal now rejects envelopes where SessionSignalOrdinal is absent or null (returns null → caller drops the item). A missing value would default to 0 and clobber a legitimate ordinal-0 row on UpsertReplace. Explicit ordinal=0 stays a valid value. - ParseTransition applies the same defense to StepIndex. - Distinguishes "key missing" from "key present with value 0" via JObject.TryGetValue + JTokenType check. Tests (+11 in Functions): - IngestTelemetryFunctionTests: empty batch, single-item batch, uniform batch, mismatch in middle, case-sensitive mismatch. - TelemetryPayloadParserTests: missing/null SessionSignalOrdinal on Signal → null; explicit ordinal 0 accepted; missing/null StepIndex on DecisionTransition → null; explicit StepIndex 0 accepted. Full Functions suite: 703 green (prev 692 + 11 new). Does not address Codex Findings 1 (V2 agent has no Signal/Transition upload path) or 2 (AdminAction=Succeeded mis-mapped to Failed locally) — those land in follow-up commits on this branch. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-21 14:13:07 +02:00
Oliver Kieselbach	c3ee0cad55	feat(backend): M5.d.2 index-reconcile queue producer + feature flag ... Adds the async fan-out producer path from Plan §2.8 / §M5.d so primary Signals + DecisionTransitions rows can be projected onto their index tables without blocking the hot ingest loop. Components: - Constants.QueueNames.TelemetryIndexReconcile (azure-queue-name-safe, all lowercase + hyphens). - IndexReconcileEnvelope (Shared DTO, flat shape with nullable fields per SourceKind="Signal"/"DecisionTransition"). Avoids polymorphic Newtonsoft deserialization — see M5.c.3 finding. - IndexReconcileEnvelopeFactory pure static helper (FromSignal / FromDecisionTransition / BuildBatch) — state-less projection. - IIndexReconcileProducer (Shared interface) contract: no-op when the feature flag is off; never rethrows underlying-queue failures. - AzureQueueIndexReconcileProducer (Functions) — Base64-encoded JSON messages, lazy CreateIfNotExistsAsync guarded by Interlocked flag, Managed-Identity-or-connection-string construction mirroring BlobStorageService / TableStorageService. - IngestTelemetryFunction ctor gains IIndexReconcileProducer; after primary Signals + DecisionTransitions commit it builds envelopes and enqueues them. Event path is untouched (events don't feed the index tables per §2.8). - Program.cs registers the producer as a singleton. Feature gate (AdminConfiguration.EnableIndexDualWrite, default false): - New bool field in AdminConfiguration. - TableConfigRepository Store + Map extension (per feedback_table_storage_serialization). - Producer checks config each call via AdminConfigurationService's 5-min memory cache. Flag off → primary behaviour is bit-exact. Fault tolerance: queue-side exceptions (network, throttling, missing queue before first CreateIfNotExists) are logged per envelope and swallowed. The primary row is the source of truth; the 2h reconcile timer (M5.d.4) is the safety-net recovery path. Tests (+11 in Functions): - 5 IndexReconcileEnvelopeFactoryTests: Signal/DecisionTransition projection, dead-end preservation, batch ordering, null/empty. - 3 IndexReconcileEnvelopeSerializationTests: Newtonsoft round-trip (Signal, full DecisionTransition, minimal-envelope tolerance). - 3 AdminConfigurationEnableIndexDualWriteTests: default-false on new config + CreateDefault, persists when set. Deferred to later sub-milestones: - M5.d.3: queue-triggered consumer that maps envelopes to 0–3 index rows using M5.d.1 TableIndexRepository. - M5.d.4: 2h reconcile timer + release-gate dual-write verification tests, then flip EnableIndexDualWrite=true in prod admin config. Full Functions suite: 672 green (prev 661 + 11 new). Full solution build green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-21 11:48:27 +02:00
Oliver Kieselbach	0936132a00	feat(backend): M5.b.3 payload size-cap guard for /api/agent/telemetry ... Defense-in-depth: reject oversized request bodies before buffering them in memory + deserialising. Closes the documented gap from M5.b — the V2 endpoint previously accepted unbounded bodies while legacy /api/agent/ingest has capped NDJSON payloads since day one. - Reuse existing TenantConfiguration.MaxNdjsonPayloadSizeMB (table-only, admin-set, default 5 MB). The property name is historical; scope is shape-agnostic — both NDJSON and JSON-array paths now share the knob. Doc comment updated to reflect that. No new config surface. - New internal static helper IngestTelemetryFunction.ReadBodyWithSizeCapAsync reads the already-gzip-decompressed request stream with a strict byte cap. Short-circuits + returns (exceeded=true, body="") the instant the cap would be exceeded, so a malicious sender can't force a 100 MB allocation. Semantics match legacy NdjsonParser: strict greater-than, equal-to accepted. - Cap exceeded → HTTP 413 Payload Too Large with IngestEventsResponse body (Success=false, Message describes limit). Agent's UploadResult parser classifies 413 as Permanent (not retried), which is correct — the same batch will fail again. The agent handles this by dropping the batch from the spool after the permanent-failure log. 4 new tests: under cap, exactly at cap, over cap (asserting the stream is not fully drained — memory bound), empty stream. 600/600 backend tests green. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-21 10:20:55 +02:00
Oliver Kieselbach	d218a6ca25	feat(backend): M5.b.2 EventIngestProcessor — V2 endpoint event-pipeline parity ... Extract the post-parse event-processing tail of IngestEventsFunction into a standalone EventIngestProcessor service so /api/agent/telemetry runs the same downstream logic as /api/agent/ingest. Deliberate copy-duplication, not a shared refactor: - Legacy /api/agent/ingest is production-hot — it serves live traffic from every deployed V1 agent and is NOT touched by this commit. Zero regression risk on the existing path. - EventIngestProcessor is a near-verbatim copy of the legacy pipeline's tail (~1000 LoC). Duplication is accepted because legacy /api/agent/ingest is scheduled for decommission post-v11 GA (see tasks/todo.md → Follow-Ups → Legacy-Agent-Ausmusterung). Until then: bugs fixed in legacy must be ported here manually. Copied pipeline steps (all 1:1 with legacy ProcessIngestAsync lines 150-566): StampServerFields → SanitizeEventTimestamps → StoreEventsBatchAsync → Index updates fire-and-forget → IME agent version history → ClassifyEvents → App install summaries → Geo-location update → UpdateSessionStatusAsync (all 7 status branches) → IncrementSessionEventCountAsync → Rule engine analysis fire-and-forget + SignalR + stats → Vulnerability correlation with WhiteGlove Part-2 merge → IncrementPlatformStatAsync TotalEvents + SuccessfulEnrollments → RecordGatherRuleStatsAsync → Diagnostics blob update → GetSession for updatedSession → Session age 4h warning → WhiteGlove IsPreProvisioned error log → SendWebhookNotificationsAsync (all 3 alert paths) → SLA breach evaluation → Admin override detection → ServerAction fetch-and-clear with TrackEvent telemetry → BuildSignalRMessages. Shared (not duplicated, to avoid drift in pure helpers): IngestEventsFunction.StampServerFields, SanitizeEventTimestamps, RecalculateAppDurations (all internal static, state-less, stable). The internal DTOs EventClassification + AppInstallAggregationState are also reused — they carry no behaviour, only fields. IngestTelemetryFunction wiring: - Return type HttpResponseData → IngestEventsOutput so the V2 endpoint can emit SignalR messages for the real-time UI push (matches legacy binding). - PersistItemsAsync partitions the batch: Events go through the processor (full pipeline + AdminAction + pending ServerActions + SignalR); Signal + Transition batches skip the processor and read control signals directly via a lean ReadControlSignalsAsync helper on ISessionRepository — no point firing the full pipeline for a batch that carries no events. - EventIngestProcessor registered in Program.cs DI. Known gap (explicitly NOT in this commit): the V2 endpoint lacks a payload size guard (legacy reads MaxNdjsonPayloadSizeMB from tenant config in ParseNdjsonRequest). Different payload shape (JSON array vs NDJSON), deferred to a dedicated follow-up. All 596 backend tests green (unchanged count — copy path has no new tests yet; the V2 endpoint's behaviour is validated by the legacy test coverage through the shared-helper surface + the TelemetryPayloadParser tests from M5.b.1). Integration coverage lands with M7 test-VM smoke. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-21 10:04:11 +02:00
Oliver Kieselbach	f0fc3696f6	feat(backend): M5.b IngestTelemetry endpoint — POST /api/agent/telemetry ... V2 ingest endpoint consuming a heterogeneous batch of TelemetryItemDtos (Event + Signal + DecisionTransition in one JSON array). gzip request bodies are decoded transparently by the existing UseRequestDecompression middleware — new code only has to read a plain stream. - TelemetryItemDto (Shared): wire-format DTO mirroring the agent's TelemetryItem JSON shape so the backend can deserialise without linking agent assemblies. - IngestTelemetryFunction: security check (cert / rate / hardware whitelist) → device & version kill-switch short-circuits with DeviceBlocked=true body → body parse → tenant-header/body match → partition by Kind → persist via the appropriate repository → 200 with AdminAction + Actions control-signal fields populated from ISessionRepository (matching the /api/agent/ingest parity needed by the M4.6.ε agent-side UploadResult parser). - TelemetryPayloadParser: pure defensive parsing layer (wire → storage record). Malformed PayloadJson on a single item returns null; one poisonous item can't poison the batch. Projects index discriminators (IsTerminal, ClassifierVerdictId, ClassifierHypothesisLevel) off the DecisionTransition payload so the later queue-driven index fan-out reads them straight off the primary row. - EndpointAccessPolicyCatalog: POST agent/telemetry registered under DeviceOrBootstrapAuth (same policy as agent/ingest). Without this the fail-closed catalog rejects the route with 403. Scope boundary — Events get stored via StoreEventsBatchAsync only; the rule engine, app-install aggregation, SignalR notification, and vulnerability correlation remain on the legacy /api/agent/ingest path. Sharing that pipeline is tracked as M5.b.2 (extract EventIngestProcessor so both functions run the same downstream steps) — M7 rollout gates on completing it. Documented inline in PersistItemsAsync. 28 new tests: defensive null-handling per kind, typed-column extraction for Signal (with envelope fallback for missing SessionTraceOrdinal), DecisionTransition dead-end + classifier-verdict nesting, IsTerminal authoritative-list guard (Completed / Failed / WhiteGloveSealed / WhiteGloveCompletedPart2 — mirrors DecisionCore SessionStageExtensions), PartitionKey parser including malformed-shape rejection. The EndpointPolicyCatalogCompletenessTests guard the route registration. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-04-21 09:33:36 +02:00